autobuild-run: do reproducible builds tests if BR2_REPRODUCIBLE=y
PATCH 1/3 autobuild-run: do reproducible builds tests if BR2_REPRODUCIBLE=y
--- scripts/autobuild-run | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)
scripts/autobuild-run +9 -1
@@ -652,7 +652,15 @@ def run_instance(**kwargs): log_write(kwargs['log'], "WARN: failed to generate configuration") continue - ret = do_build(**kwargs) + # Check if the build test is supposed to be a reproducible test + outputdir = os.path.abspath(os.path.join(idir, "output")) + with open(os.path.join(outputdir, ".config"), "r") as fconf: + if "BR2_REPRODUCIBLE=y" in fconf.read(): + log_write(kwargs['log'], "INFO: Reproducible Build Test") + ret = do_reproducible_build(**kwargs) + else: + log_write(kwargs['log'], "INFO: Non Reproducible Build Test") + ret = do_build(**kwargs) send_results(ret, **kwargs)
Signature · 1 line
2.20.1
PATCH 2/3 autobuild-run: initial implementation of do_reproducible_build()
This new function will call do_build() twice to produce two builds and then check their reproducibility
--- scripts/autobuild-run | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
scripts/autobuild-run +38 -0
@@ -455,6 +455,44 @@ def do_build(**kwargs): log_write(log, "INFO: build successful") return 0 +def do_reproducible_build(**kwargs): + """Run the builds for reproducibility testing + + Build twice with the same configuration. Calls do_build() to + perform the actual build. + """ + + idir = "instance-%d" % kwargs['instance'] + outputdir = os.path.join(idir, "output") + f = open(os.path.join(outputdir, "logfile"), "w+") + log = kwargs['log'] + + # Start the first build + log_write(log, "INFO: Reproducible Build Test, starting build 1") + ret = do_build(**kwargs) + if ret != 0: + log_write(log, "INFO: build 1 failed, skipping build 2") + return ret + + # First build has been built, move files and start build 2 + os.rename(os.path.join(outputdir, "images"), os.path.join(outputdir, "images-1")) + + # Clean up build 1 + cmd = ["make", "O=%s" % outputdir, "clean"] \ + + kwargs['make_opts'].split() + subprocess.call(cmd, stdout=f, stderr=f) + + # Start the second build + log_write(log, "INFO: Reproducible Build Test, starting build 2") + ret = do_build(**kwargs) + if ret != 0: + log_write(log, "INFO: build 2 failed") + return ret + + # Assuming both have built successfully + ret = check_reproducibility() + return ret + def send_results(result, **kwargs): """Prepare and store/send tarball with results
Signature · 1 line
2.20.1
PATCH 2/3 autobuild-run: initial implementation of do_reproducible_build()
Quoted 27 lines · Atharva Lele
This new function will call
do_build()twice to produce two builds and then check their reproducibilitySigned-off-by Atharva Lele <[email protected]>--- scripts/autobuild-run | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)scripts/autobuild-run +11
-0@@ -455,6 +455,44 @@ def do_build(**kwargs): log_write(log, "INFO: build successful") return 0 +def do_reproducible_build(**kwargs): + """Run the builds for reproducibility testing + + Build twice with the same configuration. Calls do_build() to + perform the actual build. + """ + + idir = "instance-%d" % kwargs['instance'] + outputdir = os.path.join(idir, "output") + f = open(os.path.join(outputdir, "logfile"), "w+") + log = kwargs['log']
scripts/autobuild-run def do_build(**kwargs): · line 478
Quoted 10 lines
scripts/autobuild-run +10
-0+ + # Start the first build + log_write(log, "INFO: Reproducible Build Test, starting build 1") + ret = do_build(**kwargs) + if ret != 0: + log_write(log, "INFO: build 1 failed, skipping build 2") + return ret + + # First build has been built, move files and start build 2 + os.rename(os.path.join(outputdir, "images"), os.path.join(outputdir, "images-1"))
I would actually prefer to do the second build in a different output directory,
so we have at least two differences that can be checked (time and path).
It could be done my adding outputdir to kwargs and calculating it once in the
prepare step. Note that because kwargs is a dict, it is possible for the prepare_build function to update it and it will become visible to the other functions.
However, that can be done in a later patch. Or if you want to do it now, it
should be done as a first patch (moving outputdir into kwargs).
scripts/autobuild-run def do_build(**kwargs): · line 482
Quoted 4 lines
scripts/autobuild-run +4
-0+ + # Clean up build 1 + cmd = ["make", "O=%s" % outputdir, "clean"] \ + + kwargs['make_opts'].split()
Too much copying from do_build() :-) make_opts isn't needed here.
scripts/autobuild-run def do_build(**kwargs): · line 483
Quoted 1 line
scripts/autobuild-run +1
-0+ subprocess.call(cmd, stdout=f, stderr=f)
Better move the opening of the logfile here, and do it with with:. Yes,do_build doesn't do it that way, but it should :-). Actually, I'd prefer this to be redirected to devnull like is done in gen_config(): make clean doesn't generate interesting output.
Note that if the second build would be done in a different output dir, this
wouldn't be needed. It should be sufficient to copy the .config file to the new output dir and run do_build again. But then, of course, the second output dir should also be removed by prepare.
scripts/autobuild-run def do_build(**kwargs): · line 487
Quoted 3 lines
scripts/autobuild-run +3
-0+ # Start the second build + log_write(log, "INFO: Reproducible Build Test, starting build 2") + ret = do_build(**kwargs)
It probably isn't needed for the second build to repeat legal-info. But that
shouldn't make much of a difference anyway, so OK.
Regards,
Arnout
Quoted 12 lines
scripts/autobuild-run +8
-0+ if ret != 0: + log_write(log, "INFO: build 2 failed") + return ret + + # Assuming both have built successfully + ret = check_reproducibility() + return ret + def send_results(result, **kwargs): """Prepare and store/send tarball with results
PATCH 3/3 autobuild-run: initial implementation of check_reproducibility()
In reply to Atharva LeleFind out differences in built images, if any
--- scripts/autobuild-run | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)
scripts/autobuild-run +38 -0
@@ -394,6 +394,44 @@ def stop_on_build_hang(monitor_thread_hung_build_flag, break monitor_thread_stop_flag.wait(30) +def check_reproducibility(**kwargs): + """Check reproducibility of builds + + Use diffoscope on the built images, if diffoscope is not + installed, fallback to cmp + """ + + log = kwargs['log'] + idir = "instance-%d" % kwargs['instance'] + outputdir = os.path.join(idir, "output") + diffoscope_output = os.path.join(outputdir, "diffoscope-results.txt") + # Using only tar images for now + build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar") + build_2_image = os.path.join(outputdir, "images", "rootfs.tar") + + # Prefix to point diffoscope towards cross-tools + prefix = subprocess.check_output(["make", "printvars", "VARS=TARGET_CROSS"]) + # Remove TARGET_CROSS= and \n from the string + prefix = prefix[13:-1] + + try: + log_write(log, "INFO: running diffoscope on images") + ret = subprocess.call(["diffoscope", build_1_image, build_2_image, + "--tool-prefix-binutils", prefix, + "--text", diffoscope_output]) + except OSError as e: + if e.errno == errno.ENOENT: + log_write(log, "INFO: diffoscope not installed, falling back to cmp") + ret = subprocess.call(["cmp", "build_1_image", "build_2_image", "-b", ">", diffoscope_output]) + + if ret != 0: + log_write(log, "INFO: Build is non-reproducible.") + return -1 + + # rootfs images match byte-for-byte -> reproducible image + log_write(log, "INFO: Build is reproducible!") + return 0 + def do_build(**kwargs): """Run the build itself"""
Signature · 1 line
2.20.1
PATCH 3/3 autobuild-run: initial implementation of check_reproducibility()
Hi Atharva,
Quoted 16 lines · Atharva Lele
Find out differences in built images, if any
Signed-off-by Atharva Lele <[email protected]>--- scripts/autobuild-run | 38 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+)scripts/autobuild-run +1
-0@@ -394,6 +394,44 @@ def stop_on_build_hang(monitor_thread_hung_build_flag, break monitor_thread_stop_flag.wait(30) +def check_reproducibility(**kwargs):
scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 407
Quoted 10 lines
scripts/autobuild-run +10
-0+ """Check reproducibility of builds + + Use diffoscope on the built images, if diffoscope is not + installed, fallback to cmp + """ + + log = kwargs['log'] + idir = "instance-%d" % kwargs['instance'] + outputdir = os.path.join(idir, "output") + diffoscope_output = os.path.join(outputdir, "diffoscope-results.txt")
Put it directly in the results directory, i.e.os.path.join(outputdir, "results", "diffoscope.txt")
That way, it becomes part of the results tarball and will get uploaded to the server. And it may even appear on the website without any further change.
scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 410
Quoted 3 lines
scripts/autobuild-run +3
-0+ # Using only tar images for now + build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar") + build_2_image = os.path.join(outputdir, "images", "rootfs.tar")
How can we be sure that a tarball is generated?
scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 413
Quoted 3 lines
scripts/autobuild-run +3
-0+ + # Prefix to point diffoscope towards cross-tools + prefix = subprocess.check_output(["make", "printvars", "VARS=TARGET_CROSS"])
Don't you need to pass the -C and O= options here?
scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 421
Quoted 8 lines
scripts/autobuild-run +8
-0+ # Remove TARGET_CROSS= and \n from the string + prefix = prefix[13:-1] + + try: + log_write(log, "INFO: running diffoscope on images") + ret = subprocess.call(["diffoscope", build_1_image, build_2_image, + "--tool-prefix-binutils", prefix, + "--text", diffoscope_output])
Use kwargs['sysinfo'].has("diffoscope") to check availability, rather than
handling the exception. Though Yann may havea different opinion on that.
scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 429
Quoted 8 lines
scripts/autobuild-run +8
-0+ except OSError as e: + if e.errno == errno.ENOENT: + log_write(log, "INFO: diffoscope not installed, falling back to cmp") + ret = subprocess.call(["cmp", "build_1_image", "build_2_image", "-b", ">", diffoscope_output]) + + if ret != 0: + log_write(log, "INFO: Build is non-reproducible.") + return -1
Return -3 here, and handle -3 in send_results. For the time being, it should
still be NOK, but later the web frontend can be updated to also accept NREP or something like that as a result.
scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 432
Quoted 3 lines
scripts/autobuild-run +3
-0+ + # rootfs images match byte-for-byte -> reproducible image + log_write(log, "INFO: Build is reproducible!")
We don't have a "Success" log message for the normal build, so no need to add
one for reproducible I guess.
Regards,
Arnout
Quoted 6 lines
scripts/autobuild-run +2
-0+ return 0 + def do_build(**kwargs): """Run the build itself"""
PATCH 1/3 autobuild-run: do reproducible builds tests if BR2_REPRODUCIBLE=y
In reply to Atharva LeleHi Atharva,
Quoted 10 lines · Atharva Lele
Signed-off-by Atharva Lele <[email protected]>--- scripts/autobuild-run | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-)scripts/autobuild-run +0
-0
scripts/autobuild-run def run_instance(**kwargs): · line 656
Quoted 6 lines
scripts/autobuild-run +2
-1@@ -652,7 +652,15 @@ def run_instance(**kwargs): log_write(kwargs['log'], "WARN: failed to generate configuration") continue - ret = do_build(**kwargs) + # Check if the build test is supposed to be a reproducible test + outputdir = os.path.abspath(os.path.join(idir, "output"))
These path names get constructed over and over again in a few different places.
I think it would be worth to create a Builder class that has those common things as fields, and that gets (almost?) all the preceding functions as members.
However, that's a completely different refactoring, not really a blocking point
for merging this.
scripts/autobuild-run def run_instance(**kwargs): · line 658
Quoted 2 lines
scripts/autobuild-run +2
-0+ with open(os.path.join(outputdir, ".config"), "r") as fconf: + if "BR2_REPRODUCIBLE=y" in fconf.read():
I would add a \n to the match string, but that's a minor nit of course.
scripts/autobuild-run def run_instance(**kwargs): · line 660
Quoted 2 lines
scripts/autobuild-run +2
-0+ log_write(kwargs['log'], "INFO: Reproducible Build Test") + ret = do_reproducible_build(**kwargs)
The order of your patches is wrong: this patch calls a function that is only
defined by patch 2, so patch 2 should be first. But that one again calls a function that is defined by patch 3, so patch 3 should be first.
However, I don't think it's worth splitting this in three patches. The other
two just add two new functions that don't get called, so there's not much reason to keep it separate.
Regards,
Arnout
Quoted 7 lines
scripts/autobuild-run +3
-0+ else: + log_write(kwargs['log'], "INFO: Non Reproducible Build Test") + ret = do_build(**kwargs) send_results(ret, **kwargs)