Skip to contentSkip to conversation

autobuild-run: do reproducible builds tests if BR2_REPRODUCIBLE=y

PATCH New Atharva Lele 2 participants 6 messages Jun 3, 2019 – Jun 4, 2019

lore
Share & tools
lore
#0 lore raw

PATCH 1/3 autobuild-run: do reproducible builds tests if BR2_REPRODUCIBLE=y

---
 scripts/autobuild-run | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
scripts/autobuild-run +9 -1
index ef2f2a5..9f8aabb 100755
@@ -652,7 +652,15 @@ def run_instance(**kwargs):
             log_write(kwargs['log'], "WARN: failed to generate configuration")
             continue
 
-        ret = do_build(**kwargs)
+        # Check if the build test is supposed to be a reproducible test
+        outputdir = os.path.abspath(os.path.join(idir, "output"))
+        with open(os.path.join(outputdir, ".config"), "r") as fconf:
+            if "BR2_REPRODUCIBLE=y" in fconf.read():
+                log_write(kwargs['log'], "INFO: Reproducible Build Test")
+                ret = do_reproducible_build(**kwargs)
+            else:
+                log_write(kwargs['log'], "INFO: Non Reproducible Build Test")
+                ret = do_build(**kwargs)
 
         send_results(ret, **kwargs)
 
Signature · 1 line
2.20.1
#1 lore raw

PATCH 2/3 autobuild-run: initial implementation of do_reproducible_build()

This new function will call do_build() twice to produce two builds and then check their reproducibility

---
 scripts/autobuild-run | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
scripts/autobuild-run +38 -0
index 9f8aabb..329803a 100755
@@ -455,6 +455,44 @@ def do_build(**kwargs):
     log_write(log, "INFO: build successful")
     return 0
 
+def do_reproducible_build(**kwargs):
+    """Run the builds for reproducibility testing
+
+    Build twice with the same configuration. Calls do_build() to
+    perform the actual build.
+    """
+
+    idir = "instance-%d" % kwargs['instance']
+    outputdir = os.path.join(idir, "output")
+    f = open(os.path.join(outputdir, "logfile"), "w+")
+    log = kwargs['log']
+    
+    # Start the first build
+    log_write(log, "INFO: Reproducible Build Test, starting build 1")
+    ret = do_build(**kwargs)
+    if ret != 0:
+        log_write(log, "INFO: build 1 failed, skipping build 2")
+        return ret
+    
+    # First build has been built, move files and start build 2
+    os.rename(os.path.join(outputdir, "images"), os.path.join(outputdir, "images-1"))
+
+    # Clean up build 1
+    cmd = ["make", "O=%s" % outputdir, "clean"] \
+          + kwargs['make_opts'].split()
+    subprocess.call(cmd, stdout=f, stderr=f)
+
+    # Start the second build
+    log_write(log, "INFO: Reproducible Build Test, starting build 2")
+    ret = do_build(**kwargs)
+    if ret != 0:
+        log_write(log, "INFO: build 2 failed")
+        return ret
+
+    # Assuming both have built successfully
+    ret = check_reproducibility()
+    return ret
+
 def send_results(result, **kwargs):
     """Prepare and store/send tarball with results
 
Signature · 1 line
2.20.1
#2 lore raw

PATCH 2/3 autobuild-run: initial implementation of do_reproducible_build()

Quoted 27 lines · Atharva Lele

This new function will call do_build() twice to produce two builds and then check their reproducibility

---
 scripts/autobuild-run | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
scripts/autobuild-run +11 -0
index 9f8aabb..329803a 100755
@@ -455,6 +455,44 @@ def do_build(**kwargs):
     log_write(log, "INFO: build successful")
     return 0
 
+def do_reproducible_build(**kwargs):
+    """Run the builds for reproducibility testing
+
+    Build twice with the same configuration. Calls do_build() to
+    perform the actual build.
+    """
+
+    idir = "instance-%d" % kwargs['instance']
+    outputdir = os.path.join(idir, "output")
+    f = open(os.path.join(outputdir, "logfile"), "w+")
+    log = kwargs['log']

scripts/autobuild-run def do_build(**kwargs): · line 478

Quoted 10 lines
scripts/autobuild-run +10 -0
+    
+    # Start the first build
+    log_write(log, "INFO: Reproducible Build Test, starting build 1")
+    ret = do_build(**kwargs)
+    if ret != 0:
+        log_write(log, "INFO: build 1 failed, skipping build 2")
+        return ret
+    
+    # First build has been built, move files and start build 2
+    os.rename(os.path.join(outputdir, "images"), os.path.join(outputdir, "images-1"))

I would actually prefer to do the second build in a different output directory,
so we have at least two differences that can be checked (time and path).

It could be done my adding outputdir to kwargs and calculating it once in the
prepare step. Note that because kwargs is a dict, it is possible for the prepare_build function to update it and it will become visible to the other functions.

However, that can be done in a later patch. Or if you want to do it now, it
should be done as a first patch (moving outputdir into kwargs).

scripts/autobuild-run def do_build(**kwargs): · line 482

Quoted 4 lines
scripts/autobuild-run +4 -0
+
+    # Clean up build 1
+    cmd = ["make", "O=%s" % outputdir, "clean"] \
+          + kwargs['make_opts'].split()

Too much copying from do_build() :-) make_opts isn't needed here.

scripts/autobuild-run def do_build(**kwargs): · line 483

Quoted 1 line
scripts/autobuild-run +1 -0
+    subprocess.call(cmd, stdout=f, stderr=f)

Better move the opening of the logfile here, and do it with with:. Yes,
do_build doesn't do it that way, but it should :-). Actually, I'd prefer this to be redirected to devnull like is done in gen_config(): make clean doesn't generate interesting output.

Note that if the second build would be done in a different output dir, this
wouldn't be needed. It should be sufficient to copy the .config file to the new output dir and run do_build again. But then, of course, the second output dir should also be removed by prepare.

scripts/autobuild-run def do_build(**kwargs): · line 487

Quoted 3 lines
scripts/autobuild-run +3 -0
+    # Start the second build
+    log_write(log, "INFO: Reproducible Build Test, starting build 2")
+    ret = do_build(**kwargs)

It probably isn't needed for the second build to repeat legal-info. But that
shouldn't make much of a difference anyway, so OK.

Regards,
Arnout

Quoted 12 lines
scripts/autobuild-run +8 -0
+    if ret != 0:
+        log_write(log, "INFO: build 2 failed")
+        return ret
+
+    # Assuming both have built successfully
+    ret = check_reproducibility()
+    return ret
+
 def send_results(result, **kwargs):
     """Prepare and store/send tarball with results
 
#3 lore raw

PATCH 3/3 autobuild-run: initial implementation of check_reproducibility()

In reply to Atharva Lele

Find out differences in built images, if any

---
 scripts/autobuild-run | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
scripts/autobuild-run +38 -0
index 329803a..0e24211 100755
@@ -394,6 +394,44 @@ def stop_on_build_hang(monitor_thread_hung_build_flag,
                 break
         monitor_thread_stop_flag.wait(30)
 
+def check_reproducibility(**kwargs):
+    """Check reproducibility of builds
+
+    Use diffoscope on the built images, if diffoscope is not
+    installed, fallback to cmp
+    """
+
+    log = kwargs['log']
+    idir = "instance-%d" % kwargs['instance']
+    outputdir = os.path.join(idir, "output")
+    diffoscope_output = os.path.join(outputdir, "diffoscope-results.txt")
+    # Using only tar images for now
+    build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar")
+    build_2_image = os.path.join(outputdir, "images", "rootfs.tar")
+
+    # Prefix to point diffoscope towards cross-tools
+    prefix = subprocess.check_output(["make", "printvars", "VARS=TARGET_CROSS"])
+    # Remove TARGET_CROSS= and \n from the string
+    prefix = prefix[13:-1]
+
+    try:
+        log_write(log, "INFO: running diffoscope on images")
+        ret = subprocess.call(["diffoscope", build_1_image, build_2_image,
+                            "--tool-prefix-binutils", prefix,
+                            "--text", diffoscope_output])
+    except OSError as e:
+        if e.errno == errno.ENOENT:
+            log_write(log, "INFO: diffoscope not installed, falling back to cmp")
+        ret = subprocess.call(["cmp", "build_1_image", "build_2_image", "-b", ">", diffoscope_output])
+    
+    if ret != 0:
+        log_write(log, "INFO: Build is non-reproducible.")
+        return -1
+    
+    # rootfs images match byte-for-byte -> reproducible image
+    log_write(log, "INFO: Build is reproducible!")
+    return 0
+
 def do_build(**kwargs):
     """Run the build itself"""
 
Signature · 1 line
2.20.1
#4 lore raw

PATCH 3/3 autobuild-run: initial implementation of check_reproducibility()

Hi Atharva,

Quoted 16 lines · Atharva Lele

Find out differences in built images, if any

---
 scripts/autobuild-run | 38 ++++++++++++++++++++++++++++++++++++++
 1 file changed, 38 insertions(+)
scripts/autobuild-run +1 -0
index 329803a..0e24211 100755
@@ -394,6 +394,44 @@ def stop_on_build_hang(monitor_thread_hung_build_flag,
                 break
         monitor_thread_stop_flag.wait(30)
 
+def check_reproducibility(**kwargs):

scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 407

Quoted 10 lines
scripts/autobuild-run +10 -0
+    """Check reproducibility of builds
+
+    Use diffoscope on the built images, if diffoscope is not
+    installed, fallback to cmp
+    """
+
+    log = kwargs['log']
+    idir = "instance-%d" % kwargs['instance']
+    outputdir = os.path.join(idir, "output")
+    diffoscope_output = os.path.join(outputdir, "diffoscope-results.txt")

Put it directly in the results directory, i.e.
os.path.join(outputdir, "results", "diffoscope.txt")
That way, it becomes part of the results tarball and will get uploaded to the server. And it may even appear on the website without any further change.

scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 410

Quoted 3 lines
scripts/autobuild-run +3 -0
+    # Using only tar images for now
+    build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar")
+    build_2_image = os.path.join(outputdir, "images", "rootfs.tar")

How can we be sure that a tarball is generated?

scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 413

Quoted 3 lines
scripts/autobuild-run +3 -0
+
+    # Prefix to point diffoscope towards cross-tools
+    prefix = subprocess.check_output(["make", "printvars", "VARS=TARGET_CROSS"])

Don't you need to pass the -C and O= options here?

scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 421

Quoted 8 lines
scripts/autobuild-run +8 -0
+    # Remove TARGET_CROSS= and \n from the string
+    prefix = prefix[13:-1]
+
+    try:
+        log_write(log, "INFO: running diffoscope on images")
+        ret = subprocess.call(["diffoscope", build_1_image, build_2_image,
+                            "--tool-prefix-binutils", prefix,
+                            "--text", diffoscope_output])

Use kwargs['sysinfo'].has("diffoscope") to check availability, rather than
handling the exception. Though Yann may havea different opinion on that.

scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 429

Quoted 8 lines
scripts/autobuild-run +8 -0
+    except OSError as e:
+        if e.errno == errno.ENOENT:
+            log_write(log, "INFO: diffoscope not installed, falling back to cmp")
+        ret = subprocess.call(["cmp", "build_1_image", "build_2_image", "-b", ">", diffoscope_output])
+    
+    if ret != 0:
+        log_write(log, "INFO: Build is non-reproducible.")
+        return -1

Return -3 here, and handle -3 in send_results. For the time being, it should
still be NOK, but later the web frontend can be updated to also accept NREP or something like that as a result.

scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 432

Quoted 3 lines
scripts/autobuild-run +3 -0
+    
+    # rootfs images match byte-for-byte -> reproducible image
+    log_write(log, "INFO: Build is reproducible!")

We don't have a "Success" log message for the normal build, so no need to add
one for reproducible I guess.

Regards,
Arnout

Quoted 6 lines
scripts/autobuild-run +2 -0
+    return 0
+
 def do_build(**kwargs):
     """Run the build itself"""
 
#5 lore raw

PATCH 1/3 autobuild-run: do reproducible builds tests if BR2_REPRODUCIBLE=y

In reply to Atharva Lele

Hi Atharva,

Quoted 10 lines · Atharva Lele
---
 scripts/autobuild-run | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
scripts/autobuild-run +0 -0
index ef2f2a5..9f8aabb 100755

scripts/autobuild-run def run_instance(**kwargs): · line 656

Quoted 6 lines
scripts/autobuild-run +2 -1
@@ -652,7 +652,15 @@ def run_instance(**kwargs):
             log_write(kwargs['log'], "WARN: failed to generate configuration")
             continue
 
-        ret = do_build(**kwargs)
+        # Check if the build test is supposed to be a reproducible test
+        outputdir = os.path.abspath(os.path.join(idir, "output"))

These path names get constructed over and over again in a few different places.
I think it would be worth to create a Builder class that has those common things as fields, and that gets (almost?) all the preceding functions as members.

However, that's a completely different refactoring, not really a blocking point
for merging this.

scripts/autobuild-run def run_instance(**kwargs): · line 658

Quoted 2 lines
scripts/autobuild-run +2 -0
+        with open(os.path.join(outputdir, ".config"), "r") as fconf:
+            if "BR2_REPRODUCIBLE=y" in fconf.read():

I would add a \n to the match string, but that's a minor nit of course.

scripts/autobuild-run def run_instance(**kwargs): · line 660

Quoted 2 lines
scripts/autobuild-run +2 -0
+                log_write(kwargs['log'], "INFO: Reproducible Build Test")
+                ret = do_reproducible_build(**kwargs)

The order of your patches is wrong: this patch calls a function that is only
defined by patch 2, so patch 2 should be first. But that one again calls a function that is defined by patch 3, so patch 3 should be first.

However, I don't think it's worth splitting this in three patches. The other
two just add two new functions that don't get called, so there's not much reason to keep it separate.

Regards,
Arnout

Quoted 7 lines
scripts/autobuild-run +3 -0
+            else:
+                log_write(kwargs['log'], "INFO: Non Reproducible Build Test")
+                ret = do_build(**kwargs)
 
         send_results(ret, **kwargs)