Skip to contentSkip to conversation

autobuild-run: initial implementation of check_reproducibility()

PATCH New Atharva Lele 3 participants 12 messages Jun 11, 2019 – Jun 14, 2019

lore
Share & tools
lore
#0 lore raw

PATCH v3 1/3 autobuild-run: initial implementation of check_reproducibility()

For reproducible builds, we want to find out if there are any differences between two builds having the same configuration, and to find the reason behind them. The diffoscope tool looks inside different types of files to see where the differences lie.

check_reproducibility() runs diffoscope on two output directories which are expected in output/images and output/images-1. Since it uses objdump, it needs to be provided the cross-compile prefix which is derived from the TARGET_CROSS variable.

Since diffoscope may not be installed, we fall back to cmp for byte-by-byte comparison. We add diffoscope to list of optional programs to avoid repeated checking of its presence.

---
Changes v2 -> v3:
- Use file size of reproducible_results to check reproducibility status, rather than exit status of cmp or diff (suggested by arnout)
- Rename results file (diffoscope_output.txt -> reproducible_results) to avoid confusion
- Change handling of diffoscope output text file using with open()
- Changed commit message to have all necessary info (suggested by arnout)
- Removed leftover code from when I was using an exception to handle diffoscope presense (thanks to arnout)

Changes v1 -> v2:
- move diffoscope output to results dir (suggested by arnout)
- fix make printvars call
- Add diffoscope to DEFAULT_OPTIONAL_PROGS

---
 scripts/autobuild-run | 39 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)
scripts/autobuild-run +38 -1
index 190a254..ba5b337 100755
@@ -204,7 +204,7 @@ def get_branch():
 
 class SystemInfo:
     DEFAULT_NEEDED_PROGS = ["make", "git", "gcc"]
-    DEFAULT_OPTIONAL_PROGS = ["bzr", "java", "javac", "jar"]
+    DEFAULT_OPTIONAL_PROGS = ["bzr", "diffoscope", "java", "javac", "jar"]
 
     def __init__(self):
         self.needed_progs = list(self.__class__.DEFAULT_NEEDED_PROGS)
@@ -394,6 +394,43 @@ def stop_on_build_hang(monitor_thread_hung_build_flag,
                 break
         monitor_thread_stop_flag.wait(30)
 
+def check_reproducibility(**kwargs):
+    """Check reproducibility of builds
+
+    Use diffoscope on the built images, if diffoscope is not
+    installed, fallback to cmp
+    """
+
+    log = kwargs['log']
+    idir = "instance-%d" % kwargs['instance']
+    outputdir = os.path.join(idir, "output")
+    srcdir = os.path.join(idir, "buildroot")
+    reproducible_results = os.path.join(outputdir, "results", "reproducible_results")
+    # Using only tar images for now
+    build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar")
+    build_2_image = os.path.join(outputdir, "images", "rootfs.tar")
+
+    with open(reproducible_results, 'w') as diff:
+        if kwargs['sysinfo'].has("diffoscope"):
+            # Prefix to point diffoscope towards cross-tools
+            prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", srcdir, "printvars", "VARS=TARGET_CROSS"])
+            # Remove TARGET_CROSS= and \n from the string
+            prefix = prefix[13:-1]
+            log_write(log, "INFO: running diffoscope on images")
+            subprocess.call(["diffoscope", build_1_image, build_2_image,
+                                "--tool-prefix-binutils", prefix], stdout=diff, stderr=log)
+        else:
+            log_write(log, "INFO: diffoscope not installed, falling back to cmp")
+            subprocess.call(["cmp", "-b", "build_1_image", "build_2_image"], stdout=diff, stderr=log)
+    
+    if os.stat(reproducible_results).st_size > 0:
+        log_write(log, "INFO: Build is non-reproducible.")
+        return -1
+    
+    # rootfs images match byte-for-byte -> reproducible image
+    log_write(log, "INFO: Build is reproducible!")
+    return 0
+
 def do_build(**kwargs):
     """Run the build itself"""
 
Signature · 1 line
2.20.1
#1 lore raw

PATCH v3 2/3 autobuild-run: initial implementation of do_reproducible_build()

This new function will call do_build() twice to produce two builds and then check their reproducibility.

---
Changes v2 -> v3:
- make clean outputs to logfile, similar to other make calls (suggested by y_morin)
- Output dir needs abspath (suggested by y_morin)
- Directly use make clean as arguments rather than putting it into a variable
- Added missing period at end of commit message (suggested by arnout)

Changes v1 -> v2:
- make clean outputs to devnull (suggested by arnout)
- remove unnecessary arguments to make clean (suggested by arnout)

---
 scripts/autobuild-run | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
scripts/autobuild-run +37 -0
index ba5b337..5ba6092 100755
@@ -492,6 +492,43 @@ def do_build(**kwargs):
     log_write(log, "INFO: build successful")
     return 0
 
+def do_reproducible_build(**kwargs):
+    """Run the builds for reproducibility testing
+
+    Build twice with the same configuration. Calls do_build() to
+    perform the actual build.
+    """
+
+    idir = "instance-%d" % kwargs['instance']
+    outputdir = os.path.abspath(os.path.join(idir, "output"))
+    srcdir = os.path.join(idir, "buildroot")
+    log = kwargs['log']
+    
+    # Start the first build
+    log_write(log, "INFO: Reproducible Build Test, starting build 1")
+    ret = do_build(**kwargs)
+    if ret != 0:
+        log_write(log, "INFO: build 1 failed, skipping build 2")
+        return ret
+    
+    # First build has been built, move files and start build 2
+    os.rename(os.path.join(outputdir, "images"), os.path.join(outputdir, "images-1"))
+
+    # Clean up build 1
+    f = open(os.path.join(outputdir, "logfile"), "w+")
+    subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "clean"], stdout=f, stderr=f)
+
+    # Start the second build
+    log_write(log, "INFO: Reproducible Build Test, starting build 2")
+    ret = do_build(**kwargs)
+    if ret != 0:
+        log_write(log, "INFO: build 2 failed")
+        return ret
+
+    # Assuming both have built successfully
+    ret = check_reproducibility(**kwargs)
+    return ret
+
 def send_results(result, **kwargs):
     """Prepare and store/send tarball with results
 
Signature · 1 line
2.20.1
#2 lore raw

PATCH v3 2/3 autobuild-run: initial implementation of do_reproducible_build()

Atharva, All,

On 2019-06-11 18:04 +0530, Atharva Lele spake thusly:

Quoted 5 lines

This new function will call do_build() twice to produce two builds and then check their reproducibility.

Yet, a few comments, see below...

Quoted 26 lines

---
Changes v2 -> v3:
- make clean outputs to logfile, similar to other make calls (suggested by y_morin)
- Output dir needs abspath (suggested by y_morin)
- Directly use make clean as arguments rather than putting it into a variable
- Added missing period at end of commit message (suggested by arnout)

Changes v1 -> v2:
- make clean outputs to devnull (suggested by arnout)
- remove unnecessary arguments to make clean (suggested by arnout)

---
 scripts/autobuild-run | 37 +++++++++++++++++++++++++++++++++++++
 1 file changed, 37 insertions(+)
scripts/autobuild-run +2 -0
index ba5b337..5ba6092 100755
@@ -492,6 +492,43 @@ def do_build(**kwargs):
     log_write(log, "INFO: build successful")
     return 0
 
+def do_reproducible_build(**kwargs):
+    """Run the builds for reproducibility testing

scripts/autobuild-run def do_build(**kwargs): · line 506

Quoted 10 lines
scripts/autobuild-run +10 -0
+
+    Build twice with the same configuration. Calls do_build() to
+    perform the actual build.
+    """
+
+    idir = "instance-%d" % kwargs['instance']
+    outputdir = os.path.abspath(os.path.join(idir, "output"))
+    srcdir = os.path.join(idir, "buildroot")
+    log = kwargs['log']
+    

Empty line made of spaces.

scripts/autobuild-run def do_build(**kwargs): · line 513

Quoted 7 lines
scripts/autobuild-run +7 -0
+    # Start the first build
+    log_write(log, "INFO: Reproducible Build Test, starting build 1")
+    ret = do_build(**kwargs)
+    if ret != 0:
+        log_write(log, "INFO: build 1 failed, skipping build 2")
+        return ret
+    

Ditto.

scripts/autobuild-run def do_build(**kwargs): · line 519

Quoted 6 lines
scripts/autobuild-run +6 -0
+    # First build has been built, move files and start build 2
+    os.rename(os.path.join(outputdir, "images"), os.path.join(outputdir, "images-1"))
+
+    # Clean up build 1
+    f = open(os.path.join(outputdir, "logfile"), "w+")
+    subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "clean"], stdout=f, stderr=f)

Since we do create a small Makefile wrapper in $(O), you could simplify this a little with:

    subprocess.call(["make", "-C", outputdir, "clean"], stdout=f, stderr=f)

Ditto in the existing do_build() function. Can be improved upon when you introduce the Builder class.

Regards,
Yann E. MORIN.

Quoted 17 lines
scripts/autobuild-run +11 -0
+    # Start the second build
+    log_write(log, "INFO: Reproducible Build Test, starting build 2")
+    ret = do_build(**kwargs)
+    if ret != 0:
+        log_write(log, "INFO: build 2 failed")
+        return ret
+
+    # Assuming both have built successfully
+    ret = check_reproducibility(**kwargs)
+    return ret
+
 def send_results(result, **kwargs):
     """Prepare and store/send tarball with results
 
Signature · 1 line
2.20.1
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#3 lore raw

PATCH v3 2/3 autobuild-run: initial implementation of do_reproducible_build()

Quoted 3 lines · Yann E. MORIN

Atharva, All,

On 2019-06-11 18:04 +0530, Atharva Lele spake thusly:

[snip]

Quoted 11 lines

scripts/autobuild-run def do_build(**kwargs): · line 519

Quoted 3 lines
scripts/autobuild-run +3 -0
+    # Clean up build 1
+    f = open(os.path.join(outputdir, "logfile"), "w+")
+    subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir, "clean"], stdout=f, stderr=f)

Since we do create a small Makefile wrapper in $(O), you could simplify this a little with:

    subprocess.call(["make", "-C", outputdir, "clean"], stdout=f, stderr=f)

Ditto in the existing do_build() function. Can be improved upon when you introduce the Builder class.

This is bikeshedding, obviously, but I don't like that proposal. I think an
explicit O= -C is easier to read and understand.

Shorter is not better :-)

Regards,
Arnout

#4 lore raw

PATCH v3 2/3 autobuild-run: initial implementation of do_reproducible_build()

Quoted 1 line

I think an explicit O= -C is easier to read and understand.

I do agree with this.

Regards,
Atharva Lele

Quoted 21 lines · Arnout Vandecappelle
Quoted 3 lines · Yann E. MORIN

Atharva, All,

On 2019-06-11 18:04 +0530, Atharva Lele spake thusly:

[snip]

Quoted 3 lines

scripts/autobuild-run def do_build(**kwargs):

Quoted 3 lines
scripts/autobuild-run +3 -0
+    # Clean up build 1
+    f = open(os.path.join(outputdir, "logfile"), "w+")
+    subprocess.call(["make", "O=%s" % outputdir, "-C", srcdir,

"clean"], stdout=f, stderr=f)

Quoted 5 lines

Since we do create a small Makefile wrapper in $(O), you could simplify this a little with:

    subprocess.call(["make", "-C", outputdir, "clean"], stdout=f,

stderr=f)

Quoted 3 lines

Ditto in the existing do_build() function. Can be improved upon when you introduce the Builder class.

Quoted 10 lines

This is bikeshedding, obviously, but I don't like that proposal. I think
an
explicit O= -C is easier to read and understand.

Shorter is not better :-)

Regards,
Arnout

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190614/259a3597/attachment.html>

#5 lore raw

PATCH v3 3/3 autobuild-run: do reproducible builds tests if BR2_REPRODUCIBLE=y

In reply to Atharva Lele

---
Changes v2 -> v3:
- use reproducible flag so that .config i snot kept open for the entire build duration. (suggested by arnout)
- remove reduntant logging (suggested by arnout)

Changes v1 -> v2:
- add trailing newline character to BR2_REPRODUCIBLE=y (suggested by arnout)

---
 scripts/autobuild-run | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
scripts/autobuild-run +8 -1
index 5ba6092..4fbd507 100755
@@ -727,7 +727,14 @@ def run_instance(**kwargs):
             log_write(kwargs['log'], "WARN: failed to generate configuration")
             continue
 
-        ret = do_build(**kwargs)
+        # Check if the build test is supposed to be a reproducible test
+        outputdir = os.path.abspath(os.path.join(idir, "output"))
+        with open(os.path.join(outputdir, ".config"), "r") as fconf:
+            reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read()
+            if reproducible:
+                ret = do_reproducible_build(**kwargs)
+            else:
+                ret = do_build(**kwargs)
 
         send_results(ret, **kwargs)
 
Signature · 1 line
2.20.1
#6 lore raw

PATCH v3 3/3 autobuild-run: do reproducible builds tests if BR2_REPRODUCIBLE=y

On 2019-06-11 18:04 +0530, Atharva Lele spake thusly:

Quoted 7 lines

---
Changes v2 -> v3:
- use reproducible flag so that .config i snot kept open for the entire build duration. (suggested by arnout)

Sorry, but that is still the case, see below...

Quoted 17 lines

- remove reduntant logging (suggested by arnout)

Changes v1 -> v2:
- add trailing newline character to BR2_REPRODUCIBLE=y (suggested by arnout)

---
 scripts/autobuild-run | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
scripts/autobuild-run +0 -0
index 5ba6092..4fbd507 100755
@@ -727,7 +727,14 @@ def run_instance(**kwargs):
             log_write(kwargs['log'], "WARN: failed to generate configuration")
             continue

scripts/autobuild-run def run_instance(**kwargs): · line 737

Quoted 10 lines
scripts/autobuild-run +8 -1
 
-        ret = do_build(**kwargs)
+        # Check if the build test is supposed to be a reproducible test
+        outputdir = os.path.abspath(os.path.join(idir, "output"))
+        with open(os.path.join(outputdir, ".config"), "r") as fconf:
+            reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read()
+            if reproducible:
+                ret = do_reproducible_build(**kwargs)
+            else:
+                ret = do_build(**kwargs)

This whole if reproducible block is still indented inside the with block, so you'r still having .config opened during the whole build.

You need to do (basically):

with open(os.path.join(outputdir, ".config"), "r") as fconf:
reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read()
build = do_reproducible_build if reproducible else do_build ret = build(**kwargs)

Regards,
Yann E. MORIN.

Quoted 6 lines
scripts/autobuild-run +0 -0
         send_results(ret, **kwargs)
 
Signature · 1 line
2.20.1
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#7 lore raw

PATCH v3 3/3 autobuild-run: do reproducible builds tests if BR2_REPRODUCIBLE=y

Quoted 6 lines

You need to do (basically):

with open(os.path.join(outputdir, ".config"), "r") as fconf:
reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read()
build = do_reproducible_build if reproducible else do_build ret = build(**kwargs)

It would still work if I just unindent the if-else block too right? This is what I actually intended to do
It's working for me in testing..
Like this:

with open(os.path.join(outputdir, ".config"), "r") as fconf:
reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read()
if reproducible:
ret = do_reproducible_build(**kwargs)
else:
ret = do_build(**kwargs)

Regards,
Atharva Lele

Quoted 73 lines · Yann E. MORIN

On 2019-06-11 18:04 +0530, Atharva Lele spake thusly:

Quoted 7 lines

---
Changes v2 -> v3:
- use reproducible flag so that .config i snot kept open for the entire build duration. (suggested by arnout)

Sorry, but that is still the case, see below...

Quoted 4 lines

- remove reduntant logging (suggested by arnout)

Changes v1 -> v2:
- add trailing newline character to BR2_REPRODUCIBLE=y (suggested by

arnout)

scripts/autobuild-run def run_instance(**kwargs): · line 727

Quoted 12 lines
---
 scripts/autobuild-run | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)
scripts/autobuild-run +0 -0
index 5ba6092..4fbd507 100755
@@ -727,7 +727,14 @@ def run_instance(**kwargs):
             log_write(kwargs['log'], "WARN: failed to generate

configuration")

scripts/autobuild-run def run_instance(**kwargs): · line 737

Quoted 11 lines
scripts/autobuild-run +8 -1
             continue
 
-        ret = do_build(**kwargs)
+        # Check if the build test is supposed to be a reproducible test
+        outputdir = os.path.abspath(os.path.join(idir, "output"))
+        with open(os.path.join(outputdir, ".config"), "r") as fconf:
+            reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read()
+            if reproducible:
+                ret = do_reproducible_build(**kwargs)
+            else:
+                ret = do_build(**kwargs)

This whole if reproducible block is still indented inside the with block, so you'r still having .config opened during the whole build.

You need to do (basically):

with open(os.path.join(outputdir, ".config"), "r") as fconf:
reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read()
build = do_reproducible_build if reproducible else do_build ret = build(**kwargs)

Regards,
Yann E. MORIN.

Quoted 6 lines
scripts/autobuild-run +0 -1
         send_results(ret, **kwargs)
 
--

2.20.1

--

.-----------------.--------------------.------------------.--------------------.

|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics'
conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___
     |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is
no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v
 conspiracy.  |
Quoted 2 lines

'------------------------------^-------^------------------^--------------------'

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190614/6362ef8b/attachment.html>

#8 lore raw

PATCH v3 3/3 autobuild-run: do reproducible builds tests if BR2_REPRODUCIBLE=y

Quoted 12 lines · Atharva Lele
Quoted 6 lines

You need to do (basically):

? ? with open(os.path.join(outputdir, ".config"), "r") as fconf:
? ? ? ? reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() ? ? build = do_reproducible_build if reproducible else do_build ? ? ret = build(**kwargs)

It would still work if I just unindent the if-else block too right? This is what I actually intended to do
It's working for me in testing..
Like this:

Quoted 6 lines

? ? with open(os.path.join(outputdir, ".config"), "r") as fconf:
? ? ? ? reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() ? ? if reproducible:
? ? ? ? ret = do_reproducible_build(**kwargs)
? ? else:
? ? ? ? ret = do_build(**kwargs)

Yes, and I prefer that version. Yann considers the '... if ... else ...'
construct more pythonese, but I kind of like explicit conditions :-)

Regards,
Arnout

[snip]

#9 lore raw

PATCH v3 3/3 autobuild-run: do reproducible builds tests if BR2_REPRODUCIBLE=y

Quoted 2 lines

Yann considers the '... if ... else ...' construct more pythonese, but I kind of like explicit conditions :-)

Haha I do see his point of view. For me, I've only worked with C for the last 2 years at university and my last internship so it's a bit hard to 'talk' in
pythonese.

Regards,
Atharva Lele

Quoted 23 lines · Arnout Vandecappelle
Quoted 8 lines · Atharva Lele
Quoted 6 lines

You need to do (basically):

with open(os.path.join(outputdir, ".config"), "r") as fconf:
reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read()
build = do_reproducible_build if reproducible else do_build ret = build(**kwargs)

It would still work if I just unindent the if-else block too right? This

is what

Quoted 10 lines

I actually intended to do
It's working for me in testing..
Like this:

with open(os.path.join(outputdir, ".config"), "r") as fconf:
reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read()
if reproducible:
ret = do_reproducible_build(**kwargs)
else:
ret = do_build(**kwargs)

Quoted 9 lines

Yes, and I prefer that version. Yann considers the '... if ... else ...'
construct more pythonese, but I kind of like explicit conditions :-)

Regards,
Arnout

[snip]

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190614/e12fa2c2/attachment.html>

#10 lore raw

PATCH v3 3/3 autobuild-run: do reproducible builds tests if BR2_REPRODUCIBLE=y

In reply to Arnout Vandecappelle

Arnout, All,

On 2019-06-14 10:47 +0200, Arnout Vandecappelle spake thusly:

Quoted 20 lines · Atharva Lele
Quoted 12 lines
Quoted 6 lines

You need to do (basically):

? ? with open(os.path.join(outputdir, ".config"), "r") as fconf:
? ? ? ? reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() ? ? build = do_reproducible_build if reproducible else do_build ? ? ret = build(**kwargs)

It would still work if I just unindent the if-else block too right? This is what I actually intended to do
It's working for me in testing..
Like this:

Quoted 6 lines

? ? with open(os.path.join(outputdir, ".config"), "r") as fconf:
? ? ? ? reproducible = "BR2_REPRODUCIBLE=y\n" in fconf.read() ? ? if reproducible:
? ? ? ? ret = do_reproducible_build(**kwargs)
? ? else:
? ? ? ? ret = do_build(**kwargs)

Quoted 2 lines

Yes, and I prefer that version. Yann considers the '... if ... else ...'
construct more pythonese, but I kind of like explicit conditions :-)

Well, if we use a language, we should write with the idioms of that language. Python is not C, so we should not write Python as we would C.

That notwithstanding, I'm no Python expert either, so I'm also OK with the more "traditional and conventional" ways. ;-)

Regards,
Yann E. MORIN.

Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'
#11 lore raw

PATCH v3 1/3 autobuild-run: initial implementation of check_reproducibility()

In reply to Atharva Lele

Atharva, All,

On 2019-06-11 18:04 +0530, Atharva Lele spake thusly:

Quoted 10 lines

For reproducible builds, we want to find out if there are any differences between two builds having the same configuration, and to find the reason behind them. The diffoscope tool looks inside different types of files to see where the differences lie.

check_reproducibility() runs diffoscope on two output directories which are expected in output/images and output/images-1. Since it uses objdump, it needs to be provided the cross-compile prefix which is derived from the TARGET_CROSS variable.

Quoted 5 lines

Since diffoscope may not be installed, we fall back to cmp for byte-by-byte comparison. We add diffoscope to list of optional programs to avoid repeated checking of its presence.

Except for two minor coding rules (see below):

However, for the future: it would be wonderfull if we could entice the diffoscope developers to make it possible to use it directly from
python: diffoscope *is* a python module, after all.

However, it is cutrrently not very amenable at being used as a python object (yet):

diffoscope.main():
https://salsa.debian.org/reproducible-builds/diffoscope/blob/master/diffoscope/main.py#L695

which is a glorified wrapper to diffoscope.run_diffoscope():
https://salsa.debian.org/reproducible-builds/diffoscope/blob/master/diffoscope/main.py#L600

which is itself a further glofied wrapper to diffoscope.compare_root_paths() and diffoscope.Difference()...

But that's for later.

Quoted 56 lines

---
Changes v2 -> v3:
- Use file size of reproducible_results to check reproducibility status, rather than exit status of cmp or diff (suggested by arnout)
- Rename results file (diffoscope_output.txt -> reproducible_results) to avoid confusion
- Change handling of diffoscope output text file using with open()
- Changed commit message to have all necessary info (suggested by arnout)
- Removed leftover code from when I was using an exception to handle diffoscope presense (thanks to arnout)

Changes v1 -> v2:
- move diffoscope output to results dir (suggested by arnout)
- fix make printvars call
- Add diffoscope to DEFAULT_OPTIONAL_PROGS

---
 scripts/autobuild-run | 39 ++++++++++++++++++++++++++++++++++++++-
 1 file changed, 38 insertions(+), 1 deletion(-)
scripts/autobuild-run +20 -1
index 190a254..ba5b337 100755
@@ -204,7 +204,7 @@ def get_branch():
 
 class SystemInfo:
     DEFAULT_NEEDED_PROGS = ["make", "git", "gcc"]
-    DEFAULT_OPTIONAL_PROGS = ["bzr", "java", "javac", "jar"]
+    DEFAULT_OPTIONAL_PROGS = ["bzr", "diffoscope", "java", "javac", "jar"]
 
     def __init__(self):
         self.needed_progs = list(self.__class__.DEFAULT_NEEDED_PROGS)
@@ -394,6 +394,43 @@ def stop_on_build_hang(monitor_thread_hung_build_flag,
                 break
         monitor_thread_stop_flag.wait(30)
 
+def check_reproducibility(**kwargs):
+    """Check reproducibility of builds
+
+    Use diffoscope on the built images, if diffoscope is not
+    installed, fallback to cmp
+    """
+
+    log = kwargs['log']
+    idir = "instance-%d" % kwargs['instance']
+    outputdir = os.path.join(idir, "output")
+    srcdir = os.path.join(idir, "buildroot")
+    reproducible_results = os.path.join(outputdir, "results", "reproducible_results")
+    # Using only tar images for now
+    build_1_image = os.path.join(outputdir, "images-1", "rootfs.tar")
+    build_2_image = os.path.join(outputdir, "images", "rootfs.tar")
+
+    with open(reproducible_results, 'w') as diff:
+        if kwargs['sysinfo'].has("diffoscope"):
+            # Prefix to point diffoscope towards cross-tools

scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 425

Quoted 10 lines
scripts/autobuild-run +10 -0
+            prefix = subprocess.check_output(["make", "O=%s" % outputdir, "-C", srcdir, "printvars", "VARS=TARGET_CROSS"])
+            # Remove TARGET_CROSS= and \n from the string
+            prefix = prefix[13:-1]
+            log_write(log, "INFO: running diffoscope on images")
+            subprocess.call(["diffoscope", build_1_image, build_2_image,
+                                "--tool-prefix-binutils", prefix], stdout=diff, stderr=log)
+        else:
+            log_write(log, "INFO: diffoscope not installed, falling back to cmp")
+            subprocess.call(["cmp", "-b", "build_1_image", "build_2_image"], stdout=diff, stderr=log)
+    

They are invisible, but this empty line has 4 spaces.

scripts/autobuild-run def stop_on_build_hang(monitor_thread_hung_build_flag, · line 429

Quoted 4 lines
scripts/autobuild-run +4 -0
+    if os.stat(reproducible_results).st_size > 0:
+        log_write(log, "INFO: Build is non-reproducible.")
+        return -1
+    

Ditto.

Regards,
Yann E. MORIN.

Quoted 10 lines
scripts/autobuild-run +4 -0
+    # rootfs images match byte-for-byte -> reproducible image
+    log_write(log, "INFO: Build is reproducible!")
+    return 0
+
 def do_build(**kwargs):
     """Run the build itself"""
 
Signature · 1 line
2.20.1
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------.
|  Yann E. MORIN  | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: |
| +33 662 376 056 | Software  Designer | \ / CAMPAIGN     |  ___               |
| +33 561 099 427 `------------.-------:  X  AGAINST      |  \e/  There is no  |
| http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL    |   v   conspiracy.  |
'------------------------------^-------^------------------^--------------------'