Skip to contentSkip to conversation

autobuild-run: create reason file on build failures

PATCH New Atharva Lele 2 participants 5 messages Jun 22, 2019

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/3 autobuild-run: create reason file on build failures

When a build fails, we calculate the reason of failure on the server side as well as client side. To solve this redundancy we only calculate on the client side and submit a file which contains the reason of failure.

---
NOTE: This series depends on work from builder-class series

Changes v1 -> v2:
- Split commit into two, one for adding reason file and one for accounting for reproducibility failures (next patch).

---
 scripts/autobuild-run | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)
scripts/autobuild-run +5 -2
index 92afb26..c18d8aa 100755
@@ -596,6 +596,11 @@ class Builder:
             # not found
             return None
 
+        reason = get_failure_reason()
+        if reason:
+            with open(os.path.join(self.resultdir, "reason"), "w+") as reasonf:
+                reasonf.write("-".join(reason))
+
         def extract_end_log(resultfile):
             """Save the last part of the build log, starting from the failed package"""
 
@@ -604,7 +609,6 @@ class Builder:
                                 (os.path.join(self.outputdir, "logfile"), resultfile)],
                                 shell=True)
 
-            reason = get_failure_reason()
             if not reason:
                 extract_last_500_lines()
             else:
@@ -628,7 +632,6 @@ class Builder:
         def copy_config_log_files():
             """Recursively copy any config.log files from the failing package"""
 
-            reason = get_failure_reason()
             if not reason:
                 return
 
Signature · 1 line
2.20.1
#1 lore raw

PATCH v2 2/3 autobuild-run: account for reproducibility failures when creating the reason file

---
Changes v1 -> v2:
- Split into separate commit for reproducibility failures

---
 scripts/autobuild-run | 33 +++++++++++++++++++++++----------
 1 file changed, 23 insertions(+), 10 deletions(-)
scripts/autobuild-run +23 -10
index c18d8aa..6068376 100755
@@ -582,16 +582,26 @@ class Builder:
             return
 
         def get_failure_reason():
-            # Output is a tuple (package, version), or None.
-            lastlines = decode_bytes(subprocess.Popen(
-                ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")],
-                stdout=subprocess.PIPE).communicate()[0]).splitlines()
+            # Output is a tuple (package, version), or None in case of package failure
+            # Output is "reproducible" in case of reproducibility failure
+
+            reproducible_results = os.path.join(self.resultdir, "reproducible_results")
+            if os.path.exists(reproducible_results):
+                if os.stat(reproducible_results).st_size > 0:
+                    reason = "reproducible"
+                    return reason
+                else:
+                    return None
+            else:
+                lastlines = decode_bytes(subprocess.Popen(
+                    ["tail", "-n", "3", os.path.join(self.outputdir, "logfile")],
+                    stdout=subprocess.PIPE).communicate()[0]).splitlines()
 
-            regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/')
-            for line in lastlines:
-                m = regexp.search(line)
-                if m:
-                    return m.group(1).rsplit('-', 1)
+                regexp = re.compile(r'make: \*\*\* .*/(?:build|toolchain)/([^/]*)/')
+                for line in lastlines:
+                    m = regexp.search(line)
+                    if m:
+                        return m.group(1).rsplit('-', 1)
 
             # not found
             return None
@@ -599,7 +609,10 @@ class Builder:
         reason = get_failure_reason()
         if reason:
             with open(os.path.join(self.resultdir, "reason"), "w+") as reasonf:
-                reasonf.write("-".join(reason))
+                if reason == "reproducible":
+                    reasonf.write(reason)
+                else:
+                    reasonf.write("-".join(reason))
 
         def extract_end_log(resultfile):
             """Save the last part of the build log, starting from the failed package"""
Signature · 1 line
2.20.1
#2 lore raw

PATCH v2 3/3 web/import.inc.php: support reading failure reason from reason file

In reply to Atharva Lele

---
NOTE: This patch is untested.

---
 web/import.inc.php | 26 +++++++++++++++-----------
 1 file changed, 15 insertions(+), 11 deletions(-)
web/import.inc.php +15 -11
index a9b368f..6f878e0 100644
@@ -230,17 +230,21 @@ function import_result($buildid, $filename)
     if ($status == 0)
       $reason = "none";
     else {
-	$tmp = Array();
-	exec("tail -3 " . $thisbuildfinaldir . "build-end.log | grep -v '\[_all\]' | grep 'make.*: \*\*\*' | sed 's,.*\[\([^\]*\)\] Error.*,\\1,' | sed 's,.*/build/\([^/]*\)/.*,\\1,'", $tmp);
-	if (trim($tmp[0]))
-	  $reason = $tmp[0];
-	else {
-	  exec("tail -1 " . $thisbuildfinaldir . "build-time.log | grep :start: | cut -d':' -f4", $tmp);
-	  if (trim($tmp[0]))
-	    $reason = trim($tmp[0]);
-	  else
-	    $reason = "unknown";
-	}
+      if (file_exists($thisbuildfinaldir . "reason"))
+        $reason = trim(file_get_contents($thisbuildfinaldir . "reason", "r"));
+      else {
+        $tmp = Array();
+        exec("tail -3 " . $thisbuildfinaldir . "build-end.log | grep -v '\[_all\]' | grep 'make.*: \*\*\*' | sed 's,.*\[\([^\]*\)\] Error.*,\\1,' | sed 's,.*/build/\([^/]*\)/.*,\\1,'", $tmp);
+        if (trim($tmp[0]))
+          $reason = $tmp[0];
+        else {
+          exec("tail -1 " . $thisbuildfinaldir . "build-time.log | grep :start: | cut -d':' -f4", $tmp);
+          if (trim($tmp[0]))
+            $reason = trim($tmp[0]);
+          else
+            $reason = "unknown";
+        }
+      }
     }
 
     /* Compress files that are typically too large and infrequently
Signature · 1 line
2.20.1
#3 lore raw

PATCH v2 1/3 autobuild-run: create reason file on build failures

In reply to Atharva Lele

Hello Atharva,

Thanks for the second iteration. One question below.

scripts/autobuild-run class Builder: · line 602

Quoted 4 lines · On Sun, 23 Jun 2019 00:11:16 +0530 Atharva Lele <[email protected]> wrote:
scripts/autobuild-run +4 -0
+        reason = get_failure_reason()
+        if reason:
+            with open(os.path.join(self.resultdir, "reason"), "w+") as reasonf:
+                reasonf.write("-".join(reason))

Why do we have this "-".join(reason) instead of just writing the contents of the "reason" variable to the file ?

Thomas

Signature · 3 lines
Thomas Petazzoni, CTO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

PATCH v2 1/3 autobuild-run: create reason file on build failures

Quoted 4 lines · Thomas Petazzoni

Why do we have this "-".join(reason) instead of just writing the contents of the "reason" variable to the file ?

The reason to do that is to achieve the same formatting for the failure reason
as the PHP script that is "package-version"

Regards,
Atharva Lele
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.busybox.net/pipermail/buildroot/attachments/20190622/2507a460/attachment.html>