Skip to contentSkip to conversation

Capture command-line builds which fail due to bad targets

PATCH In review Elliot Smith 3 participants 5 messages Apr 25, 2016 – Jun 9, 2016

lore
Share & tools
lore
#0 lore raw

cover Capture command-line builds which fail due to bad targets

Toaster currently can't catch build failures which are due to bad targets (e.g. "bitbake ecky-thump"), as the bitbake run doesn't reach the BuildStarted event in these cases.

Add a new BuildInit event which fires as soon as buildTargets() or buildFile() is invoked, and make the build object at this point. When BuildStarted fires, collect layer information, build variables etc. as previously.

Backwards-compatibility is maintained for jethro by still making a build object on BuildStarted events. jethro builds which fail early will therefore not be captured by the new code, but existing functionality should not be broken either.

To test:

1. Start Toaster.
2. On the command-line, run a build with a bad target like "bitbake ecky-thump".
3. The build should register as a failure on the "all builds" page.

The following changes since commit fb8349b97c736d6633b2d79a4003dd86f760bdb1 (toaster-next):

toaster: bin/toaster Fix noweb command (2016-04-21 14:19:05 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib elliot/toaster/failed_builds-8440-v6
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/failed_builds-8440-v6

Elliot Smith (2):
cooker: add BuildInit event
toasterui: use BuildInit to capture builds with bad targets

 bitbake/lib/bb/cooker.py             |  6 ++++++
 bitbake/lib/bb/event.py              |  6 +++++-
 bitbake/lib/bb/ui/buildinfohelper.py | 17 ++++++++++-------
 bitbake/lib/bb/ui/toasterui.py       | 19 ++++++++-----------
 4 files changed, 29 insertions(+), 19 deletions(-)

--
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

#1 lore raw

PATCH 1/2 cooker: add BuildInit event

In situations where a bitbake run fails before the build properly starts and BuildStarted is fired, a UI has no way to get at the targets passed to the build. This makes it difficult for the UI to report on the targets which failed.

Fire a BuildInit event before running buildTargets() or buildFile(). This enables a UI to capture targets passed to buildTargets(), even if the build fails (e.g. the targets themselves are invalid).

[YOCTO #8440]

---
 bitbake/lib/bb/cooker.py | 6 ++++++
 bitbake/lib/bb/event.py  | 6 +++++-
 2 files changed, 11 insertions(+), 1 deletion(-)
bitbake/lib/bb/cooker.py +6 -0
index 9b565fc..2ce0215 100644
@@ -1264,6 +1264,7 @@ class BBCooker:
         """
         Build the file matching regexp buildfile
         """
+        bb.event.fire(bb.event.BuildInit(), self.expanded_data)
 
         # Too many people use -b because they think it's how you normally
         # specify a target to be built, so show a warning
@@ -1367,6 +1368,11 @@ class BBCooker:
         Attempt to build the targets specified
         """
 
+        # build a list of target:task strings in format
+        # ['zlib:do_build', 'mpfr-native:do_build', ...]
+        packages = map(lambda target: "%s:do_%s" % (target, task), targets)
+        bb.event.fire(bb.event.BuildInit(packages), self.expanded_data)
+
         def buildTargetsIdle(server, rq, abort):
             msg = None
             interrupted = 0
bitbake/lib/bb/event.py +5 -1
index 5ffe89e..8377e3e 100644
@@ -373,7 +373,11 @@ class BuildBase(Event):
 
 
 
-
+class BuildInit(BuildBase):
+    """buildFile or buildTargets was invoked"""
+    def __init__(self, p=[]):
+        name = None
+        BuildBase.__init__(self, name, p)
 
 class BuildStarted(BuildBase, OperationStarted):
     """bbmake build run started"""
Signature · 12 lines
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#2 lore raw

PATCH 2/2 toasterui: use BuildInit to capture builds with bad targets

In reply to Elliot Smith

If a build has bad targets which cause bitbake to fail before BuildStarted is fired, we are unable to represent this in Toaster, as the build was not detected at all.

Use the BuildInit event so we can capture these failures.

BuildStarted is retained as the event which causes a build object to be created, to support older versions of bitbake which don't have BuildInit.

[YOCTO #8440]

---
 bitbake/lib/bb/ui/buildinfohelper.py | 17 ++++++++++-------
 bitbake/lib/bb/ui/toasterui.py       | 19 ++++++++-----------
 2 files changed, 18 insertions(+), 18 deletions(-)
bitbake/lib/bb/ui/buildinfohelper.py +10 -7
index 9397905..da8ec00 100644
@@ -1017,7 +1017,17 @@ class BuildInfoHelper(object):
 
         self.internal_state['build'] = build_obj
 
+        # create target information
+        target_information = {}
+        target_information['targets'] = event._pkgs
+        target_information['build'] = build_obj
+
+        self.internal_state['targets'] = self.orm_wrapper.get_or_create_targets(target_information)
+
+    def save_build_layers_and_variables(self, event):
         # save layer version information for this build
+        build_obj = self.internal_state['build']
+
         if not 'lvs' in self.internal_state:
             logger.error("Layer version information not found; Check if the bitbake server was configured to inherit toaster.bbclass.")
         else:
@@ -1026,13 +1036,6 @@ class BuildInfoHelper(object):
 
             del self.internal_state['lvs']
 
-        # create target information
-        target_information = {}
-        target_information['targets'] = event._pkgs
-        target_information['build'] = build_obj
-
-        self.internal_state['targets'] = self.orm_wrapper.get_or_create_targets(target_information)
-
         # Save build configuration
         data = self.server.runCommand(["getAllKeysWithFlags", ["doc", "func"]])[0]
 
bitbake/lib/bb/ui/toasterui.py +8 -11
index 6bf4c1f..6d0dea9 100644
@@ -103,6 +103,7 @@ _evt_list = [
     "bb.command.CommandFailed",
     "bb.cooker.CookerExit",
     "bb.event.BuildCompleted",
+    "bb.event.BuildInit",
     "bb.event.BuildStarted",
     "bb.event.CacheLoadCompleted",
     "bb.event.CacheLoadProgress",
@@ -231,19 +232,15 @@ def main(server, eventHandler, params):
             # pylint: disable=protected-access
             # the code will look into the protected variables of the event; no easy way around this
 
-            # we treat ParseStarted as the first event of toaster-triggered
-            # builds; that way we get the Build Configuration included in the log
-            # and any errors that occur before BuildStarted is fired
-            if isinstance(event, bb.event.ParseStarted):
+            # create a build object in buildinfohelper from either BuildInit (if available)
+            # or BuildStarted (for jethro and previous versions)
+            if isinstance(event, (bb.event.BuildStarted, bb.event.BuildInit)):
                 if not (build_log and build_log_file_path):
                     build_log, build_log_file_path = _open_build_log(log_dir)
-                continue
-
-            if isinstance(event, bb.event.BuildStarted):
-                if not (build_log and build_log_file_path):
-                    build_log, build_log_file_path = _open_build_log(log_dir)
-
-                buildinfohelper.store_started_build(event, build_log_file_path)
+                    buildinfohelper.store_started_build(event, build_log_file_path)
+                # get additional data from BuildStarted
+                if isinstance(event, bb.event.BuildStarted):
+                    buildinfohelper.save_build_layers_and_variables(event)
                 continue
 
             if isinstance(event, (bb.build.TaskStarted, bb.build.TaskSucceeded, bb.build.TaskFailedSilent)):
Signature · 12 lines
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.
#3 lore raw
In reply to Elliot Smith
Quoted 31 lines · On 25/04/2016 16:26, "[email protected] on behalf of Elliot…

Toaster currently can't catch build failures which are due to bad targets (e.g.
"bitbake ecky-thump"), as the bitbake run doesn't reach the BuildStarted event in these cases.

Add a new BuildInit event which fires as soon as buildTargets() or buildFile()
is invoked, and make the build object at this point. When BuildStarted fires,
collect layer information, build variables etc. as previously.

Backwards-compatibility is maintained for jethro by still making a build object
on BuildStarted events. jethro builds which fail early will therefore not be
captured by the new code, but existing functionality should not be broken either.

To test:

1. Start Toaster.
2. On the command-line, run a build with a bad target like "bitbake ecky-thump".
3. The build should register as a failure on the "all builds" page.

The following changes since commit
fb8349b97c736d6633b2d79a4003dd86f760bdb1
(toaster-next):

toaster: bin/toaster Fix noweb command (2016-04-21 14:19:05 +0100)

Quoted 7 lines

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib
elliot/toaster/failed_builds-8440-v6

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/fa
iled_builds-8440-v6

This works for me. Thanks!

Belén

Quoted 30 lines

Elliot Smith (2):
cooker: add BuildInit event
toasterui: use BuildInit to capture builds with bad targets

bitbake/lib/bb/cooker.py             |  6 ++++++
bitbake/lib/bb/event.py              |  6 +++++-
bitbake/lib/bb/ui/buildinfohelper.py | 17 ++++++++++-------
bitbake/lib/bb/ui/toasterui.py       | 19 ++++++++-----------
4 files changed, 29 insertions(+), 19 deletions(-)

--
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Signature · 4 lines
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster
#4 lore raw

Thanks - pushed to toaster-next

Quoted 76 lines · Belen
Quoted 31 lines · On 25/04/2016 16:26, "[email protected] on behalf of Elliot…

Toaster currently can't catch build failures which are due to bad targets (e.g.
"bitbake ecky-thump"), as the bitbake run doesn't reach the BuildStarted event in these cases.

Add a new BuildInit event which fires as soon as buildTargets() or buildFile()
is invoked, and make the build object at this point. When BuildStarted fires,
collect layer information, build variables etc. as previously.

Backwards-compatibility is maintained for jethro by still making a build object
on BuildStarted events. jethro builds which fail early will therefore not be
captured by the new code, but existing functionality should not be broken either.

To test:

1. Start Toaster.
2. On the command-line, run a build with a bad target like "bitbake ecky-thump".
3. The build should register as a failure on the "all builds" page.

The following changes since commit
fb8349b97c736d6633b2d79a4003dd86f760bdb1
(toaster-next):

toaster: bin/toaster Fix noweb command (2016-04-21 14:19:05 +0100)

Quoted 7 lines

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib
elliot/toaster/failed_builds-8440-v6

http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/fa
iled_builds-8440-v6

This works for me. Thanks!

Belén

Quoted 29 lines

Elliot Smith (2):
cooker: add BuildInit event
toasterui: use BuildInit to capture builds with bad targets

bitbake/lib/bb/cooker.py             |  6 ++++++
bitbake/lib/bb/event.py              |  6 +++++-
bitbake/lib/bb/ui/buildinfohelper.py | 17 ++++++++++-------
bitbake/lib/bb/ui/toasterui.py       | 19 ++++++++-----------
4 files changed, 29 insertions(+), 19 deletions(-)

--
1.9.3

---------------------------------------------------------------------
Intel Corporation (UK) Limited
Registered No. 1134945 (England)
Registered Office: Pipers Way, Swindon SN3 1RJ
VAT No: 860 2173 47

This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies.

Signature · 4 lines
_______________________________________________
toaster mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/toaster