Fix SDK downloads forbuilds

Hi,

Artifacts could not be downloaded for a build as they were not being listed. It was not possible to download them manually either, as the artifact download code was broken for SDK artifacts produced by command line builds.

Fix how toaster.bbclass hooks onto bitbake's build tasks, so that the SDK artifacts are picked up after do_populate_sdk runs.

Fix Toaster code for returning SDK artifacts in HTTP responses.

Changes since b53df3c (toaster-next) are in
git://git.yoctoproject.org/poky-contrib, elliot/toaster/sdk_artifacts-7603
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=elliot/toaster/sdk_artifacts-7603

NB one of the commits on this branch (fbc2024) has been submitted to oe-core:
http://lists.openembedded.org/pipermail/openembedded-core/2016-January/115366.html
This patch set contains the other commits on the branch.

Related bug: https://bugzilla.yoctoproject.org/show_bug.cgi?id=7603

To test:

1. Ensure that your database has no build artifacts in it, either by removing the database, clearing the relevant table with SQL via the command line (delete from orm_buildartifact), or removing the objects via the Django shell (orm.models.BuildArtifact.objects.all().delete()).

Note that the current behaviour, where we only associate SDK artifacts with the first build which produces them, has not been modified. So if you have already run a build and have files in tmp/deploy/sdk associated with it, subsequent builds will not be associated with those files.

2. Start a new build from the command line or Toaster, invoking the populate_sdk target:

# command line
bitbake -c populate_sdk core-image-minimal

# Toaster (NB you must use the "Local" release) run a build like "core-image-minimal:populate_sdk"

3. View the build dashboard for the build when it completes and ensure it contains an "Other artifacts" section, as shown in the bug report.

4. Click on one of the "Other artifact" links and ensure that the artifact downloads correctly.

As this code slightly modifies all other artifact downloads, you could also check downloads for the following, too:

* cooker logs
* task logs
* image files
* licence manifests

The following changes since commit d1cdf28a40489c9d730dfed3a48e67cb34e8890b:

toaster: tests Remove symlinks from toasteruitest folder (2016-01-18 15:23:51 +0200)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/toaster/submit/elliot/toaster/sdk_artifacts-7603
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/toaster/submit/elliot/toaster/sdk_artifacts-7603

Elliot Smith (3):
toaster: toasterui: listen for bb.event.MetadataEvent
toaster: toasterui: log OSErrorException metadata events
toaster: toastergui: make artifact download more robust

 lib/bb/ui/toasterui.py                             |  5 +-
 .../toastergui/templates/unavailable_artifact.html | 13 ++-
 lib/toaster/toastergui/views.py                    | 97 +++++++++-------------
 3 files changed, 51 insertions(+), 64 deletions(-)

--
Regards,
Ed

From: Elliot Smith <[email protected]>

The event mask for toasterui doesn't include MetadataEvents. This means that we're missing the ArtifactFileSize event (among others), which is the one we use to populate the SDK artifact table.

Add that event type to the toasterui event mask so we can record SDK artifacts as they are created.

[YOCTO #7603]

---
 lib/bb/ui/toasterui.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

index 0265df7..19be623 100644
@@ -99,7 +99,8 @@ _evt_list = [ "bb.runqueue.runQueueExitWait", "bb.event.LogExecTTY", "logging.Lo
               "bb.command.CommandExit", "bb.command.CommandCompleted",  "bb.cooker.CookerExit",
               "bb.event.MultipleProviders", "bb.event.NoProvider", "bb.runqueue.sceneQueueTaskStarted",
               "bb.runqueue.runQueueTaskStarted", "bb.runqueue.runQueueTaskFailed", "bb.runqueue.sceneQueueTaskFailed",
-              "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent"]
+              "bb.event.BuildBase", "bb.build.TaskStarted", "bb.build.TaskSucceeded", "bb.build.TaskFailedSilent",
+              "bb.event.MetadataEvent"]
 
 def main(server, eventHandler, params):
     # set to a logging.FileHandler instance when a build starts;

2.1.4

From: Elliot Smith <[email protected]>

OSErrors occurring in toaster.bbclass are converted to OSErrorException metadata events. They were then being swallowed as unprocessed events by toasterui, which made them difficult to spot.

Explicitly catch OSErrorException events and log them so they are easier to spot and debug.

---
 lib/bb/ui/toasterui.py | 2 ++
 1 file changed, 2 insertions(+)

index 19be623..32b1889 100644
@@ -366,6 +366,8 @@ def main(server, eventHandler, params):
                     buildinfohelper.store_license_manifest_path(event)
                 elif event.type == "SetBRBE":
                     buildinfohelper.brbe = buildinfohelper._get_data_from_event(event)
+                elif event.type == "OSErrorException":
+                    logger.error(event)
                 else:
                     logger.error("Unprocessed MetadataEvent %s ", str(event))
                 continue

2.1.4

From: Elliot Smith <[email protected]>

When an artifact download is requested, Toaster goes through a convoluted series of conditions to decide which file to push to the response. In the case of build artifact downloads for command line builds, this caused an ugly exception, as command line builds don't have a build request.

To simplify and catch more corner cases, remove the code which fetches files via the build environment (we only support the local build environment anyway). Then push all requests along a single path, catching any missing file errors, missing object errors or poorly-formed URLs in a single except clause which always returns a valid response.

Also modify the text on the "unavailable artifact" page so it says that the artifact doesn't exist, rather than it "no longer" exists (exceptions may occur because an invalid artifact was requested, rather than an artifact which was removed).

[YOCTO #7603]

---
 .../toastergui/templates/unavailable_artifact.html | 13 ++-
 lib/toaster/toastergui/views.py                    | 97 +++++++++-------------
 2 files changed, 47 insertions(+), 63 deletions(-)

index 0301a6c..2d3d02c 100644
@@ -3,15 +3,14 @@
 {% load humanize %}
 {% load static %}
 
-{% block title %} Build artifact no longer exists - Toaster {% endblock %}
+{% block title %} Build artifact does not exist - Toaster {% endblock %}
 
 {% block pagecontent %}
-
-<div class="row-fluid air">
-    <div class="alert alert-info span8 lead">
-        <p"> The build artifact you are trying to download no longer exists.</p>
-        <p><a href="javascript:window.history.back()">Back to previous page</a></p>
+    <div class="row-fluid air">
+        <div class="alert alert-info span8 lead">
+            <p>The build artifact you are trying to download does not exist.</p>
+            <p><a href="javascript:window.history.back()">Back to previous page</a></p>
+        </div>
     </div>
-</div>
 {% endblock %}
 

2.1.4

Its a small detail but when sending patches, "toasterui:" is fine in the shortlog, we don't need "toaster: toasterui:". The idea is to highlight which bit of the system is being changed and "toasterui" conveys that fine.

[cc'd to the toaster list so everyone sees this]

No need to resend these, I just tweaked the lines.

Cheers,

Richard