Skip to contentSkip to conversation

toaster: fix SDK artifact capture

PATCH In review David Reyna 2 participants 3 messages Apr 10, 2017 – Apr 11, 2017

lore
Share & tools
lore
#0 lore raw

cover toaster: fix SDK artifact capture

From: David Reyna <[email protected]>

Toaster: fix SDK artifact capture. This has been tested against qemux86, qemuarm, qemuarm64, qemumips64, and genericx86-64 for a mix of MACHINES and 64/32.

The following changes since commit 633ad6c9f436f5d2b6ee1a005b697661a054a394:

oeqa/runtime/utils/targetbuildproject: use parent classes defaults tmpdir (2017-04-06 10:13:39 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib dreyna/toaster/sdk_artifacts_10850
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=dreyna/submit/dreyna/toaster/sdk_artifacts_10850

David Reyna (1):
toaster: fix SDK artifact capture

 bitbake/lib/bb/ui/buildinfohelper.py | 30 ++++++++++++++++++++++++++++++
 bitbake/lib/bb/ui/toasterui.py       |  4 +---
 meta/classes/toaster.bbclass         | 21 ++++++++++++++++-----
 3 files changed, 47 insertions(+), 8 deletions(-)
Signature · 1 line
1.9.1
#1 lore raw

PATCH 1/1 toaster: fix SDK artifact capture

From: David Reyna <[email protected]>

Use the TaskArtifacts event to scan the SDK and ESDK manifests to cleanly collect the respective artifact files.

The previous method was broken when the SDK file deployment moved from the do_populate_sdk[_ext] tasks to their sstate tasks. That method is disabled (but not yet removed) in preparation for the rest of refactor work for the parent #10283 work.

[YOCTO #10850]

---
 bitbake/lib/bb/ui/buildinfohelper.py | 30 ++++++++++++++++++++++++++++++
 bitbake/lib/bb/ui/toasterui.py       |  4 +---
 meta/classes/toaster.bbclass         | 21 ++++++++++++++++-----
 3 files changed, 47 insertions(+), 8 deletions(-)
bitbake/lib/bb/ui/buildinfohelper.py +30 -0
index 92d1a1c..46be5a5 100644
@@ -1663,6 +1663,36 @@ class BuildInfoHelper(object):
                 break
         return endswith
 
+    def scan_task_artifacts(self, event):
+        """
+        The 'TaskArtifacts' event passes the manifest file content for the
+        tasks 'do_deploy', 'do_image_complete', 'do_populate_sdk', and
+        'do_populate_sdk_ext'. The first two will be implemented later.
+        """
+        task_vars = BuildInfoHelper._get_data_from_event(event)
+        task_name = task_vars['task'][task_vars['task'].find(':')+1:]
+        task_artifacts = task_vars['artifacts']
+
+        if task_name in ['do_populate_sdk', 'do_populate_sdk_ext']:
+            targets = [target for target in self.internal_state['targets'] \
+                if target.task == task_name[3:]]
+            if not targets:
+                logger.warning("scan_task_artifacts: SDK targets not found: %s\n", task_name)
+                return
+            for artifact_path in task_artifacts:
+                if not os.path.isfile(artifact_path):
+                    logger.warning("scan_task_artifacts: artifact file not found: %s\n", artifact_path)
+                    continue
+                for target in targets:
+                    # don't record the file if it's already been added
+                    # to this target
+                    matching_files = TargetSDKFile.objects.filter(
+                        target=target, file_name=artifact_path)
+                    if matching_files.count() == 0:
+                        artifact_size = os.stat(artifact_path).st_size
+                        self.orm_wrapper.save_target_sdk_file(
+                            target, artifact_path, artifact_size)
+
     def _get_image_files(self, deploy_dir_image, image_name, image_file_extensions):
         """
         Find files in deploy_dir_image whose basename starts with the
bitbake/lib/bb/ui/toasterui.py +1 -3
index 1729902..71f04fa 100644
@@ -438,9 +438,7 @@ def main(server, eventHandler, params):
                 elif event.type == "SetBRBE":
                     buildinfohelper.brbe = buildinfohelper._get_data_from_event(event)
                 elif event.type == "TaskArtifacts":
-                    # not implemented yet
-                    # see https://bugzilla.yoctoproject.org/show_bug.cgi?id=10283 for details
-                    pass
+                    buildinfohelper.scan_task_artifacts(event)
                 elif event.type == "OSErrorException":
                     logger.error(event)
                 else:
meta/classes/toaster.bbclass +16 -5
index 4de380b..296e476 100644
@@ -328,8 +328,18 @@ python toaster_artifacts() {
     if e.taskname in ["do_deploy", "do_image_complete", "do_populate_sdk", "do_populate_sdk_ext"]:
         d2 = d.createCopy()
         d2.setVar('FILE', e.taskfile)
-        d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}"))
+        # Use 'stamp-extra-info' if present, else use workaround
+        # to determine 'SSTATE_MANMACH'
+        extrainf = d2.getVarFlag(e.taskname, 'stamp-extra-info')
+        if extrainf:
+            d2.setVar('SSTATE_MANMACH', extrainf)
+        else:
+            if "do_populate_sdk" == e.taskname:
+                d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}${SDKMACHINE}"))
+            else:
+                d2.setVar('SSTATE_MANMACH', d2.expand("${MACHINE}"))
         manifest = oe.sstatesig.sstate_get_manifest_filename(e.taskname[3:], d2)[0]
+
         if os.access(manifest, os.R_OK):
             with open(manifest) as fmanifest:
                 artifacts = [fname.strip() for fname in fmanifest]
@@ -356,8 +366,9 @@ do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata "
 do_package[postfuncs] += "toaster_package_dumpdata "
 do_package[vardepsexclude] += "toaster_package_dumpdata "
 
-do_populate_sdk[postfuncs] += "toaster_artifact_dumpdata "
-do_populate_sdk[vardepsexclude] += "toaster_artifact_dumpdata "
+#do_populate_sdk[postfuncs] += "toaster_artifact_dumpdata "
+#do_populate_sdk[vardepsexclude] += "toaster_artifact_dumpdata "
+
+#do_populate_sdk_ext[postfuncs] += "toaster_artifact_dumpdata "
+#do_populate_sdk_ext[vardepsexclude] += "toaster_artifact_dumpdata "
 
-do_populate_sdk_ext[postfuncs] += "toaster_artifact_dumpdata "
-do_populate_sdk_ext[vardepsexclude] += "toaster_artifact_dumpdata "
Signature · 1 line
1.9.1
#2 lore raw
Quoted 12 lines · David Reyna

From: David Reyna <[email protected]>

Use the TaskArtifacts event to scan the SDK and ESDK manifests to cleanly collect the respective artifact files.

The previous method was broken when the SDK file deployment moved from the do_populate_sdk[_ext] tasks to their sstate tasks. That method is disabled (but not yet removed) in preparation for the rest of refactor work for the parent #10283 work.

[YOCTO #10850]

Quoted 7 lines

---
 bitbake/lib/bb/ui/buildinfohelper.py | 30
++++++++++++++++++++++++++++++
 bitbake/lib/bb/ui/toasterui.py       |  4 +--- meta/classes/toaster.bbclass         | 21 ++++++++++++++++-----

 3 files changed, 47 insertions(+), 8 deletions(-)

This isn't really in the right format, the patches against toaster.bbclass needs to be sent separately to OE-Core since its in a separate repository.

I've sorted this out and merged it as we're close to release builds and I appreciate this needed fixing, please just keep it in mind next time, thanks!

Richard