Skip to contentSkip to conversation

toaster: Better SDK artifacts and fstypes V2

PATCH In review bavery 2 participants 4 messages Jul 12, 2016 – Jul 19, 2016

lore
Share & tools
lore

bavery

author
#0 lore raw

cover toaster: Better SDK artifacts and fstypes V2

This is part of a patch set that is split over the openembedded-core and bitbake-devel mailing lists. This is a second version of the patch.

These patches remove complexity from the toaster.bbclass in order to move those functions to bitbake/lib/toaster.

The overall intent of this set is to improve Toaster's ability to locate artifacts, specify fstypes correctly, and add some tests for these changes.

-brian avery
an Intel employee

The following changes since commit 0659b4f780120b1459b068c85794be4f04f943bd:

toaster: fix typo in arguments for libtoaster.js function (2016-07-12 14:12:16 -0700)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib bavery/submit/elliot/toaster/8556-image_fstypesV2_foroe
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=bavery/submit/elliot/toaster/8556-image_fstypesV2_foroe

Elliot Smith (2):
toaster.bbclass: remove directory scan logic for detecting artifacts toaster.bbclass: only scan files-in-image.txt if it exists

 classes/toaster.bbclass | 99 ++++++++++++-------------------------------------
 1 file changed, 24 insertions(+), 75 deletions(-)

--
1.9.1

author
#1 lore raw

PATCH 1/2 toaster.bbclass: remove directory scan logic for detecting artifacts

From: Elliot Smith <[email protected]>

toaster.bbclass does a scan of the image deploy and SDK directories when a build finishes. However, this brings no benefit and could be better managed and made easier to modify if moved to toasterui and carried out when the BuildCompleted event occurs.

Remove the image scan code from toaster.bbclass, prior to moving it to toasterui and buildinfohelper.

Also remove the license manifest update code, as this can also be done from toasterui.

The postfuncs for do_populate_sdk are retained, but no longer do the directory scan for SDK artifacts. Instead, they fire an event with the value of the TOOLCHAIN_OUTPUTNAME variable, as this is only accessible at the point when the do_populate_sdk and do_populate_sdk_ext tasks are run. The value of this can then be used by buildinfohelper to find the SDK artifacts produced by a target.

[YOCTO #9002]

---
 classes/toaster.bbclass | 74 ++++++-------------------------------------------
 1 file changed, 8 insertions(+), 66 deletions(-)
classes/toaster.bbclass +8 -66
index 1878fe0..90ea563 100644
@@ -136,60 +136,16 @@ python toaster_package_dumpdata() {
 
 # 2. Dump output image files information
 
-python toaster_image_dumpdata() {
-    """
-    Image filename for output images is not standardized.
-    image_types.bbclass will spell out IMAGE_CMD_xxx variables that actually
-    have hardcoded ways to create image file names in them.
-    So we look for files starting with the set name.
-
-    We also look for other files in the images/ directory which don't
-    match IMAGE_NAME, such as the kernel bzImage, modules tarball etc.
-    """
-
-    dir_to_walk = d.getVar('DEPLOY_DIR_IMAGE', True);
-    image_name = d.getVar('IMAGE_NAME', True);
-    image_info_data = {}
-    artifact_info_data = {}
-
-    # collect all images and artifacts in the images directory
-    for dirpath, dirnames, filenames in os.walk(dir_to_walk):
-        for filename in filenames:
-            full_path = os.path.join(dirpath, filename)
-            try:
-                if filename.startswith(image_name):
-                    # image
-                    image_info_data[full_path] = os.stat(full_path).st_size
-                else:
-                    # other non-image artifact
-                    if not os.path.islink(full_path):
-                        artifact_info_data[full_path] = os.stat(full_path).st_size
-            except OSError as e:
-                bb.event.fire(bb.event.MetadataEvent("OSErrorException", e), d)
-
-    bb.event.fire(bb.event.MetadataEvent("ImageFileSize", image_info_data), d)
-    bb.event.fire(bb.event.MetadataEvent("ArtifactFileSize", artifact_info_data), d)
-}
-
 python toaster_artifact_dumpdata() {
     """
-    Dump data about artifacts in the SDK_DEPLOY directory
+    Dump data about SDK variables
     """
 
-    dir_to_walk = d.getVar("SDK_DEPLOY", True)
-    artifact_info_data = {}
-
-    # collect all artifacts in the sdk directory
-    for dirpath, dirnames, filenames in os.walk(dir_to_walk):
-        for filename in filenames:
-            full_path = os.path.join(dirpath, filename)
-            try:
-                if not os.path.islink(full_path):
-                    artifact_info_data[full_path] = os.stat(full_path).st_size
-            except OSError as e:
-                bb.event.fire(bb.event.MetadataEvent("OSErrorException", e), d)
+    event_data = {
+      "TOOLCHAIN_OUTPUTNAME": d.getVar("TOOLCHAIN_OUTPUTNAME", True)
+    }
 
-    bb.event.fire(bb.event.MetadataEvent("ArtifactFileSize", artifact_info_data), d)
+    bb.event.fire(bb.event.MetadataEvent("SDKArtifactInfo", event_data), d)
 }
 
 # collect list of buildstats files based on fired events; when the build completes, collect all stats and fire an event with collected data
@@ -361,17 +317,6 @@ python toaster_buildhistory_dump() {
 
 }
 
-# dump information related to license manifest path
-
-python toaster_licensemanifest_dump() {
-    deploy_dir = d.getVar('DEPLOY_DIR', True);
-    image_name = d.getVar('IMAGE_NAME', True);
-
-    data = { 'deploy_dir' : deploy_dir, 'image_name' : image_name }
-
-    bb.event.fire(bb.event.MetadataEvent("LicenseManifestPath", data), d)
-}
-
 # set event handlers
 addhandler toaster_layerinfo_dumpdata
 toaster_layerinfo_dumpdata[eventmask] = "bb.event.TreeDataPreparationCompleted"
@@ -388,11 +333,8 @@ do_packagedata_setscene[vardepsexclude] += "toaster_package_dumpdata "
 do_package[postfuncs] += "toaster_package_dumpdata "
 do_package[vardepsexclude] += "toaster_package_dumpdata "
 
-do_image_complete[postfuncs] += "toaster_image_dumpdata "
-do_image_complete[vardepsexclude] += "toaster_image_dumpdata "
-
-do_rootfs[postfuncs] += "toaster_licensemanifest_dump "
-do_rootfs[vardepsexclude] += "toaster_licensemanifest_dump "
-
 do_populate_sdk[postfuncs] += "toaster_artifact_dumpdata "
 do_populate_sdk[vardepsexclude] += "toaster_artifact_dumpdata "
+
+do_populate_sdk_ext[postfuncs] += "toaster_artifact_dumpdata "
+do_populate_sdk_ext[vardepsexclude] += "toaster_artifact_dumpdata "
\ No newline at end of file
Signature · 1 line
1.9.1
author
#2 lore raw

PATCH 2/2 toaster.bbclass: only scan files-in-image.txt if it exists

From: Elliot Smith <[email protected]>

We can reach the method in toaster.bbclass which tries to read from the files-in-image.txt file via a build which doesn't create that file (e.g. "bitbake core-image-minimal -c rootfs"). This causes the build to fail with an exception.

Check that this file exists before trying to read from it.

[YOCTO #9784]

---
 classes/toaster.bbclass | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)
classes/toaster.bbclass +16 -9
index 90ea563..972efb9 100644
@@ -288,15 +288,22 @@ python toaster_buildhistory_dump() {
                             images[target][dependsname] = {'size': 0, 'depends' : []}
                         images[target][pname]['depends'].append((dependsname, deptype))
 
-            with open("%s/files-in-image.txt" % installed_img_path, "r") as fin:
-                for line in fin:
-                    lc = [ x for x in line.strip().split(" ") if len(x) > 0 ]
-                    if lc[0].startswith("l"):
-                        files[target]['syms'].append(lc)
-                    elif lc[0].startswith("d"):
-                        files[target]['dirs'].append(lc)
-                    else:
-                        files[target]['files'].append(lc)
+            # files-in-image.txt is only generated if an image file is created,
+            # so the file entries ('syms', 'dirs', 'files') for a target will be
+            # empty for rootfs builds and other "image" tasks which don't
+            # produce image files
+            # (e.g. "bitbake core-image-minimal -c populate_sdk")
+            files_in_image_path = "%s/files-in-image.txt" % installed_img_path
+            if os.path.exists(files_in_image_path):
+                with open(files_in_image_path, "r") as fin:
+                    for line in fin:
+                        lc = [ x for x in line.strip().split(" ") if len(x) > 0 ]
+                        if lc[0].startswith("l"):
+                            files[target]['syms'].append(lc)
+                        elif lc[0].startswith("d"):
+                            files[target]['dirs'].append(lc)
+                        else:
+                            files[target]['files'].append(lc)
 
             for pname in images[target]:
                 if not pname in allpkgs:
Signature · 1 line
1.9.1
#3 lore raw
In reply to bavery
Quoted 11 lines · bavery

This is part of a patch set that is split over the openembedded-core and
bitbake-devel mailing lists. This is a second version of the patch.

These patches remove complexity from the toaster.bbclass in order to move those
functions to bitbake/lib/toaster.

The overall intent of this set is to improve Toaster's ability to locate artifacts,
specify fstypes correctly, and add some tests for these changes.

FWIW you stripped a directory level off these patches so they didn't apply. Since I'd already merged the bitbake changes, I fixed these patches by hand.

Cheers,

Richard