Support template files for wic

From: Christopher Larson <[email protected]>

This adds support for .wks.in files as WKS_FILE.

This is a re-send of the RFC PATCH series as a regular PATCH series, to attempt to get the feature merged. There's a clear value-add, there were no major objections to the RFC series, and further improvements can be added on from here.

These files are treated as the contents of a bitbake variable, so usual bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd end up largely mapping metadata variables to template fields anyway, which is a pointless indirection. Let bitbake expand the variables directly instead.

This feature lets us, for example, reference ${APPEND} in --append, and avoid hardcoding the serial console tty in the wks file, and let the user's changes to APPEND affect wic the way they do the other image construction mechanisms.

The template is read in and set in a variable at parse time, so changes to the variables referenced by the template will result in rebuilding the image.

The following changes since commit 646c366c2566bd8dd6f73681cea9f5b021589a56:

gst-player: upgrade to latest HEAD (2016-06-27 14:08:37 +0100)

are available in the git repository at:

git://github.com/kergoth/openembedded-core wks.in2
https://github.com/kergoth/openembedded-core/tree/wks.in2

Christopher Larson (2):
image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM image_types.bbclass: support template .wks.in files for wic

 meta/classes/image_types.bbclass | 49 +++++++++++++++++++++++++++++++++++++++-
 1 file changed, 48 insertions(+), 1 deletion(-)

2.8.0

From: Christopher Larson <[email protected]>

This is a bit nicer to work with, and easier to override.

---
 meta/classes/image_types.bbclass | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

index 72e8ed3..f7de756 100644
@@ -215,7 +215,8 @@ IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES"
 
 # Rebuild when the wks file or vars in WICVARS change
 USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '', d)}"
-do_image_wic[file-checksums] += "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
+WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
+do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
 
 EXTRA_IMAGECMD = ""
 

2.8.0

From: Christopher Larson <[email protected]>

These files are treated as the contents of a bitbake variable, so usual bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd end up largely mapping metadata variables to template fields anyway, which is a pointless indirection. Let bitbake expand the variables directly instead.

This feature lets us, for example, reference ${APPEND} in --append, and avoid hardcoding the serial console tty in the wks file, and let the user's changes to APPEND affect wic the way they do the other image construction mechanisms.

The template is read in and set in a variable at parse time, so changes to the variables referenced by the template will result in rebuilding the image.

---
 meta/classes/image_types.bbclass | 46 ++++++++++++++++++++++++++++++++++++++++
 1 file changed, 46 insertions(+)

index f7de756..2b97397 100644
@@ -218,6 +218,52 @@ USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s
 WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
 do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}"
 
+python do_write_wks_template () {
+    """Write out expanded template contents to WKS_FULL_PATH."""
+    import re
+
+    template_body = d.getVar('_WKS_TEMPLATE', True)
+
+    # Remove any remnant variable references left behind by the expansion
+    # due to undefined variables
+    expand_var_regexp = re.compile(r"\${[^{}@\n\t :]+}")
+    while True:
+        new_body = re.sub(expand_var_regexp, '', template_body)
+        if new_body == template_body:
+            break
+        else:
+            template_body = new_body
+
+    wks_file = d.getVar('WKS_FULL_PATH', True)
+    with open(wks_file, 'w') as f:
+        f.write(template_body)
+}
+
+python () {
+    if d.getVar('USING_WIC', True):
+        wks_file_u = d.getVar('WKS_FULL_PATH', False)
+        wks_file = d.expand(wks_file_u)
+        base, ext = os.path.splitext(wks_file)
+        if ext == '.in' and os.path.exists(wks_file):
+            wks_out_file = os.path.join(d.getVar('WORKDIR', True), os.path.basename(base))
+            d.setVar('WKS_FULL_PATH', wks_out_file)
+            d.setVar('WKS_TEMPLATE_PATH', wks_file_u)
+            d.setVar('WKS_FILE_CHECKSUM', '${WKS_TEMPLATE_PATH}:True')
+
+            try:
+                with open(wks_file, 'r') as f:
+                    body = f.read()
+            except (IOError, OSError) as exc:
+                pass
+            else:
+                # Previously, I used expandWithRefs to get the dependency list
+                # and add it to WICVARS, but there's no point re-parsing the
+                # file in process_wks_template as well, so just put it in
+                # a variable and let the metadata deal with the deps.
+                d.setVar('_WKS_TEMPLATE', body)
+                bb.build.addtask('do_write_wks_template', 'do_image_wic', None, d)
+}
+
 EXTRA_IMAGECMD = ""
 
 inherit siteinfo

2.8.0

v2 changes: simplified by dropping the prefuncs bits.

Christopher Larson
kergoth at gmail dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics

Thank you for the patchset!

A bit unrelated note. You seem to know wic functionality in meta/classes/image* better than anyone else. Would you mind adding this knowledge to wic documentation in development manual?

--
Regards,
Ed