Support template files for wic
cover Support template files for wic
From: Christopher Larson <[email protected]>
This adds support for .wks.in files as WKS_FILE.
This is a re-send of the RFC PATCH series as a regular PATCH series, to attempt to get the feature merged. There's a clear value-add, there were no major objections to the RFC series, and further improvements can be added on from here.
These files are treated as the contents of a bitbake variable, so usual bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd end up largely mapping metadata variables to template fields anyway, which is a pointless indirection. Let bitbake expand the variables directly instead.
This feature lets us, for example, reference ${APPEND} in --append, and avoid hardcoding the serial console tty in the wks file, and let the user's changes to APPEND affect wic the way they do the other image construction mechanisms.
The template is read in and set in a variable at parse time, so changes to the variables referenced by the template will result in rebuilding the image.
It's been suggested that the name should better reflect how its contents are used, i.e. .bbin or so. I've also considered including something like 'bbv' or similar, to indicate its contents are that of a bitbake value. I think this should be discussed.
I'd love to hear questions or comments on this. I think the value is clear, and it's something we need to solve, but I'd like to hear if others agree or disagree on that, and whether this implementation is appropriate.
The following changes since commit f6b0b260ce18a30d04edfb0afb7942b9f9a5480b:
lib/oe/rootfs: Fix DEBUGFS generation, without openssl (2016-05-20 10:20:42 +0100)
are available in the git repository at:
git://github.com/kergoth/openembedded-core wks.in
https://github.com/kergoth/openembedded-core/tree/wks.in
Christopher Larson (3):
image.bbclass: append to prefuncs/postfuncs for do_image_* image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM image_types.bbclass: support template .wks.in files for wic
meta/classes/image.bbclass | 4 ++-- meta/classes/image_types.bbclass | 38 +++++++++++++++++++++++++++++++++++++- 2 files changed, 39 insertions(+), 3 deletions(-)
Signature · 1 line
2.8.0
PATCH 1/3 image.bbclass: append to prefuncs/postfuncs for do_image_*
From: Christopher Larson <[email protected]>
These are list style variables, so append to them rather than blowing away any existing value.
--- meta/classes/image.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/image.bbclass +2 -2
@@ -436,9 +436,9 @@ python () { d.setVar('do_image_%s' % t, '\n'.join(cmds)) d.setVarFlag('do_image_%s' % t, 'func', '1') d.setVarFlag('do_image_%s' % t, 'fakeroot', '1') - d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size') - d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks') d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages)) + d.appendVarFlag('do_image_%s' % t, 'prefuncs', ' ' + debug + 'set_image_size') + d.appendVarFlag('do_image_%s' % t, 'postfuncs', ' create_symlinks') d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps)) d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME')
Signature · 1 line
2.8.0
PATCH 2/3 image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM
From: Christopher Larson <[email protected]>
This is a bit nicer to work with, and easier to override.
--- meta/classes/image_types.bbclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes/image_types.bbclass +2 -1
@@ -208,7 +208,8 @@ IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES" # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '', d)}" -do_image_wic[file-checksums] += "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}" +WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}" +do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}" EXTRA_IMAGECMD = ""
Signature · 1 line
2.8.0
PATCH 3/3 image_types.bbclass: support template .wks.in files for wic
In reply to Christopher LarsonFrom: Christopher Larson <[email protected]>
These files are treated as the contents of a bitbake variable, so usual bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd end up largely mapping metadata variables to template fields anyway, which is a pointless indirection. Let bitbake expand the variables directly instead.
This feature lets us, for example, reference ${APPEND} in --append, and avoid hardcoding the serial console tty in the wks file, and let the user's changes to APPEND affect wic the way they do the other image construction mechanisms.
The template is read in and set in a variable at parse time, so changes to the variables referenced by the template will result in rebuilding the image.
--- meta/classes/image_types.bbclass | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
meta/classes/image_types.bbclass +35 -0
@@ -206,6 +206,16 @@ IMAGE_CMD_wic () { } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES" +python process_wks_template () { + """Write out expanded template contents to WKS_FULL_PATH.""" + template_body = d.getVar('_WKS_TEMPLATE', True) + if template_body: + wks_file = d.getVar('WKS_FULL_PATH', True) + with open(wks_file, 'w') as f: + f.write(template_body) +} +do_image_wic[prefuncs] += 'process_wks_template' + # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '', d)}" WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
@@ -302,3 +312,28 @@ IMAGE_TYPES_MASKED ?= "" # The WICVARS variable is used to define list of bitbake variables used in wic code # variables from this list is written to <image>.env file WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS" + +python () { + """Read in and set up wks file template for wic.""" + if d.getVar('USING_WIC', True): + wks_file_u = d.getVar('WKS_FULL_PATH', False) + wks_file = d.expand(wks_file_u) + base, ext = os.path.splitext(wks_file) + if ext == '.in' and os.path.exists(wks_file): + wks_out_file = os.path.join(d.getVar('WORKDIR', True), os.path.basename(base)) + d.setVar('WKS_FULL_PATH', wks_out_file) + d.setVar('WKS_TEMPLATE_PATH', wks_file_u) + d.setVar('WKS_FILE_CHECKSUM', '${WKS_TEMPLATE_PATH}:True') + + try: + with open(wks_file, 'r') as f: + body = f.read() + except (IOError, OSError) as exc: + pass + else: + # Previously, I used expandWithRefs to get the dependency list + # and add it to WICVARS, but there's no point re-parsing the + # file in process_wks_template as well, so just put it in + # a variable and let the metadata deal with the deps. + d.setVar('_WKS_TEMPLATE', body) +}
Signature · 1 line
2.8.0
Re: [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic
Quoted 77 lines · Christopher Larson
From: Christopher Larson <[email protected]>
These files are treated as the contents of a bitbake variable, so usual
bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.Signed-off-by Christopher Larson <[email protected]>--- meta/classes/image_types.bbclass | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)meta/classes/image_types.bbclass +10
-0@@ -206,6 +206,16 @@ IMAGE_CMD_wic () { } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES" +python process_wks_template () { + """Write out expanded template contents to WKS_FULL_PATH.""" + template_body = d.getVar('_WKS_TEMPLATE', True) + if template_body: + wks_file = d.getVar('WKS_FULL_PATH', True) + with open(wks_file, 'w') as f: + f.write(template_body) +} +do_image_wic[prefuncs] += 'process_wks_template' + # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '',d)}"
WKS_FILE_CHECKSUM= "${@'${WKS_FULL_PATH}:%s' %os.path.exists('${WKS_FULL_PATH}')if '${USING_WIC}' else ''}"meta/classes/image_types.bbclass +0
-0@@ -302,3 +312,28 @@ IMAGE_TYPES_MASKED ?= "" # The WICVARS variable is used to define list of bitbake variablesused in wic code
# variables from this list is written to <image>.env file WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"meta/classes/image_types.bbclass +9
-0+ +python () { + """Read in and set up wks file template for wic.""" + if d.getVar('USING_WIC', True): + wks_file_u = d.getVar('WKS_FULL_PATH', False) + wks_file = d.expand(wks_file_u) + base, ext = os.path.splitext(wks_file) + if ext == '.in' and os.path.exists(wks_file): + wks_out_file = os.path.join(d.getVar('WORKDIR', True),
os.path.basename(base))meta/classes/image_types.bbclass +3
-0+ d.setVar('WKS_FULL_PATH', wks_out_file) + d.setVar('WKS_TEMPLATE_PATH', wks_file_u) + d.setVar('WKS_FILE_CHECKSUM','${WKS_TEMPLATE_PATH}:True')
meta/classes/image_types.bbclass IMAGE_TYPES_MASKED ?= "" · line 332
Quoted 6 lines
meta/classes/image_types.bbclass +6
-0+ + try: + with open(wks_file, 'r') as f: + body = f.read() + except (IOError, OSError) as exc: + pass
I'm a little nervous about determinism here. Shouldn't it either find the referenced file or error?
Cheers,
Richard
Re: [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic
Quoted 79 lines · Richard Purdie
Quoted 78 lines · Christopher Larson
From: Christopher Larson <[email protected]>
These files are treated as the contents of a bitbake variable, so usual
bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.Signed-off-by Christopher Larson <[email protected]>--- meta/classes/image_types.bbclass | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)meta/classes/image_types.bbclass +10
-0@@ -206,6 +206,16 @@ IMAGE_CMD_wic () { } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES" +python process_wks_template () { + """Write out expanded template contents to WKS_FULL_PATH.""" + template_body = d.getVar('_WKS_TEMPLATE', True) + if template_body: + wks_file = d.getVar('WKS_FULL_PATH', True) + with open(wks_file, 'w') as f: + f.write(template_body) +} +do_image_wic[prefuncs] += 'process_wks_template' + # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '',d)}"
WKS_FILE_CHECKSUM= "${@'${WKS_FULL_PATH}:%s' %os.path.exists('${WKS_FULL_PATH}')if '${USING_WIC}' else ''}"meta/classes/image_types.bbclass +0
-0@@ -302,3 +312,28 @@ IMAGE_TYPES_MASKED ?= "" # The WICVARS variable is used to define list of bitbake variablesused in wic code
# variables from this list is written to <image>.env file WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"meta/classes/image_types.bbclass +9
-0+ +python () { + """Read in and set up wks file template for wic.""" + if d.getVar('USING_WIC', True): + wks_file_u = d.getVar('WKS_FULL_PATH', False) + wks_file = d.expand(wks_file_u) + base, ext = os.path.splitext(wks_file) + if ext == '.in' and os.path.exists(wks_file): + wks_out_file = os.path.join(d.getVar('WORKDIR', True),
os.path.basename(base))meta/classes/image_types.bbclass +3
-0+ d.setVar('WKS_FULL_PATH', wks_out_file) + d.setVar('WKS_TEMPLATE_PATH', wks_file_u) + d.setVar('WKS_FILE_CHECKSUM','${WKS_TEMPLATE_PATH}:True')
+
Quoted 8 lines
meta/classes/image_types.bbclass IMAGE_TYPES_MASKED ?= "" · line 332
Quoted 5 lines
meta/classes/image_types.bbclass +5
-0+ try: + with open(wks_file, 'r') as f: + body = f.read() + except (IOError, OSError) as exc: + passI'm a little nervous about determinism here. Shouldn't it either find the referenced file or error?
That's a fair point, I think I was concerned about parse time errors, since they tend to blow up into a lot of unpleasantness for the user, since parse time errors in classes can cause errors for multiple recipes. If no file is read in at parse time, there'll be no contents to write out, and we could then error when trying to build the image with wic, as no wks file will exist.
Of course, the file being missing will change checksums for the image tasks, but I think that's reasonable.
Thoughts?
Signature · 5 lines
Christopher Larson kergoth at gmail dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Re: [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic
Quoted 100 lines · Christopher Larson
Quoted 79 lines · Richard Purdie
Quoted 78 lines · Christopher Larson
From: Christopher Larson <[email protected]>
These files are treated as the contents of a bitbake variable, so usual
bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.Signed-off-by Christopher Larson <[email protected]>--- meta/classes/image_types.bbclass | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)meta/classes/image_types.bbclass +10
-0@@ -206,6 +206,16 @@ IMAGE_CMD_wic () { } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES" +python process_wks_template () { + """Write out expanded template contents to WKS_FULL_PATH.""" + template_body = d.getVar('_WKS_TEMPLATE', True) + if template_body: + wks_file = d.getVar('WKS_FULL_PATH', True) + with open(wks_file, 'w') as f: + f.write(template_body) +} +do_image_wic[prefuncs] += 'process_wks_template' + # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '',d)}"
WKS_FILE_CHECKSUM= "${@'${WKS_FULL_PATH}:%s' %os.path.exists('${WKS_FULL_PATH}')if '${USING_WIC}' else ''}"meta/classes/image_types.bbclass +0
-0@@ -302,3 +312,28 @@ IMAGE_TYPES_MASKED ?= "" # The WICVARS variable is used to define list of bitbake variablesused in wic code
# variables from this list is written to <image>.env file WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"meta/classes/image_types.bbclass +9
-0+ +python () { + """Read in and set up wks file template for wic.""" + if d.getVar('USING_WIC', True): + wks_file_u = d.getVar('WKS_FULL_PATH', False) + wks_file = d.expand(wks_file_u) + base, ext = os.path.splitext(wks_file) + if ext == '.in' and os.path.exists(wks_file): + wks_out_file = os.path.join(d.getVar('WORKDIR', True),
os.path.basename(base))meta/classes/image_types.bbclass +3
-0+ d.setVar('WKS_FULL_PATH', wks_out_file) + d.setVar('WKS_TEMPLATE_PATH', wks_file_u) + d.setVar('WKS_FILE_CHECKSUM','${WKS_TEMPLATE_PATH}:True')
+Quoted 8 lines
meta/classes/image_types.bbclass IMAGE_TYPES_MASKED ?= "" · line 332
Quoted 5 lines
meta/classes/image_types.bbclass +5
-0+ try: + with open(wks_file, 'r') as f: + body = f.read() + except (IOError, OSError) as exc: + passI'm a little nervous about determinism here. Shouldn't it either find the referenced file or error?
That's a fair point, I think I was concerned about parse time errors, since they tend to blow up into a lot of unpleasantness for the user, since parse time errors in classes can cause errors for multiple recipes. If no file is read in at parse time, there'll be no contents to write out, and we could then error when trying to build the image with wic, as no wks file will exist.
Quoted 5 lines
Of course, the file being missing will change checksums for the image tasks, but I think that's reasonable.
Thoughts?
I'll do a bit more testing on the behaviors and get back to you on that later. Thanks.
Signature · 5 lines
Christopher Larson kergoth at gmail dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Re: [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic
Quoted 104 lines · Christopher Larson
Quoted 100 lines · Christopher Larson
Quoted 79 lines · Richard Purdie
Quoted 78 lines · Christopher Larson
From: Christopher Larson <[email protected]>
These files are treated as the contents of a bitbake variable, so usual
bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.Signed-off-by Christopher Larson <[email protected]>--- meta/classes/image_types.bbclass | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)meta/classes/image_types.bbclass +10
-0@@ -206,6 +206,16 @@ IMAGE_CMD_wic () { } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES" +python process_wks_template () { + """Write out expanded template contents to WKS_FULL_PATH.""" + template_body = d.getVar('_WKS_TEMPLATE', True) + if template_body: + wks_file = d.getVar('WKS_FULL_PATH', True) + with open(wks_file, 'w') as f: + f.write(template_body) +} +do_image_wic[prefuncs] += 'process_wks_template' + # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '',d)}"
WKS_FILE_CHECKSUM= "${@'${WKS_FULL_PATH}:%s' %os.path.exists('${WKS_FULL_PATH}')if '${USING_WIC}' else ''}"meta/classes/image_types.bbclass +0
-0@@ -302,3 +312,28 @@ IMAGE_TYPES_MASKED ?= "" # The WICVARS variable is used to define list of bitbake variablesused in wic code
# variables from this list is written to <image>.env file WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"meta/classes/image_types.bbclass +9
-0+ +python () { + """Read in and set up wks file template for wic.""" + if d.getVar('USING_WIC', True): + wks_file_u = d.getVar('WKS_FULL_PATH', False) + wks_file = d.expand(wks_file_u) + base, ext = os.path.splitext(wks_file) + if ext == '.in' and os.path.exists(wks_file): + wks_out_file = os.path.join(d.getVar('WORKDIR', True),
os.path.basename(base))meta/classes/image_types.bbclass +3
-0+ d.setVar('WKS_FULL_PATH', wks_out_file) + d.setVar('WKS_TEMPLATE_PATH', wks_file_u) + d.setVar('WKS_FILE_CHECKSUM','${WKS_TEMPLATE_PATH}:True')
+Quoted 8 lines
meta/classes/image_types.bbclass IMAGE_TYPES_MASKED ?= "" · line 332
Quoted 5 lines
meta/classes/image_types.bbclass +5
-0+ try: + with open(wks_file, 'r') as f: + body = f.read() + except (IOError, OSError) as exc: + passI'm a little nervous about determinism here. Shouldn't it either find the referenced file or error?
That's a fair point, I think I was concerned about parse time errors, since they tend to blow up into a lot of unpleasantness for the user, since parse time errors in classes can cause errors for multiple recipes. If no file is read in at parse time, there'll be no contents to write out, and we could then error when trying to build the image with wic, as no wks file will exist.
Quoted 8 lines
Quoted 5 lines
Of course, the file being missing will change checksums for the image tasks, but I think that's reasonable.
Thoughts?
I'll do a bit more testing on the behaviors and get back to you on that later. Thanks.
If the WKS_FILE doesn't exist, whether we're using templates or not, it'll fail in do_image_wic:
ERROR: core-image-sato-1.0-r0 do_image_wic: No kickstart files fromWKS_FILES were found: /scratch/dogwood/minnowmax/build/mkefidiskf.wks.in
core-image-sato.wks. Please set WKS_FILE or WKS_FILES appropriately.
ERROR: core-image-sato-1.0-r0 do_image_wic: Function failed: do_image_wic
(log file is located at /scratch/dogwood/minnowmax/build/tmp/work/intel_corei7_64-mel-linux/core-image-sato/1.0-r0/temp/log.do_image_wic.12181)
If the template doesn't exist, then no template handling will be done at all, so do_image_wic won't depend on any of the variables referenced in the template, and the task will fail. If it does, then we will depend on its contents and referenced variables, and the task will not fail.
I don't really see a problem with determinism there. Either the file exists at parse time or it doesn't, and the behavior seems reasonable in either case.
Signature · 5 lines
Christopher Larson kergoth at gmail dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Re: [PATCH 3/3] image_types.bbclass: support template .wks.in files for wic
Quoted 128 lines · Christopher Larson
Quoted 104 lines · Christopher Larson
Quoted 100 lines · Christopher Larson
Quoted 79 lines · Richard Purdie
Quoted 78 lines · Christopher Larson
From: Christopher Larson <[email protected]>
These files are treated as the contents of a bitbake variable, so usual
bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.Signed-off-by Christopher Larson <[email protected]>--- meta/classes/image_types.bbclass | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)meta/classes/image_types.bbclass +10
-0@@ -206,6 +206,16 @@ IMAGE_CMD_wic () { } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES" +python process_wks_template () { + """Write out expanded template contents to WKS_FULL_PATH.""" + template_body = d.getVar('_WKS_TEMPLATE', True) + if template_body: + wks_file = d.getVar('WKS_FULL_PATH', True) + with open(wks_file, 'w') as f: + f.write(template_body) +} +do_image_wic[prefuncs] += 'process_wks_template' + # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '',d)}"
WKS_FILE_CHECKSUM= "${@'${WKS_FULL_PATH}:%s' %os.path.exists('${WKS_FULL_PATH}')if '${USING_WIC}' else ''}"meta/classes/image_types.bbclass +0
-0@@ -302,3 +312,28 @@ IMAGE_TYPES_MASKED ?= "" # The WICVARS variable is used to define list of bitbake variablesused in wic code
# variables from this list is written to <image>.env file WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"meta/classes/image_types.bbclass +9
-0+ +python () { + """Read in and set up wks file template for wic.""" + if d.getVar('USING_WIC', True): + wks_file_u = d.getVar('WKS_FULL_PATH', False) + wks_file = d.expand(wks_file_u) + base, ext = os.path.splitext(wks_file) + if ext == '.in' and os.path.exists(wks_file): + wks_out_file = os.path.join(d.getVar('WORKDIR', True),
os.path.basename(base))meta/classes/image_types.bbclass +3
-0+ d.setVar('WKS_FULL_PATH', wks_out_file) + d.setVar('WKS_TEMPLATE_PATH', wks_file_u) + d.setVar('WKS_FILE_CHECKSUM','${WKS_TEMPLATE_PATH}:True')
+Quoted 8 lines
meta/classes/image_types.bbclass IMAGE_TYPES_MASKED ?= "" · line 332
Quoted 5 lines
meta/classes/image_types.bbclass +5
-0+ try: + with open(wks_file, 'r') as f: + body = f.read() + except (IOError, OSError) as exc: + passI'm a little nervous about determinism here. Shouldn't it either find the referenced file or error?
That's a fair point, I think I was concerned about parse time errors, since they tend to blow up into a lot of unpleasantness for the user, since parse time errors in classes can cause errors for multiple recipes. If no file is read in at parse time, there'll be no contents to write out, and we could then error when trying to build the image with wic, as no wks file will exist.
Quoted 8 lines
Quoted 5 lines
Of course, the file being missing will change checksums for the image tasks, but I think that's reasonable.
Thoughts?
I'll do a bit more testing on the behaviors and get back to you on that later. Thanks.
If the
WKS_FILEdoesn't exist, whether we're using templates or not, it'll fail indo_image_wic:ERROR: core-image-sato-1.0-r0
do_image_wic: No kickstart files fromWKS_FILESwere found: /scratch/dogwood/minnowmax/build/mkefidiskf.wks.in
core-image-sato.wks. Please setWKS_FILEorWKS_FILESappropriately.
ERROR: core-image-sato-1.0-r0do_image_wic: Function failed:do_image_wic
(log file is located at /scratch/dogwood/minnowmax/build/tmp/work/intel_corei7_64-mel-linux/core-image-sato/1.0-r0/temp/log.do_image_wic.12181)
Quoted 9 lines
If the template doesn't exist, then no template handling will be done at all, so
do_image_wicwon't depend on any of the variables referenced in the template, and the task will fail. If it does, then we will depend on its contents and referenced variables, and the task will not fail.I don't really see a problem with determinism there. Either the file exists at parse time or it doesn't, and the behavior seems reasonable in either case.
Update: I've occasionally seen a taskhash mismatch, so clearly I've missed something. Hold off on this until further notice. Thanks again for the feedback.
Signature · 5 lines
Christopher Larson kergoth at gmail dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics