Support template files for wic wks
cover Support template files for wic wks
From: Christopher Larson <[email protected]>
This adds support for .wks.in files as WKS_FILE.
These files are treated as the contents of a bitbake variable, so usual bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd end up largely mapping metadata variables to template fields anyway, which is a pointless indirection. Let bitbake expand the variables directly instead.
This feature lets us, for example, reference ${APPEND} in --append, and avoid hardcoding the serial console tty in the wks file, and let the user's changes to APPEND affect wic the way they do the other image construction mechanisms.
The template is read in and set in a variable at parse time, so changes to the variables referenced by the template will result in rebuilding the image.
It's been suggested that the name should better reflect how its contents are used, i.e. .bbin or so. I've also considered including something like 'bbv' or similar, to indicate its contents are that of a bitbake value. I think this should be discussed.
I'd love to hear questions or comments on this. I think the value is clear, and it's something we need to solve, but I'd like to hear if others agree or disagree on that, and whether this implementation is appropriate.
The following changes since commit 1a0e56630c5c27d8899dd0979ae0b86bbe227881:
utils.bbclass: note for deprecated base_contains (2016-04-29 07:53:58 +0100)
are available in the git repository at:
git://github.com/kergoth/openembedded-core wks.in
https://github.com/kergoth/openembedded-core/tree/wks.in
Christopher Larson (3):
image.bbclass: append to prefuncs/postfuncs for do_image_* image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM image_types.bbclass: support template .wks.in files for wic
meta/classes/image.bbclass | 4 ++-- meta/classes/image_types.bbclass | 38 +++++++++++++++++++++++++++++++++++++- 2 files changed, 39 insertions(+), 3 deletions(-)
Signature · 1 line
2.8.0
RFC PATCH 1/3 image.bbclass: append to prefuncs/postfuncs for do_image_*
From: Christopher Larson <[email protected]>
These are list style variables, so append to them rather than blowing away any existing value.
--- meta/classes/image.bbclass | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
meta/classes/image.bbclass +2 -2
@@ -436,9 +436,9 @@ python () { d.setVar('do_image_%s' % t, '\n'.join(cmds)) d.setVarFlag('do_image_%s' % t, 'func', '1') d.setVarFlag('do_image_%s' % t, 'fakeroot', '1') - d.setVarFlag('do_image_%s' % t, 'prefuncs', debug + 'set_image_size') - d.setVarFlag('do_image_%s' % t, 'postfuncs', 'create_symlinks') d.setVarFlag('do_image_%s' % t, 'subimages', ' '.join(subimages)) + d.appendVarFlag('do_image_%s' % t, 'prefuncs', ' ' + debug + 'set_image_size') + d.appendVarFlag('do_image_%s' % t, 'postfuncs', ' create_symlinks') d.appendVarFlag('do_image_%s' % t, 'vardeps', ' '.join(vardeps)) d.appendVarFlag('do_image_%s' % t, 'vardepsexclude', 'DATETIME')
Signature · 1 line
2.8.0
RFC PATCH 2/3 image_types.bbclass: add intermediate var for WKS_FILE_CHECKSUM
From: Christopher Larson <[email protected]>
This is a bit nicer to work with, and easier to override.
--- meta/classes/image_types.bbclass | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/classes/image_types.bbclass +2 -1
@@ -208,7 +208,8 @@ IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES" # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '', d)}" -do_image_wic[file-checksums] += "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}" +WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}" +do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}" EXTRA_IMAGECMD = ""
Signature · 1 line
2.8.0
RFC PATCH 3/3 image_types.bbclass: support template .wks.in files for wic
In reply to Christopher LarsonFrom: Christopher Larson <[email protected]>
These files are treated as the contents of a bitbake variable, so usual bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd end up largely mapping metadata variables to template fields anyway, which is a pointless indirection. Let bitbake expand the variables directly instead.
This feature lets us, for example, reference ${APPEND} in --append, and avoid hardcoding the serial console tty in the wks file, and let the user's changes to APPEND affect wic the way they do the other image construction mechanisms.
The template is read in and set in a variable at parse time, so changes to the variables referenced by the template will result in rebuilding the image.
--- meta/classes/image_types.bbclass | 35 +++++++++++++++++++++++++++++++++++ 1 file changed, 35 insertions(+)
meta/classes/image_types.bbclass +35 -0
@@ -206,6 +206,16 @@ IMAGE_CMD_wic () { } IMAGE_CMD_wic[vardepsexclude] = "WKS_FULL_PATH WKS_FILES" +python process_wks_template () { + """Write out expanded template contents to WKS_FULL_PATH.""" + template_body = d.getVar('_WKS_TEMPLATE', True) + if template_body: + wks_file = d.getVar('WKS_FULL_PATH', True) + with open(wks_file, 'w') as f: + f.write(template_body) +} +do_image_wic[prefuncs] += 'process_wks_template' + # Rebuild when the wks file or vars in WICVARS change USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${COMPRESSIONTYPES}'.split()), '1', '', d)}" WKS_FILE_CHECKSUM = "${@'${WKS_FULL_PATH}:%s' % os.path.exists('${WKS_FULL_PATH}') if '${USING_WIC}' else ''}"
@@ -298,3 +308,28 @@ IMAGE_TYPES_MASKED ?= "" # The WICVARS variable is used to define list of bitbake variables used in wic code # variables from this list is written to <image>.env file WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS" + +python () { + """Read in and set up wks file template for wic.""" + if d.getVar('USING_WIC', True): + wks_file_u = d.getVar('WKS_FULL_PATH', False) + wks_file = d.expand(wks_file_u) + base, ext = os.path.splitext(wks_file) + if ext == '.in' and os.path.exists(wks_file): + wks_out_file = os.path.join(d.getVar('WORKDIR', True), os.path.basename(base)) + d.setVar('WKS_FULL_PATH', wks_out_file) + d.setVar('WKS_TEMPLATE_PATH', wks_file_u) + d.setVar('WKS_FILE_CHECKSUM', '${WKS_TEMPLATE_PATH}:True') + + try: + with open(wks_file, 'r') as f: + body = f.read() + except (IOError, OSError) as exc: + pass + else: + # Previously, I used expandWithRefs to get the dependency list + # and add it to WICVARS, but there's no point re-parsing the + # file in process_wks_template as well, so just put it in + # a variable and let the metadata deal with the deps. + d.setVar('_WKS_TEMPLATE', body) +}
Signature · 1 line
2.8.0
Quoted 30 lines · Christopher Larson
From: Christopher Larson <[email protected]>
This adds support for .wks.in files as
WKS_FILE.These files are treated as the contents of a bitbake variable, so usual bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.It's been suggested that the name should better reflect how its contents are
used, i.e. .bbin or so. I've also considered including something like 'bbv' or
similar, to indicate its contents are that of a bitbake value. I think this should be discussed.
Quoted 4 lines
I'd love to hear questions or comments on this. I think the value is clear, and it's something we need to solve, but I'd like to hear if others agree or
disagree on that, and whether this implementation is appropriate.
One downside to this approach is, afaik, that it's the first serious deviation between wic outside and wic as an image type. One alternative would be to parse the template at parse time to inject vars into WICVARS, but let wic process the template, but then of course a change to the template without re-running bitabke wouldn't update WICVARS and the written env files.
Signature · 5 lines
Christopher Larson kergoth at gmail dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 17 lines · Christopher Larson
From: Christopher Larson <[email protected]>
This adds support for .wks.in files as
WKS_FILE.These files are treated as the contents of a bitbake variable, so usual bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd end up largely mapping metadata variables to template fields anyway, which is a pointless indirection. Let bitbake expand the variables directly instead.
This feature lets us, for example, reference ${APPEND} in --append, and avoid hardcoding the serial console tty in the wks file, and let the user's changes to APPEND affect wic the way they do the other image construction mechanisms.
The template is read in and set in a variable at parse time, so changes to the variables referenced by the template will result in rebuilding the image.
Quoted 9 lines
It's been suggested that the name should better reflect how its contents are used, i.e. .bbin or so. I've also considered including something like 'bbv' or similar, to indicate its contents are that of a bitbake value. I think this should be discussed.
I'd love to hear questions or comments on this. I think the value is clear, and it's something we need to solve, but I'd like to hear if others agree or disagree on that, and whether this implementation is appropriate.
The patchset looks good to me, but I'd also like to hear opinions of people more experienced in image creation internals.
Quoted 27 lines
The following changes since commit
1a0e56630c5c27d8899dd0979ae0b86bbe227881:utils.bbclass: note for deprecated
base_contains(2016-04-29 07:53:58 +0100)are available in the git repository at:
git://github.com/kergoth/openembedded-corewks.in
https://github.com/kergoth/openembedded-core/tree/wks.inChristopher Larson (3):
image.bbclass: append to prefuncs/postfuncs for do_image_* image_types.bbclass: add intermediate var forWKS_FILE_CHECKSUMimage_types.bbclass: support template .wks.in files for wicmeta/classes/image.bbclass | 4 ++-- meta/classes/image_types.bbclass | 38 +++++++++++++++++++++++++++++++++++++- 2 files changed, 39 insertions(+), 3 deletions(-)Signature · 7 lines
2.8.0 -- _______________________________________________ Openembedded-core mailing list [email protected] http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 3 lines
-- Regards, Ed
Le 02/05/2016 21:23, Christopher Larson a écrit :
Quoted 17 lines
From: Christopher Larson <[email protected]>
This adds support for .wks.in files as
WKS_FILE.These files are treated as the contents of a bitbake variable, so usual bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd end up largely mapping metadata variables to template fields anyway, which is a pointless indirection. Let bitbake expand the variables directly instead.
This feature lets us, for example, reference ${APPEND} in --append, and avoid hardcoding the serial console tty in the wks file, and let the user's changes to APPEND affect wic the way they do the other image construction mechanisms.
The template is read in and set in a variable at parse time, so changes to the variables referenced by the template will result in rebuilding the image.
Quoted 10 lines
It's been suggested that the name should better reflect how its contents are used, i.e. .bbin or so. I've also considered including something like 'bbv' or similar, to indicate its contents are that of a bitbake value. I think this should be discussed.
I'd love to hear questions or comments on this. I think the value is clear, and it's something we need to solve, but I'd like to hear if others agree or disagree on that, and whether this implementation is appropriate.
We have been discussing this a bit with Chris on IRC and i'll add a couple of thoughts we had so that discussion isn't completely lost.
As a final user of yocto, I find the need for templating in wic very usefull, it would allow me to specify disk size in the machine specification and adapt the wic image accordingly, or use different disk names ( mmcblk vs sda)
however my need is more generic and I think it would be interesting to consider a bigger problem : using yocto variables in configuration files. (I mean files in /etc/ here, not yocto configuration)
It is a pretty common problem to have to tailor configuration files for particular machines, distro features, optional dependencies etc... this is possible to do using OVERRIDES but it can only be done at the whole-file level, not within a configuration file. Moreover it makes some complex cases very tricky. let's present an example
It is pretty common to need to include the build-machine's ip address in a configuration file. This can be usefull to send logs if the build machine is also the developer's machine. The ip address of the build machine can be easily obtained via python and put into the bitbake environement, but using that variable to set a value in a random file in /etc is pretty tricky.
similarly, I like to include my public ssh key as an authorized key for root on my builds, il makes testing way easier
I think having a generic templating mechanism that would recognise some source files as needing template extension would be awesome. I don't understand fully how bitbake/yocto works, so I can't design a complete archtecture, but here are a couple of thought we had with Chris on IRC
* recognise *.in in the same way *.patch are recognized. Is there a risk of improper expansion ? maybe specify that templating is needed as a parameter in SRC_URI
* should expansion be in do_fetch or do_patch or somewhere else... images tend to skip do_patch, but do_fetch is on the bitbake side IIUC. would that be a problem ?
* interaction with sstate and rebuilding needs to be considered... changes need to be detected post-templating obviously
Overall, I really like the idea of this patch, there are lot of things in a .wks that need expansion, but I think there is even more in every other configuration file of the system.
Hope my random thoughts helped
Regards
Jérémy Rosen
Quoted 84 lines · Jeremy ROSEN
Quoted 32 lines · Christopher Larson
From: Christopher Larson <[email protected]>
This adds support for .wks.in files as
WKS_FILE.These files are treated as the contents of a bitbake variable, so usual bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.It's been suggested that the name should better reflect how its contents are
used, i.e. .bbin or so. I've also considered including something like 'bbv' or
similar, to indicate its contents are that of a bitbake value. I think this
should be discussed.Quoted 8 lines
I'd love to hear questions or comments on this. I think the value is clear,
and it's something we need to solve, but I'd like to hear if others agree or
disagree on that, and whether this implementation is appropriate.We have been discussing this a bit with Chris on IRC and i'll add a couple of thoughts we had so that discussion isn't completely lost.
As a final user of yocto, I find the need for templating in wic very usefull, it would allow me to specify disk size in the machine specification and adapt the wic image accordingly, or use different disk names ( mmcblk vs sda)
however my need is more generic and I think it would be interesting to consider a bigger problem : using yocto variables in configuration files. (I mean files in
/etc/here, not yocto configuration)It is a pretty common problem to have to tailor configuration files for particular machines, distro features, optional dependencies etc... this is possible to do using OVERRIDES but it can only be done at the whole-file level, not within a configuration file. Moreover it makes some complex cases very tricky. let's present an example
It is pretty common to need to include the build-machine's ip address in a configuration file. This can be usefull to send logs if the build machine is also the developer's machine. The ip address of the build machine can be easily obtained via python and put into the bitbake environement, but using that variable to set a value in a random file in /etc is pretty tricky.
similarly, I like to include my public ssh key as an authorized key for root on my builds, il makes testing way easier
I think having a generic templating mechanism that would recognise some source files as needing template extension would be awesome. I don't understand fully how bitbake/yocto works, so I can't design a complete archtecture, but here are a couple of thought we had with Chris on IRC
* recognise *.in in the same way *.patch are recognized. Is there a risk of improper expansion ? maybe specify that templating is needed as a parameter in
SRC_URI
* should expansion be indo_fetchordo_patchor somewhere else... images tend to skipdo_patch, butdo_fetchis on the bitbake side IIUC. would that be a problem ?
* interaction with sstate and rebuilding needs to be considered... changes need to be detected post-templating obviously
Quoted 5 lines
Overall, I really like the idea of this patch, there are lot of things in a .wks that need expansion, but I think there is even more in every other configuration file of the system.
Hope my random thoughts helped
This is an interesting concept, certainly a generic templating mechanism for source files may be of use. I expect we could get this merged, and open a new yocto feature request in bugzilla to add a more generic mechanism, possibly based on this sort of thing.
The main concern I'd have is making sure changes to the referenced variables, or changes to the list of referenced variables, changes appropriate checksums, so as was the case in this patch series, they'd have to be read in at parse time to gather that info. As long as the files are relatively small, it should be reasonable to do so, and I expect the common case will be small config files, so I doubt it'd be a blocking issue.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 38 lines · Christopher Larson
Quoted 34 lines · Jeremy ROSEN
Quoted 32 lines · Christopher Larson
From: Christopher Larson <[email protected]>
This adds support for .wks.in files as
WKS_FILE.These files are treated as the contents of a bitbake variable, so usual bitbake variable references are supported. I considered using another templating mechanism, for example the one used by yocto-layer, but then we'd
end up largely mapping metadata variables to template fields anyway, which is
a pointless indirection. Let bitbake expand the variables directly instead.This feature lets us, for example, reference ${APPEND} in --append, and avoid
hardcoding the serial console tty in the wks file, and let the user's changes
to APPEND affect wic the way they do the other image construction mechanisms.The template is read in and set in a variable at parse time, so changes to the
variables referenced by the template will result in rebuilding the image.It's been suggested that the name should better reflect how its contents are
used, i.e. .bbin or so. I've also considered including something like 'bbv' or
similar, to indicate its contents are that of a bitbake value. I think this
should be discussed.
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
I'd love to hear questions or comments on this. I think the value is clear,
and it's something we need to solve, but I'd like to hear if others agree or
disagree on that, and whether this implementation is appropriate.
Given that there don't seem to be any real objections here, I'm going to re-submit this as a non-RFC patch series to try to get it merged. If anyone has any issues adding this feature, let me know. Thanks for the feedback.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics