Build wic images with bitbake
v2 cover Build wic images with bitbake
Hi Reviewers,
This patchset adds new image type 'wic' to the list of image types supported by bitbake. This should allow to build partitioned image the same way as any other images.
New image type can be used in image recipes almost the same way as other image types. There is just one difference - <image>[.<machine>].wks file should be put to the same location as image recipe.
To enable this functionality I've implemented generating .env files with bitbake variables and reading them by wic. This seems to be the only way to get bitbake variables in wic when it's run by bitbake as running 'bitbake -e' is not possible as bitbake is locked.
Test cases for new functionality and small improvements are also included into this patchset.
Changes in V2:
- Fixed typo in the comment. Thanks Mario!
- Rebased on fresh master
The following changes since commit bb34181b42b62e0476ba9e12f9125ed9f785cf23:
bitbake: lib/bb/main: avoid importing all server/UI modules on every execution (2015-08-24 23:47:07 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib ed/wic/wic-image-type
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/wic-image-type
Ed Bartosh (17):
image.py: write bitbake variables to .env file
oe-selftest: test generation of <image>.env
wic: remove undescore from function name
wic: add BitbakeVars class
wic: create new method _parse_line
wic: add default_image attribute to BitbakeVars
wic: set default image
wic: implement getting variables from .env files
wic: implement --vars option
wic: rename variable
wic: deferred call of hlp.get_wic_plugins_help() image_types.bbclass: add wic image type
wic-image-minimal: add wic image recipe and .wks
oe-selftest: test building wic image by bitbake image.py: set bitbake variable ROOTFS_SIZE
wic: use bitbake variable ROOTFS_SIZE image.py: add script output to the rootfs log
meta/classes/image_types.bbclass | 12 ++ meta/lib/oe/image.py | 32 +++++- meta/lib/oeqa/selftest/wic.py | 33 ++++++ meta/recipes-extended/images/wic-image-minimal.bb | 14 +++ meta/recipes-extended/images/wic-image-minimal.wks | 10 ++ scripts/lib/image/help.py | 6 +- scripts/lib/wic/imager/direct.py | 14 +++ scripts/lib/wic/utils/oe/misc.py | 125 ++++++++++++++------- scripts/wic | 14 ++- 9 files changed, 212 insertions(+), 48 deletions(-) create mode 100644 meta/recipes-extended/images/wic-image-minimal.bb create mode 100644 meta/recipes-extended/images/wic-image-minimal.wks
--
Regards,
Ed
PATCH v2 1/17 image.py: write bitbake variables to .env file
Write set of bitbake variables associated with the image into build/tmp/sysroots/<machine>/imagedata/<image>.env
This is needed for wic to be able to get bitbake variables without running 'bitbake -e'.
--- meta/lib/oe/image.py | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
meta/lib/oe/image.py +23 -0
@@ -321,6 +321,27 @@ class Image(ImageDepGraph): return image_cmd_groups + def _write_env(self): + """ + Write environment variables + to tmp/sysroots/<machine>/imgdata/<image>.env + """ + stdir = self.d.getVar('STAGING_DIR_TARGET', True) + outdir = os.path.join(stdir, 'imgdata') + if not os.path.exists(outdir): + os.makedirs(outdir) + basename = self.d.getVar('IMAGE_BASENAME', True) + with open(os.path.join(outdir, basename) + '.env', 'w') as envf: + for var in sorted(self.d.keys()): + # filter out as much as we can to reduce file size + if var.startswith('_') or var.startswith('BB_') \ + or not var.isupper() or self.d.getVarFlag(var, "func") \ + or var in ('BBINCLUDED', 'SRCPV', 'MIRRORS'): + continue + value = self.d.getVar(var, True) + if value: + envf.write('%s="%s"\n' % (var, value.strip())) + def create(self): bb.note("###### Generate images #######") pre_process_cmds = self.d.getVar("IMAGE_PREPROCESS_COMMAND", True)
@@ -332,6 +353,8 @@ class Image(ImageDepGraph): image_cmd_groups = self._get_imagecmds() + self._write_env() + for image_cmds in image_cmd_groups: # create the images in parallel nproc = multiprocessing.cpu_count()
Signature · 1 line
2.1.4
PATCH v2 2/17 oe-selftest: test generation of <image>.env
Added test case to check if <image>.env file is generated and contains bitbake variables used in wic code.
--- meta/lib/oeqa/selftest/wic.py | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+)
meta/lib/oeqa/selftest/wic.py +19 -0
@@ -23,6 +23,7 @@ """Test cases for wic.""" +import os import sys from glob import glob
@@ -168,3 +169,21 @@ class Wic(oeSelfTest): self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-*.direct"))) self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-*.iso"))) + def test19_image_env(self): + """Test generation of <image>.env files.""" + image = 'core-image-minimal' + stdir = get_bb_var('STAGING_DIR_TARGET', image) + imgdatadir = os.path.join(stdir, 'imgdata') + basename = get_bb_var('IMAGE_BASENAME', image) + self.assertEqual(basename, image) + path = os.path.join(imgdatadir, basename) + '.env' + self.assertTrue(os.path.isfile(path)) + with open(path) as envfile: + content = dict(line.split("=", 1) for line in envfile) + # test if variables used by wic present in the .env file + for var in ('IMAGE_ROOTFS', 'DEPLOY_DIR_IMAGE', 'BBLAYERS', + 'STAGING_DIR_NATIVE', 'STAGING_DATADIR', 'TARGET_SYS', + 'IMAGE_BASENAME', 'INITRAMFS_FSTYPES', 'MACHINE_ARCH', + 'IMAGE_LINK_NAME', 'STAGING_LIBDIR', 'ROOTFS_SIZE'): + self.assertTrue(var in content) + self.assertTrue(content[var])
Signature · 1 line
2.1.4
PATCH v2 3/17 wic: remove undescore from function name
In reply to Ed BartoshRenamed __exec_cmd -> _exec_cmd as double underscores cause strange behaviour when function is called in class method. Python complains that __exec_cmd method(!!!) of the same class doesn't exist.
--- scripts/lib/wic/utils/oe/misc.py | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-)
scripts/lib/wic/utils/oe/misc.py +6 -6
@@ -46,13 +46,13 @@ NATIVE_RECIPES = {"mcopy": "mtools", "syslinux": "syslinux" } -def __exec_cmd(cmd_and_args, as_shell=False, catch=3): +def _exec_cmd(cmd_and_args, as_shell=False, catch=3): """ Execute command, catching stderr, stdout Need to execute as_shell if the command uses wildcards """ - msger.debug("__exec_cmd: %s" % cmd_and_args) + msger.debug("_exec_cmd: %s" % cmd_and_args) args = cmd_and_args.split() msger.debug(args)
@@ -61,7 +61,7 @@ def __exec_cmd(cmd_and_args, as_shell=False, catch=3): else: ret, out = runner.runtool(args, catch) out = out.strip() - msger.debug("__exec_cmd: output for %s (rc = %d): %s" % \ + msger.debug("_exec_cmd: output for %s (rc = %d): %s" % \ (cmd_and_args, ret, out)) return (ret, out)
@@ -73,7 +73,7 @@ def exec_cmd(cmd_and_args, as_shell=False, catch=3): Exits if rc non-zero """ - ret, out = __exec_cmd(cmd_and_args, as_shell, catch) + ret, out = _exec_cmd(cmd_and_args, as_shell, catch) if ret != 0: msger.error("exec_cmd: %s returned '%s' instead of 0" % \
@@ -99,7 +99,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3): args = cmd_and_args.split() msger.debug(args) - ret, out = __exec_cmd(native_cmd_and_args, True, catch) + ret, out = _exec_cmd(native_cmd_and_args, True, catch) if ret == 127: # shell command-not-found prog = args[0]
@@ -139,7 +139,7 @@ def get_bitbake_var(var, image=None): log_level = msger.get_loglevel() msger.set_loglevel('normal') - ret, lines = __exec_cmd(cmd) + ret, lines = _exec_cmd(cmd) msger.set_loglevel(log_level) if ret:
Signature · 1 line
2.1.4
PATCH v2 4/17 wic: add BitbakeVars class
In reply to Ed BartoshMoved code of getting bitbake variables into separate class.
Created singleton object of this class in the module namespace.
Preserved existing API get_bitbake_var.
--- scripts/lib/wic/utils/oe/misc.py | 91 +++++++++++++++++++++++----------------- 1 file changed, 53 insertions(+), 38 deletions(-)
scripts/lib/wic/utils/oe/misc.py +53 -38
@@ -124,48 +124,63 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3): BOOTDD_EXTRA_SPACE = 16384 -_BITBAKE_VARS = defaultdict(dict) +class BitbakeVars(defaultdict): + """ + Container for Bitbake variables. + """ + def __init__(self): + defaultdict.__init__(self, dict) + + def get_var(self, var, image=None): + """ + Get bitbake variable value lazy way, i.e. run + 'bitbake -e' only when variable is requested. + """ + if image not in self: + # Get bitbake -e output + cmd = "bitbake -e" + if image: + cmd += " %s" % image + + log_level = msger.get_loglevel() + msger.set_loglevel('normal') + ret, lines = _exec_cmd(cmd) + msger.set_loglevel(log_level) + + if ret: + print "Couldn't get '%s' output." % cmd + print "Bitbake failed with error:\n%s\n" % lines + return + + # Parse bitbake -e output + for line in lines.split('\n'): + if "=" not in line: + continue + try: + key, val = line.split("=") + except ValueError: + continue + key = key.strip() + val = val.strip() + if key.replace('_', '').isalnum(): + self[image][key] = val.strip('"') + + # Make first image a default set of variables + images = [key for key in self if key] + if len(images) == 1: + self[None] = self[image] + + return self[image].get(var) + +# Create BB_VARS singleton +BB_VARS = BitbakeVars() def get_bitbake_var(var, image=None): """ - Get bitbake variable value lazy way, i.e. run - 'bitbake -e' only when variable is requested. + Provide old get_bitbake_var API by wrapping + get_var method of BB_VARS singleton. """ - if image not in _BITBAKE_VARS: - # Get bitbake -e output - cmd = "bitbake -e" - if image: - cmd += " %s" % image - - log_level = msger.get_loglevel() - msger.set_loglevel('normal') - ret, lines = _exec_cmd(cmd) - msger.set_loglevel(log_level) - - if ret: - print "Couldn't get '%s' output." % cmd - print "Bitbake failed with error:\n%s\n" % lines - return - - # Parse bitbake -e output - for line in lines.split('\n'): - if "=" not in line: - continue - try: - key, val = line.split("=") - except ValueError: - continue - key = key.strip() - val = val.strip() - if key.replace('_', '').isalnum(): - _BITBAKE_VARS[image][key] = val.strip('"') - - # Make first image a default set of variables - images = [key for key in _BITBAKE_VARS if key] - if len(images) == 1: - _BITBAKE_VARS[None] = _BITBAKE_VARS[image] - - return _BITBAKE_VARS[image].get(var) + return BB_VARS.get_var(var, image) def parse_sourceparams(sourceparams): """
Signature · 1 line
2.1.4
PATCH v2 5/17 wic: create new method _parse_line
In reply to Ed BartoshMoved code that parses one line of 'bitbake -e' output to separate method _parse_line.
This method will be also used later to parse lines of .env files.
--- scripts/lib/wic/utils/oe/misc.py | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-)
scripts/lib/wic/utils/oe/misc.py +17 -10
@@ -131,6 +131,22 @@ class BitbakeVars(defaultdict): def __init__(self): defaultdict.__init__(self, dict) + def _parse_line(self, line, image): + """ + Parse one line from bitbake -e output. + Put result key-value pair into the storage. + """ + if "=" not in line: + return + try: + key, val = line.split("=") + except ValueError: + return + key = key.strip() + val = val.strip() + if key.replace('_', '').isalnum(): + self[image][key] = val.strip('"') + def get_var(self, var, image=None): """ Get bitbake variable value lazy way, i.e. run
@@ -154,16 +170,7 @@ class BitbakeVars(defaultdict): # Parse bitbake -e output for line in lines.split('\n'): - if "=" not in line: - continue - try: - key, val = line.split("=") - except ValueError: - continue - key = key.strip() - val = val.strip() - if key.replace('_', '').isalnum(): - self[image][key] = val.strip('"') + self._parse_line(line, image) # Make first image a default set of variables images = [key for key in self if key]
Signature · 1 line
2.1.4
PATCH v2 6/17 wic: add default_image attribute to BitbakeVars
In reply to Ed BartoshNew attribute is used when bitbake variable is requested without specifying image name. The attribute should be set from outside, for example when wic is called with '-e <image>' option.
--- scripts/lib/wic/utils/oe/misc.py | 6 ++++++ 1 file changed, 6 insertions(+)
scripts/lib/wic/utils/oe/misc.py +6 -0
@@ -131,6 +131,9 @@ class BitbakeVars(defaultdict): def __init__(self): defaultdict.__init__(self, dict) + # default_image attribute should be set from outside + self.default_image = None + def _parse_line(self, line, image): """ Parse one line from bitbake -e output.
@@ -152,6 +155,9 @@ class BitbakeVars(defaultdict): Get bitbake variable value lazy way, i.e. run 'bitbake -e' only when variable is requested. """ + if not image: + image = self.default_image + if image not in self: # Get bitbake -e output cmd = "bitbake -e"
Signature · 1 line
2.1.4
PATCH v2 7/17 wic: set default image
In reply to Ed BartoshSet BitbakeVars.default_image when wic is called with -e option. This makes get_bitbake_var API to use provided image as a default source of variables.
--- scripts/wic | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-)
scripts/wic +4 -2
@@ -52,7 +52,7 @@ if bitbake_exe: else: bitbake_main = None -from wic.utils.oe.misc import get_bitbake_var +from wic.utils.oe.misc import get_bitbake_var, BB_VARS from wic.utils.errors import WicError from image import engine from image import help as hlp
@@ -141,7 +141,9 @@ def wic_create_subcommand(args, usage_str): print " " + ", ".join(missed) sys.exit(1) - if not options.image_name: + if options.image_name: + BB_VARS.default_image = options.image_name + else: options.build_check = False if options.build_check:
Signature · 1 line
2.1.4
PATCH v2 8/17 wic: implement getting variables from .env files
In reply to Ed BartoshAdded functionality of getting variables from <image>.env files to BitbakeVars class. env files will be parsed if the directory with env files is known, i.e. when vars_dir attribute is set. Otherwise 'bitbake -e' output will be parsed.
--- scripts/lib/wic/utils/oe/misc.py | 59 +++++++++++++++++++++++++--------------- 1 file changed, 37 insertions(+), 22 deletions(-)
scripts/lib/wic/utils/oe/misc.py +37 -22
@@ -26,6 +26,7 @@ # """Miscellaneous functions.""" +import os from collections import defaultdict from wic import msger
@@ -131,12 +132,13 @@ class BitbakeVars(defaultdict): def __init__(self): defaultdict.__init__(self, dict) - # default_image attribute should be set from outside + # default_image and vars_dir attributes should be set from outside self.default_image = None + self.vars_dir = None def _parse_line(self, line, image): """ - Parse one line from bitbake -e output. + Parse one line from bitbake -e output or from .env file. Put result key-value pair into the storage. """ if "=" not in line:
@@ -152,31 +154,44 @@ class BitbakeVars(defaultdict): def get_var(self, var, image=None): """ - Get bitbake variable value lazy way, i.e. run - 'bitbake -e' only when variable is requested. + Get bitbake variable from 'bitbake -e' output or from .env file. + This is a lazy method, i.e. it runs bitbake or parses file only when + only when variable is requested. It also caches results. """ if not image: image = self.default_image if image not in self: - # Get bitbake -e output - cmd = "bitbake -e" - if image: - cmd += " %s" % image - - log_level = msger.get_loglevel() - msger.set_loglevel('normal') - ret, lines = _exec_cmd(cmd) - msger.set_loglevel(log_level) - - if ret: - print "Couldn't get '%s' output." % cmd - print "Bitbake failed with error:\n%s\n" % lines - return - - # Parse bitbake -e output - for line in lines.split('\n'): - self._parse_line(line, image) + if image and self.vars_dir: + fname = os.path.join(self.vars_dir, image + '.env') + if os.path.isfile(fname): + # parse .env file + with open(fname) as varsfile: + for line in varsfile: + self._parse_line(line, image) + else: + print "Couldn't get bitbake variable from %s." % fname + print "File %s doesn't exist." % fname + return + else: + # Get bitbake -e output + cmd = "bitbake -e" + if image: + cmd += " %s" % image + + log_level = msger.get_loglevel() + msger.set_loglevel('normal') + ret, lines = _exec_cmd(cmd) + msger.set_loglevel(log_level) + + if ret: + print "Couldn't get '%s' output." % cmd + print "Bitbake failed with error:\n%s\n" % lines + return + + # Parse bitbake -e output + for line in lines.split('\n'): + self._parse_line(line, image) # Make first image a default set of variables images = [key for key in self if key]
Signature · 1 line
2.1.4
PATCH v2 9/17 wic: implement --vars option
In reply to Ed BartoshThis option is used to point wic to the directory with .env files containing list of bitbake variables and their values.
If this option is used wic will get bitbake variables from files instead of parsing 'bitbake -e' output.
The main reason for this is to support new mode, when bitbake runs wic to produce wic images. In this case wic can't run bitbake again as it's locked, so it will get variables from .env files.
--- scripts/wic | 6 ++++++ 1 file changed, 6 insertions(+)
scripts/wic +6 -0
@@ -114,6 +114,9 @@ def wic_create_subcommand(args, usage_str): parser.add_option("-c", "--compress-with", choices=("gzip", "bzip2", "xz"), dest='compressor', help="compress image with specified compressor") + parser.add_option("-v", "--vars", dest='vars_dir', + help="directory with <image>.env files that store " + "bitbake variables") parser.add_option("-D", "--debug", dest="debug", action="store_true", default=False, help="output debug information")
@@ -146,6 +149,9 @@ def wic_create_subcommand(args, usage_str): else: options.build_check = False + if options.vars_dir: + BB_VARS.vars_dir = options.vars_dir + if options.build_check: print "Checking basic build environment..." if not engine.verify_build_env():
Signature · 1 line
2.1.4
PATCH v2 10/17 wic: rename variable
In reply to Ed BartoshRenamed variable help -> hlp as 'help' is a name of Python built-in function.
--- scripts/lib/image/help.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/lib/image/help.py +2 -2
@@ -41,9 +41,9 @@ def display_help(subcommand, subcommands): if subcommand not in subcommands: return False - help = subcommands.get(subcommand, subcommand_error)[2] + hlp = subcommands.get(subcommand, subcommand_error)[2] pager = subprocess.Popen('less', stdin=subprocess.PIPE) - pager.communicate(help) + pager.communicate(hlp) return True
Signature · 1 line
2.1.4
PATCH v2 11/17 wic: deferred call of hlp.get_wic_plugins_help()
In reply to Ed BartoshIf get_wic_plugins_help is called from wic main module it calls git_bitbake_var at some point. This fails when wic is called from bitbake as 'bitbake -e' can't be run.
Moved call of this method to help.py in order to call it later, when BitbakeVariables singleton is properly initialized to get variables from .env files.
--- scripts/lib/image/help.py | 2 ++ scripts/wic | 2 +- 2 files changed, 3 insertions(+), 1 deletion(-)
scripts/lib/image/help.py +2 -0
@@ -42,6 +42,8 @@ def display_help(subcommand, subcommands): return False hlp = subcommands.get(subcommand, subcommand_error)[2] + if callable(hlp): + hlp = hlp() pager = subprocess.Popen('less', stdin=subprocess.PIPE) pager.communicate(hlp)
scripts/wic +1 -1
@@ -288,7 +288,7 @@ subcommands = { hlp.wic_list_help], "plugins": [wic_help_topic_subcommand, wic_help_topic_usage, - hlp.get_wic_plugins_help()], + hlp.get_wic_plugins_help], "overview": [wic_help_topic_subcommand, wic_help_topic_usage, hlp.wic_overview_help],
Signature · 1 line
2.1.4
PATCH v2 12/17 image_types.bbclass: add wic image type
In reply to Ed Bartoshwic image type is used to produce partitioned images.
Image configuration should be stored in either <recipe>.<machine>.wks or <recipe>.wks file.
.wks file should be put to the same location as image recipe and have the same name.
[YOCTO #7672]
--- meta/classes/image_types.bbclass | 12 ++++++++++++ 1 file changed, 12 insertions(+)
meta/classes/image_types.bbclass +12 -0
@@ -154,6 +154,16 @@ IMAGE_CMD_ubi = "multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" "${UBI_VOLNAM IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS}" +IMAGE_CMD_wic () { + out=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME} + wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.${MACHINE}.wks + [ -e $wks ] || wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.wks + [ -e $wks ] || bbfatal "Kiskstart file $wks doesn't exist" + BUILDDIR=${TOPDIR} wic create $wks --vars ${STAGING_DIR_TARGET}/imgdata/ -e ${IMAGE_BASENAME} -o $out/ + mv $out/build/${IMAGE_BASENAME}*.direct $out.rootfs.wic + rm -rf $out/ +} + EXTRA_IMAGECMD = "" inherit siteinfo
@@ -182,6 +192,7 @@ IMAGE_DEPENDS_elf = "virtual/kernel mkelfimage-native" IMAGE_DEPENDS_ubi = "mtd-utils-native" IMAGE_DEPENDS_ubifs = "mtd-utils-native" IMAGE_DEPENDS_multiubi = "mtd-utils-native" +IMAGE_DEPENDS_wic = "parted-native" # This variable is available to request which values are suitable for IMAGE_FSTYPES IMAGE_TYPES = " \
@@ -201,6 +212,7 @@ IMAGE_TYPES = " \ vdi \ qcow2 \ elf \ + wic wic.gz wic.bz2 wic.lzma \ " COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum"
Signature · 1 line
2.1.4
Re: [PATCH 12/17] image_types.bbclass: add wic image type
Would it not be better approach if we don't hardcore the name and path and handle it using some variable?
Noor
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Ed Bartosh
Sent: Tuesday, August 25, 2015 2:04 PM
To: [email protected]
Subject: [OE-core] [PATCH 12/17] image_types.bbclass: add wic image type
wic image type is used to produce partitioned images.
Image configuration should be stored in either <recipe>.<machine>.wks or <recipe>.wks file. .wks file should be put to the same location as image recipe and have the same name.
[YOCTO #7672]
--- meta/classes/image_types.bbclass | 12 ++++++++++++ 1 file changed, 12 insertions(+)
meta/classes/image_types.bbclass +11 -0
@@ -154,6 +154,16 @@ IMAGE_CMD_ubi = "multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" "${UBI_VOLNAM IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS}" +IMAGE_CMD_wic () { + out=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME} + wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.${MACHINE}.wks + [ -e $wks ] || wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.wks + [ -e $wks ] || bbfatal "Kiskstart file $wks doesn't exist" + BUILDDIR=${TOPDIR} wic create $wks --vars ${STAGING_DIR_TARGET}/imgdata/ -e ${IMAGE_BASENAME} -o $out/ + mv $out/build/${IMAGE_BASENAME}*.direct $out.rootfs.wic + rm -rf $out/ +} + EXTRA_IMAGECMD = "" inherit siteinfo
@@ -182,6 +192,7 @@ IMAGE_DEPENDS_elf = "virtual/kernel mkelfimage-native" IMAGE_DEPENDS_ubi = "mtd-utils-native" IMAGE_DEPENDS_ubifs = "mtd-utils-native" IMAGE_DEPENDS_multiubi = "mtd-utils-native" +IMAGE_DEPENDS_wic = "parted-native" # This variable is available to request which values are suitable for IMAGE_FSTYPES IMAGE_TYPES = " \ @@ -201,6 +212,7 @@ IMAGE_TYPES = " \ vdi \
qcow2 \
elf \+ wic wic.gz wic.bz2 wic.lzma \
"
COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum"
--
2.1.4
--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Re: [PATCH 12/17] image_types.bbclass: add wic image type
Quoted 2 lines · Noor
Would it not be better approach if we don't hardcore the name and path and handle it using some variable?
Can you elaborate a bit? I don't see any hardcoded names and paths in this patch.
Regards,
Ed
Quoted 63 lines
Noor
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Ed Bartosh
Sent: Tuesday, August 25, 2015 2:04 PM
To: [email protected]
Subject: [OE-core] [PATCH 12/17] image_types.bbclass: add wic image typewic image type is used to produce partitioned images.
Image configuration should be stored in either <recipe>.<machine>.wks or <recipe>.wks file. .wks file should be put to the same location as image recipe and have the same name.
[YOCTO #7672]
Signed-off-by Ed Bartosh <[email protected]>--- meta/classes/image_types.bbclass | 12 ++++++++++++ 1 file changed, 12 insertions(+)meta/classes/image_types.bbclass +11
-0@@ -154,6 +154,16 @@ IMAGE_CMD_ubi = "multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" "${UBI_VOLNAM IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS}" +IMAGE_CMD_wic () { + out=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME} + wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.${MACHINE}.wks + [ -e $wks ] || wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.wks + [ -e $wks ] || bbfatal "Kiskstart file $wks doesn't exist" + BUILDDIR=${TOPDIR} wic create $wks --vars ${STAGING_DIR_TARGET}/imgdata/ -e ${IMAGE_BASENAME} -o $out/ + mv $out/build/${IMAGE_BASENAME}*.direct $out.rootfs.wic + rm -rf $out/ +} + EXTRA_IMAGECMD = "" inherit siteinfo@@ -182,6 +192,7 @@ IMAGE_DEPENDS_elf = "virtual/kernel mkelfimage-native" IMAGE_DEPENDS_ubi = "mtd-utils-native" IMAGE_DEPENDS_ubifs = "mtd-utils-native" IMAGE_DEPENDS_multiubi = "mtd-utils-native" +IMAGE_DEPENDS_wic = "parted-native" # This variable is available to request which values are suitable for IMAGE_FSTYPES IMAGE_TYPES = " \ @@ -201,6 +212,7 @@ IMAGE_TYPES = " \ vdi \qcow2 \ elf \+ wic wic.gz wic.bz2 wic.lzma \
"COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum"
--
2.1.4--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core
Signature · 3 lines
-- Regards, Ed
PATCH v2 13/17 wic-image-minimal: add wic image recipe and .wks
In reply to Ed BartoshAdded example of recipe and .wks file to create partitioned image.
This image is using quite complex partitioning scheme.
It uses its own rootfs to populate two partitions in two different ways. It also uses core-image-minimal rootfs to populate another partition.
This is how wic reports about artifacts used to create this image:
ROOTFS_DIR: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs ROOTFS_DIR["/core"]: tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs ROOTFS_DIR["/backup"]: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs
BOOTIMG_DIR: tmp/sysroots/qemux86-64/usr/share
KERNEL_DIR: tmp/deploy/images/qemux86-64
NATIVE_SYSROOT: tmp/sysroots/x86_64-linux
--- meta/recipes-extended/images/wic-image-minimal.bb | 14 ++++++++++++++ meta/recipes-extended/images/wic-image-minimal.wks | 10 ++++++++++ 2 files changed, 24 insertions(+) create mode 100644 meta/recipes-extended/images/wic-image-minimal.bb create mode 100644 meta/recipes-extended/images/wic-image-minimal.wks
meta/recipes-extended/images/wic-image-minimal.bb new file +14 -0
@@ -0,0 +1,14 @@ +SUMMARY = "An example of partitioned image." + +IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP}" + +IMAGE_FSTYPES = "wic.bz2" +RM_OLD_IMAGE = "1" + +# core-image-minimal is referenced in .wks, so we need its rootfs +# to be ready before our rootfs +do_rootfs[depends] += "core-image-minimal:do_rootfs" + +IMAGE_ROOTFS_EXTRA_SPACE = "2000" + +inherit image
meta/recipes-extended/images/wic-image-minimal.wks new file +10 -0
@@ -0,0 +1,10 @@ +# short-description: Example of partitioned image with complex layout +# long-description: This image contains boot partition and 3 rootfs partitions +# created from core-image-minimal and wic-image-minimal image recipes. + +part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024 +part / --source rootfs --ondisk sda --fstype=ext2 --label platform --align 1024 +part /core --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext2 --label core --align 1024 +part /backup --source rootfs --rootfs-dir=wic-image-minimal --ondisk sda --fstype=ext2 --label backup --align 1024 + +bootloader --timeout=0 --append="rootwait console=tty0"
Signature · 1 line
2.1.4
PATCH v2 14/17 oe-selftest: test building wic image by bitbake
In reply to Ed BartoshAdded test case to verify building of wic-image-minimal recipe and produced artifacts: manifest and bzipped partitioned image.
--- meta/lib/oeqa/selftest/wic.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
meta/lib/oeqa/selftest/wic.py +14 -0
@@ -187,3 +187,17 @@ class Wic(oeSelfTest): 'IMAGE_LINK_NAME', 'STAGING_LIBDIR', 'ROOTFS_SIZE'): self.assertTrue(var in content) self.assertTrue(content[var]) + + def test20_wic_image_type(self): + """Test building wic images by bitbake""" + self.assertEqual(0, bitbake('wic-image-minimal').status) + + deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE') + machine = get_bb_var('MACHINE') + prefix = os.path.join(deploy_dir, 'wic-image-minimal-%s.' % machine) + # check if we have result image and manifests symlinks + # pointing to existing files + for suffix in ('wic.bz2', 'manifest'): + path = prefix + suffix + self.assertTrue(os.path.islink(path)) + self.assertTrue(os.path.isfile(os.path.realpath(path)))
Signature · 1 line
2.1.4
PATCH v2 15/17 image.py: set bitbake variable ROOTFS_SIZE
In reply to Ed BartoshThis variable is going to be used by wic to set partition size. Setting it in image.py makes it possible for wic to use it without calculating it again.
--- meta/lib/oe/image.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
meta/lib/oe/image.py +3 -1
@@ -262,14 +262,16 @@ class Image(ImageDepGraph): def _write_script(self, type, cmds): tempdir = self.d.getVar('T', True) script_name = os.path.join(tempdir, "create_image." + type) + rootfs_size = self._get_rootfs_size() self.d.setVar('img_creation_func', '\n'.join(cmds)) self.d.setVarFlag('img_creation_func', 'func', 1) self.d.setVarFlag('img_creation_func', 'fakeroot', 1) + self.d.setVar('ROOTFS_SIZE', str(rootfs_size)) with open(script_name, "w+") as script: script.write("%s" % bb.build.shell_trap_code()) - script.write("export ROOTFS_SIZE=%d\n" % self._get_rootfs_size()) + script.write("export ROOTFS_SIZE=%d\n" % rootfs_size) bb.data.emit_func('img_creation_func', script, self.d) script.write("img_creation_func\n")
Signature · 1 line
2.1.4
PATCH v2 16/17 wic: use bitbake variable ROOTFS_SIZE
In reply to Ed BartoshIf bitbake image is referenced in .ks file and --size is not used there wic uses ROOTFS_SIZE variable to set minimum partition size.
ROOTFS_SIZE is calculated in meta/lib/oe/image.py when rootfs is created. The calculation is done using other image parameters:IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT, IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE.
--- scripts/lib/wic/imager/direct.py | 14 ++++++++++++++ 1 file changed, 14 insertions(+)
scripts/lib/wic/imager/direct.py +14 -0
@@ -29,6 +29,7 @@ import shutil from wic import kickstart, msger from wic.utils import fs_related +from wic.utils.oe.misc import get_bitbake_var from wic.utils.partitionedfs import Image from wic.utils.errors import CreatorError, ImageError from wic.imager.baseimager import BaseImageCreator
@@ -229,6 +230,19 @@ class DirectImageCreator(BaseImageCreator): fstab_path = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR")) for p in parts: + # get rootfs size from bitbake variable if it's not set in .ks file + if not p.size: + # and if rootfs name is specified for the partition + image_name = p.get_rootfs() + if image_name: + # Bitbake variable ROOTFS_SIZE is calculated in + # Image._get_rootfs_size method from meta/lib/oe/image.py + # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT, + # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE + rsize_bb = get_bitbake_var('ROOTFS_SIZE', image_name) + if rsize_bb: + # convert from Kb to Mb + p.size = int(rsize_bb) / 1024 # need to create the filesystems in order to get their # sizes before we can add them and do the layout. # Image.create() actually calls __format_disks() to create
Signature · 1 line
2.1.4
PATCH v2 17/17 image.py: add script output to the rootfs log
In reply to Ed BartoshLet's add output of image creation script to the bitbake log as it can contain useful information.
One good example of such an information is wic report about artifacts and .wks file used for image creation.
--- meta/lib/oe/image.py | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-)
meta/lib/oe/image.py +4 -1
@@ -11,11 +11,14 @@ def generate_image(arg): (type, create_img_cmd)) try: - subprocess.check_output(create_img_cmd, stderr=subprocess.STDOUT) + output = subprocess.check_output(create_img_cmd, + stderr=subprocess.STDOUT) except subprocess.CalledProcessError as e: return("Error: The image creation script '%s' returned %d:\n%s" % (e.cmd, e.returncode, e.output)) + bb.note("Script output:\n%s" % output) + return None
Signature · 1 line
2.1.4