Skip to contentSkip to conversation

Build wic images with bitbake

PATCH In review Ed Bartosh 2 participants 20 messages Aug 25, 2015 – Aug 26, 2015

lore
Share & tools
lore

Ed Bartosh

author
#0 lore raw

v2 cover Build wic images with bitbake

Hi Reviewers,

This patchset adds new image type 'wic' to the list of image types supported by bitbake. This should allow to build partitioned image the same way as any other images.

New image type can be used in image recipes almost the same way as other image types. There is just one difference - <image>[.<machine>].wks file should be put to the same location as image recipe.

To enable this functionality I've implemented generating .env files with bitbake variables and reading them by wic. This seems to be the only way to get bitbake variables in wic when it's run by bitbake as running 'bitbake -e' is not possible as bitbake is locked.

Test cases for new functionality and small improvements are also included into this patchset.

Changes in V2:
- Fixed typo in the comment. Thanks Mario!
- Rebased on fresh master

The following changes since commit bb34181b42b62e0476ba9e12f9125ed9f785cf23:

bitbake: lib/bb/main: avoid importing all server/UI modules on every execution (2015-08-24 23:47:07 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/wic/wic-image-type
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/wic-image-type

Ed Bartosh (17):
image.py: write bitbake variables to .env file
oe-selftest: test generation of <image>.env
wic: remove undescore from function name
wic: add BitbakeVars class
wic: create new method _parse_line
wic: add default_image attribute to BitbakeVars
wic: set default image
wic: implement getting variables from .env files
wic: implement --vars option
wic: rename variable
wic: deferred call of hlp.get_wic_plugins_help() image_types.bbclass: add wic image type
wic-image-minimal: add wic image recipe and .wks
oe-selftest: test building wic image by bitbake image.py: set bitbake variable ROOTFS_SIZE
wic: use bitbake variable ROOTFS_SIZE image.py: add script output to the rootfs log

 meta/classes/image_types.bbclass                   |  12 ++
 meta/lib/oe/image.py                               |  32 +++++-
 meta/lib/oeqa/selftest/wic.py                      |  33 ++++++
 meta/recipes-extended/images/wic-image-minimal.bb  |  14 +++
 meta/recipes-extended/images/wic-image-minimal.wks |  10 ++
 scripts/lib/image/help.py                          |   6 +-
 scripts/lib/wic/imager/direct.py                   |  14 +++
 scripts/lib/wic/utils/oe/misc.py                   | 125 ++++++++++++++-------
 scripts/wic                                        |  14 ++-
 9 files changed, 212 insertions(+), 48 deletions(-)
 create mode 100644 meta/recipes-extended/images/wic-image-minimal.bb
 create mode 100644 meta/recipes-extended/images/wic-image-minimal.wks

--
Regards,
Ed

author
#1 lore raw

PATCH v2 1/17 image.py: write bitbake variables to .env file

Write set of bitbake variables associated with the image into build/tmp/sysroots/<machine>/imagedata/<image>.env

This is needed for wic to be able to get bitbake variables without running 'bitbake -e'.

---
 meta/lib/oe/image.py | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
meta/lib/oe/image.py +23 -0
index 699c30f..a7fdefa 100644
@@ -321,6 +321,27 @@ class Image(ImageDepGraph):
 
         return image_cmd_groups
 
+    def _write_env(self):
+        """
+        Write environment variables
+        to tmp/sysroots/<machine>/imgdata/<image>.env
+        """
+        stdir = self.d.getVar('STAGING_DIR_TARGET', True)
+        outdir = os.path.join(stdir, 'imgdata')
+        if not os.path.exists(outdir):
+            os.makedirs(outdir)
+        basename = self.d.getVar('IMAGE_BASENAME', True)
+        with open(os.path.join(outdir, basename) + '.env', 'w') as envf:
+            for var in sorted(self.d.keys()):
+                # filter out as much as we can to reduce file size
+                if var.startswith('_') or var.startswith('BB_') \
+                   or not var.isupper() or self.d.getVarFlag(var, "func") \
+                   or var in ('BBINCLUDED', 'SRCPV', 'MIRRORS'):
+                    continue
+                value = self.d.getVar(var, True)
+                if value:
+                    envf.write('%s="%s"\n' % (var, value.strip()))
+
     def create(self):
         bb.note("###### Generate images #######")
         pre_process_cmds = self.d.getVar("IMAGE_PREPROCESS_COMMAND", True)
@@ -332,6 +353,8 @@ class Image(ImageDepGraph):
 
         image_cmd_groups = self._get_imagecmds()
 
+        self._write_env()
+
         for image_cmds in image_cmd_groups:
             # create the images in parallel
             nproc = multiprocessing.cpu_count()
Signature · 1 line
2.1.4
author
#2 lore raw

PATCH v2 2/17 oe-selftest: test generation of <image>.env

Added test case to check if <image>.env file is generated and contains bitbake variables used in wic code.

---
 meta/lib/oeqa/selftest/wic.py | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)
meta/lib/oeqa/selftest/wic.py +19 -0
index 117bd9d..639234b 100644
@@ -23,6 +23,7 @@
 
 """Test cases for wic."""
 
+import os
 import sys
 
 from glob import glob
@@ -168,3 +169,21 @@ class Wic(oeSelfTest):
         self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-*.direct")))
         self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-*.iso")))
 
+    def test19_image_env(self):
+        """Test generation of <image>.env files."""
+        image = 'core-image-minimal'
+        stdir = get_bb_var('STAGING_DIR_TARGET', image)
+        imgdatadir = os.path.join(stdir, 'imgdata')
+        basename = get_bb_var('IMAGE_BASENAME', image)
+        self.assertEqual(basename, image)
+        path = os.path.join(imgdatadir, basename) + '.env'
+        self.assertTrue(os.path.isfile(path))
+        with open(path) as envfile:
+            content = dict(line.split("=", 1) for line in envfile)
+            # test if variables used by wic present in the .env file
+            for var in ('IMAGE_ROOTFS', 'DEPLOY_DIR_IMAGE', 'BBLAYERS',
+                        'STAGING_DIR_NATIVE', 'STAGING_DATADIR', 'TARGET_SYS',
+                        'IMAGE_BASENAME', 'INITRAMFS_FSTYPES', 'MACHINE_ARCH',
+                        'IMAGE_LINK_NAME', 'STAGING_LIBDIR', 'ROOTFS_SIZE'):
+                self.assertTrue(var in content)
+                self.assertTrue(content[var])
Signature · 1 line
2.1.4
author
#3 lore raw

PATCH v2 3/17 wic: remove undescore from function name

In reply to Ed Bartosh

Renamed __exec_cmd -> _exec_cmd as double underscores cause strange behaviour when function is called in class method. Python complains that __exec_cmd method(!!!) of the same class doesn't exist.

---
 scripts/lib/wic/utils/oe/misc.py | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)
scripts/lib/wic/utils/oe/misc.py +6 -6
index d399f2a..9c8f52d 100644
@@ -46,13 +46,13 @@ NATIVE_RECIPES = {"mcopy": "mtools",
                   "syslinux": "syslinux"
                  }
 
-def __exec_cmd(cmd_and_args, as_shell=False, catch=3):
+def _exec_cmd(cmd_and_args, as_shell=False, catch=3):
     """
     Execute command, catching stderr, stdout
 
     Need to execute as_shell if the command uses wildcards
     """
-    msger.debug("__exec_cmd: %s" % cmd_and_args)
+    msger.debug("_exec_cmd: %s" % cmd_and_args)
     args = cmd_and_args.split()
     msger.debug(args)
 
@@ -61,7 +61,7 @@ def __exec_cmd(cmd_and_args, as_shell=False, catch=3):
     else:
         ret, out = runner.runtool(args, catch)
     out = out.strip()
-    msger.debug("__exec_cmd: output for %s (rc = %d): %s" % \
+    msger.debug("_exec_cmd: output for %s (rc = %d): %s" % \
                 (cmd_and_args, ret, out))
 
     return (ret, out)
@@ -73,7 +73,7 @@ def exec_cmd(cmd_and_args, as_shell=False, catch=3):
 
     Exits if rc non-zero
     """
-    ret, out = __exec_cmd(cmd_and_args, as_shell, catch)
+    ret, out = _exec_cmd(cmd_and_args, as_shell, catch)
 
     if ret != 0:
         msger.error("exec_cmd: %s returned '%s' instead of 0" % \
@@ -99,7 +99,7 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3):
     args = cmd_and_args.split()
     msger.debug(args)
 
-    ret, out = __exec_cmd(native_cmd_and_args, True, catch)
+    ret, out = _exec_cmd(native_cmd_and_args, True, catch)
 
     if ret == 127: # shell command-not-found
         prog = args[0]
@@ -139,7 +139,7 @@ def get_bitbake_var(var, image=None):
 
         log_level = msger.get_loglevel()
         msger.set_loglevel('normal')
-        ret, lines = __exec_cmd(cmd)
+        ret, lines = _exec_cmd(cmd)
         msger.set_loglevel(log_level)
 
         if ret:
Signature · 1 line
2.1.4
author
#4 lore raw

PATCH v2 4/17 wic: add BitbakeVars class

In reply to Ed Bartosh

Moved code of getting bitbake variables into separate class.

Created singleton object of this class in the module namespace.

Preserved existing API get_bitbake_var.

---
 scripts/lib/wic/utils/oe/misc.py | 91 +++++++++++++++++++++++-----------------
 1 file changed, 53 insertions(+), 38 deletions(-)
scripts/lib/wic/utils/oe/misc.py +53 -38
index 9c8f52d..3537a2e 100644
@@ -124,48 +124,63 @@ def exec_native_cmd(cmd_and_args, native_sysroot, catch=3):
 
 BOOTDD_EXTRA_SPACE = 16384
 
-_BITBAKE_VARS = defaultdict(dict)
+class BitbakeVars(defaultdict):
+    """
+    Container for Bitbake variables.
+    """
+    def __init__(self):
+        defaultdict.__init__(self, dict)
+
+    def get_var(self, var, image=None):
+        """
+        Get bitbake variable value lazy way, i.e. run
+        'bitbake -e' only when variable is requested.
+        """
+        if image not in self:
+            # Get bitbake -e output
+            cmd = "bitbake -e"
+            if image:
+                cmd += " %s" % image
+
+            log_level = msger.get_loglevel()
+            msger.set_loglevel('normal')
+            ret, lines = _exec_cmd(cmd)
+            msger.set_loglevel(log_level)
+
+            if ret:
+                print "Couldn't get '%s' output." % cmd
+                print "Bitbake failed with error:\n%s\n" % lines
+                return
+
+            # Parse bitbake -e output
+            for line in lines.split('\n'):
+                if "=" not in line:
+                    continue
+                try:
+                    key, val = line.split("=")
+                except ValueError:
+                    continue
+                key = key.strip()
+                val = val.strip()
+                if key.replace('_', '').isalnum():
+                    self[image][key] = val.strip('"')
+
+            # Make first image a default set of variables
+            images = [key for key in self if key]
+            if len(images) == 1:
+                self[None] = self[image]
+
+        return self[image].get(var)
+
+# Create BB_VARS singleton
+BB_VARS = BitbakeVars()
 
 def get_bitbake_var(var, image=None):
     """
-    Get bitbake variable value lazy way, i.e. run
-    'bitbake -e' only when variable is requested.
+    Provide old get_bitbake_var API by wrapping
+    get_var method of BB_VARS singleton.
     """
-    if image not in _BITBAKE_VARS:
-        # Get bitbake -e output
-        cmd = "bitbake -e"
-        if image:
-            cmd += " %s" % image
-
-        log_level = msger.get_loglevel()
-        msger.set_loglevel('normal')
-        ret, lines = _exec_cmd(cmd)
-        msger.set_loglevel(log_level)
-
-        if ret:
-            print "Couldn't get '%s' output." % cmd
-            print "Bitbake failed with error:\n%s\n" % lines
-            return
-
-        # Parse bitbake -e output
-        for line in lines.split('\n'):
-            if "=" not in line:
-                continue
-            try:
-                key, val = line.split("=")
-            except ValueError:
-                continue
-            key = key.strip()
-            val = val.strip()
-            if key.replace('_', '').isalnum():
-                _BITBAKE_VARS[image][key] = val.strip('"')
-
-        # Make first image a default set of variables
-        images = [key for key in _BITBAKE_VARS if key]
-        if len(images) == 1:
-            _BITBAKE_VARS[None] = _BITBAKE_VARS[image]
-
-    return _BITBAKE_VARS[image].get(var)
+    return BB_VARS.get_var(var, image)
 
 def parse_sourceparams(sourceparams):
     """
Signature · 1 line
2.1.4
author
#5 lore raw

PATCH v2 5/17 wic: create new method _parse_line

In reply to Ed Bartosh

Moved code that parses one line of 'bitbake -e' output to separate method _parse_line.

This method will be also used later to parse lines of .env files.

---
 scripts/lib/wic/utils/oe/misc.py | 27 +++++++++++++++++----------
 1 file changed, 17 insertions(+), 10 deletions(-)
scripts/lib/wic/utils/oe/misc.py +17 -10
index 3537a2e..41e435f 100644
@@ -131,6 +131,22 @@ class BitbakeVars(defaultdict):
     def __init__(self):
         defaultdict.__init__(self, dict)
 
+    def _parse_line(self, line, image):
+        """
+        Parse one line from bitbake -e output.
+        Put result key-value pair into the storage.
+        """
+        if "=" not in line:
+            return
+        try:
+            key, val = line.split("=")
+        except ValueError:
+            return
+        key = key.strip()
+        val = val.strip()
+        if key.replace('_', '').isalnum():
+            self[image][key] = val.strip('"')
+
     def get_var(self, var, image=None):
         """
         Get bitbake variable value lazy way, i.e. run
@@ -154,16 +170,7 @@ class BitbakeVars(defaultdict):
 
             # Parse bitbake -e output
             for line in lines.split('\n'):
-                if "=" not in line:
-                    continue
-                try:
-                    key, val = line.split("=")
-                except ValueError:
-                    continue
-                key = key.strip()
-                val = val.strip()
-                if key.replace('_', '').isalnum():
-                    self[image][key] = val.strip('"')
+                self._parse_line(line, image)
 
             # Make first image a default set of variables
             images = [key for key in self if key]
Signature · 1 line
2.1.4
author
#6 lore raw

PATCH v2 6/17 wic: add default_image attribute to BitbakeVars

In reply to Ed Bartosh

New attribute is used when bitbake variable is requested without specifying image name. The attribute should be set from outside, for example when wic is called with '-e <image>' option.

---
 scripts/lib/wic/utils/oe/misc.py | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/lib/wic/utils/oe/misc.py +6 -0
index 41e435f..040176d 100644
@@ -131,6 +131,9 @@ class BitbakeVars(defaultdict):
     def __init__(self):
         defaultdict.__init__(self, dict)
 
+        # default_image attribute should be set from outside
+        self.default_image = None
+
     def _parse_line(self, line, image):
         """
         Parse one line from bitbake -e output.
@@ -152,6 +155,9 @@ class BitbakeVars(defaultdict):
         Get bitbake variable value lazy way, i.e. run
         'bitbake -e' only when variable is requested.
         """
+        if not image:
+            image = self.default_image
+
         if image not in self:
             # Get bitbake -e output
             cmd = "bitbake -e"
Signature · 1 line
2.1.4
author
#7 lore raw

PATCH v2 7/17 wic: set default image

In reply to Ed Bartosh

Set BitbakeVars.default_image when wic is called with -e option. This makes get_bitbake_var API to use provided image as a default source of variables.

---
 scripts/wic | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)
scripts/wic +4 -2
index c1d3003..87ada36 100755
@@ -52,7 +52,7 @@ if bitbake_exe:
 else:
     bitbake_main = None
 
-from wic.utils.oe.misc import get_bitbake_var
+from wic.utils.oe.misc import get_bitbake_var, BB_VARS
 from wic.utils.errors import WicError
 from image import engine
 from image import help as hlp
@@ -141,7 +141,9 @@ def wic_create_subcommand(args, usage_str):
             print "  " + ", ".join(missed)
             sys.exit(1)
 
-    if not options.image_name:
+    if options.image_name:
+        BB_VARS.default_image = options.image_name
+    else:
         options.build_check = False
 
     if options.build_check:
Signature · 1 line
2.1.4
author
#8 lore raw

PATCH v2 8/17 wic: implement getting variables from .env files

In reply to Ed Bartosh

Added functionality of getting variables from <image>.env files to BitbakeVars class. env files will be parsed if the directory with env files is known, i.e. when vars_dir attribute is set. Otherwise 'bitbake -e' output will be parsed.

---
 scripts/lib/wic/utils/oe/misc.py | 59 +++++++++++++++++++++++++---------------
 1 file changed, 37 insertions(+), 22 deletions(-)
scripts/lib/wic/utils/oe/misc.py +37 -22
index 040176d..0a8f3f1 100644
@@ -26,6 +26,7 @@
 #
 """Miscellaneous functions."""
 
+import os
 from collections import defaultdict
 
 from wic import msger
@@ -131,12 +132,13 @@ class BitbakeVars(defaultdict):
     def __init__(self):
         defaultdict.__init__(self, dict)
 
-        # default_image attribute should be set from outside
+        # default_image and vars_dir attributes should be set from outside
         self.default_image = None
+        self.vars_dir = None
 
     def _parse_line(self, line, image):
         """
-        Parse one line from bitbake -e output.
+        Parse one line from bitbake -e output or from .env file.
         Put result key-value pair into the storage.
         """
         if "=" not in line:
@@ -152,31 +154,44 @@ class BitbakeVars(defaultdict):
 
     def get_var(self, var, image=None):
         """
-        Get bitbake variable value lazy way, i.e. run
-        'bitbake -e' only when variable is requested.
+        Get bitbake variable from 'bitbake -e' output or from .env file.
+        This is a lazy method, i.e. it runs bitbake or parses file only when
+        only when variable is requested. It also caches results.
         """
         if not image:
             image = self.default_image
 
         if image not in self:
-            # Get bitbake -e output
-            cmd = "bitbake -e"
-            if image:
-                cmd += " %s" % image
-
-            log_level = msger.get_loglevel()
-            msger.set_loglevel('normal')
-            ret, lines = _exec_cmd(cmd)
-            msger.set_loglevel(log_level)
-
-            if ret:
-                print "Couldn't get '%s' output." % cmd
-                print "Bitbake failed with error:\n%s\n" % lines
-                return
-
-            # Parse bitbake -e output
-            for line in lines.split('\n'):
-                self._parse_line(line, image)
+            if image and self.vars_dir:
+                fname = os.path.join(self.vars_dir, image + '.env')
+                if os.path.isfile(fname):
+                    # parse .env file
+                    with open(fname) as varsfile:
+                        for line in varsfile:
+                            self._parse_line(line, image)
+                else:
+                    print "Couldn't get bitbake variable from %s." % fname
+                    print "File %s doesn't exist." % fname
+                    return
+            else:
+                # Get bitbake -e output
+                cmd = "bitbake -e"
+                if image:
+                    cmd += " %s" % image
+
+                log_level = msger.get_loglevel()
+                msger.set_loglevel('normal')
+                ret, lines = _exec_cmd(cmd)
+                msger.set_loglevel(log_level)
+
+                if ret:
+                    print "Couldn't get '%s' output." % cmd
+                    print "Bitbake failed with error:\n%s\n" % lines
+                    return
+
+                # Parse bitbake -e output
+                for line in lines.split('\n'):
+                    self._parse_line(line, image)
 
             # Make first image a default set of variables
             images = [key for key in self if key]
Signature · 1 line
2.1.4
author
#9 lore raw

PATCH v2 9/17 wic: implement --vars option

In reply to Ed Bartosh

This option is used to point wic to the directory with .env files containing list of bitbake variables and their values.

If this option is used wic will get bitbake variables from files instead of parsing 'bitbake -e' output.

The main reason for this is to support new mode, when bitbake runs wic to produce wic images. In this case wic can't run bitbake again as it's locked, so it will get variables from .env files.

---
 scripts/wic | 6 ++++++
 1 file changed, 6 insertions(+)
scripts/wic +6 -0
index 87ada36..815b84e 100755
@@ -114,6 +114,9 @@ def wic_create_subcommand(args, usage_str):
     parser.add_option("-c", "--compress-with", choices=("gzip", "bzip2", "xz"),
                       dest='compressor',
                       help="compress image with specified compressor")
+    parser.add_option("-v", "--vars", dest='vars_dir',
+                      help="directory with <image>.env files that store "
+                           "bitbake variables")
     parser.add_option("-D", "--debug", dest="debug", action="store_true",
                       default=False, help="output debug information")
 
@@ -146,6 +149,9 @@ def wic_create_subcommand(args, usage_str):
     else:
         options.build_check = False
 
+    if options.vars_dir:
+        BB_VARS.vars_dir = options.vars_dir
+
     if options.build_check:
         print "Checking basic build environment..."
         if not engine.verify_build_env():
Signature · 1 line
2.1.4
author
#10 lore raw

PATCH v2 10/17 wic: rename variable

In reply to Ed Bartosh

Renamed variable help -> hlp as 'help' is a name of Python built-in function.

---
 scripts/lib/image/help.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/lib/image/help.py +2 -2
index 5fa5836..dc6ff36 100644
@@ -41,9 +41,9 @@ def display_help(subcommand, subcommands):
     if subcommand not in subcommands:
         return False
 
-    help = subcommands.get(subcommand, subcommand_error)[2]
+    hlp = subcommands.get(subcommand, subcommand_error)[2]
     pager = subprocess.Popen('less', stdin=subprocess.PIPE)
-    pager.communicate(help)
+    pager.communicate(hlp)
 
     return True
 
Signature · 1 line
2.1.4
author
#11 lore raw

PATCH v2 11/17 wic: deferred call of hlp.get_wic_plugins_help()

In reply to Ed Bartosh

If get_wic_plugins_help is called from wic main module it calls git_bitbake_var at some point. This fails when wic is called from bitbake as 'bitbake -e' can't be run.

Moved call of this method to help.py in order to call it later, when BitbakeVariables singleton is properly initialized to get variables from .env files.

---
 scripts/lib/image/help.py | 2 ++
 scripts/wic               | 2 +-
 2 files changed, 3 insertions(+), 1 deletion(-)
scripts/lib/image/help.py +2 -0
index dc6ff36..717d847 100644
@@ -42,6 +42,8 @@ def display_help(subcommand, subcommands):
         return False
 
     hlp = subcommands.get(subcommand, subcommand_error)[2]
+    if callable(hlp):
+        hlp = hlp()
     pager = subprocess.Popen('less', stdin=subprocess.PIPE)
     pager.communicate(hlp)
 
scripts/wic +1 -1
index 815b84e..25b0d67 100755
@@ -288,7 +288,7 @@ subcommands = {
                   hlp.wic_list_help],
     "plugins":   [wic_help_topic_subcommand,
                   wic_help_topic_usage,
-                  hlp.get_wic_plugins_help()],
+                  hlp.get_wic_plugins_help],
     "overview":  [wic_help_topic_subcommand,
                   wic_help_topic_usage,
                   hlp.wic_overview_help],
Signature · 1 line
2.1.4
author
#12 lore raw

PATCH v2 12/17 image_types.bbclass: add wic image type

In reply to Ed Bartosh

wic image type is used to produce partitioned images.

Image configuration should be stored in either <recipe>.<machine>.wks or <recipe>.wks file.
.wks file should be put to the same location as image recipe and have the same name.

[YOCTO #7672]

---
 meta/classes/image_types.bbclass | 12 ++++++++++++
 1 file changed, 12 insertions(+)
meta/classes/image_types.bbclass +12 -0
index 8547574..cdd66ff 100644
@@ -154,6 +154,16 @@ IMAGE_CMD_ubi = "multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" "${UBI_VOLNAM
 
 IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS}"
 
+IMAGE_CMD_wic () {
+	out=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}
+	wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.${MACHINE}.wks
+	[ -e $wks ] || wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.wks
+	[ -e $wks ] || bbfatal "Kiskstart file $wks doesn't exist"
+	BUILDDIR=${TOPDIR} wic create $wks --vars ${STAGING_DIR_TARGET}/imgdata/ -e ${IMAGE_BASENAME} -o $out/
+	mv $out/build/${IMAGE_BASENAME}*.direct $out.rootfs.wic
+	rm -rf $out/
+}
+
 EXTRA_IMAGECMD = ""
 
 inherit siteinfo
@@ -182,6 +192,7 @@ IMAGE_DEPENDS_elf = "virtual/kernel mkelfimage-native"
 IMAGE_DEPENDS_ubi = "mtd-utils-native"
 IMAGE_DEPENDS_ubifs = "mtd-utils-native"
 IMAGE_DEPENDS_multiubi = "mtd-utils-native"
+IMAGE_DEPENDS_wic = "parted-native"
 
 # This variable is available to request which values are suitable for IMAGE_FSTYPES
 IMAGE_TYPES = " \
@@ -201,6 +212,7 @@ IMAGE_TYPES = " \
     vdi \
     qcow2 \
     elf \
+    wic wic.gz wic.bz2 wic.lzma \
 "
 
 COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum"
Signature · 1 line
2.1.4
#13 lore raw

Re: [PATCH 12/17] image_types.bbclass: add wic image type

Would it not be better approach if we don't hardcore the name and path and handle it using some variable?

Noor

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Ed Bartosh
Sent: Tuesday, August 25, 2015 2:04 PM
To: [email protected]
Subject: [OE-core] [PATCH 12/17] image_types.bbclass: add wic image type

wic image type is used to produce partitioned images.

Image configuration should be stored in either <recipe>.<machine>.wks or <recipe>.wks file. .wks file should be put to the same location as image recipe and have the same name.

[YOCTO #7672]

---
 meta/classes/image_types.bbclass | 12 ++++++++++++
 1 file changed, 12 insertions(+)
meta/classes/image_types.bbclass +11 -0
index 8547574..cdd66ff 100644
@@ -154,6 +154,16 @@ IMAGE_CMD_ubi = "multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" "${UBI_VOLNAM
 
 IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS}"
 
+IMAGE_CMD_wic () {
+	out=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}
+	wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.${MACHINE}.wks
+	[ -e $wks ] || wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.wks
+	[ -e $wks ] || bbfatal "Kiskstart file $wks doesn't exist"
+	BUILDDIR=${TOPDIR} wic create $wks --vars ${STAGING_DIR_TARGET}/imgdata/ -e ${IMAGE_BASENAME} -o $out/
+	mv $out/build/${IMAGE_BASENAME}*.direct $out.rootfs.wic
+	rm -rf $out/
+}
+
 EXTRA_IMAGECMD = ""
 
 inherit siteinfo
@@ -182,6 +192,7 @@ IMAGE_DEPENDS_elf = "virtual/kernel mkelfimage-native"
 IMAGE_DEPENDS_ubi = "mtd-utils-native"
 IMAGE_DEPENDS_ubifs = "mtd-utils-native"
 IMAGE_DEPENDS_multiubi = "mtd-utils-native"
+IMAGE_DEPENDS_wic = "parted-native"
 
 # This variable is available to request which values are suitable for IMAGE_FSTYPES  IMAGE_TYPES = " \ @@ -201,6 +212,7 @@ IMAGE_TYPES = " \
     vdi \
     qcow2 \
     elf \

+ wic wic.gz wic.bz2 wic.lzma \
"

COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum"
--
2.1.4

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

author
#14 lore raw

Re: [PATCH 12/17] image_types.bbclass: add wic image type

Quoted 2 lines · Noor

Would it not be better approach if we don't hardcore the name and path and handle it using some variable?

Can you elaborate a bit? I don't see any hardcoded names and paths in this patch.

Regards,
Ed

Quoted 63 lines

Noor

-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Ed Bartosh
Sent: Tuesday, August 25, 2015 2:04 PM
To: [email protected]
Subject: [OE-core] [PATCH 12/17] image_types.bbclass: add wic image type

wic image type is used to produce partitioned images.

Image configuration should be stored in either <recipe>.<machine>.wks or <recipe>.wks file. .wks file should be put to the same location as image recipe and have the same name.

[YOCTO #7672]

---
 meta/classes/image_types.bbclass | 12 ++++++++++++
 1 file changed, 12 insertions(+)
meta/classes/image_types.bbclass +11 -0
index 8547574..cdd66ff 100644
@@ -154,6 +154,16 @@ IMAGE_CMD_ubi = "multiubi_mkfs "${MKUBIFS_ARGS}" "${UBINIZE_ARGS}" "${UBI_VOLNAM
 
 IMAGE_CMD_ubifs = "mkfs.ubifs -r ${IMAGE_ROOTFS} -o ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.rootfs.ubifs ${MKUBIFS_ARGS}"
 
+IMAGE_CMD_wic () {
+	out=${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}
+	wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.${MACHINE}.wks
+	[ -e $wks ] || wks=${FILE_DIRNAME}/${IMAGE_BASENAME}.wks
+	[ -e $wks ] || bbfatal "Kiskstart file $wks doesn't exist"
+	BUILDDIR=${TOPDIR} wic create $wks --vars ${STAGING_DIR_TARGET}/imgdata/ -e ${IMAGE_BASENAME} -o $out/
+	mv $out/build/${IMAGE_BASENAME}*.direct $out.rootfs.wic
+	rm -rf $out/
+}
+
 EXTRA_IMAGECMD = ""
 
 inherit siteinfo
@@ -182,6 +192,7 @@ IMAGE_DEPENDS_elf = "virtual/kernel mkelfimage-native"
 IMAGE_DEPENDS_ubi = "mtd-utils-native"
 IMAGE_DEPENDS_ubifs = "mtd-utils-native"
 IMAGE_DEPENDS_multiubi = "mtd-utils-native"
+IMAGE_DEPENDS_wic = "parted-native"
 
 # This variable is available to request which values are suitable for IMAGE_FSTYPES  IMAGE_TYPES = " \ @@ -201,6 +212,7 @@ IMAGE_TYPES = " \
     vdi \
     qcow2 \
     elf \

+ wic wic.gz wic.bz2 wic.lzma \
"

COMPRESSIONTYPES = "gz bz2 lzma xz lz4 sum"
--
2.1.4

--
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.openembedded.org/mailman/listinfo/openembedded-core

Signature · 3 lines
--
Regards,
Ed
author
#15 lore raw

PATCH v2 13/17 wic-image-minimal: add wic image recipe and .wks

In reply to Ed Bartosh

Added example of recipe and .wks file to create partitioned image.

This image is using quite complex partitioning scheme.
It uses its own rootfs to populate two partitions in two different ways. It also uses core-image-minimal rootfs to populate another partition.

This is how wic reports about artifacts used to create this image:
ROOTFS_DIR: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs ROOTFS_DIR["/core"]: tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs ROOTFS_DIR["/backup"]: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs
BOOTIMG_DIR: tmp/sysroots/qemux86-64/usr/share
KERNEL_DIR: tmp/deploy/images/qemux86-64
NATIVE_SYSROOT: tmp/sysroots/x86_64-linux

---
 meta/recipes-extended/images/wic-image-minimal.bb  | 14 ++++++++++++++
 meta/recipes-extended/images/wic-image-minimal.wks | 10 ++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 meta/recipes-extended/images/wic-image-minimal.bb
 create mode 100644 meta/recipes-extended/images/wic-image-minimal.wks
meta/recipes-extended/images/wic-image-minimal.bb new file +14 -0
new file mode 100644
index 0000000..073c569
@@ -0,0 +1,14 @@
+SUMMARY = "An example of partitioned image."
+
+IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP}"
+
+IMAGE_FSTYPES = "wic.bz2"
+RM_OLD_IMAGE = "1"
+
+# core-image-minimal is referenced in .wks, so we need its rootfs
+# to be ready before our rootfs
+do_rootfs[depends] += "core-image-minimal:do_rootfs"
+
+IMAGE_ROOTFS_EXTRA_SPACE = "2000"
+
+inherit image
meta/recipes-extended/images/wic-image-minimal.wks new file +10 -0
new file mode 100644
index 0000000..29cd8f2
@@ -0,0 +1,10 @@
+# short-description: Example of partitioned image with complex layout
+# long-description: This image contains boot partition and 3 rootfs partitions
+# created from core-image-minimal and wic-image-minimal image recipes.
+
+part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
+part / --source rootfs --ondisk sda --fstype=ext2 --label platform --align 1024
+part /core --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext2 --label core --align 1024
+part /backup --source rootfs --rootfs-dir=wic-image-minimal --ondisk sda --fstype=ext2 --label backup --align 1024
+
+bootloader  --timeout=0  --append="rootwait console=tty0"
Signature · 1 line
2.1.4
author
#16 lore raw

PATCH v2 14/17 oe-selftest: test building wic image by bitbake

In reply to Ed Bartosh

Added test case to verify building of wic-image-minimal recipe and produced artifacts: manifest and bzipped partitioned image.

---
 meta/lib/oeqa/selftest/wic.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
meta/lib/oeqa/selftest/wic.py +14 -0
index 639234b..b7bbc7b 100644
@@ -187,3 +187,17 @@ class Wic(oeSelfTest):
                         'IMAGE_LINK_NAME', 'STAGING_LIBDIR', 'ROOTFS_SIZE'):
                 self.assertTrue(var in content)
                 self.assertTrue(content[var])
+
+    def test20_wic_image_type(self):
+        """Test building wic images by bitbake"""
+        self.assertEqual(0, bitbake('wic-image-minimal').status)
+
+        deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE')
+        machine = get_bb_var('MACHINE')
+        prefix = os.path.join(deploy_dir, 'wic-image-minimal-%s.' % machine)
+        # check if we have result image and manifests symlinks
+        # pointing to existing files
+        for suffix in ('wic.bz2', 'manifest'):
+            path = prefix + suffix
+            self.assertTrue(os.path.islink(path))
+            self.assertTrue(os.path.isfile(os.path.realpath(path)))
Signature · 1 line
2.1.4
author
#17 lore raw

PATCH v2 15/17 image.py: set bitbake variable ROOTFS_SIZE

In reply to Ed Bartosh

This variable is going to be used by wic to set partition size. Setting it in image.py makes it possible for wic to use it without calculating it again.

---
 meta/lib/oe/image.py | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
meta/lib/oe/image.py +3 -1
index a7fdefa..7f82748 100644
@@ -262,14 +262,16 @@ class Image(ImageDepGraph):
     def _write_script(self, type, cmds):
         tempdir = self.d.getVar('T', True)
         script_name = os.path.join(tempdir, "create_image." + type)
+        rootfs_size = self._get_rootfs_size()
 
         self.d.setVar('img_creation_func', '\n'.join(cmds))
         self.d.setVarFlag('img_creation_func', 'func', 1)
         self.d.setVarFlag('img_creation_func', 'fakeroot', 1)
+        self.d.setVar('ROOTFS_SIZE', str(rootfs_size))
 
         with open(script_name, "w+") as script:
             script.write("%s" % bb.build.shell_trap_code())
-            script.write("export ROOTFS_SIZE=%d\n" % self._get_rootfs_size())
+            script.write("export ROOTFS_SIZE=%d\n" % rootfs_size)
             bb.data.emit_func('img_creation_func', script, self.d)
             script.write("img_creation_func\n")
 
Signature · 1 line
2.1.4
author
#18 lore raw

PATCH v2 16/17 wic: use bitbake variable ROOTFS_SIZE

In reply to Ed Bartosh

If bitbake image is referenced in .ks file and --size is not used there wic uses ROOTFS_SIZE variable to set minimum partition size.

ROOTFS_SIZE is calculated in meta/lib/oe/image.py when rootfs is created. The calculation is done using other image parameters:
IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT, IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE.

---
 scripts/lib/wic/imager/direct.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
scripts/lib/wic/imager/direct.py +14 -0
index 57b1335..d68fd2a 100644
@@ -29,6 +29,7 @@ import shutil
 
 from wic import kickstart, msger
 from wic.utils import fs_related
+from wic.utils.oe.misc import get_bitbake_var
 from wic.utils.partitionedfs import Image
 from wic.utils.errors import CreatorError, ImageError
 from wic.imager.baseimager import BaseImageCreator
@@ -229,6 +230,19 @@ class DirectImageCreator(BaseImageCreator):
         fstab_path = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
 
         for p in parts:
+            # get rootfs size from bitbake variable if it's not set in .ks file
+            if not p.size:
+                # and if rootfs name is specified for the partition
+                image_name = p.get_rootfs()
+                if image_name:
+                    # Bitbake variable ROOTFS_SIZE is calculated in
+                    # Image._get_rootfs_size method from meta/lib/oe/image.py
+                    # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT,
+                    # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
+                    rsize_bb = get_bitbake_var('ROOTFS_SIZE', image_name)
+                    if rsize_bb:
+                        # convert from Kb to Mb
+                        p.size = int(rsize_bb) / 1024
             # need to create the filesystems in order to get their
             # sizes before we can add them and do the layout.
             # Image.create() actually calls __format_disks() to create
Signature · 1 line
2.1.4
author
#19 lore raw

PATCH v2 17/17 image.py: add script output to the rootfs log

In reply to Ed Bartosh

Let's add output of image creation script to the bitbake log as it can contain useful information.

One good example of such an information is wic report about artifacts and .wks file used for image creation.

---
 meta/lib/oe/image.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)
meta/lib/oe/image.py +4 -1
index 7f82748..6f57df2 100644
@@ -11,11 +11,14 @@ def generate_image(arg):
             (type, create_img_cmd))
 
     try:
-        subprocess.check_output(create_img_cmd, stderr=subprocess.STDOUT)
+        output = subprocess.check_output(create_img_cmd,
+                                         stderr=subprocess.STDOUT)
     except subprocess.CalledProcessError as e:
         return("Error: The image creation script '%s' returned %d:\n%s" %
                (e.cmd, e.returncode, e.output))
 
+    bb.note("Script output:\n%s" % output)
+
     return None
 
 
Signature · 1 line
2.1.4