Skip to contentSkip to conversation

V4: Build wic images with bitbake

PATCH In review Ed Bartosh 2 participants 8 messages Aug 30, 2015 – Aug 31, 2015

lore
Share & tools
lore

Ed Bartosh

author
#0 lore raw

cover V4: Build wic images with bitbake

Hi,

This is hopefully last version of the patchset.
I've changed location of .wks and .bb files as Richard suggested.

The following changes since commit 0f84702f0811cba4b6e6e890d478f6e98937b831:

image_types.bbclass: add wic image type (2015-08-30 21:38:23 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/wic/wic-image-type
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/wic/wic-image-type

Ed Bartosh (5):
image.py: write bitbake variables to .env file
oe-selftest: test generation of <image>.env
wic-image-minimal: add wic image recipe and .wks
oe-selftest: test building wic image by bitbake
wic: use bitbake variable ROOTFS_SIZE

 .../recipes-test/images/wic-image-minimal.bb       | 14 +++++++++
 .../recipes-test/images/wic-image-minimal.wks      | 10 +++++++
 meta/classes/image_types.bbclass                   |  4 +++
 meta/lib/oe/image.py                               | 18 ++++++++++++
 meta/lib/oeqa/selftest/wic.py                      | 34 ++++++++++++++++++++++
 scripts/lib/wic/imager/direct.py                   | 14 +++++++++
 6 files changed, 94 insertions(+)
 create mode 100644 meta-selftest/recipes-test/images/wic-image-minimal.bb
 create mode 100644 meta-selftest/recipes-test/images/wic-image-minimal.wks

--
Ed

author
#1 lore raw

PATCH 1/5 image.py: write bitbake variables to .env file

Write set of bitbake variables used by wic into build/tmp/sysroots/<machine>/imagedata/<image>.env

List of variables is defined in WICVARS variable in meta/classes/image_types.bbclass.

This is needed for wic to be able to get bitbake variables without running 'bitbake -e'.

---
 meta/classes/image_types.bbclass |  4 ++++
 meta/lib/oe/image.py             | 18 ++++++++++++++++++
 2 files changed, 22 insertions(+)
meta/classes/image_types.bbclass +4 -0
index 05c45f8..ecb066b 100644
@@ -248,3 +248,7 @@ IMAGE_EXTENSION_live = "hddimg iso"
 # The IMAGE_TYPES_MASKED variable will be used to mask out from the IMAGE_FSTYPES,
 # images that will not be built at do_rootfs time: vmdk, vdi, qcow2, hddimg, iso, etc.
 IMAGE_TYPES_MASKED ?= ""
+
+# The WICVARS variable is used to define list of bitbake variables used in wic code
+# variables from this list is written to <image>.env file
+WICVARS ?= "BBLAYERS DEPLOY_DIR_IMAGE HDDDIR IMAGE_BASENAME IMAGE_BOOT_FILES IMAGE_LINK_NAME IMAGE_ROOTFS INITRAMFS_FSTYPES INITRD ISODIR MACHINE_ARCH ROOTFS_SIZE STAGING_DATADIR STAGING_DIR_NATIVE STAGING_LIBDIR TARGET_SYS"
meta/lib/oe/image.py +18 -0
index a2f94a1..95c62dc 100644
@@ -326,6 +326,22 @@ class Image(ImageDepGraph):
 
         return image_cmd_groups
 
+    def _write_env(self):
+        """
+        Write environment variables used by wic
+        to tmp/sysroots/<machine>/imgdata/<image>.env
+        """
+        stdir = self.d.getVar('STAGING_DIR_TARGET', True)
+        outdir = os.path.join(stdir, 'imgdata')
+        if not os.path.exists(outdir):
+            os.makedirs(outdir)
+        basename = self.d.getVar('IMAGE_BASENAME', True)
+        with open(os.path.join(outdir, basename) + '.env', 'w') as envf:
+            for var in self.d.getVar('WICVARS', True).split():
+                value = self.d.getVar(var, True)
+                if value:
+                    envf.write('%s="%s"\n' % (var, value.strip()))
+
     def create(self):
         bb.note("###### Generate images #######")
         pre_process_cmds = self.d.getVar("IMAGE_PREPROCESS_COMMAND", True)
@@ -337,6 +353,8 @@ class Image(ImageDepGraph):
 
         image_cmd_groups = self._get_imagecmds()
 
+        self._write_env()
+
         for image_cmds in image_cmd_groups:
             # create the images in parallel
             nproc = multiprocessing.cpu_count()
Signature · 1 line
2.1.4
#2 lore raw

Re: [PATCH 1/5] image.py: write bitbake variables to .env file

Quoted 11 lines · Ed Bartosh

Write set of bitbake variables used by wic into build/tmp/sysroots/<machine>/imagedata/<image>.env

List of variables is defined in WICVARS variable in meta/classes/image_types.bbclass.

This is needed for wic to be able to get bitbake variables without running 'bitbake -e'.

The function name (_write_env) sounds generic, as does the commit message, and the filename, yet it relies on a variable which is WIC-specific. This seems rather inconsistent to me, either this thing is for wic alone, or it isn't, and the function name, file name, and variable name should reflect that.

Signature · 5 lines
Christopher Larson
clarson at kergoth dot com
Founder - BitBake, OpenEmbedded, OpenZaurus
Maintainer - Tslib
Senior Software Engineer, Mentor Graphics
author
#3 lore raw

Re: [PATCH 1/5] image.py: write bitbake variables to .env file

Quoted 12 lines · Christopher Larson
Quoted 9 lines · Ed Bartosh

Write set of bitbake variables used by wic into build/tmp/sysroots/<machine>/imagedata/<image>.env

List of variables is defined in WICVARS variable in meta/classes/image_types.bbclass.

This is needed for wic to be able to get bitbake variables without running 'bitbake -e'.

Quoted 8 lines
Quoted 2 lines

The function name (_write_env) sounds generic, as does the commit message, and the filename, yet it relies on a variable which is WIC-specific. This seems rather inconsistent to me, either this thing is for wic alone, or it isn't, and the function name, file name, and variable name should reflect that.

Thanks for pointing this out. I'll rename _write_env to _write_wic_env it in the next patchset.

--
Regards,
Ed

author
#4 lore raw

PATCH 2/5 oe-selftest: test generation of <image>.env

In reply to Ed Bartosh

Added test case to check if <image>.env file is generated and contains bitbake variables used in wic code.

---
 meta/lib/oeqa/selftest/wic.py | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)
meta/lib/oeqa/selftest/wic.py +20 -0
index 117bd9d..fe8a2d0 100644
@@ -23,6 +23,7 @@
 
 """Test cases for wic."""
 
+import os
 import sys
 
 from glob import glob
@@ -168,3 +169,22 @@ class Wic(oeSelfTest):
         self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-*.direct")))
         self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-*.iso")))
 
+    def test19_image_env(self):
+        """Test generation of <image>.env files."""
+        image = 'core-image-minimal'
+        stdir = get_bb_var('STAGING_DIR_TARGET', image)
+        imgdatadir = os.path.join(stdir, 'imgdata')
+
+        basename = get_bb_var('IMAGE_BASENAME', image)
+        self.assertEqual(basename, image)
+        path = os.path.join(imgdatadir, basename) + '.env'
+        self.assertTrue(os.path.isfile(path))
+
+        wicvars = get_bb_var('WICVARS', image).split()
+        wicvars.remove('IMAGE_BOOT_FILES') # this variable is optional
+        with open(path) as envfile:
+            content = dict(line.split("=", 1) for line in envfile)
+            # test if variables used by wic present in the .env file
+            for var in wicvars:
+                self.assertTrue(var in content, "%s is not in .env file" % var)
+                self.assertTrue(content[var])
Signature · 1 line
2.1.4
author
#5 lore raw

PATCH 3/5 wic-image-minimal: add wic image recipe and .wks

In reply to Ed Bartosh

Added example of recipe and .wks file to create partitioned image.

This image is using quite complex partitioning scheme.
It uses its own rootfs to populate two partitions in two different ways. It also uses core-image-minimal rootfs to populate another partition.

This is how wic reports about artifacts used to create this image:
ROOTFS_DIR: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs ROOTFS_DIR["/core"]: tmp/work/qemux86_64-poky-linux/core-image-minimal/1.0-r0/rootfs ROOTFS_DIR["/backup"]: tmp/work/qemux86_64-poky-linux/wic-image-minimal/1.0-r0/rootfs
BOOTIMG_DIR: tmp/sysroots/qemux86-64/usr/share
KERNEL_DIR: tmp/deploy/images/qemux86-64
NATIVE_SYSROOT: tmp/sysroots/x86_64-linux

---
 meta-selftest/recipes-test/images/wic-image-minimal.bb  | 14 ++++++++++++++
 meta-selftest/recipes-test/images/wic-image-minimal.wks | 10 ++++++++++
 2 files changed, 24 insertions(+)
 create mode 100644 meta-selftest/recipes-test/images/wic-image-minimal.bb
 create mode 100644 meta-selftest/recipes-test/images/wic-image-minimal.wks
meta-selftest/recipes-test/images/wic-image-minimal.bb new file +14 -0
new file mode 100644
index 0000000..073c569
@@ -0,0 +1,14 @@
+SUMMARY = "An example of partitioned image."
+
+IMAGE_INSTALL = "packagegroup-core-boot ${ROOTFS_PKGMANAGE_BOOTSTRAP}"
+
+IMAGE_FSTYPES = "wic.bz2"
+RM_OLD_IMAGE = "1"
+
+# core-image-minimal is referenced in .wks, so we need its rootfs
+# to be ready before our rootfs
+do_rootfs[depends] += "core-image-minimal:do_rootfs"
+
+IMAGE_ROOTFS_EXTRA_SPACE = "2000"
+
+inherit image
meta-selftest/recipes-test/images/wic-image-minimal.wks new file +10 -0
new file mode 100644
index 0000000..29cd8f2
@@ -0,0 +1,10 @@
+# short-description: Example of partitioned image with complex layout
+# long-description: This image contains boot partition and 3 rootfs partitions
+# created from core-image-minimal and wic-image-minimal image recipes.
+
+part /boot --source bootimg-pcbios --ondisk sda --label boot --active --align 1024
+part / --source rootfs --ondisk sda --fstype=ext2 --label platform --align 1024
+part /core --source rootfs --rootfs-dir=core-image-minimal --ondisk sda --fstype=ext2 --label core --align 1024
+part /backup --source rootfs --rootfs-dir=wic-image-minimal --ondisk sda --fstype=ext2 --label backup --align 1024
+
+bootloader  --timeout=0  --append="rootwait console=tty0"
Signature · 1 line
2.1.4
author
#6 lore raw

PATCH 4/5 oe-selftest: test building wic image by bitbake

In reply to Ed Bartosh

Added test case to verify building of wic-image-minimal recipe and produced artifacts: manifest and bzipped partitioned image.

---
 meta/lib/oeqa/selftest/wic.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
meta/lib/oeqa/selftest/wic.py +14 -0
index fe8a2d0..deb2333 100644
@@ -188,3 +188,17 @@ class Wic(oeSelfTest):
             for var in wicvars:
                 self.assertTrue(var in content, "%s is not in .env file" % var)
                 self.assertTrue(content[var])
+
+    def test20_wic_image_type(self):
+        """Test building wic images by bitbake"""
+        self.assertEqual(0, bitbake('wic-image-minimal').status)
+
+        deploy_dir = get_bb_var('DEPLOY_DIR_IMAGE')
+        machine = get_bb_var('MACHINE')
+        prefix = os.path.join(deploy_dir, 'wic-image-minimal-%s.' % machine)
+        # check if we have result image and manifests symlinks
+        # pointing to existing files
+        for suffix in ('wic.bz2', 'manifest'):
+            path = prefix + suffix
+            self.assertTrue(os.path.islink(path))
+            self.assertTrue(os.path.isfile(os.path.realpath(path)))
Signature · 1 line
2.1.4
author
#7 lore raw

PATCH 5/5 wic: use bitbake variable ROOTFS_SIZE

In reply to Ed Bartosh

If bitbake image is referenced in .ks file and --size is not used there wic uses ROOTFS_SIZE variable to set minimum partition size.

ROOTFS_SIZE is calculated in meta/lib/oe/image.py when rootfs is created. The calculation is done using other image parameters:
IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT, IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE.

---
 scripts/lib/wic/imager/direct.py | 14 ++++++++++++++
 1 file changed, 14 insertions(+)
scripts/lib/wic/imager/direct.py +14 -0
index 57b1335..d68fd2a 100644
@@ -29,6 +29,7 @@ import shutil
 
 from wic import kickstart, msger
 from wic.utils import fs_related
+from wic.utils.oe.misc import get_bitbake_var
 from wic.utils.partitionedfs import Image
 from wic.utils.errors import CreatorError, ImageError
 from wic.imager.baseimager import BaseImageCreator
@@ -229,6 +230,19 @@ class DirectImageCreator(BaseImageCreator):
         fstab_path = self._write_fstab(self.rootfs_dir.get("ROOTFS_DIR"))
 
         for p in parts:
+            # get rootfs size from bitbake variable if it's not set in .ks file
+            if not p.size:
+                # and if rootfs name is specified for the partition
+                image_name = p.get_rootfs()
+                if image_name:
+                    # Bitbake variable ROOTFS_SIZE is calculated in
+                    # Image._get_rootfs_size method from meta/lib/oe/image.py
+                    # using IMAGE_ROOTFS_SIZE, IMAGE_ROOTFS_ALIGNMENT,
+                    # IMAGE_OVERHEAD_FACTOR and IMAGE_ROOTFS_EXTRA_SPACE
+                    rsize_bb = get_bitbake_var('ROOTFS_SIZE', image_name)
+                    if rsize_bb:
+                        # convert from Kb to Mb
+                        p.size = int(rsize_bb) / 1024
             # need to create the filesystems in order to get their
             # sizes before we can add them and do the layout.
             # Image.create() actually calls __format_disks() to create
Signature · 1 line
2.1.4