more flexible image conversion

In Ostro OS (https://ostroproject.org) we have a custom image class which creates a whole-disk image
(https://github.com/ostroproject/meta-ostro/blob/master/meta-ostro/classes/image-dsk.bbclass).

Currently that class is more complex than it should be because it re-implements image compression and conversion to VirtualBox, due to limitations in OE-core's image.bbclass and because we wanted to get rid of the redundant and misleading ".rootfs" suffix in image file names.

These two patches make it possible to simplify the custom code to just the actual dsk image and leverage OE-core for the rest.

The removal of the ".rootfs" suffix would also make sense for OE-core. The first commit can be seen as a first step towards that. It does not change the default, so nothing changes yet, but interested distros or users can already get rid of the suffix by overriding the IMAGE_NAME_SUFFIX variable.

https://bugzilla.yoctoproject.org/show_bug.cgi?id=9076 describes the problem and probably can be considered resolved by these changes.

The following changes since commit 331ad5c5f80044a5e82abca0e87a85b162126411:

mtools: Drop GCONV_PATH manipulation (2016-03-07 12:41:09 +0000)

are available in the git repository at:

git://github.com/pohly/openembedded-core image-conversion
https://github.com/pohly/openembedded-core/tree/image-conversion

Patrick Ohly (2):
image creation: allow overriding .rootfs suffix image.bbclass: support chaining compression (aka conversion) commands

 meta/classes/image.bbclass       | 29 +++++++++++++++---
 meta/classes/image_types.bbclass | 63 ++++++++++++++++++++++------------------
 2 files changed, 59 insertions(+), 33 deletions(-)

2.1.4

By default, the image file name contains ".rootfs" to distinguish the image file from other files created during image building. However, for certain image types (for example, .hddimg) the ".rootfs" suffix is redundant because the type suffix alone already uniquely identifies the main image file (core-image-minimal-intel-corei7-64.hddimg instead of core-image-minimal-intel-corei7-64.rootfs.hddimg).

With this change, distros that prefer the shorter image name can
override the .rootfs suffix like this:
IMAGE_NAME_ROOTFS = "${@ '' if <using special image type> else '${IMAGE_NAME_ROOTFS_DEFAULT}'}"

The exact logic when to remove the extra suffix depends on the distro and how it enables its own image type.

---
 meta/classes/image.bbclass       |  4 +--
 meta/classes/image_types.bbclass | 63 ++++++++++++++++++++++------------------
 2 files changed, 36 insertions(+), 31 deletions(-)

index 02612ad..081a0b3 100644
@@ -387,7 +387,7 @@ python () {
                     subimages.append(realt + "." + ctype)
 
         if realt not in alltypes:
-            cmds.append(localdata.expand("\trm ${IMAGE_NAME}.rootfs.${type}"))
+            cmds.append(localdata.expand("\trm ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"))
         else:
             subimages.append(realt)
 

@@ -474,7 +474,7 @@ python create_symlinks() {
     manifest_name = d.getVar('IMAGE_MANIFEST', True)
     taskname = d.getVar("BB_CURRENTTASK", True)
     subimages = (d.getVarFlag("do_" + taskname, 'subimages', False) or "").split()
-    imgsuffix = d.getVarFlag("do_" + taskname, 'imgsuffix', True) or ".rootfs."
+    imgsuffix = d.getVarFlag("do_" + taskname, 'imgsuffix', True) or "${IMAGE_NAME_SUFFIX}."
     os.chdir(deploy_dir)
 
     if not link_name:

2.1.4

It makes sense to use the compression mechanism also for conversion, for example of a whole-disk image into .vdi (VirtualBox). That part already works, like this:

   COMPRESSIONTYPES_append = " vdi"
   COMPRESS_CMD_vdi = "qemu-img convert -O vdi ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vdi"
   IMAGE_DEPENDS_vdi = "qemu-native"

But then it also makes sense to allow compressing the resulting image, which only works after enhancing the image.bbclass.

For example, suppose a custom image command produces "dsk" images. Then it becomes possible to set
IMAGE_FSTYPES = " dsk.xz dsk.vdi.xz"
and do_image_dsk will automatically produce the intermediate images, convert to dsk.xz resp. dsk.vdi -> dsk.vdi.xz and delete all intermediate images. Symlinks are also set correctly.

---
 meta/classes/image.bbclass | 27 ++++++++++++++++++++++++---
 1 file changed, 24 insertions(+), 3 deletions(-)

index 081a0b3..01b49d4 100644
@@ -303,6 +303,10 @@ python () {
                 basetype = type[:-len("." + ctype)]
                 break
 
+        if basetype != type:
+            # New base type itself might be generated by a conversion command.
+            basetype = _image_base_type(basetype)
+
         return basetype
 
     basetypes = {}

@@ -379,18 +383,35 @@ python () {
             bb.fatal("No IMAGE_CMD defined for IMAGE_FSTYPES entry '%s' - possibly invalid type name or missing support class" % t)
         cmds.append(localdata.expand("\tcd ${DEPLOY_DIR_IMAGE}"))
 
-        for bt in basetypes[t]:
+        rm_tmp_images = set()
+        def gen_conversion_cmds(bt):
             for ctype in ctypes:
                 if bt.endswith("." + ctype):
+                    type = bt[0:-len(ctype) - 1]
+                    # Create input image first.
+                    gen_conversion_cmds(type)
+                    localdata.setVar('type', type)
                     cmds.append("\t" + localdata.getVar("COMPRESS_CMD_" + ctype, True))
                     vardeps.add('COMPRESS_CMD_' + ctype)
-                    subimages.append(realt + "." + ctype)
+                    subimages.append(type + "." + ctype)
+                    if type not in alltypes:
+                        rm_tmp_images.add(localdata.expand("${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"))
+
+        for bt in basetypes[t]:
+            gen_conversion_cmds(bt)
 
+        localdata.setVar('type', realt)
         if realt not in alltypes:
-            cmds.append(localdata.expand("\trm ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"))
+            rm_tmp_images.add(localdata.expand("${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"))
         else:
             subimages.append(realt)
 
+        # Clean up after applying all conversion commands. Some of them might
+        # use the same input, therefore we cannot delete sooner without applying
+        # some complex dependency analysis.
+        for image in rm_tmp_images:
+            cmds.append("\trm " + image)
+
         after = 'do_image'
         for dep in typedeps[t]:
             after += ' do_image_%s' % dep.replace("-", "_").replace(".", "_")

2.1.4

The patch for making the .rootfs configurable was incomplete: in the python create_symlinks() method the new variable must be expanded explicitly.

Not doing so broke the symlink creation and that led to hard build failures in image types depending on the boot-directdisk.bbclass (like qcow2) because the build_boot_dd() method relied on the symlink.

---
 meta/classes/image.bbclass | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 081a0b3..8b6c30b 100644
@@ -474,7 +474,7 @@ python create_symlinks() {
     manifest_name = d.getVar('IMAGE_MANIFEST', True)
     taskname = d.getVar("BB_CURRENTTASK", True)
     subimages = (d.getVarFlag("do_" + taskname, 'subimages', False) or "").split()
-    imgsuffix = d.getVarFlag("do_" + taskname, 'imgsuffix', True) or "${IMAGE_NAME_SUFFIX}."
+    imgsuffix = d.getVarFlag("do_" + taskname, 'imgsuffix', True) or d.expand("${IMAGE_NAME_SUFFIX}.")
     os.chdir(deploy_dir)
 
     if not link_name:

2.1.4