Skip to contentSkip to conversation

image creation improvements

PATCH Superseded Ed Bartosh 1 participant 7 messages May 18, 2016

v2 of 4 Superseded Latest is v4 →

lore
Share & tools
lore

Ed Bartosh

author
#0 lore raw

v2 cover image creation improvements

Hi,

This patchset contains various improvements for the image creation functionality made by Patchick Ohly during his work on bug #9076:
- fixed dependency calculation for conversion chaining
- converted vmdk/vdi/qcow2 image types into conversion commands
- renamed COMPRESSION variables to CONVERSION as the term "compression" is no longer accurate
- prioritized specialized image creation methods over using conversion chaining

The changes look reasonable to me. However, it would be good to hear other people opinions as some changes are quite complex.

Changes in v2: rebased on top of master-next
returned image_getdepends back to image_types.bbclass

The following changes since commit a037ad93640d3d6373fe1db35f64307e9fbed4e9:

bitbake: bb.codeparser: track variable flag references (2016-05-17 23:23:32 +0100)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ed/oe-core/image-type-9076
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=ed/oe-core/image-type-9076

Patrick Ohly (6):
image.bbclass: support duplicate compression types image.bbclass: fix dependency calculation when using conversion
chaining
image.bbclass: additional output in create_symlinks
image creation: support converting masked types image.bbclass: rename COMPRESS(ION) to CONVERSION image.bbclass: prefer specialized image creation methods over chaining

 meta/classes/image-live.bbclass                    |   2 +-
 meta/classes/image-vm.bbclass                      |  33 +----
 meta/classes/image.bbclass                         | 133 ++++++++++++-------
 meta/classes/image_types.bbclass                   | 147 ++++++++++++++-------
 meta/classes/image_types_uboot.bbclass             |  18 +--
 meta/conf/documentation.conf                       |   1 -
 .../images/build-appliance-image_14.0.0.bb         |   6 +-
 7 files changed, 199 insertions(+), 141 deletions(-)

--
Regards,
Ed

author
#1 lore raw

PATCH v2 1/6 image.bbclass: support duplicate compression types

From: Patrick Ohly <[email protected]>

When a derived distro adds a certain type, say zip, to COMPRESSIONTYPES and later OE-core does the same, we end up with the type being listed twice, and that would have undesired effects (commands generated twice).

So to support such loosely coupled extension, we de-duplicated the list of types first.

Alternatively, such a situation could also be treated as error. But that seems unnecessary because typically commands for the same type will also do the same thing.

---
 meta/classes/image.bbclass | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)
meta/classes/image.bbclass +9 -1
index 6b9f979..0e81b4e 100644
@@ -297,7 +297,15 @@ python setup_debugfs () {
 
 python () {
     vardeps = set()
-    ctypes = d.getVar('COMPRESSIONTYPES', True).split()
+    # We allow COMPRESSIONTYPES to have duplicates. That avoids breaking
+    # derived distros when OE-core or some other layer independently adds
+    # the same type. There is still only one command for each type, but
+    # presumably the commands will do the same when the type is the same,
+    # even when added in different places.
+    #
+    # Without de-duplication, gen_conversion_cmds() below
+    # would create the same compression command multiple times.
+    ctypes = set(d.getVar('COMPRESSIONTYPES', True).split())
     old_overrides = d.getVar('OVERRIDES', 0)
 
     def _image_base_type(type):
Signature · 1 line
2.1.4
author
#2 lore raw

PATCH v2 2/6 image.bbclass: fix dependency calculation when using conversion chaining

In reply to Ed Bartosh

From: Patrick Ohly <[email protected]>

When using conversion chaining (for example example: .dsk.vdi.xz), the imagetypes_getdepends() did not properly detect all compression commands (thus skipping the corresponding COMPRESS_DEPENDS) and the base type, leading to missing dependencies of the image's do_rootfs task.

This is not a big problem in practice because in those cases where conversion chaining is useful (as in the example above), the missing dependency is qemu-native, which typically gets built anyway.

imagetypes_getdepends() had hard-coded special treatment for certain image types. This gets replaced with setting the normal IMAGE_DEPENDS variables for these types.

[YOCTO #9076]

---
 meta/classes/image.bbclass       | 25 +++++------------
 meta/classes/image_types.bbclass | 60 ++++++++++++++++++++++++++++------------
 2 files changed, 49 insertions(+), 36 deletions(-)
meta/classes/image.bbclass +7 -18
index 0e81b4e..06f2211 100644
@@ -143,7 +143,7 @@ IMAGE_TYPE_vm = '${@bb.utils.contains_any("IMAGE_FSTYPES", ["vmdk", "vdi", "qcow
 inherit ${IMAGE_TYPE_vm}
 
 python () {
-    deps = " " + imagetypes_getdepends(d)
+    deps = " " + image_getdepends(d)
     d.appendVarFlag('do_rootfs', 'depends', deps)
 
     deps = ""
@@ -308,19 +308,6 @@ python () {
     ctypes = set(d.getVar('COMPRESSIONTYPES', True).split())
     old_overrides = d.getVar('OVERRIDES', 0)
 
-    def _image_base_type(type):
-        basetype = type
-        for ctype in ctypes:
-            if type.endswith("." + ctype):
-                basetype = type[:-len("." + ctype)]
-                break
-
-        if basetype != type:
-            # New base type itself might be generated by a conversion command.
-            basetype = _image_base_type(basetype)
-
-        return basetype
-
     basetypes = {}
     alltypes = d.getVar('IMAGE_FSTYPES', True).split()
     typedeps = {}
@@ -331,7 +318,7 @@ python () {
             alltypes.append("debugfs_" + t)
 
     def _add_type(t):
-        baset = _image_base_type(t)
+        baset = image_split_type(t, ctypes)[0]
         input_t = t
         if baset not in basetypes:
             basetypes[baset]= []
@@ -350,7 +337,7 @@ python () {
             if dep not in alltypes:
                 alltypes.append(dep)
             _add_type(dep)
-            basedep = _image_base_type(dep)
+            basedep = image_split_type(dep, ctypes)[0]
             typedeps[baset].add(basedep)
 
         if baset != input_t:
@@ -420,8 +407,10 @@ python () {
                     if type not in alltypes:
                         rm_tmp_images.add(localdata.expand("${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"))
 
-        for bt in basetypes[t]:
-            gen_conversion_cmds(bt)
+        for type in basetypes[t]:
+            # Probably t == bt, but better check explicitly, perhaps that'll change.
+            bt = image_split_type(type, ctypes)[0]
+            gen_conversion_cmds(bt, type)
 
         localdata.setVar('type', realt)
         if t not in alltypes:
meta/classes/image_types.bbclass +42 -18

78 lines not on this page · raw message

Signature · 1 line
2.1.4
author
#3 lore raw

PATCH v2 3/6 image.bbclass: additional output in create_symlinks

In reply to Ed Bartosh

From: Patrick Ohly <[email protected]>

When a symlink does not get created, it is useful for debugging to log what would have been created and why it was skipped.

---
 meta/classes/image.bbclass | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)
meta/classes/image.bbclass +5 -3
index 06f2211..46a4d03 100644
@@ -512,9 +512,9 @@ python create_symlinks() {
     if not link_name:
         return
     for type in subimages:
-        if os.path.exists(img_name + imgsuffix + type):
-            dst = deploy_dir + "/" + link_name + "." + type
-            src = img_name + imgsuffix + type
+        dst = deploy_dir + "/" + link_name + "." + type
+        src = img_name + imgsuffix + type
+        if os.path.exists(src):
             bb.note("Creating symlink: %s -> %s" % (dst, src))
             if os.path.islink(dst):
                 if d.getVar('RM_OLD_IMAGE', True) == "1" and \
@@ -522,6 +522,8 @@ python create_symlinks() {
                     os.remove(os.path.realpath(dst))
                 os.remove(dst)
             os.symlink(src, dst)
+        else:
+            bb.note("Skipping symlink, source does not exist: %s -> %s" % (dst, src))
 }
 
 MULTILIBRE_ALLOW_REP =. "${base_bindir}|${base_sbindir}|${bindir}|${sbindir}|${libexecdir}|${sysconfdir}|${nonarch_base_libdir}/udev|/lib/modules/[^/]*/modules.*|"
Signature · 1 line
2.1.4
author
#4 lore raw

PATCH v2 4/6 image creation: support converting masked types

In reply to Ed Bartosh

From: Patrick Ohly <[email protected]>

Conversion to vmdk/vdi/qcow2 is also useful for other base images types, not just for .hdddirect. This can be achieved by definining them as conversion commands and relying on the conversion chaining to convert arbitrary base images.

For this to work when the base image gets created by a masked image type, the additional conversion commands now get executed in a do_image_complete prefunc.

With all of that in place it becomes possible to remove the special purpose code for vmdk/vdi/qcow2 types from image-vm.bbclass and several other classes. This has (intentional!) implications on the valid IMAGE_FSTYPES and the file suffices: now "hdddirect.vmdk/vdi/qcow2" must be used as IMAGE_FSTYPES to select the former special-case types "vmdk/vdi/qcow2", and the image files and links will also have the extra .hdddirect suffix.

This is intentional because it makes it makes it possible to distinguish between virtual machine images created from .hdddirect and those created from other base images.

The new support for virtual machine images can also be combined with compression, thus making it possible to create image files for publication in compressed format, for example with:
IMAGE_FSTYPES = "hdddirect.vdi.xz"

---
 meta/classes/image-live.bbclass                    |  2 +-
 meta/classes/image-vm.bbclass                      | 33 ++--------
 meta/classes/image.bbclass                         | 73 +++++++++++++++-------
 meta/classes/image_types.bbclass                   | 20 ++++--
 meta/conf/documentation.conf                       |  1 -
 .../images/build-appliance-image_14.0.0.bb         |  6 +-
 6 files changed, 75 insertions(+), 60 deletions(-)
meta/classes/image-live.bbclass +1 -1
index ea6ced2..2fbec05 100644
@@ -43,7 +43,7 @@ ROOT_LIVE ?= "root=/dev/ram0"
 INITRD_IMAGE_LIVE ?= "core-image-minimal-initramfs"
 INITRD_LIVE ?= "${DEPLOY_DIR_IMAGE}/${INITRD_IMAGE_LIVE}-${MACHINE}.cpio.gz"
 
-ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.ext4"
+ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ext4"
 
 IMAGE_TYPEDEP_live = "ext4"
 IMAGE_TYPEDEP_iso = "ext4"
meta/classes/image-vm.bbclass +4 -29
index 2bbd9d3..fbfcf03 100644
@@ -26,14 +26,11 @@ do_bootdirectdisk[depends] += "dosfstools-native:do_populate_sysroot \
                                ${PN}:do_image_${VM_ROOTFS_TYPE} \
                                "
 
-IMAGE_TYPEDEP_vmdk = "${VM_ROOTFS_TYPE}"
-IMAGE_TYPEDEP_vdi = "${VM_ROOTFS_TYPE}"
-IMAGE_TYPEDEP_qcow2 = "${VM_ROOTFS_TYPE}"
+VM_ROOTFS_TYPE ?= "ext4"
 IMAGE_TYPEDEP_hdddirect = "${VM_ROOTFS_TYPE}"
-IMAGE_TYPES_MASKED += "vmdk vdi qcow2 hdddirect"
+IMAGE_TYPES_MASKED += "hdddirect"
 
-VM_ROOTFS_TYPE ?= "ext4"
-ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.${VM_ROOTFS_TYPE}"
+ROOTFS ?= "${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.ext4"
 
 # Used by bootloader
 LABELS_VM ?= "boot"
@@ -150,27 +147,5 @@ run_qemu_img (){
     qemu-img convert -O $type ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.hdddirect ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.$type
     ln -sf ${IMAGE_NAME}.$type ${DEPLOY_DIR_IMAGE}/${IMAGE_LINK_NAME}.$type
 }
-create_vmdk_image () {
-    run_qemu_img vmdk
-}
-
-create_vdi_image () {
-    run_qemu_img vdi
-}
-
-create_qcow2_image () {
-    run_qemu_img qcow2
-}
-
-python do_vmimg() {
-    if 'vmdk' in d.getVar('IMAGE_FSTYPES', True):
-        bb.build.exec_func('create_vmdk_image', d)
-    if 'vdi' in d.getVar('IMAGE_FSTYPES', True):
-        bb.build.exec_func('create_vdi_image', d)
-    if 'qcow2' in d.getVar('IMAGE_FSTYPES', True):
-        bb.build.exec_func('create_qcow2_image', d)
-}
 
-addtask bootdirectdisk before do_vmimg
-addtask vmimg after do_bootdirectdisk before do_image_complete
-do_vmimg[depends] += "qemu-native:do_populate_sysroot"
+addtask bootdirectdisk before do_image_complete
meta/classes/image.bbclass +52 -21

103 lines not on this page · raw message

meta/classes/image_types.bbclass +15 -5
index 028bc53..2233850 100644
@@ -247,6 +247,7 @@ IMAGE_DEPENDS_cramfs = "util-linux-native"
 IMAGE_DEPENDS_ext2 = "e2fsprogs-native"
 IMAGE_DEPENDS_ext3 = "e2fsprogs-native"
 IMAGE_DEPENDS_ext4 = "e2fsprogs-native"
+IMAGE_DEPENDS_hdddirect = "${IMAGE_DEPENDS_ext4}"
 IMAGE_DEPENDS_btrfs = "btrfs-tools-native"
 IMAGE_DEPENDS_squashfs = "squashfs-tools-native"
 IMAGE_DEPENDS_squashfs-xz = "squashfs-tools-native"
@@ -265,6 +266,9 @@ IMAGE_DEPENDS_iso = "${IMAGE_DEPENDS_ext4}"
 IMAGE_DEPENDS_hddimg = "${IMAGE_DEPENDS_ext4}"
 
 # This variable is available to request which values are suitable for IMAGE_FSTYPES
+# TODO: several other variations are now also possible, for example ext4.zip,
+# hdddirect.xz, hdddirect.vdi.xz. Which variations are supposed to be listed here
+# and which not? Generate all possible variations dynamically?
 IMAGE_TYPES = " \
     jffs2 jffs2.sum \
     cramfs \
@@ -278,15 +282,15 @@ IMAGE_TYPES = " \
     ubi ubifs multiubi \
     tar tar.gz tar.bz2 tar.xz tar.lz4 \
     cpio cpio.gz cpio.xz cpio.lzma cpio.lz4 \
-    vmdk \
-    vdi \
-    qcow2 \
+    hdddirect.vmdk \
+    hdddirect.vdi \
+    hdddirect.qcow2 \
     hdddirect \
     elf \
     wic wic.gz wic.bz2 wic.lzma \
 "
 
-COMPRESSIONTYPES = "gz bz2 lzma xz lz4 zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap"
+COMPRESSIONTYPES = "gz bz2 lzma xz lz4 zip sum md5sum sha1sum sha224sum sha256sum sha384sum sha512sum bmap vmdk vdi qcow2"
 COMPRESS_CMD_lzma = "lzma -k -f -7 ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 COMPRESS_CMD_gz = "gzip -f -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.gz"
 COMPRESS_CMD_bz2 = "pbzip2 -f -k ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
@@ -294,6 +298,9 @@ COMPRESS_CMD_xz = "xz -f -k -c ${XZ_COMPRESSION_LEVEL} ${XZ_THREADS} --check=${X
 COMPRESS_CMD_lz4 = "lz4c -9 -c ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.lz4"
 COMPRESS_CMD_zip = "zip ${ZIP_COMPRESSION_LEVEL} ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.zip ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}"
 COMPRESS_CMD_sum = "sumtool -i ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} -o ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sum ${JFFS2_SUM_EXTRA_ARGS}"
+COMPRESS_CMD_vmdk = "qemu-img convert -O vmdk ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vmdk"
+COMPRESS_CMD_vdi = "qemu-img convert -O vdi ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.vdi"
+COMPRESS_CMD_qcow2 = "qemu-img convert -O qcow2 ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.qcow2"
 COMPRESS_CMD_md5sum = "md5sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.md5sum"
 COMPRESS_CMD_sha1sum = "sha1sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha1sum"
 COMPRESS_CMD_sha224sum = "sha224sum ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type} > ${IMAGE_NAME}${IMAGE_NAME_SUFFIX}.${type}.sha224sum"
@@ -309,6 +316,9 @@ COMPRESS_DEPENDS_lz4 = "lz4-native"
 COMPRESS_DEPENDS_zip = "zip-native"
 COMPRESS_DEPENDS_sum = "mtd-utils-native"
 COMPRESS_DEPENDS_bmap = "bmap-tools-native"
+COMPRESS_DEPENDS_vmdk = "qemu-native"
+COMPRESS_DEPENDS_vdi = "qemu-native"
+COMPRESS_DEPENDS_qcow2 = "qemu-native"
 
 RUNNABLE_IMAGE_TYPES ?= "ext2 ext3 ext4"
 RUNNABLE_MACHINE_PATTERNS ?= "qemu"
@@ -319,7 +329,7 @@ DEPLOYABLE_IMAGE_TYPES ?= "hddimg iso"
 IMAGE_EXTENSION_live = "hddimg iso"
 
 # The IMAGE_TYPES_MASKED variable will be used to mask out from the IMAGE_FSTYPES,
-# images that will not be built at do_rootfs time: vmdk, vdi, qcow2, hdddirect, hddimg, iso, etc.
+# images that will not be built at do_rootfs time: hdddirect, hddimg, iso, etc.
 IMAGE_TYPES_MASKED ?= ""
 
 # The WICVARS variable is used to define list of bitbake variables used in wic code
meta/conf/documentation.conf +0 -1
index 4234d75..7543362 100644
@@ -56,7 +56,6 @@ do_testsdk[doc] = "Installs an SDK and performs runtime tests on the tools insta
 do_uboot_mkimage[doc] = "Creates a uImage file from the kernel for the U-Boot bootloader"
 do_unpack[doc] = "Unpacks the source code into a working directory"
 do_validate_branches[doc] = "Ensures that the source/meta branches are on the locations specified by their SRCREV values for a linux-yocto style kernel"
-do_vmimg[doc] = "Creates an image for use with VMware or VirtualBox compatible virtual machine hosts (based on IMAGE_FSTYPES either .vmdk or .vdi)"
 
 # DESCRIPTIONS FOR VARIABLES #
 
meta/recipes-core/images/build-appliance-image_14.0.0.bb +3 -3
index 569fb3a..48f8282 100644
@@ -17,7 +17,7 @@ IMAGE_ROOTFS_EXTRA_SPACE = "41943040"
 APPEND += "rootfstype=ext4 quiet"
 
 DEPENDS = "zip-native"
-IMAGE_FSTYPES = "vmdk"
+IMAGE_FSTYPES = "hdddirect.vmdk"
 
 inherit core-image module-base
 
@@ -106,7 +106,7 @@ create_bundle_files () {
 	cd ${WORKDIR}
 	mkdir -p Yocto_Build_Appliance
 	cp *.vmx* Yocto_Build_Appliance
-	ln -sf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk
+	ln -sf ${DEPLOY_DIR_IMAGE}/${IMAGE_NAME}.hdddirect.vmdk Yocto_Build_Appliance/Yocto_Build_Appliance.vmdk
 	zip -r ${DEPLOY_DIR_IMAGE}/Yocto_Build_Appliance-${DATETIME}.zip Yocto_Build_Appliance
 	ln -sf Yocto_Build_Appliance-${DATETIME}.zip ${DEPLOY_DIR_IMAGE}/Yocto_Build_Appliance.zip 
 }
@@ -116,5 +116,5 @@ python do_bundle_files() {
     bb.build.exec_func('create_bundle_files', d)
 }
 
-addtask bundle_files after do_vmimg before do_build
+addtask bundle_files after do_image_complete before do_build
 do_bundle_files[nostamp] = "1"
Signature · 1 line
2.1.4
author
#5 lore raw

PATCH v2 5/6 image.bbclass: rename COMPRESS(ION) to CONVERSION

In reply to Ed Bartosh

From: Patrick Ohly <[email protected]>

With the enhanced functionality, the term "compression" is no longer accurate, because the mechanism also gets used for conversion operations that do not actually compress data.

It is possible to remove this naming problem in a backward-compatible manner by including COMPRESSIONTYPES in CONVERSIONTYPES and checking for the old COMPRESS_CMD/DEPENDS as fallbacks.

---
 meta/classes/image.bbclass             | 10 +++--
 meta/classes/image_types.bbclass       | 67 ++++++++++++++++++----------------
 meta/classes/image_types_uboot.bbclass | 18 ++++-----
 3 files changed, 51 insertions(+), 44 deletions(-)
meta/classes/image.bbclass +6 -4
index a1751dc..47d1c9f 100644
@@ -118,7 +118,7 @@ def rootfs_variables(d):
                  'IMAGE_ROOTFS_MAXSIZE','IMAGE_NAME','IMAGE_LINK_NAME','IMAGE_MANIFEST','DEPLOY_DIR_IMAGE','RM_OLD_IMAGE','IMAGE_FSTYPES','IMAGE_INSTALL_COMPLEMENTARY','IMAGE_LINGUAS',
                  'MULTILIBRE_ALLOW_REP','MULTILIB_TEMP_ROOTFS','MULTILIB_VARIANTS','MULTILIBS','ALL_MULTILIB_PACKAGE_ARCHS','MULTILIB_GLOBAL_VARIANTS','BAD_RECOMMENDATIONS','NO_RECOMMENDATIONS',
                  'PACKAGE_ARCHS','PACKAGE_CLASSES','TARGET_VENDOR','TARGET_ARCH','TARGET_OS','OVERRIDES','BBEXTENDVARIANT','FEED_DEPLOYDIR_BASE_URI','INTERCEPT_DIR','USE_DEVFS',
-                 'COMPRESSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED']
+                 'CONVERSIONTYPES', 'IMAGE_GEN_DEBUGFS', 'ROOTFS_RO_UNNEEDED']
     variables.extend(rootfs_command_variables(d))
     variables.extend(variable_depends(d))
     return " ".join(variables)
@@ -149,6 +149,7 @@ inherit ${IMAGE_TYPE_live}
 IMAGE_TYPE_vm = '${@ "image-vm" if image_type_active("hdddirect", d) else ""}'
 inherit ${IMAGE_TYPE_vm}
 
+
 python () {
     deps = " " + image_getdepends(d)
     d.appendVarFlag('do_rootfs', 'depends', deps)
@@ -304,7 +305,7 @@ python setup_debugfs () {
 
 python () {
     vardeps = set()
-    # We allow COMPRESSIONTYPES to have duplicates. That avoids breaking
+    # We allow CONVERSIONTYPES to have duplicates. That avoids breaking
     # derived distros when OE-core or some other layer independently adds
     # the same type. There is still only one command for each type, but
     # presumably the commands will do the same when the type is the same,
@@ -312,7 +313,7 @@ python () {
     #
     # Without de-duplication, gen_conversion_cmds() below
     # would create the same compression command multiple times.
-    ctypes = set(d.getVar('COMPRESSIONTYPES', True).split())
+    ctypes = set(d.getVar('CONVERSIONTYPES', True).split())
     old_overrides = d.getVar('OVERRIDES', 0)
 
     basetypes = {}
@@ -404,9 +405,10 @@ python () {
                     # Create input image first.
                     gen_conversion_cmds(type)
                     localdata.setVar('type', type)
-                    cmd = "\t" + localdata.getVar("COMPRESS_CMD_" + ctype, True)
+                    cmd = "\t" + (localdata.getVar("CONVERSION_CMD_" + ctype, True) or localdata.getVar("COMPRESS_CMD_" + ctype, True))
                     if cmd not in cmds:
                         cmds.append(cmd)
+                    vardeps.add('CONVERSION_CMD_' + ctype)
                     vardeps.add('COMPRESS_CMD_' + ctype)
                     subimage = type + "." + ctype
                     if subimage not in subimages:
meta/classes/image_types.bbclass +36 -31

79 lines not on this page · raw message

meta/classes/image_types_uboot.bbclass +9 -9
index 19e4aa2..f72d9b2 100644
@@ -8,19 +8,19 @@ oe_mkimage () {
     fi
 }
 
-COMPRESSIONTYPES += "gz.u-boot bz2.u-boot lzma.u-boot u-boot"
+CONVERSIONTYPES += "gz.u-boot bz2.u-boot lzma.u-boot u-boot"
 
-COMPRESS_DEPENDS_u-boot = "u-boot-mkimage-native"
-COMPRESS_CMD_u-boot      = "oe_mkimage ${IMAGE_NAME}.rootfs.${type} none"
+CONVERSION_DEPENDS_u-boot = "u-boot-mkimage-native"
+CONVERSION_CMD_u-boot      = "oe_mkimage ${IMAGE_NAME}.rootfs.${type} none"
 
-COMPRESS_DEPENDS_gz.u-boot = "u-boot-mkimage-native"
-COMPRESS_CMD_gz.u-boot      = "${COMPRESS_CMD_gz}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.gz gzip clean"
+CONVERSION_DEPENDS_gz.u-boot = "u-boot-mkimage-native"
+CONVERSION_CMD_gz.u-boot      = "${CONVERSION_CMD_gz}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.gz gzip clean"
 
-COMPRESS_DEPENDS_bz2.u-boot = "u-boot-mkimage-native"
-COMPRESS_CMD_bz2.u-boot      = "${COMPRESS_CMD_bz2}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.bz2 bzip2 clean"
+CONVERSION_DEPENDS_bz2.u-boot = "u-boot-mkimage-native"
+CONVERSION_CMD_bz2.u-boot      = "${CONVERSION_CMD_bz2}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.bz2 bzip2 clean"
 
-COMPRESS_DEPENDS_lzma.u-boot = "u-boot-mkimage-native"
-COMPRESS_CMD_lzma.u-boot      = "${COMPRESS_CMD_lzma}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.lzma lzma clean"
+CONVERSION_DEPENDS_lzma.u-boot = "u-boot-mkimage-native"
+CONVERSION_CMD_lzma.u-boot      = "${CONVERSION_CMD_lzma}; oe_mkimage ${IMAGE_NAME}.rootfs.${type}.lzma lzma clean"
 
 IMAGE_TYPES += "ext2.u-boot ext2.gz.u-boot ext2.bz2.u-boot ext2.lzma.u-boot ext3.gz.u-boot ext4.gz.u-boot cpio.gz.u-boot"
 
Signature · 1 line
2.1.4
author
#6 lore raw

PATCH v2 6/6 image.bbclass: prefer specialized image creation methods over chaining

In reply to Ed Bartosh

From: Patrick Ohly <[email protected]>

If a certain image type can be created both by applying conversion commands to a base type as well as via a specialized command (like a fictious IMAGE_CMD_ext4.xz, or some dedicated task, as indicated by IMAGE_TYPES_MASKED = ext4.xz), then pick the more specialized method and skip generating the conversion commands.

That way developers can choose between using the builtin conversion support or re-implementing certain types, for example more efficiently without intermediate files.

Fixes [YOCTO #9076]
---
 meta/classes/image.bbclass       | 17 ++++++++++-------
 meta/classes/image_types.bbclass | 16 +++++++++++++---
 2 files changed, 23 insertions(+), 10 deletions(-)
meta/classes/image.bbclass +10 -7
index 47d1c9f..ebbdf06 100644
@@ -326,7 +326,7 @@ python () {
             alltypes.append("debugfs_" + t)
 
     def _add_type(t):
-        baset = image_split_type(t, ctypes)[0]
+        baset = image_split_type(t, ctypes, d)[0]
         input_t = t
         if baset not in basetypes:
             basetypes[baset]= []
@@ -345,7 +345,7 @@ python () {
             if dep not in alltypes:
                 alltypes.append(dep)
             _add_type(dep)
-            basedep = image_split_type(dep, ctypes)[0]
+            basedep = image_split_type(dep, ctypes, d)[0]
             typedeps[baset].add(basedep)
 
         if baset != input_t:
@@ -396,14 +396,17 @@ python () {
         d.delVarFlag('IMAGE_CMD_' + realt, 'func')
 
         rm_tmp_images = set()
-        def gen_conversion_cmds(bt):
+        def gen_conversion_cmds(basetype, type):
+            if basetype == type:
+                # Done, no further conversions needed.
+                return
             for ctype in ctypes:
-                if bt.endswith("." + ctype):
-                    type = bt[0:-len(ctype) - 1]
+                if type.endswith("." + ctype):
+                    type = type[0:-len(ctype) - 1]
                     if type.startswith("debugfs_"):
                         type = type[8:]
                     # Create input image first.
-                    gen_conversion_cmds(type)
+                    gen_conversion_cmds(basetype, type)
                     localdata.setVar('type', type)
                     cmd = "\t" + (localdata.getVar("CONVERSION_CMD_" + ctype, True) or localdata.getVar("COMPRESS_CMD_" + ctype, True))
                     if cmd not in cmds:
@@ -418,7 +421,7 @@ python () {
 
         for type in basetypes[t]:
             # Probably t == bt, but better check explicitly, perhaps that'll change.
-            bt = image_split_type(type, ctypes)[0]
+            bt = image_split_type(type, ctypes, d)[0]
             gen_conversion_cmds(bt, type)
 
         localdata.setVar('type', realt)
meta/classes/image_types.bbclass +13 -3
index 225142b..8e79731 100644
@@ -9,10 +9,20 @@ IMAGE_NAME_SUFFIX ??= ".rootfs"
 # set this value to 2048 (2MiB alignment).
 IMAGE_ROOTFS_ALIGNMENT ?= "1"
 
-def image_split_type(type, allctypes):
+def image_split_type(type, allctypes, d):
     '''Returns (basetype, set of compression types in use).'''
     basetype = type
     compressiontypes = set()
+
+    # Abort stripping the type further if "basetype" has a matching
+    # IMAGE_CMD or is a masked image type. For example, if there
+    # was a fictional IMAGE_CMD_ext4.xz which creates a compressed
+    # file directly, then we should use that instead of using
+    # IMAGE_CMD_ext4 + CONVERSION_CMD_xz.
+    if d.getVar('IMAGE_CMD_' + basetype, True) or \
+        basetype in d.getVar('IMAGE_TYPES_MASKED', True).split():
+            return (basetype, compressiontypes)
+
     for ctype in allctypes:
         if type.endswith("." + ctype):
              basetype = type[:-len("." + ctype)]
@@ -21,7 +31,7 @@ def image_split_type(type, allctypes):
 
     if basetype != type:
         # New base type itself might be generated by a conversion command.
-        basetype, newctypes = image_split_type(basetype, allctypes)
+        basetype, newctypes = image_split_type(basetype, allctypes, d)
         compressiontypes.update(newctypes)
 
     return (basetype, compressiontypes)
@@ -38,7 +48,7 @@ def image_getdepends(d):
     deps = []
     ctypes = set(d.getVar('COMPRESSIONTYPES', True).split())
     for type in (d.getVar('IMAGE_FSTYPES', True) or "").split():
-        basetype, compressiontypes = image_split_type(type, ctypes)
+        basetype, compressiontypes = image_split_type(type, ctypes, d)
         for ctype in compressiontypes:
             adddep(d.getVar("COMPRESS_DEPENDS_%s" % ctype, True), deps)
         for typedepends in (d.getVar("IMAGE_TYPEDEP_%s" % basetype, True) or "").split():
Signature · 1 line
2.1.4