Implemented image compressing: poky-contrib:ed/wic/compress-with

PATCH In review Ed Bartosh 4 participants 12 messages Jun 23, 2015 – Aug 16, 2015

lore

Share & tools

lore

Ed Bartosh

author

2015-06-23 09:53 UTC #0 lore raw

cover Implemented image compressing: poky-contrib:ed/wic/compress-with

Hi reviewers,

This patchset adds --compress-with command line option to 'wic create'. It supports gzip, bzip2 and xz compressors.

Test cases for this funtionality are also included.

As always I couldn't resist making small code cleanup change while working on this. Included that as well.

Fixes [YOCTO #7593]

Note: This patchset depends on poky-contrib:ed/wic/misc

Ed Bartosh (6):
wic: Add --compress-with command line option
wic: Add new argument to wic_create function
wic: Add new argument to direct plugin
wic: Implement image compressing
wic: Make code more pythonic
wic: oe-selftest: Test image compressing

 meta/lib/oeqa/selftest/wic.py                   | 30 +++++++++++++++++++++++++
 scripts/lib/image/engine.py                     | 17 +++++---------
 scripts/lib/image/help.py                       |  4 ++++
 scripts/lib/wic/imager/direct.py                | 17 ++++++++++++--
 scripts/lib/wic/plugins/imager/direct_plugin.py |  4 +++-
 scripts/wic                                     |  5 ++++-
 6 files changed, 61 insertions(+), 16 deletions(-)

--
Regards,
Ed

Ed Bartosh

author

2015-06-23 09:54 UTC #1 lore raw

PATCH 1/6 wic: Add --compress-with command line option

Added -c/--compress-with command line option to 'wic create' subcommand. This option is used to specify compressor utility to compress the image produced by wic. gzip, bzip2 and xz compressors are supported in this implementation.

Signed-off-by Ed Bartosh <[email protected]>

scripts/lib/image/help.py +4 -0

index fe66548..c8b2fab 100644
@@ -152,6 +152,7 @@ SYNOPSIS
         [-e | --image-name] [-s, --skip-build-check] [-D, --debug]
         [-r, --rootfs-dir] [-b, --bootimg-dir]
         [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs]
+        [-c, --compress-with]
 
 DESCRIPTION
     This command creates an OpenEmbedded image based on the 'OE

@@ -218,6 +219,9 @@ DESCRIPTION
     The -o option can be used to place the image in a directory with a
     different name and location.
 
+    The -c option is used to specify compressor utility to compress
+    an image. gzip, bzip2 and xz compressors are supported.
+
     The set of properties available for a given image type can be
     listed using the 'wic list' command.
 """

scripts/wic +3 -0

index 7b0c3b2..8869fba 100755
@@ -105,6 +105,9 @@ def wic_create_subcommand(args, usage_str):
     parser.add_option("-p", "--skip-build-check", dest="build_check",
                       action="store_false", default=True, help="skip the build check")
     parser.add_option("-f", "--build-rootfs", action="store_true", help="build rootfs")
+    parser.add_option("-c", "--compress-with", choices=("gzip", "bzip2", "xz"),
+                      dest='compressor',
+                      help="compress image with specified compressor")
     parser.add_option("-D", "--debug", dest="debug", action="store_true",
                       default=False, help="output debug information")

Signature · 1 line

2.1.4

Ed Bartosh

author

2015-06-23 09:54 UTC #2 lore raw

PATCH 2/6 wic: Add new argument to wic_create function

In reply to Ed Bartosh

Added 'compressor' argument to wic_create to pass a name of compressor utility to the wic engine.

Signed-off-by Ed Bartosh <[email protected]>

scripts/lib/image/engine.py +3 -1

index 29099ee..047c40d 100644
@@ -179,7 +179,8 @@ def list_source_plugins():
 
 
 def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
-               native_sysroot, scripts_path, image_output_dir, debug):
+               native_sysroot, scripts_path, image_output_dir,
+               compressor, debug):
     """Create image
 
     wks_file - user-defined OE kickstart file

@@ -189,6 +190,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
     native_sysroot - absolute path to the build's native sysroots dir
     scripts_path - absolute path to /scripts dir
     image_output_dir - dirname to create for image
+    compressor - compressor utility to compress the image
 
     Normally, the values for the build artifacts values are determined
     by 'wic -e' from the output of the 'bitbake -e' command given an

scripts/wic +1 -1

index 8869fba..f9be972 100755
@@ -228,7 +228,7 @@ def wic_create_subcommand(args, usage_str):
     print "Creating image(s)...\n"
     engine.wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
                       native_sysroot, scripts_path, image_output_dir,
-                      options.debug)
+                      options.compressor, options.debug)
 
 
 def wic_list_subcommand(args, usage_str):

Signature · 1 line

2.1.4

Ed Bartosh

author

2015-06-23 09:54 UTC #3 lore raw

PATCH 3/6 wic: Add new argument to direct plugin

In reply to Ed Bartosh

Added 'compressor' argument to Direct plugin API to pass a name of compressor utility.

Signed-off-by Ed Bartosh <[email protected]>

scripts/lib/image/engine.py +1 -0

index 047c40d..d5c34b4 100644
@@ -217,6 +217,7 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
         sys.exit(1)
 
     direct_args = list()
+    direct_args.insert(0, compressor or "")
     direct_args.insert(0, oe_builddir)
     direct_args.insert(0, image_output_dir)
     direct_args.insert(0, wks_file)

scripts/lib/wic/imager/direct.py +1 -1

index 2290ecd..f3f20e0 100644
@@ -50,7 +50,7 @@ class DirectImageCreator(BaseImageCreator):
     """
 
     def __init__(self, oe_builddir, image_output_dir, rootfs_dir, bootimg_dir,
-                 kernel_dir, native_sysroot, creatoropts=None):
+                 kernel_dir, native_sysroot, compressor, creatoropts=None):
         """
         Initialize a DirectImageCreator instance.

scripts/lib/wic/plugins/imager/direct_plugin.py +3 -1

index 07811bd..db01c41 100644
@@ -62,7 +62,7 @@ class DirectPlugin(ImagerPlugin):
         """
         Create direct image, called from creator as 'direct' cmd
         """
-        if len(args) != 7:
+        if len(args) != 8:
             raise errors.Usage("Extra arguments given")
 
         native_sysroot = args[0]

@@ -75,6 +75,7 @@ class DirectPlugin(ImagerPlugin):
 
         image_output_dir = args[5]
         oe_builddir = args[6]
+        compressor = args[7]
 
         krootfs_dir = cls.__rootfs_dir_to_dict(rootfs_dir)

@@ -86,6 +87,7 @@ class DirectPlugin(ImagerPlugin):
                                             bootimg_dir,
                                             kernel_dir,
                                             native_sysroot,
+                                            compressor,
                                             creatoropts)
 
         try:

Signature · 1 line

2.1.4

Ed Bartosh

author

2015-06-23 09:54 UTC #4 lore raw

PATCH 4/6 wic: Implement image compressing

In reply to Ed Bartosh

Implemented compressing result image with specified compressor. Updated reporting code to show compressed image.

[YOCTO #7593]

Signed-off-by Ed Bartosh <[email protected]>

scripts/lib/wic/imager/direct.py +14 -1

index f3f20e0..2ea7e4e 100644
@@ -33,6 +33,7 @@ from wic.utils.partitionedfs import Image
 from wic.utils.errors import CreatorError, ImageError
 from wic.imager.baseimager import BaseImageCreator
 from wic.plugin import pluginmgr
+from wic.utils.oe.misc import exec_cmd
 
 disk_methods = {
     "do_install_disk":None,

@@ -71,6 +72,7 @@ class DirectImageCreator(BaseImageCreator):
         self.bootimg_dir = bootimg_dir
         self.kernel_dir = kernel_dir
         self.native_sysroot = native_sysroot
+        self.compressor = compressor
 
     def __get_part_num(self, num, parts):
         """calculate the real partition number, accounting for partitions not

@@ -318,6 +320,13 @@ class DirectImageCreator(BaseImageCreator):
                                                         self.bootimg_dir,
                                                         self.kernel_dir,
                                                         self.native_sysroot)
+        # Compress the image
+        if self.compressor:
+            for disk_name, disk in self.__image.disks.items():
+                full_path = self._full_path(self.__imgdir, disk_name, "direct")
+                msger.debug("Compressing disk %s with %s" % \
+                            (disk_name, self.compressor))
+                exec_cmd("%s %s" % (self.compressor, full_path))
 
     def print_outimage_info(self):
         """

@@ -328,7 +337,11 @@ class DirectImageCreator(BaseImageCreator):
         parts = self._get_parts()
 
         for disk_name, disk in self.__image.disks.items():
-            full_path = self._full_path(self.__imgdir, disk_name, "direct")
+            extension = "direct" + {"gzip": ".gz",
+                                    "bzip2": ".bz2",
+                                    "xz": ".xz",
+                                    "": ""}.get(self.compressor)
+            full_path = self._full_path(self.__imgdir, disk_name, extension)
             msg += '  %s\n\n' % full_path
 
         msg += 'The following build artifacts were used to create the image(s):\n'

Signature · 1 line

2.1.4

Philip Balister

2015-06-23 11:22 UTC #5 lore raw

Re: [wic][PATCH 4/6] wic: Implement image compressing

Can you add a way to enable multiple threads for xz? I found this helps speed up file creation.

Philip

Quoted 55 lines · Ed Bartosh

Implemented compressing result image with specified compressor. Updated reporting code to show compressed image.

[YOCTO #7593]

Signed-off-by Ed Bartosh <[email protected]>

scripts/lib/wic/imager/direct.py +14 -1

index f3f20e0..2ea7e4e 100644
@@ -33,6 +33,7 @@ from wic.utils.partitionedfs import Image
 from wic.utils.errors import CreatorError, ImageError
 from wic.imager.baseimager import BaseImageCreator
 from wic.plugin import pluginmgr
+from wic.utils.oe.misc import exec_cmd
 
 disk_methods = {
     "do_install_disk":None,

@@ -71,6 +72,7 @@ class DirectImageCreator(BaseImageCreator):
         self.bootimg_dir = bootimg_dir
         self.kernel_dir = kernel_dir
         self.native_sysroot = native_sysroot
+        self.compressor = compressor
 
     def __get_part_num(self, num, parts):
         """calculate the real partition number, accounting for partitions not

@@ -318,6 +320,13 @@ class DirectImageCreator(BaseImageCreator):
                                                         self.bootimg_dir,
                                                         self.kernel_dir,
                                                         self.native_sysroot)
+        # Compress the image
+        if self.compressor:
+            for disk_name, disk in self.__image.disks.items():
+                full_path = self._full_path(self.__imgdir, disk_name, "direct")
+                msger.debug("Compressing disk %s with %s" % \
+                            (disk_name, self.compressor))
+                exec_cmd("%s %s" % (self.compressor, full_path))
 
     def print_outimage_info(self):
         """

@@ -328,7 +337,11 @@ class DirectImageCreator(BaseImageCreator):
         parts = self._get_parts()
 
         for disk_name, disk in self.__image.disks.items():
-            full_path = self._full_path(self.__imgdir, disk_name, "direct")
+            extension = "direct" + {"gzip": ".gz",
+                                    "bzip2": ".bz2",
+                                    "xz": ".xz",
+                                    "": ""}.get(self.compressor)
+            full_path = self._full_path(self.__imgdir, disk_name, extension)
             msg += '  %s\n\n' % full_path
 
         msg += 'The following build artifacts were used to create the image(s):\n'

Ed Bartosh

author

2015-06-24 15:23 UTC #6 lore raw

Re: [wic][PATCH 4/6] wic: Implement image compressing

Quoted 3 lines · Philip Balister

Can you add a way to enable multiple threads for xz? I found this helps speed up file creation.

I thought about this and other compressor options, but didn't find a good way to add them. Introducing new commandline option or config value doesn't look convenient to use from my point of view.

Can you propose something better than that?

Regards,
Ed

Quoted 58 lines

Philip

Quoted 55 lines · Ed Bartosh

Implemented compressing result image with specified compressor. Updated reporting code to show compressed image.

[YOCTO #7593]

Signed-off-by Ed Bartosh <[email protected]>

scripts/lib/wic/imager/direct.py +14 -1

index f3f20e0..2ea7e4e 100644
@@ -33,6 +33,7 @@ from wic.utils.partitionedfs import Image
 from wic.utils.errors import CreatorError, ImageError
 from wic.imager.baseimager import BaseImageCreator
 from wic.plugin import pluginmgr
+from wic.utils.oe.misc import exec_cmd
 
 disk_methods = {
     "do_install_disk":None,

@@ -71,6 +72,7 @@ class DirectImageCreator(BaseImageCreator):
         self.bootimg_dir = bootimg_dir
         self.kernel_dir = kernel_dir
         self.native_sysroot = native_sysroot
+        self.compressor = compressor
 
     def __get_part_num(self, num, parts):
         """calculate the real partition number, accounting for partitions not

@@ -318,6 +320,13 @@ class DirectImageCreator(BaseImageCreator):
                                                         self.bootimg_dir,
                                                         self.kernel_dir,
                                                         self.native_sysroot)
+        # Compress the image
+        if self.compressor:
+            for disk_name, disk in self.__image.disks.items():
+                full_path = self._full_path(self.__imgdir, disk_name, "direct")
+                msger.debug("Compressing disk %s with %s" % \
+                            (disk_name, self.compressor))
+                exec_cmd("%s %s" % (self.compressor, full_path))
 
     def print_outimage_info(self):
         """

@@ -328,7 +337,11 @@ class DirectImageCreator(BaseImageCreator):
         parts = self._get_parts()
 
         for disk_name, disk in self.__image.disks.items():
-            full_path = self._full_path(self.__imgdir, disk_name, "direct")
+            extension = "direct" + {"gzip": ".gz",
+                                    "bzip2": ".bz2",
+                                    "xz": ".xz",
+                                    "": ""}.get(self.compressor)
+            full_path = self._full_path(self.__imgdir, disk_name, extension)
             msg += '  %s\n\n' % full_path
 
         msg += 'The following build artifacts were used to create the image(s):\n'

Signature · 3 lines

--
Regards,
Ed

João Henrique Ferreira de Freitas

2015-06-27 22:20 UTC #7 lore raw

Re: [wic][PATCH 4/6] wic: Implement image compressing

Hi,

Maybe using an environment variable?

Quoted 71 lines · Ed Bartosh

Quoted 3 lines · Philip Balister

Can you add a way to enable multiple threads for xz? I found this helps speed up file creation.

I thought about this and other compressor options, but didn't find a good way to add them. Introducing new commandline option or config value doesn't look convenient to use from my point of view.

Can you propose something better than that?

Regards,
Ed

Quoted 58 lines

Philip

Quoted 55 lines · Ed Bartosh

Implemented compressing result image with specified compressor. Updated reporting code to show compressed image.

[YOCTO #7593]

Signed-off-by Ed Bartosh <[email protected]>

scripts/lib/wic/imager/direct.py +14 -1

index f3f20e0..2ea7e4e 100644
@@ -33,6 +33,7 @@ from wic.utils.partitionedfs import Image
  from wic.utils.errors import CreatorError, ImageError
  from wic.imager.baseimager import BaseImageCreator
  from wic.plugin import pluginmgr
+from wic.utils.oe.misc import exec_cmd
 
  disk_methods = {
      "do_install_disk":None,

@@ -71,6 +72,7 @@ class DirectImageCreator(BaseImageCreator):
          self.bootimg_dir = bootimg_dir
          self.kernel_dir = kernel_dir
          self.native_sysroot = native_sysroot
+        self.compressor = compressor
 
      def __get_part_num(self, num, parts):
          """calculate the real partition number, accounting for partitions not

@@ -318,6 +320,13 @@ class DirectImageCreator(BaseImageCreator):
                                                          self.bootimg_dir,
                                                          self.kernel_dir,
                                                          self.native_sysroot)
+        # Compress the image
+        if self.compressor:
+            for disk_name, disk in self.__image.disks.items():
+                full_path = self._full_path(self.__imgdir, disk_name, "direct")
+                msger.debug("Compressing disk %s with %s" % \
+                            (disk_name, self.compressor))
+                exec_cmd("%s %s" % (self.compressor, full_path))
 
      def print_outimage_info(self):
          """

@@ -328,7 +337,11 @@ class DirectImageCreator(BaseImageCreator):
          parts = self._get_parts()
 
          for disk_name, disk in self.__image.disks.items():
-            full_path = self._full_path(self.__imgdir, disk_name, "direct")
+            extension = "direct" + {"gzip": ".gz",
+                                    "bzip2": ".bz2",
+                                    "xz": ".xz",
+                                    "": ""}.get(self.compressor)
+            full_path = self._full_path(self.__imgdir, disk_name, extension)
              msg += '  %s\n\n' % full_path
 
          msg += 'The following build artifacts were used to create the image(s):\n'

Philip Balister

2015-06-29 15:45 UTC #8 lore raw

Re: [wic][PATCH 4/6] wic: Implement image compressing

Quoted 4 lines · João Henrique Ferreira de Freitas

Hi,
Maybe using an environment variable?

Handle it the same way we handle PARALLEL_MAKE and BBTHREADS.

Philip

Quoted 80 lines · Ed Bartosh

Quoted 76 lines · Philip Balister

Quoted 3 lines

Can you add a way to enable multiple threads for xz? I found this helps speed up file creation.

I thought about this and other compressor options, but didn't find a good way to add them. Introducing new commandline option or config value doesn't
look convenient to use from my point of view.

Can you propose something better than that?

Regards,
Ed

Quoted 62 lines

Philip

Quoted 59 lines · Ed Bartosh

Implemented compressing result image with specified compressor. Updated reporting code to show compressed image.

[YOCTO #7593]

Signed-off-by Ed Bartosh <[email protected]>

scripts/lib/wic/imager/direct.py +2 -0

index f3f20e0..2ea7e4e 100644
@@ -33,6 +33,7 @@ from wic.utils.partitionedfs import Image
  from wic.utils.errors import CreatorError, ImageError
  from wic.imager.baseimager import BaseImageCreator
  from wic.plugin import pluginmgr
+from wic.utils.oe.misc import exec_cmd
    disk_methods = {
      "do_install_disk":None,

@@ -71,6 +72,7 @@ class DirectImageCreator(BaseImageCreator):
          self.bootimg_dir = bootimg_dir
          self.kernel_dir = kernel_dir
          self.native_sysroot = native_sysroot
+        self.compressor = compressor
        def __get_part_num(self, num, parts):
          """calculate the real partition number, accounting for

partitions not

scripts/lib/wic/imager/direct.py +4 -0

@@ -318,6 +320,13 @@ class DirectImageCreator(BaseImageCreator):
 
 self.bootimg_dir,
 
 self.kernel_dir,
 
 self.native_sysroot)
+        # Compress the image
+        if self.compressor:
+            for disk_name, disk in self.__image.disks.items():
+                full_path = self._full_path(self.__imgdir,

disk_name, "direct")

scripts/lib/wic/imager/direct.py +8 -1

+                msger.debug("Compressing disk %s with %s" % \
+                            (disk_name, self.compressor))
+                exec_cmd("%s %s" % (self.compressor, full_path))
        def print_outimage_info(self):
          """

@@ -328,7 +337,11 @@ class DirectImageCreator(BaseImageCreator):
          parts = self._get_parts()
            for disk_name, disk in self.__image.disks.items():
-            full_path = self._full_path(self.__imgdir, disk_name,
 "direct")
+            extension = "direct" + {"gzip": ".gz",
+                                    "bzip2": ".bz2",
+                                    "xz": ".xz",
+                                    "": ""}.get(self.compressor)
+            full_path = self._full_path(self.__imgdir, disk_name,
 extension)
              msg += '  %s\n\n' % full_path
            msg += 'The following build artifacts were used to

create the image(s):\n'

Khem Raj

2015-08-16 05:38 UTC #9 lore raw

Re: [wic][PATCH 4/6] wic: Implement image compressing

In reply to Ed Bartosh

Quoted 9 lines · Ed Bartosh

Quoted 3 lines · Philip Balister
Can you add a way to enable multiple threads for xz? I found this helps speed up file creation.
I thought about this and other compressor options, but didn't find a good way to add them. Introducing new commandline option or config value doesn't look convenient to use from my point of view.
Can you propose something better than that?

there is pxz that can be used on same lines as pigz and pbzip2

see
http://manpages.ubuntu.com/manpages/trusty/man1/pxz.1.html

Ed Bartosh

author

2015-06-23 09:54 UTC #10 lore raw

PATCH 5/6 wic: Make code more pythonic

In reply to Ed Bartosh

Used more pythonic way to make a list.

Signed-off-by Ed Bartosh <[email protected]>

scripts/lib/image/engine.py +2 ~~-12~~

index d5c34b4..47950f8 100644
@@ -216,23 +216,13 @@ def wic_create(wks_file, rootfs_dir, bootimg_dir, kernel_dir,
         print "BUILDDIR not found, exiting. (Did you forget to source oe-init-build-env?)"
         sys.exit(1)
 
-    direct_args = list()
-    direct_args.insert(0, compressor or "")
-    direct_args.insert(0, oe_builddir)
-    direct_args.insert(0, image_output_dir)
-    direct_args.insert(0, wks_file)
-    direct_args.insert(0, rootfs_dir)
-    direct_args.insert(0, bootimg_dir)
-    direct_args.insert(0, kernel_dir)
-    direct_args.insert(0, native_sysroot)
-    direct_args.insert(0, "direct")
-
     if debug:
         msger.set_loglevel('debug')
 
     cr = creator.Creator()
 
-    cr.main(direct_args)
+    cr.main(["direct", native_sysroot, kernel_dir, bootimg_dir, rootfs_dir,
+             wks_file, image_output_dir, oe_builddir, compressor or ""])
 
     print "\nThe image(s) were created using OE kickstart file:\n  %s" % wks_file

Signature · 1 line

2.1.4

Ed Bartosh

author

2015-06-23 09:54 UTC #11 lore raw

PATCH 6/6 wic: oe-selftest: Test image compressing

In reply to Ed Bartosh

Added 4 new testcases for 'wic --compress-with <compressor>' functionality.

Signed-off-by Ed Bartosh <[email protected]>

meta/lib/oeqa/selftest/wic.py +30 -0

index 47b70fd..5385562 100644
@@ -103,3 +103,33 @@ class Wic(oeSelfTest):
     def test11_help_kickstart(self):
         """Test wic help kickstart"""
         self.assertEqual(0, runCmd('wic help kickstart').status)
+
+    def test12_compress_gzip(self):
+        """Test compressing an image with gzip"""
+        self.assertEqual(0, runCmd("wic create directdisk "
+                                   "--image-name core-image-minimal "
+                                   "-c gzip").status)
+        self.assertEqual(1, len(glob(self.resultdir + \
+                                         "directdisk-*.direct.gz")))
+
+    def test13_compress_gzip(self):
+        """Test compressing an image with bzip2"""
+        self.assertEqual(0, runCmd("wic create directdisk "
+                                   "--image-name core-image-minimal "
+                                   "-c bzip2").status)
+        self.assertEqual(1, len(glob(self.resultdir + \
+                                         "directdisk-*.direct.bz2")))
+
+    def test14_compress_gzip(self):
+        """Test compressing an image with xz"""
+        self.assertEqual(0, runCmd("wic create directdisk "
+                                   "--image-name core-image-minimal "
+                                   "-c xz").status)
+        self.assertEqual(1, len(glob(self.resultdir + \
+                                         "directdisk-*.direct.xz")))
+
+    def test15_wrong_compressor(self):
+        """Test how wic breaks if wrong compressor is provided"""
+        self.assertEqual(2, runCmd("wic create directdisk "
+                                   "--image-name core-image-minimal "
+                                   "-c wrong", ignore_status=True).status)

Signature · 1 line

2.1.4