wic: bugfixes & --fixed-size support, tests, oe-selftest: minor fixes
v3 cover wic: bugfixes & --fixed-size support, tests, oe-selftest: minor fixes
v3 of patch series previously posted here [1].
I have noticed that Ross has cherry-picked some patches into his pull request to master. Just for reference, the patches are included in this series, but have not been changed since the previous version. The patches in question are:
oe-selftest: fix handling of test cases without ID in --list-tests-by
wic: make sure that partition size is always an integer in internal
processing
wic: use partition size when creating empty partition files
wic: check that filesystem is specified for a rootfs partition
wic: fix function comment typos
wic: add --fixed-size wks option
Changes since v2:
* COMPATIBLE_HOST workarounds now selectively skip certain wic tests for archs that cannot build images included in the test (most commonly directdisk-* image is not usable on non x86 archs), wic tests were verified to pass for qemux86-64 and beaglebone
* oe-selftest enforces en_US.UTF-8 encoding to enforce stable textual output of locaization aware programs
[1]. http://lists.openembedded.org/pipermail/openembedded-core/2016-November/128630.html
Maciej Borzecki (11):
wic: make sure that partition size is always an integer in internal
processing
wic: use partition size when creating empty partition files
wic: check that filesystem is specified for a rootfs partition
wic: fix function comment typos
wic: add --fixed-size wks option
wic: selftest: avoid COMPATIBLE_HOST issues
wic: selftest: do not repeat core-image-minimal
wic: selftest: do not assume bzImage kernel image
wic: selftest: add tests for --fixed-size partition flags
oe-selftest: fix handling of test cases without ID in --list-tests-by
oe-selftest: enforce en_US.UTF-8 locale
meta/lib/oeqa/selftest/wic.py | 230 +++++++++++++++++++++----- scripts/lib/wic/help.py | 14 +- scripts/lib/wic/imager/direct.py | 2 +- scripts/lib/wic/ksparser.py | 41 ++++- scripts/lib/wic/partition.py | 104 ++++++++---- scripts/lib/wic/plugins/source/bootimg-efi.py | 2 +- scripts/lib/wic/plugins/source/rawcopy.py | 4 +- scripts/lib/wic/utils/partitionedfs.py | 6 +- scripts/oe-selftest | 16 +- 9 files changed, 322 insertions(+), 97 deletions(-)
Signature · 1 line
2.5.0
PATCH v3 1/11 wic: make sure that partition size is always an integer in internal processing
The size field of Partition class is expected to be an integer and ought to be set inside prepare_*() method. Make sure that this is always the case.
--- scripts/lib/wic/partition.py | 12 +++++++++--- scripts/lib/wic/plugins/source/bootimg-efi.py | 2 +- scripts/lib/wic/plugins/source/rawcopy.py | 4 ++-- 3 files changed, 12 insertions(+), 6 deletions(-)
scripts/lib/wic/partition.py +9 -3
@@ -146,6 +146,12 @@ class Partition(): oe_builddir, bootimg_dir, kernel_dir, rootfs_dir, native_sysroot) + # further processing required Partition.size to be an integer, make + # sure that it is one + if type(self.size) is not int: + msger.error("Partition %s internal size is not an integer. " \ + "This a bug in source plugin %s and needs to be fixed." \ + % (self.mountpoint, self.source)) def prepare_rootfs_from_fs_image(self, cr_workdir, oe_builddir, rootfs_dir):
@@ -157,7 +163,7 @@ class Partition(): out = exec_cmd(du_cmd) rootfs_size = out.split()[0] - self.size = rootfs_size + self.size = int(rootfs_size) self.source_file = rootfs def prepare_rootfs(self, cr_workdir, oe_builddir, rootfs_dir,
@@ -194,7 +200,7 @@ class Partition(): # get the rootfs size in the right units for kickstart (kB) du_cmd = "du -Lbks %s" % rootfs out = exec_cmd(du_cmd) - self.size = out.split()[0] + self.size = int(out.split()[0]) break
@@ -379,7 +385,7 @@ class Partition(): out = exec_cmd(du_cmd) fs_size = out.split()[0] - self.size = fs_size + self.size = int(fs_size) def prepare_swap_partition(self, cr_workdir, oe_builddir, native_sysroot): """
scripts/lib/wic/plugins/source/bootimg-efi.py +1 -1
@@ -234,5 +234,5 @@ class BootimgEFIPlugin(SourcePlugin): out = exec_cmd(du_cmd) bootimg_size = out.split()[0] - part.size = bootimg_size + part.size = int(bootimg_size) part.source_file = bootimg
scripts/lib/wic/plugins/source/rawcopy.py +2 -2
@@ -78,9 +78,9 @@ class RawCopyPlugin(SourcePlugin): # get the size in the right units for kickstart (kB) du_cmd = "du -Lbks %s" % dst out = exec_cmd(du_cmd) - filesize = out.split()[0] + filesize = int(out.split()[0]) - if int(filesize) > int(part.size): + if filesize > part.size: part.size = filesize part.source_file = dst
Signature · 1 line
2.5.0
PATCH v3 2/11 wic: use partition size when creating empty partition files
In reply to Maciej BorzeckiIt seems that prepare_empty_partition_ext() and prepare_empty_partition_btrfs() got broken in commit c8669749e37fe865c197c98d5671d9de176ff4dd, thus one could observe the following backtrace:
Backtrace:
File "<snip>/poky/scripts/lib/wic/plugins/imager/direct_plugin.py", line 93, in do_create
creator.create()
File "<snip>/poky/scripts/lib/wic/imager/baseimager.py", line 159, in create
self._create()
File "<snip>/poky/scripts/lib/wic/imager/direct.py", line 290, in _create
self.bootimg_dir, self.kernel_dir, self.native_sysroot)
File "<snip>/poky/scripts/lib/wic/partition.py", line 146, in prepare
method(rootfs, oe_builddir, native_sysroot)
File "<snip>/poky/scripts/lib/wic/partition.py", line 325, in prepare_empty_partition_ext
os.ftruncate(sparse.fileno(), rootfs_size * 1024)NameError: name 'rootfs_size' is not defined
--- scripts/lib/wic/partition.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/lib/wic/partition.py +2 -2
@@ -314,7 +314,7 @@ class Partition(): Prepare an empty ext2/3/4 partition. """ with open(rootfs, 'w') as sparse: - os.ftruncate(sparse.fileno(), rootfs_size * 1024) + os.ftruncate(sparse.fileno(), self.size * 1024) extra_imagecmd = "-i 8192"
@@ -332,7 +332,7 @@ class Partition(): Prepare an empty btrfs partition. """ with open(rootfs, 'w') as sparse: - os.ftruncate(sparse.fileno(), rootfs_size * 1024) + os.ftruncate(sparse.fileno(), self.size * 1024) label_str = "" if self.label:
Signature · 1 line
2.5.0
PATCH v3 3/11 wic: check that filesystem is specified for a rootfs partition
In reply to Maciej BorzeckiWe explicitly check for --fstype if no source was provided for a partition. However, this was not the case for rootfs partitions. Make sure to raise an error if filesystem was left unspecified when preparing a rootfs partition image.
--- scripts/lib/wic/partition.py | 4 ++++ 1 file changed, 4 insertions(+)
scripts/lib/wic/partition.py +4 -0
@@ -190,6 +190,10 @@ class Partition(): if os.path.isfile(rootfs): os.remove(rootfs) + if not self.fstype: + msger.error("File system for partition %s not specified in kickstart, " \ + "use --fstype option" % (self.mountpoint)) + for prefix in ("ext", "btrfs", "vfat", "squashfs"): if self.fstype.startswith(prefix): method = getattr(self, "prepare_rootfs_" + prefix)
Signature · 1 line
2.5.0
PATCH v3 4/11 wic: fix function comment typos
In reply to Maciej BorzeckiFix typos in documentation of Image.add_partition() and Image.__format_disks().
--- scripts/lib/wic/utils/partitionedfs.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
scripts/lib/wic/utils/partitionedfs.py +2 -2
@@ -92,7 +92,7 @@ class Image(): def add_partition(self, size, disk_name, mountpoint, source_file=None, fstype=None, label=None, fsopts=None, boot=False, align=None, no_table=False, part_type=None, uuid=None, system_id=None): - """ Add the next partition. Prtitions have to be added in the + """ Add the next partition. Partitions have to be added in the first-to-last order. """ ks_pnum = len(self.partitions)
@@ -292,7 +292,7 @@ class Image(): # even number of sectors. if part['mountpoint'] == "/boot" and part['fstype'] in ["vfat", "msdos"] \ and part['size'] % 2: - msger.debug("Substracting one sector from '%s' partition to " \ + msger.debug("Subtracting one sector from '%s' partition to " \ "get even number of sectors for the partition" % \ part['mountpoint']) part['size'] -= 1
Signature · 1 line
2.5.0
PATCH v3 5/11 wic: add --fixed-size wks option
In reply to Maciej BorzeckiAdded new option --fixed-size to wks. The option can be used to indicate the exact size of a partition. The option cannot be added together with --size, in which case an error will be raised. Other options that influence automatic partition size (--extra-space, --overhead-factor), if specifiec along with --fixed-size, will raise an error.
If it partition data is larger than the amount of space specified with --fixed-size option wic will raise an error.
--- scripts/lib/wic/help.py | 14 ++++-- scripts/lib/wic/imager/direct.py | 2 +- scripts/lib/wic/ksparser.py | 41 ++++++++++++++-- scripts/lib/wic/partition.py | 88 +++++++++++++++++++++------------- scripts/lib/wic/utils/partitionedfs.py | 2 +- 5 files changed, 105 insertions(+), 42 deletions(-)
scripts/lib/wic/help.py +11 -3
@@ -646,6 +646,12 @@ DESCRIPTION not specified, the size is in MB. You do not need this option if you use --source. + --fixed-size: Exact partition size. Value format is the same + as for --size option. This option cannot be + specified along with --size. If partition data + is larger than --fixed-size and error will be + raised when assembling disk image. + --source: This option is a wic-specific option that names the source of the data that will populate the partition. The most common value for this option
@@ -719,13 +725,15 @@ DESCRIPTION space after the space filled by the content of the partition. The final size can go beyond the size specified by --size. - By default, 10MB. + By default, 10MB. This option cannot be used + with --fixed-size option. --overhead-factor: This option is specific to wic. The size of the partition is multiplied by this factor. It has to be greater than or - equal to 1. - The default value is 1.3. + equal to 1. The default value is 1.3. + This option cannot be used with --fixed-size + option. --part-type: This option is specific to wic. It specifies partition type GUID for GPT partitions.
scripts/lib/wic/imager/direct.py +1 -1
@@ -290,7 +290,7 @@ class DirectImageCreator(BaseImageCreator): self.bootimg_dir, self.kernel_dir, self.native_sysroot) - self.__image.add_partition(int(part.size), + self.__image.add_partition(part.disk_size, part.disk, part.mountpoint, part.source_file,
scripts/lib/wic/ksparser.py +37 -4
70 lines not on this page · raw message
scripts/lib/wic/partition.py +55 -33
161 lines not on this page · raw message
scripts/lib/wic/utils/partitionedfs.py +1 -1
@@ -209,7 +209,7 @@ class Image(): msger.debug("Assigned %s to %s%d, sectors range %d-%d size %d " "sectors (%d bytes)." \ % (part['mountpoint'], part['disk_name'], part['num'], - part['start'], part['start'] + part['size'] - 1, + part['start'], disk['offset'] - 1, part['size'], part['size'] * self.sector_size)) # Once all the partitions have been layed out, we can calculate the
Signature · 1 line
2.5.0
PATCH v3 6/11 wic: selftest: avoid COMPATIBLE_HOST issues
In reply to Maciej Borzeckiwic tests will unconditionally attempt to build syslinux and add configuration options that may not be compatible with current machine.
Resolve this by consulting HOST_ARCH (which defaults to TARGET_ARCH) and build recipes, add configuration options or skip tests conditionally.
A convenience decorator onlyForArch() can be used to skip test cases for specific architectures.
--- meta/lib/oeqa/selftest/wic.py | 51 +++++++++++++++++++++++++++++++++++++++---- 1 file changed, 47 insertions(+), 4 deletions(-)
meta/lib/oeqa/selftest/wic.py +47 -4
@@ -24,15 +24,33 @@ """Test cases for wic.""" import os +import unittest from glob import glob from shutil import rmtree +from functools import wraps from oeqa.selftest.base import oeSelfTest from oeqa.utils.commands import runCmd, bitbake, get_bb_var, runqemu from oeqa.utils.decorators import testcase +class onlyForArch(object): + + def __init__(self, *args): + self.archs = args + + def __call__(self,f): + @wraps(f) + def wrapped_f(*args, **kwargs): + arch = get_bb_var('HOST_ARCH', 'core-image-minimal') + if self.archs and arch not in self.archs : + raise unittest.SkipTest("Testcase arch dependency not met: %s" % arch) + return f(*args, **kwargs) + wrapped_f.__name__ = f.__name__ + return wrapped_f + + class Wic(oeSelfTest): """Wic test class."""
@@ -41,15 +59,22 @@ class Wic(oeSelfTest): def setUpLocal(self): """This code is executed before each test method.""" - self.write_config('IMAGE_FSTYPES += " hddimg"\n' - 'MACHINE_FEATURES_append = " efi"\n') + arch = get_bb_var('HOST_ARCH', 'core-image-minimal') + is_x86 = arch in ['i586', 'i686', 'x86_64'] + if is_x86: + self.write_config('IMAGE_FSTYPES += " hddimg"\n' \ + 'MACHINE_FEATURES_append = " efi"\n') # Do this here instead of in setUpClass as the base setUp does some # clean up which can result in the native tools built earlier in # setUpClass being unavailable. if not Wic.image_is_ready: - bitbake('syslinux syslinux-native parted-native gptfdisk-native ' - 'dosfstools-native mtools-native bmap-tools-native') + tools = 'parted-native gptfdisk-native ' \ + 'dosfstools-native mtools-native bmap-tools-native' + if is_x86: + tools += ' syslinux syslinux-native' + bitbake(tools) + bitbake('core-image-minimal') Wic.image_is_ready = True
@@ -71,6 +96,7 @@ class Wic(oeSelfTest): self.assertEqual(0, runCmd('wic list --help').status) @testcase(1211) + @onlyForArch('i586', 'i686', 'x86_64') def test_build_image_name(self): """Test wic create directdisk --image-name core-image-minimal""" self.assertEqual(0, runCmd("wic create directdisk "
@@ -78,6 +104,7 @@ class Wic(oeSelfTest): self.assertEqual(1, len(glob(self.resultdir + "directdisk-*.direct"))) @testcase(1212) + @onlyForArch('i586', 'i686', 'x86_64') def test_build_artifacts(self): """Test wic create directdisk providing all artifacts.""" bbvars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \
@@ -92,6 +119,7 @@ class Wic(oeSelfTest): self.assertEqual(1, len(glob(self.resultdir + "directdisk-*.direct"))) @testcase(1157) + @onlyForArch('i586', 'i686', 'x86_64') def test_gpt_image(self): """Test creation of core-image-minimal with gpt table and UUID boot""" self.assertEqual(0, runCmd("wic create directdisk-gpt "
@@ -125,6 +153,7 @@ class Wic(oeSelfTest): self.assertEqual(0, runCmd('wic help kickstart').status) @testcase(1264) + @onlyForArch('i586', 'i686', 'x86_64') def test_compress_gzip(self): """Test compressing an image with gzip""" self.assertEqual(0, runCmd("wic create directdisk "
@@ -134,6 +163,7 @@ class Wic(oeSelfTest): "directdisk-*.direct.gz"))) @testcase(1265) + @onlyForArch('i586', 'i686', 'x86_64') def test_compress_bzip2(self): """Test compressing an image with bzip2""" self.assertEqual(0, runCmd("wic create directdisk "
@@ -143,6 +173,7 @@ class Wic(oeSelfTest): "directdisk-*.direct.bz2"))) @testcase(1266) + @onlyForArch('i586', 'i686', 'x86_64') def test_compress_xz(self): """Test compressing an image with xz""" self.assertEqual(0, runCmd("wic create directdisk "
@@ -152,6 +183,7 @@ class Wic(oeSelfTest): "directdisk-*.direct.xz"))) @testcase(1267) + @onlyForArch('i586', 'i686', 'x86_64') def test_wrong_compressor(self): """Test how wic breaks if wrong compressor is provided""" self.assertEqual(2, runCmd("wic create directdisk "
@@ -159,6 +191,7 @@ class Wic(oeSelfTest): "-c wrong", ignore_status=True).status) @testcase(1268) + @onlyForArch('i586', 'i686', 'x86_64') def test_rootfs_indirect_recipes(self): """Test usage of rootfs plugin with rootfs recipes""" wks = "directdisk-multi-rootfs"
@@ -170,6 +203,7 @@ class Wic(oeSelfTest): self.assertEqual(1, len(glob(self.resultdir + "%s*.direct" % wks))) @testcase(1269) + @onlyForArch('i586', 'i686', 'x86_64') def test_rootfs_artifacts(self): """Test usage of rootfs plugin with rootfs paths""" bbvars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \
@@ -188,6 +222,7 @@ class Wic(oeSelfTest): "%(wks)s-*.direct" % bbvars))) @testcase(1346) + @onlyForArch('i586', 'i686', 'x86_64') def test_iso_image(self): """Test creation of hybrid iso image with legacy and EFI boot""" self.assertEqual(0, runCmd("wic create mkhybridiso "
@@ -220,6 +255,7 @@ class Wic(oeSelfTest): self.assertTrue(content[var]) @testcase(1351) + @onlyForArch('i586', 'i686', 'x86_64') def test_wic_image_type(self): """Test building wic images by bitbake""" self.assertEqual(0, bitbake('wic-image-minimal').status)
@@ -235,6 +271,7 @@ class Wic(oeSelfTest): self.assertTrue(os.path.isfile(os.path.realpath(path))) @testcase(1348) + @onlyForArch('i586', 'i686', 'x86_64') def test_qemux86_directdisk(self): """Test creation of qemux-86-directdisk image""" image = "qemux86-directdisk"
@@ -243,6 +280,7 @@ class Wic(oeSelfTest): self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image))) @testcase(1349) + @onlyForArch('i586', 'i686', 'x86_64') def test_mkgummidisk(self): """Test creation of mkgummidisk image""" image = "mkgummidisk"
@@ -251,6 +289,7 @@ class Wic(oeSelfTest): self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image))) @testcase(1350) + @onlyForArch('i586', 'i686', 'x86_64') def test_mkefidisk(self): """Test creation of mkefidisk image""" image = "mkefidisk"
@@ -259,6 +298,7 @@ class Wic(oeSelfTest): self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image))) @testcase(1385) + @onlyForArch('i586', 'i686', 'x86_64') def test_directdisk_bootloader_config(self): """Test creation of directdisk-bootloader-config image""" image = "directdisk-bootloader-config"
@@ -267,6 +307,7 @@ class Wic(oeSelfTest): self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image))) @testcase(1422) + @onlyForArch('i586', 'i686', 'x86_64') def test_qemu(self): """Test wic-image-minimal under qemu""" self.assertEqual(0, bitbake('wic-image-minimal').status)
@@ -277,6 +318,7 @@ class Wic(oeSelfTest): self.assertEqual(1, status, 'Failed to run command "%s": %s' % (command, output)) self.assertEqual(output, '/dev/root /\r\n/dev/vda3 /mnt') + @onlyForArch('i586', 'i686', 'x86_64') def test_bmap(self): """Test generation of .bmap file""" image = "directdisk"
@@ -285,6 +327,7 @@ class Wic(oeSelfTest): self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image))) self.assertEqual(1, len(glob(self.resultdir + "%s-*direct.bmap" % image))) + @onlyForArch('i586', 'i686', 'x86_64') def test_systemd_bootdisk(self): """Test creation of systemd-bootdisk image""" image = "systemd-bootdisk"
Signature · 1 line
2.5.0
PATCH v3 7/11 wic: selftest: do not repeat core-image-minimal
In reply to Maciej BorzeckiReplace repeated core-image-minimal with Wic class field.
--- meta/lib/oeqa/selftest/wic.py | 111 +++++++++++++++++++++++++++--------------- 1 file changed, 73 insertions(+), 38 deletions(-)
meta/lib/oeqa/selftest/wic.py +73 -38
@@ -57,6 +57,8 @@ class Wic(oeSelfTest): resultdir = "/var/tmp/wic/build/" image_is_ready = False + OE_IMAGE = "core-image-minimal" + def setUpLocal(self): """This code is executed before each test method.""" arch = get_bb_var('HOST_ARCH', 'core-image-minimal')
@@ -75,7 +77,7 @@ class Wic(oeSelfTest): tools += ' syslinux syslinux-native' bitbake(tools) - bitbake('core-image-minimal') + bitbake(self.OE_IMAGE) Wic.image_is_ready = True rmtree(self.resultdir, ignore_errors=True)
@@ -100,14 +102,14 @@ class Wic(oeSelfTest): def test_build_image_name(self): """Test wic create directdisk --image-name core-image-minimal""" self.assertEqual(0, runCmd("wic create directdisk " - "--image-name core-image-minimal").status) + "--image-name %s" % self.OE_IMAGE).status) self.assertEqual(1, len(glob(self.resultdir + "directdisk-*.direct"))) @testcase(1212) @onlyForArch('i586', 'i686', 'x86_64') def test_build_artifacts(self): """Test wic create directdisk providing all artifacts.""" - bbvars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \ + bbvars = dict((var.lower(), get_bb_var(var, self.OE_IMAGE)) \ for var in ('STAGING_DATADIR', 'DEPLOY_DIR_IMAGE', 'STAGING_DIR_NATIVE', 'IMAGE_ROOTFS')) status = runCmd("wic create directdisk "
@@ -123,7 +125,7 @@ class Wic(oeSelfTest): def test_gpt_image(self): """Test creation of core-image-minimal with gpt table and UUID boot""" self.assertEqual(0, runCmd("wic create directdisk-gpt " - "--image-name core-image-minimal").status) + "--image-name %s" % self.OE_IMAGE).status) self.assertEqual(1, len(glob(self.resultdir + "directdisk-*.direct"))) @testcase(1213)
@@ -157,8 +159,8 @@ class Wic(oeSelfTest): def test_compress_gzip(self): """Test compressing an image with gzip""" self.assertEqual(0, runCmd("wic create directdisk " - "--image-name core-image-minimal " - "-c gzip").status) + "--image-name %s " + "-c gzip" % self.OE_IMAGE).status) self.assertEqual(1, len(glob(self.resultdir + \ "directdisk-*.direct.gz")))
@@ -167,8 +169,8 @@ class Wic(oeSelfTest): def test_compress_bzip2(self): """Test compressing an image with bzip2""" self.assertEqual(0, runCmd("wic create directdisk " - "--image-name core-image-minimal " - "-c bzip2").status) + "--image-name %s " + "-c bzip2" % self.OE_IMAGE).status) self.assertEqual(1, len(glob(self.resultdir + \ "directdisk-*.direct.bz2")))
@@ -177,8 +179,8 @@ class Wic(oeSelfTest): def test_compress_xz(self): """Test compressing an image with xz""" self.assertEqual(0, runCmd("wic create directdisk " - "--image-name core-image-minimal " - "-c xz").status) + "--image-name %s " + "-c xz" % self.OE_IMAGE).status) self.assertEqual(1, len(glob(self.resultdir + \ "directdisk-*.direct.xz")))
@@ -187,26 +189,31 @@ class Wic(oeSelfTest): def test_wrong_compressor(self): """Test how wic breaks if wrong compressor is provided""" self.assertEqual(2, runCmd("wic create directdisk " - "--image-name core-image-minimal " - "-c wrong", ignore_status=True).status) + "--image-name %s " + "-c wrong" % self.OE_IMAGE, + ignore_status=True).status) @testcase(1268) @onlyForArch('i586', 'i686', 'x86_64') def test_rootfs_indirect_recipes(self): """Test usage of rootfs plugin with rootfs recipes""" wks = "directdisk-multi-rootfs" - self.assertEqual(0, runCmd("wic create %s " - "--image-name core-image-minimal " - "--rootfs rootfs1=core-image-minimal " - "--rootfs rootfs2=core-image-minimal" \ - % wks).status) + wic_cmd_vars = { + 'wks': wks, + 'image': self.OE_IMAGE, + } + self.assertEqual(0, runCmd("wic create %(wks)s " + "--image-name %(image)s " + "--rootfs rootfs1=%(image)s " + "--rootfs rootfs2=%(image)s" \ + % wic_cmd_vars).status) self.assertEqual(1, len(glob(self.resultdir + "%s*.direct" % wks))) @testcase(1269) @onlyForArch('i586', 'i686', 'x86_64') def test_rootfs_artifacts(self): """Test usage of rootfs plugin with rootfs paths""" - bbvars = dict((var.lower(), get_bb_var(var, 'core-image-minimal')) \ + bbvars = dict((var.lower(), get_bb_var(var, self.OE_IMAGE)) \ for var in ('STAGING_DATADIR', 'DEPLOY_DIR_IMAGE', 'STAGING_DIR_NATIVE', 'IMAGE_ROOTFS')) bbvars['wks'] = "directdisk-multi-rootfs"
@@ -226,24 +233,23 @@ class Wic(oeSelfTest): def test_iso_image(self): """Test creation of hybrid iso image with legacy and EFI boot""" self.assertEqual(0, runCmd("wic create mkhybridiso " - "--image-name core-image-minimal").status) + "--image-name %s" % self.OE_IMAGE).status) self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-*.direct"))) self.assertEqual(1, len(glob(self.resultdir + "HYBRID_ISO_IMG-*.iso"))) @testcase(1347) def test_image_env(self): """Test generation of <image>.env files.""" - image = 'core-image-minimal' - self.assertEqual(0, bitbake('%s -c do_rootfs_wicenv' % image).status) - stdir = get_bb_var('STAGING_DIR_TARGET', image) + self.assertEqual(0, bitbake('%s -c do_rootfs_wicenv' % self.OE_IMAGE).status) + stdir = get_bb_var('STAGING_DIR_TARGET', self.OE_IMAGE) imgdatadir = os.path.join(stdir, 'imgdata') - basename = get_bb_var('IMAGE_BASENAME', image) - self.assertEqual(basename, image) + basename = get_bb_var('IMAGE_BASENAME', self.OE_IMAGE) + self.assertEqual(basename, self.OE_IMAGE) path = os.path.join(imgdatadir, basename) + '.env' self.assertTrue(os.path.isfile(path)) - wicvars = set(get_bb_var('WICVARS', image).split()) + wicvars = set(get_bb_var('WICVARS', self.OE_IMAGE).split()) # filter out optional variables wicvars = wicvars.difference(('HDDDIR', 'IMAGE_BOOT_FILES', 'INITRD', 'ISODIR'))
@@ -275,8 +281,12 @@ class Wic(oeSelfTest): def test_qemux86_directdisk(self): """Test creation of qemux-86-directdisk image""" image = "qemux86-directdisk" - self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \ - % image).status) + wic_cmd_vars = { + 'wks': image, + 'image': self.OE_IMAGE, + } + self.assertEqual(0, runCmd("wic create %(wks)s -e %(image)s" \ + % wic_cmd_vars).status) self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image))) @testcase(1349)
@@ -284,8 +294,12 @@ class Wic(oeSelfTest): def test_mkgummidisk(self): """Test creation of mkgummidisk image""" image = "mkgummidisk" - self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \ - % image).status) + wic_cmd_vars = { + 'wks': image, + 'image': self.OE_IMAGE, + } + self.assertEqual(0, runCmd("wic create %(wks)s -e %(image)s" \ + % wic_cmd_vars).status) self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image))) @testcase(1350)
@@ -293,8 +307,12 @@ class Wic(oeSelfTest): def test_mkefidisk(self): """Test creation of mkefidisk image""" image = "mkefidisk" - self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \ - % image).status) + wic_cmd_vars = { + 'wks': image, + 'image': self.OE_IMAGE, + } + self.assertEqual(0, runCmd("wic create %(wks)s -e %(image)s" \ + % wic_cmd_vars).status) self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image))) @testcase(1385)
@@ -302,8 +320,12 @@ class Wic(oeSelfTest): def test_directdisk_bootloader_config(self): """Test creation of directdisk-bootloader-config image""" image = "directdisk-bootloader-config" - self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \ - % image).status) + wic_cmd_vars = { + 'wks': image, + 'image': self.OE_IMAGE, + } + self.assertEqual(0, runCmd("wic create %(wks)s -e %(image)s" \ + % wic_cmd_vars).status) self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image))) @testcase(1422)
@@ -322,7 +344,12 @@ class Wic(oeSelfTest): def test_bmap(self): """Test generation of .bmap file""" image = "directdisk" - status = runCmd("wic create %s -e core-image-minimal --bmap" % image).status + wic_cmd_vars = { + 'wks': image, + 'image': self.OE_IMAGE, + } + status = runCmd("wic create %(wks)s -e %(image)s --bmap" \ + % wic_cmd_vars).status self.assertEqual(0, status) self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image))) self.assertEqual(1, len(glob(self.resultdir + "%s-*direct.bmap" % image)))
@@ -331,14 +358,22 @@ class Wic(oeSelfTest): def test_systemd_bootdisk(self): """Test creation of systemd-bootdisk image""" image = "systemd-bootdisk" - self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \ - % image).status) + wic_cmd_vars = { + 'wks': image, + 'image': self.OE_IMAGE, + } + self.assertEqual(0, runCmd("wic create %(wks)s -e %(image)s" \ + % wic_cmd_vars).status) self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image))) def test_sdimage_bootpart(self): """Test creation of sdimage-bootpart image""" image = "sdimage-bootpart" self.write_config('IMAGE_BOOT_FILES = "bzImage"\n') - self.assertEqual(0, runCmd("wic create %s -e core-image-minimal" \ - % image).status) + wic_cmd_vars = { + 'wks': image, + 'image': self.OE_IMAGE, + } + self.assertEqual(0, runCmd("wic create %(wks)s -e %(image)s" \ + % wic_cmd_vars).status) self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image)))
Signature · 1 line
2.5.0
PATCH v3 8/11 wic: selftest: do not assume bzImage kernel image
In reply to Maciej BorzeckiInstead of assuming that bzImage is available, query bitbake enviroment for KERNEL_IMAGETYPE.
--- meta/lib/oeqa/selftest/wic.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
meta/lib/oeqa/selftest/wic.py +2 -1
@@ -369,7 +369,8 @@ class Wic(oeSelfTest): def test_sdimage_bootpart(self): """Test creation of sdimage-bootpart image""" image = "sdimage-bootpart" - self.write_config('IMAGE_BOOT_FILES = "bzImage"\n') + kimgtype = get_bb_var('KERNEL_IMAGETYPE', self.OE_IMAGE) + self.write_config('IMAGE_BOOT_FILES = "%s"\n' % kimgtype) wic_cmd_vars = { 'wks': image, 'image': self.OE_IMAGE,
Signature · 1 line
2.5.0
PATCH v3 9/11 wic: selftest: add tests for --fixed-size partition flags
In reply to Maciej Borzeckiwic has a new flag for setting a fixed parition size --fixed-size. Add tests that verify if partition is indeed sized properly and that errors are signaled when there is not enough space to fit partition data.
--- meta/lib/oeqa/selftest/wic.py | 65 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 65 insertions(+)
meta/lib/oeqa/selftest/wic.py +65 -0
@@ -29,6 +29,7 @@ import unittest from glob import glob from shutil import rmtree from functools import wraps +from tempfile import NamedTemporaryFile from oeqa.selftest.base import oeSelfTest from oeqa.utils.commands import runCmd, bitbake, get_bb_var, runqemu
@@ -378,3 +379,67 @@ class Wic(oeSelfTest): self.assertEqual(0, runCmd("wic create %(wks)s -e %(image)s" \ % wic_cmd_vars).status) self.assertEqual(1, len(glob(self.resultdir + "%s-*direct" % image))) + + def _make_fixed_size_wks(self, size): + """ + Create a wks of an image with a single partition. Size of the partition is set + using --fixed-size flag. Returns a tuple: (path to wks file, wks image name) + """ + with NamedTemporaryFile("w", suffix=".wks", delete=False) as tf: + wkspath = tf.name + tf.write("part " \ + "--source rootfs --ondisk hda --align 4 --fixed-size %d " + "--fstype=ext4\n" % size) + wksname = os.path.splitext(os.path.basename(wkspath))[0] + + return (wkspath, wksname) + + def test_fixed_size(self): + """ + Test creation of a simple image with partition size controlled through + --fixed-size flag + """ + wkspath, wksname = self._make_fixed_size_wks(200) + + wic_cmd_vars = { + 'wks': wkspath, + 'image': self.OE_IMAGE, + } + self.assertEqual(0, runCmd("wic create %(wks)s -e %(image)s" \ + % wic_cmd_vars).status) + os.remove(wkspath) + wicout = glob(self.resultdir + "%s-*direct" % wksname) + self.assertEqual(1, len(wicout)) + + wicimg = wicout[0] + + # verify partition size with wic + res = runCmd("parted -m %s unit mib p 2>/dev/null" % wicimg, ignore_status=True) + self.assertEqual(0, res.status) + + # parse parted output which looks like this: + # BYT;\n + # /var/tmp/wic/build/tmpfwvjjkf_-201611101222-hda.direct:200MiB:file:512:512:msdos::;\n + # 1:0.00MiB:200MiB:200MiB:ext4::;\n + partlns = res.output.splitlines()[2:] + + self.assertEqual(1, len(partlns)) + self.assertEqual("1:0.00MiB:200MiB:200MiB:ext4::;", partlns[0]) + + def test_fixed_size_error(self): + """ + Test creation of a simple image with partition size controlled through + --fixed-size flag. The size of partition is intentionally set to 1MiB + in order to trigger an error in wic. + """ + wkspath, wksname = self._make_fixed_size_wks(1) + + wic_cmd_vars = { + 'wks': wkspath, + 'image': self.OE_IMAGE, + } + self.assertEqual(1, runCmd("wic create %(wks)s -e %(image)s" \ + % wic_cmd_vars, ignore_status=True).status) + os.remove(wkspath) + wicout = glob(self.resultdir + "%s-*direct" % wksname) + self.assertEqual(0, len(wicout))
Signature · 1 line
2.5.0
Re: [PATCH v3 09/11] wic: selftest: add tests for --fixed-size partition flags
Quoted 3 lines · Maciej Borzecki
wic has a new flag for setting a fixed parition size --fixed-size. Add tests that verify if partition is indeed sized properly and that errors are signaled when there is not enough space to fit partition data.
Just failed on the AB:
FAIL: test_fixed_size (oeqa.selftest.wic.Wic)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/wic.py", line 418, in test_fixed_size
self.assertEqual(0, res.status)
AssertionError: 0 != 127
Might be useful to print res.output in the failure. :)
Ross
Re: [PATCH v3 09/11] wic: selftest: add tests for --fixed-size partition flags
Quoted 20 lines · Ross
Quoted 4 lines · Maciej Borzecki
wic has a new flag for setting a fixed parition size --fixed-size. Add tests that verify if partition is indeed sized properly and that errors are signaled when there is not enough space to fit partition data.
Just failed on the AB:
FAIL:
test_fixed_size(oeqa.selftest.wic.Wic)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/wic.py", line 418, intest_fixed_size
self.assertEqual(0, res.status)
AssertionError: 0 != 127
Quoted 2 lines
Might be useful to print res.output in the failure. :)
A bit weird. The test failed on this:
# verify partition size with wic
res = runCmd("parted -m %s unit mib p 2>/dev/null" % wicimg,
ignore_status=True)
self.assertEqual(0, res.status)So the image is there. It was listed by glob(), which is a directly before runing parted. Also wic has built it (because the test would have failed even earlier). However, when parted gets there, it either fails to understand unit mib p (unlikely), or the disk image is just not there anymore.
Is there a chance these tests are run in parallel?
Signature · 2 lines
Maciej Borzecki RnDity
Re: [PATCH v3 09/11] wic: selftest: add tests for --fixed-size partition flags
Quoted 31 lines · Maciej Borzęcki
Quoted 20 lines · Ross
Quoted 4 lines · Maciej Borzecki
wic has a new flag for setting a fixed parition size --fixed-size. Add tests that verify if partition is indeed sized properly and that errors are signaled when there is not enough space to fit partition data.
Just failed on the AB:
FAIL:
test_fixed_size(oeqa.selftest.wic.Wic)
----------------------------------------------------------------------
Traceback (most recent call last):
File
"/home/pokybuild/yocto-autobuilder/yocto-worker/nightly-oe-selftest/build/meta/lib/oeqa/selftest/wic.py", line 418, intest_fixed_size
self.assertEqual(0, res.status)
AssertionError: 0 != 127Quoted 2 lines
Might be useful to print res.output in the failure. :)
A bit weird. The test failed on this:
# verify partition size with wic res = runCmd("parted -m %s unit mib p 2>/dev/null" % wicimg, ignore_status=True) self.assertEqual(0, res.status)
Quoted 8 lines
So the image is there. It was listed by
glob(), which is a directly before runing parted. Also wic has built it (because the test would have failed even earlier). However, when parted gets there, it either fails to understandunit mib p(unlikely), or the disk image is just not there anymore.Is there a chance these tests are run in parallel?
Nevermind. I have incorrectly assumed that runCmd() sets up PATH in a way that enables running *-native tools. However, the tests ended up host parted binary rather than the one built by parted-native. Worked fine on my Fedora based build systems where parted is installed by default.
I will post a v4 of the series (just the patches that are not in master yet).
Cheers,
Signature · 2 lines
Maciej Borzecki RnDity
PATCH v3 10/11 oe-selftest: fix handling of test cases without ID in --list-tests-by
In reply to Maciej BorzeckiRunning oe-selftest --list-tests-by module wic will produce the following backtrace:
Traceback (most recent call last):
File "<snip>/poky/scripts/oe-selftest", line 668, in <module>
ret = main()
File "<snip>/poky/scripts/oe-selftest", line 486, in main
list_testsuite_by(criteria, keyword)
File "<snip>/poky/scripts/oe-selftest", line 340, in list_testsuite_by
ts = sorted([ (tc.tcid, tc.tctag, tc.tcname, tc.tcclass, tc.tcmodule) for tc in get_testsuite_by(criteria, keyword) ])
TypeError: unorderable types: int() < NoneType()
The root cause is that a test case does not necessarily have an ID assigned, hence its value is None. Since Python 3 does not allow comparison of heterogeneous types, TypeError is raised.
--- scripts/oe-selftest | 13 +++++++++---- 1 file changed, 9 insertions(+), 4 deletions(-)
scripts/oe-selftest +9 -4
@@ -336,10 +336,15 @@ def list_testsuite_by(criteria, keyword): # Get a testsuite based on 'keyword' # criteria: name, class, module, id, tag # keyword: a list of tests, classes, modules, ids, tags - - ts = sorted([ (tc.tcid, tc.tctag, tc.tcname, tc.tcclass, tc.tcmodule) for tc in get_testsuite_by(criteria, keyword) ]) - - print('%-4s\t%-20s\t%-60s\t%-25s\t%-20s' % ('id', 'tag', 'name', 'class', 'module')) + def tc_key(t): + if t[0] is None: + return (0,) + t[1:] + return t + # tcid may be None if no ID was assigned, in which case sorted() will throw + # a TypeError as Python 3 does not allow comparison (<,<=,>=,>) of + # heterogeneous types, handle this by using a custom key generator + ts = sorted([ (tc.tcid, tc.tctag, tc.tcname, tc.tcclass, tc.tcmodule) \ + for tc in get_testsuite_by(criteria, keyword) ], key=tc_key) print('_' * 150) for t in ts: if isinstance(t[1], (tuple, list)):
Signature · 1 line
2.5.0
PATCH v3 11/11 oe-selftest: enforce en_US.UTF-8 locale
In reply to Maciej BorzeckiReplicate bitbake and eforce en_US.UTF-8 locale so that ouptut of locale-aware tools remains stable.
--- scripts/oe-selftest | 3 +++ 1 file changed, 3 insertions(+)
scripts/oe-selftest +3 -0
@@ -468,6 +468,9 @@ def main(): sys.path.extend(layer_libdirs) imp.reload(oeqa.selftest) + # act like bitbake and enforce en_US.UTF-8 locale + os.environ["LC_ALL"] = "en_US.UTF-8" + if args.run_tests_by and len(args.run_tests_by) >= 2: valid_options = ['name', 'class', 'module', 'id', 'tag'] if args.run_tests_by[0] not in valid_options:
Signature · 1 line
2.5.0
Quoted 4 lines · Maciej Borzecki
I have noticed that Ross has cherry-picked some patches into his pull request to master. Just for reference, the patches are included in this
series, but have not been changed since the previous version. The
Quoted 9 lines
patches in question are:
oe-selftest: fix handling of test cases without ID in --list-tests-by
wic: make sure that partition size is always an integer in internal
processing
wic: use partition size when creating empty partition files
wic: check that filesystem is specified for a rootfs partition
wic: fix function comment typos
wic: add --fixed-size wks option
These were simple and/or fixing serious problems, hope I didn't cause too much inconvenience doing this.
Ross
Quoted 18 lines · Ross
Quoted 13 lines · Maciej Borzecki
I have noticed that Ross has cherry-picked some patches into his pull request to master. Just for reference, the patches are included in this
series, but have not been changed since the previous version. The patches in question are:
oe-selftest: fix handling of test cases without ID in --list-tests-by
wic: make sure that partition size is always an integer in internal
processing
wic: use partition size when creating empty partition files
wic: check that filesystem is specified for a rootfs partition
wic: fix function comment typos
wic: add --fixed-size wks option
Quoted 2 lines
These were simple and/or fixing serious problems, hope I didn't cause too much inconvenience doing this.
None at all. I was actually about to propose cherry picking those patches along with oe-selftest: enforce en_US.UTF-8 locale, when your pull request to OE-core came in.
Signature · 2 lines
Maciej Borzecki RnDity