Rework WIC sector size
cover Rework WIC sector size
This work is effectively based on the 'v7' of the 'standalone wic' submission. It was requested that the submission be broken into individual parts so we can review and get merge things individually.
Changes from v7:
Introduction of additional (previous implementation) vfat 4k tests.
Update the test suite to add a new cmdline argument test in addition to (not replacing) the previous variable version of the test.
Update the test case to include fat partition tests.
Cleanup Deprecation message so it only displays once, if the older style is used. In v7 the message may have been displayed twice.
Mark Hatle (1):
oeqa/selftest: wic: Add vfat to test_wic_sector_size
Trevor Woerner (1):
wic: re-implement sector-size support
meta/lib/oeqa/selftest/cases/wic.py | 105 ++++++++++++++++++++- scripts/lib/wic/engine.py | 60 +++++++----- scripts/lib/wic/help.py | 23 +++-- scripts/lib/wic/misc.py | 6 ++ scripts/lib/wic/partition.py | 97 ++++++++++++++++++- scripts/lib/wic/plugins/imager/direct.py | 18 +--- scripts/lib/wic/plugins/source/bootimg_efi.py | 5 +- scripts/lib/wic/plugins/source/bootimg_pcbios.py | 11 ++- .../lib/wic/plugins/source/isoimage_isohybrid.py | 5 +- scripts/wic | 41 ++++++++ 10 files changed, 306 insertions(+), 65 deletions(-)
Signature · 1 line
1.8.3.1
PATCH 1/2 oeqa/selftest: wic: Add vfat to test_wic_sector_size
Add an empty vfat partition to the 4k sector size test. This ensures that the -S 4096 option is passed to mkfs.vfat, and the resulting filesystem is generated.
We also now verify that the requested partitions, and names were created as expected. Size does not matter, only the partition type and name.
Note, there is a known issue in parted that 4096 fat partitions are not recognized by fstype, so report themselves as empty in the regular output. Both wic ls and parted p show an unknown filesytem type. However, the type of the partition (last field) is set to msftdata, so we can use that instead.
--- meta/lib/oeqa/selftest/cases/wic.py | 31 ++++++++++++++++++++++++++++--- 1 file changed, 28 insertions(+), 3 deletions(-)
meta/lib/oeqa/selftest/cases/wic.py +28 -3
@@ -958,6 +958,7 @@ bootloader --ptable gpt""") with NamedTemporaryFile("w", suffix=".wks") as wks: wks.writelines( ['bootloader --ptable gpt\n', + 'part --fstype vfat --fstype vfat --label emptyfat --size 1M --mkfs-extraopts "-S 4096"\n', 'part --fstype ext4 --source rootfs --label rofs-a --mkfs-extraopts "-b 4096"\n', 'part --fstype ext4 --source rootfs --use-uuid --mkfs-extraopts "-b 4096"\n']) wks.flush()
@@ -970,17 +971,21 @@ bootloader --ptable gpt""") sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools') # list partitions result = runCmd("wic ls %s -n %s" % (images[0], sysroot)) - self.assertEqual(3, len(result.output.split('\n'))) + print(result.output) + # 4 lines of output: header + 3 partition + self.assertEqual(4, len(result.output.split('\n'))) # verify partition size with wic res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p" % (wic_sector_size, images[0]), stderr=subprocess.PIPE) + print(res.output) # parse parted output which looks like this: # BYT;\n # /var/tmp/wic/build/tmpgjzzefdd-202410281021-sda.direct:78569472B:file:4096:4096:gpt::;\n - # 1:139264B:39284735B:39145472B:ext4:rofs-a:;\n - # 2:39284736B:78430207B:39145472B:ext4:primary:;\n + # 1:139264B:1187839B:1048576B::emptyfat:msftdata; + # 2:1187840B:149270527B:148082688B:ext4:rofs-a:; + # 3:149270528B:297353215B:148082688B:ext4:primary:; disk_info = res.output.splitlines()[1] # Check sector sizes sector_size_logical = int(disk_info.split(":")[3])
@@ -988,6 +993,26 @@ bootloader --ptable gpt""") self.assertEqual(wic_sector_size, sector_size_logical, "Logical sector size is not %d." % wic_sector_size) self.assertEqual(wic_sector_size, sector_size_physical, "Physical sector size is not %d." % wic_sector_size) + # It is a known issue with parsed that a 4K FAT partition does + # not have a recognized filesystem type of *fat. + part_info = res.output.splitlines()[2] + partname = part_info.split(":")[5] + parttype = part_info.split(":")[6] + self.assertEqual('emptyfat', partname) + self.assertEqual('msftdata;', parttype) + + part_info = res.output.splitlines()[3] + parttype = part_info.split(":")[4] + partname = part_info.split(":")[5] + self.assertEqual('ext4', parttype) + self.assertEqual('rofs-a', partname) + + part_info = res.output.splitlines()[4] + parttype = part_info.split(":")[4] + partname = part_info.split(":")[5] + self.assertEqual('ext4', parttype) + self.assertEqual('primary', partname) + finally: os.environ['PATH'] = oldpath
Signature · 1 line
1.8.3.1
PATCH 2/2 wic: re-implement sector-size support
In reply to Mark HatleFrom: Trevor Woerner <[email protected]>
The previous implementation had the following limitations:
- required the variable WIC_SECTOR_SIZE either be defined in a configuration file or be defined in a --vars file
- this means that every invocation of "wic ls", "wic cp", or "wic rm" needed this variable defined (config or --vars)
- required the user to create separate *wks files for every sector size they wanted to use
- required the user to specify the mkfs-extraopts by hand to specify the correct sector size: e.g.
bootloader --ptable gpt
part --fstype vfat --label emptyfat --mkfs-extraopts "-S 4096"
part --fstype ext4 --source rootfs --label rofs-a --mkfs-extraopts "-b 4096"
part --fstype ext4 --source rootfs --use-uuid --mkfs-extraopts "-b 4096"- specifying --mkfs-extraopts replaces the defaults with the user-supplied values
- it would not be possible to generate images with different sector sizes in the same build since the configuration and *wks files would need to change and the build re-run for each size
The new implementation handles the sector-size via a CLI argument, while preserving the existing variable definition previously implement:
- the sector-size may now be provided on the cmdline to the "wic ls", "wic cp", "wic rm", and "wic create" commands: default = 512
- this means the configuration and/or --vars file does not need to be changed in order to perform those operations on images with different sector sizes
- support is provided implicitly for mkdosfs and ext[234] partitions
- the user no longer needs to know and supply the sector-size magic in --mkfs-extraopts (thereby clobbering the other defaults)
As before, if the --sector-size command-line argument is not given, allow the sector-size to be provided via the WIC_SECTOR_SIZE bitbake variable. The user is warned that this behavior is deprecated. If both are given, warn the user that the cmdline argument takes precedence.
AI-Generated: codex/gpt-5.1-codex-max
restore environ test case, as it is still supported (but obsolete)
Revised commit message above.
Rework _mkdosfs_extraopts and _mkfs_ext_extraopts to verify that the sector-size and any manual size match, otherwise error. This change was needed to keep existing workflow where a user can use either (or both) WIC_SECTOR_SIZE and/or --mkfs-extraopts to select
partition sizes. Note, before it may have been possible, but
not valid, to select a different overall sector size from the individual partition sector sizes.
--- meta/lib/oeqa/selftest/cases/wic.py | 78 ++++++++++++++++- scripts/lib/wic/engine.py | 60 +++++++------ scripts/lib/wic/help.py | 23 +++-- scripts/lib/wic/misc.py | 6 ++ scripts/lib/wic/partition.py | 97 ++++++++++++++++++++-- scripts/lib/wic/plugins/imager/direct.py | 18 ++-- scripts/lib/wic/plugins/source/bootimg_efi.py | 5 +- scripts/lib/wic/plugins/source/bootimg_pcbios.py | 11 +-- .../lib/wic/plugins/source/isoimage_isohybrid.py | 5 +- scripts/wic | 41 +++++++++ 10 files changed, 280 insertions(+), 64 deletions(-)
meta/lib/oeqa/selftest/cases/wic.py +74 -4
@@ -935,8 +935,8 @@ bootloader --ptable gpt""") finally: os.remove(wks_file) - def test_wic_sector_size(self): - """Test generation image sector size""" + def test_wic_sector_size_env(self): + """Test generation image sector size via environment (obsolete)""" oldpath = os.environ['PATH'] os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
@@ -972,8 +972,8 @@ bootloader --ptable gpt""") # list partitions result = runCmd("wic ls %s -n %s" % (images[0], sysroot)) print(result.output) - # 4 lines of output: header + 3 partition - self.assertEqual(4, len(result.output.split('\n'))) + # Deprecated message + 4 lines of output: header + 3 partitions + self.assertEqual(5, len(result.output.split('\n'))) # verify partition size with wic res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p" % (wic_sector_size, images[0]),
@@ -1016,6 +1016,76 @@ bootloader --ptable gpt""") finally: os.environ['PATH'] = oldpath + def test_wic_sector_size_cli(self): + """Test sector size handling via CLI option.""" + + oldpath = os.environ['PATH'] + os.environ['PATH'] = get_bb_var("PATH", "wic-tools") + + try: + bitbake('core-image-minimal') + + with NamedTemporaryFile("w", suffix=".wks") as wks: + wks.writelines( + ['bootloader --ptable gpt\n', + 'part --fstype vfat --fstype vfat --label emptyfat --size 1M\n', + 'part --fstype ext4 --source rootfs --label rofs-a\n', + 'part --fstype ext4 --source rootfs --use-uuid\n']) + wks.flush() + cmd = "wic create %s -e core-image-minimal -o %s --sector-size 4096" % (wks.name, self.resultdir) + runCmd(cmd) + wksname = os.path.splitext(os.path.basename(wks.name))[0] + images = glob(os.path.join(self.resultdir, "%s-*direct" % wksname)) + self.assertEqual(1, len(images)) + + sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools') + # list partitions + result = runCmd("wic ls %s -n %s --sector-size 4096" % (images[0], sysroot)) + print(result.output) + # 4 lines of output: header + 3 partitions + self.assertEqual(4, len(result.output.split('\n'))) + + # verify partition size with parted output + res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p" % (4096, images[0]), + stderr=subprocess.PIPE) + + print(res.output) + # parse parted output which looks like this: + # BYT;\n + # /var/tmp/wic/build/tmpgjzzefdd-202410281021-sda.direct:78569472B:file:4096:4096:gpt::;\n + # 1:139264B:1187839B:1048576B::emptyfat:msftdata; + # 2:1187840B:149270527B:148082688B:ext4:rofs-a:; + # 3:149270528B:297353215B:148082688B:ext4:primary:; + disk_info = res.output.splitlines()[1] + # Check sector sizes + sector_size_logical = int(disk_info.split(":")[3]) + sector_size_physical = int(disk_info.split(":")[4]) + self.assertEqual(4096, sector_size_logical, "Logical sector size is not 4096.") + self.assertEqual(4096, sector_size_physical, "Physical sector size is not 4096.") + + # It is a known issue with parsed that a 4K FAT partition does + # not have a recognized filesystem type of *fat. + part_info = res.output.splitlines()[2] + partname = part_info.split(":")[5] + parttype = part_info.split(":")[6] + self.assertEqual('emptyfat', partname) + self.assertEqual('msftdata;', parttype) + + part_info = res.output.splitlines()[3] + parttype = part_info.split(":")[4] + partname = part_info.split(":")[5] + self.assertEqual('ext4', parttype) + self.assertEqual('rofs-a', partname) + + part_info = res.output.splitlines()[4] + parttype = part_info.split(":")[4] + partname = part_info.split(":")[5] + self.assertEqual('ext4', parttype) + self.assertEqual('primary', partname) + + finally: + os.environ['PATH'] = oldpath + class Wic2(WicTestCase): def test_bmap_short(self):
scripts/lib/wic/engine.py +34 -26
118 lines not on this page · raw message
scripts/lib/wic/help.py +16 -7
@@ -118,7 +118,7 @@ wic_create_usage = """ usage: wic create <wks file or image name> [-o <DIRNAME> | --outdir <DIRNAME>] [-e | --image-name] [-s, --skip-build-check] [-D, --debug] [-r, --rootfs-dir] [-b, --bootimg-dir] - [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs] + [-k, --kernel-dir] [-n, --native-sysroot] [--sector-size <bytes>] [-f, --build-rootfs] [-c, --compress-with] [-m, --bmap] This command creates an OpenEmbedded image based on the 'OE kickstart
@@ -139,13 +139,16 @@ SYNOPSIS wic create <wks file or image name> [-o <DIRNAME> | --outdir <DIRNAME>] [-e | --image-name] [-s, --skip-build-check] [-D, --debug] [-r, --rootfs-dir] [-b, --bootimg-dir] - [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs] + [-k, --kernel-dir] [-n, --native-sysroot] [--sector-size <bytes>] [-f, --build-rootfs] [-c, --compress-with] [-m, --bmap] [--no-fstab-update] DESCRIPTION This command creates an OpenEmbedded image based on the 'OE kickstart commands' found in the <wks file>. + Use the --sector-size option to select the sector size (in bytes) + used for partition layout calculations (default is 512). + In order to do this, wic needs to know the locations of the various build artifacts required to build the image.
@@ -278,7 +281,7 @@ wic_ls_usage = """ List content of a partitioned image - usage: wic ls <image>[:<partition>[<path>]] [--native-sysroot <path>] + usage: wic ls <image>[:<partition>[<path>]] [--native-sysroot <path>] [--sector-size <bytes>] This command outputs either list of image partitions or directory contents of vfat and ext* partitions.
@@ -296,7 +299,7 @@ SYNOPSIS wic ls <image> wic ls <image>:<vfat or ext* partition> wic ls <image>:<vfat or ext* partition><path> - wic ls <image>:<vfat or ext* partition><path> --native-sysroot <path> + wic ls <image>:<vfat or ext* partition><path> --native-sysroot <path> [--sector-size <bytes>] DESCRIPTION This command lists either partitions of the image or directory contents
@@ -336,6 +339,8 @@ DESCRIPTION The -n option is used to specify the path to the native sysroot containing the tools(parted and mtools) to use. + The --sector-size option sets the sector size used for partition math + (default is 512 bytes). """
@@ -343,7 +348,7 @@ wic_cp_usage = """ Copy files and directories to/from the vfat or ext* partition - usage: wic cp <src> <dest> [--native-sysroot <path>] + usage: wic cp <src> <dest> [--native-sysroot <path>] [--sector-size <bytes>] source/destination image in format <image>:<partition>[<path>]
@@ -364,7 +369,7 @@ SYNOPSIS wic cp <src> <dest>:<partition> wic cp <src>:<partition> <dest> wic cp <src> <dest-image>:<partition><path> - wic cp <src> <dest-image>:<partition><path> --native-sysroot <path> + wic cp <src> <dest-image>:<partition><path> --native-sysroot <path> [--sector-size <bytes>] DESCRIPTION This command copies files or directories either
@@ -408,13 +413,15 @@ DESCRIPTION The -n option is used to specify the path to the native sysroot containing the tools(parted and mtools) to use. + The --sector-size option sets the sector size used for partition math + (default is 512 bytes). """ wic_rm_usage = """ Remove files or directories from the vfat or ext* partitions - usage: wic rm <image>:<partition><path> [--native-sysroot <path>] + usage: wic rm <image>:<partition><path> [--native-sysroot <path>] [--sector-size <bytes>] This command removes files or directories from the vfat or ext* partitions of the partitioned image.
@@ -466,6 +473,8 @@ DESCRIPTION The -n option is used to specify the path to the native sysroot containing the tools(parted and mtools) to use. + The --sector-size option sets the sector size used for partition math + (default is 512 bytes). The -r option is used to remove directories and their contents recursively,this only applies to ext* partition.
scripts/lib/wic/misc.py +6 -0
@@ -263,4 +263,10 @@ def get_bitbake_var(var, image=None, cache=True): Provide old get_bitbake_var API by wrapping get_var method of BB_VARS singleton. """ + if var == "WIC_SECTOR_SIZE": + env_val = os.environ.get("WIC_SECTOR_SIZE") + if env_val is not None: + logger.warning("DEPRECATED: Using WIC_SECTOR_SIZE from environment; prefer --sector-size to avoid surprises.") + return env_val + return BB_VARS.get_var(var, image, cache)
scripts/lib/wic/partition.py +92 -5
136 lines not on this page · raw message
scripts/lib/wic/plugins/imager/direct.py +5 -13
@@ -67,6 +67,7 @@ class DirectPlugin(ImagerPlugin): self._image = None self.ptable_format = self.ks.bootloader.ptable self.parts = self.ks.partitions + self.sector_size = options.sector_size or 512 # as a convenience, set source to the boot partition source # instead of forcing it to be set via bootloader --source
@@ -78,7 +79,7 @@ class DirectPlugin(ImagerPlugin): image_path = self._full_path(self.workdir, self.parts[0].disk, "direct") self._image = PartitionedImage(image_path, self.ptable_format, self.ks.bootloader.diskid, self.parts, self.native_sysroot, - options.extra_space) + options.extra_space, self.sector_size) def setup_workdir(self, workdir): if workdir:
@@ -294,15 +295,13 @@ MBR_OVERHEAD = 1 # Overhead of the GPT partitioning scheme GPT_OVERHEAD = 34 -# Size of a sector in bytes -SECTOR_SIZE = 512 - class PartitionedImage(): """ Partitioned image in a file. """ - def __init__(self, path, ptable_format, disk_id, partitions, native_sysroot=None, extra_space=0): + def __init__(self, path, ptable_format, disk_id, partitions, native_sysroot=None, extra_space=0, + sector_size=512): self.path = path # Path to the image file self.numpart = 0 # Number of allocated partitions self.realpart = 0 # Number of partitions in the partition table
@@ -332,14 +331,7 @@ class PartitionedImage(): self.partitions = partitions self.partimages = [] # Size of a sector used in calculations - sector_size_str = get_bitbake_var('WIC_SECTOR_SIZE') - if sector_size_str is not None: - try: - self.sector_size = int(sector_size_str) - except ValueError: - self.sector_size = SECTOR_SIZE - else: - self.sector_size = SECTOR_SIZE + self.sector_size = sector_size self.native_sysroot = native_sysroot num_real_partitions = len([p for p in self.partitions if not p.no_table])
scripts/lib/wic/plugins/source/bootimg_efi.py +3 -2
@@ -415,8 +415,9 @@ class BootimgEFIPlugin(SourcePlugin): label = part.label if part.label else "ESP" - dosfs_cmd = "mkdosfs -v -n %s -i %s -C %s %d" % \ - (label, part.fsuuid, bootimg, blocks) + sector_size = getattr(creator, 'sector_size', 512) + dosfs_cmd = "mkdosfs -v -n %s -i %s -S %d -C %s %d" % \ + (label, part.fsuuid, sector_size, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) logger.debug("mkdosfs:\n%s" % (str(out)))
scripts/lib/wic/plugins/source/bootimg_pcbios.py +6 -5
@@ -132,7 +132,7 @@ class BootimgPcbiosPlugin(SourcePlugin): cls._do_prepare_grub(part, cr_workdir, oe_builddir, kernel_dir, rootfs_dir, native_sysroot) elif source_params['loader-bios'] == 'syslinux': - cls._do_prepare_syslinux(part, cr_workdir, bootimg_dir, + cls._do_prepare_syslinux(part, creator, cr_workdir, bootimg_dir, kernel_dir, native_sysroot) else: raise WicError("unrecognized bootimg_pcbios loader: %s" % source_params['loader-bios'])
@@ -142,7 +142,7 @@ class BootimgPcbiosPlugin(SourcePlugin): except KeyError: # Required by do_install_disk cls.loader = 'syslinux' - cls._do_prepare_syslinux(part, cr_workdir, bootimg_dir, + cls._do_prepare_syslinux(part, creator, cr_workdir, bootimg_dir, kernel_dir, native_sysroot) @classmethod
@@ -240,7 +240,7 @@ class BootimgPcbiosPlugin(SourcePlugin): cfg.close() @classmethod - def _do_prepare_syslinux(cls, part, cr_workdir, bootimg_dir, + def _do_prepare_syslinux(cls, part, creator, cr_workdir, bootimg_dir, kernel_dir, native_sysroot): """ Called to do the actual content population for a partition i.e. it
@@ -292,8 +292,9 @@ class BootimgPcbiosPlugin(SourcePlugin): label = part.label if part.label else "boot" - dosfs_cmd = "mkdosfs -n %s -i %s -S 512 -C %s %d" % \ - (label, part.fsuuid, bootimg, blocks) + sector_size = getattr(creator, 'sector_size', 512) + dosfs_cmd = "mkdosfs -n %s -i %s -S %d -C %s %d" % \ + (label, part.fsuuid, sector_size, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir)
scripts/lib/wic/plugins/source/isoimage_isohybrid.py +3 -2
@@ -367,8 +367,9 @@ class IsoImagePlugin(SourcePlugin): esp_label = source_params.get('esp_label', 'EFIimg') - dosfs_cmd = 'mkfs.vfat -n \'%s\' -S 512 -C %s %d' \ - % (esp_label, bootimg, blocks) + sector_size = getattr(creator, 'sector_size', 512) + dosfs_cmd = "mkfs.vfat -n '%s' -S %d -C %s %d" % \ + (esp_label, sector_size, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) mmd_cmd = "mmd -i %s ::/EFI" % bootimg
scripts/wic +41 -0
77 lines not on this page · raw message
Signature · 1 line
1.8.3.1
Quoted 13 lines · Mark Hatle via lists.openembedded.org
This work is effectively based on the 'v7' of the 'standalone wic' submission. It was requested that the submission be broken into individual parts so we can review and get merge things individually.
Changes from v7:
Introduction of additional (previous implementation) vfat 4k tests.
Update the test suite to add a new cmdline argument test in addition to (not replacing) the previous variable version of the test.
Update the test case to include fat partition tests.
Quoted 8 lines
Cleanup Deprecation message so it only displays once, if the older style is used. In v7 the message may have been displayed twice.
Mark Hatle (1):
oeqa/selftest: wic: Add vfat totest_wic_sector_sizeTrevor Woerner (1):
wic: re-implement sector-size support
Something in here failed in testing:
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/3519
wic.Wic2.test_mkfs_extraopts
ERROR: A sector/block size is specified in both the mkfs-extraopts argument (-S 1024) and the wic arguments (512) and they do not match.
https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/3401
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/3292
Cheers,
Richard