re-implement sector-size [was: standalone wic]
v8 cover re-implement sector-size [was: standalone wic]
Seven revisions of a patch set have been sent to split wic out from oe-core. As part of that patch set two other pieces have tagged along for the ride:
- fix a wic bug found with python 3.14 in Fedora 43
- re-implement sector-size handling support
The wic bug fix has already been sent and merged, therefore, in an attempt to make the splitting-out of wic more palatable, here is the re-implement sector-size support on its own.
The previous patch set can be references for all the (non-relevant) details:
https://lists.openembedded.org/g/openembedded-core/topic/patch_v7_0_9_standalone_wic/118086344
Trevor Woerner & Mark Hatle (1):
wic: re-implement sector-size support
meta/lib/oeqa/selftest/cases/wic.py | 78 ++++++++++++++++++- scripts/lib/wic/engine.py | 60 +++++++------- scripts/lib/wic/help.py | 23 ++++-- scripts/lib/wic/misc.py | 6 ++ scripts/lib/wic/partition.py | 63 +++++++++++++-- scripts/lib/wic/plugins/imager/direct.py | 18 ++--- scripts/lib/wic/plugins/source/bootimg_efi.py | 5 +- .../lib/wic/plugins/source/bootimg_pcbios.py | 11 +-- .../wic/plugins/source/isoimage_isohybrid.py | 5 +- scripts/wic | 41 ++++++++++ 10 files changed, 246 insertions(+), 64 deletions(-)
Signature · 1 line
2.51.0
PATCH v8 1/1 wic: re-implement sector-size support
The previous implementation had the following limitations:
- required the variable WIC_SECTOR_SIZE either be defined in a configuration file or be defined in a --vars file
- this means that every invocation of "wic ls", "wic cp", or "wic rm" needed this variable defined (config or --vars)
- required the user to create separate *wks files for every sector size they wanted to use
- required the user to specify the --mkfs-extraopts by hand to specify the correct sector size: e.g.
bootloader --ptable gpt
part --fstype vfat --label emptyfat --mkfs-extraopts "-S 4096"
part --fstype ext4 --source rootfs --label rofs-a --mkfs-extraopts "-b 4096"
part --fstype ext4 --source rootfs --use-uuid --mkfs-extraopts "-b 4096"- it would not be possible to generate images with different sector sizes in the same build since the configuration and *wks files would need to change and the build re-run for each size
The new implementation handles the sector-size via a CLI argument, while preserving the previously implemented variable definitions:
- the sector-size may now be provided on the cmdline to the "wic ls", "wic cp", "wic rm", and "wic create" commands: default = 512
- this means the configuration and/or --vars file does not need to be changed in order to perform those operations on images with different sector sizes
- support is provided implicitly for mkdosfs and ext[234] partitions
- the user no longer needs to know and supply the sector-size magic in --mkfs-extraopts (thereby clobbering the other defaults)
As before, if the --sector-size command-line argument is not given, allow the sector-size to be provided via the WIC_SECTOR_SIZE bitbake variable. The user is warned that this behavior is deprecated. If both are given, warn the user that the cmdline argument takes precedence.
AI-Generated: codex/gpt-5.1-codex-max
- restore environ test case, as it is still supported (but obsolete)
- revised commit message above
---
changes in v8:
- Mark Hatle stepped in to help provide advice and updates
- add back the code and test to support/allow users to specify a sector-size in --mkfs-extraopts, this gives the user 3 places in which the sector-size can be specified:
1. cmdline
2. --mkfs-extraopts in *.wks files
3. WIC_SECTOR_SIZE bitbake variable/--vars file
changes in v7:
- none
changes in v6:
- many comments were expanded to provide deeper explanations
- in many places we know that sector_size will be defined (for example by virtue of it being an argument with a default value) therefore checks to make sure sector_size exists can be removed
- when processing --mkfs-extraopts for fat and ext* filesystems, error out if the user provides sector-size options in the wks file since these will conflict with sector-size settings on the cmdline or via WIC_SECTOR_SIZE
- clean up the logic related to warning the user if WIC_SECTOR_SIZE is used
changes in v5:
- fix a bug in the case where WIC_SECTOR_SIZE does not exist at all and is not provided neither in the environment nor a config file
changes in v4:
- allow the sector-size to be set from WIC_SECTOR_SIZE (environment or vars file) but mark its use deprecated, suggest the user use the cmdline arg instead; if both are given point out that the cmdline value takes precedence
changes in v3:
- none
changes in v2:
- none
--- meta/lib/oeqa/selftest/cases/wic.py | 78 ++++++++++++++++++- scripts/lib/wic/engine.py | 60 +++++++------- scripts/lib/wic/help.py | 23 ++++-- scripts/lib/wic/misc.py | 6 ++ scripts/lib/wic/partition.py | 63 +++++++++++++-- scripts/lib/wic/plugins/imager/direct.py | 18 ++--- scripts/lib/wic/plugins/source/bootimg_efi.py | 5 +- .../lib/wic/plugins/source/bootimg_pcbios.py | 11 +-- .../wic/plugins/source/isoimage_isohybrid.py | 5 +- scripts/wic | 41 ++++++++++ 10 files changed, 246 insertions(+), 64 deletions(-)
meta/lib/oeqa/selftest/cases/wic.py +74 -4
@@ -935,8 +935,8 @@ bootloader --ptable gpt""") finally: os.remove(wks_file) - def test_wic_sector_size(self): - """Test generation image sector size""" + def test_wic_sector_size_env(self): + """Test generation image sector size via environment (obsolete)""" oldpath = os.environ['PATH'] os.environ['PATH'] = get_bb_var("PATH", "wic-tools")
@@ -972,8 +972,8 @@ bootloader --ptable gpt""") # list partitions result = runCmd("wic ls %s -n %s" % (images[0], sysroot)) print(result.output) - # 4 lines of output: header + 3 partition - self.assertEqual(4, len(result.output.split('\n'))) + # Deprecated message + 4 lines of output: header + 3 partitions + self.assertEqual(5, len(result.output.split('\n'))) # verify partition size with wic res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p" % (wic_sector_size, images[0]),
@@ -1016,6 +1016,76 @@ bootloader --ptable gpt""") finally: os.environ['PATH'] = oldpath + def test_wic_sector_size_cli(self): + """Test sector size handling via CLI option.""" + + oldpath = os.environ['PATH'] + os.environ['PATH'] = get_bb_var("PATH", "wic-tools") + + try: + bitbake('core-image-minimal') + + with NamedTemporaryFile("w", suffix=".wks") as wks: + wks.writelines( + ['bootloader --ptable gpt\n', + 'part --fstype vfat --fstype vfat --label emptyfat --size 1M\n', + 'part --fstype ext4 --source rootfs --label rofs-a\n', + 'part --fstype ext4 --source rootfs --use-uuid\n']) + wks.flush() + cmd = "wic create %s -e core-image-minimal -o %s --sector-size 4096" % (wks.name, self.resultdir) + runCmd(cmd) + wksname = os.path.splitext(os.path.basename(wks.name))[0] + images = glob(os.path.join(self.resultdir, "%s-*direct" % wksname)) + self.assertEqual(1, len(images)) + + sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools') + # list partitions + result = runCmd("wic ls %s -n %s --sector-size 4096" % (images[0], sysroot)) + print(result.output) + # 4 lines of output: header + 3 partitions + self.assertEqual(4, len(result.output.split('\n'))) + + # verify partition size with parted output + res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p" % (4096, images[0]), + stderr=subprocess.PIPE) + + print(res.output) + # parse parted output which looks like this: + # BYT;\n + # /var/tmp/wic/build/tmpgjzzefdd-202410281021-sda.direct:78569472B:file:4096:4096:gpt::;\n + # 1:139264B:1187839B:1048576B::emptyfat:msftdata; + # 2:1187840B:149270527B:148082688B:ext4:rofs-a:; + # 3:149270528B:297353215B:148082688B:ext4:primary:; + disk_info = res.output.splitlines()[1] + # Check sector sizes + sector_size_logical = int(disk_info.split(":")[3]) + sector_size_physical = int(disk_info.split(":")[4]) + self.assertEqual(4096, sector_size_logical, "Logical sector size is not 4096.") + self.assertEqual(4096, sector_size_physical, "Physical sector size is not 4096.") + + # It is a known issue with parsed that a 4K FAT partition does + # not have a recognized filesystem type of *fat. + part_info = res.output.splitlines()[2] + partname = part_info.split(":")[5] + parttype = part_info.split(":")[6] + self.assertEqual('emptyfat', partname) + self.assertEqual('msftdata;', parttype) + + part_info = res.output.splitlines()[3] + parttype = part_info.split(":")[4] + partname = part_info.split(":")[5] + self.assertEqual('ext4', parttype) + self.assertEqual('rofs-a', partname) + + part_info = res.output.splitlines()[4] + parttype = part_info.split(":")[4] + partname = part_info.split(":")[5] + self.assertEqual('ext4', parttype) + self.assertEqual('primary', partname) + + finally: + os.environ['PATH'] = oldpath + class Wic2(WicTestCase): def test_bmap_short(self):
scripts/lib/wic/engine.py +34 -26
118 lines not on this page · raw message
scripts/lib/wic/help.py +16 -7
@@ -118,7 +118,7 @@ wic_create_usage = """ usage: wic create <wks file or image name> [-o <DIRNAME> | --outdir <DIRNAME>] [-e | --image-name] [-s, --skip-build-check] [-D, --debug] [-r, --rootfs-dir] [-b, --bootimg-dir] - [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs] + [-k, --kernel-dir] [-n, --native-sysroot] [--sector-size <bytes>] [-f, --build-rootfs] [-c, --compress-with] [-m, --bmap] This command creates an OpenEmbedded image based on the 'OE kickstart
@@ -139,13 +139,16 @@ SYNOPSIS wic create <wks file or image name> [-o <DIRNAME> | --outdir <DIRNAME>] [-e | --image-name] [-s, --skip-build-check] [-D, --debug] [-r, --rootfs-dir] [-b, --bootimg-dir] - [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs] + [-k, --kernel-dir] [-n, --native-sysroot] [--sector-size <bytes>] [-f, --build-rootfs] [-c, --compress-with] [-m, --bmap] [--no-fstab-update] DESCRIPTION This command creates an OpenEmbedded image based on the 'OE kickstart commands' found in the <wks file>. + Use the --sector-size option to select the sector size (in bytes) + used for partition layout calculations (default is 512). + In order to do this, wic needs to know the locations of the various build artifacts required to build the image.
@@ -278,7 +281,7 @@ wic_ls_usage = """ List content of a partitioned image - usage: wic ls <image>[:<partition>[<path>]] [--native-sysroot <path>] + usage: wic ls <image>[:<partition>[<path>]] [--native-sysroot <path>] [--sector-size <bytes>] This command outputs either list of image partitions or directory contents of vfat and ext* partitions.
@@ -296,7 +299,7 @@ SYNOPSIS wic ls <image> wic ls <image>:<vfat or ext* partition> wic ls <image>:<vfat or ext* partition><path> - wic ls <image>:<vfat or ext* partition><path> --native-sysroot <path> + wic ls <image>:<vfat or ext* partition><path> --native-sysroot <path> [--sector-size <bytes>] DESCRIPTION This command lists either partitions of the image or directory contents
@@ -336,6 +339,8 @@ DESCRIPTION The -n option is used to specify the path to the native sysroot containing the tools(parted and mtools) to use. + The --sector-size option sets the sector size used for partition math + (default is 512 bytes). """
@@ -343,7 +348,7 @@ wic_cp_usage = """ Copy files and directories to/from the vfat or ext* partition - usage: wic cp <src> <dest> [--native-sysroot <path>] + usage: wic cp <src> <dest> [--native-sysroot <path>] [--sector-size <bytes>] source/destination image in format <image>:<partition>[<path>]
@@ -364,7 +369,7 @@ SYNOPSIS wic cp <src> <dest>:<partition> wic cp <src>:<partition> <dest> wic cp <src> <dest-image>:<partition><path> - wic cp <src> <dest-image>:<partition><path> --native-sysroot <path> + wic cp <src> <dest-image>:<partition><path> --native-sysroot <path> [--sector-size <bytes>] DESCRIPTION This command copies files or directories either
@@ -408,13 +413,15 @@ DESCRIPTION The -n option is used to specify the path to the native sysroot containing the tools(parted and mtools) to use. + The --sector-size option sets the sector size used for partition math + (default is 512 bytes). """ wic_rm_usage = """ Remove files or directories from the vfat or ext* partitions - usage: wic rm <image>:<partition><path> [--native-sysroot <path>] + usage: wic rm <image>:<partition><path> [--native-sysroot <path>] [--sector-size <bytes>] This command removes files or directories from the vfat or ext* partitions of the partitioned image.
@@ -466,6 +473,8 @@ DESCRIPTION The -n option is used to specify the path to the native sysroot containing the tools(parted and mtools) to use. + The --sector-size option sets the sector size used for partition math + (default is 512 bytes). The -r option is used to remove directories and their contents recursively,this only applies to ext* partition.
scripts/lib/wic/misc.py +6 -0
@@ -263,4 +263,10 @@ def get_bitbake_var(var, image=None, cache=True): Provide old get_bitbake_var API by wrapping get_var method of BB_VARS singleton. """ + if var == "WIC_SECTOR_SIZE": + env_val = os.environ.get("WIC_SECTOR_SIZE") + if env_val is not None: + logger.warning("DEPRECATED: Using WIC_SECTOR_SIZE from environment; prefer --sector-size to avoid surprises.") + return env_val + return BB_VARS.get_var(var, image, cache)
scripts/lib/wic/partition.py +58 -5
102 lines not on this page · raw message
scripts/lib/wic/plugins/imager/direct.py +5 -13
@@ -67,6 +67,7 @@ class DirectPlugin(ImagerPlugin): self._image = None self.ptable_format = self.ks.bootloader.ptable self.parts = self.ks.partitions + self.sector_size = options.sector_size or 512 # as a convenience, set source to the boot partition source # instead of forcing it to be set via bootloader --source
@@ -78,7 +79,7 @@ class DirectPlugin(ImagerPlugin): image_path = self._full_path(self.workdir, self.parts[0].disk, "direct") self._image = PartitionedImage(image_path, self.ptable_format, self.ks.bootloader.diskid, self.parts, self.native_sysroot, - options.extra_space) + options.extra_space, self.sector_size) def setup_workdir(self, workdir): if workdir:
@@ -294,15 +295,13 @@ MBR_OVERHEAD = 1 # Overhead of the GPT partitioning scheme GPT_OVERHEAD = 34 -# Size of a sector in bytes -SECTOR_SIZE = 512 - class PartitionedImage(): """ Partitioned image in a file. """ - def __init__(self, path, ptable_format, disk_id, partitions, native_sysroot=None, extra_space=0): + def __init__(self, path, ptable_format, disk_id, partitions, native_sysroot=None, extra_space=0, + sector_size=512): self.path = path # Path to the image file self.numpart = 0 # Number of allocated partitions self.realpart = 0 # Number of partitions in the partition table
@@ -332,14 +331,7 @@ class PartitionedImage(): self.partitions = partitions self.partimages = [] # Size of a sector used in calculations - sector_size_str = get_bitbake_var('WIC_SECTOR_SIZE') - if sector_size_str is not None: - try: - self.sector_size = int(sector_size_str) - except ValueError: - self.sector_size = SECTOR_SIZE - else: - self.sector_size = SECTOR_SIZE + self.sector_size = sector_size self.native_sysroot = native_sysroot num_real_partitions = len([p for p in self.partitions if not p.no_table])
scripts/lib/wic/plugins/source/bootimg_efi.py +3 -2
@@ -415,8 +415,9 @@ class BootimgEFIPlugin(SourcePlugin): label = part.label if part.label else "ESP" - dosfs_cmd = "mkdosfs -v -n %s -i %s -C %s %d" % \ - (label, part.fsuuid, bootimg, blocks) + sector_size = getattr(creator, 'sector_size', 512) + dosfs_cmd = "mkdosfs -v -n %s -i %s -S %d -C %s %d" % \ + (label, part.fsuuid, sector_size, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) logger.debug("mkdosfs:\n%s" % (str(out)))
scripts/lib/wic/plugins/source/bootimg_pcbios.py +6 -5
@@ -132,7 +132,7 @@ class BootimgPcbiosPlugin(SourcePlugin): cls._do_prepare_grub(part, cr_workdir, oe_builddir, kernel_dir, rootfs_dir, native_sysroot) elif source_params['loader-bios'] == 'syslinux': - cls._do_prepare_syslinux(part, cr_workdir, bootimg_dir, + cls._do_prepare_syslinux(part, creator, cr_workdir, bootimg_dir, kernel_dir, native_sysroot) else: raise WicError("unrecognized bootimg_pcbios loader: %s" % source_params['loader-bios'])
@@ -142,7 +142,7 @@ class BootimgPcbiosPlugin(SourcePlugin): except KeyError: # Required by do_install_disk cls.loader = 'syslinux' - cls._do_prepare_syslinux(part, cr_workdir, bootimg_dir, + cls._do_prepare_syslinux(part, creator, cr_workdir, bootimg_dir, kernel_dir, native_sysroot) @classmethod
@@ -240,7 +240,7 @@ class BootimgPcbiosPlugin(SourcePlugin): cfg.close() @classmethod - def _do_prepare_syslinux(cls, part, cr_workdir, bootimg_dir, + def _do_prepare_syslinux(cls, part, creator, cr_workdir, bootimg_dir, kernel_dir, native_sysroot): """ Called to do the actual content population for a partition i.e. it
@@ -292,8 +292,9 @@ class BootimgPcbiosPlugin(SourcePlugin): label = part.label if part.label else "boot" - dosfs_cmd = "mkdosfs -n %s -i %s -S 512 -C %s %d" % \ - (label, part.fsuuid, bootimg, blocks) + sector_size = getattr(creator, 'sector_size', 512) + dosfs_cmd = "mkdosfs -n %s -i %s -S %d -C %s %d" % \ + (label, part.fsuuid, sector_size, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir)
scripts/lib/wic/plugins/source/isoimage_isohybrid.py +3 -2
@@ -367,8 +367,9 @@ class IsoImagePlugin(SourcePlugin): esp_label = source_params.get('esp_label', 'EFIimg') - dosfs_cmd = 'mkfs.vfat -n \'%s\' -S 512 -C %s %d' \ - % (esp_label, bootimg, blocks) + sector_size = getattr(creator, 'sector_size', 512) + dosfs_cmd = "mkfs.vfat -n '%s' -S %d -C %s %d" % \ + (esp_label, sector_size, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) mmd_cmd = "mmd -i %s ::/EFI" % bootimg
scripts/wic +41 -0
77 lines not on this page · raw message
Signature · 1 line
2.51.0
Re: [OE-core] [PATCH v8 1/1] wic: re-implement sector-size support
Hi,
Quoted 40 lines · Trevor Woerner via lists.openembedded.org
The previous implementation had the following limitations:
- required the variableWIC_SECTOR_SIZEeither be defined in a configuration file or be defined in a --vars file
- this means that every invocation of "wic ls", "wic cp", or "wic rm" needed this variable defined (config or --vars)
- required the user to create separate *wks files for every sector size they wanted to use
- required the user to specify the --mkfs-extraopts by hand to specify the correct sector size: e.g.bootloader --ptable gpt part --fstype vfat --label emptyfat --mkfs-extraopts "-S 4096" part --fstype ext4 --source rootfs --label rofs-a --mkfs-extraopts "-b 4096" part --fstype ext4 --source rootfs --use-uuid --mkfs-extraopts "-b 4096"- it would not be possible to generate images with different sector sizes in the same build since the configuration and *wks files would need to change and the build re-run for each size
The new implementation handles the sector-size via a CLI argument, while preserving the previously implemented variable definitions:
- the sector-size may now be provided on the cmdline to the "wic ls", "wic cp", "wic rm", and "wic create" commands: default = 512
- this means the configuration and/or --vars file does not need to be changed in order to perform those operations on images with different sector sizes
- support is provided implicitly for mkdosfs and ext[234] partitions
- the user no longer needs to know and supply the sector-size magic in --mkfs-extraopts (thereby clobbering the other defaults)As before, if the --sector-size command-line argument is not given, allow the sector-size to be provided via the
WIC_SECTOR_SIZEbitbake variable. The user is warned that this behavior is deprecated. If both are given, warn the user that the cmdline argument takes precedence.AI-Generated: codex/gpt-5.1-codex-max
Signed-off-by Trevor Woerner <[email protected]>- restore environ test case, as it is still supported (but obsolete)
- revised commit message aboveSigned-off-by Mark Hatle <[email protected]>
Quoted 9 lines
---
changes in v8:
- Mark Hatle stepped in to help provide advice and updates
- add back the code and test to support/allow users to specify a sector-size in --mkfs-extraopts, this gives the user 3 places in which the sector-size can be specified:
1. cmdline
2. --mkfs-extraopts in *.wks files
3.WIC_SECTOR_SIZEbitbake variable/--vars file
I've written a migration note here (sent on the docs list):
https://lore.kernel.org/r/[email protected]
Could you confirm that this is the right approach?
Antonin
Re: [OE-core] [PATCH v8 1/1] wic: re-implement sector-size support
Quoted 53 lines · Antonin Godard
Hi,
Quoted 40 lines · Trevor Woerner via lists.openembedded.org
The previous implementation had the following limitations:
- required the variableWIC_SECTOR_SIZEeither be defined in a configuration file or be defined in a --vars file
- this means that every invocation of "wic ls", "wic cp", or "wic rm" needed this variable defined (config or --vars)
- required the user to create separate *wks files for every sector size they wanted to use
- required the user to specify the --mkfs-extraopts by hand to specify the correct sector size: e.g.bootloader --ptable gpt part --fstype vfat --label emptyfat --mkfs-extraopts "-S 4096" part --fstype ext4 --source rootfs --label rofs-a --mkfs-extraopts "-b 4096" part --fstype ext4 --source rootfs --use-uuid --mkfs-extraopts "-b 4096"- it would not be possible to generate images with different sector sizes in the same build since the configuration and *wks files would need to change and the build re-run for each size
The new implementation handles the sector-size via a CLI argument, while preserving the previously implemented variable definitions:
- the sector-size may now be provided on the cmdline to the "wic ls", "wic cp", "wic rm", and "wic create" commands: default = 512
- this means the configuration and/or --vars file does not need to be changed in order to perform those operations on images with different sector sizes
- support is provided implicitly for mkdosfs and ext[234] partitions
- the user no longer needs to know and supply the sector-size magic in --mkfs-extraopts (thereby clobbering the other defaults)As before, if the --sector-size command-line argument is not given, allow the sector-size to be provided via the
WIC_SECTOR_SIZEbitbake variable. The user is warned that this behavior is deprecated. If both are given, warn the user that the cmdline argument takes precedence.AI-Generated: codex/gpt-5.1-codex-max
Signed-off-by Trevor Woerner <[email protected]>- restore environ test case, as it is still supported (but obsolete)
- revised commit message aboveSigned-off-by Mark Hatle <[email protected]>Quoted 9 lines
---
changes in v8:
- Mark Hatle stepped in to help provide advice and updates
- add back the code and test to support/allow users to specify a sector-size in --mkfs-extraopts, this gives the user 3 places in which the sector-size can be specified:
1. cmdline
2. --mkfs-extraopts in *.wks files
3.WIC_SECTOR_SIZEbitbake variable/--vars file
Quoted 4 lines
I've written a migration note here (sent on the docs list):
https://lore.kernel.org/r/[email protected]Could you confirm that this is the right approach?
Yes, that should work. I was envisioning people writing their own classes to call wic with the appropriate sector-size setting, but WIC_CREATE_EXTRA_ARGS should work fine as well.
Thanks!