wic: re-implement sector-size support
PATCH 1/2 wic: re-implement sector-size support
The previous implementation to add variable sector-size support:
- required the variable WIC_SECTOR_SIZE either be defined in a configuration file or be defined in a --vars file
- this means that every invocation of "wic ls", "wic cp", or "wic rm" needed this variable defined (config or --vars)
- required the user to create separate *wks files for every sector size they wanted to use
- required the user to specify the mkfs-extraopts by hand to specify the correct sector size: e.g.
bootloader --ptable gpt part --fstype ext4 --source rootfs --label rofs-a --mkfs-extraopts "-b 4096" part --fstype ext4 --source rootfs --use-uuid --mkfs-extraopts "-b 4096"
- specifying --mkfs-extraopts replaces the defaults with the user-supplied values
- only provided details to support using variable sector-sizes with ext[234] filesystems
- it would not be possible to generate images with different sector sizes in the same build since the configuration and *wks files would need to change and the build re-run for each size
Update the sector-size handling so that:
- the sector-size will now be provided on the cmdline to the "wic ls", "wic cp", "wic rm", and "wic create" commands: default = 512
- this means the configuration and/or --vars file does not need to be changed in order to perform those operations on images with different sector sizes
- support is provided implicitly for mkdosfs and ext[234] partitions
- the user no longer needs to know and supply the sector-size magic in --mkfs-extraopts (thereby clobbering the other defaults)
AI-Generated: codex/gpt-5.1-codex-max
--- meta/lib/oeqa/selftest/cases/wic.py | 39 ++++--------- scripts/lib/wic/engine.py | 45 +++++++-------- scripts/lib/wic/help.py | 23 +++++--- scripts/lib/wic/partition.py | 55 +++++++++++++++++-- scripts/lib/wic/plugins/imager/direct.py | 18 ++---- scripts/lib/wic/plugins/source/bootimg_efi.py | 5 +- .../lib/wic/plugins/source/bootimg_pcbios.py | 11 ++-- .../wic/plugins/source/isoimage_isohybrid.py | 5 +- scripts/wic | 8 +++ 9 files changed, 125 insertions(+), 84 deletions(-)
meta/lib/oeqa/selftest/cases/wic.py +11 -28
@@ -935,33 +935,22 @@ bootloader --ptable gpt""") finally: os.remove(wks_file) - def test_wic_sector_size(self): - """Test generation image sector size""" - + def test_wic_sector_size_cli(self): + """Test sector size handling via CLI option.""" + oldpath = os.environ['PATH'] os.environ['PATH'] = get_bb_var("PATH", "wic-tools") try: - # Add WIC_SECTOR_SIZE into config - config = 'WIC_SECTOR_SIZE = "4096"\n'\ - 'WICVARS:append = " WIC_SECTOR_SIZE"\n' - self.append_config(config) bitbake('core-image-minimal') - # Check WIC_SECTOR_SIZE apply to bitbake variable - wic_sector_size_str = get_bb_var('WIC_SECTOR_SIZE', 'core-image-minimal') - wic_sector_size = int(wic_sector_size_str) - self.assertEqual(4096, wic_sector_size) - - self.logger.info("Test wic_sector_size: %d \n" % wic_sector_size) - with NamedTemporaryFile("w", suffix=".wks") as wks: wks.writelines( ['bootloader --ptable gpt\n', - 'part --fstype ext4 --source rootfs --label rofs-a --mkfs-extraopts "-b 4096"\n', - 'part --fstype ext4 --source rootfs --use-uuid --mkfs-extraopts "-b 4096"\n']) + 'part --fstype ext4 --source rootfs --label rofs-a\n', + 'part --fstype ext4 --source rootfs --use-uuid\n']) wks.flush() - cmd = "wic create %s -e core-image-minimal -o %s" % (wks.name, self.resultdir) + cmd = "wic create %s -e core-image-minimal -o %s --sector-size 4096" % (wks.name, self.resultdir) runCmd(cmd) wksname = os.path.splitext(os.path.basename(wks.name))[0] images = glob(os.path.join(self.resultdir, "%s-*direct" % wksname))
@@ -969,24 +958,18 @@ bootloader --ptable gpt""") sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools') # list partitions - result = runCmd("wic ls %s -n %s" % (images[0], sysroot)) + result = runCmd("wic ls %s -n %s --sector-size 4096" % (images[0], sysroot)) self.assertEqual(3, len(result.output.split('\n'))) - # verify partition size with wic - res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p" % (wic_sector_size, images[0]), + # verify partition size with parted output + res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p" % (4096, images[0]), stderr=subprocess.PIPE) - # parse parted output which looks like this: - # BYT;\n - # /var/tmp/wic/build/tmpgjzzefdd-202410281021-sda.direct:78569472B:file:4096:4096:gpt::;\n - # 1:139264B:39284735B:39145472B:ext4:rofs-a:;\n - # 2:39284736B:78430207B:39145472B:ext4:primary:;\n disk_info = res.output.splitlines()[1] - # Check sector sizes sector_size_logical = int(disk_info.split(":")[3]) sector_size_physical = int(disk_info.split(":")[4]) - self.assertEqual(wic_sector_size, sector_size_logical, "Logical sector size is not %d." % wic_sector_size) - self.assertEqual(wic_sector_size, sector_size_physical, "Physical sector size is not %d." % wic_sector_size) + self.assertEqual(4096, sector_size_logical, "Logical sector size is not 4096.") + self.assertEqual(4096, sector_size_physical, "Physical sector size is not 4096.") finally: os.environ['PATH'] = oldpath
scripts/lib/wic/engine.py +23 -22
102 lines not on this page · raw message
scripts/lib/wic/help.py +16 -7
@@ -118,7 +118,7 @@ wic_create_usage = """ usage: wic create <wks file or image name> [-o <DIRNAME> | --outdir <DIRNAME>] [-e | --image-name] [-s, --skip-build-check] [-D, --debug] [-r, --rootfs-dir] [-b, --bootimg-dir] - [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs] + [-k, --kernel-dir] [-n, --native-sysroot] [--sector-size <bytes>] [-f, --build-rootfs] [-c, --compress-with] [-m, --bmap] This command creates an OpenEmbedded image based on the 'OE kickstart
@@ -139,13 +139,16 @@ SYNOPSIS wic create <wks file or image name> [-o <DIRNAME> | --outdir <DIRNAME>] [-e | --image-name] [-s, --skip-build-check] [-D, --debug] [-r, --rootfs-dir] [-b, --bootimg-dir] - [-k, --kernel-dir] [-n, --native-sysroot] [-f, --build-rootfs] + [-k, --kernel-dir] [-n, --native-sysroot] [--sector-size <bytes>] [-f, --build-rootfs] [-c, --compress-with] [-m, --bmap] [--no-fstab-update] DESCRIPTION This command creates an OpenEmbedded image based on the 'OE kickstart commands' found in the <wks file>. + Use the --sector-size option to select the sector size (in bytes) + used for partition layout calculations (default is 512). + In order to do this, wic needs to know the locations of the various build artifacts required to build the image.
@@ -278,7 +281,7 @@ wic_ls_usage = """ List content of a partitioned image - usage: wic ls <image>[:<partition>[<path>]] [--native-sysroot <path>] + usage: wic ls <image>[:<partition>[<path>]] [--native-sysroot <path>] [--sector-size <bytes>] This command outputs either list of image partitions or directory contents of vfat and ext* partitions.
@@ -296,7 +299,7 @@ SYNOPSIS wic ls <image> wic ls <image>:<vfat or ext* partition> wic ls <image>:<vfat or ext* partition><path> - wic ls <image>:<vfat or ext* partition><path> --native-sysroot <path> + wic ls <image>:<vfat or ext* partition><path> --native-sysroot <path> [--sector-size <bytes>] DESCRIPTION This command lists either partitions of the image or directory contents
@@ -336,6 +339,8 @@ DESCRIPTION The -n option is used to specify the path to the native sysroot containing the tools(parted and mtools) to use. + The --sector-size option sets the sector size used for partition math + (default is 512 bytes). """
@@ -343,7 +348,7 @@ wic_cp_usage = """ Copy files and directories to/from the vfat or ext* partition - usage: wic cp <src> <dest> [--native-sysroot <path>] + usage: wic cp <src> <dest> [--native-sysroot <path>] [--sector-size <bytes>] source/destination image in format <image>:<partition>[<path>]
@@ -364,7 +369,7 @@ SYNOPSIS wic cp <src> <dest>:<partition> wic cp <src>:<partition> <dest> wic cp <src> <dest-image>:<partition><path> - wic cp <src> <dest-image>:<partition><path> --native-sysroot <path> + wic cp <src> <dest-image>:<partition><path> --native-sysroot <path> [--sector-size <bytes>] DESCRIPTION This command copies files or directories either
@@ -408,13 +413,15 @@ DESCRIPTION The -n option is used to specify the path to the native sysroot containing the tools(parted and mtools) to use. + The --sector-size option sets the sector size used for partition math + (default is 512 bytes). """ wic_rm_usage = """ Remove files or directories from the vfat or ext* partitions - usage: wic rm <image>:<partition><path> [--native-sysroot <path>] + usage: wic rm <image>:<partition><path> [--native-sysroot <path>] [--sector-size <bytes>] This command removes files or directories from the vfat or ext* partitions of the partitioned image.
@@ -466,6 +473,8 @@ DESCRIPTION The -n option is used to specify the path to the native sysroot containing the tools(parted and mtools) to use. + The --sector-size option sets the sector size used for partition math + (default is 512 bytes). The -r option is used to remove directories and their contents recursively,this only applies to ext* partition.
scripts/lib/wic/partition.py +50 -5
94 lines not on this page · raw message
scripts/lib/wic/plugins/imager/direct.py +5 -13
@@ -67,6 +67,7 @@ class DirectPlugin(ImagerPlugin): self._image = None self.ptable_format = self.ks.bootloader.ptable self.parts = self.ks.partitions + self.sector_size = options.sector_size or 512 # as a convenience, set source to the boot partition source # instead of forcing it to be set via bootloader --source
@@ -78,7 +79,7 @@ class DirectPlugin(ImagerPlugin): image_path = self._full_path(self.workdir, self.parts[0].disk, "direct") self._image = PartitionedImage(image_path, self.ptable_format, self.ks.bootloader.diskid, self.parts, self.native_sysroot, - options.extra_space) + options.extra_space, self.sector_size) def setup_workdir(self, workdir): if workdir:
@@ -294,15 +295,13 @@ MBR_OVERHEAD = 1 # Overhead of the GPT partitioning scheme GPT_OVERHEAD = 34 -# Size of a sector in bytes -SECTOR_SIZE = 512 - class PartitionedImage(): """ Partitioned image in a file. """ - def __init__(self, path, ptable_format, disk_id, partitions, native_sysroot=None, extra_space=0): + def __init__(self, path, ptable_format, disk_id, partitions, native_sysroot=None, extra_space=0, + sector_size=512): self.path = path # Path to the image file self.numpart = 0 # Number of allocated partitions self.realpart = 0 # Number of partitions in the partition table
@@ -332,14 +331,7 @@ class PartitionedImage(): self.partitions = partitions self.partimages = [] # Size of a sector used in calculations - sector_size_str = get_bitbake_var('WIC_SECTOR_SIZE') - if sector_size_str is not None: - try: - self.sector_size = int(sector_size_str) - except ValueError: - self.sector_size = SECTOR_SIZE - else: - self.sector_size = SECTOR_SIZE + self.sector_size = sector_size or 512 self.native_sysroot = native_sysroot num_real_partitions = len([p for p in self.partitions if not p.no_table])
scripts/lib/wic/plugins/source/bootimg_efi.py +3 -2
@@ -415,8 +415,9 @@ class BootimgEFIPlugin(SourcePlugin): label = part.label if part.label else "ESP" - dosfs_cmd = "mkdosfs -v -n %s -i %s -C %s %d" % \ - (label, part.fsuuid, bootimg, blocks) + sector_size = getattr(creator, 'sector_size', 512) or 512 + dosfs_cmd = "mkdosfs -v -n %s -i %s -S %d -C %s %d" % \ + (label, part.fsuuid, sector_size, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) logger.debug("mkdosfs:\n%s" % (str(out)))
scripts/lib/wic/plugins/source/bootimg_pcbios.py +6 -5
@@ -132,7 +132,7 @@ class BootimgPcbiosPlugin(SourcePlugin): cls._do_prepare_grub(part, cr_workdir, oe_builddir, kernel_dir, rootfs_dir, native_sysroot) elif source_params['loader-bios'] == 'syslinux': - cls._do_prepare_syslinux(part, cr_workdir, bootimg_dir, + cls._do_prepare_syslinux(part, creator, cr_workdir, bootimg_dir, kernel_dir, native_sysroot) else: raise WicError("unrecognized bootimg_pcbios loader: %s" % source_params['loader-bios'])
@@ -142,7 +142,7 @@ class BootimgPcbiosPlugin(SourcePlugin): except KeyError: # Required by do_install_disk cls.loader = 'syslinux' - cls._do_prepare_syslinux(part, cr_workdir, bootimg_dir, + cls._do_prepare_syslinux(part, creator, cr_workdir, bootimg_dir, kernel_dir, native_sysroot) @classmethod
@@ -240,7 +240,7 @@ class BootimgPcbiosPlugin(SourcePlugin): cfg.close() @classmethod - def _do_prepare_syslinux(cls, part, cr_workdir, bootimg_dir, + def _do_prepare_syslinux(cls, part, creator, cr_workdir, bootimg_dir, kernel_dir, native_sysroot): """ Called to do the actual content population for a partition i.e. it
@@ -292,8 +292,9 @@ class BootimgPcbiosPlugin(SourcePlugin): label = part.label if part.label else "boot" - dosfs_cmd = "mkdosfs -n %s -i %s -S 512 -C %s %d" % \ - (label, part.fsuuid, bootimg, blocks) + sector_size = getattr(creator, 'sector_size', 512) or 512 + dosfs_cmd = "mkdosfs -n %s -i %s -S %d -C %s %d" % \ + (label, part.fsuuid, sector_size, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) mcopy_cmd = "mcopy -i %s -s %s/* ::/" % (bootimg, hdddir)
scripts/lib/wic/plugins/source/isoimage_isohybrid.py +3 -2
@@ -367,8 +367,9 @@ class IsoImagePlugin(SourcePlugin): esp_label = source_params.get('esp_label', 'EFIimg') - dosfs_cmd = 'mkfs.vfat -n \'%s\' -S 512 -C %s %d' \ - % (esp_label, bootimg, blocks) + sector_size = getattr(creator, 'sector_size', 512) or 512 + dosfs_cmd = "mkfs.vfat -n '%s' -S %d -C %s %d" % \ + (esp_label, sector_size, bootimg, blocks) exec_native_cmd(dosfs_cmd, native_sysroot) mmd_cmd = "mmd -i %s ::/EFI" % bootimg
scripts/wic +8 -0
@@ -376,6 +376,8 @@ def wic_init_parser_create(subparser): default="direct", help="the wic imager plugin") subparser.add_argument("--extra-space", type=int, dest="extra_space", default=0, help="additional free disk space to add to the image") + subparser.add_argument("--sector-size", dest="sector_size", type=int, default=512, + help="sector size in bytes (default: 512)") return
@@ -413,6 +415,8 @@ def imgtype(arg): def wic_init_parser_ls(subparser): subparser.add_argument("path", type=imgtype, help="image spec: <image>[:<vfat partition>[<path>]]") + subparser.add_argument("--sector-size", dest="sector_size", type=int, default=512, + help="sector size in bytes (default: 512)") subparser.add_argument("-n", "--native-sysroot", help="path to the native sysroot containing the tools") subparser.add_argument("-e", "--image-name", dest="image_name",
@@ -433,6 +437,8 @@ def wic_init_parser_cp(subparser): help="image spec: <image>:<vfat partition>[<path>] or <file>") subparser.add_argument("dest", help="image spec: <image>:<vfat partition>[<path>] or <file>") + subparser.add_argument("--sector-size", dest="sector_size", type=int, default=512, + help="sector size in bytes (default: 512)") subparser.add_argument("-n", "--native-sysroot", help="path to the native sysroot containing the tools") subparser.add_argument("-e", "--image-name", dest="image_name",
@@ -445,6 +451,8 @@ def wic_init_parser_cp(subparser): def wic_init_parser_rm(subparser): subparser.add_argument("path", type=imgpathtype, help="path: <image>:<vfat partition><path>") + subparser.add_argument("--sector-size", dest="sector_size", type=int, default=512, + help="sector size in bytes (default: 512)") subparser.add_argument("-n", "--native-sysroot", help="path to the native sysroot containing the tools") subparser.add_argument("-r", dest="recursive_delete", action="store_true", default=False,
Signature · 1 line
2.51.0
PATCH 2/2 ufs image class: add
Add an image class and wks file that demonstrates generating a wic image with a 4096-byte sector size.
--- meta/classes-recipe/image.bbclass | 2 +- meta/classes-recipe/image_types_ufs.bbclass | 221 ++++++++++++++++++++ scripts/lib/wic/canned-wks/mkdisk-ufs.wks | 5 + 3 files changed, 227 insertions(+), 1 deletion(-) create mode 100644 meta/classes-recipe/image_types_ufs.bbclass create mode 100644 scripts/lib/wic/canned-wks/mkdisk-ufs.wks
meta/classes-recipe/image.bbclass +1 -1
@@ -18,7 +18,7 @@ inherit populate_sdk_base IMGCLASSES += "${@['', 'populate_sdk_ext']['linux' in d.getVar("SDK_OS")]}" IMGCLASSES += "${@bb.utils.contains_any('IMAGE_FSTYPES', 'live iso hddimg', 'image-live', '', d)}" IMGCLASSES += "${@bb.utils.contains('IMAGE_FSTYPES', 'container', 'image-container', '', d)}" -IMGCLASSES += "image_types_wic" +IMGCLASSES += "image_types_wic image_types_ufs" IMGCLASSES += "rootfs-postcommands" IMGCLASSES += "image-postinst-intercepts" IMGCLASSES += "overlayfs-etc"
meta/classes-recipe/image_types_ufs.bbclass new file +221 -0
221 lines not on this page · raw message
scripts/lib/wic/canned-wks/mkdisk-ufs.wks new file +5 -0
@@ -0,0 +1,5 @@ +bootloader --ptable msdos + +part /boot --source rootfs --fstype=vfat --label boot --change-directory boot/ +part / --source rootfs --fstype=ext4 --label root --exclude-path boot/ +part swap --fstype=swap --label swap --size 1M
Signature · 1 line
2.51.0
Re: [OE-core] [PATCH 2/2] ufs image class: add
Quoted 5 lines · Trevor Woerner via lists.openembedded.org
Add an image class and wks file that demonstrates generating a wic image with a 4096-byte sector size.
Signed-off-by Trevor Woerner <[email protected]>---
Thanks for your patch.
meta/classes-recipe/image_types_ufs.bbclass line 74
Quoted 3 lines
meta/classes-recipe/image_types_ufs.bbclass new file +3
-0+ +IMAGE_CMD:wic.ufs () { + out="${IMGDEPLOYDIR}/${IMAGE_NAME}"
IMAGE_CMD:wic_ufs ?
At least I believe this is the source of this warning:
WARNING: core-image-minimal-1.0-r0 do_image_wic_ufs: Function do_image_wic_ufs doesn't exist
https://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/3267
https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/3157
https://autobuilder.yoctoproject.org/valkyrie/#/builders/22/builds/3210
Thanks,
Mathieu
Signature · 3 lines
Mathieu Dubois-Briand, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
Re: [OE-core] [PATCH 2/2] ufs image class: add
Hello!
Quoted 16 lines · Mathieu Dubois-Briand
Quoted 5 lines · Trevor Woerner via lists.openembedded.org
Add an image class and wks file that demonstrates generating a wic image with a 4096-byte sector size.
Signed-off-by Trevor Woerner <[email protected]>---
Thanks for your patch.
meta/classes-recipe/image_types_ufs.bbclass line 74
Quoted 3 lines
meta/classes-recipe/image_types_ufs.bbclass new file +3
-0+ +IMAGE_CMD:wic.ufs () { + out="${IMGDEPLOYDIR}/${IMAGE_NAME}"IMAGE_CMD:wic_ufs ?
Quoted 7 lines
At least I believe this is the source of this warning:
WARNING: core-image-minimal-1.0-r0
do_image_wic_ufs: Functiondo_image_wic_ufsdoesn't existhttps://autobuilder.yoctoproject.org/valkyrie/#/builders/68/builds/3267
https://autobuilder.yoctoproject.org/valkyrie/#/builders/30/builds/3157
https://autobuilder.yoctoproject.org/valkyrie/#/builders/22/builds/3210
Thank you for your report! I had not noticed this warning since it only appears when wic.ufs is *not* requested (and since I was adding this new IMAGE_FSTYPE it did not occur to me to test without it).
I thought my v2 had fixed the problem, but it didn't; please ignore.
But I believe my v3 found the problem and fixed it.
Thanks and best regards,
Trevor