standalone wic
v9 cover standalone wic
The wic utility will no longer be integrated into the oe-core repository. However, the wic utility will continue to exist under The Yocto Project umbrella. This will allow wic to be more easily used independently of bitbake and oe-core.
The new home of wic is:
https://git.yoctoproject.org/wic
These changes have been tested with oe-selftest as follows:
$ sudo .../layers/openembedded-core/scripts/runqemu-gen-tapdevs <my numeric group id> 4
$ oe-selftest -r wic The original patchset contained several more patches. Two patches that were part of this set have already been accepted into oe-core, and several patches that added "--vars <envfile>" options to many wic invocations in the oe-selftests are no longer needed.
The order of the patches in this patchset is entirely different from the order of the patches in previous patchsets. Although it had been decided that it would be okay if patches in this series break the build (then fix the build in later patches), it was pointed out that using a different order would minimize (or eliminate) and breakages while continuing to make small, incremental changes.
This version differs from previous versions as follows:
- the patch to provide the sector-size via the cmdline has already been accepted, therefore it is dropped from this patchset
- a patch to fix a long-standing sparse_copy() bug was accepted into oe-core already, therefore no changes are needed in this patchset to work around the issue
- wic will continue to look in wic and scripts/lib/wic/canned-wks directories for *wks files
- use bitbake to learn variable values if bitbake is available, previous patchsets refused to use bitbake to discover variable values and insisted on always using a --vars file, even if the bitbake program was available
- by continuing to use bitbake (if available) most of the patches to tweak the oe-selftests are no longer required (since they are run in an environment where bitbake is available)
- reorder the patches in this series to minimize build breakages
In v8 the only patch was to upstream the sector-size cmdline option.
In v7 the code was changed to work around the sparse_copy() bug. That bug was root-caused and sent upstream independently.
Changes in v6:
- many code and comment changes based on feedback from v5 on the mailing list
- this version included many updates to add a cmdline argument for sector-size, this support was added upstream independently
- a patch to add a ufs bbclass was dropped
- move the canned-wks files before removing wic so the histories of these files are preserved
Changes in v5:
- rebased on latest master
- split commits out to make them easier to review
- a patch to add sector-size on the cmdline was added
Changes in v4:
- deprecate but allow sector-size to be set via WIC_SECTOR_SIZE
- change partition type of sample wks file to gpt
Changes in v3:
- squash all commits into one
- take another crack at trying to fix a warning if wic is not being used
Changes in v2:
- try to fix a build warning when not using wic
- add Mark as co-author of ufs class
- add wic utility to the list of native dependencies of image_types_wic.bbclass
Trevor Woerner (5):
wic: add recipe
oe-selftest/cases/wic.py: update WicTestCase
selftest/cases/wic.py: remove test_sparse_copy
wic: move canned *wks files
wic: remove to standalone repository
meta/classes-recipe/image_types_wic.bbclass | 4 +-
meta/conf/distro/include/maintainers.inc | 3 +-
meta/lib/oeqa/selftest/cases/wic.py | 64 +-
meta/recipes-core/meta/wic-tools.bb | 1 +
meta/recipes-support/wic/wic_0.3.0.bb | 18 +
.../canned-wks => meta/wic}/common.wks.inc | 0
.../wic}/directdisk-bootloader-config.cfg | 0
.../wic}/directdisk-bootloader-config.wks | 0
.../wic}/directdisk-gpt.wks | 0
.../wic}/directdisk-multi-rootfs.wks | 0
.../canned-wks => meta/wic}/directdisk.wks | 0
.../wic}/efi-bootdisk.wks.in | 0
.../wic}/efi-uki-bootdisk.wks.in | 0
.../wic/canned-wks => meta/wic}/mkefidisk.wks | 0
.../canned-wks => meta/wic}/mkhybridiso.wks | 0
.../canned-wks => meta/wic}/qemuloongarch.wks | 0
.../wic/canned-wks => meta/wic}/qemuriscv.wks | 0
.../wic}/qemux86-directdisk.wks | 0
.../wic}/sdimage-bootpart.wks | 0
.../wic}/systemd-bootdisk.wks | 0
scripts/lib/wic/__init__.py | 10 -
scripts/lib/wic/engine.py | 718 ----------
scripts/lib/wic/filemap.py | 590 --------
scripts/lib/wic/help.py | 1197 -----------------
scripts/lib/wic/ksparser.py | 322 -----
scripts/lib/wic/misc.py | 272 ----
scripts/lib/wic/partition.py | 615 ---------
scripts/lib/wic/pluginbase.py | 144 --
scripts/lib/wic/plugins/imager/direct.py | 702 ----------
.../wic/plugins/source/bootimg_biosplusefi.py | 213 ---
scripts/lib/wic/plugins/source/bootimg_efi.py | 436 ------
.../wic/plugins/source/bootimg_partition.py | 162 ---
.../lib/wic/plugins/source/bootimg_pcbios.py | 484 -------
scripts/lib/wic/plugins/source/empty.py | 89 --
.../lib/wic/plugins/source/extra_partition.py | 146 --
.../wic/plugins/source/isoimage_isohybrid.py | 464 -------
scripts/lib/wic/plugins/source/rawcopy.py | 115 --
scripts/lib/wic/plugins/source/rootfs.py | 236 ----
scripts/wic | 641 ---------
39 files changed, 58 insertions(+), 7588 deletions(-)
create mode 100644 meta/recipes-support/wic/wic_0.3.0.bb
rename {scripts/lib/wic/canned-wks => meta/wic}/common.wks.inc (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/directdisk-bootloader-config.cfg (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/directdisk-bootloader-config.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/directdisk-gpt.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/directdisk-multi-rootfs.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/directdisk.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/efi-bootdisk.wks.in (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/efi-uki-bootdisk.wks.in (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/mkefidisk.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/mkhybridiso.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/qemuloongarch.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/qemuriscv.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/qemux86-directdisk.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/sdimage-bootpart.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/systemd-bootdisk.wks (100%)
delete mode 100644 scripts/lib/wic/__init__.py
delete mode 100644 scripts/lib/wic/engine.py
delete mode 100644 scripts/lib/wic/filemap.py
delete mode 100644 scripts/lib/wic/help.py
delete mode 100644 scripts/lib/wic/ksparser.py
delete mode 100644 scripts/lib/wic/misc.py
delete mode 100644 scripts/lib/wic/partition.py
delete mode 100644 scripts/lib/wic/pluginbase.py
delete mode 100644 scripts/lib/wic/plugins/imager/direct.py
delete mode 100644 scripts/lib/wic/plugins/source/bootimg_biosplusefi.py
delete mode 100644 scripts/lib/wic/plugins/source/bootimg_efi.py
delete mode 100644 scripts/lib/wic/plugins/source/bootimg_partition.py
delete mode 100644 scripts/lib/wic/plugins/source/bootimg_pcbios.py
delete mode 100644 scripts/lib/wic/plugins/source/empty.py
delete mode 100644 scripts/lib/wic/plugins/source/extra_partition.py
delete mode 100644 scripts/lib/wic/plugins/source/isoimage_isohybrid.py
delete mode 100644 scripts/lib/wic/plugins/source/rawcopy.py
delete mode 100644 scripts/lib/wic/plugins/source/rootfs.py
delete mode 100755 scripts/wicSignature · 1 line
2.51.0
PATCH v9 1/5 wic: add recipe
Add a recipe for the external wic utility from
https://git.yoctoproject.org/wic. Add wic to the list of native wic-tools, and make it a dependency of the image bbclass for wic images.
Add myself as maintainer for this new recipe, as well as the wic-tools recipe (which did not have a maintainer).
NOTE: this patch is in preparation for removing wic from oe-core
the wic oe-selftests work fine with this patch being added now
---
changes in v9:
- updated SRCREV as required (the standalone wic repository has different commits)
- updated commit message to indicate the change of maintainership
changes in v8:
- (skipped, the patch to add sector-size cmdline arg was upstreamed)
changes in v7:
- bump SRCREV to include a work-around for wic.Wic2.test_expand_mbr_image
- changelog:
exec_cmd(): allow multiple success codes
changes in v6:
- the SRCREV was updated since any changes/revisions before the removal need to be reflected in the repository
changes in v5:
- rebase with master
- split patchset back out into smaller patches
- do not add TOPDIR and WIC_SECTOR_SIZE to image_types_wic.bbclass if a user wants them, they can add them to WICVARS in their config
changes in v4:
- (skipped)
changes in v3:
- squashed into one large patch
changes in v2:
- add wic as a native dependency of the image_types_wic.bbclass
--- meta/classes-recipe/image_types_wic.bbclass | 4 ++-- meta/conf/distro/include/maintainers.inc | 3 ++- meta/recipes-core/meta/wic-tools.bb | 1 + meta/recipes-support/wic/wic_0.3.0.bb | 18 ++++++++++++++++++ 4 files changed, 23 insertions(+), 3 deletions(-) create mode 100644 meta/recipes-support/wic/wic_0.3.0.bb
meta/classes-recipe/image_types_wic.bbclass +2 -2
@@ -111,14 +111,14 @@ do_image_wic[cleandirs] = "${WORKDIR}/build-wic" USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}" WKS_FILE_CHECKSUM = "${@wks_checksums(d.getVar('WKS_FILES').split(), d.getVar('WKS_SEARCH_PATH')) if '${USING_WIC}' else ''}" do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}" -do_image_wic[depends] += "${@' '.join('%s-native:do_populate_sysroot' % r for r in ('parted', 'gptfdisk', 'dosfstools', 'mtools'))}" +do_image_wic[depends] += "${@' '.join('%s-native:do_populate_sysroot' % r for r in ('wic', 'parted', 'gptfdisk', 'dosfstools', 'mtools'))}" # We ensure all artfacts are deployed (e.g virtual/bootloader) do_image_wic[recrdeptask] += "do_deploy" do_image_wic[deptask] += "do_image_complete" WKS_FILE_DEPENDS_DEFAULT = '${@bb.utils.contains_any("BUILD_ARCH", [ 'x86_64', 'i686' ], "syslinux-native", "",d)}' -WKS_FILE_DEPENDS_DEFAULT += "bmaptool-native cdrtools-native btrfs-tools-native squashfs-tools-native e2fsprogs-native erofs-utils-native" +WKS_FILE_DEPENDS_DEFAULT += "wic-native bmaptool-native cdrtools-native btrfs-tools-native squashfs-tools-native e2fsprogs-native erofs-utils-native" # Unified kernel images need objcopy WKS_FILE_DEPENDS_DEFAULT += "virtual/cross-binutils" WKS_FILE_DEPENDS_BOOTLOADERS = ""
meta/conf/distro/include/maintainers.inc +2 -1
@@ -892,7 +892,8 @@ RECIPE_MAINTAINER:pn-weston = "Denys Dmytriyenko <[email protected]>" RECIPE_MAINTAINER:pn-weston-init = "Denys Dmytriyenko <[email protected]>" RECIPE_MAINTAINER:pn-wget = "Yi Zhao <[email protected]>" RECIPE_MAINTAINER:pn-which = "Unassigned <[email protected]>" -RECIPE_MAINTAINER:pn-wic-tools = "Unassigned <[email protected]>" +RECIPE_MAINTAINER:pn-wic = "Trevor Woerner <[email protected]>" +RECIPE_MAINTAINER:pn-wic-tools = "Trevor Woerner <[email protected]>" RECIPE_MAINTAINER:pn-wireless-regdb = "Unassigned <[email protected]>" RECIPE_MAINTAINER:pn-wpa-supplicant = "Unassigned <[email protected]>" RECIPE_MAINTAINER:pn-wpebackend-fdo = "Unassigned <[email protected]>"
meta/recipes-core/meta/wic-tools.bb +1 -0
@@ -3,6 +3,7 @@ SUMMARY = "A meta recipe to build native tools used by wic." LICENSE = "MIT" DEPENDS = "\ + wic-native \ parted-native gptfdisk-native dosfstools-native \ mtools-native bmaptool-native grub-native cdrtools-native \ btrfs-tools-native squashfs-tools-native pseudo-native \
meta/recipes-support/wic/wic_0.3.0.bb new file +18 -0
@@ -0,0 +1,18 @@ +SUMMARY = "OpenEmbedded Image Creator (wic) standalone CLI" +HOMEPAGE = "https://git.yoctoproject.org/wic" +LICENSE = "GPL-2.0-only" +LIC_FILES_CHKSUM = "file://LICENSE;md5=4ee23c52855c222cba72583d301d2338" + +SRC_URI = "git://git.yoctoproject.org/wic.git;branch=master;protocol=https" +SRCREV = "79e2b06a1f9c97decdaf44f0806dbf3cc1e86bf9" + +inherit python_hatchling + +RDEPENDS:${PN} += " \ + python3-core \ + python3-json \ + python3-logging \ + python3-misc \ + " + +BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.51.0
Re: [OE-core] [PATCH v9 1/5] wic: add recipe
Quoted 106 lines · Trevor Woerner via lists.openembedded.org
Add a recipe for the external wic utility from
https://git.yoctoproject.org/wic. Add wic to the list of native wic-tools, and make it a dependency of the image bbclass for wic images.Add myself as maintainer for this new recipe, as well as the wic-tools recipe (which did not have a maintainer).
NOTE: this patch is in preparation for removing wic from oe-core
the wic oe-selftests work fine with this patch being added nowReviewed-by Bruce Ashfield <[email protected]>Reviewed-by Mark Hatle <[email protected]>Signed-off-by Trevor Woerner <[email protected]>---
changes in v9:
- updated SRCREV as required (the standalone wic repository has different commits)
- updated commit message to indicate the change of maintainershipchanges in v8:
- (skipped, the patch to add sector-size cmdline arg was upstreamed)changes in v7:
- bump SRCREV to include a work-around for wic.Wic2.test_expand_mbr_image
- changelog:
exec_cmd(): allow multiple success codeschanges in v6:
- the SRCREV was updated since any changes/revisions before the removal need to be reflected in the repositorychanges in v5:
- rebase with master
- split patchset back out into smaller patches
- do not add TOPDIR andWIC_SECTOR_SIZEto image_types_wic.bbclass if a user wants them, they can add them to WICVARS in their configchanges in v4:
- (skipped)changes in v3:
- squashed into one large patchchanges in v2:
- add wic as a native dependency of the image_types_wic.bbclass--- meta/classes-recipe/image_types_wic.bbclass | 4 ++-- meta/conf/distro/include/maintainers.inc | 3 ++- meta/recipes-core/meta/wic-tools.bb | 1 + meta/recipes-support/wic/wic_0.3.0.bb | 18 ++++++++++++++++++ 4 files changed, 23 insertions(+), 3 deletions(-) create mode 100644 meta/recipes-support/wic/wic_0.3.0.bbmeta/classes-recipe/image_types_wic.bbclass +2
-2@@ -111,14 +111,14 @@ do_image_wic[cleandirs] = "${WORKDIR}/build-wic" USING_WIC = "${@bb.utils.contains_any('IMAGE_FSTYPES', 'wic ' + ' '.join('wic.%s' % c for c in '${CONVERSIONTYPES}'.split()), '1', '', d)}" WKS_FILE_CHECKSUM = "${@wks_checksums(d.getVar('WKS_FILES').split(), d.getVar('WKS_SEARCH_PATH')) if '${USING_WIC}' else ''}" do_image_wic[file-checksums] += "${WKS_FILE_CHECKSUM}" -do_image_wic[depends] += "${@' '.join('%s-native:do_populate_sysroot' % r for r in ('parted', 'gptfdisk', 'dosfstools', 'mtools'))}" +do_image_wic[depends] += "${@' '.join('%s-native:do_populate_sysroot' % r for r in ('wic', 'parted', 'gptfdisk', 'dosfstools', 'mtools'))}" # We ensure all artfacts are deployed (e.g virtual/bootloader) do_image_wic[recrdeptask] += "do_deploy" do_image_wic[deptask] += "do_image_complete" WKS_FILE_DEPENDS_DEFAULT = '${@bb.utils.contains_any("BUILD_ARCH", [ 'x86_64', 'i686' ], "syslinux-native", "",d)}' -WKS_FILE_DEPENDS_DEFAULT += "bmaptool-native cdrtools-native btrfs-tools-native squashfs-tools-native e2fsprogs-native erofs-utils-native" +WKS_FILE_DEPENDS_DEFAULT += "wic-native bmaptool-native cdrtools-native btrfs-tools-native squashfs-tools-native e2fsprogs-native erofs-utils-native" # Unified kernel images need objcopy WKS_FILE_DEPENDS_DEFAULT += "virtual/cross-binutils" WKS_FILE_DEPENDS_BOOTLOADERS = ""meta/conf/distro/include/maintainers.inc +2
-1@@ -892,7 +892,8 @@ RECIPE_MAINTAINER:pn-weston = "Denys Dmytriyenko <[email protected]>" RECIPE_MAINTAINER:pn-weston-init = "Denys Dmytriyenko <[email protected]>" RECIPE_MAINTAINER:pn-wget = "Yi Zhao <[email protected]>" RECIPE_MAINTAINER:pn-which = "Unassigned <[email protected]>" -RECIPE_MAINTAINER:pn-wic-tools = "Unassigned <[email protected]>" +RECIPE_MAINTAINER:pn-wic = "Trevor Woerner <[email protected]>" +RECIPE_MAINTAINER:pn-wic-tools = "Trevor Woerner <[email protected]>" RECIPE_MAINTAINER:pn-wireless-regdb = "Unassigned <[email protected]>" RECIPE_MAINTAINER:pn-wpa-supplicant = "Unassigned <[email protected]>" RECIPE_MAINTAINER:pn-wpebackend-fdo = "Unassigned <[email protected]>"meta/recipes-core/meta/wic-tools.bb +1
-0@@ -3,6 +3,7 @@ SUMMARY = "A meta recipe to build native tools used by wic." LICENSE = "MIT" DEPENDS = "\ + wic-native \ parted-native gptfdisk-native dosfstools-native \ mtools-native bmaptool-native grub-native cdrtools-native \ btrfs-tools-native squashfs-tools-native pseudo-native \meta/recipes-support/wic/wic_0.3.0.bb new file +0
-0
meta/recipes-support/wic/wic_0.3.0.bb line 6
Quoted 6 lines
meta/recipes-support/wic/wic_0.3.0.bb new file +6
-0@@ -0,0 +1,18 @@ +SUMMARY = "OpenEmbedded Image Creator (wic) standalone CLI" +HOMEPAGE = "https://git.yoctoproject.org/wic" +LICENSE = "GPL-2.0-only" +LIC_FILES_CHKSUM = "file://LICENSE;md5=4ee23c52855c222cba72583d301d2338" + +SRC_URI = "git://git.yoctoproject.org/wic.git;branch=master;protocol=https"
(To repeat what was said during tech/review call)
Git tags in the repo for the wic versions and a ";tag=${PV}" appended to SRC_URI would be nice (at least for the 0.3.0).
Thanks!
Quoted 12 lines
meta/recipes-support/wic/wic_0.3.0.bb new file +12
-0+SRCREV = "79e2b06a1f9c97decdaf44f0806dbf3cc1e86bf9" + +inherit python_hatchling + +RDEPENDS:${PN} += " \ + python3-core \ + python3-json \ + python3-logging \ + python3-misc \ + " + +BBCLASSEXTEND = "native nativesdk"
Signature · 2 lines
Yoann Congal Smile ECS
PATCH v9 2/5 oe-selftest/cases/wic.py: update WicTestCase
In reply to Trevor WoernerThe wic oe-selftest defines its own class (WicTestCase) for handling setup, teardown, and various other pieces needed to run the individual wic oe-selftests. As part of oe-core, the wic.CLITests do not need setup and teardown. However, once wic is no longer part of oe-core, the oe-selftests will need to know where to find wic that comes from a recipe. Update PATH so wic will be available.
NOTE: this patch is in preparation for removing wic from oe-core
the wic oe-selftests work fine with this patch being added now
AI-Generated: codex/gpt-5.4 (high)
---
changes in v9:
- provide a smaller change to the wic test bbclass in order to allow the oe-selftests to work, this patch only updates the PATH
changes in v8:
- (skipped, sector-size cmdline arg upstreamed)
changes in v7:
- (none)
changes in v6:
- (none)
changes in v5:
- rebase with master
- split patch set back out into smaller patches
changes in v4:
- (skipped)
changes in v3:
- squashed into one large patch
changes in v2:
- (none)
--- meta/lib/oeqa/selftest/cases/wic.py | 36 ++++++++++++++++++++++++++++- 1 file changed, 35 insertions(+), 1 deletion(-)
meta/lib/oeqa/selftest/cases/wic.py +35 -1
@@ -57,10 +57,12 @@ class WicTestCase(OESelftestTestCase): image_is_ready = False wicenv_cache = {} + wic_bindir = None def setUpLocal(self): """This code is executed before each test method.""" self.resultdir = os.path.join(self.builddir, "wic-tmp") + self._old_path = os.environ.get('PATH') super(WicTestCase, self).setUpLocal() # Do this here instead of in setUpClass as the base setUp does some
@@ -72,13 +74,45 @@ class WicTestCase(OESelftestTestCase): bitbake('wic-tools core-image-minimal core-image-minimal-mtdutils') WicTestCase.image_is_ready = True + + os.environ['PATH'] = self._get_wic_path() rmtree(self.resultdir, ignore_errors=True) def tearDownLocal(self): """Remove resultdir as it may contain images.""" + if self._old_path is None: + os.environ.pop('PATH', None) + else: + os.environ['PATH'] = self._old_path rmtree(self.resultdir, ignore_errors=True) super(WicTestCase, self).tearDownLocal() + def _get_wic_path(self): + if WicTestCase.wic_bindir is None: + search_paths = [ + os.path.join(self.td['COREBASE'], 'scripts'), + os.path.join(get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools'), 'usr', 'bin'), + ] + + for bindir in search_paths: + if os.path.exists(os.path.join(bindir, 'wic')): + WicTestCase.wic_bindir = bindir + break + + if WicTestCase.wic_bindir is None: + self.fail("Unable to find the wic binary in %s" % ', '.join(search_paths)) + + path_entries = [] + for path_group in ( + [WicTestCase.wic_bindir], + (get_bb_var("PATH", "wic-tools") or '').split(':'), + (self._old_path or '').split(':')): + for entry in path_group: + if entry and entry not in path_entries: + path_entries.append(entry) + + return ':'.join(path_entries) + def _get_image_env_path(self, image): """Generate and obtain the path to <image>.env""" if image not in WicTestCase.wicenv_cache:
@@ -88,7 +122,7 @@ class WicTestCase(OESelftestTestCase): WicTestCase.wicenv_cache[image] = os.path.join(stdir, machine, 'imgdata') return WicTestCase.wicenv_cache[image] -class CLITests(OESelftestTestCase): +class CLITests(WicTestCase): def test_version(self): """Test wic --version""" runCmd('wic --version')
Signature · 1 line
2.51.0
PATCH v9 3/5 selftest/cases/wic.py: remove test_sparse_copy
In reply to Trevor WoernerDelete the wic.Wic2.test_sparse_copy test since it tests code that is internal to wic itself. Once wic is removed from oe-core it will not be possible to test this function from this layer.
NOTE: this patch is in preparation for removing wic from oe-core
the wic oe-selftests work fine with this patch being added now
---
changes in v9:
- in previous patchsets this was known as patch 9/9 "oe-selftest/cases/wic.py: oe-selftest -r wic.Wic2 -> PASS"
- previously this patch also included many modifications to the wic oe-selftests to add a "--vars <envfile>" option; those are no longer needed since wic will use bitbake to get variables if bitbake is available on the PATH
changes in v8:
- (skipped, the only patch in v8 was to upstream the sector-size cmdline option)
changes in v7:
- none
changes in v6:
- removed extraopts options that specify sector-size values for dos and ext* partition types
changes in v5:
- rebase with master
- re-split up into multiple, smaller patches
- now that the sector-size re-implementation is ahead of this patch, various wic subcommands (ls, cp, rm) no longer need a --vars file
- combine qemu and non-qemu patches together (were separate patches)
v4: skipped to align/combine with other patch set
changes in v3:
- squashed into 1 large commit
changes in v2:
- none
--- meta/lib/oeqa/selftest/cases/wic.py | 28 ---------------------------- 1 file changed, 28 deletions(-)
meta/lib/oeqa/selftest/cases/wic.py +0 -28
@@ -1894,34 +1894,6 @@ INITRAMFS_IMAGE = "core-image-initramfs-boot" self.remove_config(config) self.assertEqual(1, len(glob(os.path.join(self.resultdir, "sdimage-bootpart-*direct")))) - def test_sparse_copy(self): - """Test sparse_copy with FIEMAP and SEEK_HOLE filemap APIs""" - libpath = os.path.join(self.td['COREBASE'], 'scripts', 'lib', 'wic') - sys.path.insert(0, libpath) - from filemap import FilemapFiemap, FilemapSeek, sparse_copy, ErrorNotSupp - with NamedTemporaryFile("w", suffix=".wic-sparse") as sparse: - src_name = sparse.name - src_size = 1024 * 10 - sparse.truncate(src_size) - # write one byte to the file - with open(src_name, 'r+b') as sfile: - sfile.seek(1024 * 4) - sfile.write(b'\x00') - dest = sparse.name + '.out' - # copy src file to dest using different filemap APIs - for api in (FilemapFiemap, FilemapSeek, None): - if os.path.exists(dest): - os.unlink(dest) - try: - sparse_copy(sparse.name, dest, api=api) - except ErrorNotSupp: - continue # skip unsupported API - dest_stat = os.stat(dest) - self.assertEqual(dest_stat.st_size, src_size) - # 8 blocks is 4K (physical sector size) - self.assertEqual(dest_stat.st_blocks, 8) - os.unlink(dest) - def test_mkfs_extraopts(self): """Test wks option --mkfs-extraopts for empty and not empty partitions""" img = 'core-image-minimal'
Signature · 1 line
2.51.0
PATCH v9 4/5 wic: move canned *wks files
In reply to Trevor WoernerWhen "wic create ..." is invoked with a bare *wks name (i.e. without the .wks extension), wic calls engine.py:find_canned_images() to find the fully qualified *wks file. This function searches every directory formed by:
- permutating all BBLAYERS with /wic
- permutating all BBLAYERS with /scripts/lib/wic/canned-wks
- checking <scripts_path>/lib/wic/canned-wks
Where <scripts_path> is the directory containing the wic program.
When wic is part of oe-core, the last search path succeeds in finding the canned *wks files in <topdir>/scripts/lib/wic/canned-wks (since the wic program is found in oe-core's <topdir>/scripts directory, and <topdir>/scripts is not a BBLAYER).
However, once wic is removed from oe-core, this algorithm will not find these bare *wks files in any of the above-mentioned search paths since the oe-core layer will no longer be the home of the wic program, and the canned *wks files are not located in any directory relative to BBLAYERS.
Since these *wks files are specific to oe-core's meta layer, they should stay with this layer. Therefore move the *wks files so they exist in one of the locations searched relative to oe-core/meta's BBLAYERS.
NOTE: this patch is in preparation for removing wic from oe-core
the wic oe-selftests work fine with this patch being added now
---
changes in v9:
- continue to search scripts/lib/wic/canned-wks for *wks files
- commit message updated to reflect this change and to more clearly explain how wic finds *wks files and why these files are moved
changes in v8:
- (v8 was sending sector-size cmdline patch)
changes in v7:
- (none)
changes in v6:
- re-ordered so that this patch occurs before wic is removed, thus saving the commit history
changes in v5:
- rebase with master
- split patchset out again into individual, small patches
changes in v4:
- (skipped)
changes in v3:
- squashed into 1 large commit
changes in v2:
- (none)
---
{scripts/lib/wic/canned-wks => meta/wic}/common.wks.inc | 0
.../wic/canned-wks => meta/wic}/directdisk-bootloader-config.cfg | 0
.../wic/canned-wks => meta/wic}/directdisk-bootloader-config.wks | 0
{scripts/lib/wic/canned-wks => meta/wic}/directdisk-gpt.wks | 0
.../lib/wic/canned-wks => meta/wic}/directdisk-multi-rootfs.wks | 0
{scripts/lib/wic/canned-wks => meta/wic}/directdisk.wks | 0
{scripts/lib/wic/canned-wks => meta/wic}/efi-bootdisk.wks.in | 0
{scripts/lib/wic/canned-wks => meta/wic}/efi-uki-bootdisk.wks.in | 0
{scripts/lib/wic/canned-wks => meta/wic}/mkefidisk.wks | 0
{scripts/lib/wic/canned-wks => meta/wic}/mkhybridiso.wks | 0
{scripts/lib/wic/canned-wks => meta/wic}/qemuloongarch.wks | 0
{scripts/lib/wic/canned-wks => meta/wic}/qemuriscv.wks | 0
{scripts/lib/wic/canned-wks => meta/wic}/qemux86-directdisk.wks | 0
{scripts/lib/wic/canned-wks => meta/wic}/sdimage-bootpart.wks | 0
{scripts/lib/wic/canned-wks => meta/wic}/systemd-bootdisk.wks | 0
15 files changed, 0 insertions(+), 0 deletions(-)
rename {scripts/lib/wic/canned-wks => meta/wic}/common.wks.inc (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/directdisk-bootloader-config.cfg (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/directdisk-bootloader-config.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/directdisk-gpt.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/directdisk-multi-rootfs.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/directdisk.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/efi-bootdisk.wks.in (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/efi-uki-bootdisk.wks.in (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/mkefidisk.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/mkhybridiso.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/qemuloongarch.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/qemuriscv.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/qemux86-directdisk.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/sdimage-bootpart.wks (100%)
rename {scripts/lib/wic/canned-wks => meta/wic}/systemd-bootdisk.wks (100%)meta/wic/common.wks.inc renamed from scripts/lib/wic/canned-wks/common.wks.inc +0 -0
meta/wic/directdisk-bootloader-config.cfg renamed from scripts/lib/wic/canned-wks/directdisk-bootloader-config.cfg +0 -0
meta/wic/directdisk-bootloader-config.wks renamed from scripts/lib/wic/canned-wks/directdisk-bootloader-config.wks +0 -0
meta/wic/directdisk-gpt.wks renamed from scripts/lib/wic/canned-wks/directdisk-gpt.wks +0 -0
meta/wic/directdisk-multi-rootfs.wks renamed from scripts/lib/wic/canned-wks/directdisk-multi-rootfs.wks +0 -0
meta/wic/directdisk.wks renamed from scripts/lib/wic/canned-wks/directdisk.wks +0 -0
meta/wic/efi-bootdisk.wks.in renamed from scripts/lib/wic/canned-wks/efi-bootdisk.wks.in +0 -0
meta/wic/efi-uki-bootdisk.wks.in renamed from scripts/lib/wic/canned-wks/efi-uki-bootdisk.wks.in +0 -0
meta/wic/mkefidisk.wks renamed from scripts/lib/wic/canned-wks/mkefidisk.wks +0 -0
meta/wic/mkhybridiso.wks renamed from scripts/lib/wic/canned-wks/mkhybridiso.wks +0 -0
meta/wic/qemuloongarch.wks renamed from scripts/lib/wic/canned-wks/qemuloongarch.wks +0 -0
meta/wic/qemuriscv.wks renamed from scripts/lib/wic/canned-wks/qemuriscv.wks +0 -0
meta/wic/qemux86-directdisk.wks renamed from scripts/lib/wic/canned-wks/qemux86-directdisk.wks +0 -0
meta/wic/sdimage-bootpart.wks renamed from scripts/lib/wic/canned-wks/sdimage-bootpart.wks +0 -0
meta/wic/systemd-bootdisk.wks renamed from scripts/lib/wic/canned-wks/systemd-bootdisk.wks +0 -0
Signature · 1 line
2.51.0
Re: [OE-core] [PATCH v9 4/5] wic: move canned *wks files
Quoted 7 lines · Trevor Woerner via lists.openembedded.org
When "wic create ..." is invoked with a bare *wks name (i.e. without the
.wksextension), wic calls engine.py:find_canned_images() to find the fully qualified *wks file. This function searches every directory formed by:
- permutating all BBLAYERS with/wic
- permutating all BBLAYERS with/scripts/lib/wic/canned-wks
- checking<scripts_path>/lib/wic/canned-wks
Where<scripts_path>is the directory containing the wic program.
It doesn't. I just looked at the code and it uses BBPATH. That can be similar to BBLAYERS but it is different and the commit messages really should refer to BBPATH.
Quoted 9 lines
When wic is part of oe-core, the last search path succeeds in finding the canned *wks files in
<topdir>/scripts/lib/wic/canned-wks(since the wic program is found in oe-core's<topdir>/scriptsdirectory, and<topdir>/scriptsis not a BBLAYER).However, once wic is removed from oe-core, this algorithm will not find these bare *wks files in any of the above-mentioned search paths since the oe-core layer will no longer be the home of the wic program, and the canned *wks files are not located in any directory relative to BBLAYERS.
I'm a bit confused by this reasoning. Are you saying that wic will no longer search BBPATH (or WKS_SEARCH_PATH)?
Quoted 3 lines
Since these *wks files are specific to oe-core's meta layer, they should stay with this layer. Therefore move the *wks files so they exist in one of the locations searched relative to oe-core/meta's BBLAYERS.
I don't understand why these need to move given WKS_SEARCH_PATH remains unchanged, unless wic is going to ignore WKS_SEARCH_PATH going forward?
Cheers,
Richard
Re: [OE-core] [PATCH v9 4/5] wic: move canned *wks files
Quoted 9 lines · Richard Purdie
Quoted 7 lines · Trevor Woerner via lists.openembedded.org
When "wic create ..." is invoked with a bare *wks name (i.e. without the
.wksextension), wic calls engine.py:find_canned_images() to find the fully qualified *wks file. This function searches every directory formed by:
- permutating all BBLAYERS with/wic
- permutating all BBLAYERS with/scripts/lib/wic/canned-wks
- checking<scripts_path>/lib/wic/canned-wks
Where<scripts_path>is the directory containing the wic program.
Quoted 3 lines
It doesn't. I just looked at the code and it uses BBPATH. That can be similar to BBLAYERS but it is different and the commit messages really should refer to BBPATH.
In the oe-selftest wic tests, most of the "wic create ..." commands are called with bare *wks files (i.e. *wks files without the .wks extension). When this happens, as "wic create..." is called, it starts by running the code found here:
https://git.openembedded.org/openembedded-core/tree/scripts/wic#n206
which calls:
https://git.openembedded.org/openembedded-core/tree/scripts/lib/wic/engine.py#n46
which follows the algorithm that I have described above.
So for the oe-selftests to pass I wanted to move them.
Quoted 13 lines
Quoted 9 lines
When wic is part of oe-core, the last search path succeeds in finding the canned *wks files in
<topdir>/scripts/lib/wic/canned-wks(since the wic program is found in oe-core's<topdir>/scriptsdirectory, and<topdir>/scriptsis not a BBLAYER).However, once wic is removed from oe-core, this algorithm will not find these bare *wks files in any of the above-mentioned search paths since the oe-core layer will no longer be the home of the wic program, and the canned *wks files are not located in any directory relative to BBLAYERS.
I'm a bit confused by this reasoning. Are you saying that wic will no longer search BBPATH (or
WKS_SEARCH_PATH)?
Quoted 6 lines
Quoted 3 lines
Since these *wks files are specific to oe-core's meta layer, they should stay with this layer. Therefore move the *wks files so they exist in one of the locations searched relative to oe-core/meta's BBLAYERS.
I don't understand why these need to move given
WKS_SEARCH_PATHremains unchanged, unless wic is going to ignoreWKS_SEARCH_PATHgoing forward?
As an independent tool, these *wks files are rather oe-core-specific, so I thought they should stay with the oe-core/meta layer, the same way that, say, raspberry pi-specific *wks files stay in the raspberry pi layer.
My first thought was to remove them from the standalone wic repository altogether, but then decided I could keep them as examples.
I thought it would be dangerous to leave them in the wic source tree where the above algorithm could find them. If I leave them in src/wic/canned-wks, someone uses the tool and creates a *wks file with a similar name to one of the canned-wks files, but they have a typo and the canned one gets used instead of theirs...
...maybe I'm overthinking it?
Re: [OE-core] [PATCH v9 4/5] wic: move canned *wks files
Quoted 21 lines · Trevor Woerner
Quoted 9 lines · Richard Purdie
Quoted 7 lines · Trevor Woerner via lists.openembedded.org
When "wic create ..." is invoked with a bare *wks name (i.e. without the
.wksextension), wic calls engine.py:find_canned_images() to find the fully qualified *wks file. This function searches every directory formed by:
- permutating all BBLAYERS with/wic
- permutating all BBLAYERS with/scripts/lib/wic/canned-wks
- checking<scripts_path>/lib/wic/canned-wks
Where<scripts_path>is the directory containing the wic program.Quoted 3 lines
It doesn't. I just looked at the code and it uses BBPATH. That can be similar to BBLAYERS but it is different and the commit messages really should refer to BBPATH.
In the oe-selftest wic tests, most of the "wic create ..." commands are called with bare *wks files (i.e. *wks files without the
.wksextension). When this happens, as "wic create..." is called, it starts by running the code found here:https://git.openembedded.org/openembedded-core/tree/scripts/wic#n206
Quoted 7 lines
which calls:
https://git.openembedded.org/openembedded-core/tree/scripts/lib/wic/engine.py#n46
which follows the algorithm that I have described above.
So for the oe-selftests to pass I wanted to move them.
You are right, it is using BBLAYERS. This is very confusing since the wks files are using BBPATH, which is similar but subtly different.
Is there any reason it can't use BBPATH to match?
I agree the files need to move but I'd like them to end up in meta/files/wic, not meta/wic. Whilst it complicates the migration a little, I think it would be best to move them once and get them into the right place.
The questions are then firstly, how to do that simply, then secondly, how to clean up the rest of this (and when to do that).
Adding files/wic into the search path is trivial. Deprecating the existing search paths with appropriate warnings is a bit trickier and there is the question of when we should do that, before or after the current release. "after" would seem more obvious, but it will mean two different behaviours before and after release and with a standalone tool, this does become harder.
Quoted 11 lines
Quoted 2 lines
I don't understand why these need to move given
WKS_SEARCH_PATHremains unchanged, unless wic is going to ignoreWKS_SEARCH_PATHgoing forward?As an independent tool, these *wks files are rather oe-core-specific, so I thought they should stay with the oe-core/meta layer, the same way that, say, raspberry pi-specific *wks files stay in the raspberry pi layer.
My first thought was to remove them from the standalone wic repository altogether, but then decided I could keep them as examples.
Quoted 7 lines
I thought it would be dangerous to leave them in the wic source tree where the above algorithm could find them. If I leave them in src/wic/canned-wks, someone uses the tool and creates a *wks file with a similar name to one of the canned-wks files, but they have a typo and the canned one gets used instead of theirs...
...maybe I'm overthinking it?
It is definitely dangerous to leave them in the search paths so I agree with that. The files do belong in OE-Core too. It is still a little dangerous having files lying around which have the same names as the ones in core since this kind of duplication still can confuse people when they edit the wrong file. We have bigger issues I guess but some renaming of the examples might help.
Cheers,
Richard
Re: [OE-core] [PATCH v9 4/5] wic: move canned *wks files
Quoted 43 lines · Richard Purdie
Quoted 21 lines · Trevor Woerner
Quoted 9 lines · Richard Purdie
Quoted 7 lines · Trevor Woerner via lists.openembedded.org
When "wic create ..." is invoked with a bare *wks name (i.e. without the
.wksextension), wic calls engine.py:find_canned_images() to find the fully qualified *wks file. This function searches every directory formed by:
- permutating all BBLAYERS with/wic
- permutating all BBLAYERS with/scripts/lib/wic/canned-wks
- checking<scripts_path>/lib/wic/canned-wks
Where<scripts_path>is the directory containing the wic program.Quoted 3 lines
It doesn't. I just looked at the code and it uses BBPATH. That can be similar to BBLAYERS but it is different and the commit messages really should refer to BBPATH.
In the oe-selftest wic tests, most of the "wic create ..." commands are called with bare *wks files (i.e. *wks files without the
.wksextension). When this happens, as "wic create..." is called, it starts by running the code found here:https://git.openembedded.org/openembedded-core/tree/scripts/wic#n206
Quoted 7 lines
which calls:
https://git.openembedded.org/openembedded-core/tree/scripts/lib/wic/engine.py#n46
which follows the algorithm that I have described above.
So for the oe-selftests to pass I wanted to move them.
You are right, it is using BBLAYERS. This is very confusing since the wks files are using BBPATH, which is similar but subtly different.
Is there any reason it can't use BBPATH to match?
I agree the files need to move but I'd like them to end up in meta/files/wic, not meta/wic. Whilst it complicates the migration a little, I think it would be best to move them once and get them into the right place.
The questions are then firstly, how to do that simply, then secondly, how to clean up the rest of this (and when to do that).
Quoted 6 lines
Adding files/wic into the search path is trivial. Deprecating the existing search paths with appropriate warnings is a bit trickier and there is the question of when we should do that, before or after the current release. "after" would seem more obvious, but it will mean two different behaviours before and after release and with a standalone tool, this does become harder.
I'm only guessing here, but it looks like someone wrote "wic create" to run two different ways: with the wks file specified with or without the ".wks" extension.
If ".wks" is not given (e.g. "wic create directdisk-gpt ...", which is what occurs in the oe-selftest tests, i.e. when wic is run independently without being part of a bitbake build) then wic invokes its own search algorithm to try to find the wks file using BBLAYERS and script_dir. In this case the wks file is specified without the ".wks" and no path information is given on the cmdline. It is up to wic to find the actual wks file itself.
If the ".wks" is included (which is what happens when wic is called as part of a bitbake build and therefore the image_types_wic.bbclass is used) then the class adds the ".wks" at the end, and also gives wic the path to the wks file which it finds using WKS_SEARCH_PATH which is based on BBPATH and COREBASE.
Assuming the persona of someone who wants to use this new, independent wic tool, and who knows nothing about The Yocto Project or bitbake, I think having wic look in "magical" places for the wks file would be hard to understand and surprising. As an independent tool I think the user should provide the path to the actual wks file (with the ".wks" extension) and if that file can't be found as specified on the cmdline, wic should simply fail with an error.
As part of making wic an independent tool, I think wic's code to search for a wks file if the ".wks" is not provided should be removed. This means that, as part of the transition, I would have to modify each "wic create" test in oe-core's oe-selftests. I'm fine with that. That could be done cleanly if I write a little function to find the wks file in the wic.py oe-selftest itself using the same logic as is used in the image_types_wic.bbclass.
To summarize: in my opinion, wic shouldn't have search logic. From wic's point of view the wks file should be specified on the cmdline in a way that wic will find the file the user wants to use (or not find it). The oe-selftests should be updated to use the same search logic from the image_types_wic.bbclass so that when it invokes "wic create" it is providing wic with the path to a wks file that has a ".wks" extension (which is how bitbake invokes wic) instead of specifying a bare wks file and hoping wic will find it.
I'm happy to add "files/wic" as another location for image_types_wic.bbclass to search for wks files.
Quoted 31 lines
Quoted 11 lines
Quoted 2 lines
I don't understand why these need to move given
WKS_SEARCH_PATHremains unchanged, unless wic is going to ignoreWKS_SEARCH_PATHgoing forward?As an independent tool, these *wks files are rather oe-core-specific, so I thought they should stay with the oe-core/meta layer, the same way that, say, raspberry pi-specific *wks files stay in the raspberry pi layer.
My first thought was to remove them from the standalone wic repository altogether, but then decided I could keep them as examples.
Quoted 7 lines
I thought it would be dangerous to leave them in the wic source tree where the above algorithm could find them. If I leave them in src/wic/canned-wks, someone uses the tool and creates a *wks file with a similar name to one of the canned-wks files, but they have a typo and the canned one gets used instead of theirs...
...maybe I'm overthinking it?
It is definitely dangerous to leave them in the search paths so I agree with that. The files do belong in OE-Core too. It is still a little dangerous having files lying around which have the same names as the ones in core since this kind of duplication still can confuse people when they edit the wrong file. We have bigger issues I guess but some renaming of the examples might help.
Cheers,
Richard
Re: [OE-core] [PATCH v9 4/5] wic: move canned *wks files
Quoted 85 lines · Trevor Woerner
Quoted 43 lines · Richard Purdie
Quoted 21 lines · Trevor Woerner
Quoted 9 lines · Richard Purdie
Quoted 7 lines · Trevor Woerner via lists.openembedded.org
When "wic create ..." is invoked with a bare *wks name (i.e. without the
.wksextension), wic calls engine.py:find_canned_images() to find the fully qualified *wks file. This function searches every directory formed by:
- permutating all BBLAYERS with/wic
- permutating all BBLAYERS with/scripts/lib/wic/canned-wks
- checking<scripts_path>/lib/wic/canned-wks
Where<scripts_path>is the directory containing the wic program.Quoted 3 lines
It doesn't. I just looked at the code and it uses BBPATH. That can be similar to BBLAYERS but it is different and the commit messages really should refer to BBPATH.
In the oe-selftest wic tests, most of the "wic create ..." commands are called with bare *wks files (i.e. *wks files without the
.wksextension). When this happens, as "wic create..." is called, it starts by running the code found here:https://git.openembedded.org/openembedded-core/tree/scripts/wic#n206
Quoted 7 lines
which calls:
https://git.openembedded.org/openembedded-core/tree/scripts/lib/wic/engine.py#n46
which follows the algorithm that I have described above.
So for the oe-selftests to pass I wanted to move them.
You are right, it is using BBLAYERS. This is very confusing since the wks files are using BBPATH, which is similar but subtly different.
Is there any reason it can't use BBPATH to match?
I agree the files need to move but I'd like them to end up in meta/files/wic, not meta/wic. Whilst it complicates the migration a little, I think it would be best to move them once and get them into the right place.
The questions are then firstly, how to do that simply, then secondly, how to clean up the rest of this (and when to do that).
Quoted 6 lines
Adding files/wic into the search path is trivial. Deprecating the existing search paths with appropriate warnings is a bit trickier and there is the question of when we should do that, before or after the current release. "after" would seem more obvious, but it will mean two different behaviours before and after release and with a standalone tool, this does become harder.
I'm only guessing here, but it looks like someone wrote "wic create" to run two different ways: with the wks file specified with or without the ".wks" extension.
If ".wks" is not given (e.g. "wic create directdisk-gpt ...", which is what occurs in the oe-selftest tests, i.e. when wic is run independently without being part of a bitbake build) then wic invokes its own search algorithm to try to find the wks file using BBLAYERS and
script_dir. In this case the wks file is specified without the ".wks" and no path information is given on the cmdline. It is up to wic to find the actual wks file itself.If the ".wks" is included (which is what happens when wic is called as part of a bitbake build and therefore the image_types_wic.bbclass is used) then the class adds the ".wks" at the end, and also gives wic the path to the wks file which it finds using
WKS_SEARCH_PATHwhich is based on BBPATH and COREBASE.Assuming the persona of someone who wants to use this new, independent wic tool, and who knows nothing about The Yocto Project or bitbake, I think having wic look in "magical" places for the wks file would be hard to understand and surprising. As an independent tool I think the user should provide the path to the actual wks file (with the ".wks" extension) and if that file can't be found as specified on the cmdline, wic should simply fail with an error.
As part of making wic an independent tool, I think wic's code to search for a wks file if the ".wks" is not provided should be removed. This means that, as part of the transition, I would have to modify each "wic create" test in oe-core's oe-selftests. I'm fine with that. That could be done cleanly if I write a little function to find the wks file in the
wic.pyoe-selftest itself using the same logic as is used in the image_types_wic.bbclass.
Quoted 8 lines
To summarize: in my opinion, wic shouldn't have search logic. From wic's point of view the wks file should be specified on the cmdline in a way that wic will find the file the user wants to use (or not find it). The oe-selftests should be updated to use the same search logic from the image_types_wic.bbclass so that when it invokes "wic create" it is providing wic with the path to a wks file that has a ".wks" extension (which is how bitbake invokes wic) instead of specifying a bare wks file and hoping wic will find it.
Looking at the code, it's a little deeper than that. wic also has a "wic list" subcommand to "list available canned images and source plugins" and it uses the same search logic (using BBLAYERS) to find "canned images". The existing wic documentation, help text, and code refers to these bare wks names as "images" or "canned images". If you specify a bare wks to "wic create" and it can't find the wks file, the error message says (in part): "No image named %s found..." (instead of "No wks named %s found...".
Maybe I'm not understanding the history here, or maybe there's some use-case I don't understand (or has long since stopped working) but what "wic create" needs here is a wks file, not an image name, and conflating the two seems very odd.
If it were up to me I'd clean the whole thing up. "wic create" needs a wks file to guide it, not an image name that wic is going to use to find a <image>.wks file using its own search logic.
I already have two patches to do this: one to add the bbclass search logic to the oe-selftest wic tests, and a second one to remove wic's image search logic from wic itself.
The first one is fairly easy to add and justify (I think).
The second one might be harder. Maybe there is some strange use-case where we want wic to handle "wic create <bare-wks>" where <bare-wks> is thought of as an image name? Removing wic's bare-wks search logic also removes the "wic list images" subcommand since the only point of that subcommand is to invoke its search logic and find "images" (which are actually wks files).
Quoted 34 lines
I'm happy to add "files/wic" as another location for image_types_wic.bbclass to search for wks files.
Quoted 31 lines
Quoted 11 lines
Quoted 2 lines
I don't understand why these need to move given
WKS_SEARCH_PATHremains unchanged, unless wic is going to ignoreWKS_SEARCH_PATHgoing forward?As an independent tool, these *wks files are rather oe-core-specific, so I thought they should stay with the oe-core/meta layer, the same way that, say, raspberry pi-specific *wks files stay in the raspberry pi layer.
My first thought was to remove them from the standalone wic repository altogether, but then decided I could keep them as examples.
Quoted 7 lines
I thought it would be dangerous to leave them in the wic source tree where the above algorithm could find them. If I leave them in src/wic/canned-wks, someone uses the tool and creates a *wks file with a similar name to one of the canned-wks files, but they have a typo and the canned one gets used instead of theirs...
...maybe I'm overthinking it?
It is definitely dangerous to leave them in the search paths so I agree with that. The files do belong in OE-Core too. It is still a little dangerous having files lying around which have the same names as the ones in core since this kind of duplication still can confuse people when they edit the wrong file. We have bigger issues I guess but some renaming of the examples might help.
Cheers,
Richard
Re: [OE-core] [PATCH v9 4/5] wic: move canned *wks files
In reply to Trevor WoernerQuoted 94 lines · Trevor Woerner
Quoted 43 lines · Richard Purdie
Quoted 21 lines · Trevor Woerner
Quoted 9 lines · Richard Purdie
Quoted 7 lines · Trevor Woerner via lists.openembedded.org
When "wic create ..." is invoked with a bare *wks name (i.e. without the
.wksextension), wic calls engine.py:find_canned_images() to find the fully qualified *wks file. This function searches every directory formed by:
- permutating all BBLAYERS with/wic
- permutating all BBLAYERS with/scripts/lib/wic/canned-wks
- checking<scripts_path>/lib/wic/canned-wks
Where<scripts_path>is the directory containing the wic program.Quoted 3 lines
It doesn't. I just looked at the code and it uses BBPATH. That can be similar to BBLAYERS but it is different and the commit messages really should refer to BBPATH.
In the oe-selftest wic tests, most of the "wic create ..." commands are called with bare *wks files (i.e. *wks files without the
.wksextension). When this happens, as "wic create..." is called, it starts by running the code found here:https://git.openembedded.org/openembedded-core/tree/scripts/wic#n206
Quoted 7 lines
which calls:
https://git.openembedded.org/openembedded-core/tree/scripts/lib/wic/engine.py#n46
which follows the algorithm that I have described above.
So for the oe-selftests to pass I wanted to move them.
You are right, it is using BBLAYERS. This is very confusing since the wks files are using BBPATH, which is similar but subtly different.
Is there any reason it can't use BBPATH to match?
I agree the files need to move but I'd like them to end up in meta/files/wic, not meta/wic. Whilst it complicates the migration a little, I think it would be best to move them once and get them into the right place.
The questions are then firstly, how to do that simply, then secondly, how to clean up the rest of this (and when to do that).
Quoted 6 lines
Adding files/wic into the search path is trivial. Deprecating the existing search paths with appropriate warnings is a bit trickier and there is the question of when we should do that, before or after the current release. "after" would seem more obvious, but it will mean two different behaviours before and after release and with a standalone tool, this does become harder.
I'm only guessing here, but it looks like someone wrote "wic create" to run two different ways: with the wks file specified with or without the ".wks" extension.
If ".wks" is not given (e.g. "wic create directdisk-gpt ...", which is what occurs in the oe-selftest tests, i.e. when wic is run independently without being part of a bitbake build) then wic invokes its own search algorithm to try to find the wks file using BBLAYERS and
script_dir. In this case the wks file is specified without the ".wks" and no path information is given on the cmdline. It is up to wic to find the actual wks file itself.If the ".wks" is included (which is what happens when wic is called as part of a bitbake build and therefore the image_types_wic.bbclass is used) then the class adds the ".wks" at the end, and also gives wic the path to the wks file which it finds using
WKS_SEARCH_PATHwhich is based on BBPATH and COREBASE.Assuming the persona of someone who wants to use this new, independent wic tool, and who knows nothing about The Yocto Project or bitbake, I think having wic look in "magical" places for the wks file would be hard to understand and surprising. As an independent tool I think the user should provide the path to the actual wks file (with the ".wks" extension) and if that file can't be found as specified on the cmdline, wic should simply fail with an error.
As part of making wic an independent tool, I think wic's code to search for a wks file if the ".wks" is not provided should be removed. This means that, as part of the transition, I would have to modify each "wic create" test in oe-core's oe-selftests. I'm fine with that. That could be done cleanly if I write a little function to find the wks file in the
wic.pyoe-selftest itself using the same logic as is used in the image_types_wic.bbclass.To summarize: in my opinion, wic shouldn't have search logic. From wic's point of view the wks file should be specified on the cmdline in a way that wic will find the file the user wants to use (or not find it). The oe-selftests should be updated to use the same search logic from the image_types_wic.bbclass so that when it invokes "wic create" it is providing wic with the path to a wks file that has a ".wks" extension (which is how bitbake invokes wic) instead of specifying a bare wks file and hoping wic will find it.
Quoted 2 lines
I'm happy to add "files/wic" as another location for image_types_wic.bbclass to search for wks files.
I guess the first question is are we going to take this for wrynose or not? I'm not particularly happy with the files a top level wic directory, I think it needs to be meta/files/wic. To make that work, we need to tweak the search paths in wic and in image_types_wic.bbclass. If we can resolve that issue, we're probably good to move forward with merging.
I agree that the search path logic should be tidied up. Whether you can provide a search path to wic or whether that has to be done in metadata is an open question. I do see your perspective, equally, if you are using wic within OE, having some search knowledge may be ok. I would want to see it basically using files/wic from BBPATH with everything else removed. There is a question of how you warn users about old paths. This piece is too late for the release though.
So I'd propose we add files/wic to the search path, move the files there and that lets us at least get some of the changes into the release?
Cheers,
Richard
Re: [OE-core] [PATCH v9 4/5] wic: move canned *wks files
Quoted 113 lines · Richard Purdie
Quoted 94 lines · Trevor Woerner
Quoted 43 lines · Richard Purdie
Quoted 21 lines · Trevor Woerner
Quoted 9 lines · Richard Purdie
Quoted 7 lines · Trevor Woerner via lists.openembedded.org
When "wic create ..." is invoked with a bare *wks name (i.e. without the
.wksextension), wic calls engine.py:find_canned_images() to find the fully qualified *wks file. This function searches every directory formed by:
- permutating all BBLAYERS with/wic
- permutating all BBLAYERS with/scripts/lib/wic/canned-wks
- checking<scripts_path>/lib/wic/canned-wks
Where<scripts_path>is the directory containing the wic program.Quoted 3 lines
It doesn't. I just looked at the code and it uses BBPATH. That can be similar to BBLAYERS but it is different and the commit messages really should refer to BBPATH.
In the oe-selftest wic tests, most of the "wic create ..." commands are called with bare *wks files (i.e. *wks files without the
.wksextension). When this happens, as "wic create..." is called, it starts by running the code found here:https://git.openembedded.org/openembedded-core/tree/scripts/wic#n206
Quoted 7 lines
which calls:
https://git.openembedded.org/openembedded-core/tree/scripts/lib/wic/engine.py#n46
which follows the algorithm that I have described above.
So for the oe-selftests to pass I wanted to move them.
You are right, it is using BBLAYERS. This is very confusing since the wks files are using BBPATH, which is similar but subtly different.
Is there any reason it can't use BBPATH to match?
I agree the files need to move but I'd like them to end up in meta/files/wic, not meta/wic. Whilst it complicates the migration a little, I think it would be best to move them once and get them into the right place.
The questions are then firstly, how to do that simply, then secondly, how to clean up the rest of this (and when to do that).
Quoted 6 lines
Adding files/wic into the search path is trivial. Deprecating the existing search paths with appropriate warnings is a bit trickier and there is the question of when we should do that, before or after the current release. "after" would seem more obvious, but it will mean two different behaviours before and after release and with a standalone tool, this does become harder.
I'm only guessing here, but it looks like someone wrote "wic create" to run two different ways: with the wks file specified with or without the ".wks" extension.
If ".wks" is not given (e.g. "wic create directdisk-gpt ...", which is what occurs in the oe-selftest tests, i.e. when wic is run independently without being part of a bitbake build) then wic invokes its own search algorithm to try to find the wks file using BBLAYERS and
script_dir. In this case the wks file is specified without the ".wks" and no path information is given on the cmdline. It is up to wic to find the actual wks file itself.If the ".wks" is included (which is what happens when wic is called as part of a bitbake build and therefore the image_types_wic.bbclass is used) then the class adds the ".wks" at the end, and also gives wic the path to the wks file which it finds using
WKS_SEARCH_PATHwhich is based on BBPATH and COREBASE.Assuming the persona of someone who wants to use this new, independent wic tool, and who knows nothing about The Yocto Project or bitbake, I think having wic look in "magical" places for the wks file would be hard to understand and surprising. As an independent tool I think the user should provide the path to the actual wks file (with the ".wks" extension) and if that file can't be found as specified on the cmdline, wic should simply fail with an error.
As part of making wic an independent tool, I think wic's code to search for a wks file if the ".wks" is not provided should be removed. This means that, as part of the transition, I would have to modify each "wic create" test in oe-core's oe-selftests. I'm fine with that. That could be done cleanly if I write a little function to find the wks file in the
wic.pyoe-selftest itself using the same logic as is used in the image_types_wic.bbclass.To summarize: in my opinion, wic shouldn't have search logic. From wic's point of view the wks file should be specified on the cmdline in a way that wic will find the file the user wants to use (or not find it). The oe-selftests should be updated to use the same search logic from the image_types_wic.bbclass so that when it invokes "wic create" it is providing wic with the path to a wks file that has a ".wks" extension (which is how bitbake invokes wic) instead of specifying a bare wks file and hoping wic will find it.
Quoted 2 lines
I'm happy to add "files/wic" as another location for image_types_wic.bbclass to search for wks files.
I guess the first question is are we going to take this for wrynose or not? I'm not particularly happy with the files a top level wic directory, I think it needs to be meta/files/wic. To make that work, we need to tweak the search paths in wic and in image_types_wic.bbclass. If we can resolve that issue, we're probably good to move forward with merging.
I agree that the search path logic should be tidied up. Whether you can provide a search path to wic or whether that has to be done in metadata is an open question. I do see your perspective, equally, if you are using wic within OE, having some search knowledge may be ok. I would want to see it basically using files/wic from BBPATH with everything else removed. There is a question of how you warn users about old paths. This piece is too late for the release though.
Quoted 3 lines
So I'd propose we add files/wic to the search path, move the files there and that lets us at least get some of the changes into the release?
Sounds perfect, thanks!
Quoted 4 lines
Cheers,
Richard
Re: [OE-core] [PATCH v9 4/5] wic: move canned *wks files
Quoted 65 lines · Trevor Woerner
Quoted 64 lines · Richard Purdie
Quoted 45 lines · Trevor Woerner
On Sat 2026-04-04 @ 08:27:13 AM, Richard Purdie wrote:
I'm only guessing here, but it looks like someone wrote "wic create" to run two different ways: with the wks file specified with or without the ".wks" extension.
If ".wks" is not given (e.g. "wic create directdisk-gpt ...", which is what occurs in the oe-selftest tests, i.e. when wic is run independently without being part of a bitbake build) then wic invokes its own search algorithm to try to find the wks file using BBLAYERS and
script_dir. In this case the wks file is specified without the ".wks" and no path information is given on the cmdline. It is up to wic to find the actual wks file itself.If the ".wks" is included (which is what happens when wic is called as part of a bitbake build and therefore the image_types_wic.bbclass is used) then the class adds the ".wks" at the end, and also gives wic the path to the wks file which it finds using
WKS_SEARCH_PATHwhich is based on BBPATH and COREBASE.Assuming the persona of someone who wants to use this new, independent wic tool, and who knows nothing about The Yocto Project or bitbake, I think having wic look in "magical" places for the wks file would be hard to understand and surprising. As an independent tool I think the user should provide the path to the actual wks file (with the ".wks" extension) and if that file can't be found as specified on the cmdline, wic should simply fail with an error.
As part of making wic an independent tool, I think wic's code to search for a wks file if the ".wks" is not provided should be removed. This means that, as part of the transition, I would have to modify each "wic create" test in oe-core's oe-selftests. I'm fine with that. That could be done cleanly if I write a little function to find the wks file in the
wic.pyoe-selftest itself using the same logic as is used in the image_types_wic.bbclass.To summarize: in my opinion, wic shouldn't have search logic. From wic's point of view the wks file should be specified on the cmdline in a way that wic will find the file the user wants to use (or not find it). The oe-selftests should be updated to use the same search logic from the image_types_wic.bbclass so that when it invokes "wic create" it is providing wic with the path to a wks file that has a ".wks" extension (which is how bitbake invokes wic) instead of specifying a bare wks file and hoping wic will find it.
Quoted 2 lines
I'm happy to add "files/wic" as another location for image_types_wic.bbclass to search for wks files.
I guess the first question is are we going to take this for wrynose or not? I'm not particularly happy with the files a top level wic directory, I think it needs to be meta/files/wic. To make that work, we need to tweak the search paths in wic and in image_types_wic.bbclass. If we can resolve that issue, we're probably good to move forward with merging.
I agree that the search path logic should be tidied up. Whether you can provide a search path to wic or whether that has to be done in metadata is an open question. I do see your perspective, equally, if you are using wic within OE, having some search knowledge may be ok. I would want to see it basically using files/wic from BBPATH with everything else removed. There is a question of how you warn users about old paths. This piece is too late for the release though.
Quoted 5 lines
Quoted 3 lines
So I'd propose we add files/wic to the search path, move the files there and that lets us at least get some of the changes into the release?
Sounds perfect, thanks!
Since we're running out of time and I really don't want to have to think about this too much more/again, I've:
* written a patch to add files/wic as a search path in wic
* written a patch to add files/wic as a search path in the class
* dropped wic and the canned-wks paths in wic
* dropped wic and the canned-wks paths in the class
* converted wic to use BBPATH instead of BBLAYERS
* added a sanity test to detect the old paths and error
* pushed a contrib/rpurdie branch into the wic repo with the patches
This is going to mean people have to update layers to use "files/wic" but we can clearly tell people they need to do it. If we went with either of the other locations and changed BBLAYERS -> BBPATH, there could be corner cases we miss in the detection.
Autobuilder testing shows this will break a few layers but it is a clear error which people can easily fix.
How does that look to people?
Cheers,
Richard
Re: [OE-core] [PATCH v9 4/5] wic: move canned *wks files
Quoted 88 lines · Richard Purdie
Quoted 65 lines · Trevor Woerner
Quoted 64 lines · Richard Purdie
Quoted 45 lines · Trevor Woerner
On Sat 2026-04-04 @ 08:27:13 AM, Richard Purdie wrote:
I'm only guessing here, but it looks like someone wrote "wic create" to run two different ways: with the wks file specified with or without the ".wks" extension.
If ".wks" is not given (e.g. "wic create directdisk-gpt ...", which is what occurs in the oe-selftest tests, i.e. when wic is run independently without being part of a bitbake build) then wic invokes its own search algorithm to try to find the wks file using BBLAYERS and
script_dir. In this case the wks file is specified without the ".wks" and no path information is given on the cmdline. It is up to wic to find the actual wks file itself.If the ".wks" is included (which is what happens when wic is called as part of a bitbake build and therefore the image_types_wic.bbclass is used) then the class adds the ".wks" at the end, and also gives wic the path to the wks file which it finds using
WKS_SEARCH_PATHwhich is based on BBPATH and COREBASE.Assuming the persona of someone who wants to use this new, independent wic tool, and who knows nothing about The Yocto Project or bitbake, I think having wic look in "magical" places for the wks file would be hard to understand and surprising. As an independent tool I think the user should provide the path to the actual wks file (with the ".wks" extension) and if that file can't be found as specified on the cmdline, wic should simply fail with an error.
As part of making wic an independent tool, I think wic's code to search for a wks file if the ".wks" is not provided should be removed. This means that, as part of the transition, I would have to modify each "wic create" test in oe-core's oe-selftests. I'm fine with that. That could be done cleanly if I write a little function to find the wks file in the
wic.pyoe-selftest itself using the same logic as is used in the image_types_wic.bbclass.To summarize: in my opinion, wic shouldn't have search logic. From wic's point of view the wks file should be specified on the cmdline in a way that wic will find the file the user wants to use (or not find it). The oe-selftests should be updated to use the same search logic from the image_types_wic.bbclass so that when it invokes "wic create" it is providing wic with the path to a wks file that has a ".wks" extension (which is how bitbake invokes wic) instead of specifying a bare wks file and hoping wic will find it.
Quoted 2 lines
I'm happy to add "files/wic" as another location for image_types_wic.bbclass to search for wks files.
I guess the first question is are we going to take this for wrynose or not? I'm not particularly happy with the files a top level wic directory, I think it needs to be meta/files/wic. To make that work, we need to tweak the search paths in wic and in image_types_wic.bbclass. If we can resolve that issue, we're probably good to move forward with merging.
I agree that the search path logic should be tidied up. Whether you can provide a search path to wic or whether that has to be done in metadata is an open question. I do see your perspective, equally, if you are using wic within OE, having some search knowledge may be ok. I would want to see it basically using files/wic from BBPATH with everything else removed. There is a question of how you warn users about old paths. This piece is too late for the release though.
Quoted 5 lines
Quoted 3 lines
So I'd propose we add files/wic to the search path, move the files there and that lets us at least get some of the changes into the release?
Sounds perfect, thanks!
Since we're running out of time and I really don't want to have to think about this too much more/again, I've:
* written a patch to add files/wic as a search path in wic
* written a patch to add files/wic as a search path in the class
* dropped wic and the canned-wks paths in wic
* dropped wic and the canned-wks paths in the class
* converted wic to use BBPATH instead of BBLAYERS
* added a sanity test to detect the old paths and error
* pushed a contrib/rpurdie branch into the wic repo with the patchesThis is going to mean people have to update layers to use "files/wic" but we can clearly tell people they need to do it. If we went with either of the other locations and changed BBLAYERS -> BBPATH, there could be corner cases we miss in the detection.
Quoted 4 lines
Autobuilder testing shows this will break a few layers but it is a clear error which people can easily fix.
How does that look to people?
Sounds great to me!
Presumably we will move the contrib/rpurdie branch to master in git.yoctoproject.org/wic and the SRC_URI of the wic recipe in oe-core will need a tweak before making it official?
Quoted 9 lines
Cheers,
Richard
Re: [OE-core] [PATCH v9 4/5] wic: move canned *wks files
Quoted 92 lines · Trevor Woerner
Quoted 91 lines · Richard Purdie
Quoted 65 lines · Trevor Woerner
Quoted 64 lines · Richard Purdie
Quoted 45 lines · Trevor Woerner
On Sat 2026-04-04 @ 08:27:13 AM, Richard Purdie wrote:
I'm only guessing here, but it looks like someone wrote "wic create" to run two different ways: with the wks file specified with or without the ".wks" extension.
If ".wks" is not given (e.g. "wic create directdisk-gpt ...", which is what occurs in the oe-selftest tests, i.e. when wic is run independently without being part of a bitbake build) then wic invokes its own search algorithm to try to find the wks file using BBLAYERS and
script_dir. In this case the wks file is specified without the ".wks" and no path information is given on the cmdline. It is up to wic to find the actual wks file itself.If the ".wks" is included (which is what happens when wic is called as part of a bitbake build and therefore the image_types_wic.bbclass is used) then the class adds the ".wks" at the end, and also gives wic the path to the wks file which it finds using
WKS_SEARCH_PATHwhich is based on BBPATH and COREBASE.Assuming the persona of someone who wants to use this new, independent wic tool, and who knows nothing about The Yocto Project or bitbake, I think having wic look in "magical" places for the wks file would be hard to understand and surprising. As an independent tool I think the user should provide the path to the actual wks file (with the ".wks" extension) and if that file can't be found as specified on the cmdline, wic should simply fail with an error.
As part of making wic an independent tool, I think wic's code to search for a wks file if the ".wks" is not provided should be removed. This means that, as part of the transition, I would have to modify each "wic create" test in oe-core's oe-selftests. I'm fine with that. That could be done cleanly if I write a little function to find the wks file in the
wic.pyoe-selftest itself using the same logic as is used in the image_types_wic.bbclass.To summarize: in my opinion, wic shouldn't have search logic. From wic's point of view the wks file should be specified on the cmdline in a way that wic will find the file the user wants to use (or not find it). The oe-selftests should be updated to use the same search logic from the image_types_wic.bbclass so that when it invokes "wic create" it is providing wic with the path to a wks file that has a ".wks" extension (which is how bitbake invokes wic) instead of specifying a bare wks file and hoping wic will find it.
Quoted 2 lines
I'm happy to add "files/wic" as another location for image_types_wic.bbclass to search for wks files.
I guess the first question is are we going to take this for wrynose or not? I'm not particularly happy with the files a top level wic directory, I think it needs to be meta/files/wic. To make that work, we need to tweak the search paths in wic and in image_types_wic.bbclass. If we can resolve that issue, we're probably good to move forward with merging.
I agree that the search path logic should be tidied up. Whether you can provide a search path to wic or whether that has to be done in metadata is an open question. I do see your perspective, equally, if you are using wic within OE, having some search knowledge may be ok. I would want to see it basically using files/wic from BBPATH with everything else removed. There is a question of how you warn users about old paths. This piece is too late for the release though.
Quoted 5 lines
Quoted 3 lines
So I'd propose we add files/wic to the search path, move the files there and that lets us at least get some of the changes into the release?
Sounds perfect, thanks!
Since we're running out of time and I really don't want to have to think about this too much more/again, I've:
* written a patch to add files/wic as a search path in wic
* written a patch to add files/wic as a search path in the class
* dropped wic and the canned-wks paths in wic
* dropped wic and the canned-wks paths in the class
* converted wic to use BBPATH instead of BBLAYERS
* added a sanity test to detect the old paths and error
* pushed a contrib/rpurdie branch into the wic repo with the patchesThis is going to mean people have to update layers to use "files/wic" but we can clearly tell people they need to do it. If we went with either of the other locations and changed BBLAYERS -> BBPATH, there could be corner cases we miss in the detection.
Autobuilder testing shows this will break a few layers but it is a clear error which people can easily fix.
Quoted 7 lines
Quoted 1 line
How does that look to people?
Sounds great to me!
Presumably we will move the contrib/rpurdie branch to master in git.yoctoproject.org/wic and the
SRC_URIof the wic recipe in oe-core will need a tweak before making it official?
I pushed onto a contrib branch so we can make a decision on this. Assuming we go ahead, we'd merge the patches to master, the update the recipe accordingly.
Cheers,
Richard
Re: [OE-core] [PATCH v9 4/5] wic: move canned *wks files
Quoted 97 lines · Richard Purdie
Quoted 96 lines · Trevor Woerner
Quoted 88 lines · Richard Purdie
Quoted 65 lines · Trevor Woerner
Quoted 64 lines · Richard Purdie
Quoted 45 lines · Trevor Woerner
On Sat 2026-04-04 @ 08:27:13 AM, Richard Purdie wrote:
I'm only guessing here, but it looks like someone wrote "wic create" to run two different ways: with the wks file specified with or without the ".wks" extension.
If ".wks" is not given (e.g. "wic create directdisk-gpt ...", which is what occurs in the oe-selftest tests, i.e. when wic is run independently without being part of a bitbake build) then wic invokes its own search algorithm to try to find the wks file using BBLAYERS and
script_dir. In this case the wks file is specified without the ".wks" and no path information is given on the cmdline. It is up to wic to find the actual wks file itself.If the ".wks" is included (which is what happens when wic is called as part of a bitbake build and therefore the image_types_wic.bbclass is used) then the class adds the ".wks" at the end, and also gives wic the path to the wks file which it finds using
WKS_SEARCH_PATHwhich is based on BBPATH and COREBASE.Assuming the persona of someone who wants to use this new, independent wic tool, and who knows nothing about The Yocto Project or bitbake, I think having wic look in "magical" places for the wks file would be hard to understand and surprising. As an independent tool I think the user should provide the path to the actual wks file (with the ".wks" extension) and if that file can't be found as specified on the cmdline, wic should simply fail with an error.
As part of making wic an independent tool, I think wic's code to search for a wks file if the ".wks" is not provided should be removed. This means that, as part of the transition, I would have to modify each "wic create" test in oe-core's oe-selftests. I'm fine with that. That could be done cleanly if I write a little function to find the wks file in the
wic.pyoe-selftest itself using the same logic as is used in the image_types_wic.bbclass.To summarize: in my opinion, wic shouldn't have search logic. From wic's point of view the wks file should be specified on the cmdline in a way that wic will find the file the user wants to use (or not find it). The oe-selftests should be updated to use the same search logic from the image_types_wic.bbclass so that when it invokes "wic create" it is providing wic with the path to a wks file that has a ".wks" extension (which is how bitbake invokes wic) instead of specifying a bare wks file and hoping wic will find it.
Quoted 2 lines
I'm happy to add "files/wic" as another location for image_types_wic.bbclass to search for wks files.
I guess the first question is are we going to take this for wrynose or not? I'm not particularly happy with the files a top level wic directory, I think it needs to be meta/files/wic. To make that work, we need to tweak the search paths in wic and in image_types_wic.bbclass. If we can resolve that issue, we're probably good to move forward with merging.
I agree that the search path logic should be tidied up. Whether you can provide a search path to wic or whether that has to be done in metadata is an open question. I do see your perspective, equally, if you are using wic within OE, having some search knowledge may be ok. I would want to see it basically using files/wic from BBPATH with everything else removed. There is a question of how you warn users about old paths. This piece is too late for the release though.
Quoted 5 lines
Quoted 3 lines
So I'd propose we add files/wic to the search path, move the files there and that lets us at least get some of the changes into the release?
Sounds perfect, thanks!
Since we're running out of time and I really don't want to have to think about this too much more/again, I've:
* written a patch to add files/wic as a search path in wic
* written a patch to add files/wic as a search path in the class
* dropped wic and the canned-wks paths in wic
* dropped wic and the canned-wks paths in the class
* converted wic to use BBPATH instead of BBLAYERS
* added a sanity test to detect the old paths and error
* pushed a contrib/rpurdie branch into the wic repo with the patchesThis is going to mean people have to update layers to use "files/wic" but we can clearly tell people they need to do it. If we went with either of the other locations and changed BBLAYERS -> BBPATH, there could be corner cases we miss in the detection.
Quoted 4 lines
Autobuilder testing shows this will break a few layers but it is a clear error which people can easily fix.
How does that look to people?
Sounds great to me!
Quoted 7 lines
Quoted 3 lines
Presumably we will move the contrib/rpurdie branch to master in git.yoctoproject.org/wic and the
SRC_URIof the wic recipe in oe-core will need a tweak before making it official?I pushed onto a contrib branch so we can make a decision on this. Assuming we go ahead, we'd merge the patches to master, the update the recipe accordingly.
I've looked at your patches in oe-core master-next as well as in the wic repository contrib/rpurdie and they all look fine to me. This is a nice update for wic.
Quoted 3 lines
Cheers,
Richard
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: [OE-core] [PATCH v9 4/5] wic: move canned *wks files
Quoted 26 lines · Richard Purdie via lists.openembedded.org
Quoted 7 lines · Trevor Woerner via lists.openembedded.org
When "wic create ..." is invoked with a bare *wks name (i.e. without the
.wksextension), wic calls engine.py:find_canned_images() to find the fully qualified *wks file. This function searches every directory formed by:
- permutating all BBLAYERS with/wic
- permutating all BBLAYERS with/scripts/lib/wic/canned-wks
- checking<scripts_path>/lib/wic/canned-wks
Where<scripts_path>is the directory containing the wic program.It doesn't. I just looked at the code and it uses BBPATH. That can be similar to BBLAYERS but it is different and the commit messages really should refer to BBPATH.
Quoted 9 lines
When wic is part of oe-core, the last search path succeeds in finding the canned *wks files in
<topdir>/scripts/lib/wic/canned-wks(since the wic program is found in oe-core's<topdir>/scriptsdirectory, and<topdir>/scriptsis not a BBLAYER).However, once wic is removed from oe-core, this algorithm will not find these bare *wks files in any of the above-mentioned search paths since the oe-core layer will no longer be the home of the wic program, and the canned *wks files are not located in any directory relative to BBLAYERS.
I'm a bit confused by this reasoning. Are you saying that wic will no longer search BBPATH (or
WKS_SEARCH_PATH)?
Quoted 6 lines
Quoted 3 lines
Since these *wks files are specific to oe-core's meta layer, they should stay with this layer. Therefore move the *wks files so they exist in one of the locations searched relative to oe-core/meta's BBLAYERS.
I don't understand why these need to move given
WKS_SEARCH_PATHremains unchanged, unless wic is going to ignoreWKS_SEARCH_PATHgoing forward?
Looking at this a bit more, the issue is that you're removing the directory when you separate wic out so the files 'need' to move due to that?
You could in fact just leave them there if I understand correctly?
Cheers,
Richard
PATCH v9 5/5 wic: remove to standalone repository
In reply to Trevor WoernerThe wic utility will no longer be integrated into the oe-core repository. However, the wic utility will continue to exist under The Yocto Project umbrella. This will allow wic to be more easily used independently of bitbake and oe-core.
The primary repository for standalone wic will be:
https://git.yoctoproject.org/wic
---
changes in v9:
- improved commit message
- reordered patch to be last so builds and tests are less likely to break should a bisection land in the middle of this set
changes in v8:
- (in v8 the sector-size cmdline arg was upstreamed)
changes in v7:
- (none)
changes in v6:
- improved commit summary
- swap this patch with the patch to move the canned-wks files
changes in v5:
- rebase with master
- split patches back out again
changes in v4:
- (skipped)
changes in v3:
- squashed into one large patch
changes in v2:
- (none)
--- scripts/lib/wic/__init__.py | 10 - scripts/lib/wic/engine.py | 718 ---------- scripts/lib/wic/filemap.py | 590 -------- scripts/lib/wic/help.py | 1197 ----------------- scripts/lib/wic/ksparser.py | 322 ----- scripts/lib/wic/misc.py | 272 ---- scripts/lib/wic/partition.py | 615 --------- scripts/lib/wic/pluginbase.py | 144 -- scripts/lib/wic/plugins/imager/direct.py | 702 ---------- .../wic/plugins/source/bootimg_biosplusefi.py | 213 --- scripts/lib/wic/plugins/source/bootimg_efi.py | 436 ------ .../wic/plugins/source/bootimg_partition.py | 162 --- .../lib/wic/plugins/source/bootimg_pcbios.py | 484 ------- scripts/lib/wic/plugins/source/empty.py | 89 -- .../lib/wic/plugins/source/extra_partition.py | 146 -- .../wic/plugins/source/isoimage_isohybrid.py | 464 ------- scripts/lib/wic/plugins/source/rawcopy.py | 115 -- scripts/lib/wic/plugins/source/rootfs.py | 236 ---- scripts/wic | 641 --------- 19 files changed, 7556 deletions(-) delete mode 100644 scripts/lib/wic/__init__.py delete mode 100644 scripts/lib/wic/engine.py delete mode 100644 scripts/lib/wic/filemap.py delete mode 100644 scripts/lib/wic/help.py delete mode 100644 scripts/lib/wic/ksparser.py delete mode 100644 scripts/lib/wic/misc.py delete mode 100644 scripts/lib/wic/partition.py delete mode 100644 scripts/lib/wic/pluginbase.py delete mode 100644 scripts/lib/wic/plugins/imager/direct.py delete mode 100644 scripts/lib/wic/plugins/source/bootimg_biosplusefi.py delete mode 100644 scripts/lib/wic/plugins/source/bootimg_efi.py delete mode 100644 scripts/lib/wic/plugins/source/bootimg_partition.py delete mode 100644 scripts/lib/wic/plugins/source/bootimg_pcbios.py delete mode 100644 scripts/lib/wic/plugins/source/empty.py delete mode 100644 scripts/lib/wic/plugins/source/extra_partition.py delete mode 100644 scripts/lib/wic/plugins/source/isoimage_isohybrid.py delete mode 100644 scripts/lib/wic/plugins/source/rawcopy.py delete mode 100644 scripts/lib/wic/plugins/source/rootfs.py delete mode 100755 scripts/wic
scripts/lib/wic/__init__.py deleted +0 -10
@@ -1,10 +0,0 @@ -#!/usr/bin/env python3 -# -# Copyright (c) 2007 Red Hat, Inc. -# Copyright (c) 2011 Intel, Inc. -# -# SPDX-License-Identifier: GPL-2.0-only -# - -class WicError(Exception): - pass
scripts/lib/wic/engine.py deleted +0 -718
718 lines not on this page · raw message
scripts/lib/wic/filemap.py deleted +0 -590
590 lines not on this page · raw message
scripts/lib/wic/help.py deleted +0 -1197
1,197 lines not on this page · raw message
scripts/lib/wic/ksparser.py deleted +0 -322
322 lines not on this page · raw message
scripts/lib/wic/misc.py deleted +0 -272
163 lines not on this page · raw message
scripts/lib/wic/partition.py deleted +0 -615
615 lines not on this page · raw message
scripts/lib/wic/pluginbase.py deleted +0 -144
144 lines not on this page · raw message
scripts/lib/wic/plugins/imager/direct.py deleted +0 -702
702 lines not on this page · raw message
scripts/lib/wic/plugins/source/bootimg_biosplusefi.py deleted +0 -213
213 lines not on this page · raw message
scripts/lib/wic/plugins/source/bootimg_efi.py deleted +0 -436
436 lines not on this page · raw message
scripts/lib/wic/plugins/source/bootimg_partition.py deleted +0 -162
162 lines not on this page · raw message
scripts/lib/wic/plugins/source/bootimg_pcbios.py deleted +0 -484
484 lines not on this page · raw message
scripts/lib/wic/plugins/source/empty.py deleted +0 -89
89 lines not on this page · raw message
scripts/lib/wic/plugins/source/extra_partition.py deleted +0 -146
146 lines not on this page · raw message
scripts/lib/wic/plugins/source/isoimage_isohybrid.py deleted +0 -464
464 lines not on this page · raw message
scripts/lib/wic/plugins/source/rawcopy.py deleted +0 -115
115 lines not on this page · raw message
scripts/lib/wic/plugins/source/rootfs.py deleted +0 -236
236 lines not on this page · raw message
scripts/wic deleted +0 -641
641 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Signature · 1 line
2.51.0
Part of the diff is not shown here. View the raw message.
Re: [PATCH v9 5/5] wic: remove to standalone repository
Hello,
Trevor and all.
Are there any plans to add a similar oe-core style testing infrastructure in the new wic standalone repository? Or is the plan to just store the wic command and plugins in the standalone repo, but add all test cases to OE core?
Awesome thinking and work Trevor!
Re: [OE-core] [PATCH v9 5/5] wic: remove to standalone repository
Hi Vincent,
I am in the process of adding an extensive test suite to independent wic based on pytest. It will include lots of unit tests as well as a number of integration tests. The tests are mostly written at this point, I'm just going over everything in detail before submitting.
I don't know if oe-core will continue to have its own wic tests. I guess that's something we can discuss once I get wic's tests in. oe-core might always want to keep some sanity tests around, just to make sure wic is doing the basics of what it expects, especially the qemu-based ones to verify the images actually work for oe-core's purposes.
Best regards,
Trevor
Quoted 21 lines · Vincent Davis Jr via lists.openembedded.org
Hello,
Trevor and all.
Are there any plans to add a similar oe-core style testing infrastructure in the new wic standalone repository? Or is the plan to just store the wic command and plugins in the standalone repo, but add all test cases to OE core?
Awesome thinking and work Trevor!
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#235523): https://lists.openembedded.org/g/openembedded-core/message/235523
Mute This Topic: https://lists.openembedded.org/mt/118654172/900817
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-