Skip to contentSkip to conversation

wic: ship its tools, and add an SDK_FEATURES lever

PATCH Superseded Trevor Woerner 2 participants 8 messages Jul 17 – Jul 19

v3 of 4 Superseded Latest is v4 →

lore
Share & tools
lore
#0 lore raw

v3 cover wic: ship its tools, and add an SDK_FEATURES lever

wic shells out to a range of host-side tools but does not declare them, so wherever it is installed as a package (rather than run from the bitbake do_image_wic task, which has the wic-tools native sysroot on PATH) it works only by chance depending on what the host has installed.

Patch 1 fixes that directly: wic gains RDEPENDS on the tools it runs, so they are installed alongside it in every variant that packages it.

The rest of the series turns "what optional tools go into the SDK" into an explicit, self-documenting choice. It introduces SDK_FEATURES, a list in the same spirit as DISTRO_FEATURES / MACHINE_FEATURES / IMAGE_FEATURES:
a developer reads the setting and knows exactly what their SDK contains.

- patch 2 adds SDK_FEATURES and gates wic on the "wic" feature (off by

    default);
  - patch 3 moves the existing (unconditional, arch-gated) qemu addition
    behind a "qemu" feature, on by default so nothing regresses;

- patch 4 adds an opt-in "sbom" feature (SPDX + SBOM/CVE tooling);
- patch 5 moves the cross-canadian gdb behind a "gdb" feature, on by default; binutils and gcc stay unconditional since a cross-compiler is intrinsic to an SDK;
- patch 6 adds an opt-in "lldb" feature for developers who prefer it.

The default SDK_FEATURES is "gdb qemu", so an unmodified configuration produces the same SDK as before. The intent is to present the full range of knobs and let review settle which belong on by default; the split across separate patches is deliberate so each can be debated on its own. SDK_FEATURES is a plain list, so downstream layers can define their own features (for example a cgdb feature in meta-openembedded) the same way.

changes in v3:
- replace the SDK_INCLUDE_WIC yes/no knob with the general SDK_FEATURES lever, and use it for qemu, sbom, gdb and lldb as well as wic
- the v2 "buildtools-extended-tarball: drop wic helper tools" patch was merged already and is dropped from the series

changes in v2:
- dropped the wic-tools.inc refactor; folded the tool list into the wic recipe
- reworked the buildtools-extended-tarball change into a plain removal
- gated wic in the SDK behind a knob instead of adding it unconditionally

Trevor Woerner (6):
wic: add runtime dependencies on the tools it invokes nativesdk-packagegroup-sdk-host: add wic via a new SDK_FEATURES lever nativesdk-packagegroup-sdk-host: gate qemu behind SDK_FEATURES nativesdk-packagegroup-sdk-host: add an sbom SDK feature packagegroup-cross-canadian: gate gdb behind SDK_FEATURES nativesdk-packagegroup-sdk-host: add an lldb SDK feature

 meta/conf/bitbake.conf                        |  2 +-
 meta/conf/documentation.conf                  |  1 +
 .../nativesdk-packagegroup-sdk-host.bb        |  5 +++-
 .../packagegroup-cross-canadian.bb            |  2 +-
 meta/recipes-support/wic/wic_0.3.1.bb         | 23 +++++++++++++++++++
 5 files changed, 30 insertions(+), 3 deletions(-)
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#1 lore raw

PATCH v3 1/6 wic: add runtime dependencies on the tools it invokes

wic shells out to a number of host-side tools to do its work: parted, mkdosfs, mcopy, the mkfs.* family, sfdisk, e2fsck, resize2fs, debugfs, blkid and more, depending on the operation and on the partition types in the image.

None of these are declared as dependencies of wic. For the bitbake do_image_wic task this is handled by the wic-tools recipe, which builds the tools into a native sysroot. Wherever else wic is installed as a package it gets none of them, and its offline lookup (wic ls/cp/write) resolves each tool over a search path that falls back to the host PATH, so on a machine that happens to have the tool installed system-wide wic succeeds while on a machine without it wic fails:

wic.WicError: Can't find executable 'mcopy'

Falling back to the host is unreliable even when the tool is found: wic depends on specific minimum versions of some of these tools, and an arbitrary host copy may be too old, so wic can pick up a tool that does not behave as it needs. Declaring the tools as dependencies pins known-good versions instead of trusting whatever the host provides.

Add the tools wic runs as RDEPENDS so they are installed alongside it. resize2fs is listed on its own because e2fsprogs packages it separately as e2fsprogs-resize2fs, which e2fsprogs does not pull in, and wic needs it for the cp/write resize path.

syslinux (with its isohybrid helper) is only available on x86, so it is gated to those hosts; cdrtools (mkisofs) is native-only with no nativesdk variant and is only needed for ISO images, so it is not listed here.

AI-Generated: codex/claude-opus 4.8 (xhigh)

---
changes in v3:
- list the tools on all variants rather than only the nativesdk variant

changes in v2:
- merge the tool list from the v1 "wic-tools.inc: add" patch into this recipe as RDEPENDS; the separate .inc is dropped
- list the tools inline rather than via a shared WIC_TOOLS variable
- note in the commit log that a host tool may also be the wrong version, not merely absent

---
 meta/recipes-support/wic/wic_0.3.1.bb | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)
meta/recipes-support/wic/wic_0.3.1.bb +23 -0
index d9b4cc05c4bd..0911a2ae35d7 100644
@@ -17,4 +17,27 @@ RDEPENDS:${PN} += " \
     python3-misc \
     "
 
+# wic shells out to these host-side tools when it runs, so they must be
+# installed alongside it.
+RDEPENDS:${PN} += " \
+    parted \
+    gptfdisk \
+    dosfstools \
+    mtools \
+    bmaptool \
+    grub \
+    btrfs-tools \
+    squashfs-tools \
+    e2fsprogs \
+    e2fsprogs-resize2fs \
+    util-linux \
+    tar \
+    erofs-utils \
+"
+
+# syslinux (with its isohybrid helper) is only available on x86.
+RDEPENDS:${PN}:append:x86 = " syslinux syslinux-misc"
+RDEPENDS:${PN}:append:x86-64 = " syslinux syslinux-misc"
+RDEPENDS:${PN}:append:x86-x32 = " syslinux syslinux-misc"
+
 BBCLASSEXTEND = "native nativesdk"
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#2 lore raw

PATCH v3 2/6 nativesdk-packagegroup-sdk-host: add wic via a new SDK_FEATURES lever

In reply to Trevor Woerner

wic is useful from an installed SDK for assembling or inspecting a wic image, but adding it is not free: nativesdk-wic pulls in the tools it runs (parted, gptfdisk, dosfstools, mtools, bmaptool, grub, btrfs-tools, squashfs-tools, e2fsprogs, util-linux, tar, erofs-utils, and syslinux on x86). Most SDKs do not need any of that, so wic should not be in the host packagegroup unconditionally.

Introduce SDK_FEATURES, a list of optional additions to the SDK host tools, in the same spirit as DISTRO_FEATURES, MACHINE_FEATURES and
IMAGE_FEATURES: a developer reads the setting and knows exactly what is being included.

Gate wic in the host packagegroup on the "wic" feature, which is not enabled by default. A configuration that wants wic in its SDK adds the "wic" feature to SDK_FEATURES and gets a working wic, together with the tools it invokes, in the SDK.

AI-Generated: codex/claude-opus 4.8 (xhigh)

---
changes in v3:
- replace the SDK_INCLUDE_WIC yes/no knob with a new general SDK_FEATURES list; gate wic on the "wic" feature
- add the SDK_FEATURES default and documentation.conf entry

changes in v2:
- gate nativesdk-wic behind a new SDK_INCLUDE_WIC knob (default "no") instead of adding it unconditionally as in v1

---
 meta/conf/bitbake.conf                                          | 2 +-
 meta/conf/documentation.conf                                    | 1 +
 .../packagegroups/nativesdk-packagegroup-sdk-host.bb            | 1 +
 3 files changed, 3 insertions(+), 1 deletion(-)
meta/conf/bitbake.conf +1 -1
index bdf37d0da247..b7bc74887b69 100644
@@ -898,6 +898,7 @@ OES_BITBAKE_CONF = "1"
 
 MACHINE_FEATURES:append = " ${@oe.utils.filter_default_features('MACHINE_FEATURES', d)}"
 SDK_MACHINE_FEATURES ?= ""
+SDK_FEATURES ?= ""
 
 DISTRO_FEATURES:append = " ${@oe.utils.filter_default_features('DISTRO_FEATURES', d)}"
 
@@ -992,4 +993,3 @@ MULTILIB_VARIANTS ??= ""
 # what it would be anyway if the signature generator (e.g. OEEquivHash) doesn't
 # support unihashes.
 BB_UNIHASH ?= "${BB_TASKHASH}"
-
meta/conf/documentation.conf +1 -0
index 842cf31739de..604c51322813 100644
@@ -361,6 +361,7 @@ SANITY_TESTED_DISTROS[doc] = "A list of the host distribution identifiers that t
 SDK_ARCH[doc] = "The target architecture for the SDK."
 SDK_DEPLOY[doc] = "The directory set up and used by the populate_sdk_base to which the SDK is deployed."
 SDK_DIR[doc] = "The parent directory used by the OpenEmbedded build system when creating SDK output."
+SDK_FEATURES[doc] = "The list of optional features to add to the standalone SDK's host tools, for example 'wic'. Analogous to IMAGE_FEATURES but for the SDK host side."
 SDK_NAME[doc] = "The base name for SDK output files."
 SDK_OUTPUT[doc] = "The location used by the OpenEmbedded build system when creating SDK output."
 SDKIMAGE_FEATURES[doc] = "Equivalent to IMAGE_FEATURES. However, this variable applies to the SDK generated from an image using the command 'bitbake -c populate_sdk imagename'."
meta/recipes-core/packagegroups/nativesdk-packagegroup-sdk-host.bb +1 -0
index 51e48f250b19..c2974a57fe3e 100644
@@ -29,6 +29,7 @@ RDEPENDS:${PN} = "\
     nativesdk-bison \
     nativesdk-flex \
     nativesdk-perl-module-integer \
+    ${@bb.utils.contains('SDK_FEATURES', 'wic', 'nativesdk-wic', '', d)} \
     "
 
 RDEPENDS:${PN}:darwin = "\
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#3 lore raw

PATCH v3 3/6 nativesdk-packagegroup-sdk-host: gate qemu behind SDK_FEATURES

In reply to Trevor Woerner

qemu (with its helper) is added to every standalone SDK on the architectures where it builds. It is only useful for running target binaries on the host via qemu-usermode, which not every SDK user wants, and it is a large thing to carry when they do not.

Move it behind the "qemu" SDK feature, which is enabled by default so an unmodified configuration still gets qemu exactly as before on the same architectures; the architecture guard is retained. A configuration that does not want it removes the "qemu" feature from SDK_FEATURES.

AI-Generated: codex/claude-opus 4.8 (xhigh)

---
changes in v3:
- new in v3

---
 meta/conf/bitbake.conf                                          | 2 +-
 .../packagegroups/nativesdk-packagegroup-sdk-host.bb            | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
meta/conf/bitbake.conf +1 -1
index b7bc74887b69..c9317bc9722a 100644
@@ -898,7 +898,7 @@ OES_BITBAKE_CONF = "1"
 
 MACHINE_FEATURES:append = " ${@oe.utils.filter_default_features('MACHINE_FEATURES', d)}"
 SDK_MACHINE_FEATURES ?= ""
-SDK_FEATURES ?= ""
+SDK_FEATURES ?= "qemu"
 
 DISTRO_FEATURES:append = " ${@oe.utils.filter_default_features('DISTRO_FEATURES', d)}"
 
meta/recipes-core/packagegroups/nativesdk-packagegroup-sdk-host.bb +1 -1
index c2974a57fe3e..7205599288a2 100644
@@ -13,7 +13,7 @@ PACKAGEGROUP_DISABLE_COMPLEMENTARY = "1"
 # build some recent linux kernels (5.14+) for arm
 RDEPENDS:${PN} = "\
     nativesdk-pkgconfig \
-    ${@'nativesdk-qemu nativesdk-qemu-helper' if d.getVar('SDK_ARCH') in ['x86_64', 'aarch64', 'ppc64', 'ppc64le', 'riscv64', 'loongarch64', 'mips64', 's390x', 'sparc64'] else ''} \
+    ${@'nativesdk-qemu nativesdk-qemu-helper' if bb.utils.contains('SDK_FEATURES', 'qemu', True, False, d) and d.getVar('SDK_ARCH') in ['x86_64', 'aarch64', 'ppc64', 'ppc64le', 'riscv64', 'loongarch64', 'mips64', 's390x', 'sparc64'] else ''} \
     nativesdk-pseudo \
     nativesdk-unfs3 \
     nativesdk-opkg \
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#4 lore raw

PATCH v3 4/6 nativesdk-packagegroup-sdk-host: add an sbom SDK feature

In reply to Trevor Woerner

Software composition tooling is useful to have in an SDK for generating or checking SPDX documents and scanning artifacts, but it is not needed by every SDK and pulls in a Python dependency chain, so it should be opt-in rather than always present.

Add an "sbom" SDK feature that pulls in the SPDX and SBOM/CVE tools (nativesdk-python3-spdx-tools and nativesdk-python3-sbom-cve-check). It is not enabled by default; a configuration that wants it adds the "sbom" feature to SDK_FEATURES.

AI-Generated: codex/claude-opus 4.8 (xhigh)

---
changes in v3:
- new in v3

---
 .../packagegroups/nativesdk-packagegroup-sdk-host.bb             | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-core/packagegroups/nativesdk-packagegroup-sdk-host.bb +1 -0
index 7205599288a2..31559eb2cd82 100644
@@ -30,6 +30,7 @@ RDEPENDS:${PN} = "\
     nativesdk-flex \
     nativesdk-perl-module-integer \
     ${@bb.utils.contains('SDK_FEATURES', 'wic', 'nativesdk-wic', '', d)} \
+    ${@bb.utils.contains('SDK_FEATURES', 'sbom', 'nativesdk-python3-spdx-tools nativesdk-python3-sbom-cve-check', '', d)} \
     "
 
 RDEPENDS:${PN}:darwin = "\
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#5 lore raw

PATCH v3 5/6 packagegroup-cross-canadian: gate gdb behind SDK_FEATURES

In reply to Trevor Woerner

The cross-canadian gdb is added to every SDK's toolchain packagegroup unconditionally. A host cross-debugger is part of what most people expect from an SDK, but it is also large, and an SDK built only to compile does not need it.

Move it behind the "gdb" SDK feature, which is enabled by default so an unmodified configuration still gets the cross gdb exactly as before; binutils and gcc remain unconditional, as a cross-compiler is intrinsic to an SDK. A configuration that wants a compile-only SDK removes the "gdb" feature from SDK_FEATURES.

AI-Generated: codex/claude-opus 4.8 (xhigh)

---
changes in v3:
- new in v3

---
 meta/conf/bitbake.conf                                         | 2 +-
 meta/recipes-core/packagegroups/packagegroup-cross-canadian.bb | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
meta/conf/bitbake.conf +1 -1
index c9317bc9722a..ffdfcf8afa03 100644
@@ -898,7 +898,7 @@ OES_BITBAKE_CONF = "1"
 
 MACHINE_FEATURES:append = " ${@oe.utils.filter_default_features('MACHINE_FEATURES', d)}"
 SDK_MACHINE_FEATURES ?= ""
-SDK_FEATURES ?= "qemu"
+SDK_FEATURES ?= "gdb qemu"
 
 DISTRO_FEATURES:append = " ${@oe.utils.filter_default_features('DISTRO_FEATURES', d)}"
 
meta/recipes-core/packagegroups/packagegroup-cross-canadian.bb +1 -1
index 49c075eb1166..9db3cfa28831 100644
@@ -13,7 +13,7 @@ GDB = "gdb-cross-canadian-${TRANSLATED_TARGET_ARCH}"
 RDEPENDS:${PN} = "\
     ${@all_multilib_tune_values(d, 'BINUTILS')} \
     ${@all_multilib_tune_values(d, 'GCC')} \
-    ${@all_multilib_tune_values(d, 'GDB')} \
+    ${@all_multilib_tune_values(d, 'GDB') if bb.utils.contains('SDK_FEATURES', 'gdb', True, False, d) else ''} \
     meta-environment-${MACHINE} \
     "
 
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#6 lore raw

PATCH v3 6/6 nativesdk-packagegroup-sdk-host: add an lldb SDK feature

In reply to Trevor Woerner

The SDK ships the cross gdb (see the "gdb" feature), but some developers prefer lldb. It is a large addition and not wanted by default, so make it opt-in.

Add an "lldb" SDK feature that pulls in nativesdk-lldb. It is not enabled by default; a configuration that wants it adds the "lldb" feature to SDK_FEATURES.

AI-Generated: codex/claude-opus 4.8 (xhigh)

---
changes in v3:
- new in v3

---
 .../packagegroups/nativesdk-packagegroup-sdk-host.bb             | 1 +
 1 file changed, 1 insertion(+)
meta/recipes-core/packagegroups/nativesdk-packagegroup-sdk-host.bb +1 -0
index 31559eb2cd82..bf9de4201810 100644
@@ -31,6 +31,7 @@ RDEPENDS:${PN} = "\
     nativesdk-perl-module-integer \
     ${@bb.utils.contains('SDK_FEATURES', 'wic', 'nativesdk-wic', '', d)} \
     ${@bb.utils.contains('SDK_FEATURES', 'sbom', 'nativesdk-python3-spdx-tools nativesdk-python3-sbom-cve-check', '', d)} \
+    ${@bb.utils.contains('SDK_FEATURES', 'lldb', 'nativesdk-lldb', '', d)} \
     "
 
 RDEPENDS:${PN}:darwin = "\
Signature · 1 line
2.50.0.173.g8b6f19ccfc3a
#7 lore raw
In reply to Trevor Woerner
Quoted 23 lines · Trevor Woerner via lists.openembedded.org

wic shells out to a range of host-side tools but does not declare them, so wherever it is installed as a package (rather than run from the bitbake do_image_wic task, which has the wic-tools native sysroot on PATH) it works only by chance depending on what the host has installed.

Patch 1 fixes that directly: wic gains RDEPENDS on the tools it runs, so they are installed alongside it in every variant that packages it.

The rest of the series turns "what optional tools go into the SDK" into an explicit, self-documenting choice. It introduces SDK_FEATURES, a list in the same spirit as DISTRO_FEATURES / MACHINE_FEATURES / IMAGE_FEATURES:
a developer reads the setting and knows exactly what their SDK contains.

- patch 2 adds SDK_FEATURES and gates wic on the "wic" feature (off by

    default);
  - patch 3 moves the existing (unconditional, arch-gated) qemu addition
    behind a "qemu" feature, on by default so nothing regresses;

- patch 4 adds an opt-in "sbom" feature (SPDX + SBOM/CVE tooling);
- patch 5 moves the cross-canadian gdb behind a "gdb" feature, on by default; binutils and gcc stay unconditional since a cross-compiler is intrinsic to an SDK;
- patch 6 adds an opt-in "lldb" feature for developers who prefer it.

Quoted 7 lines

The default SDK_FEATURES is "gdb qemu", so an unmodified configuration produces the same SDK as before. The intent is to present the full range of knobs and let review settle which belong on by default; the split across separate patches is deliberate so each can be debated on its own. SDK_FEATURES is a plain list, so downstream layers can define their own features (for example a cgdb feature in meta-openembedded) the same way.

Hi Trevor,

Thanks for the new version. It looks like we have 2 selftest fails:

2026-07-19 07:16:57,038 - oe-selftest - INFO - sstatetests.SStateHashSameSigs2.test_sstate_allarch_samesigs (subunit.RemotedTestCase)
2026-07-19 07:16:57,040 - oe-selftest - INFO -  ... FAIL

...
ERROR: When reparsing /tmp/selftest-fetchvua2hn0x/test.bb:do_checkuri, the basehash value changed from eaf531c7d0304fd45f77fab173d0399db2b76e45cd46487426eb74bae7d4b7bb to 2cdec1b5839750d0c5998dd35b7ab7380727d2fa3a33aed864144428d3a78f7d. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake test -cdo_checkuri -Snone
ERROR: Then:
ERROR: $ bitbake test -cdo_checkuri -Sprintdiff

ERROR: When reparsing /tmp/selftest-fetchvua2hn0x/test.bb:do_fetch, the basehash value changed from 72505ac7986134cca1a5bf96409e922ca39d8b70914218c968f9a8b3d508bc81 to b34ab448c19b41b81625a8e8ffa143940fa6c6a3c9db18ecec38e83ee0e3b7bb. The metadata is not deterministic and this needs to be fixed.
ERROR: The following commands may help:
ERROR: $ bitbake test -cdo_fetch -Snone
ERROR: Then:
ERROR: $ bitbake test -cdo_fetch -Sprintdiff
...

2026-07-19 07:21:21,814 - oe-selftest - INFO - sstatetests.SStateHashSameSigs2.test_sstate_nativesdk_samesigs_multilib (subunit.RemotedTestCase)
2026-07-19 07:21:21,814 - oe-selftest - INFO -  ... FAIL

https://autobuilder.yoctoproject.org/valkyrie/#/builders/35/builds/4320
https://autobuilder.yoctoproject.org/valkyrie/#/builders/23/builds/4376
https://autobuilder.yoctoproject.org/valkyrie/#/builders/48/builds/4142

I also got this error that might be related:

ERROR: Nothing RPROVIDES 'grub' (but /srv/pokybuild/yocto-worker/check-layer/build/layers/openembedded-core/meta/recipes-support/wic/wic_0.3.1.bb RDEPENDS on or otherwise requires it) grub was skipped: incompatible with host arm-oe-linux-gnueabi (not in COMPATIBLE_HOST)
NOTE: Runtime target 'grub' is unbuildable, removing...
Missing or unbuildable dependency chain was: ['grub']
ERROR: Required build target 'meta-world-pkgdata' has no buildable providers.
Missing or unbuildable dependency chain was: ['meta-world-pkgdata', 'wic', 'grub']

https://autobuilder.yoctoproject.org/valkyrie/#/builders/27/builds/4231

Can you have a look at the issue?

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com