Support opt-out of any default machine and distro features
cover Support opt-out of any default machine and distro features
These changes should be applied *after* the corresponding changes are applied in meta-yocto [1].
[1]: https://lore.kernel.org/poky/[email protected]/T/
These changes implement the third proposal from my email on "Improving DISTRO_FEATURES backfill & opt-out" [2]:
Merge the current contents of DISTRO_FEATURES_BACKFILL into DISTRO_FEATURES_DEFAULT. This variable will be filtered to disable any items listed in DISTRO_FEATURES_OPTOUT before use, efficiently and without the use of :remove. Deprecate DISTRO_FEATURES_BACKFILL and DISTRO_FEATURES_BACKFILL_CONSIDERED, issuing warnings if either is used.
[2]: https://lists.openembedded.org/g/openembedded-architecture/topic/improving_distro_features/118009864
DISTRO_FEATURES_DEFAULT is now set from DISTRO_FEATURES_DEFAULT_RAW, excluding entries listed in DISTRO_FEATURES_OPTED_OUT. _OPTED_OUT was chosen over _OPTOUT for improved clarity.
This series doesn't include issuing warnings if DISTRO_FEATURES_BACKFILL or DISTRO_FEATURES_BACKFILL_CONSIDERED is set, that can be added as a follow up if these changes are accepted.
The change is also carried through to MACHINE_FEATURES to keep things consistent.
---
Paul Barker (4):
oelib: utils: Support filtering default features
meta: Support opting out of any distro features
meta: Support opting out of any machine features
lib: oe: Drop backfill support
meta/classes-global/base.bbclass | 7 +-- meta/classes-recipe/crosssdk.bbclass | 7 ++- meta/classes-recipe/native.bbclass | 7 ++- meta/classes-recipe/nativesdk.bbclass | 8 ++-- meta/conf/bitbake.conf | 7 ++- meta/conf/distro/include/default-distrovars.inc | 15 +++--- meta/conf/distro/include/tclibc-musl.inc | 2 +- meta/conf/distro/include/tclibc-newlib.inc | 2 +- meta/conf/distro/include/tclibc-picolibc.inc | 2 +- meta/conf/documentation.conf | 14 +++--- .../machine/include/loongarch/qemuloongarch.inc | 3 +- meta/conf/machine/include/mips/arch-mips.inc | 2 +- meta/conf/machine/include/powerpc/tune-power5.inc | 2 +- meta/conf/machine/include/powerpc/tune-power6.inc | 2 +- meta/conf/machine/include/powerpc/tune-power7.inc | 2 +- .../conf/machine/include/powerpc/tune-ppce5500.inc | 2 +- .../conf/machine/include/powerpc/tune-ppce6500.inc | 2 +- meta/conf/machine/include/qemu.inc | 4 +- meta/conf/machine/include/riscv/qemuriscv.inc | 2 +- meta/conf/machine/include/x86/arch-x86.inc | 2 +- meta/conf/machine/include/x86/x86-base.inc | 5 +- meta/conf/machine/qemux86-64.conf | 2 +- meta/conf/machine/qemux86.conf | 2 +- meta/lib/oe/utils.py | 54 ++++++++++++++-------- meta/lib/oeqa/selftest/cases/oelib/utils.py | 31 ++++++++++++- 25 files changed, 118 insertions(+), 70 deletions(-)
---
base-commit: b50d6debf7baa555fbfb3521c4f952675bba2d37
change-id: 20260328-default-features-d7f43efcfb63
Best regards,
Signature · 1 line
Paul Barker
PATCH 1/4 oelib: utils: Support filtering default features
The new library function filter_default_features() allows us to support opting out of any default features without having to use :remove.
--- meta/lib/oe/utils.py | 35 +++++++++++++++++++++++++++++ meta/lib/oeqa/selftest/cases/oelib/utils.py | 31 ++++++++++++++++++++++++- 2 files changed, 65 insertions(+), 1 deletion(-)
meta/lib/oe/utils.py +35 -0
@@ -83,6 +83,31 @@ def set_intersect(variable1, variable2, d): val2 = set(d.getVar(variable2).split()) return " ".join(val1 & val2) +def set_difference(variable1, variable2, d): + """ + Expand both variables, interpret them as lists of strings, and return the + intersection as a flattened string. + + For example: + s1 = "a b c" + s2 = "b c d" + s3 = set_difference(s1, s2) + => s3 = "a" + """ + val1 = d.getVar(variable1) + if not val1: + return "" + val2 = d.getVar(variable2) + if not val2: + return val1 + + val1 = set(val1.split()) + val2 = set(val2.split()) + + # Return a sorted string to ensure that the result is consistent between + # parser runs. + return " ".join(sorted(val1 - val2)) + def prune_suffix(var, suffixes, d): # See if var ends with any of the suffixes listed and # remove it if found
@@ -133,6 +158,16 @@ def features_backfill(var,d): if addfeatures: d.appendVar(var, " " + " ".join(addfeatures)) +def filter_default_features(varname, d): + # Process default features to exclude features which the user has opted out + # of. This should be called from an anonymous Python function and uses + # d.setVar() to avoid unnecessarily evaluating this on every data store + # update. + default_features = set_difference(varname + "_DEFAULT_RAW", + varname + "_OPTED_OUT", + d) + d.setVar(varname + "_DEFAULT", default_features) + def all_distro_features(d, features, truevalue="1", falsevalue=""): """ Returns truevalue if *all* given features are set in DISTRO_FEATURES,
meta/lib/oeqa/selftest/cases/oelib/utils.py +30 -1
@@ -8,7 +8,7 @@ import sys from unittest.case import TestCase from contextlib import contextmanager from io import StringIO -from oe.utils import packages_filter_out_system, trim_version, multiprocess_launch +from oe.utils import packages_filter_out_system, trim_version, multiprocess_launch, filter_default_features class TestPackagesFilterOutSystem(TestCase): def test_filter(self):
@@ -102,3 +102,32 @@ class TestMultiprocessLaunch(TestCase): with captured_output() as (out, err): self.assertRaises(bb.BBHandledException, multiprocess_launch, testfunction, ["1", "2", "3", "4", "5", "6"], d, extraargs=(d,)) self.assertIn("KeyError: 'Invalid number 2'", out.getvalue()) + + +class TestDefaultFeatures(TestCase): + def test_filter_default_features(self): + try: + import bb + d = bb.data_smart.DataSmart() + except ImportError: + self.skipTest("Cannot import bb") + + # Test with nothing opted out + d.setVar("DISTRO_FEATURES_DEFAULT_RAW", "alpha beta gamma") + filter_default_features("DISTRO_FEATURES", d) + self.assertEqual(d.getVar("DISTRO_FEATURES_DEFAULT"), "alpha beta gamma") + + # opt out of a single feature + d.setVar("DISTRO_FEATURES_OPTED_OUT", "beta") + filter_default_features("DISTRO_FEATURES", d) + self.assertEqual(d.getVar("DISTRO_FEATURES_DEFAULT"), "alpha gamma") + + # opt out of everything + d.setVar("DISTRO_FEATURES_OPTED_OUT", "gamma alpha beta") + filter_default_features("DISTRO_FEATURES", d) + self.assertEqual(d.getVar("DISTRO_FEATURES_DEFAULT"), "") + + # opt out of something that isn't in our defaults + d.setVar("DISTRO_FEATURES_OPTED_OUT", "omega") + filter_default_features("DISTRO_FEATURES", d) + self.assertEqual(d.getVar("DISTRO_FEATURES_DEFAULT"), "alpha beta gamma")
Signature · 1 line
2.43.0
PATCH 2/4 meta: Support opting out of any distro features
In reply to Paul BarkerConstruct DISTRO_FEATURES_DEFAULT from DISTRO_FEATURES_DEFAULT_RAW using the new filter_default_features() function.
This change obsoletes the variables DISTRO_FEATURES_BACKFILL and DISTRO_FEATURES_BACKFILL_CONSIDERED. Instead, all defaults are added to DISTRO_FEATURES_DEFAULT_RAW and users can opt out of any of these using DISTRO_FEATURES_OPTED_OUT. Hopefully the variable naming here is easier for people to understand and remember.
DISTRO_FEATURES is now weakly set to DISTRO_FEATURES_DEFAULT in bitbake.conf, after inclusion of local, DISTRO and MACHINE conf files. This means that all assignments in conf files will need to use :append instead of += to preserve the defaults but this is in line with how other global variables are modified in conf files.
Migration notes:
- If you have previously extended DISTRO_FEATURES using :append, or set DISTRO_FEATURES to a new value including ${DISTRO_FEATURES_DEFAULT}, you shouldn't need to make any changes there.
E.g. usage in poky.conf is ok, it doesn't need any change:
DISTRO_FEATURES ?= "${DISTRO_FEATURES_DEFAULT} ${POKY_DEFAULT_DISTRO_FEATURES}"- If you have previously set DISTRO_FEATURES, but did not include ${DISTRO_FEATURES_DEFAULT}, you will lose the features which were previously in DISTRO_FEATURES_BACKFILL:
pulseaudio gobject-introspection-data ldconfig opengl ptest multiarch wayland vulkan
You will need to add these to your DISTRO_FEATURES or include ${DISTRO_FEATURES_DEFAULT} as in the example above.
- If you previously set DISTRO_FEATURES_BACKFILL_CONSIDERED, use the new variable DISTRO_FEATURES_OPTED_OUT instead.
- If you previously modified DISTRO_FEATURES_BACKFILL, don't do that.
--- meta/classes-global/base.bbclass | 5 ++++- meta/classes-recipe/crosssdk.bbclass | 7 +++---- meta/classes-recipe/native.bbclass | 7 +++---- meta/classes-recipe/nativesdk.bbclass | 7 +++---- meta/conf/bitbake.conf | 3 +-- meta/conf/distro/include/default-distrovars.inc | 15 +++++++++------ meta/conf/distro/include/tclibc-musl.inc | 2 +- meta/conf/distro/include/tclibc-newlib.inc | 2 +- meta/conf/distro/include/tclibc-picolibc.inc | 2 +- meta/conf/documentation.conf | 7 ++++--- 10 files changed, 30 insertions(+), 27 deletions(-)
meta/classes-global/base.bbclass +4 -1
@@ -454,8 +454,11 @@ def set_packagetriplet(d): python () { import string, re + # Filter default features to allow users to opt out of features they don't + # want. + oe.utils.filter_default_features("DISTRO_FEATURES", d) + # Handle backfilling - oe.utils.features_backfill("DISTRO_FEATURES", d) oe.utils.features_backfill("MACHINE_FEATURES", d) # To add a recipe to the skip list , set:
meta/classes-recipe/crosssdk.bbclass +3 -4
@@ -16,13 +16,12 @@ PACKAGE_ARCH = "${SDK_ARCH}" python () { # set TUNE_PKGARCH to SDK_ARCH d.setVar('TUNE_PKGARCH', d.getVar('SDK_ARCH')) - # Set features here to prevent appends and distro features backfill - # from modifying nativesdk distro features + # Set features here to prevent DISTRO_FEATURES modifications from affecting + # crosssdk distro features features = set(d.getVar("DISTRO_FEATURES_NATIVESDK").split()) - oe.utils.features_backfill("DISTRO_FEATURES", d) + oe.utils.filter_default_features("DISTRO_FEATURES", d) filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVESDK"), d).split()) d.setVar("DISTRO_FEATURES", " ".join(sorted(features | filtered))) - d.setVar("DISTRO_FEATURES_BACKFILL", "") } STAGING_BINDIR_TOOLCHAIN = "${STAGING_DIR_NATIVE}${bindir_native}/${TARGET_ARCH}${TARGET_VENDOR}-${TARGET_OS}"
meta/classes-recipe/native.bbclass +3 -4
@@ -126,13 +126,12 @@ python native_virtclass_handler () { return bpn = d.getVar("BPN") - # Set features here to prevent appends and distro features backfill - # from modifying native distro features + # Set features here to prevent DISTRO_FEATURES modifications from affecting + # native distro features features = set(d.getVar("DISTRO_FEATURES_NATIVE").split()) - oe.utils.features_backfill("DISTRO_FEATURES", d) + oe.utils.filter_default_features("DISTRO_FEATURES", d) filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVE"), d).split()) d.setVar("DISTRO_FEATURES", " ".join(sorted(features | filtered))) - d.setVar("DISTRO_FEATURES_BACKFILL", "") classextend = d.getVar('BBCLASSEXTEND') or "" if "native" not in classextend:
meta/classes-recipe/nativesdk.bbclass +3 -4
@@ -78,13 +78,12 @@ python nativesdk_virtclass_handler () { if not (pn.endswith("-nativesdk") or pn.startswith("nativesdk-")): return - # Set features here to prevent appends and distro features backfill - # from modifying nativesdk distro features + # Set features here to prevent DISTRO_FEATURES modifications from affecting + # nativesdk distro features features = set(d.getVar("DISTRO_FEATURES_NATIVESDK").split()) - oe.utils.features_backfill("DISTRO_FEATURES", d) + oe.utils.filter_default_features("DISTRO_FEATURES", d) filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVESDK"), d).split()) d.setVar("DISTRO_FEATURES", " ".join(sorted(features | filtered))) - d.setVar("DISTRO_FEATURES_BACKFILL", "") e.data.setVar("MLPREFIX", "nativesdk-") e.data.setVar("PN", "nativesdk-" + e.data.getVar("PN").replace("-nativesdk", "").replace("nativesdk-", ""))
meta/conf/bitbake.conf +1 -2
@@ -889,7 +889,7 @@ OES_BITBAKE_CONF = "1" MACHINE_FEATURES ?= "" SDK_MACHINE_FEATURES ?= "" -DISTRO_FEATURES ?= "" +DISTRO_FEATURES ?= "${DISTRO_FEATURES_DEFAULT}" DISTRO_EXTRA_RDEPENDS ?= "" DISTRO_EXTRA_RRECOMMENDS ?= ""
@@ -912,7 +912,6 @@ DISTRO_FEATURES_NATIVESDK:mingw32 = "x11 ipv6" DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation debuginfod opencl opengl wayland" DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation debuginfod opencl opengl wayland" -DISTRO_FEATURES_BACKFILL = "pulseaudio gobject-introspection-data ldconfig opengl ptest multiarch wayland vulkan" MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode" COMBINED_FEATURES = "${@oe.utils.set_intersect('DISTRO_FEATURES', 'MACHINE_FEATURES', d)}"
meta/conf/distro/include/default-distrovars.inc +9 -6
@@ -14,19 +14,22 @@ LOCALE_UTF8_IS_DEFAULT ?= "1" LOCALE_UTF8_IS_DEFAULT:class-nativesdk = "0" # seccomp is not yet ported to rv32 -DISTRO_FEATURES_DEFAULT:remove:riscv32 = "seccomp" +DISTRO_FEATURES_OPTED_OUT:append:riscv32 = "seccomp" # seccomp is not yet ported to ARC -DISTRO_FEATURES_DEFAULT:remove:arc = "seccomp" +DISTRO_FEATURES_OPTED_OUT:append:arc = "seccomp" # seccomp is not yet ported to microblaze -DISTRO_FEATURES_DEFAULT:remove:microblaze = "seccomp" +DISTRO_FEATURES_OPTED_OUT:append:microblaze = "seccomp" # seccomp is not yet ported to loongarch64 -DISTRO_FEATURES_DEFAULT:remove:loongarch64 = "seccomp" +DISTRO_FEATURES_OPTED_OUT:append:loongarch64 = "seccomp" -DISTRO_FEATURES_DEFAULT ?= "acl alsa bluetooth debuginfod ext2 ipv4 ipv6 pcmcia usbgadget usbhost wifi xattr nfs zeroconf pci 3g nfc x11 vfat seccomp" -DISTRO_FEATURES ?= "${DISTRO_FEATURES_DEFAULT}" +DISTRO_FEATURES_DEFAULT_RAW ?= " \ + acl alsa bluetooth debuginfod ext2 ipv4 ipv6 pcmcia usbgadget usbhost \ + wifi xattr nfs zeroconf pci 3g nfc x11 vfat seccomp pulseaudio \ + gobject-introspection-data ldconfig opengl ptest multiarch wayland vulkan \ + " IMAGE_FEATURES ?= "" COMMERCIAL_AUDIO_PLUGINS ?= ""
meta/conf/distro/include/tclibc-musl.inc +1 -1
@@ -14,7 +14,7 @@ PREFERRED_PROVIDER_virtual/libc-locale ?= "musl-locales" PREFERRED_PROVIDER_virtual/nativesdk-libintl ?= "nativesdk-glibc" PREFERRED_PROVIDER_virtual/nativesdk-libiconv ?= "nativesdk-glibc" -DISTRO_FEATURES_BACKFILL_CONSIDERED += "ldconfig" +DISTRO_FEATURES_OPTED_OUT:append = "ldconfig" #USE_NLS ?= "no"
meta/conf/distro/include/tclibc-newlib.inc +1 -1
@@ -11,7 +11,7 @@ PREFERRED_PROVIDER_virtual/libintl ?= "newlib" PREFERRED_PROVIDER_virtual/nativesdk-libintl ?= "nativesdk-glibc" PREFERRED_PROVIDER_virtual/nativesdk-libiconv ?= "nativesdk-glibc" -DISTRO_FEATURES_BACKFILL_CONSIDERED += "ldconfig" +DISTRO_FEATURES_OPTED_OUT:append = "ldconfig" #USE_NLS ?= "no"
meta/conf/distro/include/tclibc-picolibc.inc +1 -1
@@ -11,7 +11,7 @@ PREFERRED_PROVIDER_virtual/libintl ?= "picolibc" PREFERRED_PROVIDER_virtual/nativesdk-libintl ?= "nativesdk-glibc" PREFERRED_PROVIDER_virtual/nativesdk-libiconv ?= "nativesdk-glibc" -DISTRO_FEATURES_BACKFILL_CONSIDERED += "ldconfig" +DISTRO_FEATURES_OPTED_OUT:append = "ldconfig" IMAGE_LINGUAS = ""
meta/conf/documentation.conf +4 -3
@@ -140,9 +140,10 @@ DESCRIPTION[doc] = "The package description used by package managers. If not set DISTRO[doc] = "The short name of the distribution. If the variable is blank, meta/conf/distro/defaultsetup.conf will be used." DISTRO_EXTRA_RDEPENDS[doc] = "Specifies a list of distro-specific packages to add to all images. The variable only applies to the images that include packagegroup-base." DISTRO_EXTRA_RRECOMMENDS[doc] = "Specifies a list of distro-specific packages to add to all images if the packages exist. The list of packages are automatically installed but you can remove them." -DISTRO_FEATURES[doc] = "The features enabled for the distribution." -DISTRO_FEATURES_BACKFILL[doc] = "Features to be added to DISTRO_FEATURES if not also present in DISTRO_FEATURES_BACKFILL_CONSIDERED. This variable is set in the meta/conf/bitbake.conf file and it is not intended to be user-configurable." -DISTRO_FEATURES_BACKFILL_CONSIDERED[doc] = "Features from DISTRO_FEATURES_BACKFILL that should not be backfilled (i.e. added to DISTRO_FEATURES) during the build." +DISTRO_FEATURES[doc] = "The features enabled for the distribution. Defaults to DISTRO_FEATURES_DEFAULT." +DISTRO_FEATURES_DEFAULT_RAW[doc] = "The default set of distribution features, prior to any filtering. Usually you should leave this alone and modify DISTRO_FEATURES and/or DISTRO_FEATURES_OPTED_OUT." +DISTRO_FEATURES_DEFAULT[doc] = "The set of default distribution features, after filtering out features listed in DISTRO_FEATURES_OPTED_OUT." +DISTRO_FEATURES_OPTED_OUT[doc] = "The set of default distribtion features to disable. Prefer opting out of features using this variable instead of using DISTRO_FEATURES:remove." DISTRO_NAME[doc] = "The long name of the distribution." DISTRO_PN_ALIAS[doc] = "Alias names used for the recipe in various Linux distributions." DISTRO_VERSION[doc] = "The version of the distribution."
Signature · 1 line
2.43.0
PATCH 3/4 meta: Support opting out of any machine features
In reply to Paul BarkerConstruct MACHINE_FEATURES_DEFAULT from MACHINE_FEATURES_DEFAULT_RAW using the new filter_default_features() function.
This change obsoletes the variables MACHINE_FEATURES_BACKFILL and MACHINE_FEATURES_BACKFILL_CONSIDERED. Instead, all defaults are added to MACHINE_FEATURES_DEFAULT_RAW and users can opt out of any of these using MACHINE_FEATURES_OPTED_OUT. Hopefully the variable naming here is easier for people to understand and remember.
MACHINE_FEATURES is now weakly set to MACHINE_FEATURES_DEFAULT in bitbake.conf, after inclusion of local, DISTRO and MACHINE conf files. This means that all assignments in conf files will need to use :append instead of += to preserve the defaults but this is in line with how other global variables are modified in conf files.
Migration notes:
- If you have previously extended MACHINE_FEATURES using :append, or set MACHINE_FEATURES to a new value including ${MACHINE_FEATURES_DEFAULT}, you shouldn't need to make any changes there.
- If you have previously set MACHINE_FEATURES, but did not include ${MACHINE_FEATURES_DEFAULT}, you will lose the features which were previously in MACHINE_FEATURES_BACKFILL:
rtc qemu-usermode
You will need to add these to your MACHINE_FEATURES or include ${MACHINE_FEATURES_DEFAULT}.
For example in meta-yocto-bsp, the MACHINE_FEATURES assignment for beaglebone-yocto must be modified to become:
MACHINE_FEATURES = "${MACHINE_FEATURES_DEFAULT} usbgadget usbhost vfat alsa"- If you previously set MACHINE_FEATURES_BACKFILL_CONSIDERED, use the new variable MACHINE_FEATURES_OPTED_OUT instead.
- If you previously modified MACHINE_FEATURES_BACKFILL, don't do that.
--- meta/classes-global/base.bbclass | 4 +--- meta/classes-recipe/nativesdk.bbclass | 1 - meta/conf/bitbake.conf | 4 ++-- meta/conf/documentation.conf | 7 ++++--- meta/conf/machine/include/loongarch/qemuloongarch.inc | 3 ++- meta/conf/machine/include/mips/arch-mips.inc | 2 +- meta/conf/machine/include/powerpc/tune-power5.inc | 2 +- meta/conf/machine/include/powerpc/tune-power6.inc | 2 +- meta/conf/machine/include/powerpc/tune-power7.inc | 2 +- meta/conf/machine/include/powerpc/tune-ppce5500.inc | 2 +- meta/conf/machine/include/powerpc/tune-ppce6500.inc | 2 +- meta/conf/machine/include/qemu.inc | 4 +++- meta/conf/machine/include/riscv/qemuriscv.inc | 2 +- meta/conf/machine/include/x86/arch-x86.inc | 2 +- meta/conf/machine/include/x86/x86-base.inc | 5 +++-- meta/conf/machine/qemux86-64.conf | 2 +- meta/conf/machine/qemux86.conf | 2 +- 17 files changed, 25 insertions(+), 23 deletions(-)
meta/classes-global/base.bbclass +1 -3
@@ -457,9 +457,7 @@ python () { # Filter default features to allow users to opt out of features they don't # want. oe.utils.filter_default_features("DISTRO_FEATURES", d) - - # Handle backfilling - oe.utils.features_backfill("MACHINE_FEATURES", d) + oe.utils.filter_default_features("MACHINE_FEATURES", d) # To add a recipe to the skip list , set: # SKIP_RECIPE[pn] = "message"
meta/classes-recipe/nativesdk.bbclass +0 -1
@@ -17,7 +17,6 @@ CLASSOVERRIDE = "class-nativesdk" MACHINEOVERRIDES = "" MACHINE_FEATURES = "${SDK_MACHINE_FEATURES}" -MACHINE_FEATURES_BACKFILL = "" MULTILIBS = ""
meta/conf/bitbake.conf +2 -2
@@ -886,7 +886,7 @@ OES_BITBAKE_CONF = "1" # Machine properties and packagegroup-base stuff ################################################################## -MACHINE_FEATURES ?= "" +MACHINE_FEATURES ?= "${MACHINE_FEATURES_DEFAULT}" SDK_MACHINE_FEATURES ?= "" DISTRO_FEATURES ?= "${DISTRO_FEATURES_DEFAULT}"
@@ -912,7 +912,7 @@ DISTRO_FEATURES_NATIVESDK:mingw32 = "x11 ipv6" DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation debuginfod opencl opengl wayland" DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation debuginfod opencl opengl wayland" -MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode" +MACHINE_FEATURES_DEFAULT_RAW = "rtc qemu-usermode" COMBINED_FEATURES = "${@oe.utils.set_intersect('DISTRO_FEATURES', 'MACHINE_FEATURES', d)}" COMBINED_FEATURES[vardeps] += "DISTRO_FEATURES MACHINE_FEATURES"
meta/conf/documentation.conf +4 -3
@@ -277,9 +277,10 @@ MACHINE_ESSENTIAL_EXTRA_RDEPENDS[doc] = "A list of required machine-specific pac MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS[doc] = "A list of recommended machine-specific packages to install as part of the image being built. Because this is a 'machine essential' variable, the list of packages are essential for the machine to boot." MACHINE_EXTRA_RDEPENDS[doc] = "A list of machine-specific packages to install as part of the image being built that are not essential for the machine to boot. However, the build process for more fully-featured images depends on the packages being present." MACHINE_EXTRA_RRECOMMENDS[doc] = "A list of machine-specific packages to install as part of the image being built that are not essential for booting the machine. The image being built has no build dependencies on the packages in this list." -MACHINE_FEATURES[doc] = "Specifies the list of hardware features the MACHINE supports." -MACHINE_FEATURES_BACKFILL[doc] = "Features to be added to MACHINE_FEATURES if not also present in MACHINE_FEATURES_BACKFILL_CONSIDERED. This variable is set in the meta/conf/bitbake.conf file and is not intended to be user-configurable." -MACHINE_FEATURES_BACKFILL_CONSIDERED[doc] = "Features from MACHINE_FEATURES_BACKFILL that should not be backfilled (i.e. added to MACHINE_FEATURES) during the build." +MACHINE_FEATURES[doc] = "Specifies the list of hardware features the MACHINE supports. Defaults to MACHINE_FEATURES_DEFAULT" +MACHINE_FEATURES_DEFAULT_RAW[doc] = "The default set of machine features, prior to any filtering. Usually you should leave this alone and modify MACHINE_FEATURES and/or MACHINE_FEATURES_OPTED_OUT." +MACHINE_FEATURES_DEFAULT[doc] = "The set of default machine features, after filtering out features listed in MACHINE_FEATURES_OPTED_OUT." +MACHINE_FEATURES_OPTED_OUT[doc] = "The set of default machine features to disable. Prefer opting out of features using this variable instead of using MACHINE_FEATURES:remove." MACHINEOVERRIDES[doc] = "Lists overrides specific to the current machine. By default, this list includes the value of MACHINE." MAINTAINER[doc] = "The email address of the distribution maintainer." MIRRORS[doc] = "Specifies additional paths from which the OpenEmbedded build system gets source code."
meta/conf/machine/include/loongarch/qemuloongarch.inc +2 -1
@@ -3,7 +3,8 @@ PREFERRED_PROVIDER_virtual/bootloader ?= "u-boot" require conf/machine/include/qemu.inc require conf/machine/include/loongarch/tune-loongarch.inc -MACHINE_FEATURES = "screen keyboard ext2 ext3 serial" +MACHINE_FEATURES:append = " keyboard ext2 ext3 serial" +MACHINE_FEATURES_OPTED_OUT:append = " alsa bluetooth usbgadget vfat" KERNEL_IMAGETYPE = "vmlinuz" KERNEL_IMAGETYPES += "vmlinuz"
meta/conf/machine/include/mips/arch-mips.inc +1 -1
@@ -21,7 +21,7 @@ ABIEXTENSION .= "${@bb.utils.filter('TUNE_FEATURES', 'n32', d)}" TUNE_CCARGS .= "${@bb.utils.contains('TUNE_FEATURES', 'n32', ' -mabi=n32', '', d)}" # user mode qemu doesn't support mips64 n32: "Invalid ELF image for this architecture" -MACHINE_FEATURES_BACKFILL_CONSIDERED:append = " ${@bb.utils.contains('TUNE_FEATURES', 'n32', 'qemu-usermode', '', d)}" +MACHINE_FEATURES_OPTED_OUT:append = " ${@bb.utils.contains('TUNE_FEATURES', 'n32', 'qemu-usermode', '', d)}" TUNEVALID[n64] = "MIPS64 n64 ABI" TUNECONFLICTS[n64] = "o32 n32"
meta/conf/machine/include/powerpc/tune-power5.inc +1 -1
@@ -21,4 +21,4 @@ GLIBC_EXTRA_OECONF:powerpc64 += "${@bb.utils.contains('TUNE_FEATURES', 'power5', GLIBC_EXTRA_OECONF:powerpc += "${@bb.utils.contains('TUNE_FEATURES', 'power5', '--with-cpu=power5', '', d)}" # QEMU usermode fails with invalid instruction error -MACHINE_FEATURES_BACKFILL_CONSIDERED:append = "${@bb.utils.contains('TUNE_FEATURES', 'power5', ' qemu-usermode', '', d)}" +MACHINE_FEATURES_OPTED_OUT:append = "${@bb.utils.contains('TUNE_FEATURES', 'power5', ' qemu-usermode', '', d)}"
meta/conf/machine/include/powerpc/tune-power6.inc +1 -1
@@ -21,4 +21,4 @@ GLIBC_EXTRA_OECONF:powerpc64 += "${@bb.utils.contains('TUNE_FEATURES', 'power6', GLIBC_EXTRA_OECONF:powerpc += "${@bb.utils.contains('TUNE_FEATURES', 'power6', '--with-cpu=power6', '', d)}" # QEMU usermode fails with invalid instruction error -MACHINE_FEATURES_BACKFILL_CONSIDERED:append = "${@bb.utils.contains('TUNE_FEATURES', 'power6', ' qemu-usermode', '', d)}" +MACHINE_FEATURES_OPTED_OUT:append = "${@bb.utils.contains('TUNE_FEATURES', 'power6', ' qemu-usermode', '', d)}"
meta/conf/machine/include/powerpc/tune-power7.inc +1 -1
@@ -21,4 +21,4 @@ GLIBC_EXTRA_OECONF:powerpc64 += "${@bb.utils.contains('TUNE_FEATURES', 'power7', GLIBC_EXTRA_OECONF:powerpc += "${@bb.utils.contains('TUNE_FEATURES', 'power7', '--with-cpu=power7', '', d)}" # QEMU usermode fails with invalid instruction error -MACHINE_FEATURES_BACKFILL_CONSIDERED:append = "${@bb.utils.contains('TUNE_FEATURES', 'power7', ' qemu-usermode', '', d)}" +MACHINE_FEATURES_OPTED_OUT:append = "${@bb.utils.contains('TUNE_FEATURES', 'power7', ' qemu-usermode', '', d)}"
meta/conf/machine/include/powerpc/tune-ppce5500.inc +1 -1
@@ -19,4 +19,4 @@ PACKAGE_EXTRA_ARCHS:tune-ppc64e5500 = "${PACKAGE_EXTRA_ARCHS:tune-powerpc64} ppc QEMU_EXTRAOPTIONS:tune-ppc64e5500 = " -cpu e500mc" # QEMU usermode fails with invalid instruction error (YOCTO: #10304) -MACHINE_FEATURES_BACKFILL_CONSIDERED:append = "${@bb.utils.contains('TUNE_FEATURES', 'e5500', ' qemu-usermode', '', d)}" +MACHINE_FEATURES_OPTED_OUT:append = "${@bb.utils.contains('TUNE_FEATURES', 'e5500', ' qemu-usermode', '', d)}"
meta/conf/machine/include/powerpc/tune-ppce6500.inc +1 -1
@@ -19,4 +19,4 @@ PACKAGE_EXTRA_ARCHS:tune-ppc64e6500 = "${PACKAGE_EXTRA_ARCHS:tune-powerpc64} ppc QEMU_EXTRAOPTIONS:tune-ppc64e6500 = " -cpu e500mc" # QEMU usermode fails with invalid instruction error (YOCTO: #10304) -MACHINE_FEATURES_BACKFILL_CONSIDERED:append = "${@bb.utils.contains('TUNE_FEATURES', 'e6500', ' qemu-usermode', '', d)}" +MACHINE_FEATURES_OPTED_OUT:append = "${@bb.utils.contains('TUNE_FEATURES', 'e6500', ' qemu-usermode', '', d)}"
meta/conf/machine/include/qemu.inc +3 -1
@@ -4,7 +4,9 @@ XSERVER ?= "xserver-xorg \ xf86-video-modesetting \ " -MACHINE_FEATURES = "alsa bluetooth usbgadget screen vfat" +# Extend the raw default features here so that individual architectures can +# easily opt-out of what they don't support. +MACHINE_FEATURES_DEFAULT_RAW:append = " alsa bluetooth usbgadget screen vfat" MACHINEOVERRIDES =. "qemuall:"
meta/conf/machine/include/riscv/qemuriscv.inc +1 -1
@@ -3,7 +3,7 @@ PREFERRED_PROVIDER_virtual/bootloader ?= "u-boot" require conf/machine/include/qemu.inc require conf/machine/include/riscv/tune-riscv.inc -MACHINE_FEATURES += "keyboard ext2 ext3 serial" +MACHINE_FEATURES:append = " keyboard ext2 ext3 serial" KERNEL_IMAGETYPE = "Image" KERNEL_IMAGETYPES += "uImage"
meta/conf/machine/include/x86/arch-x86.inc +1 -1
@@ -25,7 +25,7 @@ TUNE_CCARGS .= "${@bb.utils.contains('TUNE_FEATURES', 'mx32', ' -mx32', '', d)}" TUNE_LDARGS += "${@bb.utils.contains('TUNE_FEATURES', 'mx32', '-m elf32_x86_64', '', d)}" TUNE_ASARGS += "${@bb.utils.contains('TUNE_FEATURES', 'mx32', '-x32', '', d)}" # user mode qemu doesn't support x32 -MACHINE_FEATURES_BACKFILL_CONSIDERED:append = " ${@bb.utils.contains('TUNE_FEATURES', 'mx32', 'qemu-usermode', '', d)}" +MACHINE_FEATURES_OPTED_OUT:append = " ${@bb.utils.contains('TUNE_FEATURES', 'mx32', 'qemu-usermode', '', d)}" MACHINEOVERRIDES =. "${@bb.utils.contains('TUNE_FEATURES', 'mx32', 'x86-x32:', '', d)}" # ELF64 ABI
meta/conf/machine/include/x86/x86-base.inc +3 -2
@@ -5,8 +5,9 @@ # # common settings for X86 machines # -MACHINE_FEATURES += "screen keyboard pci usbhost ext2 ext3 x86 \ - acpi serial usbgadget alsa" +MACHINE_FEATURES:append = " \ + screen keyboard pci usbhost ext2 ext3 x86 acpi serial usbgadget alsa \ + " IMAGE_FSTYPES ?= "wic.zst"
meta/conf/machine/qemux86-64.conf +1 -1
@@ -27,7 +27,7 @@ XSERVER = "xserver-xorg \ xserver-xorg-module-libint10 \ " -MACHINE_FEATURES += "x86 pci" +MACHINE_FEATURES:append = " x86 pci" MACHINE_ESSENTIAL_EXTRA_RDEPENDS += "v86d"
meta/conf/machine/qemux86.conf +1 -1
@@ -23,7 +23,7 @@ XSERVER = "xserver-xorg \ xserver-xorg-module-libint10 \ " -MACHINE_FEATURES += "x86 pci" +MACHINE_FEATURES:append = " x86 pci" MACHINE_ESSENTIAL_EXTRA_RDEPENDS += "v86d"
Signature · 1 line
2.43.0
PATCH 4/4 lib: oe: Drop backfill support
In reply to Paul BarkerWe have removed DISTRO_FEATURES_BACKFILL and MACHINE_FEATURES_BACKFILL, so we no longer need the features_backfill() function.
--- meta/lib/oe/utils.py | 21 --------------------- 1 file changed, 21 deletions(-)
meta/lib/oe/utils.py +0 -21
@@ -137,27 +137,6 @@ def inherits(d, *classes): """Return True if the metadata inherits any of the specified classes""" return any(bb.data.inherits_class(cls, d) for cls in classes) -def features_backfill(var,d): - # This construct allows the addition of new features to variable specified - # as var - # Example for var = "DISTRO_FEATURES" - # This construct allows the addition of new features to DISTRO_FEATURES - # that if not present would disable existing functionality, without - # disturbing distributions that have already set DISTRO_FEATURES. - # Distributions wanting to elide a value in DISTRO_FEATURES_BACKFILL should - # add the feature to DISTRO_FEATURES_BACKFILL_CONSIDERED - features = (d.getVar(var) or "").split() - backfill = (d.getVar(var+"_BACKFILL") or "").split() - considered = (d.getVar(var+"_BACKFILL_CONSIDERED") or "").split() - - addfeatures = [] - for feature in backfill: - if feature not in features and feature not in considered: - addfeatures.append(feature) - - if addfeatures: - d.appendVar(var, " " + " ".join(addfeatures)) - def filter_default_features(varname, d): # Process default features to exclude features which the user has opted out # of. This should be called from an anonymous Python function and uses
Signature · 1 line
2.43.0
Quoted 23 lines · Paul Barker via lists.openembedded.org
These changes should be applied *after* the corresponding changes are applied in meta-yocto [1].
[1]:
https://lore.kernel.org/poky/[email protected]/T/These changes implement the third proposal from my email on "Improving
DISTRO_FEATURESbackfill & opt-out" [2]:Merge the current contents of
DISTRO_FEATURES_BACKFILLintoDISTRO_FEATURES_DEFAULT. This variable will be filtered to disable any items listed inDISTRO_FEATURES_OPTOUTbefore use, efficiently and without the use of:remove. DeprecateDISTRO_FEATURES_BACKFILLandDISTRO_FEATURES_BACKFILL_CONSIDERED, issuing warnings if either is used.[2]:
https://lists.openembedded.org/g/openembedded-architecture/topic/improving_distro_features/118009864
DISTRO_FEATURES_DEFAULTis now set fromDISTRO_FEATURES_DEFAULT_RAW, excluding entries listed inDISTRO_FEATURES_OPTED_OUT._OPTED_OUTwas chosen over_OPTOUTfor improved clarity.
Quoted 7 lines
This series doesn't include issuing warnings if
DISTRO_FEATURES_BACKFILL
orDISTRO_FEATURES_BACKFILL_CONSIDEREDis set, that can be added as a follow up if these changes are accepted.The change is also carried through to
MACHINE_FEATURESto keep things consistent.
Thanks for putting this together. I really don't like "DEAULT_RAW" as a variable name, we're trying to get away from unclear concepts and I think adding 'raw' into the mix may cause more confusion that it helps solve :/. I'd probably have gone with OPTEDOUT or OPTOUT instead but I know others would argue that differently too.
I did notice some changes like:
-DISTRO_FEATURES_BACKFILL_CONSIDERED += "ldconfig" +DISTRO_FEATURES_OPTED_OUT:append = "ldconfig"
and was a bit worried we were having to force append operator use? Was that necessary for ordering reasons or did += work? I spent a lot of time trying to make sure the previous code worked with += as it was catching people out and I'm a bit worried this regresses.
Perhaps most problematically, I noticed this preserves the _DEFAULT variable and behaviour. This wasn't really what I was expecting. I can understand why you've done it but I'm worried it builds up more legacy and makes the code harder to understand. Personally, I'm becoming more in favour of just making the change and letting people adapt, the complexity is becoming to great in my view. At some point we do need to switch to something clean and I think this is one of those cases even if the initial switch is more painful. I suspect this piece may be the cause of the ordering issues too.
Cheers,
Richard