Skip to contentSkip to conversation

Support opt-out of any default machine and distro features

PATCH Superseded Paul Barker 2 participants 7 messages Mar 31 – Apr 1

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover Support opt-out of any default machine and distro features

This is the second version of my patch series to improve DISTRO_FEATURES & MACHINE_FEATURES handling. The first version complicated modification of variables in conf files, requiring use of :append when we should just be able to use +=.

In this version we directly append the filtered defaults to the target variable, instead of using an intermediate variable. So, default distro features are listed in DISTRO_FEATURES_DEFAULTS, this list is filtered to remove features listed in DISTRO_FEATURES_OPTED_OUT and the result is appended to DISTRO_FEATURES. Similar behaviour exists for MACHINE_FEATURES_DEFAULTS, MACHINE_FEATURES_OPTED_OUT and MACHINE_FEATURES.

See patches 2/4 & 3/4 for updated migration notes.

This series doesn't include issuing warnings if DISTRO_FEATURES_BACKFILL or DISTRO_FEATURES_BACKFILL_CONSIDERED is set, that can be added as a follow up if these changes are accepted.

---
Changes in v2:
- Rename {DISTRO,MACHINE}_FEATURES_DEFAULT_RAW to {DISTRO,MACHINE}_FEATURES_DEFAULTS.
- {DISTRO,MACHINE}_FEATURES_DEFAULT is removed.
- filter_default_features() appends directly to
{DISTRO,MACHINE}_FEATURES.
- Follow through the consequences of the above changes.
- Changed in meta-yocto should no longer be needed.
- Link to v1: https://patch.msgid.link/[email protected]

---
Paul Barker (4):
oelib: utils: Support filtering default features
meta: Support opting out of any distro features
meta: Support opting out of any machine features
lib: oe: Drop backfill support

 meta/classes-global/base.bbclass                   |  7 +--
 meta/classes-recipe/crosssdk.bbclass               |  9 ++--
 meta/classes-recipe/native.bbclass                 |  9 ++--
 meta/classes-recipe/nativesdk.bbclass              | 10 ++--
 meta/conf/bitbake.conf                             |  3 +-
 meta/conf/distro/include/default-distrovars.inc    | 15 +++---
 meta/conf/distro/include/tclibc-musl.inc           |  2 +-
 meta/conf/distro/include/tclibc-newlib.inc         |  2 +-
 meta/conf/distro/include/tclibc-picolibc.inc       |  2 +-
 meta/conf/documentation.conf                       |  8 ++--
 meta/conf/machine/include/mips/arch-mips.inc       |  2 +-
 meta/conf/machine/include/powerpc/tune-power5.inc  |  2 +-
 meta/conf/machine/include/powerpc/tune-power6.inc  |  2 +-
 meta/conf/machine/include/powerpc/tune-power7.inc  |  2 +-
 .../conf/machine/include/powerpc/tune-ppce5500.inc |  2 +-
 .../conf/machine/include/powerpc/tune-ppce6500.inc |  2 +-
 meta/conf/machine/include/x86/arch-x86.inc         |  2 +-
 meta/lib/oe/utils.py                               | 53 ++++++++++++++--------
 meta/lib/oeqa/selftest/cases/oelib/utils.py        | 38 +++++++++++++++-
 19 files changed, 113 insertions(+), 59 deletions(-)

---
base-commit: c56990178b31b893fbf695eaf6b67de501e9d2e9
change-id: 20260328-default-features-d7f43efcfb63

Best regards,

Signature · 1 line
Paul Barker
#1 lore raw

PATCH v2 1/4 oelib: utils: Support filtering default features

The new library function filter_default_features() allows us to support opting out of any default features without having to use :remove.

---
 meta/lib/oe/utils.py                        | 34 ++++++++++++++++++++++++++
 meta/lib/oeqa/selftest/cases/oelib/utils.py | 38 ++++++++++++++++++++++++++++-
 2 files changed, 71 insertions(+), 1 deletion(-)
meta/lib/oe/utils.py +34 -0
index afcfeda0c6d5..52f2f1bd55e0 100644
@@ -83,6 +83,31 @@ def set_intersect(variable1, variable2, d):
     val2 = set(d.getVar(variable2).split())
     return " ".join(val1 & val2)
 
+def set_difference(variable1, variable2, d):
+    """
+    Expand both variables, interpret them as lists of strings, and return the
+    intersection as a flattened string.
+
+    For example:
+    s1 = "a b c"
+    s2 = "b c d"
+    s3 = set_difference(s1, s2)
+    => s3 = "a"
+    """
+    val1 = d.getVar(variable1)
+    if not val1:
+        return ""
+    val2 = d.getVar(variable2)
+    if not val2:
+        return val1
+
+    val1 = set(val1.split())
+    val2 = set(val2.split())
+
+    # Return a sorted string to ensure that the result is consistent between
+    # parser runs.
+    return " ".join(sorted(val1 - val2))
+
 def prune_suffix(var, suffixes, d):
     # See if var ends with any of the suffixes listed and
     # remove it if found
@@ -133,6 +158,15 @@ def features_backfill(var,d):
     if addfeatures:
         d.appendVar(var, " " + " ".join(addfeatures))
 
+def filter_default_features(varname, d):
+    # Process default features to exclude features which the user has opted out
+    # of. The result is appended to the target variable (e.g. DISTRO_FEATURES
+    # or MACHINE_FEATURES).
+    default_features = set_difference(varname + "_DEFAULTS",
+                                      varname + "_OPTED_OUT",
+                                      d)
+    d.appendVar(varname, " " + default_features)
+
 def all_distro_features(d, features, truevalue="1", falsevalue=""):
     """
     Returns truevalue if *all* given features are set in DISTRO_FEATURES,
meta/lib/oeqa/selftest/cases/oelib/utils.py +37 -1
index 0cb46425a02d..f886b00e18ab 100644
@@ -8,7 +8,7 @@ import sys
 from unittest.case import TestCase
 from contextlib import contextmanager
 from io import StringIO
-from oe.utils import packages_filter_out_system, trim_version, multiprocess_launch
+from oe.utils import packages_filter_out_system, trim_version, multiprocess_launch, filter_default_features
 
 class TestPackagesFilterOutSystem(TestCase):
     def test_filter(self):
@@ -102,3 +102,39 @@ class TestMultiprocessLaunch(TestCase):
         with captured_output() as (out, err):
             self.assertRaises(bb.BBHandledException, multiprocess_launch, testfunction, ["1", "2", "3", "4", "5", "6"], d, extraargs=(d,))
         self.assertIn("KeyError: 'Invalid number 2'", out.getvalue())
+
+
+class TestDefaultFeatures(TestCase):
+    def test_filter_default_features(self):
+        try:
+            import bb
+            d = bb.data_smart.DataSmart()
+        except ImportError:
+            self.skipTest("Cannot import bb")
+
+        # Test with nothing opted out
+        d.setVar("DISTRO_FEATURES", "")
+        d.setVar("DISTRO_FEATURES_DEFAULTS", "alpha beta gamma")
+        filter_default_features("DISTRO_FEATURES", d)
+        self.assertEqual(d.getVar("DISTRO_FEATURES"), "alpha beta gamma")
+
+        # opt out of a single feature
+        d.setVar("DISTRO_FEATURES", "")
+        d.setVar("DISTRO_FEATURES_DEFAULTS", "alpha beta gamma")
+        d.setVar("DISTRO_FEATURES_OPTED_OUT", "beta")
+        filter_default_features("DISTRO_FEATURES", d)
+        self.assertEqual(d.getVar("DISTRO_FEATURES"), "alpha gamma")
+
+        # opt out of everything
+        d.setVar("DISTRO_FEATURES", "")
+        d.setVar("DISTRO_FEATURES_DEFAULTS", "alpha beta gamma")
+        d.setVar("DISTRO_FEATURES_OPTED_OUT", "gamma alpha beta")
+        filter_default_features("DISTRO_FEATURES", d)
+        self.assertEqual(d.getVar("DISTRO_FEATURES"), "")
+
+        # opt out of something that isn't in our defaults
+        d.setVar("DISTRO_FEATURES", "")
+        d.setVar("DISTRO_FEATURES_DEFAULTS", "alpha beta gamma")
+        d.setVar("DISTRO_FEATURES_OPTED_OUT", "omega")
+        filter_default_features("DISTRO_FEATURES", d)
+        self.assertEqual(d.getVar("DISTRO_FEATURES"), "alpha beta gamma")
Signature · 1 line
2.43.0
#2 lore raw

PATCH v2 2/4 meta: Support opting out of any distro features

In reply to Paul Barker

Add default values to DISTRO_FEATURES using the new filter_default_features() function.

This change obsoletes the variables DISTRO_FEATURES_BACKFILL and DISTRO_FEATURES_BACKFILL_CONSIDERED. Instead, all defaults are added via DISTRO_FEATURES_DEFAULTS and users can opt out of any of these using DISTRO_FEATURES_OPTED_OUT. Hopefully the variable naming here is easier for people to understand and remember.

Migration notes:

- If you have previously assigned DISTRO_FEATURES without using DISTRO_FEATURES_DEFAULT, you will now get the default features added automatically. You will need to review these and add any features you do not want to use to DISTRO_FEATURES_OPTED_OUT.

- DISTRO_FEATURES_DEFAULT is now unused, the new variable name is slighlty different to ensure that it is not accidentally used if a layer hasn't been modified to adapt to the new naming.

- If you previously set DISTRO_FEATURES_BACKFILL_CONSIDERED, use the new variable DISTRO_FEATURES_OPTED_OUT instead.

- If you previously modified DISTRO_FEATURES_BACKFILL, don't do that.

---
 meta/classes-global/base.bbclass                |  5 ++++-
 meta/classes-recipe/crosssdk.bbclass            |  9 +++++----
 meta/classes-recipe/native.bbclass              |  9 +++++----
 meta/classes-recipe/nativesdk.bbclass           |  9 +++++----
 meta/conf/bitbake.conf                          |  1 -
 meta/conf/distro/include/default-distrovars.inc | 15 +++++++++------
 meta/conf/distro/include/tclibc-musl.inc        |  2 +-
 meta/conf/distro/include/tclibc-newlib.inc      |  2 +-
 meta/conf/distro/include/tclibc-picolibc.inc    |  2 +-
 meta/conf/documentation.conf                    |  4 ++--
 10 files changed, 33 insertions(+), 25 deletions(-)
meta/classes-global/base.bbclass +4 -1
index 22b427a5211b..0a4cfd594c72 100644
@@ -454,8 +454,11 @@ def set_packagetriplet(d):
 python () {
     import string, re
 
+    # Filter default features to allow users to opt out of features they don't
+    # want.
+    oe.utils.filter_default_features("DISTRO_FEATURES", d)
+
     # Handle backfilling
-    oe.utils.features_backfill("DISTRO_FEATURES", d)
     oe.utils.features_backfill("MACHINE_FEATURES", d)
 
     # To add a recipe to the skip list , set:
meta/classes-recipe/crosssdk.bbclass +5 -4
index a2853e6a9201..e987f1be05e9 100644
@@ -16,13 +16,14 @@ PACKAGE_ARCH = "${SDK_ARCH}"
 python () {
     # set TUNE_PKGARCH to SDK_ARCH
     d.setVar('TUNE_PKGARCH', d.getVar('SDK_ARCH'))
-    # Set features here to prevent appends and distro features backfill
-    # from modifying nativesdk distro features
+    # Reset features here to prevent DISTRO_FEATURES modifications from
+    # affecting crosssdk distro features
     features = set(d.getVar("DISTRO_FEATURES_NATIVESDK").split())
-    oe.utils.features_backfill("DISTRO_FEATURES", d)
+    d.setVar('DISTRO_FEATURES', '')
+    oe.utils.filter_default_features("DISTRO_FEATURES", d)
     filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVESDK"), d).split())
     d.setVar("DISTRO_FEATURES", " ".join(sorted(features | filtered)))
-    d.setVar("DISTRO_FEATURES_BACKFILL", "")
+    d.setVar("DISTRO_FEATURES_DEFAULTS", "")
 }
 
 STAGING_BINDIR_TOOLCHAIN = "${STAGING_DIR_NATIVE}${bindir_native}/${TARGET_ARCH}${TARGET_VENDOR}-${TARGET_OS}"
meta/classes-recipe/native.bbclass +5 -4
index 5aa9c8e4145e..47a9a066a1a8 100644
@@ -126,13 +126,14 @@ python native_virtclass_handler () {
         return
     bpn = d.getVar("BPN")
 
-    # Set features here to prevent appends and distro features backfill
-    # from modifying native distro features
+    # Reset features here to prevent DISTRO_FEATURES modifications from
+    # affecting native distro features
     features = set(d.getVar("DISTRO_FEATURES_NATIVE").split())
-    oe.utils.features_backfill("DISTRO_FEATURES", d)
+    d.setVar('DISTRO_FEATURES', '')
+    oe.utils.filter_default_features("DISTRO_FEATURES", d)
     filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVE"), d).split())
     d.setVar("DISTRO_FEATURES", " ".join(sorted(features | filtered)))
-    d.setVar("DISTRO_FEATURES_BACKFILL", "")
+    d.setVar("DISTRO_FEATURES_DEFAULTS", "")
 
     classextend = d.getVar('BBCLASSEXTEND') or ""
     if "native" not in classextend:
meta/classes-recipe/nativesdk.bbclass +5 -4
index 5adb7515bf14..f663067f3a25 100644
@@ -78,13 +78,14 @@ python nativesdk_virtclass_handler () {
     if not (pn.endswith("-nativesdk") or pn.startswith("nativesdk-")):
         return
 
-    # Set features here to prevent appends and distro features backfill
-    # from modifying nativesdk distro features
+    # Reset features here to prevent DISTRO_FEATURES modifications from
+    # affecting nativesdk distro features
     features = set(d.getVar("DISTRO_FEATURES_NATIVESDK").split())
-    oe.utils.features_backfill("DISTRO_FEATURES", d)
+    d.setVar('DISTRO_FEATURES', '')
+    oe.utils.filter_default_features("DISTRO_FEATURES", d)
     filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVESDK"), d).split())
     d.setVar("DISTRO_FEATURES", " ".join(sorted(features | filtered)))
-    d.setVar("DISTRO_FEATURES_BACKFILL", "")
+    d.setVar("DISTRO_FEATURES_DEFAULTS", "")
 
     e.data.setVar("MLPREFIX", "nativesdk-")
     e.data.setVar("PN", "nativesdk-" + e.data.getVar("PN").replace("-nativesdk", "").replace("nativesdk-", ""))
meta/conf/bitbake.conf +0 -1
index 6b3443695a04..f851e2adc9f7 100644
@@ -912,7 +912,6 @@ DISTRO_FEATURES_NATIVESDK:mingw32 = "x11 ipv6"
 DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation debuginfod opencl opengl wayland"
 DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation debuginfod opencl opengl wayland"
 
-DISTRO_FEATURES_BACKFILL = "pulseaudio gobject-introspection-data ldconfig opengl ptest multiarch wayland vulkan"
 MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode"
 
 COMBINED_FEATURES = "${@oe.utils.set_intersect('DISTRO_FEATURES', 'MACHINE_FEATURES', d)}"
meta/conf/distro/include/default-distrovars.inc +9 -6
index 7adcdfad4ff4..daa4d2a74dda 100644
@@ -14,19 +14,22 @@ LOCALE_UTF8_IS_DEFAULT ?= "1"
 LOCALE_UTF8_IS_DEFAULT:class-nativesdk = "0"
 
 # seccomp is not yet ported to rv32
-DISTRO_FEATURES_DEFAULT:remove:riscv32 = "seccomp"
+DISTRO_FEATURES_OPTED_OUT:append:riscv32 = "seccomp"
 
 # seccomp is not yet ported to ARC
-DISTRO_FEATURES_DEFAULT:remove:arc = "seccomp"
+DISTRO_FEATURES_OPTED_OUT:append:arc = "seccomp"
 
 # seccomp is not yet ported to microblaze
-DISTRO_FEATURES_DEFAULT:remove:microblaze = "seccomp"
+DISTRO_FEATURES_OPTED_OUT:append:microblaze = "seccomp"
 
 # seccomp is not yet ported to loongarch64
-DISTRO_FEATURES_DEFAULT:remove:loongarch64 = "seccomp"
+DISTRO_FEATURES_OPTED_OUT:append:loongarch64 = "seccomp"
 
-DISTRO_FEATURES_DEFAULT ?= "acl alsa bluetooth debuginfod ext2 ipv4 ipv6 pcmcia usbgadget usbhost wifi xattr nfs zeroconf pci 3g nfc x11 vfat seccomp"
-DISTRO_FEATURES ?= "${DISTRO_FEATURES_DEFAULT}"
+DISTRO_FEATURES_DEFAULTS ?= " \
+    acl alsa bluetooth debuginfod ext2 ipv4 ipv6 pcmcia usbgadget usbhost \
+    wifi xattr nfs zeroconf pci 3g nfc x11 vfat seccomp pulseaudio \
+    gobject-introspection-data ldconfig opengl ptest multiarch wayland vulkan \
+    "
 IMAGE_FEATURES ?= ""
 
 COMMERCIAL_AUDIO_PLUGINS ?= ""
meta/conf/distro/include/tclibc-musl.inc +1 -1
index 98d7a801ac84..a61a6ced0dd2 100644
@@ -14,7 +14,7 @@ PREFERRED_PROVIDER_virtual/libc-locale ?= "musl-locales"
 PREFERRED_PROVIDER_virtual/nativesdk-libintl ?= "nativesdk-glibc"
 PREFERRED_PROVIDER_virtual/nativesdk-libiconv ?= "nativesdk-glibc"
 
-DISTRO_FEATURES_BACKFILL_CONSIDERED += "ldconfig"
+DISTRO_FEATURES_OPTED_OUT += "ldconfig"
 
 #USE_NLS ?= "no"
 
meta/conf/distro/include/tclibc-newlib.inc +1 -1
index 34318b24549c..fff9b9f066fb 100644
@@ -11,7 +11,7 @@ PREFERRED_PROVIDER_virtual/libintl ?= "newlib"
 PREFERRED_PROVIDER_virtual/nativesdk-libintl ?= "nativesdk-glibc"
 PREFERRED_PROVIDER_virtual/nativesdk-libiconv ?= "nativesdk-glibc"
 
-DISTRO_FEATURES_BACKFILL_CONSIDERED += "ldconfig"
+DISTRO_FEATURES_OPTED_OUT += "ldconfig"
 
 #USE_NLS ?= "no"
 
meta/conf/distro/include/tclibc-picolibc.inc +1 -1
index 2cd26cfd7dcd..6134b0d97d6a 100644
@@ -11,7 +11,7 @@ PREFERRED_PROVIDER_virtual/libintl ?= "picolibc"
 PREFERRED_PROVIDER_virtual/nativesdk-libintl ?= "nativesdk-glibc"
 PREFERRED_PROVIDER_virtual/nativesdk-libiconv ?= "nativesdk-glibc"
 
-DISTRO_FEATURES_BACKFILL_CONSIDERED += "ldconfig"
+DISTRO_FEATURES_OPTED_OUT += "ldconfig"
 
 IMAGE_LINGUAS = ""
 
meta/conf/documentation.conf +2 -2
index 1853676fa060..5c93d0e47970 100644
@@ -141,8 +141,8 @@ DISTRO[doc] = "The short name of the distribution. If the variable is blank, met
 DISTRO_EXTRA_RDEPENDS[doc] = "Specifies a list of distro-specific packages to add to all images. The variable only applies to the images that include packagegroup-base."
 DISTRO_EXTRA_RRECOMMENDS[doc] = "Specifies a list of distro-specific packages to add to all images if the packages exist. The list of packages are automatically installed but you can remove them."
 DISTRO_FEATURES[doc] = "The features enabled for the distribution."
-DISTRO_FEATURES_BACKFILL[doc] = "Features to be added to DISTRO_FEATURES if not also present in DISTRO_FEATURES_BACKFILL_CONSIDERED. This variable is set in the meta/conf/bitbake.conf file and it is not intended to be user-configurable."
-DISTRO_FEATURES_BACKFILL_CONSIDERED[doc] = "Features from DISTRO_FEATURES_BACKFILL that should not be backfilled (i.e. added to DISTRO_FEATURES) during the build."
+DISTRO_FEATURES_DEFAULTS[doc] = "The default set of distribution features, prior to any filtering. Usually you should leave this alone and modify DISTRO_FEATURES and/or DISTRO_FEATURES_OPTED_OUT."
+DISTRO_FEATURES_OPTED_OUT[doc] = "The set of default distribtion features to disable. Prefer opting out of features using this variable instead of using DISTRO_FEATURES:remove."
 DISTRO_NAME[doc] = "The long name of the distribution."
 DISTRO_PN_ALIAS[doc] = "Alias names used for the recipe in various Linux distributions."
 DISTRO_VERSION[doc] = "The version of the distribution."
Signature · 1 line
2.43.0
#3 lore raw

Re: [OE-core] [PATCH v2 2/4] meta: Support opting out of any distro features

Quoted 20 lines · Paul Barker

Add default values to DISTRO_FEATURES using the new filter_default_features() function.

This change obsoletes the variables DISTRO_FEATURES_BACKFILL and DISTRO_FEATURES_BACKFILL_CONSIDERED. Instead, all defaults are added via DISTRO_FEATURES_DEFAULTS and users can opt out of any of these using DISTRO_FEATURES_OPTED_OUT. Hopefully the variable naming here is easier for people to understand and remember.

Migration notes:

- If you have previously assigned DISTRO_FEATURES without using DISTRO_FEATURES_DEFAULT, you will now get the default features added automatically. You will need to review these and add any features you do not want to use to DISTRO_FEATURES_OPTED_OUT.

- DISTRO_FEATURES_DEFAULT is now unused, the new variable name is slighlty different to ensure that it is not accidentally used if a layer hasn't been modified to adapt to the new naming.

Quoted 7 lines

- If you previously set DISTRO_FEATURES_BACKFILL_CONSIDERED, use the new variable DISTRO_FEATURES_OPTED_OUT instead.

- If you previously modified DISTRO_FEATURES_BACKFILL, don't do that.

---

Hi Paul,

I believe this is failing in some cases:

ERROR: libtasn1-4.21.0-r0 do_compile: Execution of '/srv/pokybuild/yocto-worker/qemux86-world-alt/build/build/tmp/work/x86-64-v3-poky-linux/libtasn1/4.21.0/temp/run.do_compile.3668072' failed with exit code 1
...

| gtkdoc-mkhtml 2>&1 --help | grep  >/dev/null "\--path"; \
| if test "$?" = "0"; then \
|   mkhtml_options="$mkhtml_options --path=\"/srv/pokybuild/yocto-worker/qemux86-world-alt/build/build/tmp/work/x86-64-v3-poky-linux/libtasn1/4.21.0/build/../sources/libtasn1-4.21.0/doc/reference\""; \
| fi; \
| cd html && gtkdoc-mkhtml $mkhtml_options  libtasn1 ../libtasn1-docs.xml
| warning: failed to load external entity "http://docbook.sourceforge.net/release/xsl/current/html/chunk.xsl"
| compilation error: file /srv/pokybuild/yocto-worker/qemux86-world-alt/build/build/tmp/work/x86-64-v3-poky-linux/libtasn1/4.21.0/recipe-sysroot-native/usr/share/gtk-doc/data/gtk-doc.xsl line 11 element import
| xsl:import : unable to load http://docbook.sourceforge.net/release/xsl/current/html/chunk.xsl

https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3345

Can you have a look at the issue?

Thanks,
Mathieu

Signature · 3 lines
Mathieu Dubois-Briand, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#4 lore raw

Re: [OE-core] [PATCH v2 2/4] meta: Support opting out of any distro features

Quoted 44 lines · Mathieu Dubois-Briand
Quoted 20 lines · Paul Barker

Add default values to DISTRO_FEATURES using the new filter_default_features() function.

This change obsoletes the variables DISTRO_FEATURES_BACKFILL and DISTRO_FEATURES_BACKFILL_CONSIDERED. Instead, all defaults are added via DISTRO_FEATURES_DEFAULTS and users can opt out of any of these using DISTRO_FEATURES_OPTED_OUT. Hopefully the variable naming here is easier for people to understand and remember.

Migration notes:

- If you have previously assigned DISTRO_FEATURES without using DISTRO_FEATURES_DEFAULT, you will now get the default features added automatically. You will need to review these and add any features you do not want to use to DISTRO_FEATURES_OPTED_OUT.

- DISTRO_FEATURES_DEFAULT is now unused, the new variable name is slighlty different to ensure that it is not accidentally used if a layer hasn't been modified to adapt to the new naming.

Quoted 7 lines

- If you previously set DISTRO_FEATURES_BACKFILL_CONSIDERED, use the new variable DISTRO_FEATURES_OPTED_OUT instead.

- If you previously modified DISTRO_FEATURES_BACKFILL, don't do that.

---

Hi Paul,

I believe this is failing in some cases:

ERROR: libtasn1-4.21.0-r0 do_compile: Execution of '/srv/pokybuild/yocto-worker/qemux86-world-alt/build/build/tmp/work/x86-64-v3-poky-linux/libtasn1/4.21.0/temp/run.do_compile.3668072' failed with exit code 1
...

Quoted 8 lines

gtkdoc-mkhtml 2>&1 --help | grep >/dev/null "\--path"; \
if test "$?" = "0"; then \
mkhtml_options="$mkhtml_options --path=\"/srv/pokybuild/yocto-worker/qemux86-world-alt/build/build/tmp/work/x86-64-v3-poky-linux/libtasn1/4.21.0/build/../sources/libtasn1-4.21.0/doc/reference\""; \
fi; \
cd html && gtkdoc-mkhtml $mkhtml_options libtasn1 ../libtasn1-docs.xml
warning: failed to load external entity "http://docbook.sourceforge.net/release/xsl/current/html/chunk.xsl"
compilation error: file /srv/pokybuild/yocto-worker/qemux86-world-alt/build/build/tmp/work/x86-64-v3-poky-linux/libtasn1/4.21.0/recipe-sysroot-native/usr/share/gtk-doc/data/gtk-doc.xsl line 11 element import xsl:import : unable to load http://docbook.sourceforge.net/release/xsl/current/html/chunk.xsl

Quoted 3 lines

https://autobuilder.yoctoproject.org/valkyrie/#/builders/17/builds/3345

Can you have a look at the issue?

Hi Mathieu,

Thanks for the heads up.

I saw some issues in an oe-selftest run after sending the patches, I have a fix for that. I'll run a qemux86-world-alt build to confirm I've also fixed the errors you saw.

Best regards,

Signature · 1 line
Paul Barker
#5 lore raw

PATCH v2 3/4 meta: Support opting out of any machine features

In reply to Paul Barker

Add default values to MACHINE_FEATURES using the new filter_default_features() function.

This change obsoletes the variables MACHINE_FEATURES_BACKFILL and MACHINE_FEATURES_BACKFILL_CONSIDERED. Instead, all defaults are added via MACHINE_FEATURES_DEFAULTS and users can opt out of any of these using MACHINE_FEATURES_OPTED_OUT. Hopefully the variable naming here is easier for people to understand and remember.

Migration notes:

- MACHINE_FEATURES will now get the default features added automatically. You will need to review these and add any features you do not want to use to MACHINE_FEATURES_OPTED_OUT.

- If you previously set MACHINE_FEATURES_BACKFILL_CONSIDERED, use the new variable MACHINE_FEATURES_OPTED_OUT instead.

- If you previously modified MACHINE_FEATURES_BACKFILL, don't do that.

---
 meta/classes-global/base.bbclass                    | 4 +---
 meta/classes-recipe/nativesdk.bbclass               | 1 -
 meta/conf/bitbake.conf                              | 2 +-
 meta/conf/documentation.conf                        | 4 ++--
 meta/conf/machine/include/mips/arch-mips.inc        | 2 +-
 meta/conf/machine/include/powerpc/tune-power5.inc   | 2 +-
 meta/conf/machine/include/powerpc/tune-power6.inc   | 2 +-
 meta/conf/machine/include/powerpc/tune-power7.inc   | 2 +-
 meta/conf/machine/include/powerpc/tune-ppce5500.inc | 2 +-
 meta/conf/machine/include/powerpc/tune-ppce6500.inc | 2 +-
 meta/conf/machine/include/x86/arch-x86.inc          | 2 +-
 11 files changed, 11 insertions(+), 14 deletions(-)
meta/classes-global/base.bbclass +1 -3
index 0a4cfd594c72..76fd0ac046a9 100644
@@ -457,9 +457,7 @@ python () {
     # Filter default features to allow users to opt out of features they don't
     # want.
     oe.utils.filter_default_features("DISTRO_FEATURES", d)
-
-    # Handle backfilling
-    oe.utils.features_backfill("MACHINE_FEATURES", d)
+    oe.utils.filter_default_features("MACHINE_FEATURES", d)
 
     # To add a recipe to the skip list , set:
     #   SKIP_RECIPE[pn] = "message"
meta/classes-recipe/nativesdk.bbclass +0 -1
index f663067f3a25..316649708781 100644
@@ -17,7 +17,6 @@ CLASSOVERRIDE = "class-nativesdk"
 MACHINEOVERRIDES = ""
 
 MACHINE_FEATURES = "${SDK_MACHINE_FEATURES}"
-MACHINE_FEATURES_BACKFILL = ""
 
 MULTILIBS = ""
 
meta/conf/bitbake.conf +1 -1
index f851e2adc9f7..84450386d984 100644
@@ -912,7 +912,7 @@ DISTRO_FEATURES_NATIVESDK:mingw32 = "x11 ipv6"
 DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation debuginfod opencl opengl wayland"
 DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation debuginfod opencl opengl wayland"
 
-MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode"
+MACHINE_FEATURES_DEFAULTS = "rtc qemu-usermode"
 
 COMBINED_FEATURES = "${@oe.utils.set_intersect('DISTRO_FEATURES', 'MACHINE_FEATURES', d)}"
 COMBINED_FEATURES[vardeps] += "DISTRO_FEATURES MACHINE_FEATURES"
meta/conf/documentation.conf +2 -2
index 5c93d0e47970..3c2088754188 100644
@@ -277,8 +277,8 @@ MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS[doc] = "A list of recommended machine-specif
 MACHINE_EXTRA_RDEPENDS[doc] = "A list of machine-specific packages to install as part of the image being built that are not essential for the machine to boot. However, the build process for more fully-featured images depends on the packages being present."
 MACHINE_EXTRA_RRECOMMENDS[doc] = "A list of machine-specific packages to install as part of the image being built that are not essential for booting the machine. The image being built has no build dependencies on the packages in this list."
 MACHINE_FEATURES[doc] = "Specifies the list of hardware features the MACHINE supports."
-MACHINE_FEATURES_BACKFILL[doc] = "Features to be added to MACHINE_FEATURES if not also present in MACHINE_FEATURES_BACKFILL_CONSIDERED. This variable is set in the meta/conf/bitbake.conf file and is not intended to be user-configurable."
-MACHINE_FEATURES_BACKFILL_CONSIDERED[doc] = "Features from MACHINE_FEATURES_BACKFILL that should not be backfilled (i.e. added to MACHINE_FEATURES) during the build."
+MACHINE_FEATURES_DEFAULTS[doc] = "The default set of machine features, prior to any filtering. Usually you should leave this alone and modify MACHINE_FEATURES and/or MACHINE_FEATURES_OPTED_OUT."
+MACHINE_FEATURES_OPTED_OUT[doc] = "The set of default machine features to disable. Prefer opting out of features using this variable instead of using MACHINE_FEATURES:remove."
 MACHINEOVERRIDES[doc] = "Lists overrides specific to the current machine. By default, this list includes the value of MACHINE."
 MAINTAINER[doc] = "The email address of the distribution maintainer."
 MIRRORS[doc] = "Specifies additional paths from which the OpenEmbedded build system gets source code."
meta/conf/machine/include/mips/arch-mips.inc +1 -1
index baadc61d652c..fff2ff639279 100644
@@ -21,7 +21,7 @@ ABIEXTENSION .= "${@bb.utils.filter('TUNE_FEATURES', 'n32', d)}"
 TUNE_CCARGS .= "${@bb.utils.contains('TUNE_FEATURES', 'n32', ' -mabi=n32', '', d)}"
 
 # user mode qemu doesn't support mips64 n32: "Invalid ELF image for this architecture"
-MACHINE_FEATURES_BACKFILL_CONSIDERED:append = " ${@bb.utils.contains('TUNE_FEATURES', 'n32', 'qemu-usermode', '', d)}"
+MACHINE_FEATURES_OPTED_OUT += "${@bb.utils.contains('TUNE_FEATURES', 'n32', 'qemu-usermode', '', d)}"
 
 TUNEVALID[n64] = "MIPS64 n64 ABI"
 TUNECONFLICTS[n64] = "o32 n32"
meta/conf/machine/include/powerpc/tune-power5.inc +1 -1
index e70e4012176c..452897d2d43d 100644
@@ -21,4 +21,4 @@ GLIBC_EXTRA_OECONF:powerpc64 += "${@bb.utils.contains('TUNE_FEATURES', 'power5',
 GLIBC_EXTRA_OECONF:powerpc += "${@bb.utils.contains('TUNE_FEATURES', 'power5', '--with-cpu=power5', '', d)}"
 
 # QEMU usermode fails with invalid instruction error
-MACHINE_FEATURES_BACKFILL_CONSIDERED:append = "${@bb.utils.contains('TUNE_FEATURES', 'power5', ' qemu-usermode', '', d)}"
+MACHINE_FEATURES_OPTED_OUT += "${@bb.utils.contains('TUNE_FEATURES', 'power5', 'qemu-usermode', '', d)}"
meta/conf/machine/include/powerpc/tune-power6.inc +1 -1
index eaf89515cad7..da8831058094 100644
@@ -21,4 +21,4 @@ GLIBC_EXTRA_OECONF:powerpc64 += "${@bb.utils.contains('TUNE_FEATURES', 'power6',
 GLIBC_EXTRA_OECONF:powerpc += "${@bb.utils.contains('TUNE_FEATURES', 'power6', '--with-cpu=power6', '', d)}"
 
 # QEMU usermode fails with invalid instruction error
-MACHINE_FEATURES_BACKFILL_CONSIDERED:append = "${@bb.utils.contains('TUNE_FEATURES', 'power6', ' qemu-usermode', '', d)}"
+MACHINE_FEATURES_OPTED_OUT += "${@bb.utils.contains('TUNE_FEATURES', 'power6', 'qemu-usermode', '', d)}"
meta/conf/machine/include/powerpc/tune-power7.inc +1 -1
index 4531ddd85f48..47887c553f97 100644
@@ -21,4 +21,4 @@ GLIBC_EXTRA_OECONF:powerpc64 += "${@bb.utils.contains('TUNE_FEATURES', 'power7',
 GLIBC_EXTRA_OECONF:powerpc += "${@bb.utils.contains('TUNE_FEATURES', 'power7', '--with-cpu=power7', '', d)}"
 
 # QEMU usermode fails with invalid instruction error
-MACHINE_FEATURES_BACKFILL_CONSIDERED:append = "${@bb.utils.contains('TUNE_FEATURES', 'power7', ' qemu-usermode', '', d)}"
+MACHINE_FEATURES_OPTED_OUT += "${@bb.utils.contains('TUNE_FEATURES', 'power7', 'qemu-usermode', '', d)}"
meta/conf/machine/include/powerpc/tune-ppce5500.inc +1 -1
index 446b344c3d25..63e745b4a29c 100644
@@ -19,4 +19,4 @@ PACKAGE_EXTRA_ARCHS:tune-ppc64e5500 = "${PACKAGE_EXTRA_ARCHS:tune-powerpc64} ppc
 QEMU_EXTRAOPTIONS:tune-ppc64e5500 = " -cpu e500mc"
 
 # QEMU usermode fails with invalid instruction error (YOCTO: #10304)
-MACHINE_FEATURES_BACKFILL_CONSIDERED:append = "${@bb.utils.contains('TUNE_FEATURES', 'e5500', ' qemu-usermode', '', d)}"
+MACHINE_FEATURES_OPTED_OUT += "${@bb.utils.contains('TUNE_FEATURES', 'e5500', 'qemu-usermode', '', d)}"
meta/conf/machine/include/powerpc/tune-ppce6500.inc +1 -1
index 4444705b2d6d..77de2f9982bc 100644
@@ -19,4 +19,4 @@ PACKAGE_EXTRA_ARCHS:tune-ppc64e6500 = "${PACKAGE_EXTRA_ARCHS:tune-powerpc64} ppc
 QEMU_EXTRAOPTIONS:tune-ppc64e6500 = " -cpu e500mc"
 
 # QEMU usermode fails with invalid instruction error (YOCTO: #10304)
-MACHINE_FEATURES_BACKFILL_CONSIDERED:append = "${@bb.utils.contains('TUNE_FEATURES', 'e6500', ' qemu-usermode', '', d)}"
+MACHINE_FEATURES_OPTED_OUT += "${@bb.utils.contains('TUNE_FEATURES', 'e6500', 'qemu-usermode', '', d)}"
meta/conf/machine/include/x86/arch-x86.inc +1 -1
index 28742e794d63..95b5fe52114c 100644
@@ -25,7 +25,7 @@ TUNE_CCARGS .= "${@bb.utils.contains('TUNE_FEATURES', 'mx32', ' -mx32', '', d)}"
 TUNE_LDARGS += "${@bb.utils.contains('TUNE_FEATURES', 'mx32', '-m elf32_x86_64', '', d)}"
 TUNE_ASARGS += "${@bb.utils.contains('TUNE_FEATURES', 'mx32', '-x32', '', d)}"
 # user mode qemu doesn't support x32
-MACHINE_FEATURES_BACKFILL_CONSIDERED:append = " ${@bb.utils.contains('TUNE_FEATURES', 'mx32', 'qemu-usermode', '', d)}"
+MACHINE_FEATURES_OPTED_OUT += "${@bb.utils.contains('TUNE_FEATURES', 'mx32', 'qemu-usermode', '', d)}"
 MACHINEOVERRIDES =. "${@bb.utils.contains('TUNE_FEATURES', 'mx32', 'x86-x32:', '', d)}"
 
 # ELF64 ABI
Signature · 1 line
2.43.0
#6 lore raw

PATCH v2 4/4 lib: oe: Drop backfill support

In reply to Paul Barker

We have removed DISTRO_FEATURES_BACKFILL and MACHINE_FEATURES_BACKFILL, so we no longer need the features_backfill() function.

---
 meta/lib/oe/utils.py | 21 ---------------------
 1 file changed, 21 deletions(-)
meta/lib/oe/utils.py +0 -21
index 52f2f1bd55e0..5c722d230842 100644
@@ -137,27 +137,6 @@ def inherits(d, *classes):
     """Return True if the metadata inherits any of the specified classes"""
     return any(bb.data.inherits_class(cls, d) for cls in classes)
 
-def features_backfill(var,d):
-    # This construct allows the addition of new features to variable specified
-    # as var
-    # Example for var = "DISTRO_FEATURES"
-    # This construct allows the addition of new features to DISTRO_FEATURES
-    # that if not present would disable existing functionality, without
-    # disturbing distributions that have already set DISTRO_FEATURES.
-    # Distributions wanting to elide a value in DISTRO_FEATURES_BACKFILL should
-    # add the feature to DISTRO_FEATURES_BACKFILL_CONSIDERED
-    features = (d.getVar(var) or "").split()
-    backfill = (d.getVar(var+"_BACKFILL") or "").split()
-    considered = (d.getVar(var+"_BACKFILL_CONSIDERED") or "").split()
-
-    addfeatures = []
-    for feature in backfill:
-        if feature not in features and feature not in considered:
-            addfeatures.append(feature)
-
-    if addfeatures:
-        d.appendVar(var, " " + " ".join(addfeatures))
-
 def filter_default_features(varname, d):
     # Process default features to exclude features which the user has opted out
     # of. The result is appended to the target variable (e.g. DISTRO_FEATURES
Signature · 1 line
2.43.0