DISTRO_FEATURES compatibility fixes

Hi,

This series contains two fixes:
- make the init manager backfilled into distro features (thanks, Marcin)
- backfill features earlier in the parse so conditional inherits work with backfilled features (patch as described by Richard over IM)

Between then this makes sysvinit images work again where distributions have set DISTRO_FEATURES, instead of appending on oe-core's base.

Ross

The following changes since commit 8d6e55bf2192524bda84138b5356a2791adbbe61:

prserv: add LOCALCOUNT to AUTOINCs migration feature (2013-01-22 16:01:32 +0000)

are available in the git repository at:

git://git.yoctoproject.org/poky-contrib ross/backfill

for you to fetch changes up to 6d174abd95de70a59f54ca28c965ca72933a3ce7:

base: make feature backfilling happen earlier (2013-01-24 17:45:38 +0000)

----------------------------------------------------------------
Marcin Juszkiewicz (1):
bitbake.conf: unbreak all builds with custom DISTRO_FEATURES

Richard Purdie (1):
base: make feature backfilling happen earlier

 meta/classes/base.bbclass |    2 ++
 meta/conf/bitbake.conf    |    5 +----
 2 files changed, 3 insertions(+), 4 deletions(-)

Marcin Juszkiewicz (1):
bitbake.conf: unbreak all builds with custom DISTRO_FEATURES

Richard Purdie (1):
base: make feature backfilling happen earlier

 meta/classes/base.bbclass |    2 ++
 meta/conf/bitbake.conf    |    5 +----
 2 files changed, 3 insertions(+), 4 deletions(-)

1.7.10.4

From: Marcin Juszkiewicz <[email protected]>

Commit 9e7c64ca9afbf27edd0d35a1830ce55ee6d778ab broke all builds where custom DISTRO_FEATURES were used.

Resulting images ended in non-bootable state due to lack of initscripts (unless someone had sysvinit or systemd in D_E already).

https://bugs.launchpad.net/linaro-oe/+bug/1102910
https://bugs.launchpad.net/linaro-oe/+bug/1099405/comments/12

---
 meta/conf/bitbake.conf |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

index 2dc50ca..607fb70 100644
@@ -738,7 +738,7 @@ MACHINE_ESSENTIAL_EXTRA_RDEPENDS ?= ""
 MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS ?= ""
 IMAGE_FEATURES += "${EXTRA_IMAGE_FEATURES}"
 
-DISTRO_FEATURES_BACKFILL = "pulseaudio"
+DISTRO_FEATURES_BACKFILL = "pulseaudio ${DISTRO_FEATURES_INITMAN}"
 DISTRO_FEATURES_append = "${@oe.utils.features_backfill("DISTRO_FEATURES",d)}"
 
 MACHINE_FEATURES_BACKFILL = "rtc"

1.7.10.4

From: Richard Purdie <[email protected]>

Backfilling DISTRO_FEATURES and MACHINE_FEATURES with _append statements happens too late to use those variables with conditional inherits, like this:

inherit ${@base_contains('DISTRO_FEATURES','sysvinit','update-rc.d_real','',d)}

Instead, do the backfilling at ConfigParse time so that it happens earlier in the parse, which results in that inherit behaving as expected when sysvinit was backfilled.

---
 meta/classes/base.bbclass |    2 ++
 meta/conf/bitbake.conf    |    3 ---
 2 files changed, 2 insertions(+), 3 deletions(-)

index b427a5e..dafded3 100644
@@ -314,6 +314,8 @@ python base_eventhandler() {
         generate_git_config(e)
         pkgarch_mapping(e.data)
         preferred_ml_updates(e.data)
+        e.data.appendVar('DISTRO_FEATURES', oe.utils.features_backfill("DISTRO_FEATURES", e.data))
+        e.data.appendVar('MACHINE_FEATURES', oe.utils.features_backfill("MACHINE_FEATURES", e.data))
 
     if isinstance(e, bb.event.BuildStarted):
         statuslines = []

index 607fb70..13a3ff9 100644
@@ -739,10 +739,7 @@ MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS ?= ""
 IMAGE_FEATURES += "${EXTRA_IMAGE_FEATURES}"
 
 DISTRO_FEATURES_BACKFILL = "pulseaudio ${DISTRO_FEATURES_INITMAN}"
-DISTRO_FEATURES_append = "${@oe.utils.features_backfill("DISTRO_FEATURES",d)}"
-
 MACHINE_FEATURES_BACKFILL = "rtc"
-MACHINE_FEATURES_append = "${@oe.utils.features_backfill("MACHINE_FEATURES",d)}"
 
 COMBINED_FEATURES = "\
     ${@base_both_contain("DISTRO_FEATURES", "MACHINE_FEATURES", "alsa", d)} \

1.7.10.4

Ross Burton <[email protected]>

This causes regressions for setups where DISTRO_FEATURES_BACKFILL uses the override mechanism. E.g. I have

| DISTRO_FEATURES_BACKFILL_mydist = "ld-is-gold ${${PROJECT_FEATURES}" | DISTRO_FEATURES_INITMAN_mydist = "systemd"

Before this commit, DISTRO_FEATURES contained 'ld-is-gold' and the project features. Now, project features + ld-is-gold vanished and pulseaudio is in again.

Enrico

I don't think its possible to make everything work :(

The question is therefore how do we want to proceed? I'll take further proposed patches but I don't know what the best thing to do here is...

Cheers,

Richard

I made my setup (--> overriding DISTRO_FEATURES_BACKFILL by distro configuration) working again by assigning it weakly in bitbake.conf:

-DISTRO_FEATURES_BACKFILL = "pulseaudio ${DISTRO_FEATURES_INITMAN}" +DISTRO_FEATURES_BACKFILL ?= "pulseaudio ${DISTRO_FEATURES_INITMAN}"

ditto for MACHINE_FEATURES_BACKFILL.

Enrico

Why are you assigning *_FEATURES_BACKFILL at all?

Cheers,
Paul

Paul Eggleton
Intel Open Source Technology Centre

There are some implicit relations between features (e.g. when selecting 'usbhost' I *usually* want 'vfat' or 'ext2' support too). So I have

  DISTRO_FEATURES_BACKFILL = "\
    largefile nfsroot modules ld-is-gold ${PROJECT_FEATURES} \
    ${DISTRO_FEATURES_INITMAN} \
    ${@base_contains('PROJECT_FEATURES', 'alsa', 'sound', '', d)} \
    ${@base_contains('PROJECT_FEATURES', 'directfb', 'screen', '', d)} \
    ${@base_contains('PROJECT_FEATURES', 'fb', 'screen', '', d)} \
    ${@base_contains('PROJECT_FEATURES', 'mmc', 'ext2 vfat', '', d)} \
    ${@base_contains('PROJECT_FEATURES', 'touchscreen', 'screen', '', d)} \
    ${@base_contains('PROJECT_FEATURES', 'ubifs', 'mtd', '', d)} \
    ${@base_contains('PROJECT_FEATURES', 'jffs2', 'mtd', '', d)} \
    ${@base_contains('PROJECT_FEATURES', 'usbclient', 'usb usbgadget', '', d)} \
    ${@base_contains('PROJECT_FEATURES', 'usbgadget', 'usb usbclient', '', d)} \
    ${@base_contains('PROJECT_FEATURES', 'usbhost', 'ext2 vfat', '', d)} \
    ${@base_contains('PROJECT_FEATURES', 'usbhost', 'usb', '', d)} \
    ${@base_contains('PROJECT_FEATURES', 'x11', 'screen', '', d)} \
  "
  
  DISTRO_FEATURES_BACKFILL_CONSIDERED = "${NO_PROJECT_FEATURES}"

in my distro configuration (I knew; some are machine but no distro features but this does not matter for me) and want to allow projects to add new features and to remove default ones.

The default 'pulseaudio' in DISTRO_FEATURES_BACKFILL does not sound sane to me either...

Enrico

SIGMA Chemnitz GmbH       Registergericht:   Amtsgericht Chemnitz HRB 1750
Am Erlenwald 13           Geschaeftsfuehrer: Grit Freitag, Frank Pyritz
09128 Chemnitz

This does seem rather like an abuse of DISTRO_FEATURES_BACKFILL. Can you explain why you are doing it this way rather than just setting DISTRO_FEATURES directly to what you wanted?

p.

I need a way to:

1. set some defaults on distribution base and avoid nasty details like mandatory ${DISTRO_FEATURES_LIBC} flags in the project configuration

2. allow to override these defaults on a per-project base

3. add features support by recipes/classes of my distribution

afaik, DISTRO_FEATURES_BACKFILL + _CONSIDERED exist to allow the first two point without an '-=' operator which lacks in bitbake.

Of course, I could reinvent the wheel and write my own _CONSIDERED mechanism. But until now (resp. without the '=' => '?=' change), it worked fine with DISTRO_FEATURES_BACKFILL.

Enrico

No they don't. They exist to allow adding new features that should be enabled for all existing distro configs without each of those having to be changed, and provide a means for distros to opt out of that enabling if they wish.

We don't support using the backfill mechanism in this manner; so if it breaks you get to keep both pieces.

It's still not clear to me why you could not just set DISTRO_FEATURES directly. If you have values you're likely to want to remove, you can put them in separate variables that you can clear out later.

Cheers,
Paul

Paul Eggleton
Intel Open Source Technology Centre

How is this different from my requirements? Ok; atm, DISTRO_FEATURES_BACKFILL is pretty useless because it contains only 'pulseaudio' which is probably unwanted on >90% of all embedded devices and it is impossible to extend DISTRO_FEATURES_BACKFILL.

But this variable exists to allow removal of features which are enabled by default.

It is very difficultly to do within distro.conf. E.g. having

| DISTRO_FEATURES = "${@subtract_set('DISTRO_FEATURES', 'UNWANTED_FEATURES')}'

will probably result into some error regarding recursive expansion.

Using ':=' will not work here because required operations are not available yet.

At the end it will reinvent the DISTRO_FEATURES_BACKFILL_CONSIDERED wheel by moving it into my-distro.bbclass.

Enrico

pulseaudio was enabled already in the recipes that care about it; that was the point of adding it in the first place so it could be turned off.

It's not intended to be extended; it's for use in OE-Core alone.

What I meant was something like:

DISTRO_FEATURES = "${DISTRO_FEATURES_BASIC} ${DISTRO_FEATURES_A} ${DISTRO_FEATURES_B}"

And then in your specific configs:
DISTRO_FEATURES_A = ""

Cheers,
Paul

Paul Eggleton
Intel Open Source Technology Centre

fwiw, I am using now

_EXT_PROJECT_FEATURES = "\
largefile nfsroot modules ld-is-gold ${PROJECT_FEATURES} \
${DISTRO_FEATURES_INITMAN} ${DISTRO_FEATURES_LIBC} \
${@base_contains('PROJECT_FEATURES', 'alsa', 'sound', '', d)} \
${@base_contains('PROJECT_FEATURES', 'directfb', 'screen', '', d)} \
...

DISTRO_FEATURES = "${@' '.join(sorted(list( \
set(d.getVar('_EXT_PROJECT_FEATURES', True).split()) - \
set(d.getVar('NO_PROJECT_FEATURES', True).split()))))}"

DISTRO_FEATURES_BACKFILL_CONSIDERED = "${DISTRO_FEATURES_BACKFILL}"

Enrico

That wasn't really the intention. The original purpose of DISTRO_FEATURES_BACKFILL was to allow behaviour that currently defaults to "on" to be brought under the control of a DISTRO_FEATURE without breaking existing distros that were relying on it.

The idea was that DISTROs would set DISTRO_FEATURES_BACKFILL_CONSIDERED to list the features that they are aware of but don't want (and would never alter the value of DISTRO_FEATURES_BACKFILL). Any feature that ends up in the latter but not in the former is considered to post-date the DISTRO and is, consequently, spliced into DISTRO_FEATURES by the backwards-compatibility code.

Admittedly I'm not sure that this was ever documented anywhere other than the mailing list archives so I guess it's understandable that you (and quite likely others) are going a bit off-piste with it. But the primary use-case is what I described above, and this is the one that needs to take precedence if there is a conflict between differing requirements.

Incidentally, regarding your other comment about pulseaudio not belonging in DISTRO_FEATURES_BACKFILL, it transpires that pulseaudio was in fact the very feature that spurred the creation of this mechanism and hence the first to be backfilled. See:

http://thread.gmane.org/gmane.comp.handhelds.openembedded.core/10941/focus=13033

and the following thread.

p.