Use fixed DISTRO_FEATURES for native

There seems to be no advantage to letting distro features affect native builds. There is a significant disadvantage: a change to DISTRO_FEATURES will trigger a lot of unnecessary native tasks. In a test like this:

  $ bitbake core-image-minimal
  # append " systemd" to DISTRO_FEATURES
  $ bitbake core-image-minimal
The latter build takes 44 minutes (28%) of cpu-time less with this
patch (skipping 135 native tasks). Sadly wall clock time was not
affected as glibc remains the bottleneck.

Set DISTRO_FEATURES to a fixed value for native recipes to avoid the unnecessary tasks: currently the default value is empty.

Do the variable setting in native_virtclass_handler() because otherwise
it could still be overridden by appends and the feature backfilling. Shuffle the early returns so DISTRO_FEATURES gets set as long as the packagename ends with "-native".

Signed-off-by Jussi Kukkonen <[email protected]>

---
 meta/classes/native.bbclass | 12 ++++++++----
 meta/conf/bitbake.conf      |  2 ++

 2 files changed, 10 insertions(+), 4 deletions(-)

meta/classes/native.bbclass +0 -0

index 1919fbc..fbca4c6 100644
@@ -121,14 +121,18 @@ PATH_prepend = "${COREBASE}/scripts/native-

intercept:"
 SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}"

 python native_virtclass_handler () {
-    classextend = e.data.getVar('BBCLASSEXTEND') or "" -    if "native" not in classextend:
-        return
-
     pn = e.data.getVar("PN")
     if not pn.endswith("-native"):
         return

+    # Set features here to prevent appends and distro features backfill
+    # from modifying native distro features
+    d.setVar("DISTRO_FEATURES", "${NATIVE_DISTRO_FEATURES}")

meta/classes/native.bbclass +5 -0

+
+    classextend = e.data.getVar('BBCLASSEXTEND') or ""
+    if "native" not in classextend:
+        return
+

     def map_dependencies(varname, d, suffix = ""):
         if suffix:
             varname = varname + "_" + suffix

meta/conf/bitbake.conf +0 -0

index 5e98d45..78a3470 100644
@@ -789,6 +789,8 @@ MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS ?= ""

 EXTRA_IMAGE_FEATURES ??= ""
 IMAGE_FEATURES += "${EXTRA_IMAGE_FEATURES}"

meta/conf/bitbake.conf +2 -0

+NATIVE_DISTRO_FEATURES ?= ""
+

 DISTRO_FEATURES_BACKFILL = "pulseaudio sysvinit bluez5 gobject- introspection-data ldconfig"
 MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode"

wrote:

Quoted 67 lines · Jussi Kukkonen

There seems to be no advantage to letting distro features affect native builds. There is a significant disadvantage: a change to DISTRO_FEATURES will trigger a lot of unnecessary native tasks. In a test like this:

  $ bitbake core-image-minimal
  # append " systemd" to DISTRO_FEATURES
  $ bitbake core-image-minimal
The latter build takes 44 minutes (28%) of cpu-time less with this
patch (skipping 135 native tasks). Sadly wall clock time was not
affected as glibc remains the bottleneck.

Set DISTRO_FEATURES to a fixed value for native recipes to avoid the unnecessary tasks: currently the default value is empty.

Do the variable setting in native_virtclass_handler() because otherwise
it could still be overridden by appends and the feature backfilling. Shuffle the early returns so DISTRO_FEATURES gets set as long as the packagename ends with "-native".

Signed-off-by Jussi Kukkonen <[email protected]>

---
 meta/classes/native.bbclass | 12 ++++++++----
 meta/conf/bitbake.conf      |  2 ++
 2 files changed, 10 insertions(+), 4 deletions(-)

meta/classes/native.bbclass +0 -0

index 1919fbc..fbca4c6 100644
@@ -121,14 +121,18 @@ PATH_prepend = "${COREBASE}/scripts/native-

intercept:"
SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}"

python native_virtclass_handler () {

meta/classes/native.bbclass +1 -4

-    classextend = e.data.getVar('BBCLASSEXTEND') or ""
-    if "native" not in classextend:
-        return
-
     pn = e.data.getVar("PN")
     if not pn.endswith("-native"):
         return
 
+    # Set features here to prevent appends and distro features

backfill

meta/classes/native.bbclass +7 -0

+    # from modifying native distro features
+    d.setVar("DISTRO_FEATURES", "${NATIVE_DISTRO_FEATURES}")
+
+    classextend = e.data.getVar('BBCLASSEXTEND') or ""
+    if "native" not in classextend:
+        return
+
     def map_dependencies(varname, d, suffix = ""):
         if suffix:
             varname = varname + "_" + suffix

meta/conf/bitbake.conf +2 -0

index 5e98d45..78a3470 100644
@@ -789,6 +789,8 @@ MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS ?= ""
 EXTRA_IMAGE_FEATURES ??= ""
 IMAGE_FEATURES += "${EXTRA_IMAGE_FEATURES}"
 
+NATIVE_DISTRO_FEATURES ?= ""
+
 DISTRO_FEATURES_BACKFILL = "pulseaudio sysvinit bluez5 gobject-

Quoted 2 lines

introspection-data ldconfig"
MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode"

Thanks for working on this, looks good.

Most native variables are called XXX_NATIVE, not NATIVE_XXX so we might want to make this match?

Also, one question. Does this have any effect on qemu-native? Does qemu-native need/use x11 today?

Quoted 77 lines · Richard Purdie

Quoted 3 lines · Jussi Kukkonen

There seems to be no advantage to letting distro features affect native builds. There is a significant disadvantage: a change to DISTRO_FEATURES will trigger a lot of unnecessary native tasks.

In a

Quoted 5 lines

test like this:

  $ bitbake core-image-minimal
  # append " systemd" to DISTRO_FEATURES
  $ bitbake core-image-minimal
The latter build takes 44 minutes (28%) of cpu-time less with

this

Quoted 4 lines

patch (skipping 135 native tasks). Sadly wall clock time was not affected as glibc remains the bottleneck.

Set DISTRO_FEATURES to a fixed value for native recipes to avoid

the

Quoted 5 lines

unnecessary tasks: currently the default value is empty.

Do the variable setting in native_virtclass_handler() because otherwise
it could still be overridden by appends and the feature

backfilling.

Quoted 49 lines

Shuffle the early returns so DISTRO_FEATURES gets set as long as the packagename ends with "-native".

Signed-off-by Jussi Kukkonen <[email protected]>

---
 meta/classes/native.bbclass | 12 ++++++++----
 meta/conf/bitbake.conf      |  2 ++

 2 files changed, 10 insertions(+), 4 deletions(-)

meta/classes/native.bbclass +0 -0

index 1919fbc..fbca4c6 100644
@@ -121,14 +121,18 @@ PATH_prepend = "${COREBASE}/scripts/native-

intercept:"
 SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}"

 python native_virtclass_handler () {
-    classextend = e.data.getVar('BBCLASSEXTEND') or "" -    if "native" not in classextend:
-        return
-
     pn = e.data.getVar("PN")
     if not pn.endswith("-native"):
         return

+    # Set features here to prevent appends and distro features backfill
+    # from modifying native distro features
+    d.setVar("DISTRO_FEATURES", "${NATIVE_DISTRO_FEATURES}")

meta/classes/native.bbclass +5 -0

+
+    classextend = e.data.getVar('BBCLASSEXTEND') or ""
+    if "native" not in classextend:
+        return
+

     def map_dependencies(varname, d, suffix = ""):
         if suffix:
             varname = varname + "_" + suffix

meta/conf/bitbake.conf +0 -0

index 5e98d45..78a3470 100644
@@ -789,6 +789,8 @@ MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS ?= ""

 EXTRA_IMAGE_FEATURES ??= ""
 IMAGE_FEATURES += "${EXTRA_IMAGE_FEATURES}"

meta/conf/bitbake.conf +2 -0

+NATIVE_DISTRO_FEATURES ?= ""
+

Quoted 3 lines

 DISTRO_FEATURES_BACKFILL = "pulseaudio sysvinit bluez5 gobject- introspection-data ldconfig"
 MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode"

Thanks for working on this, looks good.

Quoted 3 lines

Most native variables are called XXX_NATIVE, not NATIVE_XXX so we might
want to make this match?

Will do.

Quoted 2 lines

Also, one question. Does this have any effect on qemu-native? Does qemu-native need/use x11 today?

qemu itself does not use anything x11 since last year and the default native packageconfig is already not
dependent on distro features.

Now that you mentioned I had a closer look at qemu & SDL:
 * qemu-native doesn't build with "sdl" by default, but someone might change that of course
 * If that happens (and libsdl-native is not in ASSUME_PROVIDED), libsdl is built and by default depends on libx11
I completely expected this to fail now (without x11 in distro features) but it actually builds just fine. It turns out that a clever Burton has tweaked REQUIRED_DISTRO_FEATURES_class-native for some X libraries for exactly this purpose already.

Quoted 6 lines

Quoted 2 lines

Also, one question. Does this have any effect on qemu-native? Does qemu-native need/use x11 today?

qemu itself does not use anything x11 since last year and the default native packageconfig is already not
dependent on distro features.

Quoted 10 lines

Now that you mentioned I had a closer look at qemu & SDL:
* qemu-native doesn't build with "sdl" by default, but someone might
change that of course
* If that happens (and libsdl-native is not in ASSUME_PROVIDED),
libsdl is built and by default depends on libx11
I completely expected this to fail now (without x11 in distro features) but it actually builds just fine. It turns out that a clever Burton has tweaked REQUIRED_DISTRO_FEATURES_class-native for some X libraries for exactly this purpose already.

Right, we actually have a commented section for sdl and qemu in our local.conf.sample so we do need to test and figure this piece out.

Would we want to put x11 in the default native distro flags to avoid some of the other issues above?

Avoid signifant native task churn when DISTRO_FEATURES is changed by defining a fixed DISTRO_FEATURES value (default is "") for native recipes.

The xorg changes are all removals of x11 requirements (that were not real) from recipes that have native versions.

I added a configuration default in bitbake.conf, let me know if this is not a good place.

This seems to work fine (tested multiple different images) but I am not really sure how to test it further... Suggestions are welcome, as are results from autobuilder.

Quoted 14 lines · Jussi Kukkonen

Avoid signifant native task churn when DISTRO_FEATURES is changed by defining a fixed DISTRO_FEATURES value (default is "") for native
recipes.

The xorg changes are all removals of x11 requirements (that were not
real) from recipes that have native versions.

I added a configuration default in bitbake.conf, let me know if this
is not a good place.

Quoted 4 lines

This seems to work fine (tested multiple different images) but I am not really sure how to test it further... Suggestions are welcome, as
are results from autobuilder.

Close, but:

https://autobuilder.yocto.io/builders/nightly-no-x11/builds/225/steps
/BuildImages/logs/stdio