Use fixed DISTRO_FEATURES for native
v3 cover Use fixed DISTRO_FEATURES for native
Avoid signifant native task churn when DISTRO_FEATURES is changed by defining (almost) fixed DISTRO_FEATURES for native recipes.
Add DISTRO_FEATURES_FILTER_NATIVE and DISTRO_FEATURES_NATIVE to limit the changes to DISTRO_FEATURES when building native recipes.
Changes since V3:
* Use both DISTRO_FEATURES_FILTER_NATIVE (from v2) and DISTRO_FEATURES_NATIVE (from v1): see commit message for details
* Do the same thing for nativesdk
* Remove the x11 patches and solve the problem by always adding x11 to DISTRO_FEATURES_NATIVE and DISTRO_FEATURES_NATIVESDK (so it's used even when target DISTRO_FEATURES does not have it)
Thanks,
Jussi
The following changes since commit 0793c758b15e5007b4dd4b146987a0e74d6f6cba:
Revert "staging: Fix sysroot problem with populate_sysroot dependencies on do_fetch" (2017-04-11 00:57:03 +0100)
are available in the git repository at:
git://git.yoctoproject.org/poky-contrib jku/native-distro-features
http://git.yoctoproject.org/cgit.cgi/poky-contrib/log/?h=jku/native-distro-features
Jussi Kukkonen (1):
native/nativesdk: Use fixed DISTRO_FEATURES
meta/classes/native.bbclass | 14 ++++++++++---- meta/classes/nativesdk.bbclass | 6 ++++++ meta/conf/bitbake.conf | 9 +++++++++ 3 files changed, 25 insertions(+), 4 deletions(-)
Signature · 1 line
2.1.4
PATCH v3 1/1 native/nativesdk: Use fixed DISTRO_FEATURES
There seems to be little advantage to letting distro features affect native builds. There is a significant disadvantage: a change to DISTRO_FEATURES will trigger a lot of unnecessary native tasks. In a test like this:
$ bitbake core-image-minimal # append " systemd" to DISTRO_FEATURES $ bitbake core-image-minimal The latter build takes 44 minutes (28%) of cpu-time less with this patch (skipping 135 native tasks). Sadly wall clock time was not affected as glibc remains the bottleneck.
Set native distro features to DISTRO_FEATURES_NATIVE appended with an intersection of DISTRO_FEATURES and DISTRO_FEATURES_FILTER_NATIVE. Current default values (baitbake.conf) are
* DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation" (as gtk-doc-native has much less dependencies when built without it)
* DISTRO_FEATURES_NATIVE ?= "x11" (to enable native UIs even if target does not containe them)
Do the variable setting in native_virtclass_handler() because otherwise it could still be overridden by appends and the feature backfilling. Shuffle the early returns so DISTRO_FEATURES gets set as long as the packagename ends with "-native".
Add similar variables for nativesdk.
--- meta/classes/native.bbclass | 14 ++++++++++---- meta/classes/nativesdk.bbclass | 6 ++++++ meta/conf/bitbake.conf | 9 +++++++++ 3 files changed, 25 insertions(+), 4 deletions(-)
meta/classes/native.bbclass +10 -4
@@ -121,14 +121,20 @@ PATH_prepend = "${COREBASE}/scripts/native-intercept:" SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}" python native_virtclass_handler () { - classextend = e.data.getVar('BBCLASSEXTEND') or "" - if "native" not in classextend: - return - pn = e.data.getVar("PN") if not pn.endswith("-native"): return + # Set features here to prevent appends and distro features backfill + # from modifying native distro features + features = set(d.getVar("DISTRO_FEATURES_NATIVE").split()) + filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVE"), d).split()) + d.setVar("DISTRO_FEATURES", " ".join(features | filtered)) + + classextend = e.data.getVar('BBCLASSEXTEND') or "" + if "native" not in classextend: + return + def map_dependencies(varname, d, suffix = ""): if suffix: varname = varname + "_" + suffix
meta/classes/nativesdk.bbclass +6 -0
@@ -70,6 +70,12 @@ python nativesdk_virtclass_handler () { if not (pn.endswith("-nativesdk") or pn.startswith("nativesdk-")): return + # Set features here to prevent appends and distro features backfill + # from modifying nativesdk distro features + features = set(d.getVar("DISTRO_FEATURES_NATIVESDK").split()) + filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVESDK"), d).split()) + d.setVar("DISTRO_FEATURES", " ".join(features | filtered)) + e.data.setVar("MLPREFIX", "nativesdk-") e.data.setVar("PN", "nativesdk-" + e.data.getVar("PN").replace("-nativesdk", "").replace("nativesdk-", "")) e.data.setVar("OVERRIDES", e.data.getVar("OVERRIDES", False) + ":virtclass-nativesdk")
meta/conf/bitbake.conf +9 -0
@@ -789,6 +789,15 @@ MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS ?= "" EXTRA_IMAGE_FEATURES ??= "" IMAGE_FEATURES += "${EXTRA_IMAGE_FEATURES}" +# Native distro features (will always be used for -native, even if they +# are not enabled for target) +DISTRO_FEATURES_NATIVE ?= "x11" +DISTRO_FEATURES_NATIVESDK ?= "x11" +# Normally target distro features will not be applied to native builds: +# Native distro features on this list will use the target feature value +DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation" +DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation" + DISTRO_FEATURES_BACKFILL = "pulseaudio sysvinit bluez5 gobject-introspection-data ldconfig" MACHINE_FEATURES_BACKFILL = "rtc qemu-usermode"
Signature · 1 line
2.1.4
Re: [PATCHv3 1/1] native/nativesdk: Use fixed DISTRO_FEATURES
Quoted 62 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of Jussi Kukkonen
Sent: den 11 april 2017 16:36
To: [email protected]
Subject: [OE-core] [PATCHv3 1/1] native/nativesdk: Use fixedDISTRO_FEATURESThere seems to be little advantage to letting distro features affect native builds. There is a significant disadvantage: a change to
DISTRO_FEATURESwill trigger a lot of unnecessary native tasks. In a test like this:$ bitbake core-image-minimal # append " systemd" to DISTRO_FEATURES $ bitbake core-image-minimal The latter build takes 44 minutes (28%) of cpu-time less with this patch (skipping 135 native tasks). Sadly wall clock time was not affected as glibc remains the bottleneck.Set native distro features to
DISTRO_FEATURES_NATIVEappended with an intersection ofDISTRO_FEATURESandDISTRO_FEATURES_FILTER_NATIVE. Current default values (baitbake.conf) are
*DISTRO_FEATURES_FILTER_NATIVE?= "api-documentation" (as gtk-doc- native
has much less dependencies when built without it)
*DISTRO_FEATURES_NATIVE?= "x11" (to enable native UIs even if target does not containe them)Do the variable setting in
native_virtclass_handler()because otherwise it could still be overridden by appends and the feature backfilling. Shuffle the early returns soDISTRO_FEATURESgets set as long as the packagename ends with "-native".Add similar variables for nativesdk.
Signed-off-by Jussi Kukkonen <[email protected]>--- meta/classes/native.bbclass | 14 ++++++++++---- meta/classes/nativesdk.bbclass | 6 ++++++ meta/conf/bitbake.conf | 9 +++++++++ 3 files changed, 25 insertions(+), 4 deletions(-)meta/classes/native.bbclass +0
-0@@ -121,14 +121,20 @@ PATH_prepend = "${COREBASE}/scripts/native-intercept:"
SSTATE_SCAN_CMD?= "${SSTATE_SCAN_CMD_NATIVE}"python
native_virtclass_handler() {meta/classes/native.bbclass +1
-4- classextend = e.data.getVar('BBCLASSEXTEND') or "" - if "native" not in classextend: - return - pn = e.data.getVar("PN") if not pn.endswith("-native"): return + # Set features here to prevent appends and distro featuresbackfill
meta/classes/native.bbclass PATH_prepend = "${COREBASE}/scripts/native-intercept:" · line 132
Quoted 4 lines
meta/classes/native.bbclass +4
-0+ # from modifying native distro features + features = set(d.getVar("DISTRO_FEATURES_NATIVE").split()) + filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVE"), d).split()) + d.setVar("DISTRO_FEATURES", " ".join(features | filtered))
You should sort the list of features to make it deterministic.
Quoted 19 lines
meta/classes/native.bbclass +5
-0+ + classextend = e.data.getVar('BBCLASSEXTEND') or "" + if "native" not in classextend: + return + def map_dependencies(varname, d, suffix = ""): if suffix: varname = varname + "_" + suffixmeta/classes/nativesdk.bbclass +1
-0@@ -70,6 +70,12 @@ python nativesdk_virtclass_handler () { if not (pn.endswith("-nativesdk") or pn.startswith("nativesdk-")): return + # Set features here to prevent appends and distro featuresbackfill
meta/classes/nativesdk.bbclass python nativesdk_virtclass_handler () { · line 77
Quoted 4 lines
meta/classes/nativesdk.bbclass +4
-0+ # from modifying nativesdk distro features + features = set(d.getVar("DISTRO_FEATURES_NATIVESDK").split()) + filtered = set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVESDK"), d).split()) + d.setVar("DISTRO_FEATURES", " ".join(features | filtered))
Sort this one as well.
Quoted 30 lines
+
e.data.setVar("MLPREFIX", "nativesdk-") e.data.setVar("PN", "nativesdk-" + e.data.getVar("PN").replace("-nativesdk", "").replace("nativesdk-", ""))
e.data.setVar("OVERRIDES",e.data.getVar("OVERRIDES", False)+
":virtclass-nativesdk")meta/conf/bitbake.conf +1
-0@@ -789,6 +789,15 @@ MACHINE_ESSENTIAL_EXTRA_RRECOMMENDS ?= "" EXTRA_IMAGE_FEATURES ??= "" IMAGE_FEATURES += "${EXTRA_IMAGE_FEATURES}" +# Native distro features (will always be used for -native, even ifthey
meta/conf/bitbake.conf +4
-0+# are not enabled for target) +DISTRO_FEATURES_NATIVE ?= "x11" +DISTRO_FEATURES_NATIVESDK ?= "x11" +# Normally target distro features will not be applied to nativebuilds:
+# Native distro features on this list will use the target feature valuemeta/conf/bitbake.conf +3
-0+DISTRO_FEATURES_FILTER_NATIVE ?= "api-documentation" +DISTRO_FEATURES_FILTER_NATIVESDK ?= "api-documentation" + DISTRO_FEATURES_BACKFILL = "pulseaudio sysvinit bluez5 gobject-introspection-data ldconfig"
MACHINE_FEATURES_BACKFILL= "rtc qemu-usermode"
Quoted 2 lines
--
2.1.4
//Peter
Re: [PATCHv3 1/1] native/nativesdk: Use fixed DISTRO_FEATURES
Quoted 69 lines · Peter Kjellerstedt
Quoted 69 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of
Jussi Kukkonen
Sent: den 11 april 2017 16:36
To: [email protected]
Subject: [OE-core] [PATCHv3 1/1] native/nativesdk: Use fixedDISTRO_FEATURESThere seems to be little advantage to letting distro features affect
native builds. There is a significant disadvantage: a change toDISTRO_FEATURESwill trigger a lot of unnecessary native tasks. In a
test like this:$ bitbake core-image-minimal # append " systemd" to DISTRO_FEATURES $ bitbake core-image-minimal The latter build takes 44 minutes (28%) of cpu-time less with this patch (skipping 135 native tasks). Sadly wall clock time was not affected as glibc remains the bottleneck.Set native distro features to
DISTRO_FEATURES_NATIVEappended with an intersection ofDISTRO_FEATURESandDISTRO_FEATURES_FILTER_NATIVE.
Current default values (baitbake.conf) are
*DISTRO_FEATURES_FILTER_NATIVE?= "api-documentation" (as gtk-doc- native
has much less dependencies when built without it)
*DISTRO_FEATURES_NATIVE?= "x11" (to enable native UIs even if target
does not containe them)Do the variable setting in
native_virtclass_handler()because otherwise
it could still be overridden by appends and the feature backfilling.
Shuffle the early returns soDISTRO_FEATURESgets set as long as the packagename ends with "-native".Add similar variables for nativesdk.
Signed-off-by Jussi Kukkonen <[email protected]>---
meta/classes/native.bbclass | 14 ++++++++++---- meta/classes/nativesdk.bbclass | 6 ++++++ meta/conf/bitbake.conf | 9 +++++++++3 files changed, 25 insertions(+), 4 deletions(-)meta/classes/native.bbclass +0
-0@@ -121,14 +121,20 @@ PATH_prepend = "${COREBASE}/scripts/native-intercept:"
SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}"python
native_virtclass_handler() {
- classextend =e.data.getVar('BBCLASSEXTEND')or "" - if "native" not in classextend:
- return
-
pn =e.data.getVar("PN")
if notpn.endswith("-native"):
return
Quoted 8 lines
Quoted 7 lines
+ # Set features here to prevent appends and distro features backfill
+ # from modifying native distro features
+ features = set(d.getVar("DISTRO_FEATURES_NATIVE").split()) + filtered =set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVE"), d).split())+ d.setVar("DISTRO_FEATURES", " ".join(features | filtered))You should sort the list of features to make it deterministic.
Do we sort DISTRO_FEATURES anywhere else?
I thought we only accessed DISTRO_FEATURES with functions which have support in bitbake (contains/filter) which means that should be unnecessary?
Cheers,
Richard
Re: [PATCHv3 1/1] native/nativesdk: Use fixed DISTRO_FEATURES
Quoted 87 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]]
Sent: den 11 april 2017 19:16
To: Peter Kjellerstedt <[email protected]>; Jussi Kukkonen <[email protected]>; [email protected]
Subject: Re: [OE-core] [PATCHv3 1/1] native/nativesdk: Use fixedDISTRO_FEATURESQuoted 69 lines · Peter Kjellerstedt
Quoted 69 lines
-----Original Message-----
From: [email protected] [mailto:[email protected]] On Behalf Of
Jussi Kukkonen
Sent: den 11 april 2017 16:36
To: [email protected]
Subject: [OE-core] [PATCHv3 1/1] native/nativesdk: Use fixedDISTRO_FEATURESThere seems to be little advantage to letting distro features affect
native builds. There is a significant disadvantage: a change toDISTRO_FEATURESwill trigger a lot of unnecessary native tasks. In a
test like this:$ bitbake core-image-minimal # append " systemd" to DISTRO_FEATURES $ bitbake core-image-minimal The latter build takes 44 minutes (28%) of cpu-time less with this patch (skipping 135 native tasks). Sadly wall clock time was not affected as glibc remains the bottleneck.Set native distro features to
DISTRO_FEATURES_NATIVEappended with an intersection ofDISTRO_FEATURESandDISTRO_FEATURES_FILTER_NATIVE.
Current default values (baitbake.conf) are
*DISTRO_FEATURES_FILTER_NATIVE?= "api-documentation" (as gtk-doc- native
has much less dependencies when built without it)
*DISTRO_FEATURES_NATIVE?= "x11" (to enable native UIs even if target
does not containe them)Do the variable setting in
native_virtclass_handler()because otherwise
it could still be overridden by appends and the feature backfilling.
Shuffle the early returns soDISTRO_FEATURESgets set as long as the packagename ends with "-native".Add similar variables for nativesdk.
Signed-off-by Jussi Kukkonen <[email protected]>---
meta/classes/native.bbclass | 14 ++++++++++---- meta/classes/nativesdk.bbclass | 6 ++++++ meta/conf/bitbake.conf | 9 +++++++++3 files changed, 25 insertions(+), 4 deletions(-)meta/classes/native.bbclass +0
-0@@ -121,14 +121,20 @@ PATH_prepend = "${COREBASE}/scripts/native-intercept:"
SSTATE_SCAN_CMD ?= "${SSTATE_SCAN_CMD_NATIVE}"python
native_virtclass_handler() {
- classextend =e.data.getVar('BBCLASSEXTEND')or "" - if "native" not in classextend:
- return
-
pn =e.data.getVar("PN")
if notpn.endswith("-native"):
returnQuoted 8 lines
Quoted 7 lines
+ # Set features here to prevent appends and distro features backfill
+ # from modifying native distro features
+ features = set(d.getVar("DISTRO_FEATURES_NATIVE").split()) + filtered =set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVE"), d).split())+ d.setVar("DISTRO_FEATURES", " ".join(features | filtered))You should sort the list of features to make it deterministic.
Quoted 1 line
Do we sort
DISTRO_FEATURESanywhere else?
No, but it is never worked upon via set() as these are here...
Quoted 3 lines
I thought we only accessed
DISTRO_FEATURESwith functions which have support in bitbake (contains/filter) which means that should be unnecessary?
Will the lack of sorting not affect the task hashes (and bitbake -e) if these are set via set() operations, which may result in the order of the features listed in the final DISTRO_VARIABLE from varying based on how Python happens to pull the features out of the set()?
Quoted 3 lines
Cheers,
Richard
//Peter
Re: [PATCHv3 1/1] native/nativesdk: Use fixed DISTRO_FEATURES
Quoted 24 lines · Peter Kjellerstedt
Quoted 13 lines
-----Original Message-----
From: Richard Purdie [mailto:[email protected]] SeQuoted 8 lines · Peter Kjellerstedt
Quoted 7 lines
+ # Set features here to prevent appends and distro features backfill
+ # from modifying native distro features
+ features = set(d.getVar("DISTRO_FEATURES_NATIVE").split()) + filtered =set(bb.utils.filter("DISTRO_FEATURES", d.getVar("DISTRO_FEATURES_FILTER_NATIVE"), d).split())+ d.setVar("DISTRO_FEATURES", " ".join(features | filtered))Quoted 2 lines
Quoted 1 line
You should sort the list of features to make it deterministic.
Do we sort
DISTRO_FEATURESanywhere else?No, but it is never worked upon via
set()as these are here...Quoted 5 lines
I thought we only accessed
DISTRO_FEATURESwith functions which have
support in bitbake (contains/filter) which means that should be unnecessary?
Quoted 4 lines
Will the lack of sorting not affect the task hashes (and bitbake -e) if these are set via
set()operations, which may result in the order of the features listed in the finalDISTRO_VARIABLEfrom varying based on how Python happens to pull the features out of theset()?
setVar operations don't actually affect task hashes directly. We track functions which use variables through getVar/getVarFlag and only if something uses it does its value affect the hash.
We can therefore safely setVar this, its only users of getVar/getVarFlag which would see its value and with DISTRO_FEATURES, we only obtain the value through contains/filter which have special handling in bitbake (similar to getVar).
So I still think sorting is unnecessary, at least in theory.
Cheers,
Richard