variable and task/function timing
Is there a document, or better yet a diagram, showing the order in which various config files, recipes, tasks, etc are parsed?
Figuring out the order of the config files is easy enough (bitbake.conf), but trying to figure out when various other parts are processed is just a guess for me right now.
Conceptually this is what I want to do:
Currently the meson build system (class) is hard-coded to always produce a buildtype of "plain", which are (basically) production builds. But there are other buildtypes that can be specified; such as "debugoptimized". I want a recipe to be able to influence the meson buildtype, and I want the user to be able to tweak this parameter from their conf/local.conf.
Here's what I want to do:
diff --git a/meta/classes/meson.bbclass b/meta/classes/meson.bbclass
index dc8c28963c..e1a13bbbf7 100644
--- a/meta/classes/meson.bbclass
+++ b/meta/classes/meson.bbclass
@@ -12,8 +12,9 @@ MESON_SOURCEPATH = "${S}"
def noprefix(var, d):
return d.getVar(var).replace(d.getVar('prefix') + '/', '', 1)
+MESON_BUILDTYPE ?= "plain"
MESONOPTS = " --prefix ${prefix} \
- --buildtype plain \
+ --buildtype ${MESON_BUILDTYPE} \
--bindir ${@noprefix('bindir', d)} \
--sbindir ${@noprefix('sbindir', d)} \
--datadir ${@noprefix('datadir', d)} \
diff --git a/meta/recipes-graphics/mesa/mesa.inc b/meta/recipes-graphics/mesa/mesa.inc
index 5838207e6b..fb232d414e 100644
--- a/meta/recipes-graphics/mesa/mesa.inc
+++ b/meta/recipes-graphics/mesa/mesa.inc
@@ -46,6 +46,20 @@ export WANT_LLVM_RELEASE = "${MESA_LLVM_RELEASE}"
MESA_LLVM_RELEASE ?= "${LLVMVERSION}"
+# set the build type to either 'release' or 'debug'
+# the upstream mesa sources actually build a debug release by default
+# but here we assume the user will want a release build by default
+MESA_BUILD_TYPE ?= "release"
+python do_check_build_type() {
+ _buildtype = d.getVar('MESA_BUILD_TYPE')
+ if _buildtype not in ['release', 'debug']:
+ bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype)
+ if _buildtype == 'debug':
+ d.setVar('MESON_BUILDTYPE', 'debugoptimized')
+ bb.plain("setting meson build type to debugoptimized")
+}
+addtask check_build_type before do_configure
+
EXTRA_OEMESON = " \
-Dshared-glapi=true \
-Dgallium-opencl=disabled \Therefore if the user doesn't do anything explicitly, a production buildtype will be produced. However, if the user were to specify the following in their conf/local.conf:
MESA_BUILD_TYPE = "debug"
I want mesa's buildtype to be "debugoptimized".
I don't want the user to specify the following in their conf/local.conf:
MESON_BUILDTYPE = "debugoptimized"
because that would cause *all* meson-built packages to switch to the debugoptimized buildtype. I only want mesa's build to be debugoptimized.
However, the above doesn't work. If I set MESA_BUILD_TYPE to "hello" in my conf/local.conf, then the build will flag it and error out asking me to specify either 'release' or 'debug', which is great. But if I set MESA_BUILD_TYPE to 'debug' the MESON_BUILDTYPE variable is always set to "plain" regardless. So this task runs, but I guess it runs too late to influence the MESON_BUILDTYPE variable?
So, how can I run this function earlier (?) so that it can set the MESON_BUILDTYPE variable correctly? I tried the following:
diff --git a/meta/classes/meson.bbclass b/meta/classes/meson.bbclass
index dc8c28963c..e1a13bbbf7 100644
--- a/meta/classes/meson.bbclass
+++ b/meta/classes/meson.bbclass
@@ -12,8 +12,9 @@ MESON_SOURCEPATH = "${S}"
def noprefix(var, d):
return d.getVar(var).replace(d.getVar('prefix') + '/', '', 1)
+MESON_BUILDTYPE ?= "plain"
MESONOPTS = " --prefix ${prefix} \
- --buildtype plain \
+ --buildtype ${MESON_BUILDTYPE} \
--bindir ${@noprefix('bindir', d)} \
--sbindir ${@noprefix('sbindir', d)} \
--datadir ${@noprefix('datadir', d)} \
diff --git a/meta/recipes-graphics/mesa/mesa.inc b/meta/recipes-graphics/mesa/mesa.inc
index 5838207e6b..b302f8ee55 100644
--- a/meta/recipes-graphics/mesa/mesa.inc
+++ b/meta/recipes-graphics/mesa/mesa.inc
@@ -46,6 +46,20 @@ export WANT_LLVM_RELEASE = "${MESA_LLVM_RELEASE}"
MESA_LLVM_RELEASE ?= "${LLVMVERSION}"
+# set the build type to either 'release' or 'debug'
+# the upstream mesa sources actually build a debug release by default
+# but here we assume the user will want a release build by default
+MESA_BUILD_TYPE ?= "release"
+def check_buildtype(d):
+ _buildtype = d.getVar('MESA_BUILD_TYPE')
+ if _buildtype not in ['release', 'debug']:
+ bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype)
+ if _buildtype == 'debug':
+ bb.plain("setting meson build type to debugoptimized")
+ return 'debugoptimized'
+ return 'plain'
+MESON_BUILDTYPE = "${@check_buildtype(d)}"
+
EXTRA_OEMESON = " \
-Dshared-glapi=true \
-Dgallium-opencl=disabled \This will print out the bb.plain() message 16 times during parsing etc, it will fail out if MESA_BUILD_TYPE is not one of 'release' or 'debug' (which is good), but if this occurs, the error message is printed out twice plus a python traceback, which might be scary to users.
The last thing I tried was the following:
diff --git a/meta/classes/meson.bbclass b/meta/classes/meson.bbclass
index dc8c28963c..e1a13bbbf7 100644
--- a/meta/classes/meson.bbclass
+++ b/meta/classes/meson.bbclass
@@ -12,8 +12,9 @@ MESON_SOURCEPATH = "${S}"
def noprefix(var, d):
return d.getVar(var).replace(d.getVar('prefix') + '/', '', 1)
+MESON_BUILDTYPE ?= "plain"
MESONOPTS = " --prefix ${prefix} \
- --buildtype plain \
+ --buildtype ${MESON_BUILDTYPE} \
--bindir ${@noprefix('bindir', d)} \
--sbindir ${@noprefix('sbindir', d)} \
--datadir ${@noprefix('datadir', d)} \
diff --git a/meta/recipes-graphics/mesa/mesa.inc b/meta/recipes-graphics/mesa/mesa.inc
index 5838207e6b..c265ffc246 100644
--- a/meta/recipes-graphics/mesa/mesa.inc
+++ b/meta/recipes-graphics/mesa/mesa.inc
@@ -46,6 +46,12 @@ export WANT_LLVM_RELEASE = "${MESA_LLVM_RELEASE}"
MESA_LLVM_RELEASE ?= "${LLVMVERSION}"
+# set the build type to either 'release' or 'debug'
+# the upstream mesa sources actually build a debug release by default
+# but here we assume the user will want a release build by default
+MESA_BUILD_TYPE ?= "release"
+MESON_BUILDTYPE = "${@bb.utils.contains('MESA_BUILD_TYPE', 'debug', 'debugoptimized', 'plain', d)}"
+
EXTRA_OEMESON = " \
-Dshared-glapi=true \
-Dgallium-opencl=disabled \This works perfectly, but it lacks the error checking and helpful messages of the previous attempts.
Any thoughts on how I can have it all? (i.e. nice error checking and error messages with the variable tweakable from conf/local.conf?
Best regards,
Trevor
Quoted 7 lines · Trevor Woerner
Is there a document, or better yet a diagram, showing the order in which various config files, recipes, tasks, etc are parsed?
Figuring out the order of the config files is easy enough (bitbake.conf), but trying to figure out when various other parts are processed is just a guess for me right now.
well the easiest way to visualize it is to think about all variables being in a large array. Each variable (each row) having a different 'value' for each 'column' and each column representing a recipe. When bitbake references a variable it is always in the context of a given recipe, so you are looking at a specific 'column'. Initially this array is filled with default values resulting from the parsing of all global .conf files, as you noticed the list of conf files and the order is set in bitbake.conf (which itself is hardcoded in bitbake, and the first file to parse).
then bitbake will parse each recipe, one by one, and update each variable's value in that array (in the relevant column). At the end of the parsing, you have all variables which are known.
you can use bitbake -e to print variables value:
bitbake -e | grep ^MESON_BUILDTYPE
it will show you the default value (if it is set) outside the context of any recipe.
bitbake -e <bar> | grep ^MESON_BUILDTYPE
will give you the value of this variable as it is set in <bar> recipe
Of course, the parsing is dealing with all operators as well.
In a global conf file you set a variable for a specific recipe with the _pn- operator. e.g. use:
MESON_BUILDTYPE_pn-bar = "debug"
to set MESON_BUILDTYPE to debug for the <bar> recipe.
Quoted 31 lines
Conceptually this is what I want to do:
Currently the meson build system (class) is hard-coded to always produce a buildtype of "plain", which are (basically) production builds. But there are other buildtypes that can be specified; such as "debugoptimized". I want a recipe to be able to influence the meson buildtype, and I want the user to be able to tweak this parameter from their
conf/local.conf.Here's what I want to do:
diff --git a/meta/classes/meson.bbclass b/meta/classes/meson.bbclass index dc8c28963c..e1a13bbbf7 100644 --- a/meta/classes/meson.bbclass +++ b/meta/classes/meson.bbclass @@ -12,8 +12,9 @@ MESON_SOURCEPATH = "${S}" def noprefix(var, d): return d.getVar(var).replace(d.getVar('prefix') + '/', '', 1) +MESON_BUILDTYPE ?= "plain" MESONOPTS = " --prefix ${prefix} \ - --buildtype plain \ + --buildtype ${MESON_BUILDTYPE} \ --bindir ${@noprefix('bindir', d)} \ --sbindir ${@noprefix('sbindir', d)} \ --datadir ${@noprefix('datadir', d)} \ diff --git a/meta/recipes-graphics/mesa/mesa.inc b/meta/recipes-graphics/mesa/mesa.inc index 5838207e6b..fb232d414e 100644 --- a/meta/recipes-graphics/mesa/mesa.inc +++ b/meta/recipes-graphics/mesa/mesa.inc
Quoted 8 lines
@@ -46,6 +46,20 @@ export WANT_LLVM_RELEASE = "${MESA_LLVM_RELEASE}"
MESA_LLVM_RELEASE ?= "${LLVMVERSION}"
+# set the build type to either 'release' or 'debug'
+# the upstream mesa sources actually build a debug release by default
+# but here we assume the user will want a release build by default
+MESA_BUILD_TYPE ?= "release"The content of the class is parsed when it is inherited. So you need to initialize class variables *before* you inherit the class. e.g.
MESA_BUILD_TYPE ?= "release"
...
inherit meson
in your case is very different from
inherit meson
...MESA_BUILD_TYPE ?= "release"
Quoted 18 lines
+python
do_check_build_type(){
+_buildtype=d.getVar('MESA_BUILD_TYPE')
+ if_buildtypenot in ['release', 'debug']:
+bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype)
+ if_buildtype== 'debug':
+d.setVar('MESON_BUILDTYPE', 'debugoptimized')
+bb.plain("setting meson build type to debugoptimized")
+}
+addtaskcheck_build_typebeforedo_configure+
EXTRA_OEMESON= " \
-Dshared-glapi=true \
-Dgallium-opencl=disabled \Therefore if the user doesn't do anything explicitly, a production buildtype will be produced. However, if the user were to specify the following in their
conf/local.conf:
Quoted 7 lines
MESA_BUILD_TYPE= "debug"I want mesa's buildtype to be "debugoptimized".
I don't want the user to specify the following in their
conf/local.conf:
MESON_BUILDTYPE= "debugoptimized"
use _pn- operator for that, as mentioned above.
Quoted 10 lines
because that would cause *all* meson-built packages to switch to the debugoptimized buildtype. I only want mesa's build to be debugoptimized.
However, the above doesn't work. If I set
MESA_BUILD_TYPEto "hello" in myconf/local.conf, then the build will flag it and error out asking me to specify either 'release' or 'debug', which is great. But if I setMESA_BUILD_TYPEto 'debug' theMESON_BUILDTYPEvariable is always set to "plain" regardless. So this task runs, but I guess it runs too late to influence theMESON_BUILDTYPEvariable?
That is unexpected.. I would have thought it should work. You are mixing BUILD_TYPE and BUILDTYPE in your email.. maybe you are mixing that in your testing as well? What is set in local.conf will be parsed first, so if you set a variable there, it will be 'set' when you parse the recipe, so when you use ?= it should be a no-op.
Quoted 101 lines
So, how can I run this function earlier (?) so that it can set the
MESON_BUILDTYPEvariable correctly? I tried the following:diff --git a/meta/classes/meson.bbclass b/meta/classes/meson.bbclass index dc8c28963c..e1a13bbbf7 100644 --- a/meta/classes/meson.bbclass +++ b/meta/classes/meson.bbclass @@ -12,8 +12,9 @@ MESON_SOURCEPATH = "${S}" def noprefix(var, d): return d.getVar(var).replace(d.getVar('prefix') + '/', '', 1) +MESON_BUILDTYPE ?= "plain" MESONOPTS = " --prefix ${prefix} \ - --buildtype plain \ + --buildtype ${MESON_BUILDTYPE} \ --bindir ${@noprefix('bindir', d)} \ --sbindir ${@noprefix('sbindir', d)} \ --datadir ${@noprefix('datadir', d)} \ diff --git a/meta/recipes-graphics/mesa/mesa.inc b/meta/recipes-graphics/mesa/mesa.inc index 5838207e6b..b302f8ee55 100644 --- a/meta/recipes-graphics/mesa/mesa.inc +++ b/meta/recipes-graphics/mesa/mesa.inc @@ -46,6 +46,20 @@ export WANT_LLVM_RELEASE = "${MESA_LLVM_RELEASE}" MESA_LLVM_RELEASE ?= "${LLVMVERSION}" +# set the build type to either 'release' or 'debug' +# the upstream mesa sources actually build a debug release by default +# but here we assume the user will want a release build by default +MESA_BUILD_TYPE ?= "release" +def check_buildtype(d): + _buildtype = d.getVar('MESA_BUILD_TYPE') + if _buildtype not in ['release', 'debug']: + bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype) + if _buildtype == 'debug': + bb.plain("setting meson build type to debugoptimized") + return 'debugoptimized' + return 'plain' +MESON_BUILDTYPE = "${@check_buildtype(d)}" + EXTRA_OEMESON = " \ -Dshared-glapi=true \ -Dgallium-opencl=disabled \This will print out the
bb.plain()message 16 times during parsing etc, it will fail out ifMESA_BUILD_TYPEis not one of 'release' or 'debug' (which is good), but if this occurs, the error message is printed out twice plus a python traceback, which might be scary to users.The last thing I tried was the following:
diff --git a/meta/classes/meson.bbclass b/meta/classes/meson.bbclass index dc8c28963c..e1a13bbbf7 100644 --- a/meta/classes/meson.bbclass +++ b/meta/classes/meson.bbclass @@ -12,8 +12,9 @@ MESON_SOURCEPATH = "${S}" def noprefix(var, d): return d.getVar(var).replace(d.getVar('prefix') + '/', '', 1) +MESON_BUILDTYPE ?= "plain" MESONOPTS = " --prefix ${prefix} \ - --buildtype plain \ + --buildtype ${MESON_BUILDTYPE} \ --bindir ${@noprefix('bindir', d)} \ --sbindir ${@noprefix('sbindir', d)} \ --datadir ${@noprefix('datadir', d)} \ diff --git a/meta/recipes-graphics/mesa/mesa.inc b/meta/recipes-graphics/mesa/mesa.inc index 5838207e6b..c265ffc246 100644 --- a/meta/recipes-graphics/mesa/mesa.inc +++ b/meta/recipes-graphics/mesa/mesa.inc @@ -46,6 +46,12 @@ export WANT_LLVM_RELEASE = "${MESA_LLVM_RELEASE}" MESA_LLVM_RELEASE ?= "${LLVMVERSION}" +# set the build type to either 'release' or 'debug' +# the upstream mesa sources actually build a debug release by default +# but here we assume the user will want a release build by default +MESA_BUILD_TYPE ?= "release" +MESON_BUILDTYPE = "${@bb.utils.contains('MESA_BUILD_TYPE', 'debug', 'debugoptimized', 'plain', d)}" + EXTRA_OEMESON = " \ -Dshared-glapi=true \ -Dgallium-opencl=disabled \This works perfectly, but it lacks the error checking and helpful messages of the previous attempts.
Any thoughts on how I can have it all? (i.e. nice error checking and error messages with the variable tweakable from
conf/local.conf?Best regards,
Trevor
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.View/Reply Online (#47628): https://lists.yoctoproject.org/g/yocto/message/47628
Mute This Topic: https://lists.yoctoproject.org/mt/68144480/1279857
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 25 lines · Nicolas Dechesne
Quoted 7 lines · Trevor Woerner
Is there a document, or better yet a diagram, showing the order in which various config files, recipes, tasks, etc are parsed?
Figuring out the order of the config files is easy enough (bitbake.conf), but trying to figure out when various other parts are processed is just a guess for me right now.
well the easiest way to visualize it is to think about all variables being in a large array. Each variable (each row) having a different 'value' for each 'column' and each column representing a recipe. When bitbake references a variable it is always in the context of a given recipe, so you are looking at a specific 'column'. Initially this array is filled with default values resulting from the parsing of all global .conf files, as you noticed the list of conf files and the order is set in bitbake.conf (which itself is hardcoded in bitbake, and the first file to parse).
then bitbake will parse each recipe, one by one, and update each variable's value in that array (in the relevant column). At the end of the parsing, you have all variables which are known.
you can use bitbake -e to print variables value:
Quoted 8 lines
bitbake -e | grep ^MESON_BUILDTYPE
it will show you the default value (if it is set) outside the context of any recipe.
bitbake -e <bar> | grep ^MESON_BUILDTYPE
will give you the value of this variable as it is set in <bar> recipe
Ooh, I like this already! Your row/column concept is a good mental map of the process. Your explanation cleared up the "bitbake -e" vs "bitbake -e <bar>" case for me too!
Quoted 25 lines
The content of the class is parsed when it is inherited. So you need to initialize class variables *before* you inherit the class. e.g.
MESA_BUILD_TYPE?= "release"
...
inherit mesonin your case is very different from
inherit meson
...MESA_BUILD_TYPE?= "release"Quoted 13 lines
+python
do_check_build_type(){
+_buildtype=d.getVar('MESA_BUILD_TYPE')
+ if_buildtypenot in ['release', 'debug']:
+bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype)
+ if_buildtype== 'debug':
+d.setVar('MESON_BUILDTYPE', 'debugoptimized')
+bb.plain("setting meson build type to debugoptimized")
+}
+addtaskcheck_build_typebeforedo_configure+
EXTRA_OEMESON= " \
-Dshared-glapi=true \
-Dgallium-opencl=disabled \
Whether I move the above to before or after the "inherit meson..." line makes no difference. Probably because the variable is being set by a task (which, I assume, is too late to have any effect, which is a large part of why I wrote this email: when do these tasks get called with respect to how variable are being set by all the different ways they're being set?)
Quoted 5 lines
You are
mixingBUILD_TYPEand BUILDTYPE in your email.. maybe you are mixing that in your testing as well? What is set in local.conf will be parsed first, so if you set a variable there, it will be 'set' when you parse the recipe, so when you use ?= it should be a no-op.
Notice, though, that I'm using two different variables:
1) MESA_BUILD_TYPE: which can be set to "debug" or "release"
2) MESON_BUILDTYPE: which can be set to "plain" or "debugoptimized"
MESA_BUILD_TYPE is how the user tweaks the MESON_BUILDTYPE from the mesa recipe. Given what you've said above, however, I can see now that (as you say) setting:
MESON_BUILDTYPE_pn-mesa = ...
in conf/local.conf will do what I want since this variable is recipe-specific and setting it in one recipe this way won't affect it in others.
I'm sure some will say "there's your solution, do it that way, problem solved". However, I think "debug/release" are much more natural than "plain/debugoptimized". I can't change the MESON_BUILDTYPE strings, they have to be one of those two, so I introduced MESA_BUILD_TYPE as a level-of-indirection above MESON_BUILDTYPE to allow the user to use the more natural "debug/release" wording.
So my real question is (and maybe I'm just yak shaving at this point): given the row/column view of variable setting, how do we factor in the element of time? For example, *when* do the variables referenced by anonymous python functions and by tasks get set?
This works (but doesn't allow me the space for nice error checking):
MESON_BUILDTYPE = "${@bb.utils.contains('MESA_BUILD_TYPE', 'debug', 'debugoptimized', 'plain', d)}"and this doesn't:
python do_check_build_type() {
_buildtype = d.getVar('MESA_BUILD_TYPE')
if _buildtype not in ['release', 'debug']:
bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype)
if _buildtype == 'debug':
d.setVar('MESON_BUILDTYPE', 'debugoptimized')
bb.plain("setting meson build type to debugoptimized")
}
addtask check_build_type before do_configure
Quoted 17 lines · Trevor Woerner
Quoted 7 lines · Nicolas Dechesne
Quoted 7 lines
+python
do_check_build_type(){
+_buildtype=d.getVar('MESA_BUILD_TYPE')
+ if_buildtypenot in ['release', 'debug']:
+ bb.fatal("unknown build type (%s), please set to
either 'release' or 'debug'" %_buildtype)
+ if_buildtype== 'debug':
+d.setVar('MESON_BUILDTYPE', 'debugoptimized')Quoted 8 lines
Quoted 8 lines
+ bb.plain("setting meson build type to
debugoptimized")+} +addtask check_build_type before do_configure + EXTRA_OEMESON = " \ -Dshared-glapi=true \ -Dgallium-opencl=disabled \
Quoted 6 lines
Whether I move the above to before or after the "inherit meson..." line makes no difference. Probably because the variable is being set by a task (which, I assume, is too late to have any effect, which is a large part of why I wrote this email: when do these tasks get called with respect to how variable are being set by all the different ways they're being set?)
Tasks run in isolation, if you change the datastore in a task it has no way to get "seen" by other tasks. They're separate processes.
That is why the setVar in a task has no effect outside that task.
Cheers,
Richard
Quoted 26 lines · Richard Purdie
Quoted 17 lines · Trevor Woerner
Quoted 7 lines · Nicolas Dechesne
Quoted 7 lines
+python
do_check_build_type(){
+_buildtype=d.getVar('MESA_BUILD_TYPE')
+ if_buildtypenot in ['release', 'debug']:
+ bb.fatal("unknown build type (%s), please set to
either 'release' or 'debug'" %_buildtype)
+ if_buildtype== 'debug':
+d.setVar('MESON_BUILDTYPE', 'debugoptimized')Quoted 8 lines
Quoted 8 lines
+ bb.plain("setting meson build type to
debugoptimized")+} +addtask check_build_type before do_configure + EXTRA_OEMESON = " \ -Dshared-glapi=true \ -Dgallium-opencl=disabled \Quoted 6 lines
Whether I move the above to before or after the "inherit meson..." line makes no difference. Probably because the variable is being set by a task (which, I assume, is too late to have any effect, which is a large part of why I wrote this email: when do these tasks get called with respect to how variable are being set by all the different ways they're being set?)
Quoted 2 lines
Tasks run in isolation, if you change the datastore in a task it has no way to get "seen" by other tasks. They're separate processes.
right... so my array is not 2D, but 3D then! but as i just said, i missed the addtask in the first email.
Quoted 9 lines
That is why the setVar in a task has no effect outside that task.
Cheers,
Richard
Quoted 25 lines · Richard Purdie
Quoted 17 lines · Trevor Woerner
Quoted 7 lines · Nicolas Dechesne
Quoted 7 lines
+python
do_check_build_type(){
+_buildtype=d.getVar('MESA_BUILD_TYPE')
+ if_buildtypenot in ['release', 'debug']:
+ bb.fatal("unknown build type (%s), please set to
either 'release' or 'debug'" %_buildtype)
+ if_buildtype== 'debug':
+d.setVar('MESON_BUILDTYPE', 'debugoptimized')Quoted 8 lines
Quoted 8 lines
+ bb.plain("setting meson build type to
debugoptimized")+} +addtask check_build_type before do_configure + EXTRA_OEMESON = " \ -Dshared-glapi=true \ -Dgallium-opencl=disabled \Quoted 6 lines
Whether I move the above to before or after the "inherit meson..." line makes no difference. Probably because the variable is being set by a task (which, I assume, is too late to have any effect, which is a large part of why I wrote this email: when do these tasks get called with respect to how variable are being set by all the different ways they're being set?)
Quoted 4 lines
Tasks run in isolation, if you change the datastore in a task it has no way to get "seen" by other tasks. They're separate processes.
That is why the setVar in a task has no effect outside that task.
Excellent, that explains a lot, thanks!
So in this case, the answer to the question of "when" is: never :-)
hey Trevor,
Quoted 83 lines · Trevor Woerner
Quoted 25 lines · Nicolas Dechesne
Quoted 7 lines · Trevor Woerner
Is there a document, or better yet a diagram, showing the order in which various config files, recipes, tasks, etc are parsed?
Figuring out the order of the config files is easy enough (bitbake.conf), but trying to figure out when various other parts are processed is just a guess for me right now.
well the easiest way to visualize it is to think about all variables being in a large array. Each variable (each row) having a different 'value' for each 'column' and each column representing a recipe. When bitbake references a variable it is always in the context of a given recipe, so you are looking at a specific 'column'. Initially this array is filled with default values resulting from the parsing of all global .conf files, as you noticed the list of conf files and the order is set in bitbake.conf (which itself is hardcoded in bitbake, and the first file to parse).
then bitbake will parse each recipe, one by one, and update each variable's value in that array (in the relevant column). At the end of the parsing, you have all variables which are known.
you can use bitbake -e to print variables value:
Quoted 8 lines
bitbake -e | grep ^MESON_BUILDTYPE
it will show you the default value (if it is set) outside the context of any recipe.
bitbake -e <bar> | grep ^MESON_BUILDTYPE
will give you the value of this variable as it is set in <bar> recipe
Ooh, I like this already! Your row/column concept is a good mental map of the process. Your explanation cleared up the "bitbake -e" vs "bitbake -e <bar>" case for me too!
Quoted 25 lines
The content of the class is parsed when it is inherited. So you need to initialize class variables *before* you inherit the class. e.g.
MESA_BUILD_TYPE?= "release"
...
inherit mesonin your case is very different from
inherit meson
...MESA_BUILD_TYPE?= "release"Quoted 13 lines
+python
do_check_build_type(){
+_buildtype=d.getVar('MESA_BUILD_TYPE')
+ if_buildtypenot in ['release', 'debug']:
+bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype)
+ if_buildtype== 'debug':
+d.setVar('MESON_BUILDTYPE', 'debugoptimized')
+bb.plain("setting meson build type to debugoptimized")
+}
+addtaskcheck_build_typebeforedo_configure+
EXTRA_OEMESON= " \
-Dshared-glapi=true \
-Dgallium-opencl=disabled \Whether I move the above to before or after the "inherit meson..." line makes no difference. Probably because the variable is being set by a task (which, I assume, is too late to have any effect, which is a large part of why I wrote this email: when do these tasks get called with respect to how variable are being set by all the different ways they're being set?)
Quoted 5 lines
You are
mixingBUILD_TYPEand BUILDTYPE in your email.. maybe you are mixing that in your testing as well? What is set in local.conf will be parsed first, so if you set a variable there, it will be 'set' when you parse the recipe, so when you use ?= it should be a no-op.Notice, though, that I'm using two different variables:
1)
MESA_BUILD_TYPE: which can be set to "debug" or "release"
2)MESON_BUILDTYPE: which can be set to "plain" or "debugoptimized"
Quoted 8 lines
MESA_BUILD_TYPEis how the user tweaks theMESON_BUILDTYPEfrom the mesa recipe. Given what you've said above, however, I can see now that (as you say) setting:MESON_BUILDTYPE_pn-mesa = ...
in
conf/local.confwill do what I want since this variable is recipe-specific and setting it in one recipe this way won't affect it in others.
right.. I was confused, I missed a few things in your email (well, it was really long ;).
Quoted 19 lines
I'm sure some will say "there's your solution, do it that way, problem solved". However, I think "debug/release" are much more natural than "plain/debugoptimized". I can't change the
MESON_BUILDTYPEstrings, they have to be one of those two, so I introducedMESA_BUILD_TYPEas a level-of-indirection aboveMESON_BUILDTYPEto allow the user to use the more natural "debug/release" wording.So my real question is (and maybe I'm just yak shaving at this point): given the row/column view of variable setting, how do we factor in the element of time? For example, *when* do the variables referenced by anonymous python functions and by tasks get set?
This works (but doesn't allow me the space for nice error checking):
MESON_BUILDTYPE = "${@bb.utils.contains('MESA_BUILD_TYPE', 'debug', 'debugoptimized', 'plain', d)}"and this doesn't:
Quoted 9 lines
python
do_check_build_type(){
_buildtype=d.getVar('MESA_BUILD_TYPE')
if_buildtypenot in ['release', 'debug']:
bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype)
if_buildtype== 'debug':
d.setVar('MESON_BUILDTYPE', 'debugoptimized')
bb.plain("setting meson build type to debugoptimized")
}
addtaskcheck_build_typebeforedo_configure
I missed the fact that you were using a task.
You can probably call do_check_build_type() when setting MESON_BUILDTYPE instead of using a task, e.g. something like this:MESON_BUILDTYPE = "${@do_check_build_type(d.getVar('MESA_BUILD_TYPE'))}"
Quoted 2 lines · Nicolas Dechesne
right.. I was confused, I missed a few things in your email (well, it was really long ;).
lol
Apparently writing long emails is therapeutic for me.
Quoted 28 lines
Quoted 18 lines
I'm sure some will say "there's your solution, do it that way, problem solved". However, I think "debug/release" are much more natural than "plain/debugoptimized". I can't change the
MESON_BUILDTYPEstrings, they have to be one of those two, so I introducedMESA_BUILD_TYPEas a level-of-indirection aboveMESON_BUILDTYPEto allow the user to use the more natural "debug/release" wording.So my real question is (and maybe I'm just yak shaving at this point): given the row/column view of variable setting, how do we factor in the element of time? For example, *when* do the variables referenced by anonymous python functions and by tasks get set?
This works (but doesn't allow me the space for nice error checking):
MESON_BUILDTYPE = "${@bb.utils.contains('MESA_BUILD_TYPE', 'debug', 'debugoptimized', 'plain', d)}"and this doesn't:
Quoted 9 lines
python
do_check_build_type(){
_buildtype=d.getVar('MESA_BUILD_TYPE')
if_buildtypenot in ['release', 'debug']:
bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype)
if_buildtype== 'debug':
d.setVar('MESON_BUILDTYPE', 'debugoptimized')
bb.plain("setting meson build type to debugoptimized")
}
addtaskcheck_build_typebeforedo_configure
Quoted 4 lines
I missed the fact that you were using a task.
You can probably calldo_check_build_type()when settingMESON_BUILDTYPEinstead of using a task, e.g. something like this:MESON_BUILDTYPE= "${@do_check_build_type(d.getVar('MESA_BUILD_TYPE'))}"
I did propose a 3rd way (in the original email) which does work which is very similar to what you're suggesting:
# set the build type to either 'release' or 'debug'
# the upstream mesa sources actually build a debug release by default
# but here we assume the user will want a release build by default
MESA_BUILD_TYPE ?= "release"
def check_buildtype(d):
_buildtype = d.getVar('MESA_BUILD_TYPE')
if _buildtype not in ['release', 'debug']:
bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype)
if _buildtype == 'debug':
#bb.plain("setting meson build type to debugoptimized")
return 'debugoptimized'
return 'plain'
MESON_BUILDTYPE = "${@check_buildtype(d)}"The user experience is rather different though. Let's say the user sets the following in conf/local.conf:
MESA_BUILD_TYPE = "hello"
Using the "addtask" method (which doesn't work) produces a very nice error message:
$ bitbake mesa -c configure Loading cache: 100% |#########################################################################################################################| Time: 0:00:00 Loaded 2257 entries from dependency cache. Parsing recipes: 100% |#######################################################################################################################| Time: 0:00:01 Parsing of 1528 .bb files complete (1524 cached, 4 parsed). 2259 targets, 103 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependencies
Build Configuration:
BB_VERSION = "1.44.0"
BUILD_SYS = "x86_64-linux"
NATIVELSBSTRING = "opensuseleap-15.1"
TARGET_SYS = "arm-oe-linux-gnueabi"
MACHINE = "tinker-rk3288"
DISTRO = "nodistro"
DISTRO_VERSION = "nodistro.0"
TUNE_FEATURES = "arm armv7ve vfp thumb neon callconvention-hard"
TARGET_FPU = "hard"
meta-rockchip = "master:e8fd1f92ed59e0e71a7418779b912c5da342495c"
meta-tweaks = "master:b9184893949356a2da43bb2b5ec88dd573a5db0d"
meta = "master:093a1971f2ae12e1f514598da984f268607e550b"
meta-oe = "master:851321744e17e51aeb822a8d88c3532dffdf1cef"
Initialising tasks: 100% |####################################################################################################################| Time: 0:00:00 Sstate summary: Wanted 0 Found 0 Missed 0 Current 77 (0% match, 100% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
ERROR: mesa-2_19.2.4-r0 do_check_build_type: unknown build type (hello), please set to either 'release' or 'debug'
ERROR: Logfile of failure stored in: /z/build-master/tinker-rk3288/build/tmp-glibc/work/armv7vet2hf-neon-oe-linux-gnueabi/mesa/2_19.2.4-r0/temp/log.do_check_build_type.12869
ERROR: Task (/opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa_19.2.4.bb:do_check_build_type) failed with exit code '1'
NOTE: Tasks Summary: Attempted 606 tasks of which 603 didn't need to be rerun and 1 failed.
Using the method proposed above produces:
$ bitbake mesa -c configure
Loading cache: 100% |#########################################################################################################################| Time: 0:00:00
Loaded 2257 entries from dependency cache.
ERROR: /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa-gl_19.2.4.bb: unknown build type (hello), please set to either 'release' or 'debug' ETA: --:--:--
WARNING: /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa-gl_19.2.4.bb: Exception during build_dependencies for meson_do_configure
WARNING: /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa-gl_19.2.4.bb: Error during finalise of /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa-gl_19.2.4.bb
ERROR: /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa_19.2.4.bb: unknown build type (hello), please set to either 'release' or 'debug'
WARNING: /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa_19.2.4.bb: Exception during build_dependencies for meson_do_configure
WARNING: /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa_19.2.4.bb: Error during finalise of /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa_19.2.4.bb
ERROR: ExpansionError during parsing /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa-gl_19.2.4.bb
Traceback (most recent call last):
File "Var <MESON_BUILDTYPE>", line 1, in <module>
File "/opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa.inc", line 56, in check_buildtype(d=<bb.data_smart.DataSmart object at 0x7ffac8d0f4a8>):
if _buildtype not in ['release', 'debug']:
> bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype)
if _buildtype == 'debug':
File "/opt/oe/configs/z/build-master/tinker-rk3288/layers/bitbake/lib/bb/__init__.py", line 108, in fatal:
mainlogger.critical(''.join(args), extra=kwargs)
> raise BBHandledException() bb.data_smart.ExpansionError: Failure expanding variable MESON_BUILDTYPE, expression was ${@check_buildtype(d)} which triggered exception BBHandledException:
Summary: There were 4 WARNING messages shown.
Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
Quoted 119 lines · Trevor Woerner
Quoted 2 lines · Nicolas Dechesne
right.. I was confused, I missed a few things in your email (well, it was really long ;).
lol
Apparently writing long emails is therapeutic for me.Quoted 28 lines
Quoted 18 lines
I'm sure some will say "there's your solution, do it that way, problem solved". However, I think "debug/release" are much more natural than "plain/debugoptimized". I can't change the
MESON_BUILDTYPEstrings, they have to be one of those two, so I introducedMESA_BUILD_TYPEas a level-of-indirection aboveMESON_BUILDTYPEto allow the user to use the more natural "debug/release" wording.So my real question is (and maybe I'm just yak shaving at this point): given the row/column view of variable setting, how do we factor in the element of time? For example, *when* do the variables referenced by anonymous python functions and by tasks get set?
This works (but doesn't allow me the space for nice error checking):
MESON_BUILDTYPE = "${@bb.utils.contains('MESA_BUILD_TYPE', 'debug', 'debugoptimized', 'plain', d)}"and this doesn't:
Quoted 9 lines
python
do_check_build_type(){
_buildtype=d.getVar('MESA_BUILD_TYPE')
if_buildtypenot in ['release', 'debug']:
bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype)
if_buildtype== 'debug':
d.setVar('MESON_BUILDTYPE', 'debugoptimized')
bb.plain("setting meson build type to debugoptimized")
}
addtaskcheck_build_typebeforedo_configureQuoted 4 lines
I missed the fact that you were using a task.
You can probably calldo_check_build_type()when settingMESON_BUILDTYPEinstead of using a task, e.g. something like this:MESON_BUILDTYPE= "${@do_check_build_type(d.getVar('MESA_BUILD_TYPE'))}"I did propose a 3rd way (in the original email) which does work which is very similar to what you're suggesting:
# set the build type to either 'release' or 'debug' # the upstream mesa sources actually build a debug release by default # but here we assume the user will want a release build by default MESA_BUILD_TYPE ?= "release" def check_buildtype(d): _buildtype = d.getVar('MESA_BUILD_TYPE') if _buildtype not in ['release', 'debug']: bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype) if _buildtype == 'debug': #bb.plain("setting meson build type to debugoptimized") return 'debugoptimized' return 'plain' MESON_BUILDTYPE = "${@check_buildtype(d)}"The user experience is rather different though. Let's say the user sets the following in
conf/local.conf:
MESA_BUILD_TYPE= "hello"Using the "addtask" method (which doesn't work) produces a very nice error message:
$ bitbake mesa -c configure Loading cache: 100% |#########################################################################################################################| Time: 0:00:00 Loaded 2257 entries from dependency cache. Parsing recipes: 100% |#######################################################################################################################| Time: 0:00:01 Parsing of 1528 .bb files complete (1524 cached, 4 parsed). 2259 targets, 103 skipped, 0 masked, 0 errors. NOTE: Resolving any missing task queue dependenciesBuild Configuration:
BB_VERSION= "1.44.0"
BUILD_SYS= "x86_64-linux"
NATIVELSBSTRING = "opensuseleap-15.1"
TARGET_SYS= "arm-oe-linux-gnueabi"
MACHINE = "tinker-rk3288"
DISTRO = "nodistro"
DISTRO_VERSION= "nodistro.0"
TUNE_FEATURES= "arm armv7ve vfp thumb neon callconvention-hard"
TARGET_FPU= "hard"
meta-rockchip = "master:e8fd1f92ed59e0e71a7418779b912c5da342495c"
meta-tweaks = "master:b9184893949356a2da43bb2b5ec88dd573a5db0d"
meta = "master:093a1971f2ae12e1f514598da984f268607e550b"
meta-oe = "master:851321744e17e51aeb822a8d88c3532dffdf1cef"Initialising tasks: 100% |####################################################################################################################| Time: 0:00:00 Sstate summary: Wanted 0 Found 0 Missed 0 Current 77 (0% match, 100% complete)
NOTE: Executing Tasks
NOTE: Setscene tasks completed
ERROR: mesa-2_19.2.4-r0do_check_build_type: unknown build type (hello), please set to either 'release' or 'debug'
ERROR: Logfile of failure stored in: /z/build-master/tinker-rk3288/build/tmp-glibc/work/armv7vet2hf-neon-oe-linux-gnueabi/mesa/2_19.2.4-r0/temp/log.do_check_build_type.12869
ERROR: Task (/opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa_19.2.4.bb:do_check_build_type) failed with exit code '1'
NOTE: Tasks Summary: Attempted 606 tasks of which 603 didn't need to be rerun and 1 failed.Using the method proposed above produces:
$ bitbake mesa -c configure Loading cache: 100% |#########################################################################################################################| Time: 0:00:00 Loaded 2257 entries from dependency cache. ERROR: /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa-gl_19.2.4.bb: unknown build type (hello), please set to either 'release' or 'debug' ETA: --:--:-- WARNING: /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa-gl_19.2.4.bb: Exception during build_dependencies for meson_do_configure WARNING: /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa-gl_19.2.4.bb: Error during finalise of /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa-gl_19.2.4.bb ERROR: /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa_19.2.4.bb: unknown build type (hello), please set to either 'release' or 'debug' WARNING: /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa_19.2.4.bb: Exception during build_dependencies for meson_do_configure WARNING: /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa_19.2.4.bb: Error during finalise of /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa_19.2.4.bb ERROR: ExpansionError during parsing /opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa-gl_19.2.4.bb Traceback (most recent call last): File "Var <MESON_BUILDTYPE>", line 1, in <module> File "/opt/oe/configs/z/build-master/tinker-rk3288/layers/openembedded-core/meta/recipes-graphics/mesa/mesa.inc", line 56, in check_buildtype(d=<bb.data_smart.DataSmart object at 0x7ffac8d0f4a8>): if _buildtype not in ['release', 'debug']: > bb.fatal("unknown build type (%s), please set to either 'release' or 'debug'" % _buildtype) if _buildtype == 'debug': File "/opt/oe/configs/z/build-master/tinker-rk3288/layers/bitbake/lib/bb/__init__.py", line 108, in fatal: mainlogger.critical(''.join(args), extra=kwargs) > raise BBHandledException()
Quoted 5 lines
bb.data_smart.ExpansionError: Failure expanding variable
MESON_BUILDTYPE, expression was ${@check_buildtype(d)} which triggered exception BBHandledException:Summary: There were 4 WARNING messages shown.
Summary: There were 3 ERROR messages shown, returning a non-zero exit code.
then you could use an anonymous python function instead.
python () {
_buildtype = d.getVar('MESA_BUILD_TYPE') if _buildtype not in ['release', 'debug']:
bb.fatal("unknown build type (%s), please set to
either 'release' or 'debug'" % _buildtype)
if _buildtype == 'debug':
d.setVar('MESON_BUILDTYPE', 'debugoptimized')
}or something along these lines. that would be done during parsing, so the variable should be set for all tasks.
As a user, I much prefer having one MESON_BUILD_TYPE variable that I can modify using the pn-operator, even if the acceptable values are a little less pretty. The alternative is a variable that I have to guess that name of and what valid values are per-recipe. Sure it's just MESA_BUILD_TYPE now, but next week is there going to be GLIB_BUILD_TYPE and GTK_BUILD_TYPE too?
Also, if "debugoptimized" is the accepted name from the meson world, I feel it's better to keep it rather than second-guessing them with your own name.
--Aaron
Quoted 4 lines · Trevor Woerner
Quoted 1 line · Nicolas Dechesne
On Wed, Dec 11, 2019 at 8:18 AM Trevor Woerner <[email protected]>
wrote:
...
Quoted 56 lines
Quoted 5 lines
You are
mixingBUILD_TYPEand BUILDTYPE in your email.. maybe you are mixing that in your testing as well? What is set in local.conf will be parsed first, so if you set a variable there, it will be 'set' when you parse the recipe, so when you use ?= it should be a no-op.Notice, though, that I'm using two different variables:
1)
MESA_BUILD_TYPE: which can be set to "debug" or "release"
2)MESON_BUILDTYPE: which can be set to "plain" or "debugoptimized"
MESA_BUILD_TYPEis how the user tweaks theMESON_BUILDTYPEfrom the mesa recipe. Given what you've said above, however, I can see now that (as you say)
setting:MESON_BUILDTYPE_pn-mesa = ...
in
conf/local.confwill do what I want since this variable is recipe-specific
and setting it in one recipe this way won't affect it in others.I'm sure some will say "there's your solution, do it that way, problem solved". However, I think "debug/release" are much more natural than "plain/debugoptimized". I can't change the
MESON_BUILDTYPEstrings, they have to be one of those two, so I introducedMESA_BUILD_TYPEas a level-of-indirection aboveMESON_BUILDTYPEto allow the user to use the more
natural "debug/release" wording.So my real question is (and maybe I'm just yak shaving at this point):
given
the row/column view of variable setting, how do we factor in the element of time? For example, *when* do the variables referenced by anonymous python functions and by tasks get set?This works (but doesn't allow me the space for nice error checking):
MESON_BUILDTYPE = "${@bb.utils.contains('MESA_BUILD_TYPE','debug', 'debugoptimized', 'plain', d)}"
and this doesn't:
python
do_check_build_type(){
_buildtype=d.getVar('MESA_BUILD_TYPE')
if_buildtypenot in ['release', 'debug']:
bb.fatal("unknown build type (%s), please set to either
'release' or 'debug'" %_buildtype)
if_buildtype== 'debug':
d.setVar('MESON_BUILDTYPE', 'debugoptimized')bb.plain("setting meson build type to debugoptimized")
}
addtaskcheck_build_typebeforedo_configure
-=-=-=-=-=-=-=-=-=-=-=-