Layer priorities influencing default version selection
Hi all,
Andrea pointed out a situation where he has seen the layer priority overriding version selection, and I've been able to confirm it.
Basically, if you have a recipe with a lower version in a layer with a higher priority it selects the lower version. What's more after digging further I found there were some rather anomalous interactions with the version logic and BBCLASSEXTEND. Here's an example using Poky:
1. Firstly, copy meta/recipes-support/curl to meta-yocto/recipes-support, then rename the version in meta-yocto so that its version is 7.20.0. At this point both meta/ and meta-yocto/ have the same layer priority of 5.
2. "bitbake -s | grep ^curl" reports:
curl :7.21.7-r0 curl-native :7.21.7-r0 curl-nativesdk :7.21.7-r0
3. Now increase the layer priority in meta-yocto/conf/layer.conf to 6, and run "bitbake -s | grep ^curl" again:
curl :7.20.0-r0 curl-native :7.21.7-r0 curl-nativesdk :7.21.7-r0
So the latest version here is a lie, this is not the latest version available. Furthermore it seems not to have affected the BBCLASSEXTENDs.
4. Now add PREFERRED_VERSION_curl = "7.21.7" to conf/layer.conf and run "bitbake -s | grep ^curl" again:
curl :7.20.0-r0 :7.21.7-r0
curl-native :7.21.7-r0
curl-nativesdk :7.21.7-r0
So it can clearly see the other recipe, it just doesn't acknowledge it until you force the matter.
This is all rather undesirable behaviour IMHO - even if the BBCLASSEXTEND and reported "latest version available" issues were corrected, I think the policy of "latest version wins unless DEFAULT_PREFERENCE or PREFERRED_VERSION says otherwise" should not be affected by layer priority.
Thoughts?
Cheers,
Paul
(PS: this is not unique to Poky, Andrea and I first reproduced this with Angstrom using bitbake master.)
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 19 lines · Paul Eggleton
Hi all,
Andrea pointed out a situation where he has seen the layer priority overriding version selection, and I've been able to confirm it.
Basically, if you have a recipe with a lower version in a layer with a higher priority it selects the lower version. What's more after digging further I found there were some rather anomalous interactions with the version logic and BBCLASSEXTEND. Here's an example using Poky:
1. Firstly, copy meta/recipes-support/curl to meta-yocto/recipes-support, then rename the version in meta-yocto so that its version is 7.20.0. At this point both meta/ and meta-yocto/ have the same layer priority of 5.
2. "bitbake -s | grep ^curl" reports:
curl :7.21.7-r0 curl-native :7.21.7-r0 curl-nativesdk :7.21.7-r0
Quoted 5 lines
3. Now increase the layer priority in
meta-yocto/conf/layer.confto 6, and run "bitbake -s | grep ^curl" again:curl :7.20.0-r0 curl-native :7.21.7-r0 curl-nativesdk :7.21.7-r0
This is clearly broken and needs to be consistent at the very least.
Quoted 12 lines
So the latest version here is a lie, this is not the latest version available. Furthermore it seems not to have affected the BBCLASSEXTENDs.
4. Now add
PREFERRED_VERSION_curl= "7.21.7" toconf/layer.confand run "bitbake -s | grep ^curl" again:
curl :7.20.0-r0 :7.21.7-r0
curl-native :7.21.7-r0
curl-nativesdk :7.21.7-r0So it can clearly see the other recipe, it just doesn't acknowledge it until you force the matter.
Quoted 6 lines
This is all rather undesirable behaviour IMHO - even if the BBCLASSEXTEND and reported "latest version available" issues were corrected, I think the policy of "latest version wins unless
DEFAULT_PREFERENCEorPREFERRED_VERSIONsays otherwise" should not be affected by layer priority.Thoughts?
We could do with clearly documenting this in the bitbake manual. I suspect users would expect the highest version to win and we probably should change the behaviour but I'm open to other opinions.
Cheers,
Richard
Quoted 3 lines · Richard Purdie
We could do with clearly documenting this in the bitbake manual. I suspect users would expect the highest version to win and we probably should change the behaviour but I'm open to other opinions.
The original intent with BBFILE_COLLECTIONS was that the recipe from the highest-priority collection would win even if it was an older version. (The rationale for this was that the one in the higher priority collection was the most likely to have local customisations, and you wouldn't want those to suddenly be negated if a newer version popped up in some upstream collection.)
I suspect the same logic probably does apply to layers to some extent for the same reason. If there is a situation where you are just trying to aggregate the recipes together from a bunch of layers and you just want the highest version from that set then my first reaction would be that the layers should probably all have the same priority.
Actually, to some extent I consider it a bit of a misfeature that the layer priority is specified by the layer rather than by the user in bblayers.conf, since this makes it harder to vary the stack-up order without local hackery to the layer files. For example, meta-oe currently sets itself to a higher priority than oe-core, but I've found that it generally seems to suit me better if meta-oe is actually the lower-priority layer.
p.
Quoted 7 lines · Phil Blundell
Actually, to some extent I consider it a bit of a misfeature that the layer priority is specified by the layer rather than by the user in bblayers.conf, since this makes it harder to vary the stack-up order without local hackery to the layer files. For example, meta-oe currently sets itself to a higher priority than oe-core, but I've found that it generally seems to suit me better if meta-oe is actually the lower-priority layer.
Agreed. It also means that the priority knowledge is split between layer.conf and bblayers.conf today, as config/class priority is determined by order of entries in BBLAYERS, whereas recipe priority is determined by layer.conf.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 3 lines · On Tuesday 02 August 2011 15:14:41 Chris Larson wrote:
as config/class priority is
determined by order of entries in BBLAYERS, whereas recipe priority is determined by layer.conf.
Is that a good thing though? At the moment I'm not convinced that it is.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 6 lines · Paul Eggleton
Quoted 3 lines · On Tuesday 02 August 2011 15:14:41 Chris Larson wrote:
as config/class priority is
determined by order of entries in BBLAYERS, whereas recipe priority is determined by layer.conf.Is that a good thing though? At the moment I'm not convinced that it is.
I probably could have worded my reply better. I was pointing out the fact that the order is determined in two places right now as being one reason why I agreed with his statement that ordering determined by the layer isn't ideal.
Signature · 5 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics
Quoted 8 lines · On Tuesday 02 August 2011 15:27:58 Chris Larson wrote:
On Tue, Aug 2, 2011 at 7:21 AM, Paul Eggleton
Quoted 5 lines · <[email protected]> wrote:
Quoted 3 lines · On Tuesday 02 August 2011 15:14:41 Chris Larson wrote:
as config/class priority is
determined by order of entries in BBLAYERS, whereas recipe priority is determined by layer.conf.
Quoted 6 lines
Quoted 1 line
Is that a good thing though? At the moment I'm not convinced that it is.
I probably could have worded my reply better. I was pointing out the fact that the order is determined in two places right now as being one reason why I agreed with his statement that ordering determined by the layer isn't ideal.
Well, all the same I'd like to have that question answered, maybe that's a different thread.
Back to Phil's original point though, I tend to think issues with meta-oe ought to be solved by fixing meta-oe - if some things in meta-oe are causing so much trouble that we have to down-grade the priority of the layer in order to squelch them, that's bad.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
At Wind River, in our non bitbake based build system, the way layers work is based on ordering.
The last layer listed has the highest priority. So whatever is specified in that layer (lower or higher versions) wins. All of the priority information is selected based on the equivalent of the order of entries in the BBLAYERS. However, unlike bitbake -- in the WR build system we can only have one version of a package available at any time.
(There is a second level of ordering for configuration files. For instance if you are looking for foo.conf, it will search the highest priority layer first and work it's way down. If foo.conf includes itself, foo.conf, it will automatically search starting with the next lower priority layer. If you include a differently named configuration file it starts back at the highest priority layer. This is what allows a layer to provide snippets of configuration files. -- I'm not sure this is relevant to this conversation though... but I thought I'd point out a characteristic of the ordering.)
--Mark
Quoted 13 lines · Chris Larson
Quoted 6 lines · Paul Eggleton
Quoted 3 lines · On Tuesday 02 August 2011 15:14:41 Chris Larson wrote:
as config/class priority is
determined by order of entries in BBLAYERS, whereas recipe priority is determined by layer.conf.Is that a good thing though? At the moment I'm not convinced that it is.
I probably could have worded my reply better. I was pointing out the fact that the order is determined in two places right now as being one reason why I agreed with his statement that ordering determined by the layer isn't ideal.
Quoted 9 lines · Chris Larson
Quoted 7 lines · Phil Blundell
Actually, to some extent I consider it a bit of a misfeature that the layer priority is specified by the layer rather than by the user in bblayers.conf, since this makes it harder to vary the stack-up order without local hackery to the layer files. For example, meta-oe currently sets itself to a higher priority than oe-core, but I've found that it generally seems to suit me better if meta-oe is actually the lower-priority layer.
Quoted 4 lines
Agreed. It also means that the priority knowledge is split between layer.conf and bblayers.conf today, as config/class priority is determined by order of entries in BBLAYERS, whereas recipe priority is determined by layer.conf.
In my opinion the layer priority for all kind of meta data should be consistent and selected using bblayers.conf
Quoted 11 lines
Signature · 10 lines
Christopher Larson clarson at kergoth dot com Founder - BitBake, OpenEmbedded, OpenZaurus Maintainer - Tslib Senior Software Engineer, Mentor Graphics _______________________________________________ bitbake-devel mailing list [email protected] http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel
Quoted 10 lines · On Tuesday 02 August 2011 16:35:12 Khem Raj wrote:
Quoted 9 lines · Chris Larson
Quoted 7 lines · Phil Blundell
Actually, to some extent I consider it a bit of a misfeature that the layer priority is specified by the layer rather than by the user in bblayers.conf, since this makes it harder to vary the stack-up order without local hackery to the layer files. For example, meta-oe currently sets itself to a higher priority than oe-core, but I've found that it generally seems to suit me better if meta-oe is actually the lower-priority layer.
Quoted 7 lines
Quoted 4 lines
Agreed. It also means that the priority knowledge is split between layer.conf and bblayers.conf today, as config/class priority is determined by order of entries in BBLAYERS, whereas recipe priority is determined by layer.conf.
In my opinion the layer priority for all kind of meta data should be consistent and selected using bblayers.conf
So I've thought about this more, and I don't think it is a good idea, for two reasons:
1) It increases the complexity by a huge amount. New users will have to understand how to set the priorities, and if someone reports a problem, we now not only need to know what layers they are using, we also need to know what mangling of the priorities they might have done. We'd just be introducing more potentially untested configurations (as if we don't have enough already.)
2) It just papers over real problems in meta-oe or any other layer you might choose to use on top. As I said, if meta-oe has stuff in it that is not working, it should be fixed or removed. Let's fix problems instead of finding ways of ignoring them.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 4 lines · Paul Eggleton
Quoted 3 lines
In my opinion the layer priority for all kind of meta data should be consistent and selected using bblayers.conf
Quoted 8 lines
So I've thought about this more, and I don't think it is a good idea, for two reasons:
1) It increases the complexity by a huge amount. New users will have to understand how to set the priorities, and if someone reports a problem, we now not only need to know what layers they are using, we also need to know what mangling of the priorities they might have done. We'd just be introducing more potentially untested configurations (as if we don't have enough already.)
with layers we dont control policies of all the layers that may be used on top so fixing meta-oe does not solve problem completely and we can not ask for exclusive
recipes. People would want to override the recipes from other layers.
bitbake could report the layer the recipe comes from which can make it evident or may be special command to inform the layer priorities. This will guide the users to diagnose the problems quickly and help developers to identify the issues
faster. There could be a complete bill of recipes printed for a given target as well
so if someone wants to check all the recipes that were built.
Quoted 6 lines
2) It just papers over real problems in meta-oe or any other layer you might choose to use on top. As I said, if meta-oe has stuff in it that is not working, it should be fixed or removed. Let's fix problems instead of finding ways of ignoring them.
In any case I agree that problems should be fixed. However this does not scale to all
layers and we can not police all the layers and we should not. We should try to make it possible
for people to glue layers together without issues.
Quoted 4 lines · On Thursday 25 August 2011 16:56:28 you wrote:
with layers we dont control policies of all the layers that may be used on top so fixing meta-oe does not solve problem completely and we can not ask for exclusive recipes. People would want to override the recipes from other layers.
Generally, the less duplication we have across layers the better; if a maintainer is duplicating a recipe they should have a very good reason to do so. We can avoid some of this duplication by making it easier to figure out where the right place for recipes is and doing everything we can encourage people to get stuff merged there. I'm not convinced we are doing very well on either count yet, but I expect this will improve over time.
Quoted 6 lines
bitbake could report the layer the recipe comes from which can make it evident or may be special command to inform the layer priorities. This will guide the users to diagnose the problems quickly and help developers to identify the issues faster. There could be a complete bill of recipes printed for a given target as well so if someone wants to check all the recipes that were built.
I had not considered BitBake itself reporting these, that might be useful especially in the case of errors (although the full path to the recipe is already reported during the build, so you should be able to tell in most cases from that if you need to). Outputting a specific report of parsed recipes and bbappends as part of the build also might be useful.
FYI, bitbake-layers exists as a separate utility to answer a lot of these kinds of questions, and recently has become a lot more powerful - I would strongly recommend everyone check it out and if it's not able to report what you want then please tell me and I'll try to make it do so. I already have a few ideas for future improvements there which I will hopefully get time to look into in the Yocto 1.2 cycle.
Quoted 4 lines
In any case I agree that problems should be fixed. However this does not scale to all layers and we can not police all the layers and we should not. We should try to make it possible for people to glue layers together without issues.
How does merely being able to alter the priorities help here though? I could begin to understand if you were asking for some way to blacklist individual recipes in individual layers, but moving the priority of an entire layer is likely to have much more of an effect than just obscuring a few errant recipes that you don't want.
We can't "police" all the layers, no. But if those who maintain that layer can't respond to and resolve problems then they shouldn't be maintainers or those layers can't be considered well-maintained. When you add a layer to your build you are placing some trust in the maintainer of that layer; it is up to the maintainer to make sure they don't abuse that trust.
Cheers,
Paul
Signature · 2 lines
Paul Eggleton Intel Open Source Technology Centre
Quoted 38 lines · Paul Eggleton
Quoted 4 lines · On Thursday 25 August 2011 16:56:28 you wrote:
with layers we dont control policies of all the layers that may be used on top so fixing meta-oe does not solve problem completely and we can not ask for exclusive recipes. People would want to override the recipes from other layers.
Generally, the less duplication we have across layers the better; if a maintainer is duplicating a recipe they should have a very good reason to do so. We can avoid some of this duplication by making it easier to figure out where the right place for recipes is and doing everything we can encourage people to get stuff merged there. I'm not convinced we are doing very well on either count yet, but I expect this will improve over time.
Quoted 6 lines
bitbake could report the layer the recipe comes from which can make it evident or may be special command to inform the layer priorities. This will guide the users to diagnose the problems quickly and help developers to identify the issues faster. There could be a complete bill of recipes printed for a given target as well so if someone wants to check all the recipes that were built.
I had not considered BitBake itself reporting these, that might be useful especially in the case of errors (although the full path to the recipe is already reported during the build, so you should be able to tell in most cases from that if you need to). Outputting a specific report of parsed recipes and bbappends as part of the build also might be useful.
FYI, bitbake-layers exists as a separate utility to answer a lot of these kinds of questions, and recently has become a lot more powerful - I would strongly recommend everyone check it out and if it's not able to report what you want then please tell me and I'll try to make it do so. I already have a few ideas for future improvements there which I will hopefully get time to look into in the Yocto 1.2 cycle.
Quoted 4 lines
In any case I agree that problems should be fixed. However this does not scale to all layers and we can not police all the layers and we should not. We should try to make it possible for people to glue layers together without issues.
Quoted 5 lines
How does merely being able to alter the priorities help here though? I could begin to understand if you were asking for some way to blacklist individual recipes in individual layers, but moving the priority of an entire layer is likely to have much more of an effect than just obscuring a few errant recipes that you don't want.
Well in this I agree with khem that it would be more consistent to use priority from bblayers.conf. You're right that changing priority of an entire layer is having too big impact, but whoever is writing his bblayers.conf has better information about what he needs/wants from particular layer then layer maintainer who has no clue in which layer stack his layer will end up.
Quoted 5 lines
We can't "police" all the layers, no. But if those who maintain that layer can't respond to and resolve problems then they shouldn't be maintainers or those layers can't be considered well-maintained. When you add a layer to your build you are placing some trust in the maintainer of that layer; it is up to the maintainer to make sure they don't abuse that trust.
And with priority in bblayers.conf I can say how much I trust him.
my 2c
Regards,
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]