Idea for bbappend / layer organization and naming convention
Yocto community,
Triggered by the previous email request to yocto@ about best practices for layer organization I'm finally firing off this email for (hopefully) some feedback on an idea we first kicked around, discussed for rough consensus and then decided to implement in the Yocto based GDP project [1]. You can see it as a trial, anything can be changed, but so far so good...
We adopted a somewhat novel (but actually not really unique, see inside) naming convention [2] for all modifications to components that belong to other layers.
This convention shows what layers are being modified by the .bbappends by actually naming the layer it is (intending to**) modify. The initiative also aims to highlight and separate .bbappends (modifications) from uniquely added components in our project (new .bb files) in GDP.
**Note that we are well aware this does NOT control bitbake behavior (as neither does the recipe-xxx/ directory convention), and is therefore just a guide. If the project is misconfigured, bitbake could do something different than the naming convention implies and it could even be misleading, but at least the *programmer intent* is shown clearly.
I think it also makes .bbappends more explicitly visible and might trigger
the idea: Do we really need this? Are we really modifying *that* adopted
layer, and if so, why?
To Russel who wrote the previous request for guidance - feel free to consider, but I won't recommend this directly since it's by no means an agreed best practice in all of the community. I'm rather asking for other experienced OE developers to consider and comment on the idea.
For oldtimers it's probably easy to find this odd and just be negative against it since well, "standard practice", inertia, NIH and all that, but I'll stick my neck out anyway because it seems to solve some issues of organization and understanding at least for us (and I'm a sucker for some flamebait ;)
If the intro is too formal and long, just look at the final examples. I think you'll get it. Feel free to ask.
- Gunnar
--
Gunnar Andersson <[email protected]>
Development Lead
GENIVI Alliance
[1] https://github.com/GENIVI/genivi-dev-platform
[2] Naming strategy for bitbake extension layers:
https://at.projects.genivi.org/wiki/x/w4Xk
Quoted 9 lines · Gunnar Andersson
Yocto community,
Triggered by the previous email request to yocto@ about best practices for layer organization I'm finally firing off this email for (hopefully) some feedback on an idea we first kicked around, discussed for rough consensus and then decided to implement in the Yocto based GDP project [1]. You can see it as a trial, anything can be changed, but so far so good...
Quoted 4 lines
We adopted a somewhat novel (but actually not really unique, see inside) naming convention [2] for all modifications to components that belong to other layers.
I've been using a similar naming/sorting mechanism in layers that I maintain for several years now. When you have a single layer that is carrying bbappends to many other layers, I find that it really helps keep everything separated and aids finding the original recipe.
(that being said, recent work with layer index, etc, make it easier to locate the original recipe and any bbappends .. but that doesn't preclude keeping things organized in a layer).
Cheers,
Bruce
Quoted 43 lines
This convention shows what layers are being modified by the .bbappends by actually naming the layer it is (intending to**) modify. The initiative also aims to highlight and separate .bbappends (modifications) from uniquely added components in our project (new .bb files) in GDP.
**Note that we are well aware this does NOT control bitbake behavior (as neither does the recipe-xxx/ directory convention), and is therefore just a guide. If the project is misconfigured, bitbake could do something different than the naming convention implies and it could even be misleading, but at least the *programmer intent* is shown clearly.
I think it also makes .bbappends more explicitly visible and might trigger
the idea: Do we really need this? Are we really modifying *that* adopted
layer, and if so, why?To Russel who wrote the previous request for guidance - feel free to consider, but I won't recommend this directly since it's by no means an agreed best practice in all of the community. I'm rather asking for other experienced OE developers to consider and comment on the idea.
For oldtimers it's probably easy to find this odd and just be negative against it since well, "standard practice", inertia, NIH and all that, but I'll stick my neck out anyway because it seems to solve some issues of organization and understanding at least for us (and I'm a sucker for some flamebait ;)
If the intro is too formal and long, just look at the final examples. I think you'll get it. Feel free to ask.
- Gunnar
Signature · 7 lines
Gunnar Andersson <[email protected]> Development Lead GENIVI Alliance [1] https://github.com/GENIVI/genivi-dev-platform [2] Naming strategy for bitbake extension layers: https://at.projects.genivi.org/wiki/x/w4Xk
Quoted 11 lines · Bruce Ashfield
Quoted 10 lines · Gunnar Andersson
Yocto community,
Triggered by the previous email request to yocto@ about best practices for layer organization I'm finally firing off this email for (hopefully) some feedback on an idea we first kicked around, discussed for rough consensus and then decided to implement in the Yocto based GDP project [1]. You can see it as a trial, anything can be changed, but so far so good...
Quoted 7 lines
Quoted 4 lines
We adopted a somewhat novel (but actually not really unique, see inside) naming convention [2] for all modifications to components that belong to other layers.
I've been using a similar naming/sorting mechanism in layers that I maintain for several years now.
Great. Is it similar, or /exactly/ like this? I'm only asking because I think it would be useful to have some written down rules and if there are additional tweaks that will improve this proposal, I'd like to hear them.
I suspected it might actually be somewhat of a common practice.
Quoted 3 lines
When you have a single layer that is carrying bbappends to many other layers, I find that it really helps keep everything separated and aids finding the original recipe.
Yes, that is the idea.
Quoted 4 lines
(that being said, recent work with layer index, etc, make it easier to locate the original recipe and any bbappends .. but that doesn't preclude keeping things organized in a layer).
... but I didn't get any more feedback than yours, so I'll just leave it where it is for now. Maybe this is not something the other Yocto community cares about, or similar practices are actually done in practice, but not written down. Or maybe everyone is OK with the state of mixing .bb and .bbappend files within the layers...
So far I think the initial experience on our side is positive. It's something that needs to be looked after a bit (we have a few mistakes to fix). But since some directories will only have .bb files, and others only .bbappend files, it's therefore easy to script a warning system for anything that is out of place.
Best Regards
- Gunnar
Quoted 4 lines
Cheers,
Bruce
[trimmed]
Quoted 8 lines
Quoted 8 lines
--
Gunnar Andersson <[email protected]>
Development Lead
GENIVI Alliance[1] https://github.com/GENIVI/genivi-dev-platform
[2] Naming strategy for bitbake extension layers:
https://at.projects.genivi.org/wiki/x/w4Xk
Quoted 18 lines · Gunnar Andersson
Quoted 16 lines · Bruce Ashfield
Quoted 14 lines · Gunnar Andersson
Yocto community,
Triggered by the previous email request to yocto@ about best practices for layer organization I'm finally firing off this email for (hopefully) some feedback on an idea we first kicked around, discussed for rough consensus and then decided to implement in the Yocto based GDP project [1]. You can see it as a trial, anything can be changed, but so far so good...
We adopted a somewhat novel (but actually not really unique, see inside) naming convention [2] for all modifications to components that belong to other layers.
Quoted 7 lines
Quoted 2 lines
I've been using a similar naming/sorting mechanism in layers that I maintain for several years now.
Great. Is it similar, or /exactly/ like this? I'm only asking because I think it would be useful to have some written down rules and if there are additional tweaks that will improve this proposal, I'd like to hear them.
Exactly. The layer name and recipe-* structure is nested in a layer that is carrying bbappends. That way there's zero confusion about where the main recipe can be found.
Bruce
Quoted 44 lines
I suspected it might actually be somewhat of a common practice.
Quoted 3 lines
When you have a single layer that is carrying bbappends to many other layers, I find that it really helps keep everything separated and aids finding the original recipe.
Yes, that is the idea.
Quoted 4 lines
(that being said, recent work with layer index, etc, make it easier to locate the original recipe and any bbappends .. but that doesn't preclude keeping things organized in a layer).
... but I didn't get any more feedback than yours, so I'll just leave it where it is for now. Maybe this is not something the other Yocto community cares about, or similar practices are actually done in practice, but not written down. Or maybe everyone is OK with the state of mixing .bb and .bbappend files within the layers...
So far I think the initial experience on our side is positive. It's something that needs to be looked after a bit (we have a few mistakes to fix). But since some directories will only have .bb files, and others only .bbappend files, it's therefore easy to script a warning system for anything that is out of place.
Best Regards
- GunnarQuoted 4 lines
Cheers,
Bruce
[trimmed]
Quoted 8 lines
Quoted 8 lines
Signature · 7 lines
Gunnar Andersson <[email protected]> Development Lead GENIVI Alliance [1] https://github.com/GENIVI/genivi-dev-platform [2] Naming strategy for bitbake extension layers: https://at.projects.genivi.org/wiki/x/w4Xk
Quoted 19 lines · Bruce Ashfield
Quoted 18 lines · Gunnar Andersson
Quoted 16 lines · Bruce Ashfield
Quoted 14 lines · Gunnar Andersson
Yocto community,
Triggered by the previous email request to yocto@ about best practices for layer organization I'm finally firing off this email for (hopefully) some feedback on an idea we first kicked around, discussed for rough consensus and then decided to implement in the Yocto based GDP project [1]. You can see it as a trial, anything can be changed, but so far so good...
We adopted a somewhat novel (but actually not really unique, see inside) naming convention [2] for all modifications to components that belong to other layers.
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
I've been using a similar naming/sorting mechanism in layers that I maintain for several years now.
Great. Is it similar, or /exactly/ like this? I'm only asking because I think it would be useful to have some written down rules and if there are additional tweaks that will improve this proposal, I'd like to hear them.
I suspect suggestions for best practices along with examples are what people would like to see. This may be reasonable to do with both the Yocto Project and OpenEmbedded communities.
Obviously there are currently a large number of existing layers. So showing how this works, and examples of how it makes it easier to maintain and manage the work over the longer term is a really good idea.
(Keep in mind, many of the current layers were originally designed/updated before there was a large ecosystem of layers. So a lot of poor examples, of this kind of maintenance, may exist. It is a good idea to steer people away from using these as examples, and even a way to potentially help the maintainers of those layers evolve into a better and more efficient format.)
--Mark
Quoted 51 lines
Exactly. The layer name and recipe-* structure is nested in a layer that is carrying bbappends. That way there's zero confusion about where the main recipe can be found.
Bruce
Quoted 44 lines
I suspected it might actually be somewhat of a common practice.
Quoted 3 lines
When you have a single layer that is carrying bbappends to many other layers, I find that it really helps keep everything separated and aids finding the original recipe.
Yes, that is the idea.
Quoted 4 lines
(that being said, recent work with layer index, etc, make it easier to locate the original recipe and any bbappends .. but that doesn't preclude keeping things organized in a layer).
... but I didn't get any more feedback than yours, so I'll just leave it where it is for now. Maybe this is not something the other Yocto community cares about, or similar practices are actually done in practice, but not written down. Or maybe everyone is OK with the state of mixing .bb and .bbappend files within the layers...
So far I think the initial experience on our side is positive. It's something that needs to be looked after a bit (we have a few mistakes to fix). But since some directories will only have .bb files, and others only .bbappend files, it's therefore easy to script a warning system for anything that is out of place.
Best Regards
- GunnarQuoted 4 lines
Cheers,
Bruce
[trimmed]
Quoted 8 lines
Quoted 8 lines
Signature · 7 lines
Gunnar Andersson <[email protected]> Development Lead GENIVI Alliance [1] https://github.com/GENIVI/genivi-dev-platform [2] Naming strategy for bitbake extension layers: https://at.projects.genivi.org/wiki/x/w4Xk
Hi all,
I've followed this thread with interest. I'll try to bring here some ideas we had in parallel on Automotive Grade Linux (AGL) side.
What we plan to do with backports on AGL is still WIP (see
https://jira.automotivelinux.org/browse/SPEC-802).
It's still cooking, but we're going to adopt a solution close to the one proposed by Gunnar in GENIVI: I mean: we agree on the same goal, but what we end up with differs slightly.
The scheme we're thinking about would be typically something like this:
meta-agl/meta-agl-backports/<meta-origin>/<recipes-origin>/<recipe>/<feature|reason>/<recipe>.bbappend
Where:
* the first meta-agl is our main git repo (not a meta in fact): the real meta is meta-agl/meta-agl :)
* meta-agl/meta-agl-backports is a layer
* <meta-origin> is the name of the layer on which the backport is applied
* <recipes-origin> is the origin folder name for recipes in the origin layer
* <recipe> is the name of the recipe which is backported/amended
* <feature|reason> is something that defines why we have a backport here (bug id, quick desc...)
And finally, in the folder where we put the bbappend, we want to add a README.backport in YAML, similar to this one:
—
Origin: <name>
Origin-Url: <source repo url>
Origin-Ref: <rev/tag>
Destination: <AGL version name: dab, eel,...>
Expiration: <when> (if known)
Bug-AGL: <reference to bug ID>
Upstream-Status: <the same as OE's Upstream-Status tag>
Reason: <comment
on
multiple
lines>
—
With that model, we expect to smooth the workflow and make the alignment on Yocto/OE/Poky versions easier.
HTH
Best regards,
---
Stephane Desneux - CTO - IoT.bzh
[email protected] - www.iot.bzh
Quoted 96 lines · Mark Hatle
Quoted 19 lines · Bruce Ashfield
Quoted 18 lines · Gunnar Andersson
Quoted 16 lines · Bruce Ashfield
Quoted 14 lines · Gunnar Andersson
Yocto community,
Triggered by the previous email request to yocto@ about best practices for layer organization I'm finally firing off this email for (hopefully) some feedback on an idea we first kicked around, discussed for rough consensus and then decided to implement in the Yocto based GDP project [1]. You can see it as a trial, anything can be changed, but so far so good...
We adopted a somewhat novel (but actually not really unique, see inside) naming convention [2] for all modifications to components that belong to other layers.
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
I've been using a similar naming/sorting mechanism in layers that I maintain for several years now.
Great. Is it similar, or /exactly/ like this? I'm only asking because I think it would be useful to have some written down rules and if there are additional tweaks that will improve this proposal, I'd like to hear them.
I suspect suggestions for best practices along with examples are what people would like to see. This may be reasonable to do with both the Yocto Project and OpenEmbedded communities.
Obviously there are currently a large number of existing layers. So showing how this works, and examples of how it makes it easier to maintain and manage the work over the longer term is a really good idea.
(Keep in mind, many of the current layers were originally designed/updated before there was a large ecosystem of layers. So a lot of poor examples, of this kind of maintenance, may exist. It is a good idea to steer people away from using these as examples, and even a way to potentially help the maintainers of those layers evolve into a better and more efficient format.)
--Mark
Quoted 51 lines
Exactly. The layer name and recipe-* structure is nested in a layer that is carrying bbappends. That way there's zero confusion about where the main recipe can be found.
Bruce
Quoted 44 lines
I suspected it might actually be somewhat of a common practice.
Quoted 3 lines
When you have a single layer that is carrying bbappends to many other layers, I find that it really helps keep everything separated and aids finding the original recipe.
Yes, that is the idea.
Quoted 4 lines
(that being said, recent work with layer index, etc, make it easier to locate the original recipe and any bbappends .. but that doesn't preclude keeping things organized in a layer).
... but I didn't get any more feedback than yours, so I'll just leave it where it is for now. Maybe this is not something the other Yocto community cares about, or similar practices are actually done in practice, but not written down. Or maybe everyone is OK with the state of mixing .bb and .bbappend files within the layers...
So far I think the initial experience on our side is positive. It's something that needs to be looked after a bit (we have a few mistakes to fix). But since some directories will only have .bb files, and others only .bbappend files, it's therefore easy to script a warning system for anything that is out of place.
Best Regards
- GunnarQuoted 4 lines
Cheers,
Bruce
[trimmed]
Quoted 8 lines
Quoted 8 lines
Signature · 7 lines
Gunnar Andersson <[email protected]> Development Lead GENIVI Alliance [1] https://github.com/GENIVI/genivi-dev-platform [2] Naming strategy for bitbake extension layers: https://at.projects.genivi.org/wiki/x/w4Xk