RFC: Layer tooling brainstorming

Layer Tooling for OpenEmbedded
==============================

As we move forward with encouraging the use of metadata layers, a number of issues are appearing which imply the need for either infrastructure improvements to the layer construction, or better tools to handle the layers. The OS TSC discussed layers and layer tooling at the face to face TSC meeting in San Francisco. Some example problems that were identified that currently need addressing in some form are:

* There is no dependency information between layers (layer X depends on layer Y and Z)
* There is no version information present (layer X needs version A of layer Y)
* There is no layer "ABI" version number present in layers
* Layer specific sanity checks? (other types of sanity checks on Yocto 1.1 roadmap)
* Ability to fetch layers from other repositories
- Multi SCM?
- Use bitbake fetcher?
* Ability to generate repositories representing composited layers (Poky, RP has hacky scripts)
* Tool to seperate patch into one for different composited parts

* Ability to include partial components of repositories (e.g. only beagleboard from meta-ti)
* Tool to merge (flatten) several layers into one
* Able to visualise dependencies between layers
* When bitbake prints banner of branch/commit, need to cover all layer components present
* bitbake -e on steroids to highlight differences due to layer changes? (long term goal)

Quoted 23 lines · Richard Purdie

Layer Tooling for OpenEmbedded
==============================

As we move forward with encouraging the use of metadata layers, a number of issues are appearing which imply the need for either infrastructure improvements to the layer construction, or better tools to handle the layers. The OS TSC discussed layers and layer tooling at the face to face TSC meeting in San Francisco. Some example problems that were identified that currently need addressing in some form are:

* There is no dependency information between layers (layer X depends on layer Y and Z)
* There is no version information present (layer X needs version A of layer Y)
* There is no layer "ABI" version number present in layers
* Layer specific sanity checks? (other types of sanity checks on Yocto 1.1 roadmap)
* Ability to fetch layers from other repositories
- Multi SCM?
- Use bitbake fetcher?
* Ability to generate repositories representing composited layers (Poky, RP has hacky scripts)
* Tool to seperate patch into one for different composited parts

Quoted 8 lines

* Ability to include partial components of repositories (e.g. only beagleboard from meta-ti)
* Tool to merge (flatten) several layers into one
* Able to visualise dependencies between layers
* When bitbake prints banner of branch/commit, need to cover all layer components present
* bitbake -e on steroids to highlight differences due to layer changes? (long term goal)

maybe I've missed something, but
* Parsing error when ie foo_1.0.bbappend is found, but foo_1.0.bb in upper layer was upgraded to foo_1.1.bb

This can cause big problems when distribution explicitly enables something in .bbappend with EXTRA_OECONF, and someone updates all layers, builds foo_1.1.bb and installs it on targets before noticing that foo_1.0.bbappend wasn't used. Then he has to bump PR after adding foo_1.1.bbappend to distro layer (to fix foo on targets), so I would prefer parsing error in this case (won't help if upper layer keeps both versions foo_1.0.bb and foo_1.1.bb ;/).

This should be combined with your 2nd point and person who changes versioned dependency on upper layer should check that all needed .bbappends still apply.

Quoted 14 lines · Richard Purdie

maybe I've missed something, but
* Parsing error when ie foo_1.0.bbappend is found, but foo_1.0.bb in upper layer was upgraded to foo_1.1.bb

This can cause big problems when distribution explicitly enables something in .bbappend with EXTRA_OECONF, and someone updates all layers, builds foo_1.1.bb and installs it on targets before noticing that foo_1.0.bbappend wasn't used. Then he has to bump PR after adding foo_1.1.bbappend to distro layer (to fix foo on targets), so I would prefer parsing error in this case (won't help if upper layer keeps both versions foo_1.0.bb and foo_1.1.bb ;/).

Quoted 3 lines

This should be combined with your 2nd point and person who changes versioned dependency on upper layer should check that all needed .bbappends still apply.

Agreed but I thought we already had a tool to handle at least some of this though? Certainly integration of that with the above is essential though.

Op 18 apr 2011, om 09:09 heeft Richard Purdie het volgende geschreven:

Quoted 15 lines · Martin Jansa

Quoted 14 lines · Richard Purdie

maybe I've missed something, but
* Parsing error when ie foo_1.0.bbappend is found, but foo_1.0.bb in upper layer was upgraded to foo_1.1.bb

This can cause big problems when distribution explicitly enables something in .bbappend with EXTRA_OECONF, and someone updates all layers, builds foo_1.1.bb and installs it on targets before noticing that foo_1.0.bbappend wasn't used. Then he has to bump PR after adding foo_1.1.bbappend to distro layer (to fix foo on targets), so I would prefer parsing error in this case (won't help if upper layer keeps both versions foo_1.0.bb and foo_1.1.bb ;/).

Quoted 7 lines

Quoted 3 lines

This should be combined with your 2nd point and person who changes versioned dependency on upper layer should check that all needed .bbappends still apply.

Agreed but I thought we already had a tool to handle at least some of this though? Certainly integration of that with the above is essential though.

This is what I get for my setup:

koen@dominion:/OE/tentacle/build$ bitbake-layers 
<string>:1: DeprecationWarning: Call to deprecated function bb.parse.parse_py.BBHandler.vars_from_file: Please use bb.parse.vars_from_file instead.
<string>:6: DeprecationWarning: Call to deprecated function bb.which: Please use bb.utils.which instead.
WARNING: Bitbake has not been run using the bitbake wrapper (scripts/bitbake); this is likely because your PATH has been altered from that normally set up by the poky-init-build-env script. Not using the wrapper may result in failures during package installation, so it is highly recommended that you set your PATH back so that the wrapper script is being executed.
WARNING: Pseudo is not functioning correctly, which will cause failures during package installation. Please check your configuration.
Parsing recipes..<string>:1: DeprecationWarning: Call to deprecated function bb.which: Please use bb.utils.which instead.
<string>:1: DeprecationWarning: Call to deprecated function bb.which: Please use bb.utils.which instead.
<string>:1: DeprecationWarning: Call to deprecated function bb.which: Please use bb.utils.which instead.
done.

INFO: State of append files:
INFO: base-files_3.0.14.bb:
INFO:   /OE/tentacle/sources/meta-angstrom/recipes-core/base-files/base-files_3.0.14.bbappend
INFO: libusb-compat_0.1.3.bb:
INFO:   /OE/tentacle/sources/meta-openembedded/meta-oe/recipes-support/libusb/libusb-compat_0.1.3.bbappend
INFO: sysvinit_2.88dsf.bb:
INFO:   /OE/tentacle/sources/meta-angstrom/recipes-core/sysvinit/sysvinit_2.88dsf.bbappend
koen@dominion:/OE/tentacle/build$ 

Layer Tooling for OpenEmbedded
==============================

As we move forward with encouraging the use of metadata layers, a number of issues are appearing which imply the need for either infrastructure improvements to the layer construction, or better tools to handle the layers. The OS TSC discussed layers and layer tooling at the face to face TSC meeting in San Francisco. Some example problems that were identified that currently need addressing in some form are:

* There is no dependency information between layers (layer X depends on layer Y and Z)
* There is no version information present (layer X needs version A of layer Y)
* There is no layer "ABI" version number present in layers
* Layer specific sanity checks? (other types of sanity checks on Yocto 1.1 roadmap)
* Ability to fetch layers from other repositories
- Multi SCM?
- Use bitbake fetcher?
* Ability to generate repositories representing composited layers (Poky, RP has hacky scripts)
* Tool to seperate patch into one for different composited parts
* Ability to include partial components of repositories (e.g. only beagleboard from meta-ti)
* Tool to merge (flatten) several layers into one
* Able to visualise dependencies between layers
* When bitbake prints banner of branch/commit, need to cover all layer components present
* bitbake -e on steroids to highlight differences due to layer changes? (long term goal)

Implementation thoughts:

* This layer tooling probably belongs at a higher level on the stack than bitbake itself
* Maybe need to split into "bootstrap" steps (e.g where pseduo is established, layers downloaded etc)
* Need some kind of config file defining layer's properties or at least enhance layer.conf significantly
* git submodules and repo should be looked at but due to specific needs and ability to control tool evolution, likely need own tool.

This hopefully summarises the ideas we discussed in person, we therefore need to start looking at how to move forward from here with this but I wanted to document this for starters!

Cheers,

Richard

Quoted 47 lines · On Monday 18 April 2011 08:17:01 Richard Purdie wrote:

Layer Tooling for OpenEmbedded
==============================

As we move forward with encouraging the use of metadata layers, a number of issues are appearing which imply the need for either infrastructure improvements to the layer construction, or better tools to handle the layers. The OS TSC discussed layers and layer tooling at the face to face TSC meeting in San Francisco. Some example problems that were identified that currently need addressing in some form are:

* There is no dependency information between layers (layer X depends on layer Y and Z)
* There is no version information present (layer X needs version A of layer Y)
* There is no layer "ABI" version number present in layers
* Layer specific sanity checks? (other types of sanity checks on Yocto 1.1 roadmap)
* Ability to fetch layers from other repositories
- Multi SCM?
- Use bitbake fetcher?
* Ability to generate repositories representing composited layers (Poky, RP has hacky scripts)
* Tool to seperate patch into one for different composited parts
* Ability to include partial components of repositories (e.g. only beagleboard from meta-ti)
* Tool to merge (flatten) several layers into one
* Able to visualise dependencies between layers
* When bitbake prints banner of branch/commit, need to cover all layer components present
* bitbake -e on steroids to highlight differences due to layer changes? (long term goal)

Implementation thoughts:

* This layer tooling probably belongs at a higher level on the stack than bitbake itself
* Maybe need to split into "bootstrap" steps (e.g where pseduo is established, layers downloaded etc)
* Need some kind of config file defining layer's properties or at least enhance layer.conf significantly
* git submodules and repo should be looked at but due to specific needs and ability to control tool evolution, likely need own tool.

This hopefully summarises the ideas we discussed in person, we therefore need to start looking at how to move forward from here with this but I wanted to document this for starters!

Quoted 3 lines

Cheers,

Richard

Afraid being killed I didn't dare to ask - but it's getting time to do so: What exactly are layers in OE and what are their advantages?

* Show easily which recipes are being used and which are overridden. We have bitbake-layers, but this clearly needs some extension. I think long term we would want to be able to analyse overrides accross a set of layers to figure out where common stuff in non-core layers needs pushing up to oe-core.

* Allow layer maintainers to easily pull a version of a recipe into their own layer if it's about to be removed from oe-core - with some complicated recipes that could be painful/annoying to pick out all the necessary files by hand.

* Recipe maintenance tools that take advantage of bitbake's parsing logic to aid blanket updates (e.g. variable renames). This should help mitigate the increased difficulty of having recipes spread out over multiple repositories when doing these kinds of updates.

* This layer tooling probably belongs at a higher level on the stack than bitbake itself

* Maybe need to split into "bootstrap" steps (e.g where pseduo is established, layers downloaded etc)

* Maybe need to split into "bootstrap" steps (e.g where pseduo is established, layers downloaded etc)