dynamic layer dependencies in meta-security

Hello Armin!

I'm trying to use the latest meta-security master in refkit. Refkit runs yocto-compat-layer.py against its own layers, which then pulls in meta-security as one of the dependencies. That's where I am hitting a snag:

INFO: Detected layers:
INFO: meta-refkit: LayerType.DISTRO, /fast/work/intel-iot-refkit-pr/meta-refkit
INFO:
INFO: Setting up for meta-refkit(LayerType.DISTRO), /fast/work/intel-iot-refkit-pr/meta-refkit
INFO: Adding layer dependency meta-selftest
INFO: Adding layer dependency meta-intel
INFO: Adding layer dependency meta-refkit-gateway
INFO: Adding layer dependency meta-refkit-core
INFO: Adding layer dependency meta-oic
INFO: Adding layer dependency meta-iot-web
INFO: Adding layer dependency meta-refkit-computervision
INFO: Adding layer dependency meta-intel-realsense
INFO: Adding layer dependency meta-clang
INFO: Adding layer dependency meta-oe
INFO: Adding layer dependency meta-refkit-industrial
INFO: Adding layer dependency meta-python
INFO: Adding layer dependency meta-ros
INFO: Adding layer dependency meta-iotqa
INFO: Adding layer dependency meta-security
INFO: Adding layer dependency meta-perl
INFO: Adding layer dependency meta-security-isafw
INFO: Adding layer dependency meta-filesystems
INFO: Adding layer dependency meta-gnome
INFO: Adding layer dependency meta-networking
INFO: Adding layer dependency meta-flatpak
Traceback (most recent call last):
File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/yocto-compat-layer.py", line 203, in <module>
ret = main()
File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/yocto-compat-layer.py", line 151, in main

    if not add_layer_dependencies(bblayersconf, layer, dep_layers, logger) or \

File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/lib/compatlayer/__init__.py", line 195, in add_layer_dependencies
output = check_command('Getting existing layers failed.', 'bitbake-layers show-layers').decode('utf-8')
File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/lib/compatlayer/__init__.py", line 228, in check_command
raise RuntimeError(msg)
RuntimeError: Getting existing layers failed.
Command: bitbake-layers show-layers
Output:
ERROR: Layer 'security' depends on layer 'xfce-layer', but this layer is not enabled in your configuration

The reason is the way how meta-security declares its dependencies:

LAYERDEPENDS_security = "core openembedded-layer perl-layer"
LAYERDEPENDS_security += "${@bb.utils.contains("DISTRO_FEATURES", "x11", "gnome-layer xfce-layer", "", d)}" LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "tpm", "tpm-layer", "",d)}" LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "tpm2", "tpm-layer", "",d)}" LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "vtpm", "tpm-layer meta-filesystems", "",d)}"

I haven't checked how exactly yocto-compat-layer.py deals with this, but the end result is that it doesn't add meta-xfce when adding meta-security. Then when "bitbake-layers show-layers" is called, a distro (poky) has been chosen where x11 is in DISTRO_FEATURES, causing the dependency check to fail.

This highlights a conceptual issue with the dynamic LAYERDEPENDS_security above: which distro or machine config is active? Tools like yocto-compat-layer.py or the layer index have to do their work before that is defined.

This is also problematic with multiconfig. The same bblayers.conf might be used with one config where x11 is off and another where it is on.

I suspect LAYERDEPENDS simply can't be dynamic. It either has to include everything that a layer might depend on (even when the additional content doesn't end up being used), or it just depends on the essential parts and then enables additional recipes only when the optional layers they depend on are present. BBFILES_DYNAMIC can be used for that, or the recipes themselves check BBCOLLECTIONS in anonymous python code and raise a SkipRecipe exception when their pre-condition isn't satisfied.

Copying Paul (layer index) and Mark (oe-yocto-compat.py, setup tool) for their input.

Quoted 37 lines · Patrick Ohly

Hello Armin!

I'm trying to use the latest meta-security master in refkit. Refkit runs yocto-compat-layer.py against its own layers, which then pulls in meta-security as one of the dependencies. That's where I am hitting a snag:

INFO: Detected layers:
INFO: meta-refkit: LayerType.DISTRO, /fast/work/intel-iot-refkit-pr/meta-refkit
INFO:
INFO: Setting up for meta-refkit(LayerType.DISTRO), /fast/work/intel-iot-refkit-pr/meta-refkit
INFO: Adding layer dependency meta-selftest
INFO: Adding layer dependency meta-intel
INFO: Adding layer dependency meta-refkit-gateway
INFO: Adding layer dependency meta-refkit-core
INFO: Adding layer dependency meta-oic
INFO: Adding layer dependency meta-iot-web
INFO: Adding layer dependency meta-refkit-computervision
INFO: Adding layer dependency meta-intel-realsense
INFO: Adding layer dependency meta-clang
INFO: Adding layer dependency meta-oe
INFO: Adding layer dependency meta-refkit-industrial
INFO: Adding layer dependency meta-python
INFO: Adding layer dependency meta-ros
INFO: Adding layer dependency meta-iotqa
INFO: Adding layer dependency meta-security
INFO: Adding layer dependency meta-perl
INFO: Adding layer dependency meta-security-isafw
INFO: Adding layer dependency meta-filesystems
INFO: Adding layer dependency meta-gnome
INFO: Adding layer dependency meta-networking
INFO: Adding layer dependency meta-flatpak
Traceback (most recent call last):
File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/yocto-compat-layer.py", line 203, in <module>
ret = main()
File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/yocto-compat-layer.py", line 151, in main

     if not add_layer_dependencies(bblayersconf, layer, dep_layers, logger) or \

Quoted 8 lines

File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/lib/compatlayer/__init__.py", line 195, in add_layer_dependencies
output = check_command('Getting existing layers failed.', 'bitbake-layers show-layers').decode('utf-8')
File "/fast/work/intel-iot-refkit-pr/openembedded-core/scripts/lib/compatlayer/__init__.py", line 228, in check_command
raise RuntimeError(msg)
RuntimeError: Getting existing layers failed.
Command: bitbake-layers show-layers
Output:
ERROR: Layer 'security' depends on layer 'xfce-layer', but this layer is not enabled in your configuration

Quoted 9 lines

The reason is the way how meta-security declares its dependencies:

LAYERDEPENDS_security = "core openembedded-layer perl-layer"
LAYERDEPENDS_security += "${@bb.utils.contains("DISTRO_FEATURES", "x11", "gnome-layer xfce-layer", "", d)}" LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "tpm", "tpm-layer", "",d)}" LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "tpm2", "tpm-layer", "",d)}" LAYERDEPENDS_security += "${@bb.utils.contains("MACHINE_FEATURES", "vtpm", "tpm-layer meta-filesystems", "",d)}"

Quoted 5 lines

I haven't checked how exactly yocto-compat-layer.py deals with this, but the end result is that it doesn't add meta-xfce when adding meta-security. Then when "bitbake-layers show-layers" is called, a distro (poky) has been chosen where x11 is in DISTRO_FEATURES, causing the dependency check to fail.

The meta-freescale method of checking if a layer is present and adding features is how they resolve this issue. Instead of making it a dependency, they simply check if it's already present and then add to it.

I agree the above is less then optimal way of doing things, and will prevent the layer index from getting a clear list of what may be required.

Note, there is also LAYERRECOMMENDS_... in this case, the system could recommend gnome-layer, xfce-layer, tpm-layer, meta-filesystems, and then use the meta-freescale approach to only enabling thsoe features if the recommended layers are present.

Quoted 17 lines

This highlights a conceptual issue with the dynamic LAYERDEPENDS_security above: which distro or machine config is active? Tools like yocto-compat-layer.py or the layer index have to do their work before that is defined.

This is also problematic with multiconfig. The same bblayers.conf might be used with one config where x11 is off and another where it is on.

I suspect LAYERDEPENDS simply can't be dynamic. It either has to include everything that a layer might depend on (even when the additional content doesn't end up being used), or it just depends on the essential parts and then enables additional recipes only when the optional layers they depend on are present. BBFILES_DYNAMIC can be used for that, or the recipes themselves check BBCOLLECTIONS in anonymous python code and raise a SkipRecipe exception when their pre-condition isn't satisfied.

Quoted 2 lines

Copying Paul (layer index) and Mark (oe-yocto-compat.py, setup tool) for their input.

I would certainly caution against dynamic layerdepends/layerrecommends and instead focus on listing everything between what is required and what you want to add in addition -- then using the meta-freescale approach of only extending (the recommends) when present.

The setup program and layer index can handle this situaton. The yocto-compat-layer checking may have to be augmented to deal with recommends as well, but that should be fairly minor.

--Mark

Quoted 4 lines

I would certainly caution against dynamic layerdepends/layerrecommends and instead focus on listing everything between what is required and what you want to add in addition -- then using the meta-freescale approach of only extending (the recommends) when present.

This appears to be more beneficial in a bbappends than a recipe needing a package from another layer.

Quoted 4 lines · On Thu, 2017-07-20 at 14:39 -0700, akuster808 wrote:

Quoted 4 lines

I would certainly caution against dynamic layerdepends/layerrecommends and instead focus on listing everything between what is required and what you want to add in addition -- then using the meta-freescale approach of only extending (the recommends) when present.

FWIW, the meta-freescale approach has been superseeded by BBFILES_DYNAMIC.

Quoted 2 lines

This appears to be more beneficial in a bbappends than a recipe needing a package from another layer.

BBFILES_DYNAMIC can be used for both .bb and .bbappend files.

In refkit, I ended up just using it for .bbappend files:
https://github.com/intel/intel-iot-refkit/blob/master/meta-refkit-core/conf/layer.conf#L7

# All our .bbappends for other layers are in a separate # "bbappends/<layer>" hierarchy. We activate only those # bbappends for which the layer they apply to is actually # present.
#
# Sorted by layer path to keep related layers together. BBFILES_DYNAMIC += " \
clang-layer:${LAYERDIR}/bbappends/meta-clang/*/*/*.bbappend \
flatpak:${LAYERDIR}/bbappends/meta-flatpak/*/*/*.bbappend \

For recipes, I wanted to have a bit more flexibility:
- let developers decide from where they get the dependency - the well-known layer or a copy elsewhere
- let recipes decide which features they enable based on which dependencies are available

Also, "bitbake foo" should result in an error about "dependency bar required for foo not found", not a blunt "nothing provides foo". With BBFILES_DYNAMIC for .bb files one only gets the latter.

Here's how it is defined what's available:
https://github.com/intel/intel-iot-refkit/blob/master/meta-refkit-core/conf/layer.conf#L51

                # There are multiple different ways for providing some of the
                # dependencies. Here we assume that the dependencies are available if
                # the layers that the refkit distro takes them from are present.
                HAVE_META_OE = "${@ bb.utils.contains('BBFILE_COLLECTIONS', 'openembedded-layer', 'True', 'False', d) }"
                HAVE_ATOP ??= "${HAVE_META_OE}"
            HAVE_CRYPTSETUP ??= "${HAVE_META_OE}"
            ...

And here's how it is used:
https://github.com/intel/intel-iot-refkit/blob/master/meta-refkit-core/recipes-images/images/initramfs-framework-refkit-dm-verity.bb#L22

python () {
if not oe.types.boolean(d.getVar('HAVE_CRYPTSETUP') or '0'):
raise bb.parse.SkipRecipe('cryptsetup dependency not available')
}