Skip to contentSkip to conversation

How to write a distro layer?

DISCUSSION Uwe Geuder 2 participants 4 messages Aug 3, 2018

lore
Share & tools
lore

Uwe Geuder

author
#0 lore raw

Hi!

[Disclaimer: I have posted a similar message months ago. There were no replies. Let's try whether I can express myself clearer this time.]

One of the few informations I could find about how to create my own layer was something like
https://www.openembedded.org/wiki/Creating_a_new_Layer

Especially for creating a distro layer that does not give much guidance.

I could not find any examples of distro layers (except poky of course) on the net. Can somebody give me pointers?

The auto-generated local.conf mentions subclassing poky. Is that concept explained anywhere?

I inherited a distro layer, which sets a lot of its policy (selection of systemd and other feature/package selection) in layer.conf. From reading the documentation and other layer.conf files I understand that is NOT the way to do it. I would expect those settings in distro/<distro>.conf.

The yocto reference manual says for many variables: "Set it in your local.conf". For me they are policy, so why shouldn't I set them in distro/<distro>.conf?

Well, basically there is only one configuration in bitbake. So often it does not matter in what configuration file a variable is set. The setting will end up in the environment of the tasks. However, especially when the ?= operator is used order becomes significant.

Particularly tricky I found the DISTRO variable.

Our distro layer does

1.) set the DISTRO variable in layer.conf (according to the manual it should be done in local.conf)
2.) from distro/<distro>.conf include (well, require) poky.conf. Is that a big no-no??? Of course I could copy all settings I want to keep from poky (and all those I don't understand what they do...), but copying code is generally not good SW engineering, including sounds better. 3.) set DISTRO again after having included poky.conf . According to the yocto-check-layer script a distro must not set its own name. However, poky does not follow that rule and sets its name, so our code has to violate the rule once more and revert poky's wrongdoing.

In an attempt to make our code more in line with what I understand to be correct style, I tried to move all policy settings to distro/<distro>.conf and the DISTRO setting to local.conf.

However that created a couple of problems:

1. If I set DISTRO in local.conf before adding all needed layers, my own distro layer last, bitbake-layers fails in a sanity check that the distro does not exist. Fair enough, it doesn't know about before it has been added.

2. If I first modify my local.conf with all my settings except DISTRO, then run all the add-layer operations and finally modify local.conf a 2nd time to set DISTRO correctly the build works. However, all recipes are reparsed when the build starts. I guess this is caused by the config change.

So in the end having the DISTRO setting in layer.conf seems to work best. All add-layer commands succeed and recipes can be used from the cache when the build starts. Best, except that I don't think that's the way it's documented.

Any thoughts what I might be missing?

Regards,

Uwe

Uwe Geuder
Neuro Event Labs Oy
Tampere, Finland
[email protected] (Bot check: fix one obvious typo)

#1 lore raw

Hi Uwe,

2018-08-03 10:39 GMT+02:00 Uwe Geuder <[email protected]>:

Quoted 9 lines

Hi!

[Disclaimer: I have posted a similar message months ago. There were no replies. Let's try whether I can express myself clearer this time.]

One of the few informations I could find about how to create my own layer was something like
https://www.openembedded.org/wiki/Creating_a_new_Layer

Did you check
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#creating-a-general-layer-using-the-bitbake-layers-script

Quoted 8 lines

Especially for creating a distro layer that does not give much guidance.

I could not find any examples of distro layers (except poky of course) on the net. Can somebody give me pointers?

The auto-generated local.conf mentions subclassing poky. Is that concept explained anywhere?

Quoted 10 lines

I inherited a distro layer, which sets a lot of its policy (selection of systemd and other feature/package selection) in layer.conf. From reading the documentation and other layer.conf files I understand that is NOT the way to do it. I would expect those settings in distro/<distro>.conf.

The yocto reference manual says for many variables: "Set it in your local.conf". For me they are policy, so why shouldn't I set them in distro/<distro>.conf?

Did you also check
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#creating-your-own-distribution

You can look some Freescale example here:
https://github.com/Freescale/meta-freescale-distro

In fact, when Yocto manual refers to adding something to local.conf, it is the minimal way to do it,
but the cleanest is indeed to set it in machine/distro config file.

Quoted 9 lines

Well, basically there is only one configuration in bitbake. So often it does not matter in what configuration file a variable is set. The setting will end up in the environment of the tasks. However, especially when the ?= operator is used order becomes significant.

Particularly tricky I found the DISTRO variable.

Our distro layer does

Quoted 12 lines

1.) set the DISTRO variable in layer.conf (according to the manual it should be done in local.conf)
2.) from distro/<distro>.conf include (well, require) poky.conf. Is that a big no-no??? Of course I could copy all settings I want to keep from poky (and all those I don't understand what they do...), but copying code is generally not good SW engineering, including sounds better. 3.) set DISTRO again after having included poky.conf . According to the yocto-check-layer script a distro must not set its own name. However, poky does not follow that rule and sets its name, so our code has to violate the rule once more and revert poky's wrongdoing.

1.) DISTRO is local to build folder, so it should be in local.conf, and you can use a wrapper script to help creating this local.conf file, see
https://github.com/Freescale/fsl-community-bsp-base/blob/master/setup-environment
for instance.

Quoted 17 lines

In an attempt to make our code more in line with what I understand to be correct style, I tried to move all policy settings to distro/<distro>.conf and the DISTRO setting to local.conf.

However that created a couple of problems:

1. If I set DISTRO in local.conf before adding all needed layers, my own distro layer last, bitbake-layers fails in a sanity check that the distro does not exist. Fair enough, it doesn't know about before it has been added.

2. If I first modify my local.conf with all my settings except DISTRO, then run all the add-layer operations and finally modify local.conf a 2nd time to set DISTRO correctly the build works. However, all recipes are reparsed when the build starts. I guess this is caused by the config change.

Quoted 5 lines

So in the end having the DISTRO setting in layer.conf seems to work best. All add-layer commands succeed and recipes can be used from the cache when the build starts. Best, except that I don't think that's the way it's documented.

Any thoughts what I might be missing?

Quoted 1 line

You can use git submodule, or google repo to create Yocto environment, you don't need to create your layer each time and you don't have to compile before using bitbake-layers.

Quoted 1 line

Regards,

Quoted 14 lines

Uwe

Uwe Geuder
Neuro Event Labs Oy
Tampere, Finland
[email protected] (Bot check: fix one obvious typo) --
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto

author
#2 lore raw

Thanks Vincent for your quick reply!

Quoted 8 lines · Vincent Prince

2018-08-03 10:39 GMT+02:00 Uwe Geuder <[email protected]>:

Quoted 3 lines

One of the few informations I could find about how to create my own layer was something like
https://www.openembedded.org/wiki/Creating_a_new_Layer

Did you check
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#creating-a-general-layer-using-the-bitbake-layers-script

Yes. The template generated by that script is very minimal and generic. I don't see any guidance for distro layers in particular. As a matter fact I believe our layer has been created by that script (although that was long before my time).

Quoted 6 lines
Quoted 3 lines

The yocto reference manual says for many variables: "Set it in your local.conf". For me they are policy, so why shouldn't I set them in distro/<distro>.conf?

Did you also check
https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#creating-your-own-distribution

I had read that many times before, but not recently so let's check again...

| Note
| The DISTRO variable in your local.conf file determines the name of
| your distribution

[...]

| Point to Your distribution configuration file: In your local.conf
|  file in the Build Directory, set your DISTRO variable to point to
|  your distribution's configuration file. For example, if your
|  distribution's configuration file is named mydistro.conf, then you
|  point to it as follows:
|
|     DISTRO = "mydistro"

Correct, that's what I remembered. But if I set it in local.conf before adding my layer the bitbake sanity check will complain that there is no such layer. And if I modify local.conf after adding the layer all recipes will be re-parsed when building the first time.

I don't see any guidance in what order the steps should be carried out when setting up a new build area.

They do indicate that you should copy another distro's conf file, that's why I asked whether using "require" as we do is a bad/dangerous idea.

Quoted 2 lines

You can look some Freescale example here:
https://github.com/Freescale/meta-freescale-distro

Good, I hadn't found those. At least the distro/<name>.conf files look pretty much like I would expect them.

But all of them (I understand the layer defines 8 different distros) do set the DISTRO variable. I haven't run the yocto-check-layer for a while, but the last time I did it complained that a distro layer MUST NOT set its own DISTRO variable. Do you happen to know whether meta-freescale-distro passes the yocto-check-layer script without complaints? (I know I could test myself...)

Quoted 3 lines

In fact, when Yocto manual refers to adding something to local.conf, it is the minimal way to do it, but the cleanest is indeed to set it in machine/distro config file.

That has also been my thinking. If that's generally accepted we probably should send a patch to the manual. It's pretty misleading if the manual without any further explanations says "do it way A" if "way B" often is actually the better one.

Quoted 1 line

1.) DISTRO is local to build folder,

By "local to build folder", do you mean that within the same build folder one cannot build more than 1 distro? Not relevant for me, because I have only one for the time being, but just to understand the message.

Quoted 2 lines

so it should be in local.conf, and you can use a wrapper script to help creating this local.conf file,

Yes, that's what we do. Without a wrapper script builds were unlikely to be reproducible by any measure...

Quoted 1 line

https://github.com/Freescale/fsl-community-bsp-base/blob/master/setup-environment

Interesting. I cannot spot how that script handles layers.

My own script

1) sources oe-init-build-env
2) modifies the generated conf/local.conf
3) runs bitbake-layer add-layer for all layers we need in our distro

Quoted 17 lines
Quoted 17 lines

In an attempt to make our code more in line with what I understand to be correct style, I tried to move all policy settings to distro/<distro>.conf and the DISTRO setting to local.conf.

However that created a couple of problems:

1. If I set DISTRO in local.conf before adding all needed layers, my own distro layer last, bitbake-layers fails in a sanity check that the distro does not exist. Fair enough, it doesn't know about before it has been added.

2. If I first modify my local.conf with all my settings except DISTRO, then run all the add-layer operations and finally modify local.conf a 2nd time to set DISTRO correctly the build works. However, all recipes are reparsed when the build starts. I guess this is caused by the config change.

Quoted 8 lines
Quoted 6 lines

So in the end having the DISTRO setting in layer.conf seems to work best. All add-layer commands succeed and recipes can be used from the cache when the build starts. Best, except that I don't think that's the way it's documented.

Any thoughts what I might be missing?

You can use git submodule, or google repo to create Yocto environment,

Yes, we have all layers as submodules in a single git repo, so we always can do a clean build "from a single SHA1"

Quoted 1 line

you don't need to create your layer each time

What do you mean by creating a layer? Of course the source is in git and it only changes when necessary.

What I do in every new build area is run add-layer. (Currently still repeatedly for one layer at a time, but I think I read in the new release notes that bitbake-layer can now add several layers at once.)

I have thought about saving the generated layers.conf file and restoring it by my setup script. But somehow I feel more confident if the parsing is done for each new build area. Otherwise one might end up in the situation that one day you note that a change made weeks or months ago has broken something and nobody noted it.

Quoted 1 line

and you don't have to compile before using bitbake-layers.

Not sure what you mean by that. Of course I cannot build/compile anything before the build area is set-up completely.

Bitbake-layer parses all recipes of the layer it adds. If I do not modify local.conf after adding the last layer the first build starts immediately from cached recipes.

If I modify local.conf to set DISTRO after adding the last layer, the first build re-parses everything. Not that it's a huge increase to the whole build time, but it does not feel right.

Regards,

Uwe

---
Uwe Geuder
Neuro Event Labs Oy
Tampere, Finland
[email protected] (Bot check: fix one obvious typo)

#3 lore raw

You don't need to use add-layer, just create conf/bblayers.conf with
wanted layers:

LCONF_VERSION = "7"

BBPATH = "${TOPDIR}"
BSPDIR := "${@os.path.abspath(os.path.dirname(d.getVar('FILE', True)) + '/../..')}"

BBFILES ?= ""
BBLAYERS = " \
${BSPDIR}/sources/poky/meta \
${BSPDIR}/sources/poky/meta-poky \
${BSPDIR}/sources/meta-openembedded/meta-oe \
"

2018-08-03 15:43 GMT+02:00 Uwe Geuder <[email protected]>:

Quoted 176 lines

Thanks Vincent for your quick reply!

Quoted 8 lines · Vincent Prince

2018-08-03 10:39 GMT+02:00 Uwe Geuder <[email protected]>:

Quoted 3 lines

One of the few informations I could find about how to create my own layer was something like
https://www.openembedded.org/wiki/Creating_a_new_Layer

Did you check
https://www.yoctoproject.org/docs/latest/mega-manual/mega-

manual.html#creating-a-general-layer-using-the-bitbake-layers-script

Yes. The template generated by that script is very minimal and generic. I don't see any guidance for distro layers in particular. As a matter fact I believe our layer has been created by that script (although that was long before my time).

Quoted 6 lines
Quoted 3 lines

The yocto reference manual says for many variables: "Set it in your local.conf". For me they are policy, so why shouldn't I set them in distro/<distro>.conf?

Did you also check
https://www.yoctoproject.org/docs/latest/mega-manual/mega-

manual.html#creating-your-own-distribution

I had read that many times before, but not recently so let's check again...

| Note
| The DISTRO variable in your local.conf file determines the name of
| your distribution

[...]

| Point to Your distribution configuration file: In your local.conf
|  file in the Build Directory, set your DISTRO variable to point to
|  your distribution's configuration file. For example, if your
|  distribution's configuration file is named mydistro.conf, then you
|  point to it as follows:
|
|     DISTRO = "mydistro"

Correct, that's what I remembered. But if I set it in local.conf before adding my layer the bitbake sanity check will complain that there is no such layer. And if I modify local.conf after adding the layer all recipes will be re-parsed when building the first time.

I don't see any guidance in what order the steps should be carried out when setting up a new build area.

They do indicate that you should copy another distro's conf file, that's why I asked whether using "require" as we do is a bad/dangerous idea.

Quoted 2 lines

You can look some Freescale example here:
https://github.com/Freescale/meta-freescale-distro

Good, I hadn't found those. At least the distro/<name>.conf files look pretty much like I would expect them.

But all of them (I understand the layer defines 8 different distros) do set the DISTRO variable. I haven't run the yocto-check-layer for a while, but the last time I did it complained that a distro layer MUST NOT set its own DISTRO variable. Do you happen to know whether meta-freescale-distro passes the yocto-check-layer script without complaints? (I know I could test myself...)

Quoted 3 lines

In fact, when Yocto manual refers to adding something to local.conf, it is the minimal way to do it, but the cleanest is indeed to set it in machine/distro config file.

That has also been my thinking. If that's generally accepted we probably should send a patch to the manual. It's pretty misleading if the manual without any further explanations says "do it way A" if "way B" often is actually the better one.

Quoted 1 line

1.) DISTRO is local to build folder,

By "local to build folder", do you mean that within the same build folder one cannot build more than 1 distro? Not relevant for me, because I have only one for the time being, but just to understand the message.

Quoted 2 lines

so it should be in local.conf, and you can use a wrapper script to help creating this local.conf file,

Yes, that's what we do. Without a wrapper script builds were unlikely to be reproducible by any measure...

Quoted 1 line

https://github.com/Freescale/fsl-community-bsp-base/blob/

master/setup-environment

Interesting. I cannot spot how that script handles layers.

My own script

1) sources oe-init-build-env
2) modifies the generated conf/local.conf
3) runs bitbake-layer add-layer for all layers we need in our distro

Quoted 17 lines
Quoted 17 lines

In an attempt to make our code more in line with what I understand to be correct style, I tried to move all policy settings to distro/<distro>.conf and the DISTRO setting to local.conf.

However that created a couple of problems:

1. If I set DISTRO in local.conf before adding all needed layers, my own distro layer last, bitbake-layers fails in a sanity check that the distro does not exist. Fair enough, it doesn't know about before it has been added.

2. If I first modify my local.conf with all my settings except DISTRO, then run all the add-layer operations and finally modify local.conf a 2nd time to set DISTRO correctly the build works. However, all recipes are reparsed when the build starts. I guess this is caused by the config change.

Quoted 8 lines
Quoted 6 lines

So in the end having the DISTRO setting in layer.conf seems to work best. All add-layer commands succeed and recipes can be used from the cache when the build starts. Best, except that I don't think that's the way it's documented.

Any thoughts what I might be missing?

You can use git submodule, or google repo to create Yocto environment,

Yes, we have all layers as submodules in a single git repo, so we always can do a clean build "from a single SHA1"

Quoted 1 line

you don't need to create your layer each time

What do you mean by creating a layer? Of course the source is in git and it only changes when necessary.

What I do in every new build area is run add-layer. (Currently still repeatedly for one layer at a time, but I think I read in the new release notes that bitbake-layer can now add several layers at once.)

I have thought about saving the generated layers.conf file and restoring it by my setup script. But somehow I feel more confident if the parsing is done for each new build area. Otherwise one might end up in the situation that one day you note that a change made weeks or months ago has broken something and nobody noted it.

Quoted 1 line

and you don't have to compile before using bitbake-layers.

Not sure what you mean by that. Of course I cannot build/compile anything before the build area is set-up completely.

Bitbake-layer parses all recipes of the layer it adds. If I do not modify local.conf after adding the last layer the first build starts immediately from cached recipes.

If I modify local.conf to set DISTRO after adding the last layer, the first build re-parses everything. Not that it's a huge increase to the whole build time, but it does not feel right.

Regards,

Uwe

---
Uwe Geuder
Neuro Event Labs Oy
Tampere, Finland
[email protected] (Bot check: fix one obvious typo) --
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto