Skip to contentSkip to conversation

bitbake-setup: add the initial implementation

PATCH In review Alexander Kanavin 2 participants 4 messages Jun 30, 2025 – Jul 28, 2025

lore
Share & tools
lore
#0 lore raw

PATCH 1/2 bitbake-setup: add the initial implementation

From: Alexander Kanavin <[email protected]>

Preamble
========

The latest iteration of this patchset is available at
https://github.com/kanavin/bitbake
I recommend taking the patches from there to ensure that you are not trying out outdated code.

For the rationale and design guidelines please see this message:
https://lists.openembedded.org/g/openembedded-architecture/message/1913

Left out for now but will be done later:

- base bitbake configs (a way to declare the common parts between several bitbake build configurations just once, like a parent 'class') (this is inspired by a similar mechannism in yocto-autobuilder)

- official configuration repository (this probably depends on oe-core being populated with a rich, useful set of fragments, and providing sstate for official configurations)

- documentation

Amble *scratch* HOWTO
=====================

1. If you don't know where to start, list available configurations, and pick one:

===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup list
Created a new settings file in /home/alex/bitbake-builds/bitbake-setup.conf.

Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf.

Fetching configuration registry git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/bitbake-builds/.bitbake-setup-cache/configurations

Available configurations:
poky-alex Poky reference distribution, with alex fixes
poky-kirkstone Poky reference distribution, kirkstone long term support release (supported until April 2026)
poky-ng Poky-ng configuration: like poky but built from individual repositories

Run 'init' with one of the above configuration identifiers to set up a build.
===

2. Then build is initialized this way:
===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup init poky-alex
Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf.

Initializing a poky-alex build in /home/alex/bitbake-builds/poky-alex
Fetching configuration registry git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/bitbake-builds/.bitbake-setup-cache/configurations Fetching layer/tool repository poky into /home/alex/bitbake-builds/poky-alex/layers/poky
Fetching layer/tool repository meta-alex into /home/alex/bitbake-builds/poky-alex/layers/meta-alex
==============================
Setting up bitbake configuration gadget in /home/alex/bitbake-builds/poky-alex/build-gadget
Bitbake configuration summary: This configuration is intended for building gadget.
Usage instructions and additional information in /home/alex/bitbake-builds/poky-alex/build-gadget/README
==============================
Setting up bitbake configuration gizmo in /home/alex/bitbake-builds/poky-alex/build-gizmo
Bitbake configuration summary: This configuration is intended for building gizmo.
Usage instructions and additional information in /home/alex/bitbake-builds/poky-alex/build-gizmo/README
===

Note: 'init' sub-command can also take a path or a URL with a configuration file directly. You can see how those files look like here:
https://github.com/kanavin/bitbake-setup-configurations

3. The above message refers to a one-liner shell script that would build the targets specified in the chosen configuration:
===

alex@Zen2:/srv/work/alex/bitbake$ cat /home/alex/builds/poky-alex/build-gadget/build-targets
. /home/alex/builds/poky-alex/build-gadget/init-build-env && bitbake core-image-minimal
===

4. You can also source the bitbake environment, and then subsequent status/update commands will not require a --build-dir option telling bitbake-setup where the initialized build is.

5. To check if the build configuration needs to be updated, run:
===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup status --build-dir ~/bitbake-builds/poky-alex/
Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf.

Fetching configuration registry git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/bitbake-builds/.bitbake-setup-cache/configurations Configuration in /home/alex/bitbake-builds/poky-alex/ has not changed.
===

If the configuration has changed, you will see the difference:
===
...
Configuration in /home/alex/builds/poky-alex has changed:

home/alex/builds/poky-alex/config-tmp-asoubw5u/poky-alex.conf.json renamed from home/alex/builds/poky-alex/config/poky-alex.conf.json +1 -1
@@ -7,7 +7,7 @@
                         "uri": "git://git.yoctoproject.org/poky-contrib"
                     }
                 },
-                "rev": "akanavin/sstate-for-all"
+                "rev": "akanavin/bitbake-setup-testing"
             },
             "path": "poky"
         }

===

If the configuration has not changed, but layer revisions referred to it have (for example
if the configuration specifies a tip of a branch), you will see that too:
===
...
Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/sstate-for-all from 6b842ba55f996b27c900e3de78ceac8cb3b1c492 to aeb73e29379fe6007a8adc8d94c1ac18a93e68de
===

6. If the configuration has changed, you can bring it in sync with:
===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup update -build-dir ~/bitbake-builds/poky-alex/
Default parameter values are in /home/alex/.bitbake-setup/config - adjust as needed.

Fetching configuration repository git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/.bitbake-setup/configurations
Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/bitbake-setup-testing from d174acad934f8ad1fe303abc5705733e15542859 to a3d2ee10045f8c1151d680ad97994c5d6cf51ece Fetching layer/tool repository poky into /home/alex/builds/poky-alex/layers/poky

Setting up bitbake configuration gadget in /home/alex/bitbake-builds/poky-alex/build-gadget
Existing bitbake congfiguration directory renamed to /home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007 The bitbake configuration has changed:

home/alex/builds/poky-alex/build-gadget/conf/local.conf renamed from home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007/local.conf +0 -2
diff -uNr /home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007/local.conf /home/alex/builds/poky-alex/build-gadget/conf/local.conf
@@ -287,5 +287,3 @@
 # track the version of this file when it was generated. This can safely be ignored if
 # this doesn't mean anything to you.
 CONF_VERSION = "2"
-
-TCLIBC = "musl"

Bitbake configuration summary: This configuration is intended for building gadget. Usage instructions and additional information in /home/alex/bitbake-builds/poky-alex/build-gadget/README
===

Note that it will also rename/preserve the existing build/conf directory, and print changes in bitbake configuration (diff of content of build/conf/) if that has changed. I can't at the moment think of anything more clever that is also not much more brittle or complex to implement, but open to suggestions.

Terminology
===========

- 'top directory' means the place under which bitbake-setup reads and writes everything. bitbake-setup makes a promise to not touch anything outside of that, unless otherwise directed to by entries in settings (currently there is one such setting for fetcher downloads for layers and config registries). Top directory can be selected by a command line option, or otherwise assumed to be ~/bitbake-builds/. If BBPATH is in environment (e.g. we are in a bitbake environment), then the top directory is deduced from that and doesn't need to be specified by hand.

- 'settings' means bitbake-setup operational parameters that are global to all builds under a top directory. E.g. the location of configuration registry, or where the bitbake fetcher should place the downloads (DL_DIR setting). Settings are stored in a .conf file in ini format just under the top directory.

- ' build' means a tree structure set up by 'bitbake-setup init', consisting of, at least, a layers checkout, and a set of bitbake builds. It maps 1:1 to the json data it was constructed from, which is called 'configuration'. Configuration files are either standalone files, or are obtained from git repositories called 'config registries', in which case they can be listed with 'bitbake-setup list'. There can be multiple 'builds' under a top directory. Here's an example configuration that showcases this:
https://github.com/kanavin/bitbake-setup-configurations/blob/main/poky-alex.conf.json

- 'bitbake-setup status' will tell if a configuration is in sync with the build it was made from. 'bitbake-setup update' will bring a build in sync with a configuration if needed.

- 'bitbake build' means a particular sub-tree inside a build that bitbake itself operates on, e.g. what is set in BBPATH/BUILDDIR by oe-init-build-env. conf/* in that tree is 'bitbake configuration'. Bitbake configurations are constructed from templates and fragments, with existing mechanisms provided by oe-core. The configuration file format is specified such that other mechanisms to set up a bitbake build can be added; there was a mention of ability to specify local.conf content and a set of layers directly in a configuration. I think that scales poorly compared to templates and fragments, but I made sure alternative ways to configure a bitbake build are possible to add in the future :)

This commit includes a fix by Ryan Eatmon <[email protected]>:
https://github.com/kanavin/bitbake/pull/1

---
 bin/bitbake-setup | 563 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 563 insertions(+)
 create mode 100755 bin/bitbake-setup
bin/bitbake-setup new file +563 -0

563 lines not on this page · raw message

Signature · 1 line
2.39.5
#1 lore raw

PATCH 2/2 bitbake-setup: add tests to bitbake-selftest

From: Alexander Kanavin <[email protected]>

Run like this:

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-selftest -v bb.tests.setup
test_setup (bb.tests.setup.BitbakeSetupTest.test_setup) ... ok

----------------------------------------------------------------------
Ran 1 test in 9.223s

OK

The test does a basic run-through of init, then status/update on an unchanged configuration, then status/update on a configuration changed via new commits to the test layer, then status/update on configuration changed via the top level json config file.

Note that nothing whatsoever is fetched from the network; the test relies entirely on synthetic data contained inside itself, including minimal stubs for oe-setup-build and bitbake-config-build. This data is used to create temporary git repositories then clone them via local filesystem URIs.

Later on this can be supplemented by an oe-selftest that tests bitbake-setup against real config files in the official configuration repository and real layers, templates and fragments.

---
 bin/bitbake-selftest  |   1 +
 lib/bb/tests/setup.py | 251 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 252 insertions(+)
 create mode 100644 lib/bb/tests/setup.py
bin/bitbake-selftest +1 -0
index 1b7a783fd..8c1e6d3e7 100755
@@ -29,6 +29,7 @@ tests = ["bb.tests.codeparser",
          "bb.tests.fetch",
          "bb.tests.parse",
          "bb.tests.runqueue",
+         "bb.tests.setup",
          "bb.tests.siggen",
          "bb.tests.utils",
          "bb.tests.compression",
lib/bb/tests/setup.py new file +251 -0

251 lines not on this page · raw message

Signature · 1 line
2.39.5
#2 lore raw
In reply to Alexander Kanavin
Quoted 243 lines · Alexander Kanavin via lists.openembedded.org

From: Alexander Kanavin <[email protected]>

Preamble
========

The latest iteration of this patchset is available at
https://github.com/kanavin/bitbake
I recommend taking the patches from there to ensure that you are not trying out outdated code.

For the rationale and design guidelines please see this message:
https://lists.openembedded.org/g/openembedded-architecture/message/1913

Left out for now but will be done later:

- base bitbake configs (a way to declare the common parts between several bitbake build configurations just once, like a parent 'class') (this is inspired by a similar mechannism in yocto-autobuilder)

- official configuration repository (this probably depends on oe-core being populated with a rich, useful set of fragments, and providing sstate for official configurations)

- documentation

Amble *scratch* HOWTO
=====================

1. If you don't know where to start, list available configurations, and pick one:

===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup list
Created a new settings file in
/home/alex/bitbake-builds/bitbake-setup.conf.

Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf.

Fetching configuration registry git://
github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/bitbake-builds/.bitbake-setup-cache/configurations

Available configurations:
poky-alex Poky reference distribution, with alex fixes
poky-kirkstone Poky reference distribution, kirkstone long term support release (supported until April 2026)
poky-ng Poky-ng configuration: like poky but built from individual repositories

Run 'init' with one of the above configuration identifiers to set up a build.
===

2. Then build is initialized this way:
===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup init poky-alex
Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf.

Initializing a poky-alex build in /home/alex/bitbake-builds/poky-alex Fetching configuration registry git://
github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/bitbake-builds/.bitbake-setup-cache/configurations Fetching layer/tool repository poky into /home/alex/bitbake-builds/poky-alex/layers/poky
Fetching layer/tool repository meta-alex into /home/alex/bitbake-builds/poky-alex/layers/meta-alex
==============================
Setting up bitbake configuration gadget in /home/alex/bitbake-builds/poky-alex/build-gadget
Bitbake configuration summary: This configuration is intended for building gadget.
Usage instructions and additional information in /home/alex/bitbake-builds/poky-alex/build-gadget/README
==============================
Setting up bitbake configuration gizmo in /home/alex/bitbake-builds/poky-alex/build-gizmo
Bitbake configuration summary: This configuration is intended for building gizmo.
Usage instructions and additional information in /home/alex/bitbake-builds/poky-alex/build-gizmo/README
===

Note: 'init' sub-command can also take a path or a URL with a configuration file directly.
You can see how those files look like here:
https://github.com/kanavin/bitbake-setup-configurations

3. The above message refers to a one-liner shell script that would build the targets
specified in the chosen configuration:
===

alex@Zen2:/srv/work/alex/bitbake$ cat
/home/alex/builds/poky-alex/build-gadget/build-targets
. /home/alex/builds/poky-alex/build-gadget/init-build-env && bitbake
core-image-minimal
===

4. You can also source the bitbake environment, and then subsequent status/update commands
will not require a --build-dir option telling bitbake-setup where the initialized build is.

5. To check if the build configuration needs to be updated, run:
===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup status --build-dir
~/bitbake-builds/poky-alex/
Loading settings from /home/alex/bitbake-builds/bitbake-setup.conf.

Fetching configuration registry git://
github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/bitbake-builds/.bitbake-setup-cache/configurations Configuration in /home/alex/bitbake-builds/poky-alex/ has not changed.
===

If the configuration has changed, you will see the difference:
===
...
Configuration in /home/alex/builds/poky-alex has changed:
--- /home/alex/builds/poky-alex/config/poky-alex.conf.json 2024-12-16
11:43:24.077446096 +0100
+++ /home/alex/builds/poky-alex/config-tmp-asoubw5u/poky-alex.conf.json

2024-12-16 11:47:43.237104405 +0100
home/alex/builds/poky-alex/config-tmp-asoubw5u/poky-alex.conf.json renamed from home/alex/builds/poky-alex/config/poky-alex.conf.json +1 -1
@@ -7,7 +7,7 @@
                         "uri": "git://git.yoctoproject.org/poky-contrib"
                     }
                 },
-                "rev": "akanavin/sstate-for-all"
+                "rev": "akanavin/bitbake-setup-testing"
             },
             "path": "poky"
         }

===

If the configuration has not changed, but layer revisions referred to it have (for example
if the configuration specifies a tip of a branch), you will see that too:
===
...
Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/sstate-for-all from 6b842ba55f996b27c900e3de78ceac8cb3b1c492 to aeb73e29379fe6007a8adc8d94c1ac18a93e68de
===

6. If the configuration has changed, you can bring it in sync with:
===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup update -build-dir
~/bitbake-builds/poky-alex/
Default parameter values are in /home/alex/.bitbake-setup/config - adjust
as needed.

Fetching configuration repository git://
github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/.bitbake-setup/configurations
Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/bitbake-setup-testing from d174acad934f8ad1fe303abc5705733e15542859 to a3d2ee10045f8c1151d680ad97994c5d6cf51ece
Fetching layer/tool repository poky into /home/alex/builds/poky-alex/layers/poky

Setting up bitbake configuration gadget in /home/alex/bitbake-builds/poky-alex/build-gadget
Existing bitbake congfiguration directory renamed to /home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007 The bitbake configuration has changed:
diff -uNr /home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007/local.conf /home/alex/builds/poky-alex/build-gadget/conf/local.conf
---
/home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007/local.conf

    2024-12-16 11:47:51.865043102 +0100

+++ /home/alex/builds/poky-alex/build-gadget/conf/local.conf 2024-12-16
11:50:07.811942847 +0100

(quoted diff) +0 -0
@@ -287,5 +287,3 @@
 # track the version of this file when it was generated. This can safely

be ignored if

 # this doesn't mean anything to you.
 CONF_VERSION = "2"
home/alex/builds/poky-alex/build-gadget/conf/local.conf renamed from home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007/local.conf +0 -2
-
-TCLIBC = "musl"

Bitbake configuration summary: This configuration is intended for building gadget.
Usage instructions and additional information in /home/alex/bitbake-builds/poky-alex/build-gadget/README
===

Note that it will also rename/preserve the existing build/conf directory, and print changes
in bitbake configuration (diff of content of build/conf/) if that has changed. I can't
at the moment think of anything more clever that is also not much more brittle or complex
to implement, but open to suggestions.

Terminology
===========

- 'top directory' means the place under which bitbake-setup reads and writes everything. bitbake-setup makes a promise to not touch anything outside of
that, unless otherwise directed to by entries in settings (currently there is one such setting for fetcher downloads for layers and config registries). Top directory can be selected by a command line option, or otherwise assumed to be ~/bitbake-builds/. If BBPATH is in environment (e.g. we are in a bitbake environment), then the top directory is deduced from that and doesn't need to be specified by hand.

- 'settings' means bitbake-setup operational parameters that are global to all builds under a top directory. E.g. the location of configuration registry, or where the bitbake fetcher should place the downloads (DL_DIR setting). Settings are stored in a .conf file in ini format just under the top directory.

- ' build' means a tree structure set up by 'bitbake-setup init', consisting of, at least, a layers checkout, and a set of bitbake builds. It maps 1:1 to the json data it was constructed from, which is called 'configuration'. Configuration files are either standalone files, or are obtained from git repositories called 'config registries', in which case they can be listed with 'bitbake-setup list'. There can be multiple 'builds' under a top directory. Here's an example configuration that showcases this:

https://github.com/kanavin/bitbake-setup-configurations/blob/main/poky-alex.conf.json

- 'bitbake-setup status' will tell if a configuration is in sync with the build it was made from. 'bitbake-setup update' will bring a build in sync with a configuration if needed.

- 'bitbake build' means a particular sub-tree inside a build that bitbake itself operates on, e.g. what is set in BBPATH/BUILDDIR by oe-init-build-env. conf/* in that tree is 'bitbake configuration'. Bitbake configurations are constructed from templates and fragments, with existing mechanisms provided by oe-core. The configuration file format is specified such that other mechanisms to set up a bitbake build can be added; there was a mention of ability to specify local.conf content and a set of layers directly in a configuration. I think that scales poorly compared to templates and fragments, but I made sure alternative ways to configure a bitbake build are possible to add in the future :)

Quoted 5 lines

This commit includes a fix by Ryan Eatmon <[email protected]>:
https://github.com/kanavin/bitbake/pull/1

Hello Alexander,

Here is my attempt at reviewing bitbake-setup. I'll explain how I've tried to use it, what I found along the way.
I have not (yet) reviewed the code, only the user perspective.

First, I find that this is definitely the right track.
I like how it is based on bitbake tools and not directly writing in .conf files, that will definitely help maintenance long-term.

Then, here is what I've did: I've try to migrate 2 projects from Cooker (
https://github.com/cpb-/yocto-cooker) to bitbake-setup:
* our contribution project (basically yocto-master but pointed at our gitlab mirrors)
* a simple existing project of one of our customer (Scarthgap poky.git + BSP layers + some lib layers, 1 distro, 2 images)

The contribution project migrated quite easily but I was stuck on the scarthgap project.
The bitbake inside does not have bitbake-config-build which is needed to configure a build from fragments.
Maybe I could have used the oe-template mechanism instead of the fragments from config?
I finally (virtually) upgraded the project to master and then, no trouble to migrate the configuration.

I agree with the need to make some elements common between configurations:
* most config of a project will share the same layer set,
* the "dev" config should inherit the "prod" config, ...
(You mentioned this in your intro)

I think I found 2 bugs:
* Calling "bitbake-setup init non-existing-file.json" will default to cloning the default registry (where I expected to fail with a error message)
* When sh is dash, the init process fail because dash does not support the source argument:
see https://git.yoctoproject.org/poky/tree/oe-init-build-env#n11
and where the bug is (I guess):
https://github.com/kanavin/bitbake/blob/akanavin/bitbake-setup/bin/bitbake-setup#L136

Coming from Cooker, I miss some features (I'll totally accept to be told to adapt ^^):
* setting bitbake conf variable directly from the config json (but I agree the fragments are neater)
* Using already checked-out layers : For simple projects, we have the JSON configuration and the project layers inside the same git repo. IIUC, bb-setup always does a clone for layers.
Maybe this can be fixed by not handling layers as a "source" and having a way to add layer to bb configuration by their relative path to the JSON config (similar to how OEROOT template work?)
* I don't like mixing projects, so that mean I'll use different top-dir for each. Having to repeat --top-dir everytime is a bit cumbersome, maybe add an envvar that can change the topdir defaut?

Something not directly related to bb-setup I saw while testing: the variable history for fragments could maybe be nicer (I'll create a BZ bug for this.)

 $ bitbake-getvar MACHINE
# $MACHINE [2 operations]
#   set ast.py:367 [check_and_set_builtin_fragment]       <== which
fragment? from where?
#     "qemux86-64"
...
# pre-expansion value:
#   "qemux86-64"
MACHINE="qemux86-64"

Finally, a question, how do you recommend to handle my local SSTATE_DIR/DL_DIR? Setup with bbsetup, copy a site.conf with these config inside the build, then the proper build? ... or is there a better way?

Sorry for the messy mail, I hope it will be somewhat useful.

Regards,

Signature · 2 lines
Yoann Congal
Smile ECS
#3 lore raw
Quoted 2 lines · Yoann Congal

Here is my attempt at reviewing bitbake-setup. I'll explain how I've tried to use it, what I found along the way. I have not (yet) reviewed the code, only the user perspective.

Hello, and thanks for looking.

Quoted 2 lines

First, I find that this is definitely the right track.
I like how it is based on bitbake tools and not directly writing in .conf files, that will definitely help maintenance long-term.

Yes. The problem with tools like kas is that they take the easy way out of managing everything themselves, instead of first taking the trouble to add the needed pieces to the core project, having them available in a yocto release, and then using them. For bitbake-setup, the goal from the start was to build the integrated functionality from smaller concepts and tools available directly in bitbake/core. That's why bitbake-setup itself is comparatively short (again, compare with lines of code in kas).

Quoted 8 lines

Then, here is what I've did: I've try to migrate 2 projects from Cooker (https://github.com/cpb-/yocto-cooker) to bitbake-setup:
* our contribution project (basically yocto-master but pointed at our gitlab mirrors)
* a simple existing project of one of our customer (Scarthgap poky.git + BSP layers + some lib layers, 1 distro, 2 images)

The contribution project migrated quite easily but I was stuck on the scarthgap project. The bitbake inside does not have bitbake-config-build which is needed to configure a build from fragments. Maybe I could have used the oe-template mechanism instead of the fragments from config?
I finally (virtually) upgraded the project to master and then, no trouble to migrate the configuration.

Yes, scarthgap is too old for fragments, so the only option is templates. Bitbake-setup will have reduced functionality when used with older yocto releases. Something even older (like kirkstone) has no support for templates (via oe-setup-build) either, so bitbake-setup will only check out the layers, create bblayers.conf with a subset of those, and then stop.

Quoted 4 lines

I agree with the need to make some elements common between configurations:
* most config of a project will share the same layer set,
* the "dev" config should inherit the "prod" config, ...
(You mentioned this in your intro)

Yes, I'm going to implement this.

Quoted 2 lines

I think I found 2 bugs:
* Calling "bitbake-setup init non-existing-file.json" will default to cloning the default registry (where I expected to fail with a error message)

The problem is ambiguity of whether that argument is a file or a reference into the config registry.

If the file is non-existing on local disk, then bitbake-setup will try to find it in the registry. Otherwise how would it know that it's not in the registry either? I can tweak the logic, but you need to propose a better sequence (or UI).

Quoted 3 lines

* When sh is dash, the init process fail because dash does not support the source argument:
see https://git.yoctoproject.org/poky/tree/oe-init-build-env#n11
and where the bug is (I guess): https://github.com/kanavin/bitbake/blob/akanavin/bitbake-setup/bin/bitbake-setup#L136

Can you provide a way to reproduce this? sh is dash on my build machine and laptop (both debian), and I have not seen issues so far.

Quoted 2 lines

Coming from Cooker, I miss some features (I'll totally accept to be told to adapt ^^):
* setting bitbake conf variable directly from the config json (but I agree the fragments are neater)

As you may expect, the answer is a firm no. I'd like to move away from doing standard configurations with a pile of random undocumented tweaks in local.conf, and allowing this would be basically the same. If you need to tweak something in the standard config, there's always a better place for maintaining it than local.conf (or the top level project config file): distros, fragments or bbappends.

Quoted 2 lines

* Using already checked-out layers : For simple projects, we have the JSON configuration and the project layers inside the same git repo. IIUC, bb-setup always does a clone for layers. Maybe this can be fixed by not handling layers as a "source" and having a way to add layer to bb configuration by their relative path to the JSON config (similar to how OEROOT template work?)

This would break status/update commands? I'd say it's fine to put json file into a layer, and list that same layer in the json layer checkout section. Then it will be first cloned as registry for getting to the config, and then again under layers/ for using the layers in it in a build.

Quoted 1 line

* I don't like mixing projects, so that mean I'll use different top-dir for each. Having to repeat --top-dir everytime is a bit cumbersome, maybe add an envvar that can change the topdir defaut?

Bitbake-setup makes a separate directory under the top-dir for each project, so they should not step on each other. But there can be different reasons to tweak the top dir, so I'll add this. Perhaps a ~/.bitbake-setup-topdir file that sets the default as well. I don't particularly like magic environment variables, as their presence at runtime is harder to debug than presence of config files on disk.

Quoted 1 line

Finally, a question, how do you recommend to handle my local SSTATE_DIR/DL_DIR? Setup with bbsetup, copy a site.conf with these config inside the build, then the proper build? ... or is there a better way?

This is an unresolved question. I'm open to proposals. Or code.

I guess it will be something like 'bitbake-setup change-setting default site-conf /path/to/site.conf', and then bitbake-setup will add that to every build that is initialized. Site.conf need not list site settings directly, it too can add a fragment maintained in a layer. That way CI settings can be specified and maintained in a layer, but they don't need to be added to a json config, as that would impose them on everyone using that config to reproduce builds locally.

Quoted 1 line

Sorry for the messy mail, I hope it will be somewhat useful.

It certainly is!

Alex