Skip to contentSkip to conversation

bitbake-setup: add the proof of concept implementation

RFC PATCH New Alexander Kanavin 1 participant 2 messages Jan 27, 2025

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/2 bitbake-setup: add the proof of concept implementation

From: Alexander Kanavin <[email protected]>

Preamble
========

The latest iteration of this patchset is available at
https://github.com/kanavin/bitbake
I recommend taking the patches from there to ensure that you are not trying out outdated code.

For the rationale and design guidelines please see this message:
https://lists.openembedded.org/g/openembedded-architecture/message/1913

Left out for now but will be done later:

- base bitbake configs (a way to declare the common parts between several bitbake build configurations just once, like a parent 'class') (this is inspired by a similar mechannism in yocto-autobuilder)

- documentation

- official configuration repository (this probably depends on oe-core being populated with a rich, useful set of fragments, and providing sstate for official configurations)

Amble *scratch* HOWTO
=====================

1. If you don't know where to start, list available configurations, and pick one:

===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup list
Default parameter values are in /home/alex/.bitbake-setup/config - adjust as needed.

Fetching configuration repository git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/.bitbake-setup/configurations

Available configurations:
poky-alex Poky reference distribution, with alex fixes
poky-kirkstone Poky reference distribution, kirkstone long term support release (supported until April 2026)
poky-ng Poky-ng configuration: like poky but built from individual repositories
===

2. Then build is initialized this way:
===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup init poky-ng
Default parameter values are in /home/alex/.bitbake-setup/config - adjust as needed.

Initializing a poky-ng build in /home/alex/builds/poky-ng
Fetching configuration repository git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/.bitbake-setup/configurations Fetching layer/tool repository bitbake into /home/alex/builds/poky-ng/layers/bitbake
Fetching layer/tool repository openembedded-core into /home/alex/builds/poky-ng/layers/openembedded-core
Fetching layer/tool repository meta-yocto into /home/alex/builds/poky-ng/layers/meta-yocto
Fetching layer/tool repository yocto-docs into /home/alex/builds/poky-ng/layers/yocto-docs

Setting up configuration in /home/alex/builds/poky-ng/build-default Configuration summary:
This is the default build configuration for the openembedded-core layer.

Additional information is in /home/alex/builds/poky-ng/build-default/conf/conf-notes.txt

Run /home/alex/builds/poky-ng/build-default/build-targets to execute the default build targets for this configuration. Source the environment using '. /home/alex/builds/poky-ng/build-default/init-build-env' to run builds from the command line. The bitbake build configuration (local.conf, bblayers.conf and more) can be found in /home/alex/builds/poky-ng/build-default/conf
===

Note: 'init' sub-command can also take a path or a URL with a configuration file directly. You can see how those files look like here:
https://github.com/kanavin/bitbake-setup-configurations

3. The above message refers to a one-liner shell script that would build the targets specified in the chosen configuration:
===

alex@Zen2:/srv/work/alex/bitbake$ cat /home/alex/builds/poky-alex/build-gadget/build-targets
. /home/alex/builds/poky-alex/build-gadget/init-build-env && bitbake core-image-minimal
===

4. You should also source the environment, and then subsequent status/update commands will not require a parameter telling bitbake-setup where the initialized build is.

5. To check if the build configuration needs to be updated, run:
===
Default parameter values are in /home/alex/.bitbake-setup/config - adjust as needed.

Fetching configuration repository git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/.bitbake-setup/configurations Configuration in /home/alex/builds/poky-ng/ has not changed.
===

If the configuration has changed, you will see the difference:
===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup status
...
Top level configuration in /home/alex/builds/poky-alex has changed:
--- /home/alex/builds/poky-alex/config/poky-alex.conf.json	2024-12-16 11:43:24.077446096 +0100
+++ /home/alex/builds/poky-alex/config-tmp-asoubw5u/poky-alex.conf.json	2024-12-16 11:47:43.237104405 +0100
@@ -7,7 +7,7 @@
                         "uri": "git://git.yoctoproject.org/poky-contrib"
                     }
                 },
-                "rev": "akanavin/sstate-for-all"
+                "rev": "akanavin/bitbake-setup-testing"
             },
             "path": "poky"
         }
===

If the configuration has not changed, but layer revisions referred to it have (for example if the configuration specifies a tip of a branch), you will see that too:
===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup status
...
Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/sstate-for-all from 6b842ba55f996b27c900e3de78ceac8cb3b1c492 to aeb73e29379fe6007a8adc8d94c1ac18a93e68de
===

6. If the configuration has changed, you can bring it in sync with:
===

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-setup update ~/builds/poky-alex/
Default parameter values are in /home/alex/.bitbake-setup/config - adjust as needed.

Fetching configuration repository git://github.com/kanavin/bitbake-setup-configurations.git;protocol=https;branch=main;rev=main into /home/alex/.bitbake-setup/configurations
Layer repository git://git.yoctoproject.org/poky-contrib checked out into /home/alex/builds/poky-alex/layers/poky updated revision akanavin/bitbake-setup-testing from d174acad934f8ad1fe303abc5705733e15542859 to a3d2ee10045f8c1151d680ad97994c5d6cf51ece Fetching layer/tool repository poky into /home/alex/builds/poky-alex/layers/poky

Setting up configuration in /home/alex/builds/poky-alex/build-gadget
Existing bitbake congfiguration directory renamed to /home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007 The bitbake configuration has changed:

home/alex/builds/poky-alex/build-gadget/conf/local.conf renamed from home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007/local.conf +0 -2
diff -uNr /home/alex/builds/poky-alex/build-gadget/conf-backup.20241216115007/local.conf /home/alex/builds/poky-alex/build-gadget/conf/local.conf
@@ -287,5 +287,3 @@
 # track the version of this file when it was generated. This can safely be ignored if
 # this doesn't mean anything to you.
 CONF_VERSION = "2"
-
-TCLIBC = "musl"

Configuration summary:
This is the default build configuration for the Poky reference distribution.

Additional information is in /home/alex/builds/poky-alex/build-gadget/conf/conf-notes.txt

Run /home/alex/builds/poky-alex/build-gadget/build-targets to execute the default build targets for this configuration. Source the environment using '. /home/alex/builds/poky-alex/build-gadget/init-build-env' to run builds from the command line. The bitbake build configuration (local.conf, bblayers.conf and more) can be found in /home/alex/builds/poky-alex/build-gadget/conf
===

Note that it will also rename/preserve the existing build/conf directory, and print changes in bitbake configuration (diff of content of build/conf/) if that has changed. I can't at the moment think of anything more clever that is also not much more brittle or complex to implement, but open to suggestions.

7. To make it easier to review the code, please also review the data it's operating on:
===

alex@Zen2:/srv/work/alex/bitbake$ ls ~/.bitbake-setup/
cache  configurations  downloads
alex@Zen2:/srv/work/alex/bitbake$ ls ~/builds/poky-alex/
build  config  config-upstream.json  layers
===

Terminology
===========

- 'top directory' means the place under which bitbake-setup reads and writes everything. bitbake-setup makes a promise to not touch anything outside of that, unless otherwise directed to by entries in settings (currently there is one such setting for fetcher downloads for layers and config registries). Top directory can be selected by a command line option, or otherwise assumed to be ~/bitbake-builds/. If BBPATH is in environment (e.g. we are in a bitbake environment), then the top directory is deduced from that and doesn't need to be specified by hand.

- 'settings' means bitbake-setup operational parameters that are global to all builds under a top directory. E.g. the location of configuration registry, or where the bitbake fetcher should place the downloads (DL_DIR setting). Settings are stored in a .conf file in ini format just under the top directory.

- ' build' means a tree structure set up by 'bitbake-setup init', consisting of, at least, a layers checkout, and a set of bitbake builds. It maps 1:1 to the json data it was constructed from, which is called 'configuration'. Configuration files are either standalone files, or are obtained from git repositories called 'config registries', in which case they can be listed with 'bitbake-setup list'. There can be multiple 'builds' under a top directory. Here's an example configuration that showcases this:
https://github.com/kanavin/bitbake-setup-configurations/blob/main/poky-alex.conf.json

- 'bitbake-setup status' will tell if a configuration is in sync with the build it was made from. 'bitbake-setup update' will bring a build in sync with a configuration if needed.

- 'bitbake build' means a particular sub-tree inside a build that bitbake itself operates on, e.g. what is set in BBPATH/BUILDDIR by oe-init-build-env. conf/* in that tree is 'bitbake configuration'. Bitbake configurations are constructed from templates and fragments, with existing mechanisms provided by oe-core. The configuration file format is specified such that other mechanisms to set up a bitbake build can be added; there was a mention of ability to specify local.conf content and a set of layers directly in a configuration. I think that scales poorly compared to templates and fragments, but I made sure alternative ways to configure a bitbake build are possible to add in the future :)

This commit includes a fix by Ryan Eatmon <[email protected]>:
https://github.com/kanavin/bitbake/pull/1

---
 bin/bitbake-setup | 461 ++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 461 insertions(+)
 create mode 100755 bin/bitbake-setup
bin/bitbake-setup new file +461 -0

461 lines not on this page · raw message

Signature · 1 line
2.39.5
#1 lore raw

RFC PATCH 2/2 bitbake-setup: add tests to bitbake-selftest

From: Alexander Kanavin <[email protected]>

Run like this:

alex@Zen2:/srv/work/alex/bitbake$ bin/bitbake-selftest -v bb.tests.setup
test_setup (bb.tests.setup.BitbakeSetupTest.test_setup) ... ok

----------------------------------------------------------------------
Ran 1 test in 9.223s

OK

The test does a basic run-through of init, then status/update on an unchanged configuration, then status/update on a configuration changed via new commits to the test layer, then status/update on configuration changed via the top level json config file.

Note that nothing whatsoever is fetched from the network; the test relies entirely on synthetic data contained inside itself, including minimal stubs for oe-setup-build and bitbake-config-build. This data is used to create temporary git repositories then clone them via local filesystem URIs.

Later on this can be supplemented by an oe-selftest that tests bitbake-setup against real config files in the official configuration repository and real layers, templates and fragments.

---
 bin/bitbake-selftest  |   1 +
 lib/bb/tests/setup.py | 221 ++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 222 insertions(+)
 create mode 100644 lib/bb/tests/setup.py
bin/bitbake-selftest +1 -0
index 1b7a783fd..8c1e6d3e7 100755
@@ -29,6 +29,7 @@ tests = ["bb.tests.codeparser",
          "bb.tests.fetch",
          "bb.tests.parse",
          "bb.tests.runqueue",
+         "bb.tests.setup",
          "bb.tests.siggen",
          "bb.tests.utils",
          "bb.tests.compression",
lib/bb/tests/setup.py new file +221 -0

221 lines not on this page · raw message

Signature · 1 line
2.39.5