Skip to contentSkip to conversation

Package bitbake-setup for PyPI

PATCH Superseded Rob Woolley 5 participants 16 messages Jan 21 – Feb 12

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

cover Package bitbake-setup for PyPI

Adding bitbake-setup to PyPI is helpful to developers who are already comfortable with using pip in their workflow.

This series makes the necessary changes to build a wheel that may be uploaded to PyPI.

The build sequence would be:
git clone https://git.openembedded.org/bitbake

python3 -m venv venv
source venv/bin/activate

python3 -m pip install build
python3 -m build

This produces a wheel (.whl) file in the dist directory. This may be installed using pip.

Testing included "Setting Up the Environment With bitbake-setup" from the Yocto Project manual. As well as testing the ability to build using local configurations.

bitbake-setup init poky-whinlatter.conf.json

For each scenario, a build of the core-image-minimal recipe was built successfully.

Rob Woolley (5):
gitignore: ignore Python packaging files
bitbake-setup: Move into Python module
bitbake-setup: Add wrapper script for bitbake_setup module Add Python packaging files
bitbake-setup: Add the conditional script stanza

 .gitignore                    |    2 +
 bin/bitbake-setup             | 1108 +-------------------------------
 lib/bitbake_setup/__init__.py |    0
 lib/bitbake_setup/__main__.py | 1110 +++++++++++++++++++++++++++++++++
 pyproject.toml                |    3 +
 setup.cfg                     |   40 ++
 6 files changed, 1160 insertions(+), 1103 deletions(-)
 create mode 100644 lib/bitbake_setup/__init__.py
 create mode 100755 lib/bitbake_setup/__main__.py
 create mode 100644 pyproject.toml
 create mode 100644 setup.cfg
Signature · 1 line
2.49.0
#1 lore raw

PATCH 1/5 gitignore: ignore Python packaging files

---
 .gitignore | 2 ++
 1 file changed, 2 insertions(+)
.gitignore +2 -0
index a6a256b2..0d8a8e23 100644
@@ -17,3 +17,5 @@ lib/toaster/contrib/tts/backlog.txt
 lib/toaster/contrib/tts/log/*
 lib/toaster/contrib/tts/.cache/*
 lib/bb/tests/runqueue-tests/bitbake-cookerdaemon.log
+dist/
+*.egg-info/
Signature · 1 line
2.49.0
#2 lore raw

PATCH 2/5 bitbake-setup: Move into Python module

In reply to Rob Woolley

Create a separate Python module for bitbake_setup to be invoked from a wrapper script.

---
 lib/bitbake_setup/__init__.py                      | 0
 bin/bitbake-setup => lib/bitbake_setup/__main__.py | 0
 2 files changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 lib/bitbake_setup/__init__.py
 rename bin/bitbake-setup => lib/bitbake_setup/__main__.py (100%)
lib/bitbake_setup/__init__.py new file +0 -0
new file mode 100644
index 00000000..e69de29b
lib/bitbake_setup/__main__.py renamed from bin/bitbake-setup +0 -0
similarity index 100%
rename from bin/bitbake-setup
rename to lib/bitbake_setup/__main__.py
Signature · 1 line
2.49.0
#3 lore raw

PATCH 3/5 bitbake-setup: Add wrapper script for bitbake_setup module

In reply to Rob Woolley

This wrapper script is used to support the workflow where bitbake-setup is invoked from the bitbake git repository.

It uses the directory of the script to find the bitbake_setup module present in the same git repository.

---
 bin/bitbake-setup | 11 +++++++++++
 1 file changed, 11 insertions(+)
 create mode 100755 bin/bitbake-setup
bin/bitbake-setup new file +11 -0
new file mode 100755
index 00000000..e1b1cdd1
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3
+import os
+import sys
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, os.path.join(ROOT,'lib'))
+
+from bitbake_setup.__main__ import main
+
+if __name__ == "__main__":
+    main()
Signature · 1 line
2.49.0
#4 lore raw

Re: [bitbake-devel] [PATCH 3/5] bitbake-setup: Add wrapper script for bitbake_setup module

Quoted 20 lines · Rob Woolley via lists.openembedded.org

This wrapper script is used to support the workflow where bitbake-setup is invoked from the bitbake git repository.

It uses the directory of the script to find the bitbake_setup module present in the same git repository.

---
 bin/bitbake-setup | 11 +++++++++++
 1 file changed, 11 insertions(+)
 create mode 100755 bin/bitbake-setup
bin/bitbake-setup new file +1 -0
new file mode 100755
index 00000000..e1b1cdd1
@@ -0,0 +1,11 @@
+#!/usr/bin/env python3

bin/bitbake-setup line 11

Quoted 10 lines
bin/bitbake-setup new file +10 -0
+import os
+import sys
+
+ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+sys.path.insert(0, os.path.join(ROOT,'lib'))
+
+from bitbake_setup.__main__ import main
+
+if __name__ == "__main__":
+    main()

I'm not sure I understand the rationale for this and the previous patch. Why do we need to convert bitbake-setup into a module, and replace the executable with a wrapper?

Alex

#5 lore raw

PATCH 4/5 Add Python packaging files

In reply to Rob Woolley

This enables creating a pip package for PyPI with:

python3 -m build

---
 pyproject.toml |  3 +++
 setup.cfg      | 40 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+)
 create mode 100644 pyproject.toml
 create mode 100644 setup.cfg
pyproject.toml new file +3 -0
new file mode 100644
index 00000000..4aff98bb
@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools>=61.0.0", "wheel"]
+build-backend = "setuptools.build_meta"
setup.cfg new file +40 -0
new file mode 100644
index 00000000..3eaa3c49
@@ -0,0 +1,40 @@
+[metadata]
+name = bitbake-setup
+version = 2.16.0a1
+author = OpenEmbedded BitBake Developers
+author_email = [email protected]
+description = bitbake-setup
+long_description = file: README.md
+long_description_content_type = text/markdown
+url = https://github.com/openembedded/bitbake.git
+license = "GPL-2.0-only and MIT"
+classifiers =
+    Programming Language :: Python :: 3
+    License :: OSI Approved :: MIT License
+    License :: OSI Approved :: GNU General Public License v2 (GPLv2)
+    Operating System :: POSIX :: Linux
+
+[options]
+package_dir =
+    = lib
+packages = find:
+include_package_data = True
+zip_safe = False
+python_requires = >=3.6
+py_modules = codegen
+
+install_requires =
+    ; List any non-standard dependencies BitBake needs
+    ; BitBake is mostly self-contained, but check its documentation
+
+[options.packages.find]
+where = lib
+include =
+    bb*
+    bs4*
+    ply
+    bitbake_setup
+
+[options.entry_points]
+console_scripts =
+    bitbake-setup = bitbake_setup.__main__:main
Signature · 1 line
2.49.0
#6 lore raw

Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files

Quoted 1 line · Rob Woolley via lists.openembedded.org

+version = 2.16.0a1

There are other issues in setup.cfg, and the commit title should be:
'bitbake-setup: add pypi packaging', but before that I think we need to consider the release strategy and version scheme for it, and get RP's blessing :)

Here's what there is in terms of tags and branches, and from what I've heard it is not easy to change:
https://git.openembedded.org/bitbake/

I guess, bitbake-setup would only be released from the master branch, at least once when a new yocto release happens, at the point where stable and master branches diverge. Those releases should match the stable version, e.g. whinlatter release would have been 2.16.

Then the question I can't quite answer is, how to version something from the master trunk? E.g. if we want to release from master right now, what version would it have? Note that ideally it needs to contain the short commit id, for traceability.

Alex

#7 lore raw

Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files

Hi Alex, Rob,

Quoted 18 lines · Alexander Kanavin via lists.openembedded.org
Quoted 1 line · Rob Woolley via lists.openembedded.org

+version = 2.16.0a1

There are other issues in setup.cfg, and the commit title should be:
'bitbake-setup: add pypi packaging', but before that I think we need to consider the release strategy and version scheme for it, and get RP's blessing :)

Here's what there is in terms of tags and branches, and from what I've heard it is not easy to change:
https://git.openembedded.org/bitbake/

I guess, bitbake-setup would only be released from the master branch, at least once when a new yocto release happens, at the point where stable and master branches diverge. Those releases should match the stable version, e.g. whinlatter release would have been 2.16.

Quoted 5 lines

Then the question I can't quite answer is, how to version something from the master trunk? E.g. if we want to release from master right now, what version would it have? Note that ideally it needs to contain the short commit id, for traceability.

setuptools_scm should be able to handle that, see https://setuptools-scm.readthedocs.io/en/stable/usage/#default-versioning-scheme. I've yet to play with it (will try with https://github.com/yoctoproject/bmaptool/ in the next few weeks).

Note that as far as I could tell this means all versioned files in the repo are part of the sdist by default (but you can exclude them via MANIFEST.in).

The next question is whether we're planning to upload bitbake itself to PyPI in the future? If so, we need to be careful about the layout. Building two packages from the same git repo isn't that straightforward if I remember correctly so if we're moving things around, better anticipate this possibility so we don't have to move everything around again once we want to publish bitbake itself.

Cheers,
Quentin

#8 lore raw

Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files

Quoted 6 lines · Quentin Schulz via lists.openembedded.org

The next question is whether we're planning to upload bitbake itself to PyPI in the future? If so, we need to be careful about the layout. Building two packages from the same git repo isn't that straightforward if I remember correctly so if we're moving things around, better anticipate this possibility so we don't have to move everything around again once we want to publish bitbake itself.

The answer to publishing bitbake itself is a firm no.

That said, bitbake-setup packaging files still shouldn't be at the top level. There should be a script that moves them into right place before doing the packaging step, or ideally packaging should be pointed to use a sub-directory.

Alex

#9 lore raw

Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files

In reply to Quentin Schulz
Quoted 43 lines · Quentin Schulz via lists.openembedded.org
Quoted 18 lines · Alexander Kanavin via lists.openembedded.org
Quoted 1 line · Rob Woolley via lists.openembedded.org

+version = 2.16.0a1

There are other issues in setup.cfg, and the commit title should be:
'bitbake-setup: add pypi packaging', but before that I think we need to consider the release strategy and version scheme for it, and get RP's blessing :)

Here's what there is in terms of tags and branches, and from what I've heard it is not easy to change:
https://git.openembedded.org/bitbake/

I guess, bitbake-setup would only be released from the master branch, at least once when a new yocto release happens, at the point where stable and master branches diverge. Those releases should match the stable version, e.g. whinlatter release would have been 2.16.

Quoted 5 lines

Then the question I can't quite answer is, how to version something from the master trunk? E.g. if we want to release from master right now, what version would it have? Note that ideally it needs to contain the short commit id, for traceability.

setuptools_scm should be able to handle that, see

https://setuptools-scm.readthedocs.io/en/stable/usage/#default-versioning-scheme.

I've yet to play with it (will try with
https://github.com/yoctoproject/bmaptool/ in the next few weeks).

Note that as far as I could tell this means all versioned files in the repo are part of the sdist by default (but you can exclude them via MANIFEST.in).

The next question is whether we're planning to upload bitbake itself to PyPI in the future? If so, we need to be careful about the layout. Building two packages from the same git repo isn't that straightforward if I remember correctly so if we're moving things around, better anticipate this possibility so we don't have to move everything around again once we want to publish bitbake itself.

Quoted 3 lines

Cheers,
Quentin

Quick update on progress:

I have replaced nox and pytest with Python unittest so that the tests can be invoked from bitbake-selftest. The new PyPIPackagingTest still uses package-bitbake-setup.py to copy the files from contrib/pypi to a staging directory. This is needed to reorganize the layout such that pyproject.toml is at the root with the source code in subdirectories.

I experimented with using setuptools_scm to automatically detect the version number for creating the Python package. Using setuptools_scm for determining the version based on git tags conflicts with creating the staging directory. Whether it is outside the git repo (e.g. in TMPDIR) or in a sub-directory (e.g. <git repo root>/packaging_workspace) it fails to detect the latest git tag. In the case of it being in a subdirectory, I need to set root (e.g. root = "..") in the tool.setuptools_scm section of pyproject.toml for it to succeed.

If we want to use setuptools_scm with git for automatic versioning then we will need to use a fixed (temporary) directory in the git repository (eg. ./packaging_workspace). If a developer chooses another location for scratch work the version will fallback to 0.0.0, which seems acceptable to me. I added a test in PyPIPackagingTest to check for this failure case when doing "production" builds.

Are we okay with pinning the temporary staging directory inside the git repository? If so, do you have a preference as to what that directory is called and where it is located?

The setuptools_scm uses git tags rather than branches. We may want to consider updating our tagging strategy if we want to use it to automatically describe the commit for packages built on master. One option we have is to set tag_regex in pyproject.toml to a custom pattern. Thus we could create tags like bitbake-v2.16.0 or bitbake-setup-v2.16.0 to provide a reference point for results like bitbake-setup-2.16.1.dev0+gd2e159d85.d20260211 or something simpler like bitbake-setup-2.16.2.dev1 (see
https://setuptools-scm.readthedocs.io/en/latest/extending/#version-number-construction
)

Regards,
Rob

#10 lore raw

Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files

Quoted 1 line · Rob Woolley

Are we okay with pinning the temporary staging directory inside the git repository? If so, do you have a preference as to what that directory is called and where it is located?

I am okay and I have no particular preference. As long as it's in .gitignore.

Quoted 1 line

The setuptools_scm uses git tags rather than branches. We may want to consider updating our tagging strategy if we want to use it to automatically describe the commit for packages built on master. One option we have is to set tag_regex in pyproject.toml to a custom pattern. Thus we could create tags like bitbake-v2.16.0 or bitbake-setup-v2.16.0 to provide a reference point for results like bitbake-setup-2.16.1.dev0+gd2e159d85.d20260211 or something simpler like bitbake-setup-2.16.2.dev1 (see https://setuptools-scm.readthedocs.io/en/latest/extending/#version-number-construction)

The tags available in master branch can be see with 'git tag --merged master'. It seems that the two most suitable sets are at the end:

2024-04-scarthgap
2024-10-styhead
2025-04-walnascar
2025-04.1-walnascar
2025-10-whinlatter
...
yocto-5.0
yocto-5.1
yocto-5.2
yocto-5.2.1
yocto-5.3

So we could simply use one of these versions (they match exact release points), and then add a .relN+hash if that can be done automatically. And then there's no need to introduce new tag schemes or extra work for RP. I like the one with the year and release name a bit more, because it's more informative to the users.

Alex

#11 lore raw

Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files

Hi Alex,

Quoted 23 lines · Alexander Kanavin
Quoted 1 line · Rob Woolley

Are we okay with pinning the temporary staging directory inside the git repository? If so, do you have a preference as to what that directory is called and where it is located?

I am okay and I have no particular preference. As long as it's in .gitignore.

Quoted 1 line

The setuptools_scm uses git tags rather than branches. We may want to consider updating our tagging strategy if we want to use it to automatically describe the commit for packages built on master. One option we have is to set tag_regex in pyproject.toml to a custom pattern. Thus we could create tags like bitbake-v2.16.0 or bitbake-setup-v2.16.0 to provide a reference point for results like bitbake-setup-2.16.1.dev0+gd2e159d85.d20260211 or something simpler like bitbake-setup-2.16.2.dev1 (see https://setuptools-scm.readthedocs.io/en/latest/extending/#version-number-construction)

The tags available in master branch can be see with 'git tag --merged master'. It seems that the two most suitable sets are at the end:

2024-04-scarthgap
2024-10-styhead
2025-04-walnascar
2025-04.1-walnascar
2025-10-whinlatter
...
yocto-5.0
yocto-5.1
yocto-5.2
yocto-5.2.1
yocto-5.3

Quoted 6 lines

So we could simply use one of these versions (they match exact release points), and then add a .relN+hash if that can be done automatically. And then there's no need to introduce new tag schemes or extra work for RP. I like the one with the year and release name a bit more, because it's more informative to the users.

Read https://packaging.python.org/en/latest/specifications/version-specifiers/ and https://peps.python.org/pep-0440/.

Specifically:

"""
Local version identifiers SHOULD NOT be used when publishing upstream projects to a public index server """

The local version is whatever's after a +. So this isn't an option.

Also, no yocto- or -<yocto-release> string possible. See https://packaging.python.org/en/latest/specifications/version-specifiers/#public-version-identifiers

2025-04.1 will also not be a valid public version identifier. It needs to be 2025.04.1.

So the only viable option would be 5.3 for example (you can strip yocto- automatically in versioningit for example).

Now the issue for BitBake repo is that we have the initial release tag on the master branch. This means that the tag is present both on master and 2.16 branch for example. This also means that X.Y.Z.devN may point to two different commits, the N-th one after X.Y.Z release in the release branch, or the N-th one in master. This is of course illegal on PyPI so we'll need to separate them.

I'm suggesting to use .devN for master and .postN for release branches. How to make this work with the packaging tool... I don't know.
I see that versioningit has customizable hooks so maybe that could be an answer on doing this manually in some way. See https://versioningit.readthedocs.io/en/stable/configuration.html and https://versioningit.readthedocs.io/en/stable/writing-methods.html

Cheers,
Quentin

#12 lore raw

Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files

In reply to Rob Woolley

Hi Rob,

Quoted 8 lines · Rob Woolley via lists.openembedded.org

This enables creating a pip package for PyPI with:

python3 -m build

---
  pyproject.toml |  3 +++
  setup.cfg      | 40 ++++++++++++++++++++++++++++++++++++++++

Please migrate everything in setup.cfg into pyproject.toml, there's no reason to use an ancient mechanism. (This may require a more recent setuptools)

Quoted 19 lines
  2 files changed, 43 insertions(+)

create mode 100644 pyproject.toml
create mode 100644 setup.cfg

pyproject.toml new file +3 -0
new file mode 100644
index 00000000..4aff98bb
@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools>=61.0.0", "wheel"]
+build-backend = "setuptools.build_meta"
setup.cfg new file +0 -0
new file mode 100644
index 00000000..3eaa3c49

setup.cfg line 3

Quoted 3 lines
setup.cfg new file +3 -0
@@ -0,0 +1,40 @@
+[metadata]
+name = bitbake-setup
+version = 2.16.0a1

100% we are going to forget updating this field. I'm wondering if we cannot simply use setuptools_scm to have a dynamic version?

Or use the version from lib/bb/__init__.py?

setup.cfg line 10

Quoted 7 lines
setup.cfg new file +7 -0
+author = OpenEmbedded BitBake Developers
+author_email = [email protected]
+description = bitbake-setup
+long_description = file: README.md
+long_description_content_type = text/markdown
+url = https://github.com/openembedded/bitbake.git
+license = "GPL-2.0-only and MIT"

Must be uppercase, so AND. c.f. https://spdx.github.io/spdx-spec/v2.2.2/SPDX-license-expressions/#d2-case-sensitivity

setup.cfg line 14

Quoted 4 lines
setup.cfg new file +4 -0
+classifiers =
+    Programming Language :: Python :: 3
+    License :: OSI Approved :: MIT License
+    License :: OSI Approved :: GNU General Public License v2 (GPLv2)

This is deprecated since PEP639, c.f. https://peps.python.org/pep-0639/#deprecate-license-classifiers

setup.cfg line 23

Quoted 9 lines
setup.cfg new file +9 -0
+    Operating System :: POSIX :: Linux
+
+[options]
+package_dir =
+    = lib
+packages = find:
+include_package_data = True
+zip_safe = False
+python_requires = >=3.6

3.9.0 as reported in __init__.py.

setup.cfg line 33

Quoted 10 lines
setup.cfg new file +10 -0
+py_modules = codegen
+
+install_requires =
+    ; List any non-standard dependencies BitBake needs
+    ; BitBake is mostly self-contained, but check its documentation
+
+[options.packages.find]
+where = lib
+include =
+    bb*

Is this meant to match both lib/bb/ and lib/bblayers/ ?

Why not explicitly list both?

setup.cfg line 34

Quoted 1 line
setup.cfg new file +1 -0
+    bs4*

What is this supposed to match except lib/bs4/?

Cheers,
Quentin

#13 lore raw

Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files

In reply to Rob Woolley

Hi,

Quoted 27 lines · Rob Woolley via lists.openembedded.org

This enables creating a pip package for PyPI with:

python3 -m build

---
 pyproject.toml |  3 +++
 setup.cfg      | 40 ++++++++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+)
 create mode 100644 pyproject.toml
 create mode 100644 setup.cfg
pyproject.toml new file +3 -0
new file mode 100644
index 00000000..4aff98bb
@@ -0,0 +1,3 @@
+[build-system]
+requires = ["setuptools>=61.0.0", "wheel"]
+build-backend = "setuptools.build_meta"
setup.cfg new file +0 -0
new file mode 100644
index 00000000..3eaa3c49

setup.cfg line 7

Quoted 7 lines
setup.cfg new file +7 -0
@@ -0,0 +1,40 @@
+[metadata]
+name = bitbake-setup
+version = 2.16.0a1
+author = OpenEmbedded BitBake Developers
+author_email = [email protected]
+description = bitbake-setup
+long_description = file: README.md

I don't think we have a README.md file? Perhaps doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst would be a better candidate here?

setup.cfg line 8

Quoted 1 line
setup.cfg new file +1 -0
+long_description_content_type = text/markdown

This would be text/x-rst then.

setup.cfg line 9

Quoted 1 line
setup.cfg new file +1 -0
+url = https://github.com/openembedded/bitbake.git

It should probably be https://git.openembedded.org/bitbake/? The github version is supposed to be a mirror only I think.

Antonin

Signature · 3 lines
Antonin Godard, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#14 lore raw

PATCH 5/5 bitbake-setup: Add the conditional script stanza

In reply to Rob Woolley

This ensures that the main function is only executed if the module is executed in the top-level environment.

---
 lib/bitbake_setup/__main__.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
lib/bitbake_setup/__main__.py +2 -1
index a69dc750..d23582dd 100755
@@ -1106,4 +1106,5 @@ def main():
         from argparse import Namespace
         parser.print_help()
 
-main()
+if __name__ == '__main__':
+    main()
Signature · 1 line
2.49.0
#15 lore raw
In reply to Rob Woolley
Quoted 10 lines · Rob Woolley via lists.openembedded.org

Adding bitbake-setup to PyPI is helpful to developers who are already comfortable with using pip in their workflow.

This series makes the necessary changes to build a wheel that may be uploaded to PyPI.

The build sequence would be:
git clone https://git.openembedded.org/bitbake

Quoted 7 lines

python3 -m venv venv
source venv/bin/activate

python3 -m pip install build
python3 -m build
This produces a wheel (.whl) file in the dist directory. This may be installed using pip.

There should be a test for producing the .whl, ideally that also installs the resulting local .whl into a temporary location and runs 'bitbake-setup list' from that installation. Is that feasible?

The instructions to produce, publish, and install the .whl should be in a README.

Alex