Package bitbake-setup for PyPI
cover Package bitbake-setup for PyPI
Adding bitbake-setup to PyPI is helpful to developers who are already comfortable with using pip in their workflow.
This series makes the necessary changes to build a wheel that may be uploaded to PyPI.
The build sequence would be:
git clone https://git.openembedded.org/bitbake
python3 -m venv venv
source venv/bin/activate
python3 -m pip install build
python3 -m build
This produces a wheel (.whl) file in the dist directory. This may be installed using pip.
Testing included "Setting Up the Environment With bitbake-setup" from the Yocto Project manual. As well as testing the ability to build using local configurations.
bitbake-setup init poky-whinlatter.conf.json
For each scenario, a build of the core-image-minimal recipe was built successfully.
Rob Woolley (5):
gitignore: ignore Python packaging files
bitbake-setup: Move into Python module
bitbake-setup: Add wrapper script for bitbake_setup module Add Python packaging files
bitbake-setup: Add the conditional script stanza
.gitignore | 2 + bin/bitbake-setup | 1108 +------------------------------- lib/bitbake_setup/__init__.py | 0 lib/bitbake_setup/__main__.py | 1110 +++++++++++++++++++++++++++++++++ pyproject.toml | 3 + setup.cfg | 40 ++ 6 files changed, 1160 insertions(+), 1103 deletions(-) create mode 100644 lib/bitbake_setup/__init__.py create mode 100755 lib/bitbake_setup/__main__.py create mode 100644 pyproject.toml create mode 100644 setup.cfg
Signature · 1 line
2.49.0
PATCH 1/5 gitignore: ignore Python packaging files
--- .gitignore | 2 ++ 1 file changed, 2 insertions(+)
.gitignore +2 -0
@@ -17,3 +17,5 @@ lib/toaster/contrib/tts/backlog.txt lib/toaster/contrib/tts/log/* lib/toaster/contrib/tts/.cache/* lib/bb/tests/runqueue-tests/bitbake-cookerdaemon.log +dist/ +*.egg-info/
Signature · 1 line
2.49.0
PATCH 2/5 bitbake-setup: Move into Python module
In reply to Rob WoolleyCreate a separate Python module for bitbake_setup to be invoked from a wrapper script.
--- lib/bitbake_setup/__init__.py | 0 bin/bitbake-setup => lib/bitbake_setup/__main__.py | 0 2 files changed, 0 insertions(+), 0 deletions(-) create mode 100644 lib/bitbake_setup/__init__.py rename bin/bitbake-setup => lib/bitbake_setup/__main__.py (100%)
lib/bitbake_setup/__init__.py new file +0 -0
lib/bitbake_setup/__main__.py renamed from bin/bitbake-setup +0 -0
Signature · 1 line
2.49.0
PATCH 3/5 bitbake-setup: Add wrapper script for bitbake_setup module
In reply to Rob WoolleyThis wrapper script is used to support the workflow where bitbake-setup is invoked from the bitbake git repository.
It uses the directory of the script to find the bitbake_setup module present in the same git repository.
--- bin/bitbake-setup | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100755 bin/bitbake-setup
bin/bitbake-setup new file +11 -0
@@ -0,0 +1,11 @@ +#!/usr/bin/env python3 +import os +import sys + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, os.path.join(ROOT,'lib')) + +from bitbake_setup.__main__ import main + +if __name__ == "__main__": + main()
Signature · 1 line
2.49.0
Re: [bitbake-devel] [PATCH 3/5] bitbake-setup: Add wrapper script for bitbake_setup module
Quoted 20 lines · Rob Woolley via lists.openembedded.org
This wrapper script is used to support the workflow where bitbake-setup is invoked from the bitbake git repository.
It uses the directory of the script to find the
bitbake_setupmodule present in the same git repository.Signed-off-by Rob Woolley <[email protected]>--- bin/bitbake-setup | 11 +++++++++++ 1 file changed, 11 insertions(+) create mode 100755 bin/bitbake-setupbin/bitbake-setup new file +1
-0@@ -0,0 +1,11 @@ +#!/usr/bin/env python3
bin/bitbake-setup line 11
Quoted 10 lines
bin/bitbake-setup new file +10
-0+import os +import sys + +ROOT = os.path.dirname(os.path.dirname(os.path.abspath(__file__))) +sys.path.insert(0, os.path.join(ROOT,'lib')) + +from bitbake_setup.__main__ import main + +if __name__ == "__main__": + main()
I'm not sure I understand the rationale for this and the previous patch. Why do we need to convert bitbake-setup into a module, and replace the executable with a wrapper?
Alex
PATCH 4/5 Add Python packaging files
In reply to Rob WoolleyThis enables creating a pip package for PyPI with:
python3 -m build
--- pyproject.toml | 3 +++ setup.cfg | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) create mode 100644 pyproject.toml create mode 100644 setup.cfg
pyproject.toml new file +3 -0
@@ -0,0 +1,3 @@ +[build-system] +requires = ["setuptools>=61.0.0", "wheel"] +build-backend = "setuptools.build_meta"
setup.cfg new file +40 -0
@@ -0,0 +1,40 @@ +[metadata] +name = bitbake-setup +version = 2.16.0a1 +author = OpenEmbedded BitBake Developers +author_email = [email protected] +description = bitbake-setup +long_description = file: README.md +long_description_content_type = text/markdown +url = https://github.com/openembedded/bitbake.git +license = "GPL-2.0-only and MIT" +classifiers = + Programming Language :: Python :: 3 + License :: OSI Approved :: MIT License + License :: OSI Approved :: GNU General Public License v2 (GPLv2) + Operating System :: POSIX :: Linux + +[options] +package_dir = + = lib +packages = find: +include_package_data = True +zip_safe = False +python_requires = >=3.6 +py_modules = codegen + +install_requires = + ; List any non-standard dependencies BitBake needs + ; BitBake is mostly self-contained, but check its documentation + +[options.packages.find] +where = lib +include = + bb* + bs4* + ply + bitbake_setup + +[options.entry_points] +console_scripts = + bitbake-setup = bitbake_setup.__main__:main
Signature · 1 line
2.49.0
Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files
Quoted 1 line · Rob Woolley via lists.openembedded.org
+version = 2.16.0a1
There are other issues in setup.cfg, and the commit title should be:
'bitbake-setup: add pypi packaging', but before that I think we need to consider the release strategy and version scheme for it, and get RP's blessing :)
Here's what there is in terms of tags and branches, and from what I've heard it is not easy to change:
https://git.openembedded.org/bitbake/
I guess, bitbake-setup would only be released from the master branch, at least once when a new yocto release happens, at the point where stable and master branches diverge. Those releases should match the stable version, e.g. whinlatter release would have been 2.16.
Then the question I can't quite answer is, how to version something from the master trunk? E.g. if we want to release from master right now, what version would it have? Note that ideally it needs to contain the short commit id, for traceability.
Alex
Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files
Hi Alex, Rob,
Quoted 18 lines · Alexander Kanavin via lists.openembedded.org
Quoted 1 line · Rob Woolley via lists.openembedded.org
+version = 2.16.0a1
There are other issues in setup.cfg, and the commit title should be:
'bitbake-setup: add pypi packaging', but before that I think we need to consider the release strategy and version scheme for it, and get RP's blessing :)Here's what there is in terms of tags and branches, and from what I've heard it is not easy to change:
https://git.openembedded.org/bitbake/I guess, bitbake-setup would only be released from the master branch, at least once when a new yocto release happens, at the point where stable and master branches diverge. Those releases should match the stable version, e.g. whinlatter release would have been 2.16.
Quoted 5 lines
Then the question I can't quite answer is, how to version something from the master trunk? E.g. if we want to release from master right now, what version would it have? Note that ideally it needs to contain the short commit id, for traceability.
setuptools_scm should be able to handle that, see https://setuptools-scm.readthedocs.io/en/stable/usage/#default-versioning-scheme. I've yet to play with it (will try with https://github.com/yoctoproject/bmaptool/ in the next few weeks).
Note that as far as I could tell this means all versioned files in the repo are part of the sdist by default (but you can exclude them via MANIFEST.in).
The next question is whether we're planning to upload bitbake itself to PyPI in the future? If so, we need to be careful about the layout. Building two packages from the same git repo isn't that straightforward if I remember correctly so if we're moving things around, better anticipate this possibility so we don't have to move everything around again once we want to publish bitbake itself.
Cheers,
Quentin
Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files
Quoted 6 lines · Quentin Schulz via lists.openembedded.org
The next question is whether we're planning to upload bitbake itself to PyPI in the future? If so, we need to be careful about the layout. Building two packages from the same git repo isn't that straightforward if I remember correctly so if we're moving things around, better anticipate this possibility so we don't have to move everything around again once we want to publish bitbake itself.
The answer to publishing bitbake itself is a firm no.
That said, bitbake-setup packaging files still shouldn't be at the top level. There should be a script that moves them into right place before doing the packaging step, or ideally packaging should be pointed to use a sub-directory.
Alex
Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files
In reply to Quentin SchulzQuoted 43 lines · Quentin Schulz via lists.openembedded.org
Quoted 18 lines · Alexander Kanavin via lists.openembedded.org
Quoted 1 line · Rob Woolley via lists.openembedded.org
+version = 2.16.0a1
There are other issues in setup.cfg, and the commit title should be:
'bitbake-setup: add pypi packaging', but before that I think we need to consider the release strategy and version scheme for it, and get RP's blessing :)Here's what there is in terms of tags and branches, and from what I've heard it is not easy to change:
https://git.openembedded.org/bitbake/I guess, bitbake-setup would only be released from the master branch, at least once when a new yocto release happens, at the point where stable and master branches diverge. Those releases should match the stable version, e.g. whinlatter release would have been 2.16.
Quoted 5 lines
Then the question I can't quite answer is, how to version something from the master trunk? E.g. if we want to release from master right now, what version would it have? Note that ideally it needs to contain the short commit id, for traceability.
setuptools_scmshould be able to handle that, seehttps://setuptools-scm.readthedocs.io/en/stable/usage/#default-versioning-scheme.
I've yet to play with it (will try with
https://github.com/yoctoproject/bmaptool/ in the next few weeks).Note that as far as I could tell this means all versioned files in the repo are part of the sdist by default (but you can exclude them via MANIFEST.in).
The next question is whether we're planning to upload bitbake itself to PyPI in the future? If so, we need to be careful about the layout. Building two packages from the same git repo isn't that straightforward if I remember correctly so if we're moving things around, better anticipate this possibility so we don't have to move everything around again once we want to publish bitbake itself.
Quoted 3 lines
Cheers,
Quentin
Quick update on progress:
I have replaced nox and pytest with Python unittest so that the tests can be invoked from bitbake-selftest. The new PyPIPackagingTest still uses package-bitbake-setup.py to copy the files from contrib/pypi to a staging directory. This is needed to reorganize the layout such that pyproject.toml is at the root with the source code in subdirectories.
I experimented with using setuptools_scm to automatically detect the version number for creating the Python package. Using setuptools_scm for determining the version based on git tags conflicts with creating the staging directory. Whether it is outside the git repo (e.g. in TMPDIR) or in a sub-directory (e.g. <git repo root>/packaging_workspace) it fails to detect the latest git tag. In the case of it being in a subdirectory, I need to set root (e.g. root = "..") in the tool.setuptools_scm section of pyproject.toml for it to succeed.
If we want to use setuptools_scm with git for automatic versioning then we will need to use a fixed (temporary) directory in the git repository (eg. ./packaging_workspace). If a developer chooses another location for scratch work the version will fallback to 0.0.0, which seems acceptable to me. I added a test in PyPIPackagingTest to check for this failure case when doing "production" builds.
Are we okay with pinning the temporary staging directory inside the git repository? If so, do you have a preference as to what that directory is called and where it is located?
The setuptools_scm uses git tags rather than branches. We may want to consider updating our tagging strategy if we want to use it to automatically describe the commit for packages built on master. One option we have is to set tag_regex in pyproject.toml to a custom pattern. Thus we could create tags like bitbake-v2.16.0 or bitbake-setup-v2.16.0 to provide a reference point for results like bitbake-setup-2.16.1.dev0+gd2e159d85.d20260211 or something simpler like bitbake-setup-2.16.2.dev1 (see
https://setuptools-scm.readthedocs.io/en/latest/extending/#version-number-construction
)
Regards,
Rob
Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files
Quoted 1 line · Rob Woolley
Are we okay with pinning the temporary staging directory inside the git repository? If so, do you have a preference as to what that directory is called and where it is located?
I am okay and I have no particular preference. As long as it's in .gitignore.
Quoted 1 line
The
setuptools_scmuses git tags rather than branches. We may want to consider updating our tagging strategy if we want to use it to automatically describe the commit for packages built on master. One option we have is to settag_regexin pyproject.toml to a custom pattern. Thus we could create tags like bitbake-v2.16.0 or bitbake-setup-v2.16.0 to provide a reference point for results like bitbake-setup-2.16.1.dev0+gd2e159d85.d20260211 or something simpler like bitbake-setup-2.16.2.dev1 (see https://setuptools-scm.readthedocs.io/en/latest/extending/#version-number-construction)
The tags available in master branch can be see with 'git tag --merged master'. It seems that the two most suitable sets are at the end:
2024-04-scarthgap
2024-10-styhead
2025-04-walnascar
2025-04.1-walnascar
2025-10-whinlatter
...
yocto-5.0
yocto-5.1
yocto-5.2
yocto-5.2.1
yocto-5.3
So we could simply use one of these versions (they match exact release points), and then add a .relN+hash if that can be done automatically. And then there's no need to introduce new tag schemes or extra work for RP. I like the one with the year and release name a bit more, because it's more informative to the users.
Alex
Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files
Hi Alex,
Quoted 23 lines · Alexander Kanavin
Quoted 1 line · Rob Woolley
Are we okay with pinning the temporary staging directory inside the git repository? If so, do you have a preference as to what that directory is called and where it is located?
I am okay and I have no particular preference. As long as it's in .gitignore.
Quoted 1 line
The
setuptools_scmuses git tags rather than branches. We may want to consider updating our tagging strategy if we want to use it to automatically describe the commit for packages built on master. One option we have is to settag_regexin pyproject.toml to a custom pattern. Thus we could create tags like bitbake-v2.16.0 or bitbake-setup-v2.16.0 to provide a reference point for results like bitbake-setup-2.16.1.dev0+gd2e159d85.d20260211 or something simpler like bitbake-setup-2.16.2.dev1 (see https://setuptools-scm.readthedocs.io/en/latest/extending/#version-number-construction)The tags available in master branch can be see with 'git tag --merged master'. It seems that the two most suitable sets are at the end:
2024-04-scarthgap
2024-10-styhead
2025-04-walnascar
2025-04.1-walnascar
2025-10-whinlatter
...
yocto-5.0
yocto-5.1
yocto-5.2
yocto-5.2.1
yocto-5.3
Quoted 6 lines
So we could simply use one of these versions (they match exact release points), and then add a .relN+hash if that can be done automatically. And then there's no need to introduce new tag schemes or extra work for RP. I like the one with the year and release name a bit more, because it's more informative to the users.
Read https://packaging.python.org/en/latest/specifications/version-specifiers/ and https://peps.python.org/pep-0440/.
Specifically:
"""
Local version identifiers SHOULD NOT be used when publishing upstream projects to a public index server """
The local version is whatever's after a +. So this isn't an option.
Also, no yocto- or -<yocto-release> string possible. See https://packaging.python.org/en/latest/specifications/version-specifiers/#public-version-identifiers
2025-04.1 will also not be a valid public version identifier. It needs to be 2025.04.1.
So the only viable option would be 5.3 for example (you can strip yocto- automatically in versioningit for example).
Now the issue for BitBake repo is that we have the initial release tag on the master branch. This means that the tag is present both on master and 2.16 branch for example. This also means that X.Y.Z.devN may point to two different commits, the N-th one after X.Y.Z release in the release branch, or the N-th one in master. This is of course illegal on PyPI so we'll need to separate them.
I'm suggesting to use .devN for master and .postN for release branches. How to make this work with the packaging tool... I don't know.
I see that versioningit has customizable hooks so maybe that could be an answer on doing this manually in some way. See https://versioningit.readthedocs.io/en/stable/configuration.html and https://versioningit.readthedocs.io/en/stable/writing-methods.html
Cheers,
Quentin
Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files
In reply to Rob WoolleyHi Rob,
Quoted 8 lines · Rob Woolley via lists.openembedded.org
This enables creating a pip package for PyPI with:
python3 -m build
Signed-off-by Rob Woolley <[email protected]>--- pyproject.toml | 3 +++ setup.cfg | 40 ++++++++++++++++++++++++++++++++++++++++
Please migrate everything in setup.cfg into pyproject.toml, there's no reason to use an ancient mechanism. (This may require a more recent setuptools)
Quoted 19 lines
2 files changed, 43 insertions(+)create mode 100644 pyproject.toml
create mode 100644 setup.cfgpyproject.toml new file +3
-0@@ -0,0 +1,3 @@ +[build-system] +requires = ["setuptools>=61.0.0", "wheel"] +build-backend = "setuptools.build_meta"setup.cfg new file +0
-0
setup.cfg line 3
Quoted 3 lines
setup.cfg new file +3
-0@@ -0,0 +1,40 @@ +[metadata] +name = bitbake-setup +version = 2.16.0a1
100% we are going to forget updating this field. I'm wondering if we cannot simply use setuptools_scm to have a dynamic version?
Or use the version from lib/bb/__init__.py?
setup.cfg line 10
Quoted 7 lines
setup.cfg new file +7
-0+author = OpenEmbedded BitBake Developers +author_email = [email protected] +description = bitbake-setup +long_description = file: README.md +long_description_content_type = text/markdown +url = https://github.com/openembedded/bitbake.git +license = "GPL-2.0-only and MIT"
Must be uppercase, so AND. c.f. https://spdx.github.io/spdx-spec/v2.2.2/SPDX-license-expressions/#d2-case-sensitivity
setup.cfg line 14
Quoted 4 lines
setup.cfg new file +4
-0+classifiers = + Programming Language :: Python :: 3 + License :: OSI Approved :: MIT License + License :: OSI Approved :: GNU General Public License v2 (GPLv2)
This is deprecated since PEP639, c.f. https://peps.python.org/pep-0639/#deprecate-license-classifiers
setup.cfg line 23
Quoted 9 lines
setup.cfg new file +9
-0+ Operating System :: POSIX :: Linux + +[options] +package_dir = + = lib +packages = find: +include_package_data = True +zip_safe = False +python_requires = >=3.6
3.9.0 as reported in __init__.py.
setup.cfg line 33
Quoted 10 lines
setup.cfg new file +10
-0+py_modules = codegen + +install_requires = + ; List any non-standard dependencies BitBake needs + ; BitBake is mostly self-contained, but check its documentation + +[options.packages.find] +where = lib +include = + bb*
Is this meant to match both lib/bb/ and lib/bblayers/ ?
Why not explicitly list both?
setup.cfg line 34
Quoted 1 line
setup.cfg new file +1
-0+ bs4*
What is this supposed to match except lib/bs4/?
Cheers,
Quentin
Re: [bitbake-devel] [PATCH 4/5] Add Python packaging files
In reply to Rob WoolleyHi,
Quoted 27 lines · Rob Woolley via lists.openembedded.org
This enables creating a pip package for PyPI with:
python3 -m build
Signed-off-by Rob Woolley <[email protected]>--- pyproject.toml | 3 +++ setup.cfg | 40 ++++++++++++++++++++++++++++++++++++++++ 2 files changed, 43 insertions(+) create mode 100644 pyproject.toml create mode 100644 setup.cfgpyproject.toml new file +3
-0@@ -0,0 +1,3 @@ +[build-system] +requires = ["setuptools>=61.0.0", "wheel"] +build-backend = "setuptools.build_meta"setup.cfg new file +0
-0
setup.cfg line 7
Quoted 7 lines
setup.cfg new file +7
-0@@ -0,0 +1,40 @@ +[metadata] +name = bitbake-setup +version = 2.16.0a1 +author = OpenEmbedded BitBake Developers +author_email = [email protected] +description = bitbake-setup +long_description = file: README.md
I don't think we have a README.md file? Perhaps doc/bitbake-user-manual/bitbake-user-manual-environment-setup.rst would be a better candidate here?
setup.cfg line 8
Quoted 1 line
setup.cfg new file +1
-0+long_description_content_type = text/markdown
This would be text/x-rst then.
setup.cfg line 9
Quoted 1 line
setup.cfg new file +1
-0+url = https://github.com/openembedded/bitbake.git
It should probably be https://git.openembedded.org/bitbake/? The github version is supposed to be a mirror only I think.
Antonin
Signature · 3 lines
Antonin Godard, Bootlin Embedded Linux and Kernel engineering https://bootlin.com
PATCH 5/5 bitbake-setup: Add the conditional script stanza
In reply to Rob WoolleyThis ensures that the main function is only executed if the module is executed in the top-level environment.
--- lib/bitbake_setup/__main__.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
lib/bitbake_setup/__main__.py +2 -1
@@ -1106,4 +1106,5 @@ def main(): from argparse import Namespace parser.print_help() -main() +if __name__ == '__main__': + main()
Signature · 1 line
2.49.0
Quoted 10 lines · Rob Woolley via lists.openembedded.org
Adding bitbake-setup to PyPI is helpful to developers who are already comfortable with using pip in their workflow.
This series makes the necessary changes to build a wheel that may be uploaded to PyPI.
The build sequence would be:
git clone https://git.openembedded.org/bitbake
Quoted 7 lines
python3 -m venv venv
source venv/bin/activatepython3 -m pip install build
python3 -m build
This produces a wheel (.whl) file in the dist directory. This may be installed using pip.
There should be a test for producing the .whl, ideally that also installs the resulting local .whl into a temporary location and runs 'bitbake-setup list' from that installation. Is that feasible?
The instructions to produce, publish, and install the .whl should be in a README.
Alex