package/python-pykwalify: add required package for zephyr cmake
PATCH 1/1 package/python-pykwalify: add required package for zephyr cmake
When building zephyr based firmware, the cmake integrates with python scripts that can be run using the buildroot host-python dependencies. This is useful for the sdk setups and associated build steps for the firmware. In this use-case, a buildroot external project can be setup with recipes to build firmware images using eabi toolchains and the linux controller software to talk to those firmware images using one SDK. This might not be overtly obvious from the patches.
Add the required host packages and additional pykwalify
--- package/Config.in | 1 + package/python-dateutil/python-dateutil.mk | 2 ++ package/python-docopt/python-docopt.mk | 1 + package/python-pykwalify/Config.in | 9 ++++++++ .../python-pykwalify/python-pykwalify.hash | 6 +++++ package/python-pykwalify/python-pykwalify.mk | 23 +++++++++++++++++++ .../python-ruamel-yaml/python-ruamel-yaml.mk | 1 + 7 files changed, 43 insertions(+) create mode 100644 package/python-pykwalify/Config.in create mode 100644 package/python-pykwalify/python-pykwalify.hash create mode 100644 package/python-pykwalify/python-pykwalify.mk
package/Config.in +1 -0
@@ -1225,6 +1225,7 @@ menu "External python modules" source "package/python-pyicu/Config.in" source "package/python-pyinotify/Config.in" source "package/python-pyjwt/Config.in" + source "package/python-pykwalify/Config.in" source "package/python-pylibfdt/Config.in" source "package/python-pylibftdi/Config.in" source "package/python-pylru/Config.in"
package/python-dateutil/python-dateutil.mk +2 -0
@@ -10,5 +10,7 @@ PYTHON_DATEUTIL_SETUP_TYPE = setuptools PYTHON_DATEUTIL_LICENSE = BSD-3-Clause PYTHON_DATEUTIL_LICENSE_FILES = LICENSE PYTHON_DATEUTIL_DEPENDENCIES = host-python-setuptools-scm +HOST_PYTHON_DATEUTIL_DEPENDENCIES = host-python-setuptools-scm host-python-six $(eval $(python-package)) +$(eval $(host-python-package))
package/python-docopt/python-docopt.mk +1 -0
@@ -11,3 +11,4 @@ PYTHON_DOCOPT_LICENSE_FILES = LICENSE-MIT PYTHON_DOCOPT_SETUP_TYPE = setuptools $(eval $(python-package)) +$(eval $(host-python-package))
package/python-pykwalify/Config.in new file +9 -0
@@ -0,0 +1,9 @@ +config BR2_PACKAGE_PYTHON_PYKWALIFY + bool "python-pykwalify" + select BR2_PACKAGE_PYTHON_DATEUTIL # runtime + select BR2_PACKAGE_PYTHON_DOCOPT # runtime + select BR2_PACKAGE_PYTHON_RUAMEL_YAML # runtime + help + Python lib/cli for JSON/YAML schema validation. + + http://github.com/grokzen/pykwalify
package/python-pykwalify/python-pykwalify.hash new file +6 -0
@@ -0,0 +1,6 @@ +# md5, sha256 from https://pypi.org/pypi/pykwalify/json +md5 153197598f8508a49aefc0f44156c2b8 pykwalify-1.8.0.tar.gz +sha256 796b2ad3ed4cb99b88308b533fb2f559c30fa6efb4fa9fda11347f483d245884 pykwalify-1.8.0.tar.gz +# Locally computed sha256 checksums +sha256 626fafd952ab0345edc227f2d565835fe1fd14067ae0d635ee3e47289b773dc4 LICENSE +sha256 f89f247ddd2b9fdff655668148da4b6feaa9175fa44dd244d233006d0f551cd5 docs/license.rst
package/python-pykwalify/python-pykwalify.mk new file +23 -0
@@ -0,0 +1,23 @@ +################################################################################ +# +# python-pykwalify +# +################################################################################ + +PYTHON_PYKWALIFY_VERSION = 1.8.0 +PYTHON_PYKWALIFY_SOURCE = pykwalify-$(PYTHON_PYKWALIFY_VERSION).tar.gz +PYTHON_PYKWALIFY_SITE = https://files.pythonhosted.org/packages/d5/77/2d6849510dbfce5f74f1f69768763630ad0385ad7bb0a4f39b55de3920c7 +PYTHON_PYKWALIFY_SETUP_TYPE = setuptools +PYTHON_PYKWALIFY_LICENSE = MIT +PYTHON_PYKWALIFY_LICENSE_FILES = LICENSE docs/license.rst +PYTHON_PYKWALIFY_DEPENDENCIES = \ + python-dateutil \ + python-docopt \ + python-ruamel-yaml +HOST_PYTHON_PYKWALIFY_DEPENDENCIES = \ + host-python-dateutil \ + host-python-docopt \ + host-python-ruamel-yaml + +$(eval $(python-package)) +$(eval $(host-python-package))
package/python-ruamel-yaml/python-ruamel-yaml.mk +1 -0
@@ -14,3 +14,4 @@ PYTHON_RUAMEL_YAML_CPE_ID_VENDOR = ruamel.yaml_project PYTHON_RUAMEL_YAML_CPE_ID_PRODUCT = ruamel.yaml $(eval $(python-package)) +$(eval $(host-python-package))
Signature · 6 lines
2.39.2 (Apple Git-143) _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Hello Charles,
Quoted 8 lines · On Mon, 21 Aug 2023 10:58:28 -0700 Charles Hardin <[email protected]> wrote:
When building zephyr based firmware, the cmake integrates with python scripts that can be run using the buildroot host-python dependencies. This is useful for the sdk setups and associated build steps for the firmware. In this use-case, a buildroot external project can be setup with recipes to build firmware images using eabi toolchains and the linux controller software to talk to those firmware images using one SDK. This might not be overtly obvious from the patches.
Could you clarify a bit what you call "zephyr cmake" and the use case for this?
Quoted 1 line
Add the required host packages and additional pykwalify
This needs to be split into multiple patches, one per package affected. But I think we also need to understand the bigger picture of what this is useful for.
Thanks!
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
inline
Quoted 14 lines · Thomas Petazzoni
Hello Charles,
Quoted 8 lines · On Mon, 21 Aug 2023 10:58:28 -0700 Charles Hardin <[email protected]> wrote:
When building zephyr based firmware, the cmake integrates with python scripts that can be run using the buildroot host-python dependencies. This is useful for the sdk setups and associated build steps for the firmware. In this use-case, a buildroot external project can be setup with recipes to build firmware images using eabi toolchains and the linux controller software to talk to those firmware images using one SDK. This might not be overtly obvious from the patches.
Quoted 3 lines
Could you clarify a bit what you call "zephyr cmake" and the use case for this?
https://github.com/zephyrproject-rtos
The build is cmake derived but relies on calling into some python scripts to generate some build dependencies
https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/requirements-base.txt
Quoted 7 lines
Quoted 2 lines
Add the required host packages and additional pykwalify
This needs to be split into multiple patches, one per package affected. But I think we also need to understand the bigger picture of what this is useful for.
So, we have various RTOS MCUs that are hooked up into linux controllers, and this is just a simple hook so that the same SDK can be handled to build both firmware for something like an STM32 and a Raspberry-pi
Assume a reference to
https://gist.github.com/titouanc/ea0685d9cd8592deb1c49d48e33b3eee to add the gcc eabi toolchain has been done in an external package, with the patches above we can actually build both firmware and the drivers in buildroot from one tree with external recipes like below
env PATH=/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/bin:$PATH cmake \
-DCMAKE_TOOLCHAIN_FILE=/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/share/buildroot/toolchainfile.cmake
\
-DARMGNU_GCC_TOOLCHAIN_PREFIX="/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/share/gcc-arm-none-eabi" \
-DCMAKE_BUILD_TYPE=Debug \
../buildroot-externalmake VERBOSE=1 -j$(nproc)
Quoted 9 lines
Thanks!
Thomas
--
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Hello Charles,
Quoted 1 line · On Mon, 21 Aug 2023 16:45:23 -0700 Charles Hardin <[email protected]> wrote:
Yes, I know what Zephyr is :-)
Quoted 4 lines
The build is cmake derived but relies on calling into some python scripts to generate some build dependencies
https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/requirements-base.txt
OK.
Quoted 3 lines
So, we have various RTOS MCUs that are hooked up into linux controllers, and this is just a simple hook so that the same SDK can be handled to build both firmware for something like an STM32 and a Raspberry-pi
If I rephrase this: you want to use Buildroot to build the Zephyr firmware for your STM32 micro-controller, alongside with the Linux system running on the RaspberryPi?
Quoted 5 lines
Assume a reference to
https://gist.github.com/titouanc/ea0685d9cd8592deb1c49d48e33b3eee to add the gcc eabi toolchain has been done in an external package, with the patches above we can actually build both firmware and the drivers in buildroot from one tree with external recipes like below
Note: this gcc eabi toolchain is now in Buildroot upstream, no need to add it as an external package.
Quoted 7 lines
env PATH=/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/bin:$PATH cmake \
-DCMAKE_TOOLCHAIN_FILE=/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/share/buildroot/toolchainfile.cmake
\
Quoted 6 lines
-DARMGNU_GCC_TOOLCHAIN_PREFIX="/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/share/gcc-arm-none-eabi" \
-DCMAKE_BUILD_TYPE=Debug \ ../buildroot-external
Not sure to really follow in which projects this cmake command is used? To build Zephyr?
Best regards,
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Thomas,
Quoted 18 lines · Thomas Petazzoni
Hello Charles,
Quoted 4 lines
On Mon, 21 Aug 2023 16:45:23 -0700
Charles Hardin <[email protected]> wrote:Yes, I know what Zephyr is :-)
Quoted 4 lines
The build is cmake derived but relies on calling into some python scripts to generate some build dependencies
https://github.com/zephyrproject-rtos/zephyr/blob/main/scripts/requirements-base.txt
OK.
Quoted 8 lines
Quoted 3 lines
So, we have various RTOS MCUs that are hooked up into linux controllers, and this is just a simple hook so that the same SDK can be handled to build both firmware for something like an STM32 and a Raspberry-pi
If I rephrase this: you want to use Buildroot to build the Zephyr firmware for your STM32 micro-controller, alongside with the Linux system running on the RaspberryPi?
Yes, and we are doing this today with these patches.
Quoted 9 lines
Quoted 5 lines
Assume a reference to
https://gist.github.com/titouanc/ea0685d9cd8592deb1c49d48e33b3eee to add the gcc eabi toolchain has been done in an external package, with the patches above we can actually build both firmware and the drivers in buildroot from one tree with external recipes like belowNote: this gcc eabi toolchain is now in Buildroot upstream, no need to add it as an external package.
Thanks
Quoted 10 lines
Quoted 10 lines
env PATH=/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/bin:$PATH cmake \
-DCMAKE_TOOLCHAIN_FILE=/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/share/buildroot/toolchainfile.cmake
\-DARMGNU_GCC_TOOLCHAIN_PREFIX="/opt/sdk/aarch64-buildroot-linux-gnu_sdk-buildroot/share/gcc-arm-none-eabi" \
Quoted 7 lines
Quoted 3 lines
-DCMAKE_BUILD_TYPE=Debug \../buildroot-external
Not sure to really follow in which projects this cmake command is used? To build Zephyr?
There is a custom cmake file to do the zephyr build, so make does the buildroot build and we can use cmake for the zephyr build.
It is not out of the box but pretty simple glue to make it work.
Quoted 7 lines
Best regards,
Thomas
Signature · 3 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 8 lines · Charles Hardin
Thomas,
Quoted 6 lines · Thomas Petazzoni
Hello Charles,
Quoted 2 lines
On Mon, 21 Aug 2023 16:45:23 -0700
Charles Hardin <[email protected]> wrote:
[snip]
Quoted 10 lines
Quoted 8 lines
Quoted 3 lines
So, we have various RTOS MCUs that are hooked up into linux controllers, and this is just a simple hook so that the same SDK can be handled to build both firmware for something like an STM32 and a Raspberry-pi
If I rephrase this: you want to use Buildroot to build the Zephyr firmware for your STM32 micro-controller, alongside with the Linux system running on the RaspberryPi?
Yes, and we are doing this today with these patches.
Honestly, I don't know if we want to go in this direction. If Zephyr doesn't
give you the possibility to do "from scratch" builds, then in this day and age it's more appropriate to create a container image for your build environment IMHO. There's something to be said for including host packages that are likely to be used in post-build/post-image scripts, but for building a completely different OS, that is going a little far IMHO.
Specifically for Python packages, I also think it's much more appropriate to
use poetry to control them for the host. That way we just have to provide host-python-poetry, and the post-build/post-image scripts can take care of installing all the needed python tools at the exact version you need, controlled by a lock file.
In my opinion, even including those arm-none-eabi etc. compilers in Buildroot
was going a bit too far already. But I guess that that ship has sailed :-)
Regards,
Arnout
[snip]
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Arnout, Charles, All,
On 2023-09-21 22:03 +0200, Arnout Vandecappelle via buildroot spake thusly:
Quoted 10 lines · Charles Hardin
Quoted 9 lines
Quoted 9 lines · Thomas Petazzoni
Quoted 5 lines
On Mon, 21 Aug 2023 16:45:23 -0700
Charles Hardin <[email protected]> wrote:
So, we have various RTOS MCUs that are hooked up into linux controllers, and this is just a simple hook so that the same SDK can be handled to build both firmware for something like an STM32 and a Raspberry-piIf I rephrase this: you want to use Buildroot to build the Zephyr firmware for your STM32 micro-controller, alongside with the Linux system running on the RaspberryPi?
Quoted 7 lines
Quoted 1 line
Yes, and we are doing this today with these patches.
Honestly, I don't know if we want to go in this direction. If Zephyr
doesn't give you the possibility to do "from scratch" builds, then in this day and age it's more appropriate to create a container image for your build environment IMHO. There's something to be said for including host packages that are likely to be used in post-build/post-image scripts, but for building a completely different OS, that is going a little far IMHO.
I concur.
I believe in this case that external projects are better suited to handle the nitty-gritty details of running those buildsystems, even if, as Arnout suggests, that is wrapped in a docker image. With proper CI/CD, it is easy to construct pipelines that build those "other OS" stuff, export their artefacts for followup Buildroot-based jobs that integrate those artefacts (either as wrapping package that jsut instlla binary blob) or as rootfs overlays, or from post-build scripts.
I have marked the series as rejected in patchwork.
Quoted 9 lines
Specifically for Python packages, I also think it's much more appropriate
to use poetry to control them for the host. That way we just have to provide host-python-poetry, and the post-build/post-image scripts can take care of installing all the needed python tools at the exact version you need, controlled by a lock file.In my opinion, even including those arm-none-eabi etc. compilers in
Buildroot was going a bit too far already. But I guess that that ship has sailed :-)
I also heartily agree.
Regards,
Yann E. MORIN.
Signature · 10 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------' _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 1/4 package/python-dateutil: add host python package for dependencies
In reply to Charles HardinA host package for build support needs the dateutil module and so enable the dependency.
--- package/python-dateutil/python-dateutil.mk | 2 ++ 1 file changed, 2 insertions(+)
package/python-dateutil/python-dateutil.mk +2 -0
@@ -10,5 +10,7 @@ PYTHON_DATEUTIL_SETUP_TYPE = setuptools PYTHON_DATEUTIL_LICENSE = BSD-3-Clause PYTHON_DATEUTIL_LICENSE_FILES = LICENSE PYTHON_DATEUTIL_DEPENDENCIES = host-python-setuptools-scm +HOST_PYTHON_DATEUTIL_DEPENDENCIES = host-python-setuptools-scm host-python-six $(eval $(python-package)) +$(eval $(host-python-package))
Signature · 6 lines
2.39.2 (Apple Git-143) _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 2/4 package/python-docopt: add host python package for dependencies
A host package like pykwalify needs the docopt module for buid support and so enable the dependency.
--- package/python-docopt/python-docopt.mk | 1 + 1 file changed, 1 insertion(+)
package/python-docopt/python-docopt.mk +1 -0
@@ -11,3 +11,4 @@ PYTHON_DOCOPT_LICENSE_FILES = LICENSE-MIT PYTHON_DOCOPT_SETUP_TYPE = setuptools $(eval $(python-package)) +$(eval $(host-python-package))
Signature · 6 lines
2.39.2 (Apple Git-143) _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 3/4 package/python-ruamel-yaml: add host python package for dependencies
In reply to Charles HardinA host package like pykwalify needs to the ruamel module for build support and so enable the dependency.
--- package/python-ruamel-yaml/python-ruamel-yaml.mk | 1 + 1 file changed, 1 insertion(+)
package/python-ruamel-yaml/python-ruamel-yaml.mk +1 -0
@@ -14,3 +14,4 @@ PYTHON_RUAMEL_YAML_CPE_ID_VENDOR = ruamel.yaml_project PYTHON_RUAMEL_YAML_CPE_ID_PRODUCT = ruamel.yaml $(eval $(python-package)) +$(eval $(host-python-package))
Signature · 6 lines
2.39.2 (Apple Git-143) _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH v2 4/4 package/python-pykwalify: add required package for zephyr cmake
In reply to Charles HardinFrom: Charles Hardin <[email protected]>
When building zephyrproject-rtos based firmware, the cmake integrates with python scripts that can be run using the buildroot host-python dependencies. This is useful for the sdk setups and associated build steps for the firmware.
See https://github.com/zephyrproject-rtos/zephyr for the build steps that are used in the cmake and the requirements are satisfied for the majority of build minus this package.
--- package/Config.in | 1 + package/python-pykwalify/Config.in | 9 ++++++++ .../python-pykwalify/python-pykwalify.hash | 6 +++++ package/python-pykwalify/python-pykwalify.mk | 23 +++++++++++++++++++ 4 files changed, 39 insertions(+) create mode 100644 package/python-pykwalify/Config.in create mode 100644 package/python-pykwalify/python-pykwalify.hash create mode 100644 package/python-pykwalify/python-pykwalify.mk
package/Config.in +1 -0
@@ -1225,6 +1225,7 @@ menu "External python modules" source "package/python-pyicu/Config.in" source "package/python-pyinotify/Config.in" source "package/python-pyjwt/Config.in" + source "package/python-pykwalify/Config.in" source "package/python-pylibfdt/Config.in" source "package/python-pylibftdi/Config.in" source "package/python-pylru/Config.in"
package/python-pykwalify/Config.in new file +9 -0
@@ -0,0 +1,9 @@ +config BR2_PACKAGE_PYTHON_PYKWALIFY + bool "python-pykwalify" + select BR2_PACKAGE_PYTHON_DATEUTIL # runtime + select BR2_PACKAGE_PYTHON_DOCOPT # runtime + select BR2_PACKAGE_PYTHON_RUAMEL_YAML # runtime + help + Python lib/cli for JSON/YAML schema validation. + + http://github.com/grokzen/pykwalify
package/python-pykwalify/python-pykwalify.hash new file +6 -0
@@ -0,0 +1,6 @@ +# md5, sha256 from https://pypi.org/pypi/pykwalify/json +md5 153197598f8508a49aefc0f44156c2b8 pykwalify-1.8.0.tar.gz +sha256 796b2ad3ed4cb99b88308b533fb2f559c30fa6efb4fa9fda11347f483d245884 pykwalify-1.8.0.tar.gz +# Locally computed sha256 checksums +sha256 626fafd952ab0345edc227f2d565835fe1fd14067ae0d635ee3e47289b773dc4 LICENSE +sha256 f89f247ddd2b9fdff655668148da4b6feaa9175fa44dd244d233006d0f551cd5 docs/license.rst
package/python-pykwalify/python-pykwalify.mk new file +23 -0
@@ -0,0 +1,23 @@ +################################################################################ +# +# python-pykwalify +# +################################################################################ + +PYTHON_PYKWALIFY_VERSION = 1.8.0 +PYTHON_PYKWALIFY_SOURCE = pykwalify-$(PYTHON_PYKWALIFY_VERSION).tar.gz +PYTHON_PYKWALIFY_SITE = https://files.pythonhosted.org/packages/d5/77/2d6849510dbfce5f74f1f69768763630ad0385ad7bb0a4f39b55de3920c7 +PYTHON_PYKWALIFY_SETUP_TYPE = setuptools +PYTHON_PYKWALIFY_LICENSE = MIT +PYTHON_PYKWALIFY_LICENSE_FILES = LICENSE docs/license.rst +PYTHON_PYKWALIFY_DEPENDENCIES = \ + python-dateutil \ + python-docopt \ + python-ruamel-yaml +HOST_PYTHON_PYKWALIFY_DEPENDENCIES = \ + host-python-dateutil \ + host-python-docopt \ + host-python-ruamel-yaml + +$(eval $(python-package)) +$(eval $(host-python-package))
Signature · 6 lines
2.39.2 (Apple Git-143) _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot