boost: Add Boost.Python NumPy
v2 cover boost: Add Boost.Python NumPy
UPDATE[v2]: I replaced the python-numpy-native trigger with a Boost b2 feature that skips the probing for NumPy if the numpy module's include directory is supplied.
This was necessary as NumPy has a C extension which can't be safely imported when cross-compiling.
----
The recipe for Boost does not support building Boost.Python NumPy.
There is a mistake in the variables supplied to test for numpy. A new PACKAGECONFIG option is also provided to allow it to be optionally enabled.
Below are the commit messages with additional debug information:
boost: Use python3-native executable
The Boost build system executes Python in the compile task to import the Numpy module and print the include directory.
We must specify the python3-native executable in the user-config.jam so that the python.jam can execute it properly.
Without this change, it specifies a non-existant python executable and reports "No such file or directory" in log.do_compile.
``` $ grep -C4 "running" temp/log.do_compile notice: [python-cfg] interpreter command: "/ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master-poky-with-sstate-distro_poky-machine_qemuarm64/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot/usr/bin/python3" notice: [python-cfg] include path: "/ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master-poky-with-sstate-distro_poky-machine_qemuarm64/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot/usr/include/python3.14" notice: [python-cfg] library path: "/ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master-poky-with-sstate-distro_poky-machine_qemuarm64/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot/usr/lib/python3.14" notice: [python-cfg] Checking for NumPy... notice: [python-cfg] running command '/ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master-poky-with-sstate-distro_poky-machine_qemuarm64/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot/usr/bin/python3 -c "import sys; sys.stderr = sys.stdout; import numpy; print(numpy.get_include())"' sh: line 1: /ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master-poky-with-sstate-distro_poky-machine_qemuarm64/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot/usr/bin/python3: No such file or directory notice: [python-cfg] NumPy disabled. Reason: notice: [python-cfg] /ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master-poky-with-sstate-distro_poky-machine_qemuarm64/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot/usr/bin/python3 -c "import sys; sys.stderr = sys.stdout; import numpy; print(numpy.get_include())" aborted with notice: [python-cfg] ```
boost-build-native: Add python-numpy-include feature
The Boost python.jam file attempts to import the NumPy module to determine whether the Boost.Python NumPy extension should be enabled. This approach is problematic when cross-compiling, particularly when NumPy includes native extensions built for the target architecture.
Introduce the new python-numpy-include feature to avoid probing for NumPy at build time by allowing the include path to be specified explicitly.
This enables BitBake to provide the correct NumPy include directory from the recipe sysroot, eliminating the need to import the target NumPy module from site-packages.
boost: Add support for Boost.Python NumPy extension
Support for NumPy was added to Boost release 1.63.
This additional numpy PACKAGECONFIG ensures that python3-numpy is added as build-time and run-time dependencies. It uses the custom-made python-numpy-include feature to supply the directory for the numpy module headers to the b2 build system.
If numpy is not added the PACKAGECONFIG it will still attempt the probe and fail gracefully like before.
With this support in place we may add a PACKAGECONFIG to enable numpy.
Shown here in conf/local.conf:
``` PACKAGECONFIG:append:pn-boost = " numpy" ```
$ grep python-cfg -A10 log.do_compile notice: [python-cfg] Configuring python... notice: [python-cfg] user-specified version: "3.14" notice: [python-cfg] user-specified cmd-or-prefix: "/ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot-native/usr/bin/python3-native/python3" notice: [python-cfg] user-specified includes: "/ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot/usr/include/python3.14" notice: [python-cfg] user-specified libraries: "/ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot/usr/lib/python3.14" notice: [python-cfg] user-specified condition: "<python-numpy-include>/ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot/usr/lib/python3.14/site-packages/numpy/_core/include" notice: [python-cfg] Details of this Python configuration: notice: [python-cfg] interpreter command: "/ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot-native/usr/bin/python3-native/python3" notice: [python-cfg] include path: "/ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot/usr/include/python3.14" notice: [python-cfg] library path: "/ala-lpggp31/rwoolley/boost-fix/bitbake-builds/poky-master/build/tmp/work/cortexa57-poky-linux/boost/1.91.0/recipe-sysroot/usr/lib/python3.14" notice: [python-cfg] Checking for NumPy... notice: [python-cfg] NumPy enabled notice: Searching '.' for project-config configuration file 'project-config.jam'. notice: Loading project-config configuration file 'project-config.jam' from '.'. Performing configuration checks
- default address-model : 64-bit [1]
- default architecture : arm [1]
Building the Boost C++ Libraries.
$ find image \( -name libboost_py* -o -name libboost_nu* \) | xargs ls -al -rw-r--r-- 3 rwoolley users 3262 Aug 12 13:16 image/usr/lib/cmake/boost_numpy-1.91.0/libboost_numpy-variant-shared-py3.14.cmake -rw-r--r-- 3 rwoolley users 3316 Aug 12 13:16 image/usr/lib/cmake/boost_numpy-1.91.0/libboost_numpy-variant-static-py3.14.cmake -rw-r--r-- 3 rwoolley users 3280 Aug 12 13:16 image/usr/lib/cmake/boost_python-1.91.0/libboost_python-variant-shared-py3.14.cmake -rw-r--r-- 3 rwoolley users 3334 Aug 12 13:16 image/usr/lib/cmake/boost_python-1.91.0/libboost_python-variant-static-py3.14.cmake -rw-r--r-- 3 rwoolley users 1642060 Aug 12 13:15 image/usr/lib/libboost_numpy314.a lrwxrwxrwx 1 rwoolley users 27 Aug 12 13:15 image/usr/lib/libboost_numpy314.so -> libboost_numpy314.so.1.91.0 -rwxr-xr-x 1 rwoolley users 702176 Aug 12 13:15 image/usr/lib/libboost_numpy314.so.1.91.0 -rw-r--r-- 3 rwoolley users 10658572 Aug 12 13:15 image/usr/lib/libboost_python314.a lrwxrwxrwx 1 rwoolley users 28 Aug 12 13:15 image/usr/lib/libboost_python314.so -> libboost_python314.so.1.91.0 -rwxr-xr-x 1 rwoolley users 4727368 Aug 12 13:15 image/usr/lib/libboost_python314.so.1.91.0
---
Rob Woolley (3):
boost: Use python3-native executable
boost-build-native: Add python-numpy-include feature
boost: Add support for Boost.Python NumPy extension
.../boost/boost-build-native_1.91.0.bb | 3 +- .../add-python-numpy-include-feature.patch | 74 ++++++++++++++++++++++ meta/recipes-support/boost/boost.inc | 9 ++- 3 files changed, 84 insertions(+), 2 deletions(-)
---
base-commit: 96872cac87349528fa0516fa2f6265adf0a15dd7
change-id: 20260806-fix-boost-fca31915ebc4
Best regards,
Signature · 1 line
Rob Woolley <[email protected]>
PATCH v2 1/3 boost: Use python3-native executable
The Boost build system executes Python in the compile task to import the Numpy module and print the include directory.
We must specify the python3-native executable in the user-config.jam so that the python.jam can execute it properly.
Without this change, it specifies a non-existant python executable and reports "No such file or directory" in log.do_compile.
--- meta/recipes-support/boost/boost.inc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
meta/recipes-support/boost/boost.inc +1 -1
@@ -173,7 +173,7 @@ do_configure() { # If we want Python then we need to tell Boost *exactly* where to find it if ${@bb.utils.contains('BOOST_LIBS', 'python', 'true', 'false', d)}; then - echo "using python : ${PYTHON_BASEVERSION} : ${STAGING_DIR_HOST}${bindir}/python3 : ${STAGING_DIR_HOST}${includedir}/${PYTHON_DIR}${PYTHON_ABI} : ${STAGING_DIR_HOST}${libdir}/${PYTHON_DIR} ;" >> ${WORKDIR}/user-config.jam + echo "using python : ${PYTHON_BASEVERSION} : ${STAGING_DIR_NATIVE}${bindir}/python3-native/python3 : ${STAGING_DIR_HOST}${includedir}/${PYTHON_DIR}${PYTHON_ABI} : ${STAGING_DIR_HOST}${libdir}/${PYTHON_DIR} ;" >> ${WORKDIR}/user-config.jam fi if ${@bb.utils.contains('BOOST_LIBS', 'mpi', 'true', 'false', d)}; then
Signature · 1 line
2.49.0
Patchtest results for [PATCH v2 1/3] boost: Use python3-native executable
Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:
---
Testing patch /home/patchtest/share/mboxes/v2-1-3-boost-Use-python3-native-executable.patch
FAIL: test max line length: Patch line too long (current length 240, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test auh changelog truncation notice (test_mbox.TestMbox.test_auh_changelog_truncation_notice)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test commit message user tags (test_mbox.TestMbox.test_commit_message_user_tags)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)
SKIP: pretest pylint: No python related patches, skipping test (test_python_pylint.PyLint.pretest_pylint)
SKIP: pretest src uri left files: No modified recipes, skipping pretest (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new source patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new source patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new source patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test pylint: No python related patches, skipping test (test_python_pylint.PyLint.test_pylint)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: No modified recipes, skipping test (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
---
Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!
PATCH v2 2/3 boost-build-native: Add python-numpy-include feature
In reply to Rob WoolleyThe Boost python.jam file attempts to import the NumPy module to determine whether the Boost.Python NumPy extension should be enabled. This approach is problematic when cross-compiling, particularly when NumPy includes native extensions built for the target architecture.
Introduce the new python-numpy-include feature to avoid probing for NumPy at build time by allowing the include path to be specified explicitly.
This enables BitBake to provide the correct NumPy include directory from the recipe sysroot, eliminating the need to import the target NumPy module from site-packages.
--- .../boost/boost-build-native_1.91.0.bb | 3 +- .../add-python-numpy-include-feature.patch | 74 ++++++++++++++++++++++ 2 files changed, 76 insertions(+), 1 deletion(-)
meta/recipes-support/boost/boost-build-native_1.91.0.bb +2 -1
@@ -6,7 +6,8 @@ SECTION = "devel" LICENSE = "BSL-1.0" LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=e4224ccaecb14d942c71d31bef20d78c" -SRC_URI = "git://github.com/boostorg/build;protocol=https;branch=master;tag=boost-${PV}" +SRC_URI = "git://github.com/boostorg/build;protocol=https;branch=master;tag=boost-${PV} \ + file://add-python-numpy-include-feature.patch" SRCREV = "85252c9d6a3817daca23ddae94ce16c7be4d2c12" PE = "1"
meta/recipes-support/boost/boost-build/add-python-numpy-include-feature.patch new file +74 -0
74 lines not on this page · raw message
Signature · 1 line
2.49.0
Re: [PATCH v2 2/3] boost-build-native: Add python-numpy-include feature
Quoted 1 line · Rob Woolley
+Upstream-Status: Pending
Please do not use Pending without a reason for it. This should be submitted upstream first.
Alex
Re: [OE-core] [PATCH v2 2/3] boost-build-native: Add python-numpy-include feature
Hi Alex,
Thanks for taking a look at this. I have submitted the patch here (
https://github.com/bfgroup/b2/pull/611 ) and will revise the Upstream-Status in the next series.
Do you have any technical concerns with the changes?
Regards,
Rob
Quoted 20 lines · Alexander Kanavin via lists.openembedded.org
Quoted 1 line · Rob Woolley
+Upstream-Status: Pending
Please do not use Pending without a reason for it. This should be submitted upstream first.
Alex
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#243343):
https://lists.openembedded.org/g/openembedded-core/message/243343
Mute This Topic: https://lists.openembedded.org/mt/120724337/556952
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Re: [OE-core] [PATCH v2 2/3] boost-build-native: Add python-numpy-include feature
Quoted 3 lines · Rob Woolley
Thanks for taking a look at this. I have submitted the patch here ( https://github.com/bfgroup/b2/pull/611 ) and will revise the Upstream-Status in the next series.
Do you have any technical concerns with the changes?
Thanks. The patch itself is fine.
We're trying to get the amount of Pending patches to zero in oe-core :) Historically it's been misused to solve the submitter's immediate problem without taking the trouble of actual upstream submission (or opening a ticket if the patch is clearly inappropriate).
Alex
PATCH v2 3/3 boost: Add support for Boost.Python NumPy extension
In reply to Rob WoolleySupport for NumPy was added to Boost release 1.63.
This additional numpy PACKAGECONFIG ensures that python3-numpy is added as build-time and run-time dependencies. It uses the custom-made python-numpy-include feature to supply the directory for the numpy module headers to the b2 build system.
If numpy is not added the PACKAGECONFIG it will still attempt the probe and fail gracefully like before.
--- meta/recipes-support/boost/boost.inc | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-)
meta/recipes-support/boost/boost.inc +8 -1
@@ -53,6 +53,7 @@ PACKAGECONFIG ??= "locale python" PACKAGECONFIG[locale] = ",,icu" PACKAGECONFIG[graph_parallel] = ",,,boost-mpi mpich" PACKAGECONFIG[mpi] = ",,mpich" +PACKAGECONFIG[numpy] = ",,python3-numpy" PACKAGECONFIG[python] = ",,python3" inherit python3-dir
@@ -86,6 +87,8 @@ FILES:${PN}-serialization = "${libdir}/libboost_serialization*.so.* \ ${libdir}/libboost_wserialization*.so.*" FILES:${PN}-test = "${libdir}/libboost_prg_exec_monitor*.so.* \ ${libdir}/libboost_unit_test_framework*.so.*" +FILES:${PN}-python = "${libdir}/libboost_python*.so.* \ + ${libdir}/libboost_numpy*.so.*" # -dev last to pick up the remaining stuff PACKAGES += "${PN}-dev ${PN}-staticdev"
@@ -173,7 +176,11 @@ do_configure() { # If we want Python then we need to tell Boost *exactly* where to find it if ${@bb.utils.contains('BOOST_LIBS', 'python', 'true', 'false', d)}; then - echo "using python : ${PYTHON_BASEVERSION} : ${STAGING_DIR_NATIVE}${bindir}/python3-native/python3 : ${STAGING_DIR_HOST}${includedir}/${PYTHON_DIR}${PYTHON_ABI} : ${STAGING_DIR_HOST}${libdir}/${PYTHON_DIR} ;" >> ${WORKDIR}/user-config.jam + if ${@bb.utils.contains('PACKAGECONFIG', 'numpy', 'true', 'false', d)}; then + # Use the python-numpy-include feature patched in to boost-build-native + ENABLE_NUMPY="<python-numpy-include>${STAGING_LIBDIR}/${PYTHON_DIR}/site-packages/numpy/_core/include" + fi + echo "using python : ${PYTHON_BASEVERSION} : ${STAGING_DIR_NATIVE}${bindir}/python3-native/python3 : ${STAGING_DIR_HOST}${includedir}/${PYTHON_DIR}${PYTHON_ABI} : ${STAGING_DIR_HOST}${libdir}/${PYTHON_DIR} : ${ENABLE_NUMPY} ;" >> ${WORKDIR}/user-config.jam fi if ${@bb.utils.contains('BOOST_LIBS', 'mpi', 'true', 'false', d)}; then
Signature · 1 line
2.49.0
Patchtest results for [PATCH v2 3/3] boost: Add support for Boost.Python NumPy extension
Thank you for your submission. Patchtest identified one or more issues with the patch. Please see the log below for more information:
---
Testing patch /home/patchtest/share/mboxes/v2-3-3-boost-Add-support-for-Boost.Python-NumPy-extension.patch
FAIL: test max line length: Patch line too long (current length 258, maximum is 200) (test_metadata.TestMetadata.test_max_line_length)
PASS: test Signed-off-by presence (test_mbox.TestMbox.test_signed_off_by_presence)
PASS: test auh changelog truncation notice (test_mbox.TestMbox.test_auh_changelog_truncation_notice)
PASS: test author valid (test_mbox.TestMbox.test_author_valid)
PASS: test commit message presence (test_mbox.TestMbox.test_commit_message_presence)
PASS: test commit message user tags (test_mbox.TestMbox.test_commit_message_user_tags)
PASS: test mbox format (test_mbox.TestMbox.test_mbox_format)
PASS: test non-AUH upgrade (test_mbox.TestMbox.test_non_auh_upgrade)
PASS: test shortlog format (test_mbox.TestMbox.test_shortlog_format)
PASS: test shortlog length (test_mbox.TestMbox.test_shortlog_length)
PASS: test target mailing list (test_mbox.TestMbox.test_target_mailing_list)
SKIP: pretest pylint: No python related patches, skipping test (test_python_pylint.PyLint.pretest_pylint)
SKIP: pretest src uri left files: Patch cannot be merged (test_metadata.TestMetadata.pretest_src_uri_left_files)
SKIP: test CVE check ignore: No modified recipes or older target branch, skipping test (test_metadata.TestMetadata.test_cve_check_ignore)
SKIP: test CVE tag format: No new source patches introduced (test_patch.TestPatch.test_cve_tag_format)
SKIP: test Signed-off-by presence: No new source patches introduced (test_patch.TestPatch.test_signed_off_by_presence)
SKIP: test Upstream-Status presence: No new source patches introduced (test_patch.TestPatch.test_upstream_status_presence_format)
SKIP: test bugzilla entry format: No bug ID found (test_mbox.TestMbox.test_bugzilla_entry_format)
SKIP: test lic files chksum modified not mentioned: No modified recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_modified_not_mentioned)
SKIP: test lic files chksum presence: No added recipes, skipping test (test_metadata.TestMetadata.test_lic_files_chksum_presence)
SKIP: test license presence: No added recipes, skipping test (test_metadata.TestMetadata.test_license_presence)
SKIP: test pylint: No python related patches, skipping test (test_python_pylint.PyLint.test_pylint)
SKIP: test series merge on head: Merge test is disabled for now (test_mbox.TestMbox.test_series_merge_on_head)
SKIP: test src uri left files: Patch cannot be merged (test_metadata.TestMetadata.test_src_uri_left_files)
SKIP: test summary presence: No added recipes, skipping test (test_metadata.TestMetadata.test_summary_presence)
---
Please address the issues identified and
submit a new revision of the patch, or alternatively, reply to this email with an explanation of why the patch should be accepted. If you believe these results are due to an error in patchtest, please submit a bug at https://bugzilla.yoctoproject.org/ (use the 'Patchtest' category under 'Yocto Project Subprojects'). For more information on specific failures, see: https://wiki.yoctoproject.org/wiki/Patchtest. Thank you!