Bump of python-cryptography and other Rust related fun
cover Bump of python-cryptography and other Rust related fun
Hello,
This series bumps the python-cryptography package to a new version. It is based on earlier work from James Hilliard at:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/
But it did require quite some more work, which is presented in this patch series.
The unique challenge caused by this version bump is that python-cryptography now has some Rust code integrated, so suddenly this package, and all its reverse dependencies need to depend on Rust being available.
There is also a host variant of python-cryptography, which is used solely by ca-certificates, which has a significant number of reverse dependencies. To avoid adding Rust dependencies to these, we broke the dependency of ca-certificates on host-python-cryptography using a simple patch, and dropped host-python-cryptography altogether.
This patch series depends on the "Go/Cargo vendoring" patch series posted recently. Indeed, the Rust code in python-cryptography depends on other Cargo crates, which we don't want to download during the build, but really during the download step.
There is however still one problem with the last patch of the series, see below.
This series is available at:
https://github.com/tpetazzoni/buildroot/tree/python-cryptography
The patch series goes like this:
- PATCH 1 modifies ca-certificates to no longer need host-python-cryptography
- PATCH 2 drops host-python-cryptography itselef
- PATCH 3 and 4 fix some issues in package/rustc/ in the description of the ARMv5 and RISC-V 64-bit Rust support. We noticed this because Rust binaries were not working on ARMv5 ("Illegal instruction"), this was due to the fact that we were using the pre-compiled Rust standard library for ARMv6, due to a bogus architecture name. This is fixed by PATCH 3. And while reviewing this, we spotted a similar issue for RISC-V 64-bit, which is fixed in PATCH 4.
- PATCH 5 slightly improves the Cargo download post-process script to make it possible to pass a custom path to the Cargo.toml file, instead of assuming it's located at the root of the package source tree.
- PATCH 6 modifies the runtime test cases that involve python-cryptography directly or indirectly to use a glibc toolchain instead of the default uclibc toolchain. Indeed, Rust is for now not available on uClibc platforms. This could have been done in the patch doing the version bump, but the version bump patch is already quite large, and the update of the runtime tests could be done as a separate preparation step.
- PATCH 7 finally bumps python-cryptography
itself. python-cryptography itself requires a bit of trickery to leverage the Cargo download post-process logic, to vendor dependencies during the download step. Then the patch also touches the entire reverse dependency tree (sometimes up to 5 levels deep!). This patch should NOT be applied for now, due to issues (see below).
This series was tested by building the following configuration:
BR2_arm=y
BR2_cortex_a9=y
BR2_ARM_ENABLE_VFP=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
# BR2_PACKAGE_BUSYBOX is not set
BR2_PACKAGE_PYTHON_CHANNELS_REDIS=y
BR2_PACKAGE_PYTHON_CROSSBAR=y
BR2_PACKAGE_PYTHON_KEYRING=y
BR2_PACKAGE_PYTHON_OPCUA_ASYNCIO=y
BR2_PACKAGE_PYTHON_TXDBUS=y
BR2_PACKAGE_DOCKER_COMPOSE=y
# BR2_TARGET_ROOTFS_TAR is not set
that builds all affected packages.
It was also tested by running all Python runtime test cases:
https://gitlab.com/tpetazzoni/buildroot/-/pipelines/443445436
The one thing that fails is when building python-cryptography on a target arch that is the same as the host. This can be reproduced by running the tests.package.test_docker_compose.TestDockerCompose test case.
We get this:
Running `/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build`
error: failed to run custom build command for proc-macro-hack v0.5.19
Caused by:
process didn't exit successfully: /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build (exi
t status: 1)
--- stderr /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: versio
n `GLIBC_2.33' not found (required by /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build)
/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: versio
n `GLIBC_2.32' not found (required by /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build)
/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: versio
n `GLIBC_2.34' not found (required by /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build)
warning: build failed, waiting for other jobs to finish...
error: build failed
Which looks terribly similar to the issue on ripgrep which was fixed by commit 4b2be770b8a853a7dd97b5788d837f0d84923fa1. So I tried doing the same, i.e passing:
__CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly" \ CARGO_TARGET_APPLIES_TO_HOST="false"
in the environment, and -Z target-applies-to-host to cargo (by passing it to the args argument of the RustExtension class in python-cryptography's setup.py). But cargo doesn't like that, the build log goes like this:
Quoted 1 line
Quoted 1 line
Quoted 1 line
python-cryptography 36.0.1 BuildingESC[27m
... calling setup.py ...
cargo rustc --lib --manifest-path src/rust/Cargo.toml --features pyo3/abi3-py36 pyo3/extension-module --target x86_64-unknown-linux-gnu -Z target-applies-to-host --release --verbose -- --crate-type cdylib
...
error: unknown -Z flag specified: target-applies-to-host
The difference that I see with the cargo infrastructure is that this is not using "cargo build" but "cargo rustc". I would really appreciate some help on solving this (last?) issue.
Best regards,
Thomas Petazzoni
Thomas Petazzoni (7):
package/ca-certificates: remove dependency on host-python-cryptography package/python-cryptography: drop host variant package/rustc: fix BR2_PACKAGE_HOST_RUSTC_ARCH for ARMv5 package/rustc: fix the riscv64gc architecture handling support/download/cargo-post-process: make manifest path configurable support/testing/tests: switch python-cryptography related tests to
glibc
DO-NOT-APPLY package/python-cryptography: bump version to 36.0.1
...2pem.py-make-cryptography-module-opt.patch | 59 +++++++++++++++++++ package/ca-certificates/ca-certificates.mk | 2 +- package/docker-compose/Config.in | 8 ++- package/python-autobahn/Config.in | 6 +- package/python-channels-redis/Config.in | 7 +-- package/python-channels/Config.in | 7 +-- package/python-crossbar/Config.in | 6 +- package/python-cryptography/Config.in | 14 +---- .../python-cryptography.hash | 5 +- .../python-cryptography.mk | 20 ++++--- package/python-daphne/Config.in | 7 +-- package/python-keyring/Config.in | 7 +-- package/python-opcua-asyncio/Config.in | 1 + package/python-paramiko/Config.in | 8 +-- package/python-pyopenssl/Config.in | 8 +-- package/python-secretstorage/Config.in | 6 +- package/python-service-identity/Config.in | 10 ++-- package/python-treq/Config.in | 10 ++-- package/python-twisted/Config.in | 6 +- package/python-txdbus/Config.in | 5 +- package/python-txtorcon/Config.in | 7 +-- package/rustc/Config.in.host | 4 +- support/download/cargo-post-process | 2 +- .../tests/package/test_python_autobahn.py | 10 +++- .../tests/package/test_python_crossbar.py | 10 +++- .../tests/package/test_python_cryptography.py | 10 +++- .../package/test_python_service_identity.py | 10 +++- .../testing/tests/package/test_python_treq.py | 10 +++- .../tests/package/test_python_txtorcon.py | 10 +++- 29 files changed, 170 insertions(+), 105 deletions(-) create mode 100644 package/ca-certificates/0001-mozilla-certdata2pem.py-make-cryptography-module-opt.patch
Signature · 6 lines
2.33.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 1/7 package/ca-certificates: remove dependency on host-python-cryptography
The host-python-cryptography module is only used by ca-certificates for a check of the expiration date of certificates, which is only a warning not even causing the build to abort, i.e something that Buildroot users are most likely never going to see.
Since the host-python-cryptography dependency would soon require a dependency on rust, it's a lot simpler to just patch the certdata2pem.py script to no longer require cryptography, but only make use of it if available.
--- ...2pem.py-make-cryptography-module-opt.patch | 59 +++++++++++++++++++ package/ca-certificates/ca-certificates.mk | 2 +- 2 files changed, 60 insertions(+), 1 deletion(-) create mode 100644 package/ca-certificates/0001-mozilla-certdata2pem.py-make-cryptography-module-opt.patch
package/ca-certificates/0001-mozilla-certdata2pem.py-make-cryptography-module-opt.patch new file +59 -0
@@ -0,0 +1,59 @@ +From bf18b564122e8f976681a2398862fde1eafd84ba Mon Sep 17 00:00:00 2001 +From: Thomas Petazzoni <[email protected]> +Date: Thu, 6 Jan 2022 23:15:00 +0100 +Subject: [PATCH] mozilla/certdata2pem.py: make cryptography module optional + +The Python cryptography module is only used to verify if trusted +certificates have expired, but this is only a warning. For some build +systems and distributions, providing Python cryptography is costly, +especially since it's now partly written in Rust. + +As the check is only a warning, it's anyway going to be overlooked by +most people. This commit changes the check to be optional: if the +cryptography Python module is there, we perform the check, otherwise +the check is skipped. + +Signed-off-by: Thomas Petazzoni <[email protected]> +--- + mozilla/certdata2pem.py | 18 ++++++++++-------- + 1 file changed, 10 insertions(+), 8 deletions(-) + +diff --git a/mozilla/certdata2pem.py b/mozilla/certdata2pem.py +index ede23d4..a6261f8 100644 +--- a/mozilla/certdata2pem.py ++++ b/mozilla/certdata2pem.py +@@ -28,9 +28,6 @@ import sys + import textwrap + import io + +-from cryptography import x509 +- +- + objects = [] + + # Dirty file parser. +@@ -122,11 +119,16 @@ for obj in objects: + if not obj['CKA_LABEL'] in trust or not trust[obj['CKA_LABEL']]: + continue + +- cert = x509.load_der_x509_certificate(obj['CKA_VALUE']) +- if cert.not_valid_after < datetime.datetime.now(): +- print('!'*74) +- print('Trusted but expired certificate found: %s' % obj['CKA_LABEL']) +- print('!'*74) ++ try: ++ from cryptography import x509 ++ ++ cert = x509.load_der_x509_certificate(obj['CKA_VALUE']) ++ if cert.not_valid_after < datetime.datetime.now(): ++ print('!'*74) ++ print('Trusted but expired certificate found: %s' % obj['CKA_LABEL']) ++ print('!'*74) ++ except ImportError: ++ pass + + bname = obj['CKA_LABEL'][1:-1].replace('/', '_')\ + .replace(' ', '_')\ +-- +2.33.1 +
package/ca-certificates/ca-certificates.mk +1 -1
@@ -7,7 +7,7 @@ CA_CERTIFICATES_VERSION = 20211016 CA_CERTIFICATES_SOURCE = ca-certificates_$(CA_CERTIFICATES_VERSION).tar.xz CA_CERTIFICATES_SITE = https://snapshot.debian.org/archive/debian/20211022T144903Z/pool/main/c/ca-certificates -CA_CERTIFICATES_DEPENDENCIES = host-openssl host-python3 host-python-cryptography +CA_CERTIFICATES_DEPENDENCIES = host-openssl host-python3 CA_CERTIFICATES_LICENSE = GPL-2.0+ (script), MPL-2.0 (data) CA_CERTIFICATES_LICENSE_FILES = debian/copyright
Signature · 6 lines
2.33.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 2/7 package/python-cryptography: drop host variant
In reply to Thomas PetazzoniThe host variant was only needed by ca-certificates, which no longer needs it, so drop this variant.
--- package/python-cryptography/python-cryptography.mk | 6 ------ 1 file changed, 6 deletions(-)
package/python-cryptography/python-cryptography.mk +0 -6
@@ -13,11 +13,5 @@ PYTHON_CRYPTOGRAPHY_LICENSE_FILES = LICENSE LICENSE.APACHE LICENSE.BSD PYTHON_CRYPTOGRAPHY_CPE_ID_VENDOR = cryptography_project PYTHON_CRYPTOGRAPHY_CPE_ID_PRODUCT = cryptography PYTHON_CRYPTOGRAPHY_DEPENDENCIES = host-python-cffi openssl -HOST_PYTHON_CRYPTOGRAPHY_NEEDS_HOST_PYTHON = python3 -HOST_PYTHON_CRYPTOGRAPHY_DEPENDENCIES = \ - host-python3-cffi \ - host-python3-six \ - host-openssl $(eval $(python-package)) -$(eval $(host-python-package))
Signature · 6 lines
2.33.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 3/7 package/rustc: fix BR2_PACKAGE_HOST_RUSTC_ARCH for ARMv5
In reply to Thomas PetazzoniBR2_PACKAGE_HOST_RUSTC_ARCH only had a special value for BR2_ARM_CPU_ARMV7A, but it also needs a special value for BR2_ARM_CPU_ARMV5, as the pre-compiled Rust standard library for ARMv5TE is identified with the "armv5te" architecture name, see
https://doc.rust-lang.org/nightly/rustc/platform-support.html.
We noticed this because Rust binaries wouldn't work on an ARMv5 platform (Illegal instruction). This was due to the usage of the arm-unknown-linux-gnueabi variant of the Rust standard library, which is for ARMv6. Thanks to this commit, we correctly use the armv5te-unknown-linux-gnueabi variant, and Rust binaries work properly on ARMv5TE.
--- package/rustc/Config.in.host | 1 + 1 file changed, 1 insertion(+)
package/rustc/Config.in.host +1 -0
@@ -102,6 +102,7 @@ config BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS config BR2_PACKAGE_HOST_RUSTC_ARCH string + default "armv5te" if BR2_ARM_CPU_ARMV5 default "armv7" if BR2_ARM_CPU_ARMV7A default BR2_ARCH if !BR2_ARM_CPU_ARMV7A
Signature · 6 lines
2.33.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 3/7] package/rustc: fix BR2_PACKAGE_HOST_RUSTC_ARCH for ARMv5
Quoted 23 lines · On Fri, 7 Jan 2022 18:13:13 +0100 Thomas Petazzoni…
BR2_PACKAGE_HOST_RUSTC_ARCHonly had a special value forBR2_ARM_CPU_ARMV7A, but it also needs a special value forBR2_ARM_CPU_ARMV5, as the pre-compiled Rust standard library for ARMv5TE is identified with the "armv5te" architecture name, see
https://doc.rust-lang.org/nightly/rustc/platform-support.html.We noticed this because Rust binaries wouldn't work on an ARMv5 platform (Illegal instruction). This was due to the usage of the arm-unknown-linux-gnueabi variant of the Rust standard library, which is for ARMv6. Thanks to this commit, we correctly use the armv5te-unknown-linux-gnueabi variant, and Rust binaries work properly on ARMv5TE.
Signed-off-by Thomas Petazzoni <[email protected]>--- package/rustc/Config.in.host | 1 + 1 file changed, 1 insertion(+)package/rustc/Config.in.host +0
-0
package/rustc/Config.in.host config BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS · line 107
Quoted 6 lines
package/rustc/Config.in.host +1
-0@@ -102,6 +102,7 @@ config BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS config BR2_PACKAGE_HOST_RUSTC_ARCH string + default "armv5te" if BR2_ARM_CPU_ARMV5 default "armv7" if BR2_ARM_CPU_ARMV7A default BR2_ARCH if !BR2_ARM_CPU_ARMV7A
I need to drop the "if !BR2_ARM_CPU_ARMV7A" here. Will do in the next iteration, or can be done when applying if there are no other comments.
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 4/7 package/rustc: fix the riscv64gc architecture handling
In reply to Thomas Petazzoniriscv64gc was made available for BR2_riscv, but RISC-V can be 32-bit or 64-bit, so we need to restrict it to BR2_RISCV_64. There's no need to keep the BR2_riscv dependency, as BR2_RISCV_64 can only be true when BR2_riscv is true.
Also, BR2_PACKAGE_HOST_RUSTC_ARCH needs to be set to riscv64gc to allow rust-bin to download its pre-compiled standard library correctly.
--- package/rustc/Config.in.host | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
package/rustc/Config.in.host +2 -1
@@ -49,7 +49,7 @@ config BR2_PACKAGE_HOST_RUSTC_TARGET_TIER2_HOST_TOOLS_PLATFORMS default y if BR2_powerpc64le && BR2_TOOLCHAIN_USES_GLIBC # riscv64gc-unknown-linux-gnu # "g" stands for imafd, and we also need "c". - default y if BR2_riscv && BR2_RISCV_ISA_RVI && BR2_RISCV_ISA_RVM && \ + default y if BR2_RISCV_64 && BR2_RISCV_ISA_RVI && BR2_RISCV_ISA_RVM && \ BR2_RISCV_ISA_RVA && BR2_RISCV_ISA_RVF && \ BR2_RISCV_ISA_RVD && BR2_RISCV_ISA_RVC && \ BR2_TOOLCHAIN_USES_GLIBC
@@ -104,6 +104,7 @@ config BR2_PACKAGE_HOST_RUSTC_ARCH string default "armv5te" if BR2_ARM_CPU_ARMV5 default "armv7" if BR2_ARM_CPU_ARMV7A + default "riscv64gc" if BR2_RISCV_64 default BR2_ARCH if !BR2_ARM_CPU_ARMV7A config BR2_PACKAGE_HOST_RUSTC_ABI
Signature · 6 lines
2.33.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Re: [Buildroot] [PATCH 4/7] package/rustc: fix the riscv64gc architecture handling
Quoted 11 lines · Thomas Petazzoni
riscv64gc was made available for
BR2_riscv, but RISC-V can be 32-bit or 64-bit, so we need to restrict it toBR2_RISCV_64. There's no need to keep theBR2_riscvdependency, asBR2_RISCV_64can only be true whenBR2_riscvis true.Also,
BR2_PACKAGE_HOST_RUSTC_ARCHneeds to be set to riscv64gc to allow rust-bin to download its pre-compiled standard library correctly.Signed-off-by Thomas Petazzoni <[email protected]>
Alistair
Quoted 28 lines
--- package/rustc/Config.in.host | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)package/rustc/Config.in.host +2
-2@@ -49,7 +49,7 @@ config BR2_PACKAGE_HOST_RUSTC_TARGET_TIER2_HOST_TOOLS_PLATFORMS default y if BR2_powerpc64le && BR2_TOOLCHAIN_USES_GLIBC # riscv64gc-unknown-linux-gnu # "g" stands for imafd, and we also need "c". - default y if BR2_riscv && BR2_RISCV_ISA_RVI && BR2_RISCV_ISA_RVM && \ + default y if BR2_RISCV_64 && BR2_RISCV_ISA_RVI && BR2_RISCV_ISA_RVM && \ BR2_RISCV_ISA_RVA && BR2_RISCV_ISA_RVF && \ BR2_RISCV_ISA_RVD && BR2_RISCV_ISA_RVC && \ BR2_TOOLCHAIN_USES_GLIBC@@ -104,6 +104,7 @@ config BR2_PACKAGE_HOST_RUSTC_ARCH string default "armv5te" if BR2_ARM_CPU_ARMV5 default "armv7" if BR2_ARM_CPU_ARMV7A + default "riscv64gc" if BR2_RISCV_64 default BR2_ARCH if !BR2_ARM_CPU_ARMV7A config BR2_PACKAGE_HOST_RUSTC_ABI --2.33.1
Quoted 4 lines
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 5/7 support/download/cargo-post-process: make manifest path configurable
In reply to Thomas PetazzoniIn most pure Rust packages, the Cargo.toml manifest is at the root directory, which is why we could call "cargo vendor" without specifying the path of the manifest.
However, other packages, such as python-cryptography, which have parts implemented in Rust, have their Cargo.toml located in a specific subdirectory.
This commit extends the cargo-post-process download script to understand a BR_CARGO_MANIFEST_PATH environment variable, which allows a package to pass the location of the Cargo.toml file. If not passed, "Cargo.toml" is used, preserving the existing behavior for other packages.
--- support/download/cargo-post-process | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
support/download/cargo-post-process +1 -1
@@ -22,7 +22,7 @@ post_process_unpack ${base_name} ${output} # Do the Cargo vendoring pushd ${base_name} > /dev/null -cargo vendor --locked VENDOR +cargo vendor --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} --locked VENDOR # Create the local .cargo/config with vendor info mkdir -p .cargo/
Signature · 6 lines
2.33.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 6/7 support/testing/tests: switch python-cryptography related tests to glibc
In reply to Thomas PetazzoniIn a follow-up commit, we are about to bump python-cryptography to a new version, which has the interesting charateristic of using Rust code. This means python-cryptography will now only be available on platforms supported by Rust, which for now excludes uclibc-based configurations (none of the Rust Tier1/Tier2 platforms use uClibc, there is some uClibc support in Tier3 platforms but they have not been added to Buildroot for now).
So in preparation for this bump, we switch the few test cases of Python packages that directly or indirectly use python-cryptography to use a glibc toolchain. Another impacted test case is the docker-compose test case, but it already uses a glibc toolchain;
--- support/testing/tests/package/test_python_autobahn.py | 10 +++++++++- support/testing/tests/package/test_python_crossbar.py | 10 +++++++++- .../testing/tests/package/test_python_cryptography.py | 10 +++++++++- .../tests/package/test_python_service_identity.py | 10 +++++++++- support/testing/tests/package/test_python_treq.py | 10 +++++++++- support/testing/tests/package/test_python_txtorcon.py | 10 +++++++++- 6 files changed, 54 insertions(+), 6 deletions(-)
support/testing/tests/package/test_python_autobahn.py +9 -1
@@ -3,9 +3,17 @@ from tests.package.test_python import TestPythonPackageBase class TestPythonPy3Autobahn(TestPythonPackageBase): __test__ = True - config = TestPythonPackageBase.config + \ + # Need to use a different toolchain than the default due to + # python-cryptography using Rust (not available with uclibc) + config = \ """ + BR2_arm=y + BR2_TOOLCHAIN_EXTERNAL=y + BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y + BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_AUTOBAHN=y + BR2_TARGET_ROOTFS_CPIO=y + # BR2_TARGET_ROOTFS_TAR is not set """ sample_scripts = ["tests/package/sample_python_autobahn.py"]
support/testing/tests/package/test_python_crossbar.py +9 -1
@@ -3,10 +3,18 @@ from tests.package.test_python import TestPythonPackageBase class TestPythonPy3Crossbar(TestPythonPackageBase): __test__ = True - config = TestPythonPackageBase.config + \ + # Need to use a different toolchain than the default due to + # python-cryptography using Rust (not available with uclibc) + config = \ """ + BR2_arm=y + BR2_TOOLCHAIN_EXTERNAL=y + BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y + BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_CROSSBAR=y + BR2_TARGET_ROOTFS_CPIO=y + # BR2_TARGET_ROOTFS_TAR is not set """ sample_scripts = ["tests/package/sample_python_crossbar.py"] timeout = 60
support/testing/tests/package/test_python_cryptography.py +9 -1
@@ -3,10 +3,18 @@ from tests.package.test_python import TestPythonPackageBase class TestPythonPy3Cryptography(TestPythonPackageBase): __test__ = True - config = TestPythonPackageBase.config + \ + # Need to use a different toolchain than the default due to + # python-cryptography using Rust (not available with uclibc) + config = \ """ + BR2_arm=y + BR2_TOOLCHAIN_EXTERNAL=y + BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y + BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_CRYPTOGRAPHY=y + BR2_TARGET_ROOTFS_CPIO=y + # BR2_TARGET_ROOTFS_TAR is not set """ sample_scripts = ["tests/package/sample_python_cryptography.py"] timeout = 40
support/testing/tests/package/test_python_service_identity.py +9 -1
@@ -3,10 +3,18 @@ from tests.package.test_python import TestPythonPackageBase class TestPythonPy3ServiceIdentity(TestPythonPackageBase): __test__ = True - config = TestPythonPackageBase.config + \ + # Need to use a different toolchain than the default due to + # python-cryptography using Rust (not available with uclibc) + config = \ """ + BR2_arm=y + BR2_TOOLCHAIN_EXTERNAL=y + BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y + BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_SERVICE_IDENTITY=y + BR2_TARGET_ROOTFS_CPIO=y + # BR2_TARGET_ROOTFS_TAR is not set """ sample_scripts = ["tests/package/sample_python_service_identity.py"] timeout = 30
support/testing/tests/package/test_python_treq.py +9 -1
@@ -14,8 +14,16 @@ class TestPythonTreq(TestPythonPackageBase): class TestPythonPy3Treq(TestPythonTreq): __test__ = True - config = TestPythonTreq.config + \ + # Need to use a different toolchain than the default due to + # python-cryptography using Rust (not available with uclibc) + config = \ """ + BR2_arm=y + BR2_TOOLCHAIN_EXTERNAL=y + BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y + BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_TREQ=y + BR2_TARGET_ROOTFS_CPIO=y + # BR2_TARGET_ROOTFS_TAR is not set """
support/testing/tests/package/test_python_txtorcon.py +9 -1
@@ -3,10 +3,18 @@ from tests.package.test_python import TestPythonPackageBase class TestPythonPy3Txtorcon(TestPythonPackageBase): __test__ = True - config = TestPythonPackageBase.config + \ + # Need to use a different toolchain than the default due to + # python-cryptography using Rust (not available with uclibc) + config = \ """ + BR2_arm=y + BR2_TOOLCHAIN_EXTERNAL=y + BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y + BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y BR2_PACKAGE_PYTHON3=y BR2_PACKAGE_PYTHON_TXTORCON=y + BR2_TARGET_ROOTFS_CPIO=y + # BR2_TARGET_ROOTFS_TAR is not set """ sample_scripts = ["tests/package/sample_python_txtorcon.py"] timeout = 30
Signature · 6 lines
2.33.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
PATCH 7/7 DO-NOT-APPLY package/python-cryptography: bump version to 36.0.1
In reply to Thomas PetazzoniThis version bump requires significant changes because python-cryptography is now partially implemented in Rust.
This means that:
- The C++ dependency is no longer needed.
- We need to ensure we are on an architecture where Rust is available (BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS)
- Almost all Python dependencies are no longer relevant, except for the python-cffi.
- A number of magic environment variables are needed to make the Rust part build correctly (CARGO_BUILD_TARGET, PYO3_CROSS_LIB_DIR, CARGO_HOME)
- We need to invoke the "cargo" download post-process hook to vendor the Cargo dependencies at download time.
- We need to propagate to relatively significant reverse dependency tree the changes of dependencies on python-cryptography.
DO-NOT-APPLY: this commit is currently broken when host arch == target arch.
--- package/docker-compose/Config.in | 8 +++++--- package/python-autobahn/Config.in | 6 +----- package/python-channels-redis/Config.in | 7 ++----- package/python-channels/Config.in | 7 ++----- package/python-crossbar/Config.in | 6 +----- package/python-cryptography/Config.in | 14 ++------------ .../python-cryptography/python-cryptography.hash | 5 ++--- package/python-cryptography/python-cryptography.mk | 14 +++++++++++--- package/python-daphne/Config.in | 7 ++----- package/python-keyring/Config.in | 7 ++----- package/python-opcua-asyncio/Config.in | 1 + package/python-paramiko/Config.in | 8 ++------ package/python-pyopenssl/Config.in | 8 ++------ package/python-secretstorage/Config.in | 6 +----- package/python-service-identity/Config.in | 10 ++++------ package/python-treq/Config.in | 10 ++++------ package/python-twisted/Config.in | 6 ++---- package/python-txdbus/Config.in | 5 ++++- package/python-txtorcon/Config.in | 7 ++----- 19 files changed, 52 insertions(+), 90 deletions(-)
package/docker-compose/Config.in +5 -3
@@ -5,7 +5,8 @@ config BR2_PACKAGE_DOCKER_COMPOSE depends on BR2_USE_WCHAR # python depends on BR2_TOOLCHAIN_HAS_THREADS # python depends on !BR2_STATIC_LIBS # python - depends on BR2_INSTALL_LIBSTDCPP # python-paramiko -> python-cryptography + # python-paramiko -> python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS select BR2_PACKAGE_PYTHON3 select BR2_PACKAGE_PYTHON_CACHED_PROPERTY # runtime select BR2_PACKAGE_PYTHON_DOCOPT # runtime
@@ -23,7 +24,8 @@ config BR2_PACKAGE_DOCKER_COMPOSE https://www.docker.com/ -comment "docker-compose needs a toolchain w/ C++, wchar, threads, dynamic library" +comment "docker-compose needs a toolchain w/ wchar, threads, dynamic library" depends on BR2_USE_MMU + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS depends on !BR2_USE_WCHAR || !BR2_TOOLCHAIN_HAS_THREADS || \ - BR2_STATIC_LIBS || !BR2_INSTALL_LIBSTDCPP + BR2_STATIC_LIBS
package/python-autobahn/Config.in +1 -5
@@ -1,6 +1,6 @@ config BR2_PACKAGE_PYTHON_AUTOBAHN bool "python-autobahn" - depends on BR2_INSTALL_LIBSTDCPP # python-cryptography -> python-pyasn + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS # python-cryptography depends on BR2_PACKAGE_PYTHON3 select BR2_PACKAGE_PYTHON_CFFI # runtime select BR2_PACKAGE_PYTHON_CRYPTOGRAPHY # runtime
@@ -11,7 +11,3 @@ config BR2_PACKAGE_PYTHON_AUTOBAHN framework. https://pypi.python.org/pypi/autobahn - -comment "python-autobahn needs a toolchain w/ C++" - depends on !BR2_INSTALL_LIBSTDCPP - depends on BR2_PACKAGE_PYTHON3
package/python-channels-redis/Config.in +2 -5
@@ -1,6 +1,7 @@ config BR2_PACKAGE_PYTHON_CHANNELS_REDIS bool "python-channels-redis" - depends on BR2_INSTALL_LIBSTDCPP # python-channels -> python-daphne + # python-channels -> python-daphne -> python-autobahn -> python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS depends on BR2_PACKAGE_PYTHON3 # python-channels select BR2_PACKAGE_PYTHON_AIOREDIS # runtime select BR2_PACKAGE_PYTHON_ASGIREF # runtime
@@ -12,7 +13,3 @@ config BR2_PACKAGE_PYTHON_CHANNELS_REDIS sharded configurations, as well as group support. http://github.com/django/channels_redis/ - -comment "python-channels-redis needs a toolchain w/ C++" - depends on !BR2_INSTALL_LIBSTDCPP - depends on BR2_PACKAGE_PYTHON3
package/python-channels/Config.in +2 -5
@@ -1,6 +1,7 @@ config BR2_PACKAGE_PYTHON_CHANNELS bool "python-channels" - depends on BR2_INSTALL_LIBSTDCPP # python-daphne -> python-autobahn -> python-cryptography + # python-daphne -> python-autobahn -> python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS depends on BR2_PACKAGE_PYTHON3 # python-daphne select BR2_PACKAGE_PYTHON_ASGIREF # runtime select BR2_PACKAGE_PYTHON_DAPHNE # runtime
@@ -14,7 +15,3 @@ config BR2_PACKAGE_PYTHON_CHANNELS own protocols and needs. http://github.com/django/channels - -comment "python-channels needs a toolchain w/ C++" - depends on !BR2_INSTALL_LIBSTDCPP - depends on BR2_PACKAGE_PYTHON3
package/python-crossbar/Config.in +1 -5
@@ -1,6 +1,6 @@ config BR2_PACKAGE_PYTHON_CROSSBAR bool "python-crossbar" - depends on BR2_INSTALL_LIBSTDCPP # python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS # python-cryptography depends on BR2_PACKAGE_PYTHON3 # All the following dependencies are runtime dependencies select BR2_PACKAGE_PYTHON_ATTRS
@@ -48,7 +48,3 @@ config BR2_PACKAGE_PYTHON_CROSSBAR components that can talk in real-time with each other. https://pypi.python.org/pypi/crossbar - -comment "python-crossbar needs a toolchain w/ C++" - depends on !BR2_INSTALL_LIBSTDCPP - depends on BR2_PACKAGE_PYTHON3
package/python-cryptography/Config.in +2 -12
@@ -1,21 +1,11 @@ config BR2_PACKAGE_PYTHON_CRYPTOGRAPHY bool "python-cryptography" - depends on BR2_PACKAGE_PYTHON3 # python-idna - depends on BR2_INSTALL_LIBSTDCPP # python-pyasn + depends on BR2_PACKAGE_PYTHON3 + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS select BR2_PACKAGE_OPENSSL select BR2_PACKAGE_PYTHON_CFFI # runtime - select BR2_PACKAGE_PYTHON_IDNA # runtime - select BR2_PACKAGE_PYTHON_ASN1CRYPTO # runtime - select BR2_PACKAGE_PYTHON3_PYEXPAT # runtime - select BR2_PACKAGE_PYTHON_SETUPTOOLS # runtime - select BR2_PACKAGE_PYTHON_SIX # runtime - select BR2_PACKAGE_PYTHON3_SSL # runtime help cryptography is a package designed to expose cryptographic primitives and recipes to Python developers. https://cryptography.io - -comment "python-cryptography needs a toolchain w/ C++" - depends on BR2_PACKAGE_PYTHON3 - depends on !BR2_INSTALL_LIBSTDCPP
package/python-cryptography/python-cryptography.hash +2 -3
@@ -1,6 +1,5 @@ -# md5, sha256 from https://pypi.org/pypi/cryptography/json -md5 e2ce2ec8a63965fad351f36ed70fde4b cryptography-3.3.2.tar.gz -sha256 5a60d3780149e13b7a6ff7ad6526b38846354d11a15e21068e57073e29e19bed cryptography-3.3.2.tar.gz +# Locally calculated after vendoring +sha256 196bba703cebc052a19f5353614fcfa9d680471990c10305f110adcc05744eeb cryptography-36.0.1.tar.gz # Locally computed sha256 checksums sha256 43dad2cc752ab721cd9a9f36ece70fb53ab7713551f2d3d8694d8e8c5a06d6e2 LICENSE sha256 aac73b3148f6d1d7111dbca32099f68d26c644c6813ae1e4f05f6579aa2663fe LICENSE.APACHE
package/python-cryptography/python-cryptography.mk +11 -3
@@ -4,14 +4,22 @@ # ################################################################################ -PYTHON_CRYPTOGRAPHY_VERSION = 3.3.2 +PYTHON_CRYPTOGRAPHY_VERSION = 36.0.1 PYTHON_CRYPTOGRAPHY_SOURCE = cryptography-$(PYTHON_CRYPTOGRAPHY_VERSION).tar.gz -PYTHON_CRYPTOGRAPHY_SITE = https://files.pythonhosted.org/packages/d4/85/38715448253404186029c575d559879912eb8a1c5d16ad9f25d35f7c4f4c +PYTHON_CRYPTOGRAPHY_SITE = https://files.pythonhosted.org/packages/f9/4b/1cf8e281f7ae4046a59e5e39dd7471d46db9f61bb564fddbff9084c4334f PYTHON_CRYPTOGRAPHY_SETUP_TYPE = setuptools PYTHON_CRYPTOGRAPHY_LICENSE = Apache-2.0 or BSD-3-Clause PYTHON_CRYPTOGRAPHY_LICENSE_FILES = LICENSE LICENSE.APACHE LICENSE.BSD PYTHON_CRYPTOGRAPHY_CPE_ID_VENDOR = cryptography_project PYTHON_CRYPTOGRAPHY_CPE_ID_PRODUCT = cryptography -PYTHON_CRYPTOGRAPHY_DEPENDENCIES = host-python-cffi openssl +PYTHON_CRYPTOGRAPHY_DEPENDENCIES = host-python-setuptools-rust host-python-cffi host-rustc +PYTHON_CRYPTOGRAPHY_ENV = CARGO_BUILD_TARGET="$(RUSTC_TARGET_NAME)" \ + PYO3_CROSS_LIB_DIR="$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)" \ + CARGO_HOME="$(HOST_DIR)/share/cargo" +# We need to vendor the Cargo crates at download time +PYTHON_CRYPTOGRAPHY_DOWNLOAD_POST_PROCESS = cargo +PYTHON_CRYPTOGRAPHY_DOWNLOAD_DEPENDENCIES = host-rustc +PYTHON_CRYPTOGRAPHY_DL_ENV = \ + BR_CARGO_MANIFEST_PATH=src/rust/Cargo.toml $(eval $(python-package))
package/python-daphne/Config.in +2 -5
@@ -1,6 +1,7 @@ config BR2_PACKAGE_PYTHON_DAPHNE bool "python-daphne" - depends on BR2_INSTALL_LIBSTDCPP # python-autobahn -> python-cryptography + # python-autobahn -> python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS depends on BR2_PACKAGE_PYTHON3 select BR2_PACKAGE_PYTHON_ASGIREF # runtime select BR2_PACKAGE_PYTHON_AUTOBAHN # runtime
@@ -14,7 +15,3 @@ config BR2_PACKAGE_PYTHON_DAPHNE versus HTTP endpoints. https://github.com/django/daphne - -comment "python-daphne needs a toolchain w/ C++" - depends on !BR2_INSTALL_LIBSTDCPP - depends on BR2_PACKAGE_PYTHON3
package/python-keyring/Config.in +2 -5
@@ -1,7 +1,8 @@ config BR2_PACKAGE_PYTHON_KEYRING bool "python-keyring" + # python-secretstorage -> python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS depends on BR2_PACKAGE_PYTHON3 - depends on BR2_INSTALL_LIBSTDCPP # python-secretstorage select BR2_PACKAGE_PYTHON_ENTRYPOINTS # runtime select BR2_PACKAGE_PYTHON_SECRETSTORAGE # runtime help
@@ -9,7 +10,3 @@ config BR2_PACKAGE_PYTHON_KEYRING system keyring service from Python. https://pypi.python.org/pypi/keyring - -comment "python-keyring needs a toolchain w/ C++" - depends on BR2_PACKAGE_PYTHON3 - depends on !BR2_INSTALL_LIBSTDCPP
package/python-opcua-asyncio/Config.in +1 -0
@@ -1,5 +1,6 @@ config BR2_PACKAGE_PYTHON_OPCUA_ASYNCIO bool "python-opcua-asyncio" + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS # python-cryptography depends on BR2_PACKAGE_PYTHON3 select BR2_PACKAGE_PYTHON_AIOFILES # runtime select BR2_PACKAGE_PYTHON_CRYPTOGRAPHY # runtime
package/python-paramiko/Config.in +2 -6
@@ -1,7 +1,7 @@ config BR2_PACKAGE_PYTHON_PARAMIKO bool "python-paramiko" - depends on BR2_PACKAGE_PYTHON3 # python-bcrypt - depends on BR2_INSTALL_LIBSTDCPP # python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS # python-cryptography + depends on BR2_PACKAGE_PYTHON3 # python-bcrypt, python-cryptography select BR2_PACKAGE_PYTHON_BCRYPT # runtime select BR2_PACKAGE_PYTHON_CRYPTOGRAPHY # runtime select BR2_PACKAGE_PYTHON_PYNACL # runtime
@@ -10,7 +10,3 @@ config BR2_PACKAGE_PYTHON_PARAMIKO SSH2 protocol library. https://github.com/paramiko/paramiko/ - -comment "python-paramiko needs a toolchain w/ C++" - depends on BR2_PACKAGE_PYTHON3 - depends on !BR2_INSTALL_LIBSTDCPP
package/python-pyopenssl/Config.in +2 -6
@@ -1,14 +1,10 @@ config BR2_PACKAGE_PYTHON_PYOPENSSL bool "python-pyopenssl" - depends on BR2_PACKAGE_PYTHON3 # python-cryptography -> python-idna - depends on BR2_INSTALL_LIBSTDCPP # python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS # python-cryptography + depends on BR2_PACKAGE_PYTHON3 # python-cryptography select BR2_PACKAGE_PYTHON_CRYPTOGRAPHY # runtime select BR2_PACKAGE_PYTHON_SIX # runtime help Python wrapper module around the OpenSSL library. https://github.com/pyca/pyopenssl - -comment "python-pyopenssl needs a toolchain w/ C++" - depends on BR2_PACKAGE_PYTHON3 - depends on !BR2_INSTALL_LIBSTDCPP
package/python-secretstorage/Config.in +1 -5
@@ -1,13 +1,9 @@ config BR2_PACKAGE_PYTHON_SECRETSTORAGE bool "python-secretstorage" depends on BR2_PACKAGE_PYTHON3 - depends on BR2_INSTALL_LIBSTDCPP # python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS # python-cryptography select BR2_PACKAGE_PYTHON_CRYPTOGRAPHY # runtime help Python bindings to FreeDesktop.org Secret Service API. https://github.com/mitya57/secretstorage - -comment "python-secretstorage needs a toolchain w/ C++" - depends on BR2_PACKAGE_PYTHON3 - depends on !BR2_INSTALL_LIBSTDCPP
package/python-service-identity/Config.in +4 -6
@@ -1,7 +1,9 @@ config BR2_PACKAGE_PYTHON_SERVICE_IDENTITY bool "python-service-identity" - depends on BR2_PACKAGE_PYTHON3 # python-pyopenssl -> python-cryptography -> python-idna - depends on BR2_INSTALL_LIBSTDCPP # python-pyopenssl + # python-pyopenssl -> python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS + # python-pyopenssl -> python-cryptography + depends on BR2_PACKAGE_PYTHON3 select BR2_PACKAGE_PYTHON_ATTRS # runtime select BR2_PACKAGE_PYTHON_CHARACTERISTIC # runtime select BR2_PACKAGE_PYTHON_PYASN1 # runtime
@@ -11,7 +13,3 @@ config BR2_PACKAGE_PYTHON_SERVICE_IDENTITY Service identity verification for pyOpenSSL. https://pypi.python.org/pypi/service_identity - -comment "python-service-identify needs a toolchain w/ C++" - depends on BR2_PACKAGE_PYTHON3 - depends on !BR2_INSTALL_LIBSTDCPP
package/python-treq/Config.in +4 -6
@@ -1,7 +1,9 @@ config BR2_PACKAGE_PYTHON_TREQ bool "python-treq" - depends on BR2_PACKAGE_PYTHON3 # python-idna - depends on BR2_INSTALL_LIBSTDCPP # python-pyopenssl + # python-pyopenssl -> python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS + # python-idna, python-pyopenssl -> python-cryptography + depends on BR2_PACKAGE_PYTHON3 select BR2_PACKAGE_PYTHON_ATTRS # runtime select BR2_PACKAGE_PYTHON_IDNA # runtime select BR2_PACKAGE_PYTHON_INCREMENTAL # runtime
@@ -16,7 +18,3 @@ config BR2_PACKAGE_PYTHON_TREQ API for making HTTP requests when using Twisted. https://github.com/twisted/treq - -comment "python-treq needs a toolchain w/ C++" - depends on BR2_PACKAGE_PYTHON3 - depends on !BR2_INSTALL_LIBSTDCPP
package/python-twisted/Config.in +2 -4
@@ -23,12 +23,10 @@ config BR2_PACKAGE_PYTHON_TWISTED_HTTP2 config BR2_PACKAGE_PYTHON_TWISTED_TLS bool "TLS support" - depends on BR2_INSTALL_LIBSTDCPP # python-{pyopenssl,service-identity} + # python-{pyopenssl,service-identity} + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS select BR2_PACKAGE_PYTHON_IDNA # runtime select BR2_PACKAGE_PYTHON_PYOPENSSL # runtime select BR2_PACKAGE_PYTHON_SERVICE_IDENTITY # runtime -comment "TLS support needs a toolchain w/ C++" - depends on !BR2_INSTALL_LIBSTDCPP - endif
package/python-txdbus/Config.in +4 -1
@@ -1,6 +1,9 @@ config BR2_PACKAGE_PYTHON_TXDBUS bool "python-txdbus" - depends on BR2_PACKAGE_PYTHON3 # python-twisted -> python-cryptography -> python-idna + # python-twisted -> python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS + # python-twisted -> python-cryptography + depends on BR2_PACKAGE_PYTHON3 select BR2_PACKAGE_PYTHON_TWISTED # runtime select BR2_PACKAGE_PYTHON_SIX # runtime help
package/python-txtorcon/Config.in +2 -5
@@ -1,7 +1,8 @@ config BR2_PACKAGE_PYTHON_TXTORCON bool "python-txtorcon" + # python-pyopenssl -> python-cryptography + depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS depends on BR2_PACKAGE_PYTHON3 # python-idna - depends on BR2_INSTALL_LIBSTDCPP # python-pyopenssl, python-service-identity select BR2_PACKAGE_PYTHON_AUTOMAT # runtime select BR2_PACKAGE_PYTHON_IDNA # runtime select BR2_PACKAGE_PYTHON_INCREMENTAL # runtime
@@ -14,7 +15,3 @@ config BR2_PACKAGE_PYTHON_TXTORCON configuration abstractions. https://github.com/meejah/txtorcon - -comment "python-txtorcon needs a toolchain w/ C++" - depends on BR2_PACKAGE_PYTHON3 - depends on !BR2_INSTALL_LIBSTDCPP
Signature · 6 lines
2.33.1 _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot
Quoted 40 lines · Thomas Petazzoni
Hello,
This series bumps the python-cryptography package to a new version. It is based on earlier work from James Hilliard at:
https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/But it did require quite some more work, which is presented in this patch series.
The unique challenge caused by this version bump is that python-cryptography now has some Rust code integrated, so suddenly this package, and all its reverse dependencies need to depend on Rust being available.
There is also a host variant of python-cryptography, which is used solely by ca-certificates, which has a significant number of reverse dependencies. To avoid adding Rust dependencies to these, we broke the dependency of ca-certificates on host-python-cryptography using a simple patch, and dropped host-python-cryptography altogether.
This patch series depends on the "Go/Cargo vendoring" patch series posted recently. Indeed, the Rust code in python-cryptography depends on other Cargo crates, which we don't want to download during the build, but really during the download step.
There is however still one problem with the last patch of the series, see below.
This series is available at:
https://github.com/tpetazzoni/buildroot/tree/python-cryptography
The patch series goes like this:
- PATCH 1 modifies ca-certificates to no longer need host-python-cryptography
- PATCH 2 drops host-python-cryptography itselef
Quoted 8 lines
- PATCH 3 and 4 fix some issues in package/rustc/ in the description of the ARMv5 and RISC-V 64-bit Rust support. We noticed this because Rust binaries were not working on ARMv5 ("Illegal instruction"), this was due to the fact that we were using the pre-compiled Rust standard library for ARMv6, due to a bogus architecture name. This is fixed by PATCH 3. And while reviewing this, we spotted a similar issue for RISC-V 64-bit, which is fixed in PATCH 4.
First four patches applied to master, with the fixup that you mentioned on
patch 3.
Regards,
Arnout
Quoted 136 lines
- PATCH 5 slightly improves the Cargo download post-process script to make it possible to pass a custom path to the Cargo.toml file, instead of assuming it's located at the root of the package source tree.
- PATCH 6 modifies the runtime test cases that involve python-cryptography directly or indirectly to use a glibc toolchain instead of the default uclibc toolchain. Indeed, Rust is for now not available on uClibc platforms. This could have been done in the patch doing the version bump, but the version bump patch is already quite large, and the update of the runtime tests could be done as a separate preparation step.
- PATCH 7 finally bumps python-cryptography
itself. python-cryptography itself requires a bit of trickery to leverage the Cargo download post-process logic, to vendor dependencies during the download step. Then the patch also touches the entire reverse dependency tree (sometimes up to 5 levels deep!). This patch should NOT be applied for now, due to issues (see below).This series was tested by building the following configuration:
BR2_arm=y
BR2_cortex_a9=y
BR2_ARM_ENABLE_VFP=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
#BR2_PACKAGE_BUSYBOXis not set
BR2_PACKAGE_PYTHON_CHANNELS_REDIS=y
BR2_PACKAGE_PYTHON_CROSSBAR=y
BR2_PACKAGE_PYTHON_KEYRING=y
BR2_PACKAGE_PYTHON_OPCUA_ASYNCIO=y
BR2_PACKAGE_PYTHON_TXDBUS=y
BR2_PACKAGE_DOCKER_COMPOSE=y
#BR2_TARGET_ROOTFS_TARis not setthat builds all affected packages.
It was also tested by running all Python runtime test cases:
https://gitlab.com/tpetazzoni/buildroot/-/pipelines/443445436The one thing that fails is when building python-cryptography on a target arch that is the same as the host. This can be reproduced by running the tests.package.test_docker_compose.TestDockerCompose test case.
We get this:
Running `/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build`error: failed to run custom build command for
proc-macro-hack v0.5.19Caused by:
process didn't exit successfully:/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build(exi
t status: 1)
--- stderr/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build:/lib/x86_64-linux-gnu/libc.so.6: versio
n `GLIBC_2.33' not found (required by/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build)
/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build:/lib/x86_64-linux-gnu/libc.so.6: versio
n `GLIBC_2.32' not found (required by/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build)
/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build:/lib/x86_64-linux-gnu/libc.so.6: versio
n `GLIBC_2.34' not found (required by/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build)
warning: build failed, waiting for other jobs to finish...
error: build failedWhich looks terribly similar to the issue on ripgrep which was fixed by commit
4b2be770b8a853a7dd97b5788d837f0d84923fa1. So I tried doing the same, i.e passing:__CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly" \ CARGO_TARGET_APPLIES_TO_HOST="false"in the environment, and -Z target-applies-to-host to cargo (by passing it to the args argument of the RustExtension class in python-cryptography's
setup.py). But cargo doesn't like that, the build log goes like this:Quoted 1 line
Quoted 1 line
Quoted 1 line
python-cryptography 36.0.1 BuildingESC[27m
... calling
setup.py...
cargo rustc --lib --manifest-pathsrc/rust/Cargo.toml--features pyo3/abi3-py36 pyo3/extension-module --target x86_64-unknown-linux-gnu -Z target-applies-to-host --release --verbose -- --crate-type cdylib
...
error: unknown-Zflag specified: target-applies-to-hostThe difference that I see with the cargo infrastructure is that this is not using "cargo build" but "cargo rustc". I would really appreciate some help on solving this (last?) issue.
Best regards,
Thomas Petazzoni
Thomas Petazzoni (7):
package/ca-certificates: remove dependency on host-python-cryptography package/python-cryptography: drop host variant package/rustc: fixBR2_PACKAGE_HOST_RUSTC_ARCHfor ARMv5 package/rustc: fix the riscv64gc architecture handling support/download/cargo-post-process: make manifest path configurable support/testing/tests: switch python-cryptography related tests to
glibc
DO-NOT-APPLY package/python-cryptography: bump version to 36.0.1...2pem.py-make-cryptography-module-opt.patch | 59 +++++++++++++++++++ package/ca-certificates/ca-certificates.mk | 2 +- package/docker-compose/Config.in | 8 ++- package/python-autobahn/Config.in | 6 +- package/python-channels-redis/Config.in | 7 +-- package/python-channels/Config.in | 7 +-- package/python-crossbar/Config.in | 6 +- package/python-cryptography/Config.in | 14 +---- .../python-cryptography.hash | 5 +- .../python-cryptography.mk | 20 ++++--- package/python-daphne/Config.in | 7 +-- package/python-keyring/Config.in | 7 +-- package/python-opcua-asyncio/Config.in | 1 + package/python-paramiko/Config.in | 8 +-- package/python-pyopenssl/Config.in | 8 +-- package/python-secretstorage/Config.in | 6 +- package/python-service-identity/Config.in | 10 ++-- package/python-treq/Config.in | 10 ++-- package/python-twisted/Config.in | 6 +- package/python-txdbus/Config.in | 5 +- package/python-txtorcon/Config.in | 7 +-- package/rustc/Config.in.host | 4 +- support/download/cargo-post-process | 2 +- .../tests/package/test_python_autobahn.py | 10 +++- .../tests/package/test_python_crossbar.py | 10 +++- .../tests/package/test_python_cryptography.py | 10 +++- .../package/test_python_service_identity.py | 10 +++- .../testing/tests/package/test_python_treq.py | 10 +++- .../tests/package/test_python_txtorcon.py | 10 +++- 29 files changed, 170 insertions(+), 105 deletions(-)
Quoted 2 lines
create mode 100644
package/ca-certificates/0001-mozilla-certdata2pem.py-make-cryptography-module-opt.patch
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Hello,
Quoted 9 lines · On Fri, 7 Jan 2022 18:13:10 +0100 Thomas Petazzoni…
Thomas Petazzoni (7):
package/ca-certificates: remove dependency on host-python-cryptography package/python-cryptography: drop host variant package/rustc: fixBR2_PACKAGE_HOST_RUSTC_ARCHfor ARMv5 package/rustc: fix the riscv64gc architecture handling support/download/cargo-post-process: make manifest path configurable support/testing/tests: switch python-cryptography related tests to
glibc
DO-NOT-APPLY package/python-cryptography: bump version to 36.0.1
Thanks to the help of James, we were able to figure out the last issues, so I pushed those last 3 patches, but with a number of additional preparation patches:
01764876e3d4e7ab95c8365f1c83e6c12d49757c package/python-cryptography: bump version to 36.0.1 b6378631c2609742382984f6f7b93c1d9d2cdb78 package/pkg-cargo.mk: use CARGO_TARGET_<target>_LINKER instead of cargo config file 91a5a2d317a02d7fb41e706a490fe5063039d55c package/pkg-cargo.mk: use CARGO_BUILD_TARGET instead of --target 98c7ea3ddf11d08acd11e30be36b4b56c682d239 package/pkg-cargo.mk: enable -Z target-applies-to-host via env 0894937f7fc4ac9d04d65bc6cdd5fc1927483976 package/pkg-cargo.mk: split PKG_CARGO_ENV into target/host variables e5ead094e18cd4a5a9a22b69a42e687f43ee8a8b support/testing/tests: switch python-cryptography related tests to glibc 36e8743da181420775b79d5be9bbe872b00f2c01 support/download/cargo-post-process: make manifest path configurable
Best regards,
Thomas
Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin Embedded Linux and Kernel engineering and training https://bootlin.com _______________________________________________ buildroot mailing list [email protected] https://lists.buildroot.org/mailman/listinfo/buildroot