Skip to contentSkip to conversation

Bump of python-cryptography and other Rust related fun

PATCH In review Thomas Petazzoni 3 participants 12 messages Jan 7, 2022 – Jan 9, 2022

lore
Share & tools
lore
#0 lore raw

cover Bump of python-cryptography and other Rust related fun

Hello,

This series bumps the python-cryptography package to a new version. It is based on earlier work from James Hilliard at:

https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

But it did require quite some more work, which is presented in this patch series.

The unique challenge caused by this version bump is that python-cryptography now has some Rust code integrated, so suddenly this package, and all its reverse dependencies need to depend on Rust being available.

There is also a host variant of python-cryptography, which is used solely by ca-certificates, which has a significant number of reverse dependencies. To avoid adding Rust dependencies to these, we broke the dependency of ca-certificates on host-python-cryptography using a simple patch, and dropped host-python-cryptography altogether.

This patch series depends on the "Go/Cargo vendoring" patch series posted recently. Indeed, the Rust code in python-cryptography depends on other Cargo crates, which we don't want to download during the build, but really during the download step.

There is however still one problem with the last patch of the series, see below.

This series is available at:

https://github.com/tpetazzoni/buildroot/tree/python-cryptography

The patch series goes like this:

- PATCH 1 modifies ca-certificates to no longer need host-python-cryptography

- PATCH 2 drops host-python-cryptography itselef

- PATCH 3 and 4 fix some issues in package/rustc/ in the description of the ARMv5 and RISC-V 64-bit Rust support. We noticed this because Rust binaries were not working on ARMv5 ("Illegal instruction"), this was due to the fact that we were using the pre-compiled Rust standard library for ARMv6, due to a bogus architecture name. This is fixed by PATCH 3. And while reviewing this, we spotted a similar issue for RISC-V 64-bit, which is fixed in PATCH 4.

- PATCH 5 slightly improves the Cargo download post-process script to make it possible to pass a custom path to the Cargo.toml file, instead of assuming it's located at the root of the package source tree.

- PATCH 6 modifies the runtime test cases that involve python-cryptography directly or indirectly to use a glibc toolchain instead of the default uclibc toolchain. Indeed, Rust is for now not available on uClibc platforms. This could have been done in the patch doing the version bump, but the version bump patch is already quite large, and the update of the runtime tests could be done as a separate preparation step.

- PATCH 7 finally bumps python-cryptography
itself. python-cryptography itself requires a bit of trickery to leverage the Cargo download post-process logic, to vendor dependencies during the download step. Then the patch also touches the entire reverse dependency tree (sometimes up to 5 levels deep!). This patch should NOT be applied for now, due to issues (see below).

This series was tested by building the following configuration:

BR2_arm=y
BR2_cortex_a9=y
BR2_ARM_ENABLE_VFP=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
# BR2_PACKAGE_BUSYBOX is not set
BR2_PACKAGE_PYTHON_CHANNELS_REDIS=y
BR2_PACKAGE_PYTHON_CROSSBAR=y
BR2_PACKAGE_PYTHON_KEYRING=y
BR2_PACKAGE_PYTHON_OPCUA_ASYNCIO=y
BR2_PACKAGE_PYTHON_TXDBUS=y
BR2_PACKAGE_DOCKER_COMPOSE=y
# BR2_TARGET_ROOTFS_TAR is not set

that builds all affected packages.

It was also tested by running all Python runtime test cases:

https://gitlab.com/tpetazzoni/buildroot/-/pipelines/443445436

The one thing that fails is when building python-cryptography on a target arch that is the same as the host. This can be reproduced by running the tests.package.test_docker_compose.TestDockerCompose test case.

We get this:

     Running `/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build`

error: failed to run custom build command for proc-macro-hack v0.5.19

Caused by:
process didn't exit successfully: /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build (exi
t status: 1)
--- stderr /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: versio
n `GLIBC_2.33' not found (required by /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build)
/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: versio
n `GLIBC_2.32' not found (required by /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build)
/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: versio
n `GLIBC_2.34' not found (required by /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build)
warning: build failed, waiting for other jobs to finish...
error: build failed

Which looks terribly similar to the issue on ripgrep which was fixed by commit 4b2be770b8a853a7dd97b5788d837f0d84923fa1. So I tried doing the same, i.e passing:

	__CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly" \
	CARGO_TARGET_APPLIES_TO_HOST="false"

in the environment, and -Z target-applies-to-host to cargo (by passing it to the args argument of the RustExtension class in python-cryptography's setup.py). But cargo doesn't like that, the build log goes like this:

Quoted 1 line
Quoted 1 line
Quoted 1 line

python-cryptography 36.0.1 BuildingESC[27m

... calling setup.py ...
cargo rustc --lib --manifest-path src/rust/Cargo.toml --features pyo3/abi3-py36 pyo3/extension-module --target x86_64-unknown-linux-gnu -Z target-applies-to-host --release --verbose -- --crate-type cdylib
...
error: unknown -Z flag specified: target-applies-to-host

The difference that I see with the cargo infrastructure is that this is not using "cargo build" but "cargo rustc". I would really appreciate some help on solving this (last?) issue.

Best regards,

Thomas Petazzoni

Thomas Petazzoni (7):
package/ca-certificates: remove dependency on host-python-cryptography package/python-cryptography: drop host variant package/rustc: fix BR2_PACKAGE_HOST_RUSTC_ARCH for ARMv5 package/rustc: fix the riscv64gc architecture handling support/download/cargo-post-process: make manifest path configurable support/testing/tests: switch python-cryptography related tests to
glibc
DO-NOT-APPLY package/python-cryptography: bump version to 36.0.1

 ...2pem.py-make-cryptography-module-opt.patch | 59 +++++++++++++++++++
 package/ca-certificates/ca-certificates.mk    |  2 +-
 package/docker-compose/Config.in              |  8 ++-
 package/python-autobahn/Config.in             |  6 +-
 package/python-channels-redis/Config.in       |  7 +--
 package/python-channels/Config.in             |  7 +--
 package/python-crossbar/Config.in             |  6 +-
 package/python-cryptography/Config.in         | 14 +----
 .../python-cryptography.hash                  |  5 +-
 .../python-cryptography.mk                    | 20 ++++---
 package/python-daphne/Config.in               |  7 +--
 package/python-keyring/Config.in              |  7 +--
 package/python-opcua-asyncio/Config.in        |  1 +
 package/python-paramiko/Config.in             |  8 +--
 package/python-pyopenssl/Config.in            |  8 +--
 package/python-secretstorage/Config.in        |  6 +-
 package/python-service-identity/Config.in     | 10 ++--
 package/python-treq/Config.in                 | 10 ++--
 package/python-twisted/Config.in              |  6 +-
 package/python-txdbus/Config.in               |  5 +-
 package/python-txtorcon/Config.in             |  7 +--
 package/rustc/Config.in.host                  |  4 +-
 support/download/cargo-post-process           |  2 +-
 .../tests/package/test_python_autobahn.py     | 10 +++-
 .../tests/package/test_python_crossbar.py     | 10 +++-
 .../tests/package/test_python_cryptography.py | 10 +++-
 .../package/test_python_service_identity.py   | 10 +++-
 .../testing/tests/package/test_python_treq.py | 10 +++-
 .../tests/package/test_python_txtorcon.py     | 10 +++-
 29 files changed, 170 insertions(+), 105 deletions(-)
 create mode 100644 package/ca-certificates/0001-mozilla-certdata2pem.py-make-cryptography-module-opt.patch
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#1 lore raw

PATCH 1/7 package/ca-certificates: remove dependency on host-python-cryptography

The host-python-cryptography module is only used by ca-certificates for a check of the expiration date of certificates, which is only a warning not even causing the build to abort, i.e something that Buildroot users are most likely never going to see.

Since the host-python-cryptography dependency would soon require a dependency on rust, it's a lot simpler to just patch the certdata2pem.py script to no longer require cryptography, but only make use of it if available.

---
 ...2pem.py-make-cryptography-module-opt.patch | 59 +++++++++++++++++++
 package/ca-certificates/ca-certificates.mk    |  2 +-
 2 files changed, 60 insertions(+), 1 deletion(-)
 create mode 100644 package/ca-certificates/0001-mozilla-certdata2pem.py-make-cryptography-module-opt.patch
package/ca-certificates/0001-mozilla-certdata2pem.py-make-cryptography-module-opt.patch new file +59 -0
new file mode 100644
index 0000000000..b76c1bfd7f
@@ -0,0 +1,59 @@
+From bf18b564122e8f976681a2398862fde1eafd84ba Mon Sep 17 00:00:00 2001
+From: Thomas Petazzoni <[email protected]>
+Date: Thu, 6 Jan 2022 23:15:00 +0100
+Subject: [PATCH] mozilla/certdata2pem.py: make cryptography module optional
+
+The Python cryptography module is only used to verify if trusted
+certificates have expired, but this is only a warning. For some build
+systems and distributions, providing Python cryptography is costly,
+especially since it's now partly written in Rust.
+
+As the check is only a warning, it's anyway going to be overlooked by
+most people. This commit changes the check to be optional: if the
+cryptography Python module is there, we perform the check, otherwise
+the check is skipped.
+
+Signed-off-by: Thomas Petazzoni <[email protected]>
+---
+ mozilla/certdata2pem.py | 18 ++++++++++--------
+ 1 file changed, 10 insertions(+), 8 deletions(-)
+
+diff --git a/mozilla/certdata2pem.py b/mozilla/certdata2pem.py
+index ede23d4..a6261f8 100644
+--- a/mozilla/certdata2pem.py
++++ b/mozilla/certdata2pem.py
+@@ -28,9 +28,6 @@ import sys
+ import textwrap
+ import io
+ 
+-from cryptography import x509
+-
+-
+ objects = []
+ 
+ # Dirty file parser.
+@@ -122,11 +119,16 @@ for obj in objects:
+         if not obj['CKA_LABEL'] in trust or not trust[obj['CKA_LABEL']]:
+             continue
+ 
+-        cert = x509.load_der_x509_certificate(obj['CKA_VALUE'])
+-        if cert.not_valid_after < datetime.datetime.now():
+-            print('!'*74)
+-            print('Trusted but expired certificate found: %s' % obj['CKA_LABEL'])
+-            print('!'*74)
++        try:
++            from cryptography import x509
++
++            cert = x509.load_der_x509_certificate(obj['CKA_VALUE'])
++            if cert.not_valid_after < datetime.datetime.now():
++                print('!'*74)
++                print('Trusted but expired certificate found: %s' % obj['CKA_LABEL'])
++                print('!'*74)
++        except ImportError:
++            pass
+ 
+         bname = obj['CKA_LABEL'][1:-1].replace('/', '_')\
+                                       .replace(' ', '_')\
+-- 
+2.33.1
+
package/ca-certificates/ca-certificates.mk +1 -1
index 7084ab781e..0b6962ab7b 100644
@@ -7,7 +7,7 @@
 CA_CERTIFICATES_VERSION = 20211016
 CA_CERTIFICATES_SOURCE = ca-certificates_$(CA_CERTIFICATES_VERSION).tar.xz
 CA_CERTIFICATES_SITE = https://snapshot.debian.org/archive/debian/20211022T144903Z/pool/main/c/ca-certificates
-CA_CERTIFICATES_DEPENDENCIES = host-openssl host-python3 host-python-cryptography
+CA_CERTIFICATES_DEPENDENCIES = host-openssl host-python3
 CA_CERTIFICATES_LICENSE = GPL-2.0+ (script), MPL-2.0 (data)
 CA_CERTIFICATES_LICENSE_FILES = debian/copyright
 
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#2 lore raw

PATCH 2/7 package/python-cryptography: drop host variant

In reply to Thomas Petazzoni

The host variant was only needed by ca-certificates, which no longer needs it, so drop this variant.

---
 package/python-cryptography/python-cryptography.mk | 6 ------
 1 file changed, 6 deletions(-)
package/python-cryptography/python-cryptography.mk +0 -6
index 081dfc06a0..9c00efb44f 100644
@@ -13,11 +13,5 @@ PYTHON_CRYPTOGRAPHY_LICENSE_FILES = LICENSE LICENSE.APACHE LICENSE.BSD
 PYTHON_CRYPTOGRAPHY_CPE_ID_VENDOR = cryptography_project
 PYTHON_CRYPTOGRAPHY_CPE_ID_PRODUCT = cryptography
 PYTHON_CRYPTOGRAPHY_DEPENDENCIES = host-python-cffi openssl
-HOST_PYTHON_CRYPTOGRAPHY_NEEDS_HOST_PYTHON = python3
-HOST_PYTHON_CRYPTOGRAPHY_DEPENDENCIES = \
-	host-python3-cffi \
-	host-python3-six \
-	host-openssl
 
 $(eval $(python-package))
-$(eval $(host-python-package))
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#3 lore raw

PATCH 3/7 package/rustc: fix BR2_PACKAGE_HOST_RUSTC_ARCH for ARMv5

In reply to Thomas Petazzoni

BR2_PACKAGE_HOST_RUSTC_ARCH only had a special value for BR2_ARM_CPU_ARMV7A, but it also needs a special value for BR2_ARM_CPU_ARMV5, as the pre-compiled Rust standard library for ARMv5TE is identified with the "armv5te" architecture name, see
https://doc.rust-lang.org/nightly/rustc/platform-support.html.

We noticed this because Rust binaries wouldn't work on an ARMv5 platform (Illegal instruction). This was due to the usage of the arm-unknown-linux-gnueabi variant of the Rust standard library, which is for ARMv6. Thanks to this commit, we correctly use the armv5te-unknown-linux-gnueabi variant, and Rust binaries work properly on ARMv5TE.

---
 package/rustc/Config.in.host | 1 +
 1 file changed, 1 insertion(+)
package/rustc/Config.in.host +1 -0
index 90ea56f626..05324ad755 100644
@@ -102,6 +102,7 @@ config BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
 
 config BR2_PACKAGE_HOST_RUSTC_ARCH
 	string
+	default "armv5te" if BR2_ARM_CPU_ARMV5
 	default "armv7"  if BR2_ARM_CPU_ARMV7A
 	default BR2_ARCH if !BR2_ARM_CPU_ARMV7A
 
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#4 lore raw

Re: [Buildroot] [PATCH 3/7] package/rustc: fix BR2_PACKAGE_HOST_RUSTC_ARCH for ARMv5

Quoted 23 lines · On Fri, 7 Jan 2022 18:13:13 +0100 Thomas Petazzoni…

BR2_PACKAGE_HOST_RUSTC_ARCH only had a special value for BR2_ARM_CPU_ARMV7A, but it also needs a special value for BR2_ARM_CPU_ARMV5, as the pre-compiled Rust standard library for ARMv5TE is identified with the "armv5te" architecture name, see
https://doc.rust-lang.org/nightly/rustc/platform-support.html.

We noticed this because Rust binaries wouldn't work on an ARMv5 platform (Illegal instruction). This was due to the usage of the arm-unknown-linux-gnueabi variant of the Rust standard library, which is for ARMv6. Thanks to this commit, we correctly use the armv5te-unknown-linux-gnueabi variant, and Rust binaries work properly on ARMv5TE.

---
 package/rustc/Config.in.host | 1 +
 1 file changed, 1 insertion(+)
package/rustc/Config.in.host +0 -0
index 90ea56f626..05324ad755 100644

package/rustc/Config.in.host config BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS · line 107

Quoted 6 lines
package/rustc/Config.in.host +1 -0
@@ -102,6 +102,7 @@ config BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
 
 config BR2_PACKAGE_HOST_RUSTC_ARCH
 	string
+	default "armv5te" if BR2_ARM_CPU_ARMV5
 	default "armv7"  if BR2_ARM_CPU_ARMV7A
 	default BR2_ARCH if !BR2_ARM_CPU_ARMV7A

I need to drop the "if !BR2_ARM_CPU_ARMV7A" here. Will do in the next iteration, or can be done when applying if there are no other comments.

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#5 lore raw

PATCH 4/7 package/rustc: fix the riscv64gc architecture handling

In reply to Thomas Petazzoni

riscv64gc was made available for BR2_riscv, but RISC-V can be 32-bit or 64-bit, so we need to restrict it to BR2_RISCV_64. There's no need to keep the BR2_riscv dependency, as BR2_RISCV_64 can only be true when BR2_riscv is true.

Also, BR2_PACKAGE_HOST_RUSTC_ARCH needs to be set to riscv64gc to allow rust-bin to download its pre-compiled standard library correctly.

---
 package/rustc/Config.in.host | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
package/rustc/Config.in.host +2 -1
index 05324ad755..8c8b75c0a2 100644
@@ -49,7 +49,7 @@ config BR2_PACKAGE_HOST_RUSTC_TARGET_TIER2_HOST_TOOLS_PLATFORMS
 	default y if BR2_powerpc64le && BR2_TOOLCHAIN_USES_GLIBC
 	# riscv64gc-unknown-linux-gnu
 	# "g" stands for imafd, and we also need "c".
-	default y if BR2_riscv && BR2_RISCV_ISA_RVI && BR2_RISCV_ISA_RVM && \
+	default y if BR2_RISCV_64 && BR2_RISCV_ISA_RVI && BR2_RISCV_ISA_RVM && \
 			BR2_RISCV_ISA_RVA && BR2_RISCV_ISA_RVF && \
 			BR2_RISCV_ISA_RVD && BR2_RISCV_ISA_RVC && \
 			BR2_TOOLCHAIN_USES_GLIBC
@@ -104,6 +104,7 @@ config BR2_PACKAGE_HOST_RUSTC_ARCH
 	string
 	default "armv5te" if BR2_ARM_CPU_ARMV5
 	default "armv7"  if BR2_ARM_CPU_ARMV7A
+	default "riscv64gc" if BR2_RISCV_64
 	default BR2_ARCH if !BR2_ARM_CPU_ARMV7A
 
 config BR2_PACKAGE_HOST_RUSTC_ABI
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
Reviewed-by on v1 4/7
#6 lore raw

Re: [Buildroot] [PATCH 4/7] package/rustc: fix the riscv64gc architecture handling

Quoted 11 lines · Thomas Petazzoni

riscv64gc was made available for BR2_riscv, but RISC-V can be 32-bit or 64-bit, so we need to restrict it to BR2_RISCV_64. There's no need to keep the BR2_riscv dependency, as BR2_RISCV_64 can only be true when BR2_riscv is true.

Also, BR2_PACKAGE_HOST_RUSTC_ARCH needs to be set to riscv64gc to allow rust-bin to download its pre-compiled standard library correctly.

Alistair

Quoted 28 lines
---
 package/rustc/Config.in.host | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
package/rustc/Config.in.host +2 -2
index 05324ad755..8c8b75c0a2 100644
@@ -49,7 +49,7 @@ config BR2_PACKAGE_HOST_RUSTC_TARGET_TIER2_HOST_TOOLS_PLATFORMS
        default y if BR2_powerpc64le && BR2_TOOLCHAIN_USES_GLIBC
        # riscv64gc-unknown-linux-gnu
        # "g" stands for imafd, and we also need "c".
-       default y if BR2_riscv && BR2_RISCV_ISA_RVI && BR2_RISCV_ISA_RVM && \
+       default y if BR2_RISCV_64 && BR2_RISCV_ISA_RVI && BR2_RISCV_ISA_RVM && \
                        BR2_RISCV_ISA_RVA && BR2_RISCV_ISA_RVF && \
                        BR2_RISCV_ISA_RVD && BR2_RISCV_ISA_RVC && \
                        BR2_TOOLCHAIN_USES_GLIBC
@@ -104,6 +104,7 @@ config BR2_PACKAGE_HOST_RUSTC_ARCH
        string
        default "armv5te" if BR2_ARM_CPU_ARMV5
        default "armv7"  if BR2_ARM_CPU_ARMV7A
+       default "riscv64gc" if BR2_RISCV_64
        default BR2_ARCH if !BR2_ARM_CPU_ARMV7A
 
 config BR2_PACKAGE_HOST_RUSTC_ABI
--

2.33.1

Quoted 4 lines

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#7 lore raw

PATCH 5/7 support/download/cargo-post-process: make manifest path configurable

In reply to Thomas Petazzoni

In most pure Rust packages, the Cargo.toml manifest is at the root directory, which is why we could call "cargo vendor" without specifying the path of the manifest.

However, other packages, such as python-cryptography, which have parts implemented in Rust, have their Cargo.toml located in a specific subdirectory.

This commit extends the cargo-post-process download script to understand a BR_CARGO_MANIFEST_PATH environment variable, which allows a package to pass the location of the Cargo.toml file. If not passed, "Cargo.toml" is used, preserving the existing behavior for other packages.

---
 support/download/cargo-post-process | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
support/download/cargo-post-process +1 -1
index 25dbbedbfa..51d9dd24c1 100755
@@ -22,7 +22,7 @@ post_process_unpack ${base_name} ${output}
 
 # Do the Cargo vendoring
 pushd ${base_name} > /dev/null
-cargo vendor --locked VENDOR
+cargo vendor --manifest-path ${BR_CARGO_MANIFEST_PATH-Cargo.toml} --locked VENDOR
 
 # Create the local .cargo/config with vendor info
 mkdir -p .cargo/
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#8 lore raw

PATCH 6/7 support/testing/tests: switch python-cryptography related tests to glibc

In reply to Thomas Petazzoni

In a follow-up commit, we are about to bump python-cryptography to a new version, which has the interesting charateristic of using Rust code. This means python-cryptography will now only be available on platforms supported by Rust, which for now excludes uclibc-based configurations (none of the Rust Tier1/Tier2 platforms use uClibc, there is some uClibc support in Tier3 platforms but they have not been added to Buildroot for now).

So in preparation for this bump, we switch the few test cases of Python packages that directly or indirectly use python-cryptography to use a glibc toolchain. Another impacted test case is the docker-compose test case, but it already uses a glibc toolchain;

---
 support/testing/tests/package/test_python_autobahn.py  | 10 +++++++++-
 support/testing/tests/package/test_python_crossbar.py  | 10 +++++++++-
 .../testing/tests/package/test_python_cryptography.py  | 10 +++++++++-
 .../tests/package/test_python_service_identity.py      | 10 +++++++++-
 support/testing/tests/package/test_python_treq.py      | 10 +++++++++-
 support/testing/tests/package/test_python_txtorcon.py  | 10 +++++++++-
 6 files changed, 54 insertions(+), 6 deletions(-)
support/testing/tests/package/test_python_autobahn.py +9 -1
index a237ce465d..6c1678a615 100644
@@ -3,9 +3,17 @@ from tests.package.test_python import TestPythonPackageBase
 
 class TestPythonPy3Autobahn(TestPythonPackageBase):
     __test__ = True
-    config = TestPythonPackageBase.config + \
+    # Need to use a different toolchain than the default due to
+    # python-cryptography using Rust (not available with uclibc)
+    config = \
         """
+        BR2_arm=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_AUTOBAHN=y
+        BR2_TARGET_ROOTFS_CPIO=y
+        # BR2_TARGET_ROOTFS_TAR is not set
         """
     sample_scripts = ["tests/package/sample_python_autobahn.py"]
support/testing/tests/package/test_python_crossbar.py +9 -1
index d7843ddd8b..83649aeec2 100644
@@ -3,10 +3,18 @@ from tests.package.test_python import TestPythonPackageBase
 
 class TestPythonPy3Crossbar(TestPythonPackageBase):
     __test__ = True
-    config = TestPythonPackageBase.config + \
+    # Need to use a different toolchain than the default due to
+    # python-cryptography using Rust (not available with uclibc)
+    config = \
         """
+        BR2_arm=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_CROSSBAR=y
+        BR2_TARGET_ROOTFS_CPIO=y
+        # BR2_TARGET_ROOTFS_TAR is not set
         """
     sample_scripts = ["tests/package/sample_python_crossbar.py"]
     timeout = 60
support/testing/tests/package/test_python_cryptography.py +9 -1
index 62f0ca0def..bf11c02e5c 100644
@@ -3,10 +3,18 @@ from tests.package.test_python import TestPythonPackageBase
 
 class TestPythonPy3Cryptography(TestPythonPackageBase):
     __test__ = True
-    config = TestPythonPackageBase.config + \
+    # Need to use a different toolchain than the default due to
+    # python-cryptography using Rust (not available with uclibc)
+    config = \
         """
+        BR2_arm=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_CRYPTOGRAPHY=y
+        BR2_TARGET_ROOTFS_CPIO=y
+        # BR2_TARGET_ROOTFS_TAR is not set
         """
     sample_scripts = ["tests/package/sample_python_cryptography.py"]
     timeout = 40
support/testing/tests/package/test_python_service_identity.py +9 -1
index 09cfbe0a26..dbad5f931f 100644
@@ -3,10 +3,18 @@ from tests.package.test_python import TestPythonPackageBase
 
 class TestPythonPy3ServiceIdentity(TestPythonPackageBase):
     __test__ = True
-    config = TestPythonPackageBase.config + \
+    # Need to use a different toolchain than the default due to
+    # python-cryptography using Rust (not available with uclibc)
+    config = \
         """
+        BR2_arm=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_SERVICE_IDENTITY=y
+        BR2_TARGET_ROOTFS_CPIO=y
+        # BR2_TARGET_ROOTFS_TAR is not set
         """
     sample_scripts = ["tests/package/sample_python_service_identity.py"]
     timeout = 30
support/testing/tests/package/test_python_treq.py +9 -1
index 307c9932b2..515f1ea304 100644
@@ -14,8 +14,16 @@ class TestPythonTreq(TestPythonPackageBase):
 
 class TestPythonPy3Treq(TestPythonTreq):
     __test__ = True
-    config = TestPythonTreq.config + \
+    # Need to use a different toolchain than the default due to
+    # python-cryptography using Rust (not available with uclibc)
+    config = \
         """
+        BR2_arm=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_TREQ=y
+        BR2_TARGET_ROOTFS_CPIO=y
+        # BR2_TARGET_ROOTFS_TAR is not set
         """
support/testing/tests/package/test_python_txtorcon.py +9 -1
index 8e74190ee1..6f0c4ca1ed 100644
@@ -3,10 +3,18 @@ from tests.package.test_python import TestPythonPackageBase
 
 class TestPythonPy3Txtorcon(TestPythonPackageBase):
     __test__ = True
-    config = TestPythonPackageBase.config + \
+    # Need to use a different toolchain than the default due to
+    # python-cryptography using Rust (not available with uclibc)
+    config = \
         """
+        BR2_arm=y
+        BR2_TOOLCHAIN_EXTERNAL=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
+        BR2_TOOLCHAIN_EXTERNAL_BOOTLIN_ARMV5_EABI_GLIBC_STABLE=y
         BR2_PACKAGE_PYTHON3=y
         BR2_PACKAGE_PYTHON_TXTORCON=y
+        BR2_TARGET_ROOTFS_CPIO=y
+        # BR2_TARGET_ROOTFS_TAR is not set
         """
     sample_scripts = ["tests/package/sample_python_txtorcon.py"]
     timeout = 30
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#9 lore raw

PATCH 7/7 DO-NOT-APPLY package/python-cryptography: bump version to 36.0.1

In reply to Thomas Petazzoni

This version bump requires significant changes because python-cryptography is now partially implemented in Rust.

This means that:

- The C++ dependency is no longer needed.

- We need to ensure we are on an architecture where Rust is available (BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS)

- Almost all Python dependencies are no longer relevant, except for the python-cffi.

- A number of magic environment variables are needed to make the Rust part build correctly (CARGO_BUILD_TARGET, PYO3_CROSS_LIB_DIR, CARGO_HOME)

- We need to invoke the "cargo" download post-process hook to vendor the Cargo dependencies at download time.

- We need to propagate to relatively significant reverse dependency tree the changes of dependencies on python-cryptography.

DO-NOT-APPLY: this commit is currently broken when host arch == target arch.

---
 package/docker-compose/Config.in                   |  8 +++++---
 package/python-autobahn/Config.in                  |  6 +-----
 package/python-channels-redis/Config.in            |  7 ++-----
 package/python-channels/Config.in                  |  7 ++-----
 package/python-crossbar/Config.in                  |  6 +-----
 package/python-cryptography/Config.in              | 14 ++------------
 .../python-cryptography/python-cryptography.hash   |  5 ++---
 package/python-cryptography/python-cryptography.mk | 14 +++++++++++---
 package/python-daphne/Config.in                    |  7 ++-----
 package/python-keyring/Config.in                   |  7 ++-----
 package/python-opcua-asyncio/Config.in             |  1 +
 package/python-paramiko/Config.in                  |  8 ++------
 package/python-pyopenssl/Config.in                 |  8 ++------
 package/python-secretstorage/Config.in             |  6 +-----
 package/python-service-identity/Config.in          | 10 ++++------
 package/python-treq/Config.in                      | 10 ++++------
 package/python-twisted/Config.in                   |  6 ++----
 package/python-txdbus/Config.in                    |  5 ++++-
 package/python-txtorcon/Config.in                  |  7 ++-----
 19 files changed, 52 insertions(+), 90 deletions(-)
package/docker-compose/Config.in +5 -3
index 8e16b60521..761befea4d 100644
@@ -5,7 +5,8 @@ config BR2_PACKAGE_DOCKER_COMPOSE
 	depends on BR2_USE_WCHAR # python
 	depends on BR2_TOOLCHAIN_HAS_THREADS # python
 	depends on !BR2_STATIC_LIBS # python
-	depends on BR2_INSTALL_LIBSTDCPP # python-paramiko -> python-cryptography
+	# python-paramiko -> python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
 	select BR2_PACKAGE_PYTHON3
 	select BR2_PACKAGE_PYTHON_CACHED_PROPERTY # runtime
 	select BR2_PACKAGE_PYTHON_DOCOPT # runtime
@@ -23,7 +24,8 @@ config BR2_PACKAGE_DOCKER_COMPOSE
 
 	  https://www.docker.com/
 
-comment "docker-compose needs a toolchain w/ C++, wchar, threads, dynamic library"
+comment "docker-compose needs a toolchain w/ wchar, threads, dynamic library"
 	depends on BR2_USE_MMU
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
 	depends on !BR2_USE_WCHAR || !BR2_TOOLCHAIN_HAS_THREADS || \
-		BR2_STATIC_LIBS || !BR2_INSTALL_LIBSTDCPP
+		BR2_STATIC_LIBS
package/python-autobahn/Config.in +1 -5
index f719dcc8f1..92ddad0ea4 100644
@@ -1,6 +1,6 @@
 config BR2_PACKAGE_PYTHON_AUTOBAHN
 	bool "python-autobahn"
-	depends on BR2_INSTALL_LIBSTDCPP # python-cryptography -> python-pyasn
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS # python-cryptography
 	depends on BR2_PACKAGE_PYTHON3
 	select BR2_PACKAGE_PYTHON_CFFI # runtime
 	select BR2_PACKAGE_PYTHON_CRYPTOGRAPHY # runtime
@@ -11,7 +11,3 @@ config BR2_PACKAGE_PYTHON_AUTOBAHN
 	  framework.
 
 	  https://pypi.python.org/pypi/autobahn
-
-comment "python-autobahn needs a toolchain w/ C++"
-	depends on !BR2_INSTALL_LIBSTDCPP
-	depends on BR2_PACKAGE_PYTHON3
package/python-channels-redis/Config.in +2 -5
index 8946670180..bab801d2a9 100644
@@ -1,6 +1,7 @@
 config BR2_PACKAGE_PYTHON_CHANNELS_REDIS
 	bool "python-channels-redis"
-	depends on BR2_INSTALL_LIBSTDCPP # python-channels -> python-daphne
+	# python-channels -> python-daphne -> python-autobahn -> python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
 	depends on BR2_PACKAGE_PYTHON3 # python-channels
 	select BR2_PACKAGE_PYTHON_AIOREDIS # runtime
 	select BR2_PACKAGE_PYTHON_ASGIREF # runtime
@@ -12,7 +13,3 @@ config BR2_PACKAGE_PYTHON_CHANNELS_REDIS
 	  sharded configurations, as well as group support.
 
 	  http://github.com/django/channels_redis/
-
-comment "python-channels-redis needs a toolchain w/ C++"
-	depends on !BR2_INSTALL_LIBSTDCPP
-	depends on BR2_PACKAGE_PYTHON3
package/python-channels/Config.in +2 -5
index 5c70f20d1c..6cd6e5f60d 100644
@@ -1,6 +1,7 @@
 config BR2_PACKAGE_PYTHON_CHANNELS
 	bool "python-channels"
-	depends on BR2_INSTALL_LIBSTDCPP # python-daphne -> python-autobahn -> python-cryptography
+	# python-daphne -> python-autobahn -> python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
 	depends on BR2_PACKAGE_PYTHON3 # python-daphne
 	select BR2_PACKAGE_PYTHON_ASGIREF # runtime
 	select BR2_PACKAGE_PYTHON_DAPHNE # runtime
@@ -14,7 +15,3 @@ config BR2_PACKAGE_PYTHON_CHANNELS
 	  own protocols and needs.
 
 	  http://github.com/django/channels
-
-comment "python-channels needs a toolchain w/ C++"
-	depends on !BR2_INSTALL_LIBSTDCPP
-	depends on BR2_PACKAGE_PYTHON3
package/python-crossbar/Config.in +1 -5
index 50b70b8e02..b72d64c19d 100644
@@ -1,6 +1,6 @@
 config BR2_PACKAGE_PYTHON_CROSSBAR
 	bool "python-crossbar"
-	depends on BR2_INSTALL_LIBSTDCPP # python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS # python-cryptography
 	depends on BR2_PACKAGE_PYTHON3
 	# All the following dependencies are runtime dependencies
 	select BR2_PACKAGE_PYTHON_ATTRS
@@ -48,7 +48,3 @@ config BR2_PACKAGE_PYTHON_CROSSBAR
 	  components that can talk in real-time with each other.
 
 	  https://pypi.python.org/pypi/crossbar
-
-comment "python-crossbar needs a toolchain w/ C++"
-	depends on !BR2_INSTALL_LIBSTDCPP
-	depends on BR2_PACKAGE_PYTHON3
package/python-cryptography/Config.in +2 -12
index da4252d6da..07a22bf846 100644
@@ -1,21 +1,11 @@
 config BR2_PACKAGE_PYTHON_CRYPTOGRAPHY
 	bool "python-cryptography"
-	depends on BR2_PACKAGE_PYTHON3 # python-idna
-	depends on BR2_INSTALL_LIBSTDCPP # python-pyasn
+	depends on BR2_PACKAGE_PYTHON3
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
 	select BR2_PACKAGE_OPENSSL
 	select BR2_PACKAGE_PYTHON_CFFI # runtime
-	select BR2_PACKAGE_PYTHON_IDNA # runtime
-	select BR2_PACKAGE_PYTHON_ASN1CRYPTO # runtime
-	select BR2_PACKAGE_PYTHON3_PYEXPAT # runtime
-	select BR2_PACKAGE_PYTHON_SETUPTOOLS # runtime
-	select BR2_PACKAGE_PYTHON_SIX # runtime
-	select BR2_PACKAGE_PYTHON3_SSL # runtime
 	help
 	  cryptography is a package designed to expose cryptographic
 	  primitives and recipes to Python developers.
 
 	  https://cryptography.io
-
-comment "python-cryptography needs a toolchain w/ C++"
-	depends on BR2_PACKAGE_PYTHON3
-	depends on !BR2_INSTALL_LIBSTDCPP
package/python-cryptography/python-cryptography.hash +2 -3
index 7cb38c8d3d..1219fa9b5f 100644
@@ -1,6 +1,5 @@
-# md5, sha256 from https://pypi.org/pypi/cryptography/json
-md5  e2ce2ec8a63965fad351f36ed70fde4b  cryptography-3.3.2.tar.gz
-sha256  5a60d3780149e13b7a6ff7ad6526b38846354d11a15e21068e57073e29e19bed  cryptography-3.3.2.tar.gz
+# Locally calculated after vendoring
+sha256  196bba703cebc052a19f5353614fcfa9d680471990c10305f110adcc05744eeb  cryptography-36.0.1.tar.gz
 # Locally computed sha256 checksums
 sha256  43dad2cc752ab721cd9a9f36ece70fb53ab7713551f2d3d8694d8e8c5a06d6e2  LICENSE
 sha256  aac73b3148f6d1d7111dbca32099f68d26c644c6813ae1e4f05f6579aa2663fe  LICENSE.APACHE
package/python-cryptography/python-cryptography.mk +11 -3
index 9c00efb44f..d302f87405 100644
@@ -4,14 +4,22 @@
 #
 ################################################################################
 
-PYTHON_CRYPTOGRAPHY_VERSION = 3.3.2
+PYTHON_CRYPTOGRAPHY_VERSION = 36.0.1
 PYTHON_CRYPTOGRAPHY_SOURCE = cryptography-$(PYTHON_CRYPTOGRAPHY_VERSION).tar.gz
-PYTHON_CRYPTOGRAPHY_SITE = https://files.pythonhosted.org/packages/d4/85/38715448253404186029c575d559879912eb8a1c5d16ad9f25d35f7c4f4c
+PYTHON_CRYPTOGRAPHY_SITE = https://files.pythonhosted.org/packages/f9/4b/1cf8e281f7ae4046a59e5e39dd7471d46db9f61bb564fddbff9084c4334f
 PYTHON_CRYPTOGRAPHY_SETUP_TYPE = setuptools
 PYTHON_CRYPTOGRAPHY_LICENSE = Apache-2.0 or BSD-3-Clause
 PYTHON_CRYPTOGRAPHY_LICENSE_FILES = LICENSE LICENSE.APACHE LICENSE.BSD
 PYTHON_CRYPTOGRAPHY_CPE_ID_VENDOR = cryptography_project
 PYTHON_CRYPTOGRAPHY_CPE_ID_PRODUCT = cryptography
-PYTHON_CRYPTOGRAPHY_DEPENDENCIES = host-python-cffi openssl
+PYTHON_CRYPTOGRAPHY_DEPENDENCIES = host-python-setuptools-rust host-python-cffi host-rustc
+PYTHON_CRYPTOGRAPHY_ENV = CARGO_BUILD_TARGET="$(RUSTC_TARGET_NAME)" \
+	PYO3_CROSS_LIB_DIR="$(STAGING_DIR)/usr/lib/python$(PYTHON3_VERSION_MAJOR)" \
+	CARGO_HOME="$(HOST_DIR)/share/cargo"
+# We need to vendor the Cargo crates at download time
+PYTHON_CRYPTOGRAPHY_DOWNLOAD_POST_PROCESS = cargo
+PYTHON_CRYPTOGRAPHY_DOWNLOAD_DEPENDENCIES = host-rustc
+PYTHON_CRYPTOGRAPHY_DL_ENV = \
+	BR_CARGO_MANIFEST_PATH=src/rust/Cargo.toml
 
 $(eval $(python-package))
package/python-daphne/Config.in +2 -5
index 601b2e1db8..c4479417e9 100644
@@ -1,6 +1,7 @@
 config BR2_PACKAGE_PYTHON_DAPHNE
 	bool "python-daphne"
-	depends on BR2_INSTALL_LIBSTDCPP # python-autobahn -> python-cryptography
+	# python-autobahn -> python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
 	depends on BR2_PACKAGE_PYTHON3
 	select BR2_PACKAGE_PYTHON_ASGIREF # runtime
 	select BR2_PACKAGE_PYTHON_AUTOBAHN # runtime
@@ -14,7 +15,3 @@ config BR2_PACKAGE_PYTHON_DAPHNE
 	  versus HTTP endpoints.
 
 	  https://github.com/django/daphne
-
-comment "python-daphne needs a toolchain w/ C++"
-	depends on !BR2_INSTALL_LIBSTDCPP
-	depends on BR2_PACKAGE_PYTHON3
package/python-keyring/Config.in +2 -5
index f7bb1ae820..61cb98952f 100644
@@ -1,7 +1,8 @@
 config BR2_PACKAGE_PYTHON_KEYRING
 	bool "python-keyring"
+	# python-secretstorage -> python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
 	depends on BR2_PACKAGE_PYTHON3
-	depends on BR2_INSTALL_LIBSTDCPP # python-secretstorage
 	select BR2_PACKAGE_PYTHON_ENTRYPOINTS # runtime
 	select BR2_PACKAGE_PYTHON_SECRETSTORAGE # runtime
 	help
@@ -9,7 +10,3 @@ config BR2_PACKAGE_PYTHON_KEYRING
 	  system keyring service from Python.
 
 	  https://pypi.python.org/pypi/keyring
-
-comment "python-keyring needs a toolchain w/ C++"
-	depends on BR2_PACKAGE_PYTHON3
-	depends on !BR2_INSTALL_LIBSTDCPP
package/python-opcua-asyncio/Config.in +1 -0
index 3700e00e1d..95715fdd08 100644
@@ -1,5 +1,6 @@
 config BR2_PACKAGE_PYTHON_OPCUA_ASYNCIO
 	bool "python-opcua-asyncio"
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS # python-cryptography
 	depends on BR2_PACKAGE_PYTHON3
 	select BR2_PACKAGE_PYTHON_AIOFILES # runtime
 	select BR2_PACKAGE_PYTHON_CRYPTOGRAPHY # runtime
package/python-paramiko/Config.in +2 -6
index 4b8495d76d..e44237b0c2 100644
@@ -1,7 +1,7 @@
 config BR2_PACKAGE_PYTHON_PARAMIKO
 	bool "python-paramiko"
-	depends on BR2_PACKAGE_PYTHON3 # python-bcrypt
-	depends on BR2_INSTALL_LIBSTDCPP # python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS # python-cryptography
+	depends on BR2_PACKAGE_PYTHON3 # python-bcrypt, python-cryptography
 	select BR2_PACKAGE_PYTHON_BCRYPT # runtime
 	select BR2_PACKAGE_PYTHON_CRYPTOGRAPHY # runtime
 	select BR2_PACKAGE_PYTHON_PYNACL # runtime
@@ -10,7 +10,3 @@ config BR2_PACKAGE_PYTHON_PARAMIKO
 	  SSH2 protocol library.
 
 	  https://github.com/paramiko/paramiko/
-
-comment "python-paramiko needs a toolchain w/ C++"
-	depends on BR2_PACKAGE_PYTHON3
-	depends on !BR2_INSTALL_LIBSTDCPP
package/python-pyopenssl/Config.in +2 -6
index 99ea4e9408..65d516813f 100644
@@ -1,14 +1,10 @@
 config BR2_PACKAGE_PYTHON_PYOPENSSL
 	bool "python-pyopenssl"
-	depends on BR2_PACKAGE_PYTHON3 # python-cryptography -> python-idna
-	depends on BR2_INSTALL_LIBSTDCPP # python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS # python-cryptography
+	depends on BR2_PACKAGE_PYTHON3 # python-cryptography
 	select BR2_PACKAGE_PYTHON_CRYPTOGRAPHY # runtime
 	select BR2_PACKAGE_PYTHON_SIX # runtime
 	help
 	  Python wrapper module around the OpenSSL library.
 
 	  https://github.com/pyca/pyopenssl
-
-comment "python-pyopenssl needs a toolchain w/ C++"
-	depends on BR2_PACKAGE_PYTHON3
-	depends on !BR2_INSTALL_LIBSTDCPP
package/python-secretstorage/Config.in +1 -5
index 8ae8c2d598..059ea097bb 100644
@@ -1,13 +1,9 @@
 config BR2_PACKAGE_PYTHON_SECRETSTORAGE
 	bool "python-secretstorage"
 	depends on BR2_PACKAGE_PYTHON3
-	depends on BR2_INSTALL_LIBSTDCPP # python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS # python-cryptography
 	select BR2_PACKAGE_PYTHON_CRYPTOGRAPHY # runtime
 	help
 	  Python bindings to FreeDesktop.org Secret Service API.
 
 	  https://github.com/mitya57/secretstorage
-
-comment "python-secretstorage needs a toolchain w/ C++"
-	depends on BR2_PACKAGE_PYTHON3
-	depends on !BR2_INSTALL_LIBSTDCPP
package/python-service-identity/Config.in +4 -6
index ab012347dc..c571065e92 100644
@@ -1,7 +1,9 @@
 config BR2_PACKAGE_PYTHON_SERVICE_IDENTITY
 	bool "python-service-identity"
-	depends on BR2_PACKAGE_PYTHON3 # python-pyopenssl -> python-cryptography -> python-idna
-	depends on BR2_INSTALL_LIBSTDCPP # python-pyopenssl
+	# python-pyopenssl -> python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
+	# python-pyopenssl -> python-cryptography
+	depends on BR2_PACKAGE_PYTHON3
 	select BR2_PACKAGE_PYTHON_ATTRS # runtime
 	select BR2_PACKAGE_PYTHON_CHARACTERISTIC # runtime
 	select BR2_PACKAGE_PYTHON_PYASN1 # runtime
@@ -11,7 +13,3 @@ config BR2_PACKAGE_PYTHON_SERVICE_IDENTITY
 	  Service identity verification for pyOpenSSL.
 
 	  https://pypi.python.org/pypi/service_identity
-
-comment "python-service-identify needs a toolchain w/ C++"
-	depends on BR2_PACKAGE_PYTHON3
-	depends on !BR2_INSTALL_LIBSTDCPP
package/python-treq/Config.in +4 -6
index 02c26a53f7..6e54c19e16 100644
@@ -1,7 +1,9 @@
 config BR2_PACKAGE_PYTHON_TREQ
 	bool "python-treq"
-	depends on BR2_PACKAGE_PYTHON3 # python-idna
-	depends on BR2_INSTALL_LIBSTDCPP # python-pyopenssl
+	# python-pyopenssl -> python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
+	# python-idna, python-pyopenssl -> python-cryptography
+	depends on BR2_PACKAGE_PYTHON3
 	select BR2_PACKAGE_PYTHON_ATTRS # runtime
 	select BR2_PACKAGE_PYTHON_IDNA # runtime
 	select BR2_PACKAGE_PYTHON_INCREMENTAL # runtime
@@ -16,7 +18,3 @@ config BR2_PACKAGE_PYTHON_TREQ
 	  API for making HTTP requests when using Twisted.
 
 	  https://github.com/twisted/treq
-
-comment "python-treq needs a toolchain w/ C++"
-	depends on BR2_PACKAGE_PYTHON3
-	depends on !BR2_INSTALL_LIBSTDCPP
package/python-twisted/Config.in +2 -4
index 29536ecf9d..89298b2e0c 100644
@@ -23,12 +23,10 @@ config BR2_PACKAGE_PYTHON_TWISTED_HTTP2
 
 config BR2_PACKAGE_PYTHON_TWISTED_TLS
 	bool "TLS support"
-	depends on BR2_INSTALL_LIBSTDCPP # python-{pyopenssl,service-identity}
+	# python-{pyopenssl,service-identity}
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
 	select BR2_PACKAGE_PYTHON_IDNA # runtime
 	select BR2_PACKAGE_PYTHON_PYOPENSSL # runtime
 	select BR2_PACKAGE_PYTHON_SERVICE_IDENTITY # runtime
 
-comment "TLS support needs a toolchain w/ C++"
-	depends on !BR2_INSTALL_LIBSTDCPP
-
 endif
package/python-txdbus/Config.in +4 -1
index c7de322f91..ffd00f153e 100644
@@ -1,6 +1,9 @@
 config BR2_PACKAGE_PYTHON_TXDBUS
 	bool "python-txdbus"
-	depends on BR2_PACKAGE_PYTHON3 # python-twisted -> python-cryptography -> python-idna
+	# python-twisted -> python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
+	# python-twisted -> python-cryptography
+	depends on BR2_PACKAGE_PYTHON3
 	select BR2_PACKAGE_PYTHON_TWISTED # runtime
 	select BR2_PACKAGE_PYTHON_SIX # runtime
 	help
package/python-txtorcon/Config.in +2 -5
index e8a1a233db..0606e5f442 100644
@@ -1,7 +1,8 @@
 config BR2_PACKAGE_PYTHON_TXTORCON
 	bool "python-txtorcon"
+	# python-pyopenssl -> python-cryptography
+	depends on BR2_PACKAGE_HOST_RUSTC_TARGET_ARCH_SUPPORTS
 	depends on BR2_PACKAGE_PYTHON3 # python-idna
-	depends on BR2_INSTALL_LIBSTDCPP # python-pyopenssl, python-service-identity
 	select BR2_PACKAGE_PYTHON_AUTOMAT # runtime
 	select BR2_PACKAGE_PYTHON_IDNA # runtime
 	select BR2_PACKAGE_PYTHON_INCREMENTAL # runtime
@@ -14,7 +15,3 @@ config BR2_PACKAGE_PYTHON_TXTORCON
 	  configuration abstractions.
 
 	  https://github.com/meejah/txtorcon
-
-comment "python-txtorcon needs a toolchain w/ C++"
-	depends on BR2_PACKAGE_PYTHON3
-	depends on !BR2_INSTALL_LIBSTDCPP
Signature · 6 lines
2.33.1

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot
#10 lore raw
In reply to Thomas Petazzoni
Quoted 40 lines · Thomas Petazzoni

Hello,

This series bumps the python-cryptography package to a new version. It is based on earlier work from James Hilliard at:

   https://patchwork.ozlabs.org/project/buildroot/patch/[email protected]/

But it did require quite some more work, which is presented in this patch series.

The unique challenge caused by this version bump is that python-cryptography now has some Rust code integrated, so suddenly this package, and all its reverse dependencies need to depend on Rust being available.

There is also a host variant of python-cryptography, which is used solely by ca-certificates, which has a significant number of reverse dependencies. To avoid adding Rust dependencies to these, we broke the dependency of ca-certificates on host-python-cryptography using a simple patch, and dropped host-python-cryptography altogether.

This patch series depends on the "Go/Cargo vendoring" patch series posted recently. Indeed, the Rust code in python-cryptography depends on other Cargo crates, which we don't want to download during the build, but really during the download step.

There is however still one problem with the last patch of the series, see below.

This series is available at:

https://github.com/tpetazzoni/buildroot/tree/python-cryptography

The patch series goes like this:

- PATCH 1 modifies ca-certificates to no longer need host-python-cryptography

- PATCH 2 drops host-python-cryptography itselef

Quoted 8 lines

- PATCH 3 and 4 fix some issues in package/rustc/ in the description of the ARMv5 and RISC-V 64-bit Rust support. We noticed this because Rust binaries were not working on ARMv5 ("Illegal instruction"), this was due to the fact that we were using the pre-compiled Rust standard library for ARMv6, due to a bogus architecture name. This is fixed by PATCH 3. And while reviewing this, we spotted a similar issue for RISC-V 64-bit, which is fixed in PATCH 4.

First four patches applied to master, with the fixup that you mentioned on
patch 3.

Regards,
Arnout

Quoted 136 lines

- PATCH 5 slightly improves the Cargo download post-process script to make it possible to pass a custom path to the Cargo.toml file, instead of assuming it's located at the root of the package source tree.

- PATCH 6 modifies the runtime test cases that involve python-cryptography directly or indirectly to use a glibc toolchain instead of the default uclibc toolchain. Indeed, Rust is for now not available on uClibc platforms. This could have been done in the patch doing the version bump, but the version bump patch is already quite large, and the update of the runtime tests could be done as a separate preparation step.

- PATCH 7 finally bumps python-cryptography
itself. python-cryptography itself requires a bit of trickery to leverage the Cargo download post-process logic, to vendor dependencies during the download step. Then the patch also touches the entire reverse dependency tree (sometimes up to 5 levels deep!). This patch should NOT be applied for now, due to issues (see below).

This series was tested by building the following configuration:

BR2_arm=y
BR2_cortex_a9=y
BR2_ARM_ENABLE_VFP=y
BR2_TOOLCHAIN_EXTERNAL=y
BR2_TOOLCHAIN_EXTERNAL_BOOTLIN=y
BR2_INIT_NONE=y
BR2_SYSTEM_BIN_SH_NONE=y
# BR2_PACKAGE_BUSYBOX is not set
BR2_PACKAGE_PYTHON_CHANNELS_REDIS=y
BR2_PACKAGE_PYTHON_CROSSBAR=y
BR2_PACKAGE_PYTHON_KEYRING=y
BR2_PACKAGE_PYTHON_OPCUA_ASYNCIO=y
BR2_PACKAGE_PYTHON_TXDBUS=y
BR2_PACKAGE_DOCKER_COMPOSE=y
# BR2_TARGET_ROOTFS_TAR is not set

that builds all affected packages.

It was also tested by running all Python runtime test cases:

    https://gitlab.com/tpetazzoni/buildroot/-/pipelines/443445436

The one thing that fails is when building python-cryptography on a target arch that is the same as the host. This can be reproduced by running the tests.package.test_docker_compose.TestDockerCompose test case.

We get this:

      Running `/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build`

error: failed to run custom build command for proc-macro-hack v0.5.19

Caused by:
process didn't exit successfully: /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build (exi
t status: 1)
--- stderr /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: versio
n `GLIBC_2.33' not found (required by /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build)
/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: versio
n `GLIBC_2.32' not found (required by /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build)
/home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: versio
n `GLIBC_2.34' not found (required by /home/thomas/buildroot/buildroot/output/build/python-cryptography-36.0.1/src/rust/target/release/build/proc-macro-hack-f35f1313688107b0/build-script-build)
warning: build failed, waiting for other jobs to finish...
error: build failed

Which looks terribly similar to the issue on ripgrep which was fixed by commit 4b2be770b8a853a7dd97b5788d837f0d84923fa1. So I tried doing the same, i.e passing:

	__CARGO_TEST_CHANNEL_OVERRIDE_DO_NOT_USE_THIS="nightly" \
	CARGO_TARGET_APPLIES_TO_HOST="false"

in the environment, and -Z target-applies-to-host to cargo (by passing it to the args argument of the RustExtension class in python-cryptography's setup.py). But cargo doesn't like that, the build log goes like this:

Quoted 1 line
Quoted 1 line
Quoted 1 line

python-cryptography 36.0.1 BuildingESC[27m

... calling setup.py ...
cargo rustc --lib --manifest-path src/rust/Cargo.toml --features pyo3/abi3-py36 pyo3/extension-module --target x86_64-unknown-linux-gnu -Z target-applies-to-host --release --verbose -- --crate-type cdylib
...
error: unknown -Z flag specified: target-applies-to-host

The difference that I see with the cargo infrastructure is that this is not using "cargo build" but "cargo rustc". I would really appreciate some help on solving this (last?) issue.

Best regards,

Thomas Petazzoni

Thomas Petazzoni (7):
package/ca-certificates: remove dependency on host-python-cryptography package/python-cryptography: drop host variant package/rustc: fix BR2_PACKAGE_HOST_RUSTC_ARCH for ARMv5 package/rustc: fix the riscv64gc architecture handling support/download/cargo-post-process: make manifest path configurable support/testing/tests: switch python-cryptography related tests to
glibc
DO-NOT-APPLY package/python-cryptography: bump version to 36.0.1

  ...2pem.py-make-cryptography-module-opt.patch | 59 +++++++++++++++++++
  package/ca-certificates/ca-certificates.mk    |  2 +-
  package/docker-compose/Config.in              |  8 ++-
  package/python-autobahn/Config.in             |  6 +-
  package/python-channels-redis/Config.in       |  7 +--
  package/python-channels/Config.in             |  7 +--
  package/python-crossbar/Config.in             |  6 +-
  package/python-cryptography/Config.in         | 14 +----
  .../python-cryptography.hash                  |  5 +-
  .../python-cryptography.mk                    | 20 ++++---
  package/python-daphne/Config.in               |  7 +--
  package/python-keyring/Config.in              |  7 +--
  package/python-opcua-asyncio/Config.in        |  1 +
  package/python-paramiko/Config.in             |  8 +--
  package/python-pyopenssl/Config.in            |  8 +--
  package/python-secretstorage/Config.in        |  6 +-
  package/python-service-identity/Config.in     | 10 ++--
  package/python-treq/Config.in                 | 10 ++--
  package/python-twisted/Config.in              |  6 +-
  package/python-txdbus/Config.in               |  5 +-
  package/python-txtorcon/Config.in             |  7 +--
  package/rustc/Config.in.host                  |  4 +-
  support/download/cargo-post-process           |  2 +-
  .../tests/package/test_python_autobahn.py     | 10 +++-
  .../tests/package/test_python_crossbar.py     | 10 +++-
  .../tests/package/test_python_cryptography.py | 10 +++-
  .../package/test_python_service_identity.py   | 10 +++-
  .../testing/tests/package/test_python_treq.py | 10 +++-
  .../tests/package/test_python_txtorcon.py     | 10 +++-
  29 files changed, 170 insertions(+), 105 deletions(-)
Quoted 2 lines

create mode 100644 package/ca-certificates/0001-mozilla-certdata2pem.py-make-cryptography-module-opt.patch

_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot

#11 lore raw
In reply to Thomas Petazzoni

Hello,

Quoted 9 lines · On Fri, 7 Jan 2022 18:13:10 +0100 Thomas Petazzoni…

Thomas Petazzoni (7):
package/ca-certificates: remove dependency on host-python-cryptography package/python-cryptography: drop host variant package/rustc: fix BR2_PACKAGE_HOST_RUSTC_ARCH for ARMv5 package/rustc: fix the riscv64gc architecture handling support/download/cargo-post-process: make manifest path configurable support/testing/tests: switch python-cryptography related tests to
glibc
DO-NOT-APPLY package/python-cryptography: bump version to 36.0.1

Thanks to the help of James, we were able to figure out the last issues, so I pushed those last 3 patches, but with a number of additional preparation patches:

01764876e3d4e7ab95c8365f1c83e6c12d49757c package/python-cryptography: bump version to 36.0.1 b6378631c2609742382984f6f7b93c1d9d2cdb78 package/pkg-cargo.mk: use CARGO_TARGET_<target>_LINKER instead of cargo config file 91a5a2d317a02d7fb41e706a490fe5063039d55c package/pkg-cargo.mk: use CARGO_BUILD_TARGET instead of --target 98c7ea3ddf11d08acd11e30be36b4b56c682d239 package/pkg-cargo.mk: enable -Z target-applies-to-host via env 0894937f7fc4ac9d04d65bc6cdd5fc1927483976 package/pkg-cargo.mk: split PKG_CARGO_ENV into target/host variables e5ead094e18cd4a5a9a22b69a42e687f43ee8a8b support/testing/tests: switch python-cryptography related tests to glibc 36e8743da181420775b79d5be9bbe872b00f2c01 support/download/cargo-post-process: make manifest path configurable

Best regards,

Thomas

Signature · 7 lines
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering and training
https://bootlin.com
_______________________________________________
buildroot mailing list
[email protected]
https://lists.buildroot.org/mailman/listinfo/buildroot