Rust Oe-Selftest implementation
PATCH v11 1/1 Rust Oe-Selftest implementation
From: pgowda <[email protected]>
The patch implements Rust testing framework similar to other selftest, specifically the gcc selftest in OE. It uses the client and server based method to test the binaries for cross-target on the image. The test framework is a wrapper around the Rust build system as ./x.py test. It tests many functionalities of Rust distribution like tools, documentation, libraries, packages, tools, Cargo, Crater etc. Please refer the following link for detailed description of Rust testing:-
https://rustc-dev-guide.rust-lang.org/tests/intro.html#tool-tests
To support the rust tests in oe-core, the following functions were added:-setup_cargo_environment(): Build bootstrap and some early stage tools. do_rust_setup_snapshot(): Install the snapshot version of rust binaries. do_configure(): To generate config.tomldo_compile(): To build "remote-test-server" for qemu target image.
Approximate Number of Tests Run in the Rust Testsuite :- 18000 Approximate Number of Tests that FAIL in bitbake environment :- 100-150 Normally majority of the testcases are present in major folder "test/" It contributes to more than 80% of the testcases present in Rust test framework. These tests pass as expected on any Rust versions without much fuss. The tests that fail are of less important and contribute to less than 2% of the total testcases. These minor tests are observed to work on some versions and fail on others. They have to be added, ignored or excluded for different versions as per the behavior. These tests have been ignored or excluded in the Rust selftest environment to generate success of completing the testsuite.
These tests work in parallel mode even in the skipped test mode as expected. Although the patch to disable tests is large, it is very simple in that it only disables tests. When updating to a newer version of Rust, the patch can usually be ported in a day.
--- meta/lib/oeqa/selftest/cases/rust.py | 54 ++ meta/recipes-devtools/rust/rust-source.inc | 3 +- meta/recipes-devtools/rust/rust.inc | 1 + .../rust/rust/rust-oe-selftest.patch | 508 ++++++++++++++++++ meta/recipes-devtools/rust/rust_1.63.0.bb | 5 + 5 files changed, 570 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/rust.py create mode 100644 meta/recipes-devtools/rust/rust/rust-oe-selftest.patch
meta/lib/oeqa/selftest/cases/rust.py new file +54 -0
@@ -0,0 +1,54 @@ +# SPDX-License-Identifier: MIT +import os +import subprocess +from oeqa.core.decorator import OETestTag +from oeqa.core.case import OEPTestResultTestCase +from oeqa.selftest.case import OESelftestTestCase +from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu, Command +from oeqa.utils.sshcontrol import SSHControl + +# Total time taken for testing is of about 2hr 20min, with PARALLEL_MAKE set to 40 number of jobs. +class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase): + def test_rust(self, *args, **kwargs): + # build remote-test-server before image build + recipe = "rust" + bitbake("{} -c test_compile".format(recipe)) + builddir = get_bb_var("RUSTSRC", "rust") + # build core-image-minimal with required packages + default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp"] + features = [] + features.append('IMAGE_FEATURES += "ssh-server-openssh"') + features.append('CORE_IMAGE_EXTRA_INSTALL += "{0}"'.format(" ".join(default_installed_packages))) + self.write_config("\n".join(features)) + bitbake("core-image-minimal") + # wrap the execution with a qemu instance. + # Tests are run with 512 tasks in parallel to execute all tests very quickly + with runqemu("core-image-minimal", runqemuparams = "nographic", qemuparams = "-m 512") as qemu: + # Copy remote-test-server to image through scp + ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root") + ssh.copy_to(builddir + "/" + "build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-server","~/") + # Execute remote-test-server on image through background ssh + command = '~/remote-test-server -v remote' + sshrun=subprocess.Popen(("ssh", '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-f', "root@%s" % qemu.ip, command), shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE) + # Get the values of variables. + tcpath = get_bb_var("TARGET_SYS", "rust") + targetsys = get_bb_var("RUST_TARGET_SYS", "rust") + rustlibpath = get_bb_var("WORKDIR", "rust") + tmpdir = get_bb_var("TMPDIR", "rust") + + # Exclude the test folders that error out while building + # TODO: Fix the errors and include them for testing + # no-fail-fast: Run all tests regardless of failure. + # bless: First runs rustfmt to format the codebase, + # then runs tidy checks. + testargs = "--exclude src/test/rustdoc --exclude src/test/rustdoc-json --exclude src/test/run-make-fulldeps --exclude src/tools/tidy --exclude src/tools/rustdoc-themes --exclude src/rustdoc-json-types --exclude src/librustdoc --exclude src/doc/unstable-book --exclude src/doc/rustdoc --exclude src/doc/rustc --exclude compiler/rustc --exclude library/panic_abort --exclude library/panic_unwind --exclude src/tools/lint-docs --exclude src/test/rustdoc-js-std --no-doc --no-fail-fast --bless" + + # Set path for target-poky-linux-gcc, RUST_TARGET_PATH and hosttools. + cmd = " export PATH=%s/recipe-sysroot-native/usr/bin:$PATH;" % rustlibpath + cmd = cmd + " export TARGET_VENDOR=\"-poky\";" + cmd = cmd + " export PATH=%s/recipe-sysroot-native/usr/bin/%s:%s/hosttools:$PATH;" % (rustlibpath, tcpath, tmpdir) + cmd = cmd + " export RUST_TARGET_PATH=%s/rust-targets;" % rustlibpath + # Trigger testing. + cmd = cmd + " export TEST_DEVICE_ADDR=\"%s:12345\";" % qemu.ip + cmd = cmd + " cd %s; python3 src/bootstrap/bootstrap.py test %s --target %s ;" % (builddir, testargs, targetsys) + result = runCmd(cmd)
meta/recipes-devtools/rust/rust-source.inc +2 -1
@@ -4,7 +4,8 @@ SRC_URI[rust.sha256sum] = "8f44af6dc44cc4146634a4dd5e4cc5470b3052a2337019b870c0e SRC_URI:append:class-target:pn-rust = " \ file://hardcodepaths.patch \ file://crossbeam_atomic.patch \ - file://0001-Add-ENOTSUP-constant-for-riscv32-musl.patch" + file://0001-Add-ENOTSUP-constant-for-riscv32-musl.patch \ + file://rust-oe-selftest.patch;striplevel=1" SRC_URI:append:class-nativesdk:pn-nativesdk-rust = " file://hardcodepaths.patch" RUSTSRC = "${WORKDIR}/rustc-${PV}-src"
meta/recipes-devtools/rust/rust.inc +1 -0
@@ -53,6 +53,7 @@ do_rust_setup_snapshot () { fi } addtask rust_setup_snapshot after do_unpack before do_configure +addtask do_test_compile after do_configure do_rust_gen_targets do_rust_setup_snapshot[dirs] += "${WORKDIR}/rust-snapshot" do_rust_setup_snapshot[vardepsexclude] += "UNINATIVE_LOADER"
meta/recipes-devtools/rust/rust/rust-oe-selftest.patch new file +508 -0
508 lines not on this page · raw message
meta/recipes-devtools/rust/rust_1.63.0.bb +5 -0
@@ -14,6 +14,11 @@ do_compile () { rust_runx build --stage 2 } +do_test_compile[dirs] = "${B}" +do_test_compile () { + rust_runx build src/tools/remote-test-server --target "${RUST_TARGET_SYS}" +} + do_compile:append:class-target () { rust_runx build --stage 2 src/tools/clippy rust_runx build --stage 2 src/tools/rustfmt
Signature · 1 line
2.25.1
Quoted 38 lines · On Wed, 2022-09-07 at 20:27 +0530, [email protected] wrote:
From: pgowda <[email protected]>
The patch implements Rust testing framework similar to other selftest, specifically the gcc selftest in OE. It uses the client and server based method to test the binaries for cross-target on the image. The test framework is a wrapper around the Rust build system as ./x.py test. It tests many functionalities of Rust distribution like tools, documentation, libraries, packages, tools, Cargo, Crater etc. Please refer the following link for detailed description of Rust testing:-
https://rustc-dev-guide.rust-lang.org/tests/intro.html#tool-testsTo support the rust tests in oe-core, the following functions were added:-
setup_cargo_environment(): Build bootstrap and some early stage tools.do_rust_setup_snapshot(): Install the snapshot version of rust binaries.do_configure(): To generate config.tomldo_compile(): To build "remote-test-server" for qemu target image.Approximate Number of Tests Run in the Rust Testsuite :- 18000 Approximate Number of Tests that FAIL in bitbake environment :- 100-150 Normally majority of the testcases are present in major folder "test/" It contributes to more than 80% of the testcases present in Rust test framework. These tests pass as expected on any Rust versions without much fuss. The tests that fail are of less important and contribute to less than 2% of the total testcases. These minor tests are observed to work on some versions and fail on others. They have to be added, ignored or excluded for different versions as per the behavior. These tests have been ignored or excluded in the Rust selftest environment to generate success of completing the testsuite.
These tests work in parallel mode even in the skipped test mode as expected. Although the patch to disable tests is large, it is very simple in that it only disables tests. When updating to a newer version of Rust, the patch can usually be ported in a day.
Signed-off-by pgowda <[email protected]>Signed-off-by Vinay Kumar <[email protected]>
Quoted 9 lines
--- meta/lib/oeqa/selftest/cases/rust.py | 54 ++ meta/recipes-devtools/rust/rust-source.inc | 3 +- meta/recipes-devtools/rust/rust.inc | 1 + .../rust/rust/rust-oe-selftest.patch | 508 ++++++++++++++++++ meta/recipes-devtools/rust/rust_1.63.0.bb | 5 + 5 files changed, 570 insertions(+), 1 deletion(-) create mode 100644 meta/lib/oeqa/selftest/cases/rust.py create mode 100644 meta/recipes-devtools/rust/rust/rust-oe-selftest.patch
I ran this through some testing on the autobuilder. The first time was ok, it only ran on qemux86 and qemux86-64 as it was only enabled for toolchain-system but ran successfully.
On the second run I added @OETestTag("toolchain-user") as well which meant it ran for arm/mips/powerpc. It looks like there was one worrying intermittent failure:
qemux86-64 intermittent issue:
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/5788/steps/19/logs/stdio
and then some arch specific failures:
qemuppc failed in the test:
https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/5790/steps/19/logs/stdio
qemumips64 failed in the test:
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/5786/steps/20/logs/stdio
qemumips failed in the test:
https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/5804/steps/20/logs/stdio
qemuarm64, qemuarm and qemux86 passed on this run.
We need to understand and fix that qemux86-64 intermittent issue before this can merge. I wasn't sure what we want to do with mips and ppc.
Cheers,
Richard
Quoted 48 lines · Richard Purdie
Quoted 45 lines · On Wed, 2022-09-07 at 20:27 +0530, [email protected] wrote:
From: pgowda <[email protected]>
The patch implements Rust testing framework similar to other selftest, specifically the gcc selftest in OE. It uses the client and server based method to test the binaries for cross-target on the image. The test framework is a wrapper around the Rust build system as ./x.py test. It tests many functionalities of Rust distribution like tools, documentation, libraries, packages, tools, Cargo, Crater etc. Please refer the following link for detailed description of Rust testing:-
https://rustc-dev-guide.rust-lang.org/tests/intro.html#tool-testsTo support the rust tests in oe-core, the following functions were added:-
setup_cargo_environment(): Build bootstrap and some early stage tools.do_rust_setup_snapshot(): Install the snapshot version of rust binaries.do_configure(): To generate config.tomldo_compile(): To build "remote-test-server" for qemu target image.Approximate Number of Tests Run in the Rust Testsuite :- 18000 Approximate Number of Tests that FAIL in bitbake environment :- 100-150 Normally majority of the testcases are present in major folder "test/" It contributes to more than 80% of the testcases present in Rust test framework. These tests pass as expected on any Rust versions without much fuss. The tests that fail are of less important and contribute to less than 2% of the total testcases. These minor tests are observed to work on some versions and fail on others. They have to be added, ignored or excluded for different versions as per the behavior. These tests have been ignored or excluded in the Rust selftest environment to generate success of completing the testsuite.
These tests work in parallel mode even in the skipped test mode as expected. Although the patch to disable tests is large, it is very simple in that it only disables tests. When updating to a newer version of Rust, the patch can usually be ported in a day.
Signed-off-by pgowda <[email protected]>Signed-off-by Vinay Kumar <[email protected]>--- meta/lib/oeqa/selftest/cases/rust.py | 54 ++ meta/recipes-devtools/rust/rust-source.inc | 3 +- meta/recipes-devtools/rust/rust.inc | 1 + .../rust/rust/rust-oe-selftest.patch | 508 ++++++++++++++++++ meta/recipes-devtools/rust/rust_1.63.0.bb | 5 + 5 files changed, 570 insertions(+), 1 deletion(-)Quoted 2 lines
create mode 100644
meta/lib/oeqa/selftest/cases/rust.pycreate mode 100644meta/recipes-devtools/rust/rust/rust-oe-selftest.patch
Quoted 3 lines
I ran this through some testing on the autobuilder. The first time was ok, it only ran on qemux86 and qemux86-64 as it was only enabled for toolchain-system but ran successfully.
Yay, progress!
Quoted 7 lines
On the second run I added @OETestTag("toolchain-user") as well which meant it ran for arm/mips/powerpc. It looks like there was one worrying intermittent failure:
qemux86-64 intermittent issue:
https://autobuilder.yoctoproject.org/typhoon/#/builders/73/builds/5788/steps/19/logs/stdio
A copy failed so Naveen should figure out why and fix the root cause.
This is the top priority to deal with and if you fix that, you should re-submit before spending significant time on the non-x86 tests.
Quoted 6 lines
and then some arch specific failures:
qemuppc failed in the test:
https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/5790/steps/19/logs/stdio
Quoted 7 lines
qemumips64 failed in the test:
https://autobuilder.yoctoproject.org/typhoon/#/builders/74/builds/5786/steps/20/logs/stdioqemumips failed in the test:
https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/5804/steps/20/logs/stdioqemuarm64, qemuarm and qemux86 passed on this run.
Naveen,
Does qemuriscv64 pass all tests? If not summarize as described below.
Quoted 3 lines
We need to understand and fix that qemux86-64 intermittent issue before this can merge. I wasn't sure what we want to do with mips and ppc.
It's tempting to just skip ppc, mips* since they are 2rd tier platforms for
Rust which means that:
"Rust's continuous integration checks that tier 2 targets will always build, but they may or may not pass tests."
- https://doc.rust-lang.org/nightly/rustc/target-tier-policy.html
- https://doc.rust-lang.org/nightly/rustc/platform-support.html
Also, in practice, they are not top tier targets for Yocto.
Regardless , I'd like Naveen to figure out what the problems are and summarize them here. Naveen you don't need to look at more tha say the first 5 or 10 test failures since we're just trying to get a sense of whether these are problems that are unique to Yocto or known upstream.
If any of the issues are known issues upstream, please provide a link that we could track.
../Randy
Quoted 5 lines
Cheers,
Richard
Signature · 2 lines
# Randy MacLeod # Wind River Linux