Rust Oe-Selftest implementation
PATCH v3 1/1 Rust Oe-Selftest implementation
The patch implements Oe-selftest framework for Rust test. Some of the functions are as follows:- setup_cargo_environment(): Build bootstrap and some early stage tools. do_rust_setup_snapshot(): Install the snapshot version of rust binaries. do_configure(): To generate config.tomldo_compile(): To build "remote-test-server" for qemutarget image.
The python file builds remote-test-server and executes rust testing remotely using background ssh. It adds the necessary test environment and variables to run the rust oe-selftest.
Print the results in case of failure of runCmd().
The patch has been run and tested for X86, X86_64, Arm & Arm64 targets on Ubuntu-18 successfully.
Most of the tests excluded in rust.py are due to the following error:- These tests will be included in Oe-selftest after fixing the issue.
===============================================================
"build-st/tmp/work/core2-64-poky-linux/rust-testsuite/1.58.1-r0/ rustc-1.58.1-src/build/bootstrap/debug/rustc -vV` (exit status: 101) --- stderr
thread 'main' panicked at '
kind: NotFound, message: "No such file or directory" }', src/bootstrap/bin/rustc.rs:175:37
===============================================================
--- meta/lib/oeqa/selftest/cases/rust.py | 57 + meta/recipes-devtools/rust/rust-testsuite.inc | 163 +++ .../rust-testsuite/rust-oe-selftest.patch | 995 ++++++++++++++++++ .../rust/rust-testsuite_1.58.1.bb | 3 + 4 files changed, 1218 insertions(+) create mode 100644 meta/lib/oeqa/selftest/cases/rust.py create mode 100644 meta/recipes-devtools/rust/rust-testsuite.inc create mode 100644 meta/recipes-devtools/rust/rust-testsuite/rust-oe-selftest.patch create mode 100644 meta/recipes-devtools/rust/rust-testsuite_1.58.1.bb
meta/lib/oeqa/selftest/cases/rust.py new file +57 -0
@@ -0,0 +1,57 @@ +# SPDX-License-Identifier: MIT +import os +import subprocess +from oeqa.core.decorator import OETestTag +from oeqa.core.case import OEPTestResultTestCase +from oeqa.selftest.case import OESelftestTestCase +from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu, Command +from oeqa.utils.sshcontrol import SSHControl + +# Total time taken for testing is of about 2hr 20min, with PARALLEL_MAKE set to 40 number of jobs. +class RustSelfTestBase(OESelftestTestCase, OEPTestResultTestCase): + + def run_check_emulated(self, *args, **kwargs): + # build remote-test-server before image build + recipe = "rust-testsuite" + bitbake("{} -c compile".format(recipe)) + builddir = get_bb_var("B", "rust-testsuite") + # build core-image-minimal with required packages + default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp"] + features = [] + features.append('IMAGE_FEATURES += "ssh-server-openssh"') + features.append('CORE_IMAGE_EXTRA_INSTALL += "{0}"'.format(" ".join(default_installed_packages))) + self.write_config("\n".join(features)) + bitbake("core-image-minimal") + # wrap the execution with a qemu instance + with runqemu("core-image-minimal", runqemuparams = "nographic", qemuparams = "-m 512") as qemu: + # Copy remote-test-server to image through scp + ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root") + ssh.copy_to(builddir + "/" + "build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-server","~/") + # Execute remote-test-server on image through background ssh + command = '~/remote-test-server -v remote' + sshrun=subprocess.Popen(("ssh", '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-f', "root@%s" % qemu.ip, command), + shell=False, + stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + # Get the values of variables. + targetsys = get_bb_var("TARGET_SYS", "rust-testsuite") + rustlibpath = get_bb_var("STAGING_LIBDIR_NATIVE", "rust-testsuite") + tmpdir = get_bb_var("TMPDIR", "rust-testsuite") + + # Exclude the test folders that error out while building + # Need to fix these errors and include them for testing + testargs = "--exclude src/test/rustdoc-json --exclude src/test/run-make-fulldeps --exclude src/tools/tidy --exclude src/tools/rustfmt --exclude src/tools/rustdoc-themes --exclude src/tools/rust-demangler --exclude src/tools/rls --exclude src/tools/miri --exclude src/tools/linkchecker --exclude src/tools/expand-yaml-anchors --exclude src/tools/compiletest --exclude src/tools/clippy --exclude src/tools/cargotest --exclude src/tools/cargo --exclude src/test/rustdoc-gui --exclude src/rustdoc-json-types --exclude src/tools/rustdoc --exclude src/librustdoc --exclude src/doc/unstable-book --exclude src/doc/rustdoc --exclude src/doc/rustc --exclude compiler/rustc --exclude library/panic_abort --exclude library/panic_unwind --no-doc --no-fail-fast --bless" + + # Set path for target-poky-linux-gcc, RUST_TARGET_PATH and hosttools. + cmd = " export PATH=%s/../bin:$PATH;" % rustlibpath + cmd = cmd + " export PATH=%s/../bin/%s:%s/hosttools:$PATH;" % (rustlibpath, targetsys, tmpdir) + cmd = cmd + " export RUST_TARGET_PATH=%s/rustlib;" % rustlibpath + # Trigger testing. + cmd = cmd + " export TEST_DEVICE_ADDR=\"%s:12345\";" % qemu.ip + cmd = cmd + " cd %s; python3 src/bootstrap/bootstrap.py test %s --target %s ;" % (builddir, testargs, targetsys) + result = runCmd(cmd) + +@OETestTag("toolchain-system") +class RustSelfTestSystemEmulated(RustSelfTestBase): + def test_rust(self): + self.run_check_emulated("rust")
meta/recipes-devtools/rust/rust-testsuite.inc new file +163 -0
163 lines not on this page · raw message
meta/recipes-devtools/rust/rust-testsuite/rust-oe-selftest.patch new file +995 -0
995 lines not on this page · raw message
meta/recipes-devtools/rust/rust-testsuite_1.58.1.bb new file +3 -0
@@ -0,0 +1,3 @@ +require rust-testsuite.inc +require rust-source.inc +require rust-snapshot.inc
Signature · 1 line
2.31.1
Hi Naveen,
Quoted 1 line · Pgowda
The patch implements Oe-selftest framework for Rust test.
I think you need a bit more of an introduction to the patch. I assume that you and Vinay based this in part on the gcc.py test suite so you might say so and explain what is different and why.
Additionally, people will not be familiar with the rust test suite so you should try to summarize that in a few sentences if possible.
I have some suggestion below but in addition to that maybe you can include a link in this commit log to, and outline how this would differ from:
https://rustc-dev-guide.rust-lang.org/tests/intro.html
without discussing the implementation.
Quoted 1 line
Some of the functions are as follows:-
Rather than just listing the functions, say:
To support the rust tests in oe-core, the following functions were added:
Quoted 13 lines
setup_cargo_environment(): Build bootstrap and some early stage tools.do_rust_setup_snapshot(): Install the snapshot version of rust binaries.do_configure(): To generate config.tomldo_compile(): To build "remote-test-server" for qemutarget image.The python file builds remote-test-server and executes rust testing remotely using background ssh. It adds the necessary test environment and variables to run the rust oe-selftest.
Print the results in case of failure ofrunCmd().The patch has been run and tested for X86,
X86_64, Arm & Arm64 targets on Ubuntu-18 successfully.
Quoted 10 lines
Most of the tests excluded in
rust.pyare due to the following error:- These tests will be included in Oe-selftest after fixing the issue.
===============================================================
"build-st/tmp/work/core2-64-poky-linux/rust-testsuite/1.58.1-r0/ rustc-1.58.1-src/build/bootstrap/debug/rustc -vV` (exit status: 101) --- stderr
thread 'main' panicked at '
kind: NotFound, message: "No such file or directory" }',src/bootstrap/bin/rustc.rs:175:37
===============================================================
In this commit log, add a summary of APPROXIMATELY how many tests are run when building rust outside of a bitbake environment and compare that to:
1. tests run before skipping the broken ones
2. tests run with the patch below that skips higher level modules that are broken.
i.e.
on ubuntu, there are 12000 tests and 11,555 pass
before the disable patch, 9000 tests ran and 7777 passed
with the patch, 5500 test ran and they all passed.
I just want the reader (myself) to have an understanding of how much we have disabled. If you have exact numbers, that's great but I'm NOT asking you to spend hours or days gathering that data.
Quoted 36 lines
Signed-off-by pgowda <[email protected]>Signed-off-by Vinay Kumar <[email protected]>--- meta/lib/oeqa/selftest/cases/rust.py | 57 + meta/recipes-devtools/rust/rust-testsuite.inc | 163 +++ .../rust-testsuite/rust-oe-selftest.patch | 995 ++++++++++++++++++ .../rust/rust-testsuite_1.58.1.bb | 3 + 4 files changed, 1218 insertions(+)create mode 100644
meta/lib/oeqa/selftest/cases/rust.py
create mode 100644 meta/recipes-devtools/rust/rust-testsuite.inc create mode 100644meta/recipes-devtools/rust/rust-testsuite/rust-oe-selftest.patchcreate mode 100644 meta/recipes-devtools/rust/rust-testsuite_1.58.1.bbmeta/lib/oeqa/selftest/cases/rust.py new file +16
-0@@ -0,0 +1,57 @@ +# SPDX-License-Identifier: MIT +import os +import subprocess +from oeqa.core.decorator import OETestTag +from oeqa.core.case import OEPTestResultTestCase +from oeqa.selftest.case import OESelftestTestCase +from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu, Command +from oeqa.utils.sshcontrol import SSHControl + +# Total time taken for testing is of about 2hr 20min, with PARALLEL_MAKE set to 40 number of jobs. +class RustSelfTestBase(OESelftestTestCase, OEPTestResultTestCase): + + def run_check_emulated(self, *args, **kwargs): + # build remote-test-server before image build + recipe = "rust-testsuite" + bitbake("{} -c compile".format(recipe))
meta/lib/oeqa/selftest/cases/rust.py line 26
Quoted 10 lines
meta/lib/oeqa/selftest/cases/rust.py new file +10
-0+ builddir = get_bb_var("B", "rust-testsuite") + # build core-image-minimal with required packages + default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp"] + features = [] + features.append('IMAGE_FEATURES += "ssh-server-openssh"') + features.append('CORE_IMAGE_EXTRA_INSTALL += "{0}"'.format(" ".join(default_installed_packages))) + self.write_config("\n".join(features)) + bitbake("core-image-minimal") + # wrap the execution with a qemu instance + with runqemu("core-image-minimal", runqemuparams = "nographic", qemuparams = "-m 512") as qemu:
meta/lib/oeqa/selftest/cases/gcc.py has a quick test:
# validate that SSH is working
Given that you are about to run a long test, that seems worthwhile even if your next step is to copy over the test server.
Quoted 7 lines
meta/lib/oeqa/selftest/cases/rust.py new file +7
-0+ # Copy remote-test-server to image through scp + ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root") + ssh.copy_to(builddir + "/" + "build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-server","~/") + # Execute remote-test-server on image through background ssh + command = '~/remote-test-server -v remote' + sshrun=subprocess.Popen(("ssh", '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-f', "root@%s" % qemu.ip, command), + shell=False,
meta/lib/oeqa/selftest/cases/rust.py line 43
Quoted 10 lines
meta/lib/oeqa/selftest/cases/rust.py new file +10
-0+ stdout=subprocess.PIPE, + stderr=subprocess.PIPE) + # Get the values of variables. + targetsys = get_bb_var("TARGET_SYS", "rust-testsuite") + rustlibpath = get_bb_var("STAGING_LIBDIR_NATIVE", "rust-testsuite") + tmpdir = get_bb_var("TMPDIR", "rust-testsuite") + + # Exclude the test folders that error out while building + # Need to fix these errors and include them for testing + testargs = "--exclude src/test/rustdoc-json --exclude src/test/run-make-fulldeps --exclude src/tools/tidy --exclude src/tools/rustfmt --exclude src/tools/rustdoc-themes --exclude src/tools/rust-demangler --exclude src/tools/rls --exclude src/tools/miri --exclude src/tools/linkchecker --exclude src/tools/expand-yaml-anchors --exclude src/tools/compiletest --exclude src/tools/clippy --exclude src/tools/cargotest --exclude src/tools/cargo --exclude src/test/rustdoc-gui --exclude src/rustdoc-json-types --exclude src/tools/rustdoc --exclude src/librustdoc --exclude src/doc/unstable-book --exclude src/doc/rustdoc --exclude src/doc/rustc --exclude compiler/rustc --exclude library/panic_abort --exclude library/panic_unwind --no-doc --no-fail-fast --bless"
Can you add a comment in the commit log listing why these are excluded?
It seems that they are extra features that we haven't worked on supporting yet so if that's right, just say so.
Quoted 4 lines
meta/lib/oeqa/selftest/cases/rust.py new file +4
-0+ + # Set path for target-poky-linux-gcc, RUST_TARGET_PATH and hosttools. + cmd = " export PATH=%s/../bin:$PATH;" % rustlibpath + cmd = cmd + " export PATH=%s/../bin/%s:%s/hosttools:$PATH;" % (rustlibpath, targetsys, tmpdir)
meta/lib/oeqa/selftest/cases/rust.py line 57
Quoted 10 lines
meta/lib/oeqa/selftest/cases/rust.py new file +10
-0+ cmd = cmd + " export RUST_TARGET_PATH=%s/rustlib;" % rustlibpath + # Trigger testing. + cmd = cmd + " export TEST_DEVICE_ADDR=\"%s:12345\";" % qemu.ip + cmd = cmd + " cd %s; python3 src/bootstrap/bootstrap.py test %s --target %s ;" % (builddir, testargs, targetsys) + result = runCmd(cmd) + +@OETestTag("toolchain-system") +class RustSelfTestSystemEmulated(RustSelfTestBase): + def test_rust(self): + self.run_check_emulated("rust")
I haven't actually worked with it but in:
meta/lib/oeqa/selftest/cases/gcc.py
the tests can be run using qemu-system or qemu-user mode. It seems that either mode is possible for the gcc tests.
Nathan, Richard,
Should we support both modes and which is the default if you know offhand.
Are these gcc tests actually run in parallel in practice? We should look into what parallelism is possible in these rust tests but first we need to make it work even in this degraded (skipped tests) mode.
No comments on the rest of the patch so I've removed that.
I assume that you have moved the patch from one release to another so just out of curiosity, is that minutes, an hour, many hours or days of work to fix up when we next have to update the recipe?
Good work, hopefully it's almost ready to be merged.
Since all I've asked from you is changes to the commit log, I hope you can provide that quickly as v4 on this thread, and then people CCed here can provide additional comments.
Signature · 2 lines
# Randy MacLeod # Wind River Linux