Skip to contentSkip to conversation

Rust Oe-Selftest implementation

PATCH Superseded Yash Shinde 2 participants 3 messages Apr 18, 2023 – Apr 19, 2023

v13 of 14 Superseded Latest is v14 →

lore
Share & tools
lore
#0 lore raw

PATCH v13 1/1 Rust Oe-Selftest implementation

The patch implements Rust testing framework similar to other selftest, specifically the gcc selftest in OE. It uses the client and server based method to test the binaries for cross-target on the image. The test framework is a wrapper around the Rust build system as ./x.py test. It tests many functionalities of Rust distribution like tools, documentation, libraries, packages, tools, Cargo, Crater etc. Please refer the following link for detailed description of Rust testing:-
https://rustc-dev-guide.rust-lang.org/tests/intro.html#tool-tests

To support the rust tests in oe-core, the following functions were added:-
setup_cargo_environment(): Build bootstrap and some early stage tools. do_rust_setup_snapshot(): Install the snapshot version of rust binaries. do_configure(): To generate config.toml
do_compile(): To build "remote-test-server" for qemu target image.

Approximate Number of Tests Run in the Rust Testsuite :- 18000 Approximate Number of Tests that FAIL in bitbake environment :- 100-150 Normally majority of the testcases are present in major folder "test/" It contributes to more than 80% of the testcases present in Rust test framework. These tests pass as expected on any Rust versions without much fuss. The tests that fail are of less important and contribute to less than 2% of the total testcases. These minor tests are observed to work on some versions and fail on others. They have to be added, ignored or excluded for different versions as per the behavior. These tests have been ignored or excluded in the Rust selftest environment to generate success of completing the testsuite.

These tests work in parallel mode even in the skipped test mode as expected. Although the patch to disable tests is large, it is very simple in that it only disables tests. When updating to a newer version of Rust, the patch can usually be ported in a day.

Tested for X86, X86-64, ARM, ARM64 and MIPS64 on CentOS release 6.10

---
 meta/lib/oeqa/selftest/cases/rust.py          |  85 +++
 .../rust/files/rust-oe-selftest.patch         | 715 ++++++++++++++++++
 meta/recipes-devtools/rust/rust-source.inc    |   1 +
 meta/recipes-devtools/rust/rust_1.68.2.bb     |  18 +
 4 files changed, 819 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/rust.py
 create mode 100644 meta/recipes-devtools/rust/files/rust-oe-selftest.patch
meta/lib/oeqa/selftest/cases/rust.py new file +85 -0
new file mode 100644
index 0000000000..f328d2d887
@@ -0,0 +1,85 @@
+# SPDX-License-Identifier: MIT
+import os
+import subprocess
+from oeqa.core.decorator import OETestTag
+from oeqa.core.case import OEPTestResultTestCase
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu, Command
+from oeqa.utils.sshcontrol import SSHControl
+
+def parse_results(filename):
+    tests = []
+    with open(filename, "r") as f:
+        lines = f.readlines()
+        for line in lines:
+            if "..." in line and "test [" in line:
+                test = line.split("test ")[1].split(" ... ")[0]
+                result = line.split(" ... ")[1].strip()
+                if result == "ok":
+                    result = "PASS"
+                elif result == "falied":
+                    result = "FAIL"
+                elif result == "ignored":
+                    result = "SKIP"
+                tests.append((test, result))
+    return tests
+
+# Total time taken for testing is of about 2hr 20min, with PARALLEL_MAKE set to 40 number of jobs.
+class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase):
+    def test_rust(self, *args, **kwargs):
+        # build remote-test-server before image build
+        recipe = "rust"
+        bitbake("{} -c test_compile".format(recipe))
+        builddir = get_bb_var("RUSTSRC", "rust")
+        # build core-image-minimal with required packages
+        default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp"]
+        features = []
+        features.append('IMAGE_FEATURES += "ssh-server-dropbear"')
+        features.append('CORE_IMAGE_EXTRA_INSTALL += "{0}"'.format(" ".join(default_installed_packages)))
+        self.write_config("\n".join(features))
+        bitbake("core-image-minimal")
+        # wrap the execution with a qemu instance.
+        # Tests are run with 512 tasks in parallel to execute all tests very quickly
+        with runqemu("core-image-minimal", runqemuparams = "nographic", qemuparams = "-m 512") as qemu:
+            # Copy remote-test-server to image through scp
+            buildsys = get_bb_var("RUST_BUILD_SYS", "rust")
+            ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root")
+            ssh.copy_to(builddir + "/" + "build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-server","~/")
+            # Execute remote-test-server on image through background ssh
+            command = '~/remote-test-server --bind 0.0.0.0:12345 -v'
+            sshrun=subprocess.Popen(("ssh", '-o',  'UserKnownHostsFile=/dev/null', '-o',  'StrictHostKeyChecking=no', '-f', "root@%s" % qemu.ip, command), shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            # Get the values of variables.
+            tcpath = get_bb_var("TARGET_SYS", "rust")
+            targetsys = get_bb_var("RUST_TARGET_SYS", "rust")
+            rustlibpath = get_bb_var("WORKDIR", "rust")
+            tmpdir = get_bb_var("TMPDIR", "rust")
+
+            # Exclude the test folders that error out while building
+            # TODO: Fix the errors and include them for testing
+            # no-fail-fast: Run all tests regardless of failure.
+            # bless: First runs rustfmt to format the codebase,
+            # then runs tidy checks.
+            testargs = "--exclude src/test/rustdoc --exclude src/tools/rust-analyzer  --exclude src/test/rustdoc-json  --exclude src/test/run-make-fulldeps --exclude src/tools/tidy --exclude src/tools/rustdoc-themes --exclude src/rustdoc-json-types --exclude src/librustdoc --exclude src/doc/unstable-book --exclude src/doc/rustdoc --exclude src/doc/rustc --exclude compiler/rustc --exclude library/panic_abort --exclude library/panic_unwind --exclude src/tools/lint-docs --exclude src/test/rustdoc-js-std --no-doc --no-fail-fast --bless"
+
+            # Set path for target-poky-linux-gcc, RUST_TARGET_PATH and hosttools.
+            cmd = " export PATH=%s/recipe-sysroot-native/usr/bin:$PATH;" % rustlibpath
+            cmd = cmd + " export TARGET_VENDOR=\"-poky\";"
+            cmd = cmd + " export PATH=%s/recipe-sysroot-native/usr/bin/%s:%s/hosttools:$PATH;" % (rustlibpath, tcpath, tmpdir)
+            cmd = cmd + " export RUST_TARGET_PATH=%s/rust-targets;" % rustlibpath
+            # Trigger testing.
+            cmd = cmd + " export TEST_DEVICE_ADDR=\"%s:12345\";" % qemu.ip
+            cmd = cmd + " cd %s;  python3 src/bootstrap/bootstrap.py test %s --target %s > summary.txt 2>&1 ;" % (builddir, testargs, targetsys)
+            runCmd(cmd)
+
+            ptestsuite = "rust"
+            self.ptest_section(ptestsuite, logfile = builddir + "/summary.txt")
+            filename = builddir + "/summary.txt"
+            test_results = parse_results(filename)
+            for test, result in test_results:
+                self.ptest_result(ptestsuite, test, result)
+
+@OETestTag("toolchain-system")
+@OETestTag("runqemu")
+class RustSelfTestBase(RustSelfTestSystemEmulated):
+    def test_check(self):
+        self.test_rust()
meta/recipes-devtools/rust/files/rust-oe-selftest.patch new file +715 -0

715 lines not on this page · raw message

meta/recipes-devtools/rust/rust-source.inc +1 -0
index b25b5c17e8..c0c02c4594 100644
@@ -7,6 +7,7 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n
             file://0001-Do-not-use-LFS64-on-linux-with-musl.patch;patchdir=${RUSTSRC} \
             file://zlib-off64_t.patch;patchdir=${RUSTSRC} \
             file://0001-musl-Define-SOCK_SEQPACKET-in-common-place.patch;patchdir=${RUSTSRC} \
+            file://rust-oe-selftest.patch;patchdir=${RUSTSRC} \
 "
 SRC_URI[rust.sha256sum] = "ce1a115f6aafa912b4622906a92b626354973afa9288e2c7750df4dcf3390fc0"
 
meta/recipes-devtools/rust/rust_1.68.2.bb +18 -0
index 83f9bec68a..27ba44f7bd 100644
@@ -66,6 +66,7 @@ do_rust_setup_snapshot () {
     fi
 }
 addtask rust_setup_snapshot after do_unpack before do_configure
+addtask do_test_compile after do_configure do_rust_gen_targets
 do_rust_setup_snapshot[dirs] += "${WORKDIR}/rust-snapshot"
 do_rust_setup_snapshot[vardepsexclude] += "UNINATIVE_LOADER"
 
@@ -221,8 +222,25 @@ FILES:${PN} += "${libdir}/*.so"
 FILES:${PN}-dev = ""
 
 do_compile () {
+    rust_runx build --stage 2
 }
 
+do_test_compile[dirs] = "${B}"
+do_test_compile () {
+    rust_runx build src/tools/remote-test-server --target "${RUST_TARGET_SYS}"
+}
+
+do_compile:append:class-target () {
+    rust_runx build --stage 2 src/tools/clippy
+    rust_runx build --stage 2 src/tools/rustfmt
+}
+
+do_compile:append:class-nativesdk () {
+    rust_runx build --stage 2 src/tools/clippy
+    rust_runx build --stage 2 src/tools/rustfmt
+}
+
+
 ALLOW_EMPTY:${PN} = "1"
 
 PACKAGES =+ "${PN}-tools-clippy ${PN}-tools-rustfmt"
Signature · 1 line
2.34.1
#1 lore raw

Hello Yash,

Quoted 34 lines · On Tue, 18 Apr 2023 14:40:13 +0530 "Yash Shinde" <[email protected]> wrote:

The patch implements Rust testing framework similar to other selftest, specifically the gcc selftest in OE. It uses the client and server based method to test the binaries for cross-target on the image. The test framework is a wrapper around the Rust build system as ./x.py test. It tests many functionalities of Rust distribution like tools, documentation, libraries, packages, tools, Cargo, Crater etc. Please refer the following link for detailed description of Rust testing:-
https://rustc-dev-guide.rust-lang.org/tests/intro.html#tool-tests

To support the rust tests in oe-core, the following functions were added:-
setup_cargo_environment(): Build bootstrap and some early stage tools. do_rust_setup_snapshot(): Install the snapshot version of rust binaries. do_configure(): To generate config.toml
do_compile(): To build "remote-test-server" for qemu target image.

Approximate Number of Tests Run in the Rust Testsuite :- 18000 Approximate Number of Tests that FAIL in bitbake environment :- 100-150 Normally majority of the testcases are present in major folder "test/" It contributes to more than 80% of the testcases present in Rust test framework. These tests pass as expected on any Rust versions without much fuss. The tests that fail are of less important and contribute to less than 2% of the total testcases. These minor tests are observed to work on some versions and fail on others. They have to be added, ignored or excluded for different versions as per the behavior. These tests have been ignored or excluded in the Rust selftest environment to generate success of completing the testsuite.

These tests work in parallel mode even in the skipped test mode as expected. Although the patch to disable tests is large, it is very simple in that it only disables tests. When updating to a newer version of Rust, the patch can usually be ported in a day.

Quoted 5 lines

Tested for X86, X86-64, ARM, ARM64 and MIPS64 on CentOS release 6.10

Testing with this patch is causing a lot of failures. Here are some logs:

https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/6944/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/6344/steps/16/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/100/builds/4337/steps/12/logs/stdio
https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/6998/steps/11/logs/stdio

And here's the entire a-full build in case you need more info:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5194

Best regards,
Luca

Signature · 3 lines
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com
#2 lore raw

Thanks for sharing the logs. I will check the failures.

Regards,
Yash

Quoted 69 lines · Luca Ceresoli,

Hello Yash,

Quoted 34 lines · On Tue, 18 Apr 2023 14:40:13 +0530 "Yash Shinde" <[email protected]> wrote:

The patch implements Rust testing framework similar to other selftest, specifically the gcc selftest in OE. It uses the client and server based method to test the binaries for cross-target on the image. The test framework is a wrapper around the Rust build system as ./x.py test. It tests many functionalities of Rust distribution like tools, documentation, libraries, packages, tools, Cargo, Crater etc. Please refer the following link for detailed description of Rust testing:-
https://rustc-dev-guide.rust-lang.org/tests/intro.html#tool-tests

To support the rust tests in oe-core, the following functions were added:-
setup_cargo_environment(): Build bootstrap and some early stage tools. do_rust_setup_snapshot(): Install the snapshot version of rust binaries. do_configure(): To generate config.toml
do_compile(): To build "remote-test-server" for qemu target image.

Approximate Number of Tests Run in the Rust Testsuite :- 18000 Approximate Number of Tests that FAIL in bitbake environment :- 100-150 Normally majority of the testcases are present in major folder "test/" It contributes to more than 80% of the testcases present in Rust test framework. These tests pass as expected on any Rust versions without much fuss. The tests that fail are of less important and contribute to less than 2% of the total testcases. These minor tests are observed to work on some versions and fail on others. They have to be added, ignored or excluded for different versions as per the behavior. These tests have been ignored or excluded in the Rust selftest environment to generate success of completing the testsuite.

These tests work in parallel mode even in the skipped test mode as expected. Although the patch to disable tests is large, it is very simple in that it only disables tests. When updating to a newer version of Rust, the patch can usually be ported in a day.

Quoted 5 lines

Tested for X86, X86-64, ARM, ARM64 and MIPS64 on CentOS release 6.10

Testing with this patch is causing a lot of failures. Here are some logs:

https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/6944/steps/12/logs/stdio

https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/6344/steps/16/logs/stdio

https://autobuilder.yoctoproject.org/typhoon/#/builders/100/builds/4337/steps/12/logs/stdio

https://autobuilder.yoctoproject.org/typhoon/#/builders/37/builds/6998/steps/11/logs/stdio

And here's the entire a-full build in case you need more info:

https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/5194

Best regards,
Luca

--
Luca Ceresoli, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com