Skip to contentSkip to conversation

Rust Oe-Selftest implementation

PATCH In review [email protected] 3 participants 5 messages Feb 23, 2023 – Feb 24, 2023 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

PATCH v12 1/1 Rust Oe-Selftest implementation

From: pgowda <[email protected]>

The patch implements Rust testing framework similar to other selftest, specifically the gcc selftest in OE. It uses the client and server based method to test the binaries for cross-target on the image. The test framework is a wrapper around the Rust build system as ./x.py test. It tests many functionalities of Rust distribution like tools, documentation, libraries, packages, tools, Cargo, Crater etc. Please refer the following link for detailed description of Rust testing:-
https://rustc-dev-guide.rust-lang.org/tests/intro.html#tool-tests

To support the rust tests in oe-core, the following functions were added:-
setup_cargo_environment(): Build bootstrap and some early stage tools. do_rust_setup_snapshot(): Install the snapshot version of rust binaries. do_configure(): To generate config.toml
do_compile(): To build "remote-test-server" for qemu target image.

Approximate Number of Tests Run in the Rust Testsuite :- 18000 Approximate Number of Tests that FAIL in bitbake environment :- 100-150 Normally majority of the testcases are present in major folder "test/" It contributes to more than 80% of the testcases present in Rust test framework. These tests pass as expected on any Rust versions without much fuss. The tests that fail are of less important and contribute to less than 2% of the total testcases. These minor tests are observed to work on some versions and fail on others. They have to be added, ignored or excluded for different versions as per the behavior. These tests have been ignored or excluded in the Rust selftest environment to generate success of completing the testsuite.

These tests work in parallel mode even in the skipped test mode as expected. Although the patch to disable tests is large, it is very simple in that it only disables tests. When updating to a newer version of Rust, the patch can usually be ported in a day.

Tested for X86, X86-64, ARM, ARM64 and MIPS64 on CentOS release 6.10

---
 meta/lib/oeqa/selftest/cases/rust.py          |  54 ++
 .../rust/files/rust-oe-selftest.patch         | 736 ++++++++++++++++++
 meta/recipes-devtools/rust/rust-source.inc    |   1 +
 meta/recipes-devtools/rust/rust_1.67.1.bb     |   6 +
 4 files changed, 797 insertions(+)
 create mode 100644 meta/lib/oeqa/selftest/cases/rust.py
 create mode 100644 meta/recipes-devtools/rust/files/rust-oe-selftest.patch
meta/lib/oeqa/selftest/cases/rust.py new file +54 -0
new file mode 100644
index 0000000000..f9889b840c
@@ -0,0 +1,54 @@
+# SPDX-License-Identifier: MIT
+import os
+import subprocess
+from oeqa.core.decorator import OETestTag
+from oeqa.core.case import OEPTestResultTestCase
+from oeqa.selftest.case import OESelftestTestCase
+from oeqa.utils.commands import runCmd, bitbake, get_bb_var, get_bb_vars, runqemu, Command
+from oeqa.utils.sshcontrol import SSHControl
+
+# Total time taken for testing is of about 2hr 20min, with PARALLEL_MAKE set to 40 number of jobs.
+class RustSelfTestSystemEmulated(OESelftestTestCase, OEPTestResultTestCase):
+    def test_rust(self, *args, **kwargs):
+        # build remote-test-server before image build
+        recipe = "rust"
+        bitbake("{} -c test_compile".format(recipe))
+        builddir = get_bb_var("RUSTSRC", "rust")
+        # build core-image-minimal with required packages
+        default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp"]
+        features = []
+        features.append('IMAGE_FEATURES += "ssh-server-dropbear"')
+        features.append('CORE_IMAGE_EXTRA_INSTALL += "{0}"'.format(" ".join(default_installed_packages)))
+        self.write_config("\n".join(features))
+        bitbake("core-image-minimal")
+        # wrap the execution with a qemu instance.
+        # Tests are run with 512 tasks in parallel to execute all tests very quickly
+        with runqemu("core-image-minimal", runqemuparams = "nographic", qemuparams = "-m 512") as qemu:
+            # Copy remote-test-server to image through scp
+            ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root")
+            ssh.copy_to(builddir + "/" + "build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-server","~/")
+            # Execute remote-test-server on image through background ssh
+            command = '~/remote-test-server --bind 0.0.0.0:12345 -v'
+            sshrun=subprocess.Popen(("ssh", '-o',  'UserKnownHostsFile=/dev/null', '-o',  'StrictHostKeyChecking=no', '-f', "root@%s" % qemu.ip, command), shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+            # Get the values of variables.
+            tcpath = get_bb_var("TARGET_SYS", "rust")
+            targetsys = get_bb_var("RUST_TARGET_SYS", "rust")
+            rustlibpath = get_bb_var("WORKDIR", "rust")
+            tmpdir = get_bb_var("TMPDIR", "rust")
+
+            # Exclude the test folders that error out while building
+            # TODO: Fix the errors and include them for testing
+            # no-fail-fast: Run all tests regardless of failure.
+            # bless: First runs rustfmt to format the codebase,
+            # then runs tidy checks.
+            testargs = "--exclude src/test/rustdoc --exclude src/tools/rust-analyzer  --exclude src/test/rustdoc-json  --exclude src/test/run-make-fulldeps --exclude src/tools/tidy --exclude src/tools/rustdoc-themes --exclude src/rustdoc-json-types --exclude src/librustdoc --exclude src/doc/unstable-book --exclude src/doc/rustdoc --exclude src/doc/rustc --exclude compiler/rustc --exclude library/panic_abort --exclude library/panic_unwind --exclude src/tools/lint-docs --exclude src/test/rustdoc-js-std --no-doc --no-fail-fast --bless"
+
+            # Set path for target-poky-linux-gcc, RUST_TARGET_PATH and hosttools.
+            cmd = " export PATH=%s/recipe-sysroot-native/usr/bin:$PATH;" % rustlibpath
+            cmd = cmd + " export TARGET_VENDOR=\"-poky\";"
+            cmd = cmd + " export PATH=%s/recipe-sysroot-native/usr/bin/%s:%s/hosttools:$PATH;" % (rustlibpath, tcpath, tmpdir)
+            cmd = cmd + " export RUST_TARGET_PATH=%s/rust-targets;" % rustlibpath
+            # Trigger testing.
+            cmd = cmd + " export TEST_DEVICE_ADDR=\"%s:12345\";" % qemu.ip
+            cmd = cmd + " cd %s;  python3 src/bootstrap/bootstrap.py test %s --target %s ;" % (builddir, testargs, targetsys)
+            result = runCmd(cmd)
meta/recipes-devtools/rust/files/rust-oe-selftest.patch new file +736 -0

736 lines not on this page · raw message

meta/recipes-devtools/rust/rust-source.inc +1 -0
index 942978e6bf..b79e5009a4 100644
@@ -6,6 +6,7 @@ SRC_URI += "https://static.rust-lang.org/dist/rustc-${RUST_VERSION}-src.tar.xz;n
             file://getrandom-open64.patch;patchdir=${RUSTSRC} \
             file://0001-Do-not-use-LFS64-on-linux-with-musl.patch;patchdir=${RUSTSRC} \
             file://zlib-off64_t.patch;patchdir=${RUSTSRC} \
+            file://rust-oe-selftest.patch;patchdir=${RUSTSRC} \
 "
 SRC_URI[rust.sha256sum] = "77e0615011f887d9533d5374bf9c15c590c3caf32bbb035b392d1c2ae502a682"
 
meta/recipes-devtools/rust/rust_1.67.1.bb +6 -0
index 2118faec8f..f8b64af4b7 100644
@@ -64,6 +64,7 @@ do_rust_setup_snapshot () {
     fi
 }
 addtask rust_setup_snapshot after do_unpack before do_configure
+addtask do_test_compile after do_configure do_rust_gen_targets
 do_rust_setup_snapshot[dirs] += "${WORKDIR}/rust-snapshot"
 do_rust_setup_snapshot[vardepsexclude] += "UNINATIVE_LOADER"
 
@@ -222,6 +223,11 @@ do_compile () {
     rust_runx build --stage 2
 }
 
+do_test_compile[dirs] = "${B}"
+do_test_compile () {
+    rust_runx build src/tools/remote-test-server --target "${RUST_TARGET_SYS}"
+}
+
 do_compile:append:class-target () {
     rust_runx build --stage 2 src/tools/clippy
     rust_runx build --stage 2 src/tools/rustfmt
Signature · 1 line
2.34.1
#1 lore raw

Thanks Naveen,

With a large patch like this it would be good if you could provide a cover letter that summarized what is different in v12, compared to v11.

I've looked at the diff and my summary is below. There are some changes which I see but would benefit from an explanation.

Finally, please change your SOB line to @windriver.com.

Summary of what changed in v12:
=====================

Rebase Rust tests to 1.67.1.

Switch from ssh-server-openssh to ssh-server-dropbear. --> Why?
I know you were having problems with scp not working for qemux86 but either server should work in theory.

Tell the Rust remote-test-server to bind to port 12345. --> Why?

Also add: --exclude src/tools/rust-analyzer to testargs. --> Why?
I don't object but ideally the commit log would explain this decision.

As far as the tests go, you've :

 -  added 3 more 'ignore-stage1' comments :

$ rg ignore-stage1 /tmp/rust-test-v11.eml | wc -l
41
$ rg ignore-stage1 /tmp/rust-test-v12.eml | wc -l
44

 - added 20 more #[ignore] directives:

$ rg "ignore]" /tmp/rust-test-v11.eml | wc -l
4
$ rg "ignore]" /tmp/rust-test-v12.eml  | wc -l
24

Those additional test skip directives doesn't change the overall pass rate significantly since the test suite has ~ 18,000 tests so they're find with me .

I think that's about it. Did I miss anything?

Key, very abbreviated differences b/w v11, v12 are shown below .

../Randy

+Tested for X86, X86-64, ARM, ARM64 and MIPS64 on CentOS release 6.10 +

 ---
  meta/lib/oeqa/selftest/cases/rust.py          |  54 ++
- meta/recipes-devtools/rust/rust-source.inc    |   3 +-
- meta/recipes-devtools/rust/rust.inc           |   1 +
- .../rust/rust/rust-oe-selftest.patch          | 508 ++++++++++++++++++
- meta/recipes-devtools/rust/rust_1.63.0.bb     |   5 +
- 5 files changed, 570 insertions(+), 1 deletion(-)
+ .../rust/files/rust-oe-selftest.patch         | 736 ++++++++++++++++++
+ meta/recipes-devtools/rust/rust-source.inc    |   1 +
+ meta/recipes-devtools/rust/rust_1.67.1.bb     |   6 +
+ 4 files changed, 797 insertions(+)
  create mode 100644 meta/lib/oeqa/selftest/cases/rust.py
- create mode 100644 meta/recipes-devtools/rust/rust/rust-oe-selftest.patch
+ create mode 100644 meta/recipes-devtools/rust/files/rust-oe-selftest.patch

 diff --git a/meta/lib/oeqa/selftest/cases/rust.py b/meta/lib/oeqa/selftest/cases/rust.py new file mode 100644
-index 0000000000..e06ddaa6d7
+index 0000000000..f9889b840c
 --- /dev/null
 +++ b/meta/lib/oeqa/selftest/cases/rust.py
 @@ -0,0 +1,54 @@
@@ -237,7 +239,7 @@
 +        # build core-image-minimal with required packages
 +        default_installed_packages = ["libgcc", "libstdc++", "libatomic", "libgomp"]
 +        features = []
-+        features.append('IMAGE_FEATURES += "ssh-server-openssh"')
++        features.append('IMAGE_FEATURES += "ssh-server-dropbear"')
 +        features.append('CORE_IMAGE_EXTRA_INSTALL += "{0}"'.format(" ".join(default_installed_packages))) +        self.write_config("\n".join(features))
 +        bitbake("core-image-minimal")

@@ -248,7 +250,7 @@
 +            ssh = SSHControl(ip=qemu.ip, logfile=qemu.sshlog, user="root")
 +            ssh.copy_to(builddir + "/" + "build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-server","~/") +            # Execute remote-test-server on image through background ssh
-+            command = '~/remote-test-server -v remote'
++            command = '~/remote-test-server --bind 0.0.0.0:12345 -v'

Signature · 2 lines
# Randy MacLeod
# Wind River Linux
#2 lore raw
Quoted 6 lines · Randy MacLeod

Thanks Naveen,

With a large patch like this it would be good if you could provide a cover letter that summarized what is different in v12, compared to v11.

Quoted 6 lines

I've looked at the diff and my summary is below. There are some changes
 which I see but would benefit from an explanation.

Finally, please change your SOB line to @windriver.com.

Changelogs definitely help, thanks Randy!

Thanks for putting this patch together Naveen, we are getting there. I ran this through the autobuilder and the good news is we only saw one failure related to the patch, in oe-selftest on an ARM host:

https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1004
(from
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4962
)

We could disable the rust tests on arm host systems to get this to merge however ideally it would be fixed to run there properly as well. I'm not even sure this should actually be running there at all now I look at things. The other toolchain/machine specific tests are tagged as "machine", e.g. meta_ide.py has:

@OETestTag("machine")
class MetaIDE(OESelftestTestCase):

or they're marked as @OETestTag("toolchain-user") or @OETestTag("toolchain-system") in the case of the glibc/gcc tests. The difference is how the tests are run so "toolchain-system" would be most appropriate for rust. This means that the tests are run per MACHINE target instead of once overall.

The second issue I noticed is that the test results are not appearing in the test report. For the build above, this is here:

https://autobuilder.yocto.io/pub/non-release/20230223-9/testresults/testresult-report.txt

and I can't see the rust tests at all. This will need to be fixed before the patches can merge. It may be changing to be machine/toolchain might fix that, I don't remember the details.

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 43 lines · Richard Purdie via lists.openembedded.org
Quoted 6 lines · Randy MacLeod

Thanks Naveen,

With a large patch like this it would be good if you could provide a cover letter that summarized what is different in v12, compared to v11.

Quoted 6 lines

I've looked at the diff and my summary is below. There are some changes
 which I see but would benefit from an explanation.

Finally, please change your SOB line to @windriver.com.

Changelogs definitely help, thanks Randy!

Thanks for putting this patch together Naveen, we are getting there. I ran this through the autobuilder and the good news is we only saw one failure related to the patch, in oe-selftest on an ARM host:

https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1004
(from
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4962
)

We could disable the rust tests on arm host systems to get this to merge however ideally it would be fixed to run there properly as well. I'm not even sure this should actually be running there at all now I look at things. The other toolchain/machine specific tests are tagged as "machine", e.g. meta_ide.py has:

@OETestTag("machine")
class MetaIDE(OESelftestTestCase):

or they're marked as @OETestTag("toolchain-user") or @OETestTag("toolchain-system") in the case of the glibc/gcc tests. The difference is how the tests are run so "toolchain-system" would be most appropriate for rust. This means that the tests are run per MACHINE target instead of once overall.

The second issue I noticed is that the test results are not appearing in the test report. For the build above, this is here:

Quoted 6 lines

https://autobuilder.yocto.io/pub/non-release/20230223-9/testresults/testresult-report.txt

and I can't see the rust tests at all. This will need to be fixed before the patches can merge. It may be changing to be machine/toolchain might fix that, I don't remember the details.

I found some time to check into this a little further. We definitely need to mark the test as a toolchain-system test so it is run in the right places.

Aside from that, there is also no individual test result reporting in place for it, either the test suite passes or it does not.

With the gcc tests, you'll see there is the self.ptest_result() call which logs the individual test pass/fail/skip value. We'll need to do something similar with rust so we don't just have one overall pass/fail.

Cheers,

Richard

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 56 lines · Richard Purdie via lists.openembedded.org
Quoted 43 lines · Richard Purdie via lists.openembedded.org
Quoted 6 lines · Randy MacLeod

Thanks Naveen,

With a large patch like this it would be good if you could provide a cover letter that summarized what is different in v12, compared to v11.

Quoted 6 lines

I've looked at the diff and my summary is below. There are some changes
 which I see but would benefit from an explanation.

Finally, please change your SOB line to @windriver.com.

Changelogs definitely help, thanks Randy!

Thanks for putting this patch together Naveen, we are getting there. I ran this through the autobuilder and the good news is we only saw one failure related to the patch, in oe-selftest on an ARM host:

https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/1004
(from
https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/4962
)

We could disable the rust tests on arm host systems to get this to merge however ideally it would be fixed to run there properly as well. I'm not even sure this should actually be running there at all now I look at things. The other toolchain/machine specific tests are tagged as "machine", e.g. meta_ide.py has:

@OETestTag("machine")
class MetaIDE(OESelftestTestCase):

or they're marked as @OETestTag("toolchain-user") or @OETestTag("toolchain-system") in the case of the glibc/gcc tests. The difference is how the tests are run so "toolchain-system" would be most appropriate for rust. This means that the tests are run per MACHINE target instead of once overall.

The second issue I noticed is that the test results are not appearing in the test report. For the build above, this is here:

Quoted 6 lines

https://autobuilder.yocto.io/pub/non-release/20230223-9/testresults/testresult-report.txt

and I can't see the rust tests at all. This will need to be fixed before the patches can merge. It may be changing to be machine/toolchain might fix that, I don't remember the details.

I found some time to check into this a little further. We definitely need to mark the test as a toolchain-system test so it is run in the right places.

Quoted 7 lines

Aside from that, there is also no individual test result reporting in place for it, either the test suite passes or it does not.

With the gcc tests, you'll see there is the self.ptest_result() call which logs the individual test pass/fail/skip value. We'll need to do something similar with rust so we don't just have one overall pass/fail.

I reran this with the OTTestTag set. With that set, the tests run in the correct places and we then see failures for x86, mips and ppc:

https://autobuilder.yoctoproject.org/typhoon/#/builders/59/builds/6663
https://autobuilder.yoctoproject.org/typhoon/#/builders/60/builds/6682
https://autobuilder.yoctoproject.org/typhoon/#/builders/63/builds/6672

AssertionError: Command '['scp', '-o', 'UserKnownHostsFile=/dev/null', '-o', 'StrictHostKeyChecking=no', '-o', 'LogLevel=ERROR', '/home/pokybuild/yocto-worker/qemux86/build/build-st-2169186/tmp/work/core2-32-poky-linux/rust/1.67.1-r0/rustc-1.67.1-src/build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-server', '[email protected]:~/']' returned non-zero exit status 1:
b'scp: dest open(""): No such file or directory\r\nscp: failed to upload file /home/pokybuild/yocto-worker/qemux86/build/build-st-2169186/tmp/work/core2-32-poky-linux/rust/1.67.1-r0/rustc-1.67.1-src/build/x86_64-unknown-linux-gnu/stage1-tools-bin/remote-test-server to'

which might be because it is trying to copy a file from /x86_64- unknown-linux-gnu/ when it is a 32 bit target?

Cheers,

Richard