docs/manual: run-tests test framework
PATCH v4 1/1 docs/manual: run-tests test framework
This patch adds a new manual section that captures an overview of the run-tests tool, how to manually run a test and where to find the test case script.
A brief set of steps is included to go through how to add a new test case and suggestions on how to test/debug.
---
Changes
v3 -> v4
[Yegor
- Fixed some typos/wording
v2 -> v3
[Thomas
- Created new sections for use / debug / create test cases
- Rewording of how gitlab ci is used in the descriptions
- Moved around section material for use vs test creation
- Expanded gitlab branch naming for job execution section instead of referencing commit hash description
- Added brief notes on test creation
[Romain
- capitalized Buildroot
- added notes about default defconfig
--- docs/manual/contribute.txt | 167 +++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+)
docs/manual/contribute.txt +167 -0
@@ -487,3 +487,170 @@ preserve Unix-style line terminators when downloading raw pastes. Following pastebin services are known to work correctly: - https://gist.github.com/ - http://code.bulix.org/ + +=== Using the run-tests framework + +Buildroot includes a run-time testing framework called run-tests built +upon Python scripting and QEMU runtime execution. There are two types of +test cases within the framework, one for build time tests and another for +run-time tests that have a QEMU dependency. The goals of the framework are +the following. + +* Build a well defined configuration +* Optionally, verify some properties of the build output +* If a run-time test, boot it under QEMU +* Lastly, run some test condition to verify that a given feature is working + +The run-tests tool has a series of options documented in the tool's help '-h' +description. Some common options include setting the download folder, the +output folder, keeping build output, and for multiple test cases, you can set +the JLEVEL for each. + +Here is an example walk through of running a test case. + +* For a first step, let us see what all the test case options are. The test +cases can be listed by executing +support/testing/run-tests -l+. These tests +can all be run individually during test development from the console. Both +one at a time and selectively as a group of a subset of tests. + +--------------------- +$ support/testing/run-tests -l +List of tests +test_run (tests.utils.test_check_package.TestCheckPackage) +Test the various ways the script can be called in a simple top to ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainBuildrootMusl) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainBuildrootuClibc) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainCCache) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainCtngMusl) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainLinaroArm) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv4) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv5) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv7) ... ok +[snip] +test_run (tests.init.test_systemd.TestInitSystemSystemdRoFull) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRoIfupdown) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRoNetworkd) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRwFull) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRwIfupdown) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRwNetworkd) ... ok +test_run (tests.init.test_busybox.TestInitSystemBusyboxRo) ... ok +test_run (tests.init.test_busybox.TestInitSystemBusyboxRoNet) ... ok +test_run (tests.init.test_busybox.TestInitSystemBusyboxRw) ... ok +test_run (tests.init.test_busybox.TestInitSystemBusyboxRwNet) ... ok + +Ran 157 tests in 0.021s + +OK +--------------------- + +Those runtime tests are regularly executed by Buildroot Gitlab CI +infrastructure, see .gitlab.yml and https://gitlab.com/buildroot.org/buildroot/-/jobs. + +==== Debugging a test case + +Within the Buildroot repository, the testing framework is organized at the +top level in +support/testing/+ by folders of +conf+, +infra+ and +tests+. +All the test cases live under the +test+ folder and are organized in various +folders representing the catagory of test. + +Lets walk through an example. + +* Using the Busybox Init system test case with a read/write rootfs ++tests.init.test_busybox.TestInitSystemBusyboxRw+ +* A minimal set of command line arguments when debugging a test case would +include '-d' which points to your dl folder, '-o' to an output folder, and +'-k' to keep any output on both pass/fail. With those options, the test will +retain logging and build artifacts providing status of the build and +execution of the test case. + +--------------------- +$ support/testing/run-tests -d dl -o output_folder -k tests.init.test_busybox.TestInitSystemBusyboxRw +15:03:26 TestInitSystemBusyboxRw Starting +15:03:28 TestInitSystemBusyboxRw Building +15:08:18 TestInitSystemBusyboxRw Building done +15:08:27 TestInitSystemBusyboxRw Cleaning up +. +Ran 1 test in 301.140s + +OK +--------------------- + +* For the case of a successful build, the +output_folder+ would contain a +<test name> folder with the Buildroot build, build log and run-time log. If +the build failed, the console output would show the stage at which it failed +(setup / build / run). Depending on the failure stage, the build/run logs +and/or Buildroot build artifacts can be inspected and instrumented. If the +QEMU instance needs to be launched for additional testing, the first few +lines of the run-time log capture it and it would allow some incremental +testing without re-running +support/testing/run-tests+. + +* You can also make modifications to the current sources inside the ++output_folder+ (e.g. for debug purposes) and rerun the standard +Buildroot make targets (in order to regenerate the complete image with +the new modifications) and then rerun the test. Modifying the sources +directly can speed up debugging compared to adding patch files, wiping the +output directoy, and starting the test again. + +--------------------- +$ ls output_folder/ +TestInitSystemBusyboxRw/ +TestInitSystemBusyboxRw-build.log +TestInitSystemBusyboxRw-run.log +--------------------- + +* The source file used to implement this example test is found under ++support/testing/tests/init/test_busybox.py+. This file outlines the +minimal defconfig that creates the build, QEMU configuration to launch +the built images and the test case assertions. + +To test an existing or new test case within Gitlab CI, there is a method of +invoking a specific test by creating a Buildroot fork in Gitlab under your +account. This can be handy when adding/changing a run-time test or fixing a +bug on a use case tested by a run-time test case. + + +In the examples below, the <name> component of the branch name is a unique +string you choose to identify this specific job being created. + +* to trigger all run-test test case jobs: + +--------------------- + $ git push gitlab HEAD:<name>-runtime-tests +--------------------- + +* to trigger one test case job, a specific branch naming string is used that +includes the full test case name. + +--------------------- + $ git push gitlab HEAD:<name>-<test case name> +--------------------- + +==== Creating a test case + +The best way to get familiar with how to create a test case is to look at a +few of the basic file system +support/testing/tests/fs/+ and init ++support/testing/tests/init/+ test scripts. Those tests give good examples +of a basic build and build with run type of tests. There are other more +advanced cases that use things like nested +br2-external+ folders to provide +skeletons and additional packages. + +The test cases by default use a br-arm-full-* uClibc-ng toolchain and the +prebuild kernel for a armv5/7 cpu. It is recommended to use the default +defconfig test configuration except when Glibc/musl or a newer kernel are +necessary. By using the default it saves build time and the test would +automatically inherit a kernel/std library upgrade when the default is +updated. + +The basic test case definition involves + +* Creation of a new test file +* Defining a unique test class +* Determining if the default defconfig plus test options can be used +* Implementing a +def test_run(self):+ function to optionally startup the +emulator and provide test case conditions. + +Beyond creating the test script, there are a couple of additional steps that +should be taken once you have your initial test case script. The first is +to add yourself to the +DEVELOPERS+ file to be the maintainer of that test +case. The second is to update the Gitlab CI yml by executing ++make .gitlab-ci.yml+.
Signature · 1 line
2.18.0
Yann E. MORIN
AppliedPATCH v4 1/1 docs/manual: run-tests test framework
Matthew, All,
On 2020-01-14 09:04 -0600, Matt Weber spake thusly:
Quoted 10 lines
This patch adds a new manual section that captures an overview of the run-tests tool, how to manually run a test and where to find the test case script.
A brief set of steps is included to go through how to add a new test case and suggestions on how to test/debug.
Cc Ricardo Martincoski <[email protected]>Cc Yegor Yefremov <[email protected]>Signed-off-by Matthew Weber <[email protected]>
Applied to master with those changes:
- switch the creating and debugging sections
- minor reformatting
Thanks!
Regards,
Yann E. MORIN.
Quoted 205 lines
---
Changes
v3 -> v4
[Yegor
- Fixed some typos/wordingv2 -> v3
[Thomas
- Created new sections for use / debug / create test cases
- Rewording of how gitlab ci is used in the descriptions
- Moved around section material for use vs test creation
- Expanded gitlab branch naming for job execution section instead of referencing commit hash description
- Added brief notes on test creation[Romain
- capitalized Buildroot
- added notes about default defconfig--- docs/manual/contribute.txt | 167 +++++++++++++++++++++++++++++++++++++ 1 file changed, 167 insertions(+)docs/manual/contribute.txt +167
-0@@ -487,3 +487,170 @@ preserve Unix-style line terminators when downloading raw pastes. Following pastebin services are known to work correctly: - https://gist.github.com/ - http://code.bulix.org/ + +=== Using the run-tests framework + +Buildroot includes a run-time testing framework called run-tests built +upon Python scripting and QEMU runtime execution. There are two types of +test cases within the framework, one for build time tests and another for +run-time tests that have a QEMU dependency. The goals of the framework are +the following. + +* Build a well defined configuration +* Optionally, verify some properties of the build output +* If a run-time test, boot it under QEMU +* Lastly, run some test condition to verify that a given feature is working + +The run-tests tool has a series of options documented in the tool's help '-h' +description. Some common options include setting the download folder, the +output folder, keeping build output, and for multiple test cases, you can set +the JLEVEL for each. + +Here is an example walk through of running a test case. + +* For a first step, let us see what all the test case options are. The test +cases can be listed by executing +support/testing/run-tests -l+. These tests +can all be run individually during test development from the console. Both +one at a time and selectively as a group of a subset of tests. + +--------------------- +$ support/testing/run-tests -l +List of tests +test_run (tests.utils.test_check_package.TestCheckPackage) +Test the various ways the script can be called in a simple top to ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainBuildrootMusl) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainBuildrootuClibc) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainCCache) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainCtngMusl) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainLinaroArm) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv4) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv5) ... ok +test_run (tests.toolchain.test_external.TestExternalToolchainSourceryArmv7) ... ok +[snip] +test_run (tests.init.test_systemd.TestInitSystemSystemdRoFull) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRoIfupdown) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRoNetworkd) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRwFull) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRwIfupdown) ... ok +test_run (tests.init.test_systemd.TestInitSystemSystemdRwNetworkd) ... ok +test_run (tests.init.test_busybox.TestInitSystemBusyboxRo) ... ok +test_run (tests.init.test_busybox.TestInitSystemBusyboxRoNet) ... ok +test_run (tests.init.test_busybox.TestInitSystemBusyboxRw) ... ok +test_run (tests.init.test_busybox.TestInitSystemBusyboxRwNet) ... ok + +Ran 157 tests in 0.021s + +OK +--------------------- + +Those runtime tests are regularly executed by Buildroot Gitlab CI +infrastructure, see .gitlab.yml and https://gitlab.com/buildroot.org/buildroot/-/jobs. + +==== Debugging a test case + +Within the Buildroot repository, the testing framework is organized at the +top level in +support/testing/+ by folders of +conf+, +infra+ and +tests+. +All the test cases live under the +test+ folder and are organized in various +folders representing the catagory of test. + +Lets walk through an example. + +* Using the Busybox Init system test case with a read/write rootfs ++tests.init.test_busybox.TestInitSystemBusyboxRw+ +* A minimal set of command line arguments when debugging a test case would +include '-d' which points to your dl folder, '-o' to an output folder, and +'-k' to keep any output on both pass/fail. With those options, the test will +retain logging and build artifacts providing status of the build and +execution of the test case. + +--------------------- +$ support/testing/run-tests -d dl -o output_folder -k tests.init.test_busybox.TestInitSystemBusyboxRw +15:03:26 TestInitSystemBusyboxRw Starting +15:03:28 TestInitSystemBusyboxRw Building +15:08:18 TestInitSystemBusyboxRw Building done +15:08:27 TestInitSystemBusyboxRw Cleaning up +. +Ran 1 test in 301.140s + +OK +--------------------- + +* For the case of a successful build, the +output_folder+ would contain a +<test name> folder with the Buildroot build, build log and run-time log. If +the build failed, the console output would show the stage at which it failed +(setup / build / run). Depending on the failure stage, the build/run logs +and/or Buildroot build artifacts can be inspected and instrumented. If the +QEMU instance needs to be launched for additional testing, the first few +lines of the run-time log capture it and it would allow some incremental +testing without re-running +support/testing/run-tests+. + +* You can also make modifications to the current sources inside the ++output_folder+ (e.g. for debug purposes) and rerun the standard +Buildroot make targets (in order to regenerate the complete image with +the new modifications) and then rerun the test. Modifying the sources +directly can speed up debugging compared to adding patch files, wiping the +output directoy, and starting the test again. + +--------------------- +$ ls output_folder/ +TestInitSystemBusyboxRw/ +TestInitSystemBusyboxRw-build.log +TestInitSystemBusyboxRw-run.log +--------------------- + +* The source file used to implement this example test is found under ++support/testing/tests/init/test_busybox.py+. This file outlines the +minimal defconfig that creates the build, QEMU configuration to launch +the built images and the test case assertions. + +To test an existing or new test case within Gitlab CI, there is a method of +invoking a specific test by creating a Buildroot fork in Gitlab under your +account. This can be handy when adding/changing a run-time test or fixing a +bug on a use case tested by a run-time test case. + + +In the examples below, the <name> component of the branch name is a unique +string you choose to identify this specific job being created. + +* to trigger all run-test test case jobs: + +--------------------- + $ git push gitlab HEAD:<name>-runtime-tests +--------------------- + +* to trigger one test case job, a specific branch naming string is used that +includes the full test case name. + +--------------------- + $ git push gitlab HEAD:<name>-<test case name> +--------------------- + +==== Creating a test case + +The best way to get familiar with how to create a test case is to look at a +few of the basic file system +support/testing/tests/fs/+ and init ++support/testing/tests/init/+ test scripts. Those tests give good examples +of a basic build and build with run type of tests. There are other more +advanced cases that use things like nested +br2-external+ folders to provide +skeletons and additional packages. + +The test cases by default use a br-arm-full-* uClibc-ng toolchain and the +prebuild kernel for a armv5/7 cpu. It is recommended to use the default +defconfig test configuration except when Glibc/musl or a newer kernel are +necessary. By using the default it saves build time and the test would +automatically inherit a kernel/std library upgrade when the default is +updated. + +The basic test case definition involves + +* Creation of a new test file +* Defining a unique test class +* Determining if the default defconfig plus test options can be used +* Implementing a +def test_run(self):+ function to optionally startup the +emulator and provide test case conditions. + +Beyond creating the test script, there are a couple of additional steps that +should be taken once you have your initial test case script. The first is +to add yourself to the +DEVELOPERS+ file to be the maintainer of that test +case. The second is to update the Gitlab CI yml by executing ++make .gitlab-ci.yml+.Signature · 6 lines
2.18.0 _______________________________________________ buildroot mailing list buildroot at busybox.net http://lists.busybox.net/mailman/listinfo/buildroot
Signature · 6 lines
.-----------------.--------------------.------------------.--------------------. | Yann E. MORIN | Real-Time Embedded | /"\ ASCII RIBBON | Erics' conspiracy: | | +33 662 376 056 | Software Designer | \ / CAMPAIGN | ___ | | +33 561 099 427 `------------.-------: X AGAINST | \e/ There is no | | http://ymorin.is-a-geek.org/ | _/*\_ | / \ HTML MAIL | v conspiracy. | '------------------------------^-------^------------------^--------------------'