Discussion: Package testing
Hi.
Many source packages include their own package test suites. We are looking at running these tests on target in a structured way, and I would like a discussion about how to best fit this into the Yocto framework.
The actions that need to be performed for a package test are roughly:
1) Build the test suite
2) Make the test suite appear on target
3) Run the test suite
4) Parse the results
Each action can be done in several ways, and there are different considerations for each solution.
1) Build the test suite
-----------------------
Many package tests are simply bash scripts that run the packaged binaries in various ways, but often the test suites also include binary tools that are not part of the normal package build. These tools have to be built for package testing to work.
Additionally, many packages build and run the test in a single command, such as "make check", which is obviously unsuitable for cross-compiled packages.
We can solve this in different ways:
a) Run the test build+run jobs on target. This avoids the need to modify packages, but building code on target can get quite expensive in terms of disk space. This in turn means many tests would require a harddisk or network disk to run.
b) Patch the makefiles to split test building and test running. Patching makefiles mean we get an additional maintenance and/or upstreaming burden, but we should be able to do this in ways that are acceptable to upstream. This is our suggestion.
2) Make the test suite appear on target
---------------------------------------
The test suite and utilities obviously have to be executable by the target system in order to run. There are a few options for this:
a) Copy all test files to the target at test run time, from the build dir, using whatever means available (scp, ftp etc). This limits testing to targets/images with easy automatic file transfer abilities installed.
b) NFS mount the build dir to access the full work dir and hence the test code. this limits testing to targets (and images) with network+nfs support. Plus it blends the build env and runtime env in an unpleasant way.
c) Create ${PN}-ptest packages (in the model of -dev and -dbg) that contain all test files and add those to the image and hence the rootfs. This is our suggestion.
3) Run the test suite
---------------------
Depending on how the test files are presented to the target, the way we run them can take different shapes:
a) (scp) A top-level run-all-tests.sh runs on the build host, copies all test files from build dir to target, logs in and runs each test.
b) (nfs) run-all-tests.sh is executed in the nfs-mounted build dir on target and build-runs each test in its work dir.
c) (-ptest) Install all test files to /opt/ptest/${PN}-${PV} (for example). Make a package "ptest-runner" that has a script /opt/ptest/run-all-tests to iterate over all installed tests and run them. This is our suggestion.
4) Parse the test results
-------------------------
Just running the tests doesn't give us much. We have to be able to look at the results and make them meaningful to us on a system-global level. Packages present their test results in very different ways, and we need to convert that to a generic format:
a) Patch each package test to produce a generic ptest output format. This is likely difficult to get accepted upstream.
b) Patch the test code minimally and instead use a simple per-packet translate script that converts test suite output from the package-specific format to a generic ptest format. This is our suggestion.
Opinions?
Note that this mail is about the test suites for/in/by specific packages. Standalone test suites such as LTP are a slightly different topic, since they are separate packages (${BPN} == ${PN}) rather than package companion suites.
Signature · 1 line
Björn
Bjorn, these are some interesting ideas.
Some feedback below.
This is from the perspective of asomeone who develops for embedded systems boards
2012/6/14 Björn Stenberg <[email protected]>
Quoted 27 lines
Hi.
Many source packages include their own package test suites. We are looking at running these tests on target in a structured way, and I would like a discussion about how to best fit this into the Yocto framework.
The actions that need to be performed for a package test are roughly:
1) Build the test suite
2) Make the test suite appear on target
3) Run the test suite
4) Parse the resultsEach action can be done in several ways, and there are different considerations for each solution.
1) Build the test suite
-----------------------
Many package tests are simply bash scripts that run the packaged binaries in various ways, but often the test suites also include binary tools that are not part of the normal package build. These tools have to be built for package testing to work.Additionally, many packages build and run the test in a single command, such as "make check", which is obviously unsuitable for cross-compiled packages.
Quoted 8 lines
We can solve this in different ways:
a) Run the test build+run jobs on target. This avoids the need to modify packages, but building code on target can get quite expensive in terms of disk space. This in turn means many tests would require a harddisk or network disk to run.
Not only storage space can be an issue. Some targets also do have limited
RAM and are low on CPU power.
Quoted 5 lines
b) Patch the makefiles to split test building and test running. Patching makefiles mean we get an additional maintenance and/or upstreaming burden, but we should be able to do this in ways that are acceptable to upstream. This is our suggestion.
This seems the most plexible approach. I'd say this could generate additional -test packages
Quoted 11 lines
2) Make the test suite appear on target
---------------------------------------
The test suite and utilities obviously have to be executable by the target system in order to run. There are a few options for this:a) Copy all test files to the target at test run time, from the build dir, using whatever means available (scp, ftp etc). This limits testing to targets/images with easy automatic file transfer abilities installed.
It also somewhat couples the building and the copying (or an additional command or step would be needed.
Quoted 9 lines
b) NFS mount the build dir to access the full work dir and hence the test code. this limits testing to targets (and images) with network+nfs support. Plus it blends the build env and runtime env in an unpleasant way.
c) Create ${PN}-ptest packages (in the model of -dev and -dbg) that contain all test files and add those to the image and hence the rootfs. This is our suggestion.
Agree, then one can decide whether to add them to an image, use a package feed to install things or whatever means.
In our case some of our systems do not have network support
Quoted 10 lines
3) Run the test suite
---------------------
Depending on how the test files are presented to the target, the way we run them can take different shapes:a) (scp) A top-level
run-all-tests.shruns on the build host, copies all test files from build dir to target, logs in and runs each test.
Quoted 8 lines
b) (nfs)
run-all-tests.shis executed in the nfs-mounted build dir on target and build-runs each test in its work dir.c) (-ptest) Install all test files to
/opt/ptest/${PN}-${PV}(for example). Make a package "ptest-runner" that has a script/opt/ptest/run-all-teststo iterate over all installed tests and run them. This is our suggestion.
Seems good to me. a run-all-tests.sh script is somewhat problematic as different systems have different packages installed so different tests. E.g. t is not too interesting to run C++ tests if your target system does not have/need C++
That makes creating a run-all-tests.sh script somewhat nastier (of course it could be conditional e.g. test -f ./testA && ./testA)
Quoted 9 lines
4) Parse the test results
-------------------------
Just running the tests doesn't give us much. We have to be able to look at the results and make them meaningful to us on a system-global level. Packages present their test results in very different ways, and we need to convert that to a generic format:
Quoted 7 lines
a) Patch each package test to produce a generic ptest output format. This is likely difficult to get accepted upstream.
b) Patch the test code minimally and instead use a simple per-packet translate script that converts test suite output from the package-specific format to a generic ptest format. This is our suggestion.
I suspect most tests would indicate success/failure by their return code. Maybe the test-runner could use the return code. Optionally the recipe could add a way to specify how to measure success (e.g by means of specifying a TEST_SUCCESS variable in the recipe that contains a command or shell script that when executed will return true otherwise false.
I think testers are mainly interested in success (all tests give ok is good, if a test fails, it would be fine with me if I would have to go to a package specific place to find log files etc (or it could be written to a common log file)
Quoted 4 lines
Opinions?
Good proposal!
The only specific point I can think of is cleaning up after a test is run. If you are low on resources like storage space and a test creates some larger outputs and does not delete them, the tests might run out of storage. So (maybe as an option) a test should contain (or be supplemented with) code to remove all its output (maybe only on success).
Hope this helps.
Frans
Quoted 13 lines
Note that this mail is about the test suites for/in/by specific packages. Standalone test suites such as LTP are a slightly different topic, since they are separate packages (${BPN} == ${PN}) rather than package companion suites.
--
Björn
_______________________________________________
yocto mailing list
[email protected]
https://lists.yoctoproject.org/listinfo/yocto
Quoted 2 lines · Frans Meulenbroeks
a
run-all-tests.shscript is somewhat problematic as different systems have different packages installed so different tests.
The idea is that the top level run-all script looks in the /opt/ptest directory and runs all package tests that are installed there. It has no built-in knowledge of any tests.
Quoted 4 lines
I think testers are mainly interested in success (all tests give ok is good, if a test fails, it would be fine with me if I would have to go to a package specific place to find log files etc (or it could be written to a common log file)
While failing tests are the most interesting during development, keeping track of both failed and successful test cases is important for regression tracking and test development over time. Therefore we want to produce an output that clearly identifies each test case ran and its result.
Quoted 1 line
The only specific point I can think of is cleaning up after a test is run.
Good point, I forgot to mention that. It is the responsibility of the per-package "run-ptest" script to clean up after the test.
Signature · 1 line
Björn
Quoted 33 lines · Björn Stenberg
Many source packages include their own package test suites. We are looking at running these tests on target in a structured way, and I would like a discussion about how to best fit this into the Yocto framework.
The actions that need to be performed for a package test are roughly:
1) Build the test suite
2) Make the test suite appear on target
3) Run the test suite
4) Parse the resultsEach action can be done in several ways, and there are different considerations for each solution.
1) Build the test suite
-----------------------
Many package tests are simply bash scripts that run the packaged binaries in various ways, but often the test suites also include binary tools that are not part of the normal package build. These tools have to be built for package testing to work.Additionally, many packages build and run the test in a single command, such as "make check", which is obviously unsuitable for cross-compiled packages.
We can solve this in different ways:
a) Run the test build+run jobs on target. This avoids the need to modify packages, but building code on target can get quite expensive in terms of disk space. This in turn means many tests would require a harddisk or network disk to run.
Quoted 4 lines
b) Patch the makefiles to split test building and test running. Patching makefiles mean we get an additional maintenance and/or upstreaming burden, but we should be able to do this in ways that are acceptable to upstream. This is our suggestion.
There are some good questions here :)
I think in general, we should be aiming for b) since that falls well into some of the ideas below. Upstreams should be amenable to splitting these into two targets (assuming "make check" just runs one after the other) so we shouldn't have to carry patches in most cases long term.
Quoted 10 lines
2) Make the test suite appear on target
---------------------------------------
The test suite and utilities obviously have to be executable by the target system in order to run. There are a few options for this:a) Copy all test files to the target at test run time, from the build dir, using whatever means available (scp, ftp etc). This limits testing to targets/images with easy automatic file transfer abilities installed.
Quoted 8 lines
b) NFS mount the build dir to access the full work dir and hence the test code. this limits testing to targets (and images) with network +nfs support. Plus it blends the build env and runtime env in an unpleasant way.
c) Create ${PN}-ptest packages (in the model of -dev and -dbg) that contain all test files and add those to the image and hence the rootfs. This is our suggestion.
I like the idea of test packages.
Quoted 9 lines
3) Run the test suite
---------------------
Depending on how the test files are presented to the target, the way we run them can take different shapes:a) (scp) A top-level
run-all-tests.shruns on the build host, copies all test files from build dir to target, logs in and runs each test.
Quoted 7 lines
b) (nfs)
run-all-tests.shis executed in the nfs-mounted build dir on target and build-runs each test in its work dir.c) (-ptest) Install all test files to
/opt/ptest/${PN}-${PV}(for example). Make a package "ptest-runner" that has a script/opt/ptest/run-all-teststo iterate over all installed tests and run them. This is our suggestion.
Can we suitably extract the tests out of the source code to be able to do this? I'm worried a little about the maintenance burden but I do like option c).
Quoted 10 lines
4) Parse the test results
-------------------------
Just running the tests doesn't give us much. We have to be able to look at the results and make them meaningful to us on a system-global level. Packages present their test results in very different ways, and we need to convert that to a generic format:a) Patch each package test to produce a generic ptest output format. This is likely difficult to get accepted upstream.
Quoted 7 lines
b) Patch the test code minimally and instead use a simple per-packet translate script that converts test suite output from the package-specific format to a generic ptest format. This is our suggestion.
Opinions?
I think the approach described is along the right lines. We'll probably have to try it and see how it goes. Over time, if upstreams see value in standard formats and so on, we may be able to effect some change but right now, we need to start somewhere!
I firmly believe working on these areas will add a lot of value to the project and attract many more users, as well as help Linux in general so its something we should work on.
Cheers,
Richard
Quoted 34 lines · Richard Purdie
Quoted 33 lines · Björn Stenberg
Many source packages include their own package test suites. We are looking at running these tests on target in a structured way, and I would like a discussion about how to best fit this into the Yocto framework.
The actions that need to be performed for a package test are roughly:
1) Build the test suite
2) Make the test suite appear on target
3) Run the test suite
4) Parse the resultsEach action can be done in several ways, and there are different considerations for each solution.
1) Build the test suite
-----------------------
Many package tests are simply bash scripts that run the packaged binaries in various ways, but often the test suites also include binary tools that are not part of the normal package build. These tools have to be built for package testing to work.Additionally, many packages build and run the test in a single command, such as "make check", which is obviously unsuitable for cross-compiled packages.
We can solve this in different ways:
a) Run the test build+run jobs on target. This avoids the need to modify packages, but building code on target can get quite expensive in terms of disk space. This in turn means many tests would require a harddisk or network disk to run.
Quoted 6 lines
Quoted 4 lines
b) Patch the makefiles to split test building and test running. Patching makefiles mean we get an additional maintenance and/or upstreaming burden, but we should be able to do this in ways that are acceptable to upstream. This is our suggestion.
There are some good questions here :)
Yes there are!
Quoted 5 lines
I think in general, we should be aiming for b) since that falls well into some of the ideas below. Upstreams should be amenable to splitting these into two targets (assuming "make check" just runs one after the other) so we shouldn't have to carry patches in most cases long term.
I think you both might be missing some cases here. The problem is that 'make check' does not have a standardized meaning.
From the top of my head, I can think of these different variants:
1) Function test level with testing run on target, package cross-compiled
2) Function test level with testing run on host, package cross-compiled
3) Function test level with testing run on host, package compiled for host
4) Unit test level with testing run on target, package cross-compiled
5) Unit test level with testing run on host, package compiled for host
I think that we need to be able to separate the different uses of 'make check' to be able to support testing nicely.
Quoted 15 lines
Quoted 15 lines
2) Make the test suite appear on target
---------------------------------------
The test suite and utilities obviously have to be executable by the target system in order to run. There are a few options for this:a) Copy all test files to the target at test run time, from the build dir, using whatever means available (scp, ftp etc). This limits testing to targets/images with easy automatic file transfer abilities installed.
b) NFS mount the build dir to access the full work dir and hence the test code. this limits testing to targets (and images) with network +nfs support. Plus it blends the build env and runtime env in an unpleasant way.
Quoted 5 lines
Quoted 3 lines
c) Create ${PN}-ptest packages (in the model of -dev and -dbg) that contain all test files and add those to the image and hence the rootfs. This is our suggestion.
I like the idea of test packages.
Would be very nice for functional tests, but it will not be so simple for unit testing. For unit tests I belive you need some build target too, which leads back to the comment above.
It might be a lot of autoconf hacking to get this working, don't know if there are any widely accepted standard targets for it.
/Regards
Fredrik Hugosson
Quoted 2 lines · Fredrik Hugosson
I think you both might be missing some cases here. The problem is that 'make check' does not have a standardized meaning.
Even if there was a standardized meaning to "make check", there are lots of packages that use completely differents systems for their building and testing. Hence there is no generic solution, we have to adapt each individual package manually.
Quoted 5 lines
1) Function test level with testing run on target, package cross-compiled
2) Function test level with testing run on host, package cross-compiled
3) Function test level with testing run on host, package compiled for host
4) Unit test level with testing run on target, package cross-compiled
5) Unit test level with testing run on host, package compiled for host
Our proposal is focused on testing the target binaries on the target, in an attempt to limit complexity. Testing -native and -nativesdk packages is therefore outside this scope.
Quoted 3 lines
Would be very nice for functional tests, but it will not be so simple for unit testing. For unit tests I belive you need some build target too, which leads back to the comment above.
Yes, this proposal only works for tests that run on a single machine. But from what I have seen so far, that covers the vast majority of included package tests. Do you have examples to the contrary?
There will certainly be room and need for other catergories of tests, which is one reason I've used the specific "ptest" moniker for this. This is not intended to be a covers-all-bases test solution. It's just one part.
Signature · 1 line
Björn
Quoted 8 lines · Richard Purdie
Quoted 4 lines
c) (-ptest) Install all test files to
/opt/ptest/${PN}-${PV}(for example). Make a package "ptest-runner" that has a script/opt/ptest/run-all-teststo iterate over all installed tests and run them. This is our suggestion.Can we suitably extract the tests out of the source code to be able to do this? I'm worried a little about the maintenance burden but I do like option c).
Typically the tests don't mix production and test code, since the purpose is to test the production code. Rather the tests are isolated in separate files that we relatively easily can build and install separately from the base package.
However, this is still early in the process. We have only processed a limited number of packages yet. We wanted to involve the community early to align our goals and directions.
Signature · 1 line
Björn
Quoted 1 line · Björn Stenberg
Typically the tests don't mix production and test code, since the purpose is to test the production code. Rather the tests are isolated in separate files that we relatively easily can build and install separately from the base package.
The problem with that is that you can only test the external interfaces, not the inner state (black box vs white box). i.e. DBus has test cases for each C file inside that C file, so it can exercise both the external API and internal state.
Ross
Quoted 4 lines · Ross
The problem with that is that you can only test the external interfaces, not the inner state (black box vs white box). i.e. DBus has test cases for each C file inside that C file, so it can exercise both the external API and internal state.
I haven't examined dbus in detail yet, but in principle I don't think it should be a problem. As long as the test binaries are compiled on the build host, in the normal work dir, we don't need to know or care what source code they are built from.
Signature · 1 line
Björn