regression testing
Hi. I was wondering if there has been any discussion of kernel regression testing. Wouldn't it be great if we didn't have to depend on human testers to verify every change didn't break something?
OK, I'll admit I haven't given this a lot of thought. What I'm wondering is whether the user-mode linux could help here (allow a way to simulate controlled activity).
Quoted 8 lines · On 22 Mar 2001 [email protected] wrote:
Hi. I was wondering if there has been any discussion of kernel regression testing. Wouldn't it be great if we didn't have to depend on human testers to verify every change didn't break something?
OK, I'll admit I haven't given this a lot of thought. What I'm wondering is whether the user-mode linux could help here (allow a way to simulate controlled activity).
-
Regression testing __is__ what happens when 10,000 testers independently try to break the software!
Canned so-called "regression-test" schemes will fail to test at least 90 percent of the code paths, while attempting to "test" 100 percent of the code!
Cheers,
Dick Johnson
Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
"Memory is like gasoline. You use it up when you are running. Of course you get it all back when you reboot..."; Actual explanation obtained from the Micro$oft help desk.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Richard" == Richard B Johnson <[email protected]> writes:
Richard> On 22 Mar 2001 [email protected] wrote:
>> Hi. I was wondering if there has been any discussion of kernel >> regression testing. Wouldn't it be great if we didn't have to depend >> on human testers to verify every change didn't break something? >>
>> OK, I'll admit I haven't given this a lot of thought. What I'm >> wondering is whether the user-mode linux could help here (allow a way >> to simulate controlled activity).
>> -
The problem I see, based on reading this list, is that the testing is not so thorough as you say. First of all, do we really know how many people test the pre-relase kernels? How about which features they actually test?
In principle, we could find out how much test coverage there is.
The problem with an army of testers, is that after an individual has gone through the build/install/screw-around-with-it process a few hundred times, it starts to get a bit repetitive. They start to get sloppy and haphazard regarding what they test or whether they even try out xxx-pre<put_big_number_here> at all. If we could automate at least some basic testing that would be a big win.
Quoted 2 lines
Regression testing
__is__what happens when 10,000 testers independently try to break the software!
Nope. Thats stress testing and a limited amount of coverage testing.
Quoted 3 lines
Canned so-called "regression-test" schemes will fail to test at least 90 percent of the code paths, while attempting to "test" 100 percent of the code!
Then they are not well designed. Tools like gprof and the kernel profiling will let you measure code path coverage of a test series
Quoted 12 lines · Alan Cox
Quoted 2 lines
Regression testing
__is__what happens when 10,000 testers independently try to break the software!Nope. Thats stress testing and a limited amount of coverage testing.
Quoted 3 lines
Canned so-called "regression-test" schemes will fail to test at least 90 percent of the code paths, while attempting to "test" 100 percent of the code!
Then they are not well designed. Tools like gprof and the kernel profiling will let you measure code path coverage of a test series
Alan's back!
Cheers,
Dick Johnson
Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
"Memory is like gasoline. You use it up when you are running. Of course you get it all back when you reboot..."; Actual explanation obtained from the Micro$oft help desk.
Quoted 4 lines · [email protected] wrote:
Hi. I was wondering if there has been any discussion of kernel regression testing. Wouldn't it be great if we didn't have to depend on human testers to verify every change didn't break something?
IMHO, much of the strength of Linux is the very large, extremely diverse population of folks using it, testing it, beating on the latest release, etc.
However, a lab dedicated to testing the linux kernel, properly funded, staffed, and containing the most common hardware and software would be a good idea. Does anyone have any idea how this could be accomplished? Who could do it? IBM? What would it cost to setup a reasonable lab? My guess would be dozens of machines of various architectures, a staff of at least 10, several thousand square feet of space, and a good budget.... Any takers?
Much of the kernel COULD be tested such as file systems, network stack, SMP, compile options on various platforms, etc. More obscure hardware, some older hardware, etc., would be out of scope for such an effort.
Cheers,
Signature · 5 lines
W. Wade, Hampton <[email protected]> If Microsoft Built Cars: Every time they repainted the lines on the road, you'd have to buy a new car. Occasionally your car would just die for no reason, and you'd have to restart it, but you'd just accept this.
Quoted 7 lines · Wade Hampton
Quoted 3 lines · [email protected] wrote:
Hi. I was wondering if there has been any discussion of kernel regression testing. Wouldn't it be great if we didn't have to depend on human testers to verify every change didn't break something?
IMHO, much of the strength of Linux is the very large, extremely diverse population of folks using it, testing it, beating on the latest release, etc.
The Linux community definately provides the configuration testing that could never be done in a lab setting. At least one that most companies could afford.
Quoted 8 lines
However, a lab dedicated to testing the linux kernel, properly funded, staffed, and containing the most common hardware and software would be a good idea. Does anyone have any idea how this could be accomplished? Who could do it? IBM? What would it cost to setup a reasonable lab? My guess would be dozens of machines of various architectures, a staff of at least 10, several thousand square feet of space, and a good budget.... Any takers?
SGI is working on regression testing for Linux. We have released some of our tests and utilities under the Linux Test Project. IMHO, a few hundred tests aren't enough. I need to make another big push with the tests I've ported over the last few months.
I believe there is some work in the Open Source Development Lab that some IBMers could comment on. I don't know if there is a web site detailing their efforts yet.
Signature · 3 lines
Nate Straz [email protected] sgi, inc http://www.sgi.com/ Linux Test Project http://oss.sgi.com/projects/ltp/
Quoted 10 lines · Nathan Straz
Quoted 8 lines · Wade Hampton
However, a lab dedicated to testing the linux kernel, properly funded, staffed, and containing the most common hardware and software would be a good idea. Does anyone have any idea how this could be accomplished? Who could do it? IBM? What would it cost to setup a reasonable lab? My guess would be dozens of machines of various architectures, a staff of at least 10, several thousand square feet of space, and a good budget.... Any takers?
Quoted 8 lines
SGI is working on regression testing for Linux. We have released some of our tests and utilities under the Linux Test Project. IMHO, a few hundred tests aren't enough. I need to make another big push with the tests I've ported over the last few months.
I believe there is some work in the Open Source Development Lab that some IBMers could comment on. I don't know if there is a web site detailing their efforts yet.
www.osdlab.org is the closest you will get right now.
Since we have only recently opened we have been working to get the machines configured and ready for beating. We have only recently started opening up the larger servers to developers and testers.
Any regression/scalability development and testing people want to do should be workable. Anyone interested in coordinating development or testing at our lab can contact me for info.
-Nathan Dabney
Open Source Development Lab
Quoted 3 lines · On 22 Mar 2001 [email protected] wrote:
Hi. I was wondering if there has been any discussion of kernel regression testing. Wouldn't it be great if we didn't have to depend on human testers to verify every change didn't break something?
This is definately a great idea. A relatively easy start should be testing things like:
- automated crashme runs
- automated heavy stress testing
- automated compilation of the kernel with random config options (done by Arjan v/d Ven?)
- automated testing of certain kernel behaviour (didn't SGI have a project to look at this? could they use help?)
- ... ?
Note that some of these testing options are almost trivial to set up and could be run by any sysadmin with a few spare/test machines. It would be great if testing like this could be done on the Linux kernel.
If there isn't already some place to coordinate this testing, I'll be able to host everything needed on NL.linux.org.
regards,
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/
Quoted 1 line
- automated heavy stress testing
This would be an interesting one to me, from a benchmarking POV. I'd like to know what my hardware can really do, for one thing - it's all very well saying this box can do X Whetstones and has a 100Mbit NIC, but it's a much more solid thing to be able to say "my box handled the official Foobar stress-test in Y hours, handling Z widgets per second".
The lmbench statistics are nice, but most of those numbers mean absolutely nothing to all but übergeeks (what they say to my partially-trained ears is that my gateway box sucks, but then again it's five-year-old Intel hardware so that could probably have been predicted). Are there already stress-testing kits for user-level processes I could play with?
What would be *really* interesting would be to make such tests as portable as practical, so that at least some of them can be run on "rival platforms" (such as *cough* Redmondware) to see if they stand up to the challenge as well.
However, if the focus is on *kernel* stress-testing, portability might be a little difficult to arrange for many tests. The basics could probably be written portably or ported easily, though - things that lmbench already (briefly) benchmarks, for example.
--------------------------------------------------------------
from: Jonathan "Chromatix" Morton mail: [email protected] (not for attachments) big-mail: [email protected] uni-mail: [email protected]
The key to knowledge is not to rely on people to teach you it.
Get VNC Server for Macintosh from http://www.chromatix.uklinux.net/vnc/
-----BEGIN GEEK CODE BLOCK-----
Version 3.12
GCS$/E/S dpu(!) s:- a20 C+++ UL++ P L+++ E W+ N- o? K? w--- O-- M++$ V? PS PE- Y+ PGP++ t- 5- X- R !tv b++ DI+++ D G e+ h+ r++ y+(*) -----END GEEK CODE BLOCK-----
} On 22 Mar 2001 [email protected] wrote:
}
} > Hi. I was wondering if there has been any discussion of kernel } > regression testing. Wouldn't it be great if we didn't have to depend } > on human testers to verify every change didn't break something? } >
} > OK, I'll admit I haven't given this a lot of thought. What I'm } > wondering is whether the user-mode linux could help here (allow a way } > to simulate controlled activity).
} > -
}
} Regression testing __is__ what happens when 10,000 testers independently } try to break the software!
No, in fact that is not a regression test.
} Canned so-called "regression-test" schemes will fail to test at least } 90 percent of the code paths, while attempting to "test" 100 percent } of the code!
A canned set of regression tests would actually do what they're supposed to
- prevent the kernel from regressing. If you fix a bug - write a test for that bug and keep running it. Something we follow for RTLinux that has helped us immensely.
We have a start for PPC. It has the title "Regression Tester" but is actually a "compiles and boots tester". The aim is a automated regression test.
Take a look at http://altus.drgw.net/
It pulls directly from our BitKeeper archive every time we push a change and goes through the build targeted for a number of platforms.
} - automated compilation of the kernel with random config
} options (done by Arjan v/d Ven?)
} - automated testing of certain kernel behaviour (didn't
} SGI have a project to look at this? could they use help?)
} - ... ?
Quoted 3 lines · [email protected] writes:
Hi. I was wondering if there has been any discussion of kernel regression testing. Wouldn't it be great if we didn't have to depend on human testers to verify every change didn't break something?
There is a some truth to this. However for kernel development there is one thing to keep in mind: most bugs are in the drivers (buggy hardware with a software workaround counts as a driver bug). Having an army of human testers with weird machines with all kinds of drivers is the only economical way of doing driver regression testing.
Quoted 3 lines
OK, I'll admit I haven't given this a lot of thought. What I'm wondering is whether the user-mode linux could help here (allow a way to simulate controlled activity).
The most devastating bugs are in the core kernel code, and a regression test for that code is more likely. Yes user-mode linux could help here (you could stress test the core kernel without worry that when it crashes your machine will crash as well).
Additionally a good test suite that give the kernel a good going over in a manner that exercises standard kinds of hardware could also help.
Unless you are using profiling to verify you have covered all paths in the code there will always be question such as does that code path work in practice.
The other thing that can help a lot looks like the augmented static analysis of the kernel. Once that is generally available we should be able to verify at compile time that a lot of little rules are always obeyed. It will never get the really hard things but the compiler pointing out that there is an sti on all paths after a cli could improve things significantly (especially with the drivers).
Eric
Quoted 7 lines · Jonathan Morton
Quoted 1 line
- automated heavy stress testing
This would be an interesting one to me, from a benchmarking POV. I'd like to know what my hardware can really do, for one thing - it's all very well saying this box can do X Whetstones and has a 100Mbit NIC, but it's a much more solid thing to be able to say "my box handled the official Foobar stress-test in Y hours, handling Z widgets per second".
Which would tell you exactly that, nothing more and nothing less. No idea how many users it would support reading mail via IMAP, if playing Quake on it is doable, or whether it is a good idea to use it as a firewall or development machine. Computer systems are very complex, their uses much more so (and extremely varied to boot); to believe some number (or even a small set of numbers) will tell you everything there is to know about a particular box is just an illusion. And a box rarely stands in isolation.
Signature · 4 lines
Dr. Horst H. von Brand mailto:[email protected] Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Quoted 10 lines · Horst von Brand
Quoted 7 lines · Jonathan Morton
Quoted 1 line
- automated heavy stress testing
This would be an interesting one to me, from a benchmarking POV. I'd like to know what my hardware can really do, for one thing - it's all very well saying this box can do X Whetstones and has a 100Mbit NIC, but it's a much more solid thing to be able to say "my box handled the official Foobar stress-test in Y hours, handling Z widgets per second".
Which would tell you exactly that, nothing more and nothing less.
But if we get 500 people to test _different_ things on a regular basis, we might be able to get more meaningful results.
regards,
Rik
--
Virtual memory is like a game you can't win; However, without VM there's truly nothing to lose...
http://www.surriel.com/
Quoted 3 lines · Eric W. Biederman
Yes user-mode linux
could help here (you could stress test the core kernel without worry that when it crashes your machine will crash as well).
A similar approach can be used for very detailed tests of specific subsystems. E.g. that's what we've started doing, kind of as a by-product of some other work, with tcsim, which runs most of the traffic control subsystem in user space. (ftp://icaftp.epfl.ch/pub/linux/tcng/)
The advantage over using UML is that we have a much simpler environment, allowing for more straightforward integration, and we can exercise tight control over infrastructure parts like timers or network events. Once UML is part of the mainstream kernel, it would be interesting to try to obtain the same functionality with UML too.
Some added goodies include the ability to run kernel code with malloc debuggers like Electric Fence, which has already helped to find a few real bugs. (Does EFence work with UML ?)
- Werner
Signature · 3 lines
_________________________________________________________________________ / Werner Almesberger, ICA, EPFL, CH [email protected] / /_IN_N_032__Tel_+41_21_693_6621__Fax_+41_21_693_6610_____________________/