the Linux kernel, testsuites, and maybe *you*
is there any sort of standard for testing and integration into mainline ? in the Blackfin world, we've been developing little external kernel modules and adding them to our own testsuite, but often times these things are not Blackfin specific. case in point, we're integrating a string testsuite to make sure all of the fun str* and mem* functions are sane and operate as they expected, but rather than having just Blackfin benefit here, i'd like to see this pushed upstream ...
i'm fully aware of LTP (as i work on it), but i feel that serves a great purpose for exercising the API/ABI exposed to userspace either directly through the kernel or indirectly through the system libc ... it isnt a very good tool for testing kernel internals, especially as the kernel changes and evolves.
is there a framework already in place i'm not aware of ? should there be ? should this all live in LTP ? i wouldnt mind an option under kernel hacking "Enable testsuites" ... as far as i can tell, the testing process is really based extensively on feedback from people, nothing really automated.
-mike
On Fri 31 Aug 2007 17:22, Mike Frysinger pondered:
Quoted 8 lines
is there any sort of standard for testing and integration into mainline ? in the Blackfin world, we've been developing little external kernel modules and adding them to our own testsuite, but often times these things are not Blackfin specific. case in point, we're integrating a string testsuite to make sure all of the fun str* and mem* functions are sane and operate as they expected, but rather than having just Blackfin benefit here, i'd like to see this pushed upstream ...
I know there have been some discussions at past OLS about some testing that Martin was working on.
http://www.linuxsymposium.org/2006/view_abstract.php?content_key=13
But I think this was more functional tests, less unit tests that what you are talking/asking about.
http://test.kernel.org/functional/index.html
Also - If I remember - most of the existing tests were for a self hosted environment - and might not be well suited to embedded (like Blackfin) which requires cross compile, and pretty thin runtime environment (uClibc + busybox's msh as shell).
Martin?
-Robin
Quoted 18 lines · Robin Getz
On Fri 31 Aug 2007 17:22, Mike Frysinger pondered:
Quoted 8 lines
is there any sort of standard for testing and integration into mainline ? in the Blackfin world, we've been developing little external kernel modules and adding them to our own testsuite, but often times these things are not Blackfin specific. case in point, we're integrating a string testsuite to make sure all of the fun str* and mem* functions are sane and operate as they expected, but rather than having just Blackfin benefit here, i'd like to see this pushed upstream ...
I know there have been some discussions at past OLS about some testing that Martin was working on.
http://www.linuxsymposium.org/2006/view_abstract.php?content_key=13
But I think this was more functional tests, less unit tests that what you are talking/asking about.
Quoted 6 lines
http://test.kernel.org/functional/index.html
Also - If I remember - most of the existing tests were for a self hosted environment - and might not be well suited to embedded (like Blackfin) which requires cross compile, and pretty thin runtime environment (uClibc + busybox's msh as shell).
i flipped through the autotest code a bit but i dont think this can do what i'm looking for. it appears to be a mix of LTP provides and what we have for our Blackfin test infrastructure -- iow, all userspace stuff and no real way of exercising kernel internals.
-mike
Quoted 2 lines · Mike Frysinger
is there any sort of standard for testing and integration into mainline ?
Everybody does their own.
Quoted 7 lines
in the Blackfin world, we've been developing little external kernel modules and adding them to our own testsuite, but often times these things are not Blackfin specific. case in point, we're integrating a string testsuite to make sure all of the fun str* and mem* functions are sane and operate as they expected, but rather than having just Blackfin benefit here, i'd like to see this pushed upstream ...
I would like to see this too. I wrote a couple of unit tests during x86-64 development too and they would be handy to have in a central place.
The SUSE kernel also has a crasher module that exercises the allocators and is pretty useful to stress code in general.
Disadvantage is that test code tends to be hackish and ugly
and very specialized and will probably not pass standard review procedures.
Also the rt people seem to have pushed a couple of tests in already, but I always hated it that they're in the standard directories. RCU also has, in fact they added a "eat all my CPU in tests" CONFIG option. Just making that dependent on a CONFIG_UNIT_TESTS would be a good change in itself. And then there are the slab failure inducers of course.
BTW string functions are best tested in user space. That's a relatively bad example.
-Andi
Quoted 2 lines · Andi Kleen
BTW string functions are best tested in user space. That's a relatively bad example.
in theory, maybe ... in reality, i really dont think so
the string implementations are spread out over the kernel ... there's implementations in lib/, include/asm-*/, and arch/*/lib/ ... so any test code that would use these sources is going to be an ugly hack to make sure it grabs all the right pieces from all the right places.
then there's the issue of API ... compiling code that is designed to be in the kernel and nowhere else is going to require a mighty bit of crappy glue logic in order for it to compile properly in userspace
but let's cheat and just use the kernel build system to produce .o's for us ... then there's the issue of ABI. there is no guarantee the ABI the kernel uses internally is the same as the ABI userspace uses (and in fact, on some architectures it is simply not possible). so you're left with a solution that may work for some, but not for others.
or we put one common piece of C test code right into the kernel and be done. no worrying about the tested API/ABI/environment being different from the actual runtime the kernel itself has. -mike
Quoted 10 lines · Mike Frysinger
Quoted 2 lines · Andi Kleen
BTW string functions are best tested in user space. That's a relatively bad example.
in theory, maybe ... in reality, i really dont think so
the string implementations are spread out over the kernel ... there's implementations in
lib/,include/asm-*/, andarch/*/lib/... so any test code that would use these sources is going to be an ugly hack to make sure it grabs all the right pieces from all the right places.
string functions tend to be self contained.
The other issue to test some of them properly you need unmapped pages etc. That gets much easier to do in user space. There are some other issues.
-Andi
Quoted 5 lines · Andi Kleen
Quoted 4 lines · Mike Frysinger
Quoted 2 lines · Andi Kleen
BTW string functions are best tested in user space. That's a relatively bad example.
Quoted 8 lines
Quoted 6 lines
in theory, maybe ... in reality, i really dont think so
the string implementations are spread out over the kernel ... there's implementations in
lib/,include/asm-*/, andarch/*/lib/... so any test code that would use these sources is going to be an ugly hack to make sure it grabs all the right pieces from all the right places.string functions tend to be self contained.
"tend to be" does not lend itself to being cleanly tested ... and i imagine i'd get some pretty heavy resistance if i proposed re-organzing code just so that i could compile it outside of the kernel
there is still the ABI issue ... code written in kernel space in pure asm cannot always be compiled in userspace and work properly/the same
Quoted 3 lines
The other issue to test some of them properly you need unmapped pages etc. That gets much easier to do in user space. There are some other issues.
you mean testing boundary overflows ? can be handled with canaries rather than segfaults i imagine ...
-mike
Quoted 2 lines · Mike Frysinger
there is still the ABI issue ... code written in kernel space in pure asm cannot always be compiled in userspace and work properly/the same
Is that a blackfin weirdness?
Quoted 7 lines
Quoted 4 lines
The other issue to test some of them properly you need unmapped pages etc. That gets much easier to do in user space. There are some other issues.
you mean testing boundary overflows ? can be handled with canaries rather than segfaults i imagine ...
Not for reads, no.
-Andi
Quoted 5 lines · Andi Kleen
Quoted 2 lines · Mike Frysinger
there is still the ABI issue ... code written in kernel space in pure asm cannot always be compiled in userspace and work properly/the same
Is that a blackfin weirdness?
yes, Blackfin is weird in this respect due to being a no-mmu and wanting ELF, but i was being sufficiently general to bolster my claims and to leave the door open for any other port out there i'm not familiar with and any port that comes along
Quoted 8 lines
Quoted 6 lines
Quoted 3 lines
The other issue to test some of them properly you need unmapped pages etc. That gets much easier to do in user space. There are some other issues.
you mean testing boundary overflows ? can be handled with canaries rather than segfaults i imagine ...
Not for reads, no.
true, but generally the string functions arent searching for crashes in the boundary conditions ... they just happen to sometimes find them :)
-mike
Quoted 11 lines · Mike Frysinger
Quoted 5 lines · Andi Kleen
Quoted 2 lines · Mike Frysinger
there is still the ABI issue ... code written in kernel space in pure asm cannot always be compiled in userspace and work properly/the same
Is that a blackfin weirdness?
yes, Blackfin is weird in this respect due to being a no-mmu and wanting ELF, but i was being sufficiently general to bolster my claims and to leave the door open for any other port out there i'm not familiar with and any port that comes along
actually, there was another case i was thinking of ... it isnt uncommon for architectures to use compiler flags which affect the ABI and thus make userland not a 1-to-1 test example for the kernel ... consider the regparam stuff we use on i386
-mike
Quoted 3 lines · Andi Kleen
The other issue to test some of them properly you need unmapped pages etc. That gets much easier to do in user space. There are some other issues.
vmalloc, vmap, etc. always put a guard page after the allocation. So if you test string operations on vmalloc()'ed memory, you have unmapped pages in both ends.
I like the idea of a test suite for kernel internals, but I also think it would be best to run it in kernel mode since things like atomics, etc. may work differently in user space on some architectures.
Haavard
On Sat 1 Sep 2007 18:08, Andi Kleen pondered:
Quoted 6 lines
Quoted 2 lines · Mike Frysinger
is there any sort of standard for testing and integration into mainline ?
Everybody does their own.
That kind of stinks - and seems to be a potential duplication of effort all over the place.
Quoted 12 lines
Quoted 7 lines
in the Blackfin world, we've been developing little external kernel modules and adding them to our own testsuite, but often times these things are not Blackfin specific. case in point, we're integrating a string testsuite to make sure all of the fun str* and mem* functions are sane and operate as they expected, but rather than having just Blackfin benefit here, i'd like to see this pushed upstream ...
I would like to see this too. I wrote a couple of unit tests during x86-64 development too and they would be handy to have in a central place.
Quoted 6 lines
The SUSE kernel also has a crasher module that exercises the allocators and is pretty useful to stress code in general.
Disadvantage is that test code tends to be hackish and ugly and very specialized and will probably not pass standard review procedures.
Then this makes perfect sense to actually have it in the proper places, and have it go through the standard review process. I have run into problems recently, where ugly hackish tests seem never really catch actual faults. (Since they were written by the same person who wrote the driver - Design a fault in, code the fault, test the fault is there).
With many eyes - and all that stuff - hopefully we would catch things.
Quoted 6 lines
Also the rt people seem to have pushed a couple of tests in already, but I always hated it that they're in the standard directories. RCU also has, in fact they added a "eat all my CPU in tests" CONFIG option. Just making that dependent on a
CONFIG_UNIT_TESTSwould be a good change in itself. And then there are the slab failure inducers of course.
Putting them close to the standard directories might be OK - keeping the tests close to the thing that is being tested may help with maintainers keeping an eye on things that might be missed otherwise.
Maybe these types of tests should go into a separate directory - power on self tests (post) directories in the respective places? ./kernel/post and ./drivers/xxxx/post?
So Mike's proposed string tests would go into ./lib/post/strings.c
When things pass - you get a single indication (all tests for unit X passed) or a pass for each test?
-Robin
Quoted 15 lines · Mike Frysinger
is there any sort of standard for testing and integration into mainline ? in the Blackfin world, we've been developing little external kernel modules and adding them to our own testsuite, but often times these things are not Blackfin specific. case in point, we're integrating a string testsuite to make sure all of the fun str* and mem* functions are sane and operate as they expected, but rather than having just Blackfin benefit here, i'd like to see this pushed upstream ...
i'm fully aware of LTP (as i work on it), but i feel that serves a great purpose for exercising the API/ABI exposed to userspace either directly through the kernel or indirectly through the system libc ... it isnt a very good tool for testing kernel internals, especially as the kernel changes and evolves.
Quoted 6 lines
is there a framework already in place i'm not aware of ? should there be ? should this all live in LTP ? i wouldnt mind an option under kernel hacking "Enable testsuites" ... as far as i can tell, the testing process is really based extensively on feedback from people, nothing really automated.
If you want to test that stuff and run it on the current code in the kernel, how about a kernel module? You could "modprobe sanitytest" or something and report to syslog at module load time. And maybe have a parameter which does something drastic if something core is so hosed that filesystem damage is likely. Or just optional init code run by a kernel option, perhaps sanity testing after boot is not a great idea.
Signature · 3 lines
Bill Davidsen <[email protected]> "We have more to fear from the bungling of the incompetent than from the machinations of the wicked." - from Slashdot
Quoted 6 lines · Bill Davidsen
If you want to test that stuff and run it on the current code in the kernel, how about a kernel module? You could "modprobe sanitytest" or something and report to syslog at module load time. And maybe have a parameter which does something drastic if something core is so hosed that filesystem damage is likely. Or just optional init code run by a kernel option, perhaps sanity testing after boot is not a great idea.
i'm fully capable of doing my own testing right now; that isnt the point of this thread ... the point is for everyone to benefit, not just the Blackfin architecture
-mike
Quoted 10 lines · Mike Frysinger
Quoted 7 lines · Bill Davidsen
If you want to test that stuff and run it on the current code in the kernel, how about a kernel module? You could "modprobe sanitytest" or something and report to syslog at module load time. And maybe have a parameter which does something drastic if something core is so hosed that filesystem damage is likely. Or just optional init code run by a kernel option, perhaps sanity testing after boot is not a great idea.
Quoted 4 lines
i'm fully capable of doing my own testing right now; that isnt the point of this thread ... the point is for everyone to benefit, not just the Blackfin architecture
That's exactly the point I was making, if your tests and any others people feel are useful were in a module and/or could be run as init code only when wanted, everyone could benefit from the tests. Anything related to build sanity could be in one place, to guard against surprises like new versions of gcc, typos in patches, etc.
Signature · 3 lines
bill davidsen <[email protected]> CTO TMR Associates, Inc Doing interesting things with small computers since 1979