runltp-ng hackweek experiment
Hi!
I've tried to experiment with a proof of concept for new LTP test execution framework during Hackweek[1]. And I've ended up with something that is more or less usable and has features missing in runltp + ltp-pan script we have now.
It's in the proof of concept state at this point and polishing it would require more work, but it looks promising at this point.
The key desing points are:
* We use the ltp-wrapper[2] instead of ltp-pan, which runs oldlib and openposix testcases from a test() function from a newlib test.
This provides us test timeouts and killing whole test process tree, etc. basically for free.
* The ltp-runner[3] then does the runtest files parsing, openposix test discovery, test execution, writing test results, etc.
- The one intrusive change is that I've added code to both oldlib and newlib to support '-q' flag, so that we can query test for details i.e. if it's oldlib/newlib test, if it requires root, etc. This is would be also needed for parallel test execution, where the test would report if it can be executed in parallel and/or report what kind of system resources it monopolizes. The bottom line is that all testcases should understand the -q option and at least exit without writing any output, which is not the case at the moment.
- The code is split into several modules, the parsers prepare list of tests to be executed, then there is code to execute the tests and store logs, then the output writers can produce logs in several different formats. I've experimented a bit with a better html output[4][5] (table sorting is not implemented yet, but planned).
- The code is prepared for parallel test execution which is not implemented at the moment but basically boils down to marking down testcases that cannot be executed concurently. As I wrote I would like to keep this information inside the test code and export it via the -q options.
- We miss random test execution and execute tests for defined amount of time at this point, but other than that the basic features are there.
And as usuall any comments are welcomed ;-).
[1] https://hackweek.suse.com/
[2] https://github.com/metan-ucw/ltp/blob/master/tools/runltp-ng/bin/ltp-wrapper.c
[3] https://github.com/metan-ucw/ltp/blob/master/tools/runltp-ng/bin/ltp-runner.c
[4] http://metan.ucw.cz/outgoing/ltp-openposix.html
[5] http://metan.ucw.cz/outgoing/ltp-syscalls.html
Signature · 2 lines
Cyril Hrubis [email protected]
Hi Cyril,
Quoted 7 lines
I've tried to experiment with a proof of concept for new LTP test execution framework during Hackweek[1]. And I've ended up with something that is more or less usable and has features missing in runltp + ltp-pan script we have now.
It's in the proof of concept state at this point and polishing it would require more work, but it looks promising at this point.
It looks really interesting.
BTW while Open Posix Testcases (-o switch) are working, "normal" tests doesn't. Do I miss something?
$ /opt/ltp/bin/ltp-runner -S /opt/ltp/runtest/io
Overall results:
-------------------------------------
Passed 0 out of 0
And shouldn't there be a different check as otherwise it segfaults?
$ /opt/ltp/bin/ltp-runner -S /opt/ltp/runtest/mm Segmentation fault (core dumped)
runltp-ng/bin/runner_test_filter.c +1 -1
@@ -100,7 +100,7 @@ static void trie_insert(const char *str) str =trie_search (str, &node); -while (*str) { +while (str) {int key =char2key (*str);if (key ==-1 )
Quoted 5 lines
- The code is split into several modules, the parsers prepare list of tests to be executed, then there is code to execute the tests and store logs, then the output writers can produce logs in several different formats. I've experimented a bit with a better html output[4][5] (table sorting is not implemented yet, but planned).
Tables looks very nice. I'd be for more features like hover highlighting a and slightly different color for odd and even rows, but thats a detail (see
http://adminer.sourceforge.net/adminer.php?username=&db=a264133_6rgs7fa4 as example).
Kind regards,
Petr
Hi!
Quoted 6 lines
It looks really interesting.
BTW while Open Posix Testcases (-o switch) are working, "normal" tests doesn't. Do I miss something?
$ /opt/ltp/bin/ltp-runner -S /opt/ltp/runtest/io
The -S option sets a skipfile, runtest files are passed as positional parameters.
Quoted 10 lines
Overall results:
-------------------------------------Passed 0 out of 0
And shouldn't there be a different check as otherwise it segfaults?
$ /opt/ltp/bin/ltp-runner -S /opt/ltp/runtest/mm Segmentation fault (core dumped)
Hmm, looks like a bug.
runltp-ng/bin/runner_test_filter.c static void trie_insert(const char *str) · line 106
Quoted 11 lines
runltp-ng/bin/runner_test_filter.c +1
-1@@ -100,7 +100,7 @@ static void trie_insert(const char *str) str =trie_search (str, &node); -while (*str) { +while (str) {int key =char2key (*str);if (key ==-1 )
Does not look correct, we are looping over the string characters here inserting nodes into the tree. Here we likely segfault since the trie_search() encountered unexpected character and returned NULL since I deliberately allow only printable ascii characters in the test id. So we should do something as:
str = trie_search(str, &node);
if (!str) {
WARN("Invalid character found in '%s'", str);
return;
}Quoted 9 lines
Quoted 5 lines
- The code is split into several modules, the parsers prepare list of tests to be executed, then there is code to execute the tests and store logs, then the output writers can produce logs in several different formats. I've experimented a bit with a better html output[4][5] (table sorting is not implemented yet, but planned).
Tables looks very nice. I'd be for more features like hover highlighting a and slightly different color for odd and even rows, but thats a detail (see
http://adminer.sourceforge.net/adminer.php?username=&db=a264133_6rgs7fa4 as example).
I have something like that in mind as well, the table was done in a half of a day and I've spend most of the time looking into javascript documentation...
But I've managed to write a table sorting function yesterday evenning, I will push that code into my github repo soon enough.
Signature · 2 lines
Cyril Hrubis [email protected]
Hi,
Quoted 7 lines
Quoted 3 lines
Tables looks very nice. I'd be for more features like hover highlighting a and slightly different color for odd and even rows, but thats a detail (see
http://adminer.sourceforge.net/adminer.php?username=&db=a264133_6rgs7fa4 as example).I have something like that in mind as well, the table was done in a half of a day and I've spend most of the time looking into javascript documentation...
Sure, web UI development is a bit different from low level C code :-). This reporting feature is really nice.
Quoted 2 lines
But I've managed to write a table sorting function yesterday evenning, I will push that code into my github repo soon enough.
Great.
Kind regards,
Petr