First step in formalizing the test library
cover First step in formalizing the test library
This is partly motivated by the issue #462 where we ended up with a loose corer cases that are not well defined.
So as a first step this patches:
- Do fix old problem with TBROK propagation
- Start a library description that hopefully captures so-far unwritten desing decisions and rules
Cyril Hrubis (2):
lib: tst_test.c: Add TBROK counter.
lib: Add test library design document
lib/README.md | 130 ++++++++++++++++++++++++++++++++++++ lib/newlib_tests/.gitignore | 1 + lib/newlib_tests/test22.c | 34 ++++++++++ lib/tst_test.c | 13 +++- 4 files changed, 176 insertions(+), 2 deletions(-) create mode 100644 lib/README.md create mode 100644 lib/newlib_tests/test22.c
Signature · 1 line
2.26.2
PATCH 1/2 lib: tst_test.c: Add TBROK counter.
This ensures that TBROK is never lost.
If test process forks a child and the child calls SAFE_MACRO() the failure will be lost unless the test process handles the exit value properly and propagates the TBROK.
It is also strange that TBROK is the only return value that is solely propagated by the exit value and not by the counters. This has been mistake to begin with.
--- lib/newlib_tests/.gitignore | 1 + lib/newlib_tests/test22.c | 34 ++++++++++++++++++++++++++++++++++ lib/tst_test.c | 13 +++++++++++-- 3 files changed, 46 insertions(+), 2 deletions(-) create mode 100644 lib/newlib_tests/test22.c
lib/newlib_tests/.gitignore +1 -0
@@ -25,6 +25,7 @@ test18 test19 test20 test21 +test22 tst_expiration_timer test_assert test_timer
lib/newlib_tests/test22.c new file +34 -0
@@ -0,0 +1,34 @@ +// SPDX-License-Identifier: GPL-2.0-or-later +/* +* Copyright (c) 2020 Cyril Hrubis <[email protected]> +*/ + +/* +* Test that TBROK is propagated correctly to the results even if we wait on +* child and throw away the status. +*/ +#include "tst_test.h" + +static void do_test (void ) +{ +int pid =SAFE_FORK (); + +if (pid) { +tst_res (TPASS ,"Test main pid" ); +SAFE_WAITPID (pid,NULL ,0 ); +return ; + } + +if (tst_variant ==1 ) +tst_brk (TBROK ,"Test child!" ); +else +tst_brk (TCONF ,"Test child!" ); + +tst_res (TPASS ,"Test child" ); +} + +static struct tst_test test = { + .test_all = do_test, + .test_variants =2 , + .forks_child =1 , +};
lib/tst_test.c +11 -2
@@ -56,6 +56,7 @@ struct results {int skipped;int failed;int warnings; +int broken;unsigned int timeout; };
@@ -179,6 +180,9 @@ static void update_results(int ttype)case TFAIL :tst_atomic_inc (&results->failed);break ; +case TBROK : +tst_atomic_inc (&results->broken); +break ; } }
@@ -368,10 +372,8 @@ static void check_child_status(pid_t pid, int status) ret =WEXITSTATUS (status);switch (ret) {case TPASS : -break ;case TBROK :case TCONF : -tst_brk (ret,"Reported by child (%i)" , pid);break ;default :tst_brk (TBROK ,"Invalid child (%i) exit value %i" , pid, ret);
@@ -698,9 +700,13 @@ static void do_exit(int ret)if (results->warnings) ret |=TWARN ; +if (results->broken) + ret |=TBROK ; +printf ("\nSummary:\n" );printf ("passed %d\n" , results->passed);printf ("failed %d\n" , results->failed); +printf ("broken %d\n" , results->broken);printf ("skipped %d\n" , results->skipped);printf ("warnings %d\n" , results->warnings); }
@@ -737,6 +743,9 @@ static int results_equal(struct results *a, struct results *b)if (a->skipped != b->skipped)return 0 ; +if (a->broken != b->broken) +return 0 ; +return 1 ; }
Signature · 1 line
2.26.2
PATCH 1/2 lib: tst_test.c: Add TBROK counter.
Hi!
Quoted 9 lines
This ensures that TBROK is never lost.
If test process forks a child and the child calls
SAFE_MACRO()the failure will be lost unless the test process handles the exit value properly and propagates the TBROK.It is also strange that TBROK is the only return value that is solely propagated by the exit value and not by the counters. This has been mistake to begin with.
Anybody to review this?
I consider it important and at the same time quite safe to apply since I've tried to keep the changes minimal.
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH 1/2 lib: tst_test.c: Add TBROK counter.
Quoted 7 lines
Hi!
Quoted 5 lines
This ensures that TBROK is never lost.
If test process forks a child and the child calls
SAFE_MACRO()the failure will be lost unless the test process handles the exit value properly and propagates the TBROK.
Quoted 8 lines
Quoted 3 lines
It is also strange that TBROK is the only return value that is solely propagated by the exit value and not by the counters. This has been mistake to begin with.
Anybody to review this?
I consider it important and at the same time quite safe to apply since I've tried to keep the changes minimal.
+1
Kind regards,
Petr
PATCH 1/2 lib: tst_test.c: Add TBROK counter.
Hi!
Quoted 1 line
Reviewed-by Petr Vorel <[email protected]>
Pushed, thanks.
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH 2/2 lib: Add test library design document
In reply to Cyril HrubisWhich tries to explain high level overview and design choices for the test library.
--- lib/README.md | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 lib/README.md
lib/README.md new file +130 -0
@@ -0,0 +1,130 @@ +# Test library design document + +## Test lifetime overview + +When a test is executed the very first thing to happen is that the we check for +various test pre-requisities. These are described in the tst\_test structure +and range from simple '.require\_root' to a more complicated kernel .config +boolean expressions such as: +"CONFIG\_X86\_INTEL\_UMIP=y | CONFIG\_X86\_UMIP=y". + +If all checks are passed the process carries on with setting up the test +environment as requested in the tst\_test structure. There are many different +setup steps that have been put into the test library again ranging from rather +simple creation of a unique test temporary directory to a bit more complicated +ones such as preparing, formatting, and mounting a block device. + +The test library also intializes shrared memory used for IPC at this step. + +Once all the prerequisities are checked and test environment has been prepared +we can move on executing the testcase itself. The actual test is executed in a +forked process, however there are a few hops before we get there. + +First of all there are test variants, which means that the test is re-executed +several times with a slightly different settings. This is usually used to test +a family of similar syscalls, where we test each of these syscalls exactly the +same, but without re-executing the test binary itself. Test varianst are +implemented as a simple global variable counter that gets increased on each +iteration. In a case of syscall tests we switch between which syscall to call +based on the global counter. + +Then there is all\_filesystems flag which is mostly the same as test variants +but executes the test for each filesystem supported by the system. Note that we +can get cartesian product between test variants and all filesystems as well. + +In a pseoudo code it could be expressed as: + +``` +for test_variants: + for all_filesystems: + fork_testrun() +``` + +Before we fork() the test process the test library sets up a timeout alarm and +also a heartbeat signal handlers and also sets up an alarm(2) accordingly to +the test timeout. When a test timeouts the test library gets SIGALRM and the +alarm handler mercilesly kills all forked children by sending SIGKILL to the +whole process group. The heartbeat handler is used by the test process to reset +this timer for example when the test functions runs in a loop. + +With that done we finally fork() the test process. The test process firstly +resets signal handlers and sets its pid to be a process group leader so that we +can slaughter all children if needed. The test library proceeds with suspending +itself in waitpid() syscall and waits for the child to finish at this point. + +The test process goes ahead and call the test setup() function if present in +the tst\_test structure. It's important that we execute all test callbacks +after we have forked the process, that way we cannot crash the test library +process. The setup can also cause the the test to exit prematurely by either +direct or indirect (SAFE\_MACROS()) call to tst\_brk(). In this case the +fork\_testrun() function exits, but the loops for test variants or filesystems +carries on. + +All that is left to be done is to actually execute the tests, what happnes now +depends on the -i and -I command line parameters that can request that the +run() or run\_all() callbacks are executed N times or for a N seconds. Again +the test can exit at any time by direct or indirect call to tst\_brk(). + +Once the test is finished all that is left for the test process is the test +cleanup(). So if a there is a cleanup() callback in the tst\_test strucuture +it's executed. Callback runs in a special context where the tst\_brk(TBROK, +...) calls are converted into tst\_res(TWARN, ...) calls. This is because we +found out that carrying up with partially broken cleanup is usually better +option than exitting it in the middle. + +The test cleanup() is also called by the tst\_brk() handler in order to cleanup +before exitting the test process, hence it must be able to cope even with +partiall test setup. Usually it suffices to make sure to clean up only +resources that already have been set up and to do that in an inverse order that +we did in setup(). + +Once the test process exits or leaves the run() or run\_all() function the test +library wakes up from the waitpid() call, and checks if the test process +exitted normally. + +Once the testrun is finished the test library does a cleanup() as well to clean +up resources set up in the test library setup(), reports test results and +finally exits the process. + +### Test library and fork()-ing + +Things are a bit more complicated when fork()-ing is involved, however the +tests results are stored in a page of a shared memory and incremented by atomic +operations, hence the results are stored rigth after the test reporting +fucntion returns from the test library and the access is, by definition, +race-free as well. + +On the other hand the test library, apart from sending a SIGKILL to the whole +process group on timeout, does not track granchildren. + +This especially means that: + +- The test exits once the main test process exits. + +- While the test results are, by the desing, propagated to the test library + we may still miss a child that gets killed by a signal or exits unexpectedly. + +The test writer should, because of these, take care for mourning these proceses +properly, in most cases this could be simply done by calling +tst\_reap\_children() to collect and dissect deceased. + +Also note that tst\_brk() does exit only the current process, so if child +process calls tst\_brk() the counters are incremented and the process exits. + +### Test library and exec() + +The piece of mapped memory to store the results to is not preserved over +exec(2), hence to use the test library from a binary started by an exec() it +has to be remaped. In this case the process must to call tst\_reinit() before +calling any other library functions. In order to make this happen the program +environment carries LTP\_IPC\_PATH variable with a path to the backing file on +tmpfs. This also allows us to use the test library from a shell testcases. + +### Test library and process synchronization + +The piece of mapped memory is also used as a base for a futex-based +synchronization primitives called checkpoints. And as said previously the +memory can be mapped to any process by calling the tst\_reinit() function. As a +matter of a fact there is even a tst\_checkpoint binary that allows use to use +the checkpoints from shell code as well. +
Signature · 1 line
2.26.2
PATCH 2/2 lib: Add test library design document
Quoted 109 lines · Cyril Hrubis
Which tries to explain high level overview and design choices for the test library.
Signed-off-by Cyril Hrubis <[email protected]>---
lib/README.md| 130 ++++++++++++++++++++++++++++++++++++++++++++++++++1 file changed, 130 insertions(+)create mode 100644
lib/README.mdlib/README.md new file +94
-0@@ -0,0 +1,130 @@ +# Test library design document + +## Test lifetime overview + +When a test is executed the very first thing to happen is that the we check for +various test pre-requisities. These are described in the tst\_test structure +and range from simple '.require\_root' to a more complicated kernel .config +boolean expressions such as: +"CONFIG\_X86\_INTEL\_UMIP=y | CONFIG\_X86\_UMIP=y". + +If all checks are passed the process carries on with setting up the test +environment as requested in the tst\_test structure. There are many different +setup steps that have been put into the test library again ranging from rather +simple creation of a unique test temporary directory to a bit more complicated +ones such as preparing, formatting, and mounting a block device. + +The test library also intializes shrared memory used for IPC at this step. + +Once all the prerequisities are checked and test environment has been prepared +we can move on executing the testcase itself. The actual test is executed in a +forked process, however there are a few hops before we get there. + +First of all there are test variants, which means that the test is re-executed +several times with a slightly different settings. This is usually used to test +a family of similar syscalls, where we test each of these syscalls exactly the +same, but without re-executing the test binary itself. Test varianst are +implemented as a simple global variable counter that gets increased on each +iteration. In a case of syscall tests we switch between which syscall to call +based on the global counter. + +Then there is all\_filesystems flag which is mostly the same as test variants +but executes the test for each filesystem supported by the system. Note that we +can get cartesian product between test variants and all filesystems as well. + +In a pseoudo code it could be expressed as: + +``` +for test_variants: + for all_filesystems: + fork_testrun() +``` + +Before we fork() the test process the test library sets up a timeout alarm and +also a heartbeat signal handlers and also sets up an alarm(2) accordingly to +the test timeout. When a test timeouts the test library gets SIGALRM and the +alarm handler mercilesly kills all forked children by sending SIGKILL to the +whole process group. The heartbeat handler is used by the test process to reset +this timer for example when the test functions runs in a loop. + +With that done we finally fork() the test process. The test process firstly +resets signal handlers and sets its pid to be a process group leader so that we +can slaughter all children if needed. The test library proceeds with suspending +itself in waitpid() syscall and waits for the child to finish at this point. + +The test process goes ahead and call the test setup() function if present in +the tst\_test structure. It's important that we execute all test callbacks +after we have forked the process, that way we cannot crash the test library +process. The setup can also cause the the test to exit prematurely by either +direct or indirect (SAFE\_MACROS()) call to tst\_brk(). In this case the +fork\_testrun() function exits, but the loops for test variants or filesystems +carries on. + +All that is left to be done is to actually execute the tests, what happnes now +depends on the -i and -I command line parameters that can request that the +run() or run\_all() callbacks are executed N times or for a N seconds. Again +the test can exit at any time by direct or indirect call to tst\_brk(). + +Once the test is finished all that is left for the test process is the test +cleanup(). So if a there is a cleanup() callback in the tst\_test strucuture +it's executed. Callback runs in a special context where the tst\_brk(TBROK, +...) calls are converted into tst\_res(TWARN, ...) calls. This is because we +found out that carrying up with partially broken cleanup is usually better +option than exitting it in the middle. + +The test cleanup() is also called by the tst\_brk() handler in order to cleanup +before exitting the test process, hence it must be able to cope even with +partiall test setup. Usually it suffices to make sure to clean up only +resources that already have been set up and to do that in an inverse order that +we did in setup(). + +Once the test process exits or leaves the run() or run\_all() function the test +library wakes up from the waitpid() call, and checks if the test process +exitted normally. + +Once the testrun is finished the test library does a cleanup() as well to clean +up resources set up in the test library setup(), reports test results and +finally exits the process. + +### Test library and fork()-ing + +Things are a bit more complicated when fork()-ing is involved, however the +tests results are stored in a page of a shared memory and incremented by atomic +operations, hence the results are stored rigth after the test reporting +fucntion returns from the test library and the access is, by definition,
lib/README.md line 104
Quoted 10 lines
lib/README.md new file +10
-0+race-free as well. + +On the other hand the test library, apart from sending a SIGKILL to the whole +process group on timeout, does not track granchildren. + +This especially means that: + +- The test exits once the main test process exits. + +- While the test results are, by the desing, propagated to the test library
^^ typo
Quoted 16 lines
lib/README.md new file +16
-0+ we may still miss a child that gets killed by a signal or exits unexpectedly. + +The test writer should, because of these, take care for mourning these proceses +properly, in most cases this could be simply done by calling +tst\_reap\_children() to collect and dissect deceased. + +Also note that tst\_brk() does exit only the current process, so if child +process calls tst\_brk() the counters are incremented and the process exits. + +### Test library and exec() + +The piece of mapped memory to store the results to is not preserved over +exec(2), hence to use the test library from a binary started by an exec() it +has to be remaped. In this case the process must to call tst\_reinit() before +calling any other library functions. In order to make this happen the program +environment carries LTP\_IPC\_PATH variable with a path to the backing file on
lib/README.md line 130
Quoted 10 lines
lib/README.md new file +10
-0+tmpfs. This also allows us to use the test library from a shell testcases. + +### Test library and process synchronization + +The piece of mapped memory is also used as a base for a futex-based +synchronization primitives called checkpoints. And as said previously the +memory can be mapped to any process by calling the tst\_reinit() function. As a +matter of a fact there is even a tst\_checkpoint binary that allows use to use +the checkpoints from shell code as well. +
Looks good to me.
What do you think about adding a small ascii picture(s)? For example, one that shows outline of what's called in library vs. test process:
lib process
+----------------------------+
| main |
| tst_run_tcases |
| do_setup |
| for_each_variant |
| for_each_filesystem | test process
| fork_testrun ---------------------+--------------------------------------------+
| waitpid | | testrun |
| | | do_test_setup |
| | | tst_test->setup |
| | | run_tests |
| | | tst_test->test(i) or tst_test->test_all |
| | | do_test_cleanup |
| | | tst_test->cleanup |
| | | exit(0) |
| do_exit | +--------------------------------------------+
| do_cleanup |
| exit(ret) |
+----------------------------+
PATCH 2/2 lib: Add test library design document
Hi!
Quoted 1 line
Looks good to me.
Thanks.
Quoted 16 lines
What do you think about adding a small ascii picture(s)? For example, one that shows outline of what's called in library vs. test process:
lib process
+----------------------------+
| main |
|tst_run_tcases|
|do_setup|
|for_each_variant|
|for_each_filesystem| test process
|fork_testrun---------------------+--------------------------------------------+
| waitpid | | testrun |
| | |do_test_setup|
| | |tst_test->setup|
| | |run_tests|
Quoted 8 lines
| | |
tst_test->test(i)ortst_test->test_all|
| | |do_test_cleanup|
| | |tst_test->cleanup|
| | | exit(0) |
|do_exit| +--------------------------------------------+
|do_cleanup|
| exit(ret) |
+----------------------------+
I would love that, feel free to send v2 based on my patch.
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH v2 2/2 lib: Add test library design document
In reply to Cyril HrubisFrom: Cyril Hrubis <[email protected]>
Which tries to explain high level overview and design choices for the test library (also know as "newlib").
--- lib/README.md | 153 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 lib/README.md
Add ascii picture and fix small typo.
lib/README.md new file +153 -0
@@ -0,0 +1,153 @@ +# Test library design document + +## High-level picture + + library process + +----------------------------+ + | main | + | tst_run_tcases | + | do_setup | + | for_each_variant | + | for_each_filesystem | test process + | fork_testrun ------------->+--------------------------------------------+ + | waitpid | | testrun | + | | | do_test_setup | + | | | tst_test->setup | + | | | run_tests | + | | | tst_test->test(i) or tst_test->test_all | + | | | do_test_cleanup | + | | | tst_test->cleanup | + | | | exit(0) | + | do_exit | +--------------------------------------------+ + | do_cleanup | + | exit(ret) | + +----------------------------+ + +## Test lifetime overview + +When a test is executed the very first thing to happen is that the we check for +various test pre-requisities. These are described in the tst\_test structure +and range from simple '.require\_root' to a more complicated kernel .config +boolean expressions such as: +"CONFIG\_X86\_INTEL\_UMIP=y | CONFIG\_X86\_UMIP=y". + +If all checks are passed the process carries on with setting up the test +environment as requested in the tst\_test structure. There are many different +setup steps that have been put into the test library again ranging from rather +simple creation of a unique test temporary directory to a bit more complicated +ones such as preparing, formatting, and mounting a block device. + +The test library also intializes shrared memory used for IPC at this step. + +Once all the prerequisities are checked and test environment has been prepared +we can move on executing the testcase itself. The actual test is executed in a +forked process, however there are a few hops before we get there. + +First of all there are test variants, which means that the test is re-executed +several times with a slightly different settings. This is usually used to test +a family of similar syscalls, where we test each of these syscalls exactly the +same, but without re-executing the test binary itself. Test varianst are +implemented as a simple global variable counter that gets increased on each +iteration. In a case of syscall tests we switch between which syscall to call +based on the global counter. + +Then there is all\_filesystems flag which is mostly the same as test variants +but executes the test for each filesystem supported by the system. Note that we +can get cartesian product between test variants and all filesystems as well. + +In a pseoudo code it could be expressed as: + +``` +for test_variants: + for all_filesystems: + fork_testrun() +``` + +Before we fork() the test process the test library sets up a timeout alarm and +also a heartbeat signal handlers and also sets up an alarm(2) accordingly to +the test timeout. When a test timeouts the test library gets SIGALRM and the +alarm handler mercilesly kills all forked children by sending SIGKILL to the +whole process group. The heartbeat handler is used by the test process to reset +this timer for example when the test functions runs in a loop. + +With that done we finally fork() the test process. The test process firstly +resets signal handlers and sets its pid to be a process group leader so that we +can slaughter all children if needed. The test library proceeds with suspending +itself in waitpid() syscall and waits for the child to finish at this point. + +The test process goes ahead and call the test setup() function if present in +the tst\_test structure. It's important that we execute all test callbacks +after we have forked the process, that way we cannot crash the test library +process. The setup can also cause the the test to exit prematurely by either +direct or indirect (SAFE\_MACROS()) call to tst\_brk(). In this case the +fork\_testrun() function exits, but the loops for test variants or filesystems +carries on. + +All that is left to be done is to actually execute the tests, what happnes now +depends on the -i and -I command line parameters that can request that the +run() or run\_all() callbacks are executed N times or for a N seconds. Again +the test can exit at any time by direct or indirect call to tst\_brk(). + +Once the test is finished all that is left for the test process is the test +cleanup(). So if a there is a cleanup() callback in the tst\_test strucuture +it's executed. Callback runs in a special context where the tst\_brk(TBROK, +...) calls are converted into tst\_res(TWARN, ...) calls. This is because we +found out that carrying up with partially broken cleanup is usually better +option than exitting it in the middle. + +The test cleanup() is also called by the tst\_brk() handler in order to cleanup +before exitting the test process, hence it must be able to cope even with +partiall test setup. Usually it suffices to make sure to clean up only +resources that already have been set up and to do that in an inverse order that +we did in setup(). + +Once the test process exits or leaves the run() or run\_all() function the test +library wakes up from the waitpid() call, and checks if the test process +exitted normally. + +Once the testrun is finished the test library does a cleanup() as well to clean +up resources set up in the test library setup(), reports test results and +finally exits the process. + +### Test library and fork()-ing + +Things are a bit more complicated when fork()-ing is involved, however the +tests results are stored in a page of a shared memory and incremented by atomic +operations, hence the results are stored rigth after the test reporting +fucntion returns from the test library and the access is, by definition, +race-free as well. + +On the other hand the test library, apart from sending a SIGKILL to the whole +process group on timeout, does not track granchildren. + +This especially means that: + +- The test exits once the main test process exits. + +- While the test results are, by the design, propagated to the test library + we may still miss a child that gets killed by a signal or exits unexpectedly. + +The test writer should, because of these, take care for mourning these proceses +properly, in most cases this could be simply done by calling +tst\_reap\_children() to collect and dissect deceased. + +Also note that tst\_brk() does exit only the current process, so if child +process calls tst\_brk() the counters are incremented and the process exits. + +### Test library and exec() + +The piece of mapped memory to store the results to is not preserved over +exec(2), hence to use the test library from a binary started by an exec() it +has to be remaped. In this case the process must to call tst\_reinit() before +calling any other library functions. In order to make this happen the program +environment carries LTP\_IPC\_PATH variable with a path to the backing file on +tmpfs. This also allows us to use the test library from a shell testcases. + +### Test library and process synchronization + +The piece of mapped memory is also used as a base for a futex-based +synchronization primitives called checkpoints. And as said previously the +memory can be mapped to any process by calling the tst\_reinit() function. As a +matter of a fact there is even a tst\_checkpoint binary that allows use to use +the checkpoints from shell code as well. +
Signature · 1 line
2.18.1
PATCH v2 2/2 lib: Add test library design document
Hi!
Quoted 12 lines
lib/README.md new file +12
-0+## High-level picture + + library process + +----------------------------+ + | main | + | tst_run_tcases | + | do_setup | + | for_each_variant | + | for_each_filesystem | test process + | fork_testrun ------------->+--------------------------------------------+ + | waitpid | | testrun | + | | | do_test_setup |
lib/README.md line 24
Quoted 10 lines
lib/README.md new file +10
-0+ | | | tst_test->setup | + | | | run_tests | + | | | tst_test->test(i) or tst_test->test_all | + | | | do_test_cleanup | + | | | tst_test->cleanup | + | | | exit(0) | + | do_exit | +--------------------------------------------+ + | do_cleanup | + | exit(ret) | + +----------------------------+
Shouldn't we wrap this to ``` so that it renders nicely on the web?
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH v2 2/2 lib: Add test library design document
----- Original Message -----
Quoted 27 lines
Hi!
Quoted 26 lines
lib/README.md new file +11
-1+## High-level picture + + library process + +----------------------------+ + | main | + | tst_run_tcases | + | do_setup | + | for_each_variant | + | for_each_filesystem | test process + | fork_testrun ------------->+--------------------------------------------+ + | waitpid | | testrun|
+ | | |do_test_setup
|
+ | | |tst_test->setup
|
+ | | |run_tests
|
+ | | |tst_test->test(i)ortst_test->test_all|
+ | | |do_test_cleanup
|
+ | | |tst_test->cleanup
|
+ | | | exit(0)
Quoted 8 lines
lib/README.md line 24
Quoted 6 lines
|
lib/README.md new file +5
-0+ | do_exit | +--------------------------------------------+ + | do_cleanup | + | exit(ret) | + +----------------------------+Shouldn't we wrap this to ``` so that it renders nicely on the web?
I thought the 4 spaces prefix will already do that. I was testing it online on dillinger.io and stackedit.io. But I'm fine with adding ```, if that's better for github.
PATCH v2 2/2 lib: Add test library design document
Hi!
Quoted 35 lines
Quoted 26 lines
Quoted 26 lines
lib/README.md new file +11
-1+## High-level picture + + library process + +----------------------------+ + | main | + | tst_run_tcases | + | do_setup | + | for_each_variant | + | for_each_filesystem | test process + | fork_testrun ------------->+--------------------------------------------+ + | waitpid | | testrun|
+ | | |do_test_setup
|
+ | | |tst_test->setup
|
+ | | |run_tests
|
+ | | |tst_test->test(i)ortst_test->test_all|
+ | | |do_test_cleanup
|
+ | | |tst_test->cleanup
|
+ | | | exit(0)Quoted 8 lines
lib/README.md line 24
Quoted 6 lines
|
lib/README.md new file +5
-0+ | do_exit | +--------------------------------------------+ + | do_cleanup | + | exit(ret) | + +----------------------------+Shouldn't we wrap this to ``` so that it renders nicely on the web?
Quoted 3 lines
I thought the 4 spaces prefix will already do that. I was testing it online on dillinger.io and stackedit.io. But I'm fine with adding ```, if that's better for github.
My bad, four spaces should work here.
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH v2 2/2 lib: Add test library design document
In reply to Jan StancekHi,
thanks for adding documentation, that is really useful.
some typos and a few comments
Quoted 38 lines · Jan Stancek
From: Cyril Hrubis <[email protected]>
Which tries to explain high level overview and design choices for the test library (also know as "newlib").
Signed-off-by Cyril Hrubis <[email protected]>Acked-by Jan Stancek <[email protected]>--- lib/README.md | 153 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+)create mode 100644
lib/README.mdAdd ascii picture and fix small typo.
lib/README.md new file +18
-0@@ -0,0 +1,153 @@ +# Test library design document + +## High-level picture + + library process + +----------------------------+ + | main | + | tst_run_tcases | + | do_setup | + | for_each_variant | + | for_each_filesystem | test process + | fork_testrun ------------->+--------------------------------------------+ + | waitpid | | testrun | + | | | do_test_setup | + | | | tst_test->setup | + | | | run_tests | + | | | tst_test->test(i) or tst_test->test_all | + | | | do_test_cleanup |
lib/README.md line 28
Quoted 10 lines
lib/README.md new file +10
-0+ | | | tst_test->cleanup | + | | | exit(0) | + | do_exit | +--------------------------------------------+ + | do_cleanup | + | exit(ret) | + +----------------------------+ + +## Test lifetime overview + +When a test is executed the very first thing to happen is that the we check for
the we
lib/README.md line 29
Quoted 1 line
lib/README.md new file +1
-0+various test pre-requisities. These are described in the tst\_test structure
prerequisites (better without hyphen and typo)
lib/README.md line 42
Quoted 13 lines
lib/README.md new file +13
-0+and range from simple '.require\_root' to a more complicated kernel .config +boolean expressions such as: +"CONFIG\_X86\_INTEL\_UMIP=y | CONFIG\_X86\_UMIP=y". + +If all checks are passed the process carries on with setting up the test +environment as requested in the tst\_test structure. There are many different +setup steps that have been put into the test library again ranging from rather +simple creation of a unique test temporary directory to a bit more complicated +ones such as preparing, formatting, and mounting a block device. + +The test library also intializes shrared memory used for IPC at this step. + +Once all the prerequisities are checked and test environment has been prepared
prerequisites
lib/README.md line 47
Quoted 5 lines
lib/README.md new file +5
-0+we can move on executing the testcase itself. The actual test is executed in a +forked process, however there are a few hops before we get there. + +First of all there are test variants, which means that the test is re-executed +several times with a slightly different settings. This is usually used to test
a setting
lib/README.md line 49
Quoted 2 lines
lib/README.md new file +2
-0+a family of similar syscalls, where we test each of these syscalls exactly the +same, but without re-executing the test binary itself. Test varianst are
variants
Quoted 9 lines
lib/README.md new file +9
-0+implemented as a simple global variable counter that gets increased on each +iteration. In a case of syscall tests we switch between which syscall to call +based on the global counter. + +Then there is all\_filesystems flag which is mostly the same as test variants +but executes the test for each filesystem supported by the system. Note that we +can get cartesian product between test variants and all filesystems as well. + +In a pseoudo code it could be expressed as:
lib/README.md line 68
Quoted 10 lines
lib/README.md new file +10
-0+ +``` +for test_variants: + for all_filesystems: + fork_testrun() +``` + +Before we fork() the test process the test library sets up a timeout alarm and +also a heartbeat signal handlers and also sets up an alarm(2) accordingly to +the test timeout. When a test timeouts the test library gets SIGALRM and the
"times out", I guess there is no verb timeout
lib/README.md line 69
Quoted 1 line
lib/README.md new file +1
-0+alarm handler mercilesly kills all forked children by sending SIGKILL to the
mercilessly
lib/README.md line 71
Quoted 2 lines
lib/README.md new file +2
-0+whole process group. The heartbeat handler is used by the test process to reset +this timer for example when the test functions runs in a loop.
either "function runs" or "functions run"
lib/README.md line 78
Quoted 7 lines
lib/README.md new file +7
-0+ +With that done we finally fork() the test process. The test process firstly +resets signal handlers and sets its pid to be a process group leader so that we +can slaughter all children if needed. The test library proceeds with suspending +itself in waitpid() syscall and waits for the child to finish at this point. + +The test process goes ahead and call the test setup() function if present in
calls
lib/README.md line 81
Quoted 3 lines
lib/README.md new file +3
-0+the tst\_test structure. It's important that we execute all test callbacks +after we have forked the process, that way we cannot crash the test library +process. The setup can also cause the the test to exit prematurely by either
double the
lib/README.md line 88
Quoted 7 lines
lib/README.md new file +7
-0+direct or indirect (SAFE\_MACROS()) call to tst\_brk(). In this case the +fork\_testrun() function exits, but the loops for test variants or filesystems +carries on. + +All that is left to be done is to actually execute the tests, what happnes now +depends on the -i and -I command line parameters that can request that the +run() or run\_all() callbacks are executed N times or for a N seconds. Again
"for N seconds"
lib/README.md line 92
Quoted 4 lines
lib/README.md new file +4
-0+the test can exit at any time by direct or indirect call to tst\_brk(). + +Once the test is finished all that is left for the test process is the test +cleanup(). So if a there is a cleanup() callback in the tst\_test strucuture
structure
lib/README.md line 93
Quoted 1 line
lib/README.md new file +1
-0+it's executed. Callback runs in a special context where the tst\_brk(TBROK,
Which callbacks? Only the cleanup callback, right? Then this should be "This callback runs"
lib/README.md line 95
Quoted 2 lines
lib/README.md new file +2
-0+...) calls are converted into tst\_res(TWARN, ...) calls. This is because we +found out that carrying up with partially broken cleanup is usually better
"carrying on"
lib/README.md line 100
Quoted 5 lines
lib/README.md new file +5
-0+option than exitting it in the middle. + +The test cleanup() is also called by the tst\_brk() handler in order to cleanup +before exitting the test process, hence it must be able to cope even with +partiall test setup. Usually it suffices to make sure to clean up only
partial
Quoted 5 lines
lib/README.md new file +5
-0+resources that already have been set up and to do that in an inverse order that +we did in setup(). + +Once the test process exits or leaves the run() or run\_all() function the test +library wakes up from the waitpid() call, and checks if the test process
lib/README.md line 115
Quoted 10 lines
lib/README.md new file +10
-0+exitted normally. + +Once the testrun is finished the test library does a cleanup() as well to clean +up resources set up in the test library setup(), reports test results and +finally exits the process. + +### Test library and fork()-ing + +Things are a bit more complicated when fork()-ing is involved, however the +tests results are stored in a page of a shared memory and incremented by atomic
either "test's results" or "test results"
lib/README.md line 116
Quoted 1 line
lib/README.md new file +1
-0+operations, hence the results are stored rigth after the test reporting
right
lib/README.md line 117
Quoted 1 line
lib/README.md new file +1
-0+fucntion returns from the test library and the access is, by definition,
function
lib/README.md line 121
Quoted 4 lines
lib/README.md new file +4
-0+race-free as well. + +On the other hand the test library, apart from sending a SIGKILL to the whole +process group on timeout, does not track granchildren.
grandchildren
lib/README.md line 130
Quoted 9 lines
lib/README.md new file +9
-0+ +This especially means that: + +- The test exits once the main test process exits. + +- While the test results are, by the design, propagated to the test library + we may still miss a child that gets killed by a signal or exits unexpectedly. + +The test writer should, because of these, take care for mourning these proceses
"because of this" and processes
mourning sounds strange maybe just "take care of these processes"?
lib/README.md line 134
Quoted 4 lines
lib/README.md new file +4
-0+properly, in most cases this could be simply done by calling +tst\_reap\_children() to collect and dissect deceased. + +Also note that tst\_brk() does exit only the current process, so if child
"if a child"
lib/README.md line 144
Quoted 10 lines
lib/README.md new file +10
-0+process calls tst\_brk() the counters are incremented and the process exits. + +### Test library and exec() + +The piece of mapped memory to store the results to is not preserved over +exec(2), hence to use the test library from a binary started by an exec() it +has to be remaped. In this case the process must to call tst\_reinit() before +calling any other library functions. In order to make this happen the program +environment carries LTP\_IPC\_PATH variable with a path to the backing file on +tmpfs. This also allows us to use the test library from a shell testcases.
"from shell testcases"
lib/README.md line 151
Quoted 7 lines
lib/README.md new file +7
-0+ +### Test library and process synchronization + +The piece of mapped memory is also used as a base for a futex-based +synchronization primitives called checkpoints. And as said previously the +memory can be mapped to any process by calling the tst\_reinit() function. As a +matter of a fact there is even a tst\_checkpoint binary that allows use to use
"us to use"
lib/README.md line 153
Quoted 2 lines
lib/README.md new file +2
-0+the checkpoints from shell code as well. +
J?rg
PATCH v2 2/2 lib: Add test library design document
Hi!
Quoted 3 lines
thanks for adding documentation, that is really useful.
some typos and a few comments
Fixed the typos and clarified bits you pointed out (and couple more I've found today) and pushed, thanks a lot.
Signature · 2 lines
Cyril Hrubis [email protected]