Skip to contentSkip to conversation

First step in formalizing the test library

PATCH New Cyril Hrubis 4 participants 14 messages Nov 27, 2020 – Dec 9, 2020 Library code

lore
Share & tools
lore
#0 lore raw

cover First step in formalizing the test library

This is partly motivated by the issue #462 where we ended up with a loose corer cases that are not well defined.

So as a first step this patches:

- Do fix old problem with TBROK propagation

- Start a library description that hopefully captures so-far unwritten desing decisions and rules

Cyril Hrubis (2):
lib: tst_test.c: Add TBROK counter.
lib: Add test library design document

 lib/README.md               | 130 ++++++++++++++++++++++++++++++++++++
 lib/newlib_tests/.gitignore |   1 +
 lib/newlib_tests/test22.c   |  34 ++++++++++
 lib/tst_test.c              |  13 +++-
 4 files changed, 176 insertions(+), 2 deletions(-)
 create mode 100644 lib/README.md
 create mode 100644 lib/newlib_tests/test22.c
Signature · 1 line
2.26.2
#1 lore raw

PATCH 1/2 lib: tst_test.c: Add TBROK counter.

This ensures that TBROK is never lost.

If test process forks a child and the child calls SAFE_MACRO() the failure will be lost unless the test process handles the exit value properly and propagates the TBROK.

It is also strange that TBROK is the only return value that is solely propagated by the exit value and not by the counters. This has been mistake to begin with.

---
 lib/newlib_tests/.gitignore |  1 +
 lib/newlib_tests/test22.c   | 34 ++++++++++++++++++++++++++++++++++
 lib/tst_test.c              | 13 +++++++++++--
 3 files changed, 46 insertions(+), 2 deletions(-)
 create mode 100644 lib/newlib_tests/test22.c
lib/newlib_tests/.gitignore +1 -0
index ac1d19be0..6fc549cf2 100644
@@ -25,6 +25,7 @@ test18
 test19
 test20
 test21
+test22
 tst_expiration_timer
 test_assert
 test_timer
lib/newlib_tests/test22.c new file +34 -0
new file mode 100644
index 000000000..520b8dad8
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (c) 2020 Cyril Hrubis <[email protected]>
+ */
+
+/*
+ * Test that TBROK is propagated correctly to the results even if we wait on
+ * child and throw away the status.
+ */
+#include "tst_test.h"
+
+static void do_test(void)
+{
+	int pid = SAFE_FORK();
+
+	if (pid) {
+		tst_res(TPASS, "Test main pid");
+		SAFE_WAITPID(pid, NULL, 0);
+		return;
+	}
+
+	if (tst_variant == 1)
+		tst_brk(TBROK, "Test child!");
+	else
+		tst_brk(TCONF, "Test child!");
+
+	tst_res(TPASS, "Test child");
+}
+
+static struct tst_test test = {
+	.test_all = do_test,
+	.test_variants = 2,
+	.forks_child = 1,
+};
lib/tst_test.c +11 -2
index 535c0ff4c..128058e6e 100644
@@ -56,6 +56,7 @@ struct results {
 	int skipped;
 	int failed;
 	int warnings;
+	int broken;
 	unsigned int timeout;
 };
 
@@ -179,6 +180,9 @@ static void update_results(int ttype)
 	case TFAIL:
 		tst_atomic_inc(&results->failed);
 	break;
+	case TBROK:
+		tst_atomic_inc(&results->broken);
+	break;
 	}
 }
 
@@ -368,10 +372,8 @@ static void check_child_status(pid_t pid, int status)
 	ret = WEXITSTATUS(status);
 	switch (ret) {
 	case TPASS:
-	break;
 	case TBROK:
 	case TCONF:
-		tst_brk(ret, "Reported by child (%i)", pid);
 	break;
 	default:
 		tst_brk(TBROK, "Invalid child (%i) exit value %i", pid, ret);
@@ -698,9 +700,13 @@ static void do_exit(int ret)
 		if (results->warnings)
 			ret |= TWARN;
 
+		if (results->broken)
+			ret |= TBROK;
+
 		printf("\nSummary:\n");
 		printf("passed   %d\n", results->passed);
 		printf("failed   %d\n", results->failed);
+		printf("broken   %d\n", results->broken);
 		printf("skipped  %d\n", results->skipped);
 		printf("warnings %d\n", results->warnings);
 	}
@@ -737,6 +743,9 @@ static int results_equal(struct results *a, struct results *b)
 	if (a->skipped != b->skipped)
 		return 0;
 
+	if (a->broken != b->broken)
+		return 0;
+
 	return 1;
 }
 
Signature · 1 line
2.26.2
#2 lore raw

PATCH 1/2 lib: tst_test.c: Add TBROK counter.

Hi!

Quoted 9 lines

This ensures that TBROK is never lost.

If test process forks a child and the child calls SAFE_MACRO() the failure will be lost unless the test process handles the exit value properly and propagates the TBROK.

It is also strange that TBROK is the only return value that is solely propagated by the exit value and not by the counters. This has been mistake to begin with.

Anybody to review this?

I consider it important and at the same time quite safe to apply since I've tried to keep the changes minimal.

Signature · 2 lines
Cyril Hrubis
[email protected]
#3 lore raw

PATCH 1/2 lib: tst_test.c: Add TBROK counter.

Quoted 7 lines

Hi!

Quoted 5 lines

This ensures that TBROK is never lost.

If test process forks a child and the child calls SAFE_MACRO() the failure will be lost unless the test process handles the exit value properly and propagates the TBROK.

Quoted 8 lines
Quoted 3 lines

It is also strange that TBROK is the only return value that is solely propagated by the exit value and not by the counters. This has been mistake to begin with.

Anybody to review this?

I consider it important and at the same time quite safe to apply since I've tried to keep the changes minimal.

+1

Kind regards,
Petr

#5 lore raw

PATCH 2/2 lib: Add test library design document

In reply to Cyril Hrubis

Which tries to explain high level overview and design choices for the test library.

---
 lib/README.md | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 130 insertions(+)
 create mode 100644 lib/README.md
lib/README.md new file +130 -0
new file mode 100644
index 000000000..6efd3cf33
@@ -0,0 +1,130 @@
+# Test library design document
+
+## Test lifetime overview
+
+When a test is executed the very first thing to happen is that the we check for
+various test pre-requisities. These are described in the tst\_test structure
+and range from simple '.require\_root' to a more complicated kernel .config
+boolean expressions such as:
+"CONFIG\_X86\_INTEL\_UMIP=y | CONFIG\_X86\_UMIP=y".
+
+If all checks are passed the process carries on with setting up the test
+environment as requested in the tst\_test structure. There are many different
+setup steps that have been put into the test library again ranging from rather
+simple creation of a unique test temporary directory to a bit more complicated
+ones such as preparing, formatting, and mounting a block device.
+
+The test library also intializes shrared memory used for IPC at this step.
+
+Once all the prerequisities are checked and test environment has been prepared
+we can move on executing the testcase itself. The actual test is executed in a
+forked process, however there are a few hops before we get there.
+
+First of all there are test variants, which means that the test is re-executed
+several times with a slightly different settings. This is usually used to test
+a family of similar syscalls, where we test each of these syscalls exactly the
+same, but without re-executing the test binary itself. Test varianst are
+implemented as a simple global variable counter that gets increased on each
+iteration. In a case of syscall tests we switch between which syscall to call
+based on the global counter.
+
+Then there is all\_filesystems flag which is mostly the same as test variants
+but executes the test for each filesystem supported by the system. Note that we
+can get cartesian product between test variants and all filesystems as well.
+
+In a pseoudo code it could be expressed as:
+
+```
+for test_variants:
+	for all_filesystems:
+		fork_testrun()
+```
+
+Before we fork() the test process the test library sets up a timeout alarm and
+also a heartbeat signal handlers and also sets up an alarm(2) accordingly to
+the test timeout. When a test timeouts the test library gets SIGALRM and the
+alarm handler mercilesly kills all forked children by sending SIGKILL to the
+whole process group. The heartbeat handler is used by the test process to reset
+this timer for example when the test functions runs in a loop.
+
+With that done we finally fork() the test process. The test process firstly
+resets signal handlers and sets its pid to be a process group leader so that we
+can slaughter all children if needed. The test library proceeds with suspending
+itself in waitpid() syscall and waits for the child to finish at this point.
+
+The test process goes ahead and call the test setup() function if present in
+the tst\_test structure. It's important that we execute all test callbacks
+after we have forked the process, that way we cannot crash the test library
+process. The setup can also cause the the test to exit prematurely by either
+direct or indirect (SAFE\_MACROS()) call to tst\_brk().  In this case the
+fork\_testrun() function exits, but the loops for test variants or filesystems
+carries on.
+
+All that is left to be done is to actually execute the tests, what happnes now
+depends on the -i and -I command line parameters that can request that the
+run() or run\_all() callbacks are executed N times or for a N seconds. Again
+the test can exit at any time by direct or indirect call to tst\_brk().
+
+Once the test is finished all that is left for the test process is the test
+cleanup(). So if a there is a cleanup() callback in the tst\_test strucuture
+it's executed. Callback runs in a special context where the tst\_brk(TBROK,
+...) calls are converted into tst\_res(TWARN, ...) calls. This is because we
+found out that carrying up with partially broken cleanup is usually better
+option than exitting it in the middle.
+
+The test cleanup() is also called by the tst\_brk() handler in order to cleanup
+before exitting the test process, hence it must be able to cope even with
+partiall test setup. Usually it suffices to make sure to clean up only
+resources that already have been set up and to do that in an inverse order that
+we did in setup().
+
+Once the test process exits or leaves the run() or run\_all() function the test
+library wakes up from the waitpid() call, and checks if the test process
+exitted normally.
+
+Once the testrun is finished the test library does a cleanup() as well to clean
+up resources set up in the test library setup(), reports test results and
+finally exits the process.
+
+### Test library and fork()-ing
+
+Things are a bit more complicated when fork()-ing is involved, however the
+tests results are stored in a page of a shared memory and incremented by atomic
+operations, hence the results are stored rigth after the test reporting
+fucntion returns from the test library and the access is, by definition,
+race-free as well.
+
+On the other hand the test library, apart from sending a SIGKILL to the whole
+process group on timeout, does not track granchildren.
+
+This especially means that:
+
+- The test exits once the main test process exits.
+
+- While the test results are, by the desing, propagated to the test library
+  we may still miss a child that gets killed by a signal or exits unexpectedly.
+
+The test writer should, because of these, take care for mourning these proceses
+properly, in most cases this could be simply done by calling
+tst\_reap\_children() to collect and dissect deceased.
+
+Also note that tst\_brk() does exit only the current process, so if child
+process calls tst\_brk() the counters are incremented and the process exits.
+
+### Test library and exec()
+
+The piece of mapped memory to store the results to is not preserved over
+exec(2), hence to use the test library from a binary started by an exec() it
+has to be remaped. In this case the process must to call tst\_reinit() before
+calling any other library functions. In order to make this happen the program
+environment carries LTP\_IPC\_PATH variable with a path to the backing file on
+tmpfs. This also allows us to use the test library from a shell testcases.
+
+### Test library and process synchronization
+
+The piece of mapped memory is also used as a base for a futex-based
+synchronization primitives called checkpoints. And as said previously the
+memory can be mapped to any process by calling the tst\_reinit() function. As a
+matter of a fact there is even a tst\_checkpoint binary that allows use to use
+the checkpoints from shell code as well.
+
Signature · 1 line
2.26.2
#6 lore raw

PATCH 2/2 lib: Add test library design document

Quoted 109 lines · Cyril Hrubis

Which tries to explain high level overview and design choices for the test library.

---
lib/README.md | 130 ++++++++++++++++++++++++++++++++++++++++++++++++++

1 file changed, 130 insertions(+)

create mode 100644 lib/README.md

lib/README.md new file +94 -0
new file mode 100644
index 000000000..6efd3cf33
@@ -0,0 +1,130 @@
+# Test library design document
+
+## Test lifetime overview
+
+When a test is executed the very first thing to happen is that the we check for
+various test pre-requisities. These are described in the tst\_test structure
+and range from simple '.require\_root' to a more complicated kernel .config
+boolean expressions such as:
+"CONFIG\_X86\_INTEL\_UMIP=y | CONFIG\_X86\_UMIP=y".
+
+If all checks are passed the process carries on with setting up the test
+environment as requested in the tst\_test structure. There are many different
+setup steps that have been put into the test library again ranging from rather
+simple creation of a unique test temporary directory to a bit more complicated
+ones such as preparing, formatting, and mounting a block device.
+
+The test library also intializes shrared memory used for IPC at this step.
+
+Once all the prerequisities are checked and test environment has been prepared
+we can move on executing the testcase itself. The actual test is executed in a
+forked process, however there are a few hops before we get there.
+
+First of all there are test variants, which means that the test is re-executed
+several times with a slightly different settings. This is usually used to test
+a family of similar syscalls, where we test each of these syscalls exactly the
+same, but without re-executing the test binary itself. Test varianst are
+implemented as a simple global variable counter that gets increased on each
+iteration. In a case of syscall tests we switch between which syscall to call
+based on the global counter.
+
+Then there is all\_filesystems flag which is mostly the same as test variants
+but executes the test for each filesystem supported by the system. Note that we
+can get cartesian product between test variants and all filesystems as well.
+
+In a pseoudo code it could be expressed as:
+
+```
+for test_variants:
+	for all_filesystems:
+		fork_testrun()
+```
+
+Before we fork() the test process the test library sets up a timeout alarm and
+also a heartbeat signal handlers and also sets up an alarm(2) accordingly to
+the test timeout. When a test timeouts the test library gets SIGALRM and the
+alarm handler mercilesly kills all forked children by sending SIGKILL to the
+whole process group. The heartbeat handler is used by the test process to reset
+this timer for example when the test functions runs in a loop.
+
+With that done we finally fork() the test process. The test process firstly
+resets signal handlers and sets its pid to be a process group leader so that we
+can slaughter all children if needed. The test library proceeds with suspending
+itself in waitpid() syscall and waits for the child to finish at this point.
+
+The test process goes ahead and call the test setup() function if present in
+the tst\_test structure. It's important that we execute all test callbacks
+after we have forked the process, that way we cannot crash the test library
+process. The setup can also cause the the test to exit prematurely by either
+direct or indirect (SAFE\_MACROS()) call to tst\_brk().  In this case the
+fork\_testrun() function exits, but the loops for test variants or filesystems
+carries on.
+
+All that is left to be done is to actually execute the tests, what happnes now
+depends on the -i and -I command line parameters that can request that the
+run() or run\_all() callbacks are executed N times or for a N seconds. Again
+the test can exit at any time by direct or indirect call to tst\_brk().
+
+Once the test is finished all that is left for the test process is the test
+cleanup(). So if a there is a cleanup() callback in the tst\_test strucuture
+it's executed. Callback runs in a special context where the tst\_brk(TBROK,
+...) calls are converted into tst\_res(TWARN, ...) calls. This is because we
+found out that carrying up with partially broken cleanup is usually better
+option than exitting it in the middle.
+
+The test cleanup() is also called by the tst\_brk() handler in order to cleanup
+before exitting the test process, hence it must be able to cope even with
+partiall test setup. Usually it suffices to make sure to clean up only
+resources that already have been set up and to do that in an inverse order that
+we did in setup().
+
+Once the test process exits or leaves the run() or run\_all() function the test
+library wakes up from the waitpid() call, and checks if the test process
+exitted normally.
+
+Once the testrun is finished the test library does a cleanup() as well to clean
+up resources set up in the test library setup(), reports test results and
+finally exits the process.
+
+### Test library and fork()-ing
+
+Things are a bit more complicated when fork()-ing is involved, however the
+tests results are stored in a page of a shared memory and incremented by atomic
+operations, hence the results are stored rigth after the test reporting
+fucntion returns from the test library and the access is, by definition,

lib/README.md line 104

Quoted 10 lines
lib/README.md new file +10 -0
+race-free as well.
+
+On the other hand the test library, apart from sending a SIGKILL to the whole
+process group on timeout, does not track granchildren.
+
+This especially means that:
+
+- The test exits once the main test process exits.
+
+- While the test results are, by the desing, propagated to the test library
                                       ^^ typo
Quoted 16 lines
lib/README.md new file +16 -0
+  we may still miss a child that gets killed by a signal or exits unexpectedly.
+
+The test writer should, because of these, take care for mourning these proceses
+properly, in most cases this could be simply done by calling
+tst\_reap\_children() to collect and dissect deceased.
+
+Also note that tst\_brk() does exit only the current process, so if child
+process calls tst\_brk() the counters are incremented and the process exits.
+
+### Test library and exec()
+
+The piece of mapped memory to store the results to is not preserved over
+exec(2), hence to use the test library from a binary started by an exec() it
+has to be remaped. In this case the process must to call tst\_reinit() before
+calling any other library functions. In order to make this happen the program
+environment carries LTP\_IPC\_PATH variable with a path to the backing file on

lib/README.md line 130

Quoted 10 lines
lib/README.md new file +10 -0
+tmpfs. This also allows us to use the test library from a shell testcases.
+
+### Test library and process synchronization
+
+The piece of mapped memory is also used as a base for a futex-based
+synchronization primitives called checkpoints. And as said previously the
+memory can be mapped to any process by calling the tst\_reinit() function. As a
+matter of a fact there is even a tst\_checkpoint binary that allows use to use
+the checkpoints from shell code as well.
+

Looks good to me.

What do you think about adding a small ascii picture(s)? For example, one that shows outline of what's called in library vs. test process:

lib process
+----------------------------+
| main |
| tst_run_tcases |
| do_setup |
| for_each_variant |
| for_each_filesystem | test process
| fork_testrun ---------------------+--------------------------------------------+
| waitpid | | testrun |
| | | do_test_setup |
| | | tst_test->setup |
| | | run_tests |
| | | tst_test->test(i) or tst_test->test_all |
| | | do_test_cleanup |
| | | tst_test->cleanup |
| | | exit(0) |
| do_exit | +--------------------------------------------+
| do_cleanup |
| exit(ret) |
+----------------------------+

#7 lore raw

PATCH 2/2 lib: Add test library design document

Hi!

Quoted 1 line

Looks good to me.

Thanks.

Quoted 16 lines

What do you think about adding a small ascii picture(s)? For example, one that shows outline of what's called in library vs. test process:

lib process
+----------------------------+
| main |
| tst_run_tcases |
| do_setup |
| for_each_variant |
| for_each_filesystem | test process
| fork_testrun ---------------------+--------------------------------------------+
| waitpid | | testrun |
| | | do_test_setup |
| | | tst_test->setup |
| | | run_tests |

Quoted 8 lines

| | | tst_test->test(i) or tst_test->test_all |
| | | do_test_cleanup |
| | | tst_test->cleanup |
| | | exit(0) |
| do_exit | +--------------------------------------------+
| do_cleanup |
| exit(ret) |
+----------------------------+

I would love that, feel free to send v2 based on my patch.

Signature · 2 lines
Cyril Hrubis
[email protected]
#8 lore raw

PATCH v2 2/2 lib: Add test library design document

In reply to Cyril Hrubis

From: Cyril Hrubis <[email protected]>

Which tries to explain high level overview and design choices for the test library (also know as "newlib").

---
 lib/README.md | 153 ++++++++++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 153 insertions(+)
 create mode 100644 lib/README.md

Add ascii picture and fix small typo.

lib/README.md new file +153 -0
new file mode 100644
index 000000000000..2b81ec9aea33
@@ -0,0 +1,153 @@
+# Test library design document
+
+## High-level picture
+
+    library process
+    +----------------------------+
+    | main                       |
+    |  tst_run_tcases            |
+    |   do_setup                 |
+    |   for_each_variant         |
+    |    for_each_filesystem     |   test process
+    |     fork_testrun ------------->+--------------------------------------------+
+    |      waitpid               |   | testrun                                    |
+    |                            |   |  do_test_setup                             |
+    |                            |   |   tst_test->setup                          |
+    |                            |   |  run_tests                                 |
+    |                            |   |   tst_test->test(i) or tst_test->test_all  |
+    |                            |   |  do_test_cleanup                           |
+    |                            |   |   tst_test->cleanup                        |
+    |                            |   |  exit(0)                                   |
+    |   do_exit                  |   +--------------------------------------------+
+    |    do_cleanup              |
+    |     exit(ret)              |
+    +----------------------------+
+
+## Test lifetime overview
+
+When a test is executed the very first thing to happen is that the we check for
+various test pre-requisities. These are described in the tst\_test structure
+and range from simple '.require\_root' to a more complicated kernel .config
+boolean expressions such as:
+"CONFIG\_X86\_INTEL\_UMIP=y | CONFIG\_X86\_UMIP=y".
+
+If all checks are passed the process carries on with setting up the test
+environment as requested in the tst\_test structure. There are many different
+setup steps that have been put into the test library again ranging from rather
+simple creation of a unique test temporary directory to a bit more complicated
+ones such as preparing, formatting, and mounting a block device.
+
+The test library also intializes shrared memory used for IPC at this step.
+
+Once all the prerequisities are checked and test environment has been prepared
+we can move on executing the testcase itself. The actual test is executed in a
+forked process, however there are a few hops before we get there.
+
+First of all there are test variants, which means that the test is re-executed
+several times with a slightly different settings. This is usually used to test
+a family of similar syscalls, where we test each of these syscalls exactly the
+same, but without re-executing the test binary itself. Test varianst are
+implemented as a simple global variable counter that gets increased on each
+iteration. In a case of syscall tests we switch between which syscall to call
+based on the global counter.
+
+Then there is all\_filesystems flag which is mostly the same as test variants
+but executes the test for each filesystem supported by the system. Note that we
+can get cartesian product between test variants and all filesystems as well.
+
+In a pseoudo code it could be expressed as:
+
+```
+for test_variants:
+	for all_filesystems:
+		fork_testrun()
+```
+
+Before we fork() the test process the test library sets up a timeout alarm and
+also a heartbeat signal handlers and also sets up an alarm(2) accordingly to
+the test timeout. When a test timeouts the test library gets SIGALRM and the
+alarm handler mercilesly kills all forked children by sending SIGKILL to the
+whole process group. The heartbeat handler is used by the test process to reset
+this timer for example when the test functions runs in a loop.
+
+With that done we finally fork() the test process. The test process firstly
+resets signal handlers and sets its pid to be a process group leader so that we
+can slaughter all children if needed. The test library proceeds with suspending
+itself in waitpid() syscall and waits for the child to finish at this point.
+
+The test process goes ahead and call the test setup() function if present in
+the tst\_test structure. It's important that we execute all test callbacks
+after we have forked the process, that way we cannot crash the test library
+process. The setup can also cause the the test to exit prematurely by either
+direct or indirect (SAFE\_MACROS()) call to tst\_brk().  In this case the
+fork\_testrun() function exits, but the loops for test variants or filesystems
+carries on.
+
+All that is left to be done is to actually execute the tests, what happnes now
+depends on the -i and -I command line parameters that can request that the
+run() or run\_all() callbacks are executed N times or for a N seconds. Again
+the test can exit at any time by direct or indirect call to tst\_brk().
+
+Once the test is finished all that is left for the test process is the test
+cleanup(). So if a there is a cleanup() callback in the tst\_test strucuture
+it's executed. Callback runs in a special context where the tst\_brk(TBROK,
+...) calls are converted into tst\_res(TWARN, ...) calls. This is because we
+found out that carrying up with partially broken cleanup is usually better
+option than exitting it in the middle.
+
+The test cleanup() is also called by the tst\_brk() handler in order to cleanup
+before exitting the test process, hence it must be able to cope even with
+partiall test setup. Usually it suffices to make sure to clean up only
+resources that already have been set up and to do that in an inverse order that
+we did in setup().
+
+Once the test process exits or leaves the run() or run\_all() function the test
+library wakes up from the waitpid() call, and checks if the test process
+exitted normally.
+
+Once the testrun is finished the test library does a cleanup() as well to clean
+up resources set up in the test library setup(), reports test results and
+finally exits the process.
+
+### Test library and fork()-ing
+
+Things are a bit more complicated when fork()-ing is involved, however the
+tests results are stored in a page of a shared memory and incremented by atomic
+operations, hence the results are stored rigth after the test reporting
+fucntion returns from the test library and the access is, by definition,
+race-free as well.
+
+On the other hand the test library, apart from sending a SIGKILL to the whole
+process group on timeout, does not track granchildren.
+
+This especially means that:
+
+- The test exits once the main test process exits.
+
+- While the test results are, by the design, propagated to the test library
+  we may still miss a child that gets killed by a signal or exits unexpectedly.
+
+The test writer should, because of these, take care for mourning these proceses
+properly, in most cases this could be simply done by calling
+tst\_reap\_children() to collect and dissect deceased.
+
+Also note that tst\_brk() does exit only the current process, so if child
+process calls tst\_brk() the counters are incremented and the process exits.
+
+### Test library and exec()
+
+The piece of mapped memory to store the results to is not preserved over
+exec(2), hence to use the test library from a binary started by an exec() it
+has to be remaped. In this case the process must to call tst\_reinit() before
+calling any other library functions. In order to make this happen the program
+environment carries LTP\_IPC\_PATH variable with a path to the backing file on
+tmpfs. This also allows us to use the test library from a shell testcases.
+
+### Test library and process synchronization
+
+The piece of mapped memory is also used as a base for a futex-based
+synchronization primitives called checkpoints. And as said previously the
+memory can be mapped to any process by calling the tst\_reinit() function. As a
+matter of a fact there is even a tst\_checkpoint binary that allows use to use
+the checkpoints from shell code as well.
+
Signature · 1 line
2.18.1
#9 lore raw

PATCH v2 2/2 lib: Add test library design document

Hi!

Quoted 12 lines
lib/README.md new file +12 -0
+## High-level picture
+
+    library process
+    +----------------------------+
+    | main                       |
+    |  tst_run_tcases            |
+    |   do_setup                 |
+    |   for_each_variant         |
+    |    for_each_filesystem     |   test process
+    |     fork_testrun ------------->+--------------------------------------------+
+    |      waitpid               |   | testrun                                    |
+    |                            |   |  do_test_setup                             |

lib/README.md line 24

Quoted 10 lines
lib/README.md new file +10 -0
+    |                            |   |   tst_test->setup                          |
+    |                            |   |  run_tests                                 |
+    |                            |   |   tst_test->test(i) or tst_test->test_all  |
+    |                            |   |  do_test_cleanup                           |
+    |                            |   |   tst_test->cleanup                        |
+    |                            |   |  exit(0)                                   |
+    |   do_exit                  |   +--------------------------------------------+
+    |    do_cleanup              |
+    |     exit(ret)              |
+    +----------------------------+

Shouldn't we wrap this to ``` so that it renders nicely on the web?

Signature · 2 lines
Cyril Hrubis
[email protected]
#10 lore raw

PATCH v2 2/2 lib: Add test library design document

----- Original Message -----

Quoted 27 lines

Hi!

Quoted 26 lines
lib/README.md new file +11 -1
+## High-level picture
+
+    library process
+    +----------------------------+
+    | main                       |
+    |  tst_run_tcases            |
+    |   do_setup                 |
+    |   for_each_variant         |
+    |    for_each_filesystem     |   test process
+    |     fork_testrun
------------->+--------------------------------------------+
+    |      waitpid               |   | testrun

|
+ | | | do_test_setup
|
+ | | | tst_test->setup
|
+ | | | run_tests
|
+ | | | tst_test->test(i) or
tst_test->test_all |
+ | | | do_test_cleanup
|
+ | | | tst_test->cleanup
|
+ | | | exit(0)

Quoted 8 lines

lib/README.md line 24

Quoted 6 lines

|

lib/README.md new file +5 -0
+    |   do_exit                  |
+--------------------------------------------+
+    |    do_cleanup              |
+    |     exit(ret)              |
+    +----------------------------+

Shouldn't we wrap this to ``` so that it renders nicely on the web?

I thought the 4 spaces prefix will already do that. I was testing it online on dillinger.io and stackedit.io. But I'm fine with adding ```, if that's better for github.

#11 lore raw

PATCH v2 2/2 lib: Add test library design document

Hi!

Quoted 35 lines
Quoted 26 lines
Quoted 26 lines
lib/README.md new file +11 -1
+## High-level picture
+
+    library process
+    +----------------------------+
+    | main                       |
+    |  tst_run_tcases            |
+    |   do_setup                 |
+    |   for_each_variant         |
+    |    for_each_filesystem     |   test process
+    |     fork_testrun
------------->+--------------------------------------------+
+    |      waitpid               |   | testrun

|
+ | | | do_test_setup
|
+ | | | tst_test->setup
|
+ | | | run_tests
|
+ | | | tst_test->test(i) or
tst_test->test_all |
+ | | | do_test_cleanup
|
+ | | | tst_test->cleanup
|
+ | | | exit(0)

Quoted 8 lines

lib/README.md line 24

Quoted 6 lines

|

lib/README.md new file +5 -0
+    |   do_exit                  |
+--------------------------------------------+
+    |    do_cleanup              |
+    |     exit(ret)              |
+    +----------------------------+

Shouldn't we wrap this to ``` so that it renders nicely on the web?

Quoted 3 lines

I thought the 4 spaces prefix will already do that. I was testing it online on dillinger.io and stackedit.io. But I'm fine with adding ```, if that's better for github.

My bad, four spaces should work here.

Signature · 2 lines
Cyril Hrubis
[email protected]
#12 lore raw

PATCH v2 2/2 lib: Add test library design document

In reply to Jan Stancek

Hi,

thanks for adding documentation, that is really useful.

some typos and a few comments

Quoted 38 lines · Jan Stancek

From: Cyril Hrubis <[email protected]>

Which tries to explain high level overview and design choices for the test library (also know as "newlib").

---
  lib/README.md | 153 ++++++++++++++++++++++++++++++++++++++++++++++++++
  1 file changed, 153 insertions(+)

create mode 100644 lib/README.md

Add ascii picture and fix small typo.

lib/README.md new file +18 -0
new file mode 100644
index 000000000000..2b81ec9aea33
@@ -0,0 +1,153 @@
+# Test library design document
+
+## High-level picture
+
+    library process
+    +----------------------------+
+    | main                       |
+    |  tst_run_tcases            |
+    |   do_setup                 |
+    |   for_each_variant         |
+    |    for_each_filesystem     |   test process
+    |     fork_testrun ------------->+--------------------------------------------+
+    |      waitpid               |   | testrun                                    |
+    |                            |   |  do_test_setup                             |
+    |                            |   |   tst_test->setup                          |
+    |                            |   |  run_tests                                 |
+    |                            |   |   tst_test->test(i) or tst_test->test_all  |
+    |                            |   |  do_test_cleanup                           |

lib/README.md line 28

Quoted 10 lines
lib/README.md new file +10 -0
+    |                            |   |   tst_test->cleanup                        |
+    |                            |   |  exit(0)                                   |
+    |   do_exit                  |   +--------------------------------------------+
+    |    do_cleanup              |
+    |     exit(ret)              |
+    +----------------------------+
+
+## Test lifetime overview
+
+When a test is executed the very first thing to happen is that the we check for

the we

lib/README.md line 29

Quoted 1 line
lib/README.md new file +1 -0
+various test pre-requisities. These are described in the tst\_test structure

prerequisites (better without hyphen and typo)

lib/README.md line 42

Quoted 13 lines
lib/README.md new file +13 -0
+and range from simple '.require\_root' to a more complicated kernel .config
+boolean expressions such as:
+"CONFIG\_X86\_INTEL\_UMIP=y | CONFIG\_X86\_UMIP=y".
+
+If all checks are passed the process carries on with setting up the test
+environment as requested in the tst\_test structure. There are many different
+setup steps that have been put into the test library again ranging from rather
+simple creation of a unique test temporary directory to a bit more complicated
+ones such as preparing, formatting, and mounting a block device.
+
+The test library also intializes shrared memory used for IPC at this step.
+
+Once all the prerequisities are checked and test environment has been prepared

prerequisites

lib/README.md line 47

Quoted 5 lines
lib/README.md new file +5 -0
+we can move on executing the testcase itself. The actual test is executed in a
+forked process, however there are a few hops before we get there.
+
+First of all there are test variants, which means that the test is re-executed
+several times with a slightly different settings. This is usually used to test

a setting

lib/README.md line 49

Quoted 2 lines
lib/README.md new file +2 -0
+a family of similar syscalls, where we test each of these syscalls exactly the
+same, but without re-executing the test binary itself. Test varianst are

variants

Quoted 9 lines
lib/README.md new file +9 -0
+implemented as a simple global variable counter that gets increased on each
+iteration. In a case of syscall tests we switch between which syscall to call
+based on the global counter.
+
+Then there is all\_filesystems flag which is mostly the same as test variants
+but executes the test for each filesystem supported by the system. Note that we
+can get cartesian product between test variants and all filesystems as well.
+
+In a pseoudo code it could be expressed as:

lib/README.md line 68

Quoted 10 lines
lib/README.md new file +10 -0
+
+```
+for test_variants:
+	for all_filesystems:
+		fork_testrun()
+```
+
+Before we fork() the test process the test library sets up a timeout alarm and
+also a heartbeat signal handlers and also sets up an alarm(2) accordingly to
+the test timeout. When a test timeouts the test library gets SIGALRM and the

"times out", I guess there is no verb timeout

lib/README.md line 69

Quoted 1 line
lib/README.md new file +1 -0
+alarm handler mercilesly kills all forked children by sending SIGKILL to the

mercilessly

lib/README.md line 71

Quoted 2 lines
lib/README.md new file +2 -0
+whole process group. The heartbeat handler is used by the test process to reset
+this timer for example when the test functions runs in a loop.

either "function runs" or "functions run"

lib/README.md line 78

Quoted 7 lines
lib/README.md new file +7 -0
+
+With that done we finally fork() the test process. The test process firstly
+resets signal handlers and sets its pid to be a process group leader so that we
+can slaughter all children if needed. The test library proceeds with suspending
+itself in waitpid() syscall and waits for the child to finish at this point.
+
+The test process goes ahead and call the test setup() function if present in

calls

lib/README.md line 81

Quoted 3 lines
lib/README.md new file +3 -0
+the tst\_test structure. It's important that we execute all test callbacks
+after we have forked the process, that way we cannot crash the test library
+process. The setup can also cause the the test to exit prematurely by either

double the

lib/README.md line 88

Quoted 7 lines
lib/README.md new file +7 -0
+direct or indirect (SAFE\_MACROS()) call to tst\_brk().  In this case the
+fork\_testrun() function exits, but the loops for test variants or filesystems
+carries on.
+
+All that is left to be done is to actually execute the tests, what happnes now
+depends on the -i and -I command line parameters that can request that the
+run() or run\_all() callbacks are executed N times or for a N seconds. Again

"for N seconds"

lib/README.md line 92

Quoted 4 lines
lib/README.md new file +4 -0
+the test can exit at any time by direct or indirect call to tst\_brk().
+
+Once the test is finished all that is left for the test process is the test
+cleanup(). So if a there is a cleanup() callback in the tst\_test strucuture

structure

lib/README.md line 93

Quoted 1 line
lib/README.md new file +1 -0
+it's executed. Callback runs in a special context where the tst\_brk(TBROK,

Which callbacks? Only the cleanup callback, right? Then this should be "This callback runs"

lib/README.md line 95

Quoted 2 lines
lib/README.md new file +2 -0
+...) calls are converted into tst\_res(TWARN, ...) calls. This is because we
+found out that carrying up with partially broken cleanup is usually better

"carrying on"

lib/README.md line 100

Quoted 5 lines
lib/README.md new file +5 -0
+option than exitting it in the middle.
+
+The test cleanup() is also called by the tst\_brk() handler in order to cleanup
+before exitting the test process, hence it must be able to cope even with
+partiall test setup. Usually it suffices to make sure to clean up only

partial

Quoted 5 lines
lib/README.md new file +5 -0
+resources that already have been set up and to do that in an inverse order that
+we did in setup().
+
+Once the test process exits or leaves the run() or run\_all() function the test
+library wakes up from the waitpid() call, and checks if the test process

lib/README.md line 115

Quoted 10 lines
lib/README.md new file +10 -0
+exitted normally.
+
+Once the testrun is finished the test library does a cleanup() as well to clean
+up resources set up in the test library setup(), reports test results and
+finally exits the process.
+
+### Test library and fork()-ing
+
+Things are a bit more complicated when fork()-ing is involved, however the
+tests results are stored in a page of a shared memory and incremented by atomic

either "test's results" or "test results"

lib/README.md line 116

Quoted 1 line
lib/README.md new file +1 -0
+operations, hence the results are stored rigth after the test reporting

right

lib/README.md line 117

Quoted 1 line
lib/README.md new file +1 -0
+fucntion returns from the test library and the access is, by definition,

function

lib/README.md line 121

Quoted 4 lines
lib/README.md new file +4 -0
+race-free as well.
+
+On the other hand the test library, apart from sending a SIGKILL to the whole
+process group on timeout, does not track granchildren.

grandchildren

lib/README.md line 130

Quoted 9 lines
lib/README.md new file +9 -0
+
+This especially means that:
+
+- The test exits once the main test process exits.
+
+- While the test results are, by the design, propagated to the test library
+  we may still miss a child that gets killed by a signal or exits unexpectedly.
+
+The test writer should, because of these, take care for mourning these proceses

"because of this" and processes

mourning sounds strange maybe just "take care of these processes"?

lib/README.md line 134

Quoted 4 lines
lib/README.md new file +4 -0
+properly, in most cases this could be simply done by calling
+tst\_reap\_children() to collect and dissect deceased.
+
+Also note that tst\_brk() does exit only the current process, so if child

"if a child"

lib/README.md line 144

Quoted 10 lines
lib/README.md new file +10 -0
+process calls tst\_brk() the counters are incremented and the process exits.
+
+### Test library and exec()
+
+The piece of mapped memory to store the results to is not preserved over
+exec(2), hence to use the test library from a binary started by an exec() it
+has to be remaped. In this case the process must to call tst\_reinit() before
+calling any other library functions. In order to make this happen the program
+environment carries LTP\_IPC\_PATH variable with a path to the backing file on
+tmpfs. This also allows us to use the test library from a shell testcases.

"from shell testcases"

lib/README.md line 151

Quoted 7 lines
lib/README.md new file +7 -0
+
+### Test library and process synchronization
+
+The piece of mapped memory is also used as a base for a futex-based
+synchronization primitives called checkpoints. And as said previously the
+memory can be mapped to any process by calling the tst\_reinit() function. As a
+matter of a fact there is even a tst\_checkpoint binary that allows use to use

"us to use"

lib/README.md line 153

Quoted 2 lines
lib/README.md new file +2 -0
+the checkpoints from shell code as well.
+

J?rg

#13 lore raw

PATCH v2 2/2 lib: Add test library design document

Hi!

Quoted 3 lines

thanks for adding documentation, that is really useful.

some typos and a few comments

Fixed the typos and clarified bits you pointed out (and couple more I've found today) and pushed, thanks a lot.

Signature · 2 lines
Cyril Hrubis
[email protected]