Skip to contentSkip to conversation

tester: Add framework for serialized test execution

RFC PATCH Superseded Inga Stotland 3 participants 3 messages Mar 5, 2021 – Mar 9, 2021 Kbuild

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 tester: Add framework for serialized test execution

This adds a framework for running series of noninteractive tests that are capable of executing more complex scenarios than a usual unit test.

The framework enables staging of a test case processing:
- test pre-setup: can be used to prepare an initial testing environment potentially common to between several test cases, e.g., put an emulator in a certain well known state.
- test setup: used to setup conditions specific for the test case
- test run: execute actual test case
- test teardown: Cleans up after test setup
- test post-teardown: Cleans up after test pre-setup

Additional command line options allow to run a selected group of tests based either on a group prefix of the test name or a substring in test description.

---
 Makefile.am  |   6 +-
 ell/tester.c | 786 +++++++++++++++++++++++++++++++++++++++++++++++++++
 ell/tester.h |  94 ++++++
 3 files changed, 884 insertions(+), 2 deletions(-)
 create mode 100644 ell/tester.c
 create mode 100644 ell/tester.h
Makefile.am +4 -2
index 2f9a4ce..4c65e9d 100644
@@ -58,7 +58,8 @@ pkginclude_HEADERS = ell/ell.h \
 			ell/gpio.h \
 			ell/path.h \
 			ell/icmp6.h \
-			ell/acd.h
+			ell/acd.h \
+			ell/tester.h
 
 lib_LTLIBRARIES = ell/libell.la
 
@@ -141,7 +142,8 @@ ell_libell_la_SOURCES = $(linux_headers) \
 			ell/path.c \
 			ell/icmp6.c \
 			ell/icmp6-private.h \
-			ell/acd.c
+			ell/acd.c \
+			ell/tester.c
 
 ell_libell_la_LDFLAGS = -Wl,--no-undefined \
 			-Wl,--version-script=$(top_srcdir)/ell/ell.sym \
ell/tester.c new file +786 -0

786 lines not on this page · raw message

ell/tester.h new file +94 -0

94 lines not on this page · raw message

Signature · 1 line
2.26.2
#1 lore raw

Hi Inga,

Quoted 13 lines · Inga Stotland

This adds a framework for running series of noninteractive tests that are capable of executing more complex scenarios than a usual unit test.

The framework enables staging of a test case processing:
- test pre-setup: can be used to prepare an initial testing environment potentially common to between several test cases, e.g., put an emulator in a certain well known state.
- test setup: used to setup conditions specific for the test case
- test run: execute actual test case
- test teardown: Cleans up after test setup
- test post-teardown: Cleans up after test pre-setup

Quoted 7 lines

Additional command line options allow to run a selected group of tests based either on a group prefix of the test name or a substring in test description.

---
  Makefile.am  |   6 +-
  ell/tester.c | 786 +++++++++++++++++++++++++++++++++++++++++++++++++++
  ell/tester.h |  94 ++++++

Hmm, ell.sym entries?

Quoted 4 lines
  3 files changed, 884 insertions(+), 2 deletions(-)

create mode 100644 ell/tester.c
create mode 100644 ell/tester.h

<snip>

ell/tester.c line 104

Quoted 13 lines
ell/tester.c new file +13 -0
+
+static char *tester_name;
+
+static struct l_queue *test_list;
+static const struct l_queue_entry *test_entry;
+static struct timeval tester_start;
+
+static bool option_list;
+static const char *option_prefix;
+static const char *option_string;
+
+static bool terminated;
+

Curious. This tester framework is part of the main ell library, so I would have assumed it is a 'class' so to speak. But you use static vars here. So is l_tester only used from a dedicated binary?

If not, do the l_tester_* signatures need to be updated to take an l_tester instance and appropriate constructors / destructors defined?

<snip>

Review

Quoted 2 lines
(quoted diff) +2 -0
+
+void *l_tester_get_data(void)

LIB_EXPORT?

Quoted 8 lines
ell/tester.c new file +8 -0
+{
+	struct test_case *test;
+
+	if (!test_entry)
+		return NULL;
+
+	test = test_entry->data;
+

ell/tester.c line 156

Quoted 10 lines
ell/tester.c new file +10 -0
+	return test->user_data;
+}
+
+static double get_elapsed_time(struct timeval *base)
+{
+	static struct timeval now, elapsed;
+
+	gettimeofday(&now, NULL);
+	timersub(&now, base, &elapsed);
+

Have you considered using l_time* from ell/time.h?

ell/tester.c line 159

Quoted 3 lines
ell/tester.c new file +3 -0
+	return elapsed.tv_sec + ((double) elapsed.tv_usec) / 1000000;
+}
+

<snip>

The rest of the code seems straightforward and nothing else really jumped out at me.

Regards,
-Denis

#2 lore raw

Hi Denis,

On Mon, 2021-03-08 at 16:19 -0600, Denis Kenzior wrote:

Hi Inga,

On 3/5/21 3:37 PM, Inga Stotland wrote:

This adds a framework for running series of noninteractive tests

that are capable of executing more complex scenarios than a usual unit

test.

The framework enables staging of a test case processing:

- test pre-setup: can be used to prepare an initial testing environment

potentially common to between several test cases, e.g.,

put an emulator in a certain well known state.

- test setup: used to setup conditions specific for the test case

- test run: execute actual test case

- test teardown: Cleans up after test setup

- test post-teardown: Cleans up after test pre-setup

Additional command line options allow to run a selected group of tests

based either on a group prefix of the test name or a substring in test

description.

---

  Makefile.am  |   6 +-
  ell/tester.c | 786 +++++++++++++++++++++++++++++++++++++++++++++++++++
  ell/tester.h |  94 ++++++

Hmm, ell.sym entries?

  3 files changed, 884 insertions(+), 2 deletions(-)

create mode 100644 ell/tester.c

create mode 100644 ell/tester.h

<snip>

+

+static char *tester_name;

+

+static struct l_queue *test_list;

+static const struct l_queue_entry *test_entry;

+static struct timeval tester_start;

+

+static bool option_list;

+static const char *option_prefix;

+static const char *option_string;

+

+static bool terminated;

+

Curious. This tester framework is part of the main ell library, so I would have

assumed it is a 'class' so to speak. But you use static vars here. So is

l_tester only used from a dedicated binary?

If not, do the l_tester_* signatures need to be updated to take an l_tester

instance and appropriate constructors / destructors defined?

<snip>

+

+void *l_tester_get_data(void)

LIB_EXPORT?

+{

+ struct test_case *test;

+

+ if (!test_entry)

+ return NULL;

+

+ test = test_entry->data;

+

+ return test->user_data;

+}

+

+static double get_elapsed_time(struct timeval *base)

+{

+ static struct timeval now, elapsed;

+

+ gettimeofday(&now, NULL);

+ timersub(&now, base, &elapsed);

+

Have you considered using l_time* from ell/time.h?

+ return elapsed.tv_sec + ((double) elapsed.tv_usec) / 1000000;

+}

+

<snip>

The rest of the code seems straightforward and nothing else really jumped out at me.

Regards,

-Denis

Thanks for reviewing. Will work on v2.

Regards,

Inga