Skip to contentSkip to conversation

Proof-of-concept for test documentation format

RFC PATCH New Cyril Hrubis 3 participants 7 messages Nov 28, 2018 – Apr 18, 2019 Library code

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 Proof-of-concept for test documentation format

This is just hacked up proof of concept in order to continue our discussion on test documentation. The code itself is just meant to prove that something is possible and it's not mean for inclusion.

With that in mind, let me describe what I attempted to prove to be possible here:

The test metadata are written in special comment in the test source, see test_docs.c and top level comment starting with /*RST tag.

There is a hacked up tool that parses the comment prior to the test compilation and picks up relevant parts and formats them into C data structures, the results can be seen in test_docs.gen.c once you run make in the newlib_tests directory. This data structures are then linked with the actual test e.g. test_docs and used by the test library.

The upside of this approach is that all the metadata can now reside in the special comment used to describe the test and at the same time the test can access this information at runtime, there is no need to repeat any piece of information in this setup. Even the needs_root fileds can be moved to the comment after this and indeed that works nicely here. At the same time the test can print the test description when passed -h flag, etc.

The only downside is that this is automagical process that adds complexity to the compliation process.

---
 include/tst_metadata.h       |  28 +++++
 include/tst_test.h           |   1 +
 lib/newlib_tests/.gitignore  |   3 +
 lib/newlib_tests/Makefile    |   7 ++
 lib/newlib_tests/docparse.c  | 220 +++++++++++++++++++++++++++++++++++
 lib/newlib_tests/test_docs.c |  40 +++++++
 lib/tst_test.c               |  21 +++-
 7 files changed, 318 insertions(+), 2 deletions(-)
 create mode 100644 include/tst_metadata.h
 create mode 100644 lib/newlib_tests/docparse.c
 create mode 100644 lib/newlib_tests/test_docs.c
include/tst_metadata.h new file +28 -0
new file mode 100644
index 000000000..8a7363320
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (C) 2018 Cyril Hrubis <[email protected]>
+ */
+
+#ifndef TST_METADATA_H__
+#define TST_METADATA_H__
+
+struct tst_tag {
+	const char *name;
+	const char *value;
+};
+
+struct tst_metadata {
+	/* help string printed with -h */
+	const char *help;
+
+	/* general test tags */
+	const struct tst_tag *tags;
+
+	/* set if test needs root */
+	int needs_root:1;
+
+	/* NULL-terminated list of required kernel config option */
+	const char *const *kconfigs;
+};
+
+#endif /* TST_METADATA_H__ */
include/tst_test.h +1 -0
index 2ebf746eb..82f607a37 100644
@@ -43,6 +43,7 @@
 #include "tst_get_bad_addr.h"
 #include "tst_path_has_mnt_flags.h"
 #include "tst_sys_conf.h"
+#include "tst_metadata.h"
 
 /*
  * Reports testcase result.
lib/newlib_tests/.gitignore +3 -0
index c702644f0..931d05b1b 100644
@@ -24,3 +24,6 @@ test19
 tst_expiration_timer
 test_exec
 test_exec_child
+*.gen.c
+test_docs
+docparse
lib/newlib_tests/Makefile +7 -0
index 2fc50160a..3d4f1e25b 100644
@@ -12,6 +12,13 @@ test16: CFLAGS+=-pthread
 test16: LDLIBS+=-lrt
 tst_expiration_timer: LDLIBS+=-lrt
 
+FILTER_OUT_MAKE_TARGETS += $(subst .c,,$(wildcard *.gen.c))
+
+%.gen.c: %.c
+	./docparse $< > $@
+
+test_docs: test_docs.gen.c
+
 ifeq ($(ANDROID),1)
 FILTER_OUT_MAKE_TARGETS	+= test08
 endif
lib/newlib_tests/docparse.c new file +220 -0

220 lines not on this page · raw message

lib/newlib_tests/test_docs.c new file +40 -0
new file mode 100644
index 000000000..3ee715cb4
@@ -0,0 +1,40 @@
+/*
+ * Copyright (c) 2018 Cyril Hrubis <[email protected]>
+ */
+#include "tst_test.h"
+
+/*RST
+
+===========
+Description
+===========
+
+This is a test for the comment parser.
+
+* This is a list
+* And one more
+
+====
+Tags
+====
+
+CVE-2007-2002
+
+============
+Requirements
+============
+
+* root
+* CONFIG_FOO=y
+
+*/
+
+static void do_test(void)
+{
+	tst_res(TPASS, "Passed");
+}
+
+static struct tst_test test = {
+	.test_all = do_test,
+	.needs_root = 1,
+};
lib/tst_test.c +19 -2
index 661fbbfce..15c825df0 100644
@@ -409,10 +409,17 @@ static struct option {
 	{"C:", "-C ARG   Run child process with ARG arguments (used internally)"},
 };
 
+extern const struct tst_metadata tst_metadata;
+
+const struct tst_metadata tst_metadata __attribute__((weak));
+
 static void print_help(void)
 {
 	unsigned int i;
 
+	if (tst_metadata.help)
+		fprintf(stderr, "%s", tst_metadata.help);
+
 	for (i = 0; i < ARRAY_SIZE(options); i++)
 		fprintf(stderr, "%s\n", options[i].help);
 
@@ -762,6 +769,17 @@ static void prepare_device(void)
 	}
 }
 
+static int needs_root(void)
+{
+	if (tst_test->needs_root)
+		return 1;
+
+	if (tst_metadata.needs_root)
+		return 1;
+
+	return 0;
+}
+
 static void do_setup(int argc, char *argv[])
 {
 	if (!tst_test)
@@ -779,7 +797,7 @@ static void do_setup(int argc, char *argv[])
 
 	parse_opts(argc, argv);
 
-	if (tst_test->needs_root && geteuid() != 0)
+	if (needs_root() && geteuid() != 0)
 		tst_brk(TCONF, "Test needs to be run as root");
 
 	if (tst_test->min_kver)
@@ -1191,7 +1209,6 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)
 	do_exit(ret);
 }
 
-
 void tst_flush(void)
 {
 	int rval;
Signature · 1 line
2.18.1
#1 lore raw

RFC PATCH 1/1 Proof-of-concept for test documentation format

Quoted 58 lines · Cyril Hrubis

This is just hacked up proof of concept in order to continue our discussion on test documentation. The code itself is just meant to prove that something is possible and it's not mean for inclusion.

With that in mind, let me describe what I attempted to prove to be possible here:

The test metadata are written in special comment in the test source, see test_docs.c and top level comment starting with /*RST tag.

There is a hacked up tool that parses the comment prior to the test compilation and picks up relevant parts and formats them into C data structures, the results can be seen in test_docs.gen.c once you run make in the newlib_tests directory. This data structures are then linked with the actual test e.g. test_docs and used by the test library.

The upside of this approach is that all the metadata can now reside in the special comment used to describe the test and at the same time the test can access this information at runtime, there is no need to repeat any piece of information in this setup. Even the needs_root fileds can be moved to the comment after this and indeed that works nicely here. At the same time the test can print the test description when passed -h flag, etc.

The only downside is that this is automagical process that adds complexity to the compliation process.

---
 include/tst_metadata.h       |  28 +++++
 include/tst_test.h           |   1 +
 lib/newlib_tests/.gitignore  |   3 +
 lib/newlib_tests/Makefile    |   7 ++
 lib/newlib_tests/docparse.c  | 220 +++++++++++++++++++++++++++++++++++
 lib/newlib_tests/test_docs.c |  40 +++++++
 lib/tst_test.c               |  21 +++-
 7 files changed, 318 insertions(+), 2 deletions(-)
 create mode 100644 include/tst_metadata.h
 create mode 100644 lib/newlib_tests/docparse.c
 create mode 100644 lib/newlib_tests/test_docs.c
include/tst_metadata.h new file +9 -0
new file mode 100644
index 000000000..8a7363320
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright (C) 2018 Cyril Hrubis <[email protected]>
+ */
+
+#ifndef TST_METADATA_H__
+#define TST_METADATA_H__
+
+struct tst_tag {

include/tst_metadata.h line 19

Quoted 10 lines
include/tst_metadata.h new file +10 -0
+       const char *name;
+       const char *value;
+};
+
+struct tst_metadata {
+       /* help string printed with -h */
+       const char *help;
+
+       /* general test tags */
+       const struct tst_tag *tags;

Maybe we can add a new field .fix_kver to make this hints more valuable/precise?

/* bug fix on kernel version */
const char *fix_kver;

I'm thinking that if the regression test fails on a kernel less than .fix_kver that probably meas hit the same bug; and if the regression test fails on a kernel newer than .fix_kver that probably a new bug/issue in kernel/testcase, then we'd better skip to print the hints in case to misleading LTP user.

Signature · 2 lines
Regards,
Li Wang
#2 lore raw

RFC PATCH 1/1 Proof-of-concept for test documentation format

In reply to Cyril Hrubis

Hi!
Has anybody any comments for this?

Even simple "concept looks OK" would be nice.

If I got at least some possitive feedback about it I would go ahead and implement something less hacky to kick this forward...

Signature · 2 lines
Cyril Hrubis
[email protected]
#3 lore raw

RFC PATCH 1/1 Proof-of-concept for test documentation format

Quoted 8 lines · Cyril Hrubis

Hi!
Has anybody any comments for this?

Even simple "concept looks OK" would be nice.

If I got at least some possitive feedback about it I would go ahead and implement something less hacky to kick this forward...

Personally, I think this is a good idea at least for regression tests. If there is any inconvenient perhaps to make the test author spent more energy on the comment definition. But from the template you have given, it seems not very complex. So I feel that we can have a try, and if you like I can provide some test and code review for this then.

Quoted 8 lines

--
Cyril Hrubis
[email protected]

--
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 5 lines
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20190418/c8efa978/attachment.html>
#4 lore raw

RFC PATCH 1/1 Proof-of-concept for test documentation format

Hi!

Quoted 8 lines
Quoted 7 lines

Has anybody any comments for this?

Even simple "concept looks OK" would be nice.

If I got at least some possitive feedback about it I would go ahead and implement something less hacky to kick this forward...

Quoted 5 lines

Personally, I think this is a good idea at least for regression tests. If there is any inconvenient perhaps to make the test author spent more energy on the comment definition. But from the template you have given, it seems not very complex. So I feel that we can have a try, and if you like I can provide some test and code review for this then.

Thanks for the feedback, I will try to make something cleaner and post v2 probably once we are done with the upcomming release.

Signature · 2 lines
Cyril Hrubis
[email protected]
#5 lore raw

RFC PATCH 1/1 Proof-of-concept for test documentation format

In reply to Cyril Hrubis
Quoted 7 lines

-----Original Message-----
From: Cyril Hrubis
Has anybody any comments for this?

Even simple "concept looks OK" would be nice.

Sorry to have not responded sooner. I assume you are referring to
https://lists.yoctoproject.org/pipermail/automated-testing/2018-November/000359.html

I think having an RST-formatted comment inside the test, along with some conventions (or even a standard) for the schema for the comment contents, is very useful. It would be superior to what we started doing in Fuego with RST-formatted documents that were separate from the code (indeed, they were separate from the whole upstream project - which makes maintenance a real burden.)

With regard to the possible movement of needs_root into a declarative statement inside the RST, there are some pros and cons. In Fuego we are trying to move to as much declarative syntax as possible for test dependencies (or requirements/pre-requisites). This allows a test scheduler to examine the code ahead of time, and avoid executing the test if the pre-requisites are not met.

In this case, with the comment parseable prior to compile time, it would even allow for the build system to avoid compiling the code and deploying it, if the static dependencies were not met. This is particularly useful for kernel config dependencies, to avoid compilation errors. So, that's a "strong support" vote from me on that part of the concept.

This allows other parts of the test stack (the scheduler and the builder) to take into account this information, which is extremely useful. That's completely aside from being able to show this information easily to humans (e.g. incorporate it into the visualization part of the stack) which is also extremely useful.

In the short term, it's quite handy for the test itself to continue to check its pre-requisites at runtime. It would be nice to avoid having this be reliant on complicated build infrastructure. So, IMHO it would be better to auto-populate either the tst_test structure from the meta-data or the meta-data from the tst_test structure. (My vote would be for the latter, in the short term). This would mean, however, that there should be a policy that the tst_test attributes are the single source for this information, and that the 'Requirements' section of the RST document should not be edited by humans (unless you intend to put additional advisory or explanatory elements in that section, just for documentation purposes.)

The TL;DR of all this is that I think this is a good idea.
-- Tim

#6 lore raw

RFC PATCH 1/1 Proof-of-concept for test documentation format

Hi!

Quoted 2 lines

Sorry to have not responded sooner. I assume you are referring to
https://lists.yoctoproject.org/pipermail/automated-testing/2018-November/000359.html

Yes.

Quoted 12 lines

I think having an RST-formatted comment inside the test, along with some conventions (or even a standard) for the schema for the comment contents, is very useful. It would be superior to what we started doing in Fuego with RST-formatted documents that were separate from the code (indeed, they were separate from the whole upstream project - which makes maintenance a real burden.)

With regard to the possible movement of needs_root into a declarative statement inside the RST, there are some pros and cons. In Fuego we are trying to move to as much declarative syntax as possible for test dependencies (or requirements/pre-requisites). This allows a test scheduler to examine the code ahead of time, and avoid executing the test if the pre-requisites are not met.

Quoted 4 lines

In this case, with the comment parseable prior to compile time, it would even allow for the build system to avoid compiling the code and deploying it, if the static dependencies were not met. This is particularly useful for kernel config dependencies, to avoid compilation errors. So, that's a "strong support" vote from me on that part of the concept.

Good.

Quoted 4 lines

This allows other parts of the test stack (the scheduler and the builder) to take into account this information, which is extremely useful. That's completely aside from being able to show this information easily to humans (e.g. incorporate it into the visualization part of the stack) which is also extremely useful.

That is exactly what I'm aiming for. One of my motivations is that this is a stepping stone for running tests in parallel.

Quoted 2 lines

In the short term, it's quite handy for the test itself to continue to check its pre-requisites at runtime.

The run-time checking is something that cannot go away at any time. The main reason is that kernel devs will still need to be able to execute a single testcase outside of any automated framework to reproduce bugs.

Quoted 7 lines

It would be nice to avoid having this be reliant on complicated build infrastructure. So, IMHO it would be better to auto-populate either the tst_test structure from the meta-data or the meta-data from the tst_test structure. (My vote would be for the latter, in the short term). This would mean, however, that there should be a policy that the tst_test attributes are the single source for this information, and that the 'Requirements' section of the RST document should not be edited by humans (unless you intend to put additional advisory or explanatory elements in that section, just for documentation purposes.)

This part is something I'm not decided on yet, I guess that I will have to keep experimenting a bit, then we can argue which solution is the best.

I thing that it would be bettr to have all the metadata in the top level comment and generate the C structures out of that, but I can be proved wrong :-).

All in all we can desing it so that:

* make metadata rebuilds the metadata for a given targets
* add git hooks that checks for metadata consitency

With that it should be fairly easy to keep the comment and C structures in sync and we would avoid any build-time magic.

Quoted 1 line

The TL;DR of all this is that I think this is a good idea.

Great!

Signature · 2 lines
Cyril Hrubis
[email protected]