Proof-of-concept for test documentation format
RFC PATCH 1/1 Proof-of-concept for test documentation format
This is just hacked up proof of concept in order to continue our discussion on test documentation. The code itself is just meant to prove that something is possible and it's not mean for inclusion.
With that in mind, let me describe what I attempted to prove to be possible here:
The test metadata are written in special comment in the test source, see test_docs.c and top level comment starting with /*RST tag.
There is a hacked up tool that parses the comment prior to the test compilation and picks up relevant parts and formats them into C data structures, the results can be seen in test_docs.gen.c once you run make in the newlib_tests directory. This data structures are then linked with the actual test e.g. test_docs and used by the test library.
The upside of this approach is that all the metadata can now reside in the special comment used to describe the test and at the same time the test can access this information at runtime, there is no need to repeat any piece of information in this setup. Even the needs_root fileds can be moved to the comment after this and indeed that works nicely here. At the same time the test can print the test description when passed -h flag, etc.
The only downside is that this is automagical process that adds complexity to the compliation process.
--- include/tst_metadata.h | 28 +++++ include/tst_test.h | 1 + lib/newlib_tests/.gitignore | 3 + lib/newlib_tests/Makefile | 7 ++ lib/newlib_tests/docparse.c | 220 +++++++++++++++++++++++++++++++++++ lib/newlib_tests/test_docs.c | 40 +++++++ lib/tst_test.c | 21 +++- 7 files changed, 318 insertions(+), 2 deletions(-) create mode 100644 include/tst_metadata.h create mode 100644 lib/newlib_tests/docparse.c create mode 100644 lib/newlib_tests/test_docs.c
include/tst_metadata.h new file +28 -0
@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Copyright (C) 2018 Cyril Hrubis <[email protected]> +*/ + +#ifndef TST_METADATA_H__ +#define TST_METADATA_H__ + +struct tst_tag { +const char *name; +const char *value; +}; + +struct tst_metadata { +/* help string printed with -h */ +const char *help; + +/* general test tags */ +const struct tst_tag *tags; + +/* set if test needs root */ +int needs_root:1 ; + +/* NULL-terminated list of required kernel config option */ +const char *const *kconfigs; +}; + +#endif /* TST_METADATA_H__ */
include/tst_test.h +1 -0
@@ -43,6 +43,7 @@#include "tst_get_bad_addr.h" #include "tst_path_has_mnt_flags.h" #include "tst_sys_conf.h" +#include "tst_metadata.h" /* * Reports testcase result.
lib/newlib_tests/.gitignore +3 -0
@@ -24,3 +24,6 @@ test19 tst_expiration_timer test_exec test_exec_child +*.gen.c +test_docs +docparse
lib/newlib_tests/Makefile +7 -0
@@ -12,6 +12,13 @@ test16: CFLAGS+=-pthread test16: LDLIBS+=-lrt tst_expiration_timer: LDLIBS+=-lrt +FILTER_OUT_MAKE_TARGETS += $(subst .c,,$(wildcard *.gen.c)) + +%.gen.c: %.c + ./docparse $< > $@ + +test_docs: test_docs.gen.c + ifeq ($(ANDROID),1) FILTER_OUT_MAKE_TARGETS += test08 endif
lib/newlib_tests/docparse.c new file +220 -0
220 lines not on this page · raw message
lib/newlib_tests/test_docs.c new file +40 -0
@@ -0,0 +1,40 @@ +/* +* Copyright (c) 2018 Cyril Hrubis <[email protected]> +*/ +#include "tst_test.h" + +/*RST + +=========== +Description +=========== + +This is a test for the comment parser. + +* This is a list +* And one more + +==== +Tags +==== + +CVE-2007-2002 + +============ +Requirements +============ + +* root +* CONFIG_FOO=y + +*/ + +static void do_test (void ) +{ +tst_res (TPASS ,"Passed" ); +} + +static struct tst_test test = { + .test_all = do_test, + .needs_root =1 , +};
lib/tst_test.c +19 -2
@@ -409,10 +409,17 @@ static struct option { {"C:", "-C ARG Run child process with ARG arguments (used internally)"}, }; +extern const struct tst_metadata tst_metadata; + +const struct tst_metadata tst_metadata __attribute__((weak)); +static void print_help (void ) {unsigned int i; +if (tst_metadata.help) +fprintf (stderr,"%s" , tst_metadata.help); +for (i =0 ; i <ARRAY_SIZE (options); i++)fprintf (stderr,"%s\n" , options[i].help);
@@ -762,6 +769,17 @@ static void prepare_device(void) } } +static int needs_root (void ) +{ +if (tst_test->needs_root) +return 1 ; + +if (tst_metadata.needs_root) +return 1 ; + +return 0 ; +} +static void do_setup (int argc,char *argv[]) {if (!tst_test)
@@ -779,7 +797,7 @@ static void do_setup(int argc, char *argv[])parse_opts (argc, argv); -if (tst_test->needs_root &&geteuid () !=0 ) +if (needs_root () &&geteuid () !=0 )tst_brk (TCONF ,"Test needs to be run as root" );if (tst_test->min_kver)
@@ -1191,7 +1209,6 @@ void tst_run_tcases(int argc, char *argv[], struct tst_test *self)do_exit (ret); } -void tst_flush (void ) {int rval;
Signature · 1 line
2.18.1
RFC PATCH 1/1 Proof-of-concept for test documentation format
Quoted 58 lines · Cyril Hrubis
This is just hacked up proof of concept in order to continue our discussion on test documentation. The code itself is just meant to prove that something is possible and it's not mean for inclusion.
With that in mind, let me describe what I attempted to prove to be possible here:
The test metadata are written in special comment in the test source, see
test_docs.cand top level comment starting with /*RST tag.There is a hacked up tool that parses the comment prior to the test compilation and picks up relevant parts and formats them into C data structures, the results can be seen in
test_docs.gen.conce you run make in thenewlib_testsdirectory. This data structures are then linked with the actual test e.g.test_docsand used by the test library.The upside of this approach is that all the metadata can now reside in the special comment used to describe the test and at the same time the test can access this information at runtime, there is no need to repeat any piece of information in this setup. Even the
needs_rootfileds can be moved to the comment after this and indeed that works nicely here. At the same time the test can print the test description when passed -h flag, etc.The only downside is that this is automagical process that adds complexity to the compliation process.
Signed-off-by Cyril Hrubis <[email protected]>--- include/tst_metadata.h | 28 +++++ include/tst_test.h | 1 + lib/newlib_tests/.gitignore | 3 + lib/newlib_tests/Makefile | 7 ++ lib/newlib_tests/docparse.c | 220 +++++++++++++++++++++++++++++++++++ lib/newlib_tests/test_docs.c | 40 +++++++ lib/tst_test.c | 21 +++- 7 files changed, 318 insertions(+), 2 deletions(-) create mode 100644 include/tst_metadata.h create mode 100644 lib/newlib_tests/docparse.c create mode 100644 lib/newlib_tests/test_docs.cinclude/tst_metadata.h new file +9
-0@@ -0,0 +1,28 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ +/* +* Copyright (C) 2018 Cyril Hrubis <[email protected]> +*/ + +#ifndef TST_METADATA_H__ +#define TST_METADATA_H__ + +struct tst_tag {
include/tst_metadata.h line 19
Quoted 10 lines
include/tst_metadata.h new file +10
-0+const char *name; +const char *value; +}; + +struct tst_metadata { +/* help string printed with -h */ +const char *help; + +/* general test tags */ +const struct tst_tag *tags;
Maybe we can add a new field .fix_kver to make this hints more valuable/precise?
/* bug fix on kernel version */
const char *fix_kver;
I'm thinking that if the regression test fails on a kernel less than .fix_kver that probably meas hit the same bug; and if the regression test fails on a kernel newer than .fix_kver that probably a new bug/issue in kernel/testcase, then we'd better skip to print the hints in case to misleading LTP user.
Signature · 2 lines
Regards, Li Wang
RFC PATCH 1/1 Proof-of-concept for test documentation format
In reply to Cyril HrubisHi!
Has anybody any comments for this?
Even simple "concept looks OK" would be nice.
If I got at least some possitive feedback about it I would go ahead and implement something less hacky to kick this forward...
Signature · 2 lines
Cyril Hrubis [email protected]
RFC PATCH 1/1 Proof-of-concept for test documentation format
Quoted 8 lines · Cyril Hrubis
Hi!
Has anybody any comments for this?Even simple "concept looks OK" would be nice.
If I got at least some possitive feedback about it I would go ahead and implement something less hacky to kick this forward...
Personally, I think this is a good idea at least for regression tests. If there is any inconvenient perhaps to make the test author spent more energy on the comment definition. But from the template you have given, it seems not very complex. So I feel that we can have a try, and if you like I can provide some test and code review for this then.
Quoted 8 lines
--
Cyril Hrubis
[email protected]--
Mailing list info: https://lists.linux.it/listinfo/ltp
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20190418/c8efa978/attachment.html>
RFC PATCH 1/1 Proof-of-concept for test documentation format
Hi!
Quoted 8 lines
Quoted 7 lines
Has anybody any comments for this?
Even simple "concept looks OK" would be nice.
If I got at least some possitive feedback about it I would go ahead and implement something less hacky to kick this forward...
Quoted 5 lines
Personally, I think this is a good idea at least for regression tests. If there is any inconvenient perhaps to make the test author spent more energy on the comment definition. But from the template you have given, it seems not very complex. So I feel that we can have a try, and if you like I can provide some test and code review for this then.
Thanks for the feedback, I will try to make something cleaner and post v2 probably once we are done with the upcomming release.
Signature · 2 lines
Cyril Hrubis [email protected]
RFC PATCH 1/1 Proof-of-concept for test documentation format
In reply to Cyril HrubisQuoted 7 lines
-----Original Message-----
From: Cyril Hrubis
Has anybody any comments for this?Even simple "concept looks OK" would be nice.
Sorry to have not responded sooner. I assume you are referring to
https://lists.yoctoproject.org/pipermail/automated-testing/2018-November/000359.html
I think having an RST-formatted comment inside the test, along with some conventions (or even a standard) for the schema for the comment contents, is very useful. It would be superior to what we started doing in Fuego with RST-formatted documents that were separate from the code (indeed, they were separate from the whole upstream project - which makes maintenance a real burden.)
With regard to the possible movement of needs_root into a declarative statement inside the RST, there are some pros and cons. In Fuego we are trying to move to as much declarative syntax as possible for test dependencies (or requirements/pre-requisites). This allows a test scheduler to examine the code ahead of time, and avoid executing the test if the pre-requisites are not met.
In this case, with the comment parseable prior to compile time, it would even allow for the build system to avoid compiling the code and deploying it, if the static dependencies were not met. This is particularly useful for kernel config dependencies, to avoid compilation errors. So, that's a "strong support" vote from me on that part of the concept.
This allows other parts of the test stack (the scheduler and the builder) to take into account this information, which is extremely useful. That's completely aside from being able to show this information easily to humans (e.g. incorporate it into the visualization part of the stack) which is also extremely useful.
In the short term, it's quite handy for the test itself to continue to check its pre-requisites at runtime. It would be nice to avoid having this be reliant on complicated build infrastructure. So, IMHO it would be better to auto-populate either the tst_test structure from the meta-data or the meta-data from the tst_test structure. (My vote would be for the latter, in the short term). This would mean, however, that there should be a policy that the tst_test attributes are the single source for this information, and that the 'Requirements' section of the RST document should not be edited by humans (unless you intend to put additional advisory or explanatory elements in that section, just for documentation purposes.)
The TL;DR of all this is that I think this is a good idea.
-- Tim
RFC PATCH 1/1 Proof-of-concept for test documentation format
Hi!
Quoted 2 lines
Sorry to have not responded sooner. I assume you are referring to
https://lists.yoctoproject.org/pipermail/automated-testing/2018-November/000359.html
Yes.
Quoted 12 lines
I think having an RST-formatted comment inside the test, along with some conventions (or even a standard) for the schema for the comment contents, is very useful. It would be superior to what we started doing in Fuego with RST-formatted documents that were separate from the code (indeed, they were separate from the whole upstream project - which makes maintenance a real burden.)
With regard to the possible movement of
needs_rootinto a declarative statement inside the RST, there are some pros and cons. In Fuego we are trying to move to as much declarative syntax as possible for test dependencies (or requirements/pre-requisites). This allows a test scheduler to examine the code ahead of time, and avoid executing the test if the pre-requisites are not met.
Quoted 4 lines
In this case, with the comment parseable prior to compile time, it would even allow for the build system to avoid compiling the code and deploying it, if the static dependencies were not met. This is particularly useful for kernel config dependencies, to avoid compilation errors. So, that's a "strong support" vote from me on that part of the concept.
Good.
Quoted 4 lines
This allows other parts of the test stack (the scheduler and the builder) to take into account this information, which is extremely useful. That's completely aside from being able to show this information easily to humans (e.g. incorporate it into the visualization part of the stack) which is also extremely useful.
That is exactly what I'm aiming for. One of my motivations is that this is a stepping stone for running tests in parallel.
Quoted 2 lines
In the short term, it's quite handy for the test itself to continue to check its pre-requisites at runtime.
The run-time checking is something that cannot go away at any time. The main reason is that kernel devs will still need to be able to execute a single testcase outside of any automated framework to reproduce bugs.
Quoted 7 lines
It would be nice to avoid having this be reliant on complicated build infrastructure. So, IMHO it would be better to auto-populate either the
tst_teststructure from the meta-data or the meta-data from thetst_teststructure. (My vote would be for the latter, in the short term). This would mean, however, that there should be a policy that thetst_testattributes are the single source for this information, and that the 'Requirements' section of the RST document should not be edited by humans (unless you intend to put additional advisory or explanatory elements in that section, just for documentation purposes.)
This part is something I'm not decided on yet, I guess that I will have to keep experimenting a bit, then we can argue which solution is the best.
I thing that it would be bettr to have all the metadata in the top level comment and generate the C structures out of that, but I can be proved wrong :-).
All in all we can desing it so that:
* make metadata rebuilds the metadata for a given targets
* add git hooks that checks for metadata consitency
With that it should be fairly easy to keep the comment and C structures in sync and we would avoid any build-time magic.
Quoted 1 line
The TL;DR of all this is that I think this is a good idea.
Great!
Signature · 2 lines
Cyril Hrubis [email protected]