Test metadata extraction
cover Test metadata extraction
This patchset adds a test metadata extraction into LTP and also documentation generator that produces browseable HTML documentation from the exported metadata. For detailed description of the idea and implementation see the patch that adds README.md.
While the idea is quite new the code is mature enough to be included in the upstream repository and I'm also worried that we will not get any feedback or users of the metadata unless it's included in the upstream git.
The next step is to use the extracted metadata in runltp-ng in the proof of concept parallel executor that has been written by Ritchie and posted to this mailing list as well.
Cyril Hrubis (4):
docparse: Add test documentation parser
docparse: Add README
syscalls: Add a few documentation comments
syscalls: Move needs_drivers inside of the tst_test struct
Petr Vorel (7):
make: Support compiling native build tools
travis: Add git
make: Allow {INSTALL,MAKE}_TARGETS be a directory
make: Allow CLEAN_TARGETS to remove directories
travis: Install docparse dependencies
docparse: Add configure options
docparse: Generate html and pdf using asciidoc{,tor}
Makefile | 4 + configure.ac | 32 +- docparse/.gitignore | 7 + docparse/Makefile | 77 ++++ docparse/README.md | 248 ++++++++++ docparse/data_storage.h | 299 ++++++++++++ docparse/docparse.c | 415 +++++++++++++++++ docparse/parse.sh | 41 ++ docparse/testinfo.pl | 424 ++++++++++++++++++ include/mk/config.mk.in | 21 + include/mk/env_post.mk | 3 +- include/mk/features.mk.in | 5 + include/mk/functions.mk | 3 +- include/mk/generic_leaf_target.inc | 16 +- include/mk/rules.mk | 8 + m4/ax_compare_version.m4 | 177 ++++++++ m4/ax_prog_perl_modules.m4 | 77 ++++ m4/ltp-docparse.m4 | 112 +++++ testcases/kernel/syscalls/abort/abort01.c | 16 +- testcases/kernel/syscalls/accept/accept01.c | 8 +- testcases/kernel/syscalls/accept/accept02.c | 7 +- testcases/kernel/syscalls/acct/acct01.c | 5 + testcases/kernel/syscalls/acct/acct02.c | 6 +- .../kernel/syscalls/fsetxattr/fsetxattr02.c | 10 +- testcases/kernel/syscalls/ioctl/ioctl08.c | 9 +- travis/alpine.sh | 4 + travis/debian.minimal.sh | 8 +- travis/debian.sh | 10 +- travis/fedora.sh | 12 +- travis/tumbleweed.sh | 9 +- 30 files changed, 2034 insertions(+), 39 deletions(-) create mode 100644 docparse/.gitignore create mode 100644 docparse/Makefile create mode 100644 docparse/README.md create mode 100644 docparse/data_storage.h create mode 100644 docparse/docparse.c create mode 100755 docparse/parse.sh create mode 100755 docparse/testinfo.pl create mode 100644 m4/ax_compare_version.m4 create mode 100644 m4/ax_prog_perl_modules.m4 create mode 100644 m4/ltp-docparse.m4
Signature · 1 line
2.26.2
PATCH 1/11 make: Support compiling native build tools
From: Petr Vorel <[email protected]>
Add HOST_MAKE_TARGETS make target and HOSTCC,
HOST_{CFLAGS,LDFLAGS} make variables.
Needed for cross-compilation.
NOTE: detect cross compilation with comparing $(build) and $(host) instead of using $cross_compiling configure variable, which would require move the detection into m4 macro.
--- configure.ac | 2 ++ include/mk/config.mk.in | 21 +++++++++++++++++++++ include/mk/env_post.mk | 3 +-- include/mk/generic_leaf_target.inc | 6 ++++++ include/mk/rules.mk | 8 ++++++++ 5 files changed, 38 insertions(+), 2 deletions(-)
configure.ac +2 -0
@@ -13,6 +13,8 @@ AC_CONFIG_FILES([ \ execltp \ ]) +AC_ARG_VAR(HOSTCC, [The C compiler on the host]) + AM_MAINTAINER_MODE([enable]) AC_CANONICAL_HOST
include/mk/config.mk.in +21 -0
@@ -31,6 +31,19 @@ RANLIB := @RANLIB@ STRIP := @STRIP@ YACC := @YACC@ +HOSTCC = @HOSTCC@ +build := @build@ +host := @host@ +ifeq ($(strip $(HOSTCC)),) +# native build, respect CC +ifeq ($(build),$(host)) +HOSTCC := $(CC) +else +# cross compilation +HOSTCC := cc +endif +endif + AIO_LIBS := @AIO_LIBS@ CAP_LIBS := @CAP_LIBS@ ACL_LIBS := @ACL_LIBS@
@@ -70,6 +83,14 @@ WCFLAGS ?= -Wall -W @GCC_WARN_OLDSTYLE@ LDFLAGS += $(WLDFLAGS) CFLAGS += $(DEBUG_CFLAGS) $(OPT_CFLAGS) $(WCFLAGS) +ifeq ($(strip $(HOST_CFLAGS)),) +HOST_CFLAGS := $(CFLAGS) +endif + +ifeq ($(strip $(HOST_LDFLAGS)),) +HOST_LDFLAGS := $(LDFLAGS) +endif + LINUX_VERSION := @LINUX_VERSION@ LINUX_DIR := @LINUX_DIR@ LINUX_VERSION_MAJOR := @LINUX_VERSION_MAJOR@
include/mk/env_post.mk +1 -2
@@ -48,11 +48,10 @@ LDFLAGS += -L$(top_builddir)/lib/android_librt endif MAKE_TARGETS ?= $(notdir $(patsubst %.c,%,$(wildcard $(abs_srcdir)/*.c))) - MAKE_TARGETS := $(filter-out $(FILTER_OUT_MAKE_TARGETS),$(MAKE_TARGETS)) # with only *.dwo, .[0-9]+.dwo can not be cleaned -CLEAN_TARGETS += $(MAKE_TARGETS) *.o *.pyc .cache.mk *.dwo .*.dwo +CLEAN_TARGETS += $(MAKE_TARGETS) $(HOST_MAKE_TARGETS) *.o *.pyc .cache.mk *.dwo .*.dwo # Majority of the files end up in testcases/bin... INSTALL_DIR ?= testcases/bin
include/mk/generic_leaf_target.inc +6 -0
@@ -57,6 +57,8 @@ # rope to hang one's self in the event of # unwanted behavior. # +# $(HOST_MAKE_TARGETS) : Host tools which use $HOSTCC. +# # $(CLEAN_TARGETS) : What targets should be cleaned (must be # real files). This will automatically append # adds the .o suffix to all files referenced
@@ -92,6 +94,10 @@ .PHONY: all clean install +ifneq ($(strip $(MAKE_TARGETS)),) +$(MAKE_TARGETS) += $(HOST_MAKE_TARGETS) +endif + $(MAKE_TARGETS): | $(MAKE_DEPS) all: $(MAKE_TARGETS)
include/mk/rules.mk +8 -0
@@ -22,6 +22,14 @@ else @echo LD $(target_rel_dir)$@ endif +$(HOST_MAKE_TARGETS): %: %.c +ifdef VERBOSE + $(HOSTCC) $(HOST_CFLAGS) $(HOST_LDFLAGS) $< $(HOST_LDLIBS) -o $@ +else + @$(HOSTCC) $(HOST_CFLAGS) $(HOST_LDFLAGS) $< $(HOST_LDLIBS) -o $@ + @echo HOSTCC $(target_rel_dir)$@ +endif + %: %.c ifdef VERBOSE $(CC) $(CPPFLAGS) $(CFLAGS) $(LDFLAGS) $^ $(LTPLDLIBS) $(LDLIBS) -o $@
Signature · 1 line
2.26.2
PATCH 2/11 travis: Add git
In reply to Cyril HrubisFrom: Petr Vorel <[email protected]>
Needed in parse.sh script in next commit.
--- travis/alpine.sh | 1 + travis/debian.sh | 1 + travis/fedora.sh | 1 + travis/tumbleweed.sh | 1 + 4 files changed, 4 insertions(+)
travis/alpine.sh +1 -0
@@ -10,6 +10,7 @@ apk add \ automake \ clang \ gcc \ + git \ keyutils-dev \ libaio-dev \ libacl \
travis/debian.sh +1 -0
@@ -17,6 +17,7 @@ apt install -y --no-install-recommends \ devscripts \ clang \ gcc \ + git \ libacl1 \ libacl1-dev \ libaio-dev \
travis/fedora.sh +1 -0
@@ -8,6 +8,7 @@ yum -y install \ make \ clang \ gcc \ + git \ findutils \ libtirpc \ libtirpc-devel \
travis/tumbleweed.sh +1 -0
@@ -8,6 +8,7 @@ zypper --non-interactive install --force-resolution --no-recommends \ clang \ findutils \ gcc \ + git \ gzip \ make \ kernel-default-devel \
Signature · 1 line
2.26.2
PATCH 3/11 docparse: Add test documentation parser
In reply to Cyril HrubisFrom: Cyril Hrubis <[email protected]>
This commit implements a simple parser to pick up metadata that are nicely structured in the tst_test structure. In order to do that it implements very simple and very basic C tokenizer.
The result of the metadata extraction is one big json file that contains data extracted from all new library testcases plus some information about the LTP version etc.
The tokenizer also extracts special documentation comments that are then used for a test description. The documentation format has been choosen to be asciidoc.
Special documentation comments starts with string "/*\\n" and is, by convention ended with the same, so it looks like:
/*\
* [DOCUMENTATION]
* - foo
* - bar
*
* ...
*
\*/
[ pvorel: fixes
- parse.sh: git related fixes, out-of-tree build
- docparse: Eat also space following *
- Makefile: remove header rule (broken for clang)
]
--- Makefile | 2 +- docparse/.gitignore | 2 + docparse/Makefile | 19 ++ docparse/data_storage.h | 299 +++++++++++++++++++++++++++++ docparse/docparse.c | 415 ++++++++++++++++++++++++++++++++++++++++ docparse/parse.sh | 41 ++++ docparse/testinfo.pl | 40 ++++ 7 files changed, 817 insertions(+), 1 deletion(-) create mode 100644 docparse/.gitignore create mode 100644 docparse/Makefile create mode 100644 docparse/data_storage.h create mode 100644 docparse/docparse.c create mode 100755 docparse/parse.sh create mode 100755 docparse/testinfo.pl
Makefile +1 -1
@@ -62,7 +62,7 @@ $(1):: | $$(abs_top_builddir)/$$(basename $$(subst -,.,$(1))) endif endef -COMMON_TARGETS += testcases tools +COMMON_TARGETS += testcases tools docparse # Don't want to nuke the original files if we're installing in-build-tree. ifneq ($(BUILD_TREE_STATE),$(BUILD_TREE_SRCDIR_INSTALL)) INSTALL_TARGETS += runtest scenario_groups testscripts
docparse/.gitignore new file +2 -0
@@ -0,0 +1,2 @@ +/docparse +/metadata.json
docparse/Makefile new file +19 -0
@@ -0,0 +1,19 @@ +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2019 Cyril Hrubis <[email protected]> + +top_srcdir ?= .. + +include $(top_srcdir)/include/mk/env_pre.mk +include $(top_srcdir)/include/mk/functions.mk + +MAKE_TARGETS := metadata.json +HOST_MAKE_TARGETS := docparse + +INSTALL_DIR = metadata + +.PHONY: metadata.json + +metadata.json: docparse + $(abs_srcdir)/parse.sh > metadata.json + +include $(top_srcdir)/include/mk/generic_leaf_target.mk
docparse/data_storage.h new file +299 -0
299 lines not on this page · raw message
docparse/docparse.c new file +415 -0
415 lines not on this page · raw message
docparse/parse.sh new file +41 -0
41 lines not on this page · raw message
docparse/testinfo.pl new file +40 -0
@@ -0,0 +1,40 @@ +#!/usr/bin/perl +# SPDX-License-Identifier: GPL-2.0-or-later +# Copyright (c) 2019 Cyril Hrubis <[email protected]> + +use strict; +use warnings; + +use JSON; +use Data::Dumper; + +sub load_json +{ + my ($fname) = @_; + local $/; + + open(my $fh, '<', $fname) or die("Can't open $fname $!"); + + return <$fh>; +} + +sub query_flag +{ + my ($json, $flag) = @_; + + my $tests = $json->{'tests'}; + + foreach my $key (sort(keys %$tests)) { + if ($tests->{$key}->{$flag}) { + if ($tests->{$key}->{$flag} eq "1") { + print("$key\n"); + } else { + print("$key:\n" . Dumper($tests->{$key}->{$flag}) . "\n"); + } + } + } +} + +my $json = decode_json(load_json($ARGV[0])); + +query_flag($json, $ARGV[1]);
Signature · 1 line
2.26.2
PATCH 3/11 docparse: Add test documentation parser
Quoted 8 lines · Cyril Hrubis
docparse/docparse.c new file +8
-0+const char *next_token (FILE *f,struct data_node *doc) +{ +size_t i =0 ; +static char buf[4096 ]; +int c; +int in_str =0 ; + +for (;;) {
docparse/docparse.c line 142
Quoted 10 lines
docparse/docparse.c new file +10
-0+ c =fgetc (f); + +if (c ==EOF ) + goto exit; + +if (in_str) { +if (c =='"' ) { +if (i ==0 || buf[i-1 ] !='\\' ) + goto exit; + }
There is a problem in handle a special string token here, which can not parse the "" correctly in many test cases.
e.g.
# ./docparse ../testcases/kernel/syscalls/fsopen/fsopen01.c # ./docparse ../testcases/kernel/fs/ftest/ftest02.c ....
We got nothing output from the above two tests parsing because they contains "" in their sentence, it makes next_token() exit too early.
TEST(move_mount(fsmfd, "", AT_FDCWD, MNTPOINT,
MOVE_MOUNT_F_EMPTY_PATH));
Quoted 96 lines
docparse/docparse.c new file +96
-0+ + buf[i++] = c; +continue ; + } + +switch (c) { +case '{' : +case '}' : +case ';' : +case '(' : +case ')' : +case '=' : +case ',' : +case '[' : +case ']' : +if (i) { +ungetc (c, f); + goto exit; + } + + buf[i++]=c; + goto exit; + case '0' ... '9': + case 'a' ... 'z': + case 'A' ... 'Z': +case '.' : +case '_' : +case '-' : + buf[i++]=c; +break ; +case '/' : +maybe_comment (f, doc); +break ; +case '"' : + in_str =1 ; +break ; +case ' ' : +case '\n' : +case '\t' : +if (i) + goto exit; +break ; + } + } + +exit: +if (i ==0 ) +return NULL ; + + buf[i] =0 ; +return buf; +} + +#define WARN (str) fprintf(stderr, str "\n") + +static int parse_array (FILE *f,struct data_node *node) +{ +const char *token; + +for (;;) { +if (!(token =next_token (f,NULL ))) +return 1 ; + +if (!strcmp (token,"{" )) { +struct data_node *ret =data_node_array (); +parse_array (f, ret); + +if (data_node_array_len (ret)) +data_node_array_add (node, ret); +else +data_node_free (ret); + +continue ; + } + +if (!strcmp (token,"}" )) +return 0 ; + +if (!strcmp (token,"," )) +continue ; + +if (!strcmp (token,"NULL" )) +continue ; + +struct data_node *str =data_node_string (token); + +data_node_array_add (node, str); + } + +return 0 ; +} + +static const char *tokens[] = { +"static" , +"struct" , +"tst_test" ,
docparse/docparse.c line 307
Quoted 10 lines
docparse/docparse.c new file +10
-0+"test" , +"=" , + "{", +}; + +static struct data_node *parse_file (const char *fname) +{ +int state =0 , found =0 ; +const char *token; +
Seems we'd better check the fname is valid before opening it.
if (access(fname, F_OK)) {
fprintf(stderr, "file %s is not exist\n", fname);
return NULL;
}docparse/docparse.c line 351
Quoted 8 lines
docparse/docparse.c new file +8
-0+static const char *filter_out[] = { +"test" , +"test_all" , +"setup" , +"cleanup" , +"tcnt" , +"mntpoint" , +"bufs" ,
I guess the "options" should be also filtered out here?
--
Regards,
Li Wang
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linux.it/pipermail/ltp/attachments/20201023/483b507f/attachment-0001.htm>
PATCH 3/11 docparse: Add test documentation parser
Quoted 25 lines · Li Wang
Quoted 8 lines · Cyril Hrubis
docparse/docparse.c new file +8
-0+const char *next_token (FILE *f,struct data_node *doc) +{ +size_t i =0 ; +static char buf[4096 ]; +int c; +int in_str =0 ; + +for (;;) {docparse/docparse.c line 142
Quoted 10 lines
docparse/docparse.c new file +10
-0+ c =fgetc (f); + +if (c ==EOF ) + goto exit; + +if (in_str) { +if (c =='"' ) { +if (i ==0 || buf[i-1 ] !='\\' ) + goto exit; + }There is a problem in handle a special string token here, which can not parse the "" correctly in many test cases.
Quoted 5 lines
e.g.
# ./docparse ../testcases/kernel/syscalls/fsopen/fsopen01.c # ./docparse ../testcases/kernel/fs/ftest/ftest02.c
The ftest02.c has not been converted to new API, please ignore.
To fix this problem I propose to add a simple line as:
@@ -137,8 +138,10 @@ const char *next_token(FILE *f, struct data_node *doc)
if (in_str) {
if (c == '"') {
- if (i == 0 || buf[i-1] != '\\')
+ if (i == 0 || buf[i-1] != '\\') {
+ buf[i++] = c;
goto exit;
+ }
}Quoted 8 lines
....
We got nothing output from the above two tests parsing because they contains "" in their sentence, it makes
next_token()exit too early.TEST(move_mount(fsmfd, "",
AT_FDCWD, MNTPOINT,
MOVE_MOUNT_F_EMPTY_PATH));
Signature · 5 lines
Regards, Li Wang -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.linux.it/pipermail/ltp/attachments/20201023/6111a36b/attachment.htm>
PATCH 4/11 docparse: Add README
In reply to Cyril HrubisFrom: Cyril Hrubis <[email protected]>
* example of C source and JSON
* note about exporting timeouts
--- docparse/README.md | 248 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 248 insertions(+) create mode 100644 docparse/README.md
docparse/README.md new file +248 -0
@@ -0,0 +1,248 @@ +Motivation for metadata exctraction +=================================== + +Exporting documentation +----------------------- + +This allow us to build browseable documentation for the testcases, e.g. a +catalogue of test information that would be searchable etc. At this point there +is a single page generated from the extracted data that tries to outline the +intent. + + +Propagating test requirements +----------------------------- + +Some subtests require differnt hardware resources/software versions/etc. the +test execution framework needs to consume these so that it can locate proper +hardware, install proper software, etc. + +Some examples of requriments are: + +* Test needs at least 1GB of RAM. + +* Test needs a block device at least 512MB in size + +* Test needs a NUMA machine with two memory nodes and at least 300 free pages on each node + +* Test needs i2c eeprom connected on a i2c bus + +* Test needs two serial ports connected via null-modem cable + + +With this information extracted from the tests the testrunner can then map the +requiremnts on the available machines in a lab and select a proper machine for +the particular (sub)set of testcases as well as supply a particular test with +additional information needed for the test, such as address of the i2c device, +paths to the serial devices, etc. In the case of virtual machines the test could +also dynamically prepare the correct environment for the test on demand. + + +Parallel test execution +----------------------- + +An LTP testrun on a modern hardware wastes most of the machine resources +because the testcases are running sequentially. However in order to execute +tests in parallel we need to know which system resources are utilized by a +given test, as obviously we cannot run two tests that monopolize the same +resource. In some cases we would also need to partition the system resource +accordingly, e.g. if we have two memory stress tests running at the same time +we will need to cap each of these tests on half of the available memory, or +make sure that sum of the memory used by these two tests is not greater an +available memory. + +Examples of such tests are: + +* Tests that mess with global system state + - system time (e.g. settimeofday() test and leap second test) + - SysV SHM + - ... + +* Tests that use block device + +* Tests that work with a particular hardware resource + - i2c eeprom test + - serial port tests + - ... + +Exporting test runtime/timeout to the testrunner +------------------------------------------------ + +Currently most of the testrunners usually do not know for how long is the test +supposed to run, this means that we have to guess some upper limit on how long +a test is supposed to run. The value is usually twice of the maximal runtime +for all testcases or whole suite or even larger. This means that we are wasting +time in the case that the test ends up stuck and we could have failed it much +sooner in most of the cases. This becomes quite important for a kernel +regression tests that do crash the host, if the information that the test is +supposed to crash a kernel under a minute is exported to the testrunner we can +reboot the machine much faster in an event of a crash. + +Getting rid of runtest files +---------------------------- + +This would also allow us to get rid of the unflexible and hard to maintain +runtest files. Once this system is in place we will have a list of all tests +along with their respective metadata - which means that we will be able to +generate subsets of the test easily on the fly. + +In order to achieve this we need two things: + +Each test will describe which syscall/functionality it tests in the metadata. +Then we could define groups of tests based on that. I.e. instead of having +syscall runtest file we would ask the testrunner to run all test that have a +defined which syscall they test, or whose filename matches a particular syscall name. + +Secondly we will have to store the test variants in the test metadata instead +of putting them in a file that is unrelated to the test. + +For example: + +* To run CVE related test we would select testcases with CVE tag + +* To run IPC test we will define a list of IPC syscalls and run all syscall + test that are in the list + +* And many more... + + +Implementation +============== + +The docparser is implemented as a minimal C tokenizer that can parse and +extract code comments and C structures. The docparser then runs over all C +sources in the testcases directory and if tst\_test structure is present in the +source it's parsed and the result is included in the resulting metadata. + +During parsing the metadata is stored in a simple key/value storage that more +or less follows C structure layout, i.e. can include hash, array, and string. +Once the parsing is finished the result is filtered so that only interesting +fields of the tst\_test structure are included and then converted into JSON +output. + +This process produces one big JSON file with metadata for all tests, that +is then installed along with the testcases. This would then be used by the +testrunner. + +The test requirements are stored in the tst\_test structure either as +bitflags, integers or arrays of strings: + +```c +struct tst_test test = { + ... + /* tests needs to run with UID=1 */ + .needs_root = 1, + + /* + * Tests needs a block device at least 1024MB in size and also + * mkfs.ext4 installed. + */ + .needs_device = 1, + .dev_min_size = 1024, + .dev_fs_type = ext4, + + /* Indicates that the test is messing with system wall clock */ + .restore_wallclock = 1, + + /* Tests needs uinput either compiled in or loaded as a module */ + .needs_drivers = (const char *[]) { + "uinput", + NULL + }, + + /* Tests needs enabled kernel config flags */ + .needs_kconfigs = (const char *[]) { + "CONFIG_X86_INTEL_UMIP=y", + NULL + }, + + /* Additional array of key value pairs */ + .tags = (const struct tst_tag[]) { + {"linux-git", "43a6684519ab"}, + {"CVE", "2017-2671"}, + {NULL, NULL} + } +}; +``` + +The test documentation is stored in a special comment such as: + +``` +/*\ + * Test description + * + * This is a test description. + * Consisting of several lines. +\*/ +``` + +Which will yield following json output: + +```json + "testcaseXY": { + "needs_root": "1", + "needs_device": "1", + "dev_min_size": "1024", + "dev_fs_type": "ext4", + "restore_wallclock": "1", + "needs_drivers": [ + "uinput", + ], + "needs_kconfigs": [ + "CONFIG_X86_INTEL_UMIP=y", + ], + "tags": [ + [ + "linux-git", + "43a6684519ab" + ], + [ + "CVE", + "2017-2671" + ], + ], + "doc": [ + "Test description", + "", + "This is a test description.", + "Consisting of several lines." + ], + "fname": "testcases/kernel/syscalls/foo/testcaseXY.c" + }, +``` + +The final JSON file is JSON object of test descriptions indexed by a test name +with a header describing the testsuite: + +```json +{ + "testsuite": "Linux Test Project", + "testsuite_short": "LTP", + "url": "https://github.com/linux-test-project/ltp/", + "scm_url_base": "https://github.com/linux-test-project/ltp/tree/master/", + "timeout": 300, + "version": "20200930", + "tests": { + "testcaseXY": { + ... + }, + ... + } +} +``` + +Open Points +=========== + +There are stil some loose ends. Mostly it's not well defined where to put +things and how to format them. + +* Some of the hardware requirements are already listed in the tst\_test. Should + we put all of them there? + +* What would be the format for test documentation and how to store things such + as test variants there? + +So far this proof of concept generates a metadata file. I guess that we need +actual consumers which will help to settle things down, I will try to look into +making use of this in the runltp-ng at least as a reference implementation.
Signature · 1 line
2.26.2
PATCH 4/11 docparse: Add README
----- Original Message -----
Quoted 11 lines
docparse/README.md new file +7
-0+Open Points +=========== + +There are stil some loose ends. Mostly it's not well defined where to put +things and how to format them. + +* Some of the hardware requirements are already listed in the tst\_test.Should
docparse/README.md new file +3
-0+ we put all of them there? + +* What would be the format for test documentation and how to store things
Quoted 2 lines
such
+ as test variants there?
I'm assuming you don't mean ".test_variants" here, but runtest entries using same binary with different parameters. Currently we have a "tag" that can refer to binary+parameters pair. Which is also useful, for skipping tests - e.g. they run long, they are broken, or older kernel is known to crash - we have a list of checks that modify runtest files before actual run to avoid running into known issues (or save time).
Quoted 13 lines
docparse/README.md new file +3
-0+ +So far this proof of concept generates a metadata file. I guess that we need +actual consumers which will help to settle things down, I will try to lookinto
+making use of this in the runltp-ng at least as a reference implementation. --
2.26.2--
Mailing list info: https://lists.linux.it/listinfo/ltp
PATCH 4/11 docparse: Add README
Hi!
Quoted 12 lines
Quoted 12 lines
docparse/README.md new file +7
-0+Open Points +=========== + +There are stil some loose ends. Mostly it's not well defined where to put +things and how to format them. + +* Some of the hardware requirements are already listed in the tst\_test.Should
docparse/README.md new file +3
-0+ we put all of them there? + +* What would be the format for test documentation and how to store thingssuch
Quoted 8 lines
Quoted 1 line
+ as test variants there?
I'm assuming you don't mean ".test_variants" here, but runtest entries using same binary with different parameters. Currently we have a "tag" that can refer to binary+parameters pair. Which is also useful, for skipping tests - e.g. they run long, they are broken, or older kernel is known to crash - we have a list of checks that modify runtest files before actual run to avoid running into known issues (or save time).
Yes I do mean binary parameters here. And yes we will need a way how to express skiplist for this as well.
Signature · 2 lines
Cyril Hrubis [email protected]
PATCH 5/11 syscalls: Add a few documentation comments
In reply to Cyril HrubisFrom: Cyril Hrubis <[email protected]>
So that it shows up in the resulting json file.
--- testcases/kernel/syscalls/abort/abort01.c | 16 ++++++++++------ testcases/kernel/syscalls/accept/accept01.c | 8 +++++--- testcases/kernel/syscalls/accept/accept02.c | 7 +++++-- testcases/kernel/syscalls/acct/acct01.c | 5 +++++ testcases/kernel/syscalls/acct/acct02.c | 6 ++++-- 5 files changed, 29 insertions(+), 13 deletions(-)
testcases/kernel/syscalls/abort/abort01.c +10 -6
@@ -5,14 +5,18 @@* 01/02/2003 Port to LTP [email protected] * 11/11/2002: Ported to LTP Suite by Ananda * 06/30/2001 Port to Linux [email protected] -* -* ALGORITHM -* Fork child. Have child abort, check return status. -* -* RESTRICTIONS -* The ulimit for core file size must be greater than 0. */ +/*\ +* [DESCRIPTION] +* Checks that process which called abort() gets killed by SIGIOT and dumps core. +* +* [ALGORITHM] +* - Fork child. +* - Child calls abort. +* - Parent checks return status. +\*/ +#include <sys/types.h> #include <sys/wait.h> #include <errno.h>
testcases/kernel/syscalls/accept/accept01.c +5 -3
@@ -3,11 +3,13 @@/* * Copyright (c) International Business Machines Corp., 2001 * 07/2001 Ported by Wayne Boyer -* -* Description: -* Verify that accept() returns the proper errno for various failure cases */ +/*\ +* [DESCRIPTION] +* Verify that accept() returns the proper errno for various failure cases. +\*/ +#include <stdio.h> #include <unistd.h> #include <errno.h>
testcases/kernel/syscalls/accept/accept02.c +5 -2
@@ -3,7 +3,10 @@* Copyright (c) 2019 SUSE LLC * Author: Christian Amann <[email protected]> */ -/* Test for CVE-2017-8890 +/*\ +* [DESCRIPTION] +* +* Test for CVE-2017-8890 * * In Kernels up to 4.10.15 missing commit 657831ff the multicast * group information of a socket gets copied over to a newly created
@@ -16,7 +19,7 @@* * For more information about this CVE see: * https://www.suse.com/security/cve/CVE-2017-8890/ -*/ +\*/ #include <errno.h> #include <sys/socket.h>
testcases/kernel/syscalls/acct/acct01.c +5 -0
@@ -7,6 +7,11 @@/* 12/03/2002 Port to LTP [email protected] */ /* 06/30/2001 Port to Linux [email protected] */ +/*\ +* [DOCUMENTATION] +* Verify that acct() returns proper errno on failure. +\*/ +#include <sys/types.h> #include <sys/stat.h> #include <errno.h>
testcases/kernel/syscalls/acct/acct02.c +4 -2
@@ -3,7 +3,9 @@* Copyright (c) SUSE LLC, 2019 * Author: Christian Amann <[email protected]> */ -/* +/*\ +* [DOCUMENTATION] +* * This tests if the kernel writes correct data to the * process accounting file. *
@@ -19,7 +21,7 @@* * This is also accidental regression test for: * 4d9570158b626 kernel/acct.c: fix the acct->needcheck check in check_free_space() -*/ +\*/ #include <sys/stat.h> #include <errno.h>
Signature · 1 line
2.26.2
PATCH 6/11 syscalls: Move needs_drivers inside of the tst_test struct
In reply to Cyril Hrubis--- testcases/kernel/syscalls/fsetxattr/fsetxattr02.c | 10 ++++------ testcases/kernel/syscalls/ioctl/ioctl08.c | 9 ++++----- 2 files changed, 8 insertions(+), 11 deletions(-)
testcases/kernel/syscalls/fsetxattr/fsetxattr02.c +4 -6
@@ -241,11 +241,6 @@ static void cleanup(void) } } -static const char *const needed_drivers[] = { -"brd" , -NULL , -}; -static struct tst_test test = { .setup = setup, .test = verify_fsetxattr,
@@ -254,7 +249,10 @@ static struct tst_test test = { .needs_devfs = 1, .mntpoint = MNTPOINT, .needs_root = 1, - .needs_drivers = needed_drivers, + .needs_drivers = (const char *const []) { +"brd" , +NULL , + }, };#else /* HAVE_SYS_XATTR_H */
testcases/kernel/syscalls/ioctl/ioctl08.c +4 -5
@@ -112,10 +112,6 @@ static void setup(void) sizeof(struct file_dedupe_range_info)); } -static const char *const needed_drivers[] = { -"btrfs" , -NULL , -};static struct tst_test test = { .test = verify_ioctl,
@@ -127,7 +123,10 @@ static struct tst_test test = { .mount_device = 1, .mntpoint = MNTPOINT, .dev_fs_type = "btrfs", - .needs_drivers = needed_drivers, + .needs_drivers = (const char *const []) { +"btrfs" , +NULL , + }, };#else TST_TEST_TCONF(
Signature · 1 line
2.26.2
PATCH 7/11 make: Allow {INSTALL, MAKE}_TARGETS be a directory
In reply to Cyril HrubisFrom: Petr Vorel <[email protected]>
by detecting it and adding required -d parameter for install and installing whole directory with -t.
This will be needed for metadata metadata.chunked target.
--- include/mk/functions.mk | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
include/mk/functions.mk +2 -1
@@ -35,7 +35,8 @@ INSTALL_FILES += $$(abspath $$(DESTDIR)/$(3)/$(1)) $$(abspath $$(DESTDIR)/$(3)/$(1)): \ $$(abspath $$(dir $$(DESTDIR)/$(3)/$(1))) - install -m $$(INSTALL_MODE) "$(2)/$(1)" "$$@" + install -m $$(INSTALL_MODE) $(shell test -d "$(2)/$(1)" && echo "-d") $(PARAM) "$(2)/$(1)" $$@ + $(shell test -d "$(2)/$(1)" && echo "install -m "'$$(INSTALL_MODE) $(PARAM)' "$(2)/$(1)/*" -t '$$@') endef #
Signature · 1 line
2.26.2
PATCH 8/11 make: Allow CLEAN_TARGETS to remove directories
In reply to Cyril HrubisFrom: Petr Vorel <[email protected]>
make clean on leaf target can contain also directories.
Changing delete to recursive delete (-r parameter for rm) should be safe thus detection whether actually requited is not implemented.
This will be needed for metadata metadata.chunked target.
--- include/mk/generic_leaf_target.inc | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-)
include/mk/generic_leaf_target.inc +5 -5
@@ -60,10 +60,10 @@ # $(HOST_MAKE_TARGETS) : Host tools which use $HOSTCC. # # $(CLEAN_TARGETS) : What targets should be cleaned (must be -# real files). This will automatically append -# adds the .o suffix to all files referenced -# by $(MAKE_TARGETS)) to CLEAN_TARGETS, if -# MAKE_TARGETS wasn't defined (see +# real files or directories). This will automatically append +# adds the .o suffix to all files referenced by +# $(MAKE_TARGETS)) to CLEAN_TARGETS, if MAKE_TARGETS wasn't +# defined (see # $(MAKE_TARGETS)). # $(INSTALL_MODE) : What mode should we using when calling # install(1)?
@@ -103,7 +103,7 @@ $(MAKE_TARGETS): | $(MAKE_DEPS) all: $(MAKE_TARGETS) clean:: $(CLEAN_DEPS) - -$(RM) -f $(CLEAN_TARGETS) + -$(RM) -f -r $(CLEAN_TARGETS) $(INSTALL_FILES): | $(INSTALL_DEPS)
Signature · 1 line
2.26.2
PATCH 9/11 travis: Install docparse dependencies
In reply to Cyril HrubisFrom: Petr Vorel <[email protected]>
--- travis/alpine.sh | 3 +++ travis/debian.minimal.sh | 8 +++++++- travis/debian.sh | 9 ++++++++- travis/fedora.sh | 11 ++++++++--- travis/tumbleweed.sh | 8 +++++++- 5 files changed, 33 insertions(+), 6 deletions(-)
travis/alpine.sh +3 -0
@@ -6,6 +6,8 @@ apk update apk add \ acl-dev \ + asciidoc \ + asciidoctor \ autoconf \ automake \ clang \
@@ -23,6 +25,7 @@ apk add \ musl-dev \ numactl-dev \ openssl-dev \ + perl-json \ pkgconfig cat /etc/os-release
travis/debian.minimal.sh +7 -1
@@ -2,7 +2,11 @@ # Copyright (c) 2018-2020 Petr Vorel <[email protected]> set -ex -apt remove -y \ +apt="apt remove -y" + +$apt \ + asciidoc \ + asciidoctor \ libacl1-dev \ libaio-dev \ libaio1 \
@@ -17,3 +21,5 @@ apt remove -y \ libsepol1-dev \ libssl-dev \ libtirpc-dev + +$apt asciidoc-base ruby-asciidoctor || true
travis/debian.sh +8 -1
@@ -8,8 +8,12 @@ grep -v oldstable-updates /etc/apt/sources.list > /tmp/sources.list && mv /tmp/s apt update -apt install -y --no-install-recommends \ +apt="apt install -y --no-install-recommends" + +$apt \ acl-dev \ + asciidoc \ + asciidoctor \ autoconf \ automake \ build-essential \
@@ -26,6 +30,7 @@ apt install -y --no-install-recommends \ libcap2 \ libc6 \ libc6-dev \ + libjson-perl \ libkeyutils-dev \ libkeyutils1 \ libmm-dev \
@@ -40,4 +45,6 @@ apt install -y --no-install-recommends \ lsb-release \ pkg-config +$apt ruby-asciidoctor-pdf || true + df -hT
travis/fedora.sh +8 -3
@@ -2,7 +2,10 @@ # Copyright (c) 2018-2020 Petr Vorel <[email protected]> set -ex -yum -y install \ +yum="yum -y install" + +$yum \ + asciidoc \ autoconf \ automake \ make \
@@ -12,8 +15,10 @@ yum -y install \ findutils \ libtirpc \ libtirpc-devel \ + perl-JSON \ pkg-config \ redhat-lsb-core -# CentOS 8 doesn't have libmnl-devel -yum -y install libmnl-devel || yum -y install libmnl +# CentOS 8 fixes +$yum libmnl-devel || $yum libmnl +$yum rubygem-asciidoctor || true
travis/tumbleweed.sh +7 -1
@@ -2,7 +2,10 @@ # Copyright (c) 2018-2020 Petr Vorel <[email protected]> set -ex -zypper --non-interactive install --force-resolution --no-recommends \ +zyp="zypper --non-interactive install --force-resolution --no-recommends" + +$zyp \ + asciidoc \ autoconf \ automake \ clang \
@@ -23,4 +26,7 @@ zypper --non-interactive install --force-resolution --no-recommends \ libtirpc-devel \ linux-glibc-devel \ lsb-release \ + perl-JSON \ pkg-config + +$zyp ruby2.7-rubygem-asciidoctor || $zyp ruby2.5-rubygem-asciidoctor
Signature · 1 line
2.26.2
PATCH 9/11 travis: Install docparse dependencies
Cyril Hrubis <[email protected]> wrote:
...
Quoted 15 lines
travis/fedora.sh +4
-1@@ -2,7 +2,10 @@ # Copyright (c) 2018-2020 Petr Vorel <[email protected]> set -ex -yum -y install \ +yum="yum -y install" + +$yum \ + asciidoc \ autoconf \ automake \ make \
travis/fedora.sh yum -y install \ · line 18
Quoted 4 lines
travis/fedora.sh +1
-0@@ -12,8 +15,10 @@ yum -y install \ findutils \ libtirpc \ libtirpc-devel \ + perl-JSON \
Package perl-libwww-perl is needed for fedora/centos as well.
Signature · 2 lines
Regards, Li Wang
PATCH 10/11 docparse: Add configure options
In reply to Cyril HrubisFrom: Petr Vorel <[email protected]>
We build HTML and PDF docs with:
- asciidoctor (the default as it produces better output, any version)
- asciidoc (required >= 9)
For asciidoctor we support only native asciidoctor-pdf plugin (we don't support DocBook, Apache FO, or LaTeX intermediary formats).
The goal is to have smart default: try to generate HTML (or also PDF if enabled) docs if suitable generator available.
But allow user to chose:
1) disable whole metadata generation (--disable-metadata)
2) disable metadata HTML generation (--disable-metadata-html)
3) enable metadata PDF generation (--enable-metadata-pdf)
4) generator (--with-metadata-generator=asciidoc|asciidoctor)
Used 2 ax_* (third party) macros from autoconf-archive.
--- Makefile | 6 +- configure.ac | 30 ++++++- include/mk/features.mk.in | 5 ++ m4/ax_compare_version.m4 | 177 +++++++++++++++++++++++++++++++++++++ m4/ax_prog_perl_modules.m4 | 77 ++++++++++++++++ m4/ltp-docparse.m4 | 112 +++++++++++++++++++++++ 6 files changed, 405 insertions(+), 2 deletions(-) create mode 100644 m4/ax_compare_version.m4 create mode 100644 m4/ax_prog_perl_modules.m4 create mode 100644 m4/ltp-docparse.m4
Makefile +5 -1
@@ -62,7 +62,11 @@ $(1):: | $$(abs_top_builddir)/$$(basename $$(subst -,.,$(1))) endif endef -COMMON_TARGETS += testcases tools docparse +COMMON_TARGETS += testcases tools +ifeq ($(WITH_METADATA),yes) +COMMON_TARGETS += docparse +endif + # Don't want to nuke the original files if we're installing in-build-tree. ifneq ($(BUILD_TREE_STATE),$(BUILD_TREE_SRCDIR_INSTALL)) INSTALL_TARGETS += runtest scenario_groups testscripts
configure.ac +29 -1
@@ -189,7 +189,7 @@ AC_CHECK_TYPES([struct xt_entry_match, struct xt_entry_target],,,[ # Tools knobs -# Expect +# Bash AC_ARG_WITH([bash], [AC_HELP_STRING([--with-bash], [have the Bourne Again Shell interpreter])],
@@ -202,6 +202,34 @@ else AC_SUBST([WITH_BASH],["no"]) fi +# metadata +AC_ARG_ENABLE([metadata], + [AC_HELP_STRING([--disable-metadata], + [Disable metadata generation (both HTML and PDF, default no)])], + [], [enable_metadata=yes] +) +AC_ARG_ENABLE([metadata_html], + [AC_HELP_STRING([--disable-metadata-html], + [Disable metadata HTML generation (default no)])], + [], [enable_metadata_html=yes] +) + +AC_ARG_ENABLE([metadata_pdf], + [AC_HELP_STRING([--enable-metadata-pdf], + [Enable metadata PDF generation (default no)])], + [], [enable_metadata_pdf=no] +) + +AC_ARG_WITH([metadata_generator], + [AC_HELP_STRING([--with-metadata-generator=asciidoc|asciidoctor], + [Specify metadata generator to use (default autodetect)])], + [with_metadata_generator=$withval], + [with_metadata_generator=detect] +) + +LTP_DOCPARSE + +# Expect AC_ARG_WITH([expect], [AC_HELP_STRING([--with-expect], [have the Tcl/expect library])],
include/mk/features.mk.in +5 -0
@@ -27,6 +27,11 @@ WITH_PERL := @WITH_PERL@ WITH_PYTHON := @WITH_PYTHON@ +METADATA_GENERATOR := @METADATA_GENERATOR@ +WITH_METADATA := @WITH_METADATA@ +WITH_METADATA_HTML := @WITH_METADATA_HTML@ +WITH_METADATA_PDF := @WITH_METADATA_PDF@ + # Features knobs # Test suite knobs
m4/ax_compare_version.m4 new file +177 -0
177 lines not on this page · raw message
m4/ax_prog_perl_modules.m4 new file +77 -0
77 lines not on this page · raw message
m4/ltp-docparse.m4 new file +112 -0
112 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH 11/11 docparse: Generate html and pdf using asciidoc{, tor}
In reply to Cyril HrubisFrom: Petr Vorel <[email protected]>
Rewrite testinfo.pl to generate *.txt pages in asciidoc format which is then regenerated to html (and pdf if enabled) using asciidoc,{tor}.
Replace getting Linux kernel git commit messages from local git repository (needed after having all tests in single page, because API has access limits; it's also better to generate everything once thus don't depend on network connection).
--- docparse/.gitignore | 5 + docparse/Makefile | 58 +++++++ docparse/testinfo.pl | 406 +++++++++++++++++++++++++++++++++++++++++-- 3 files changed, 458 insertions(+), 11 deletions(-)
docparse/.gitignore +5 -0
@@ -1,2 +1,7 @@ +/*.txt +/docbook-xsl.css /docparse /metadata.json +/metadata.html +/metadata.pdf +/metadata.chunked/
docparse/Makefile +58 -0
77 lines not on this page · raw message
docparse/testinfo.pl +395 -11
434 lines not on this page · raw message
Signature · 1 line
2.26.2
PATCH 11/11 docparse: Generate html and pdf using asciidoc{, tor}
Quoted 24 lines · Cyril Hrubis
docparse/Makefile +13
-0@@ -1,19 +1,77 @@ # SPDX-License-Identifier: GPL-2.0-or-later # Copyright (c) 2019 Cyril Hrubis <[email protected]> +# Copyright (c) 2020 Petr Vorel <[email protected]> top_srcdir ?= .. include $(top_srcdir)/include/mk/env_pre.mk include $(top_srcdir)/include/mk/functions.mk +ifeq ($(METADATA_GENERATOR),asciidoctor) +METADATA_GENERATOR_CMD := asciidoctor +METADATA_GENERATOR_PARAMS := -d book metadata.txt +METADATA_GENERATOR_PARAMS_HTML := -b xhtml +METADATA_GENERATOR_PARAMS_PDF := -b pdf -r asciidoctor-pdf +else ifeq ($(METADATA_GENERATOR),asciidoc) +METADATA_GENERATOR_CMD := a2x +METADATA_GENERATOR_PARAMS := --xsltproc-opts "--stringparam toc.section.depth 1" -d book -L --resource="$(PWD)" metadata.txt +METADATA_GENERATOR_PARAMS_HTML := -f xhtml +METADATA_GENERATOR_PARAMS_PDF := -f pdf +METADATA_GENERATOR_PARAMS_HTML_CHUNKED := -f chunked +else ifeq ($(METADATA_GENERATOR),)
docparse/Makefile line 31
Quoted 10 lines
docparse/Makefile +10
-0+$(error 'METADATA_GENERATOR' not not configured, run ./configure in the root directory) +else +$(error '$(METADATA_GENERATOR)' not supported, only asciidoctor and asciidoc are supported) +endif + +ifdef VERBOSE +METADATA_GENERATOR_PARAMS += -v +endif + +CLEAN_TARGETS := *.txt
I guess the generated *.css *.js files should be deleted as well.
--
Regards,
Li Wang
PATCH 11/11 docparse: Generate html and pdf using asciidoc{, tor}
Hi,
Quoted 6 lines
Review
Quoted 2 lines · Cyril Hrubis
(quoted diff) +2
-0+ +CLEAN_TARGETS := *.txtI guess the generated *.css *.js files should be deleted as well.
+1
Kind regards,
Petr
cover Test metadata extraction
In reply to Cyril HrubisHi!
The applied patchset can also be seen at:
https://github.com/metan-ucw/ltp/tree/master/docparse
Also please ignore the [email protected] address in the sign-offs that should have been [email protected] I will fix that before applying if we decide to go with this version.
Signature · 2 lines
Cyril Hrubis [email protected]
cover Test metadata extraction
In reply to Cyril HrubisHi,
Quoted 9 lines
This patchset adds a test metadata extraction into LTP and also documentation generator that produces browseable HTML documentation from the exported metadata. For detailed description of the idea and implementation see the patch that adds README.md.
While the idea is quite new the code is mature enough to be included in the upstream repository and I'm also worried that we will not get any feedback or users of the metadata unless it's included in the upstream git.
Example of the output:
(generally I prefer asciidoctor, but PDF support is not always available) HTML could have custom javascript (probably jquery) based search/filtering.
= asciidoctor
* PDF
https://pevik.github.io/asciidoctor/metadata.pdf
There is missing TOC (as :toc: does not work and adding '-a toc' to generate as:
asciidoctor -d book metadata.txt -b pdf -r asciidoctor-pdf -a toc
puts TOC at the top.
* HTML single page
https://pevik.github.io/asciidoctor/metadata.html
= asciidoc
* PDF
https://pevik.github.io/asciidoc/metadata.pdf
There is ugly revision history at the top.
* HTML single page
https://pevik.github.io/asciidoc/metadata.html
* HTML chunked
https://pevik.github.io/asciidoc/metadata.chunked/index.html
https://pevik.github.io/asciidoc/metadata.chunked/ch01.html
https://pevik.github.io/asciidoc/metadata.chunked/ch02.html
https://pevik.github.io/asciidoc/metadata.chunked/ch03.html
Quoted 3 lines
The next step is to use the extracted metadata in runltp-ng in the proof of concept parallel executor that has been written by Ritchie and posted to this mailing list as well.
Kind regards,
Petr