Skip to contentSkip to conversation

valgrind: integration of regression tests to ptest

PATCH New Dave Lerner 1 participant 2 messages Feb 15, 2014

lore
Share & tools
lore
#0 lore raw

v2 cover valgrind: integration of regression tests to ptest

Hello Saul,

V2 patch includes following changes:
1) no PR bump in recipe
2) lower-cased commit header

Regards,
Dave Lerner

--------------
The attached patch adds PTEST integration to valgrind. The patch was built and tested as shown in the "CONFIGURATION", "BUILD TESTS", and "RUNTIME TESTS" sections below. A summary of changes is shown in "COMMIT STAT".

Thanks,
Dave Lerner

COMMIT STAT
===========
commit 2e539fd064c4ef91a1b9593edfadffffe0f628d1

Author: Dave Lerner <[email protected]>
Date:   Mon Jan 27 11:46:19 2014 -0600

valgrind: integration of regression tests to ptest

Modifies valgrind's regression test framework to be compatible with the yocto PTEST framework as follows:
* existing recipe valgrind*bb adds new methods: do_compile_ptest and do_install_ptest.
* new file run-ptest adds the wrapper interface to the valgrind regression test script vg_regtest.
* existing valgrind regression test script 'vg_regtest' changes to report the status of the valgrind component tests in the format that PTEST expects, instead of the valgrind formats, but only when vg_regtest is invoked with an optional --yocto-ptest argument
* four new patches disable building tests that don't compile with the yocto compiler and default options. See the patches for details.

 .../valgrind/valgrind/add-ptest.patch              |  205 ++++++++++++++++++++
 .../valgrind/valgrind/force-nostabs.patch          |   42 ++++
 .../valgrind/remove-arm-variant-specific.patch     |   67 +++++++
 .../valgrind/remove-ppc-tests-failing-build.patch  |   75 +++++++
 meta/recipes-devtools/valgrind/valgrind/run-ptest  |   11 ++
 meta/recipes-devtools/valgrind/valgrind_3.9.0.bb   |   56 +++++-
 6 files changed, 455 insertions(+), 1 deletion(-)

CONFIGURATION
=============
Appended to each conf/local.conf ...
MACHINE = 'some-machine-name'
SSTATE_DIR = "/ord-dlerner-l81/dlerner/poky_16/sstate-cache" DISTRO_FEATURES_append = " ptest" EXTRA_IMAGE_FEATURES = "debug-tweaks dbg-pkgs ptest-pkgs" IMAGE_INSTALL_append = " valgrind ptest-runner"

BUILD TESTS
===========
Configuration fails with reports that valgrind is not supported for:
qemuarm, qemumips*, routerstationpro
ERROR: Nothing RPROVIDES 'valgrind' (but ...core-image-basic.bb
RDEPENDS on or otherwise requires it)
ERROR: valgrind was skipped: incompatible with host
mips-poky-linux (not in COMPATIBLE_HOST)
NOTE: Runtime target 'valgrind' is unbuildable, removing...

Configuration and 'bitbake core-image-basic' passes for:

    beagleboard, mpc8316-rdb, qemuppc, qemux86, qemux86-64

RUNTIME TESTS
=============
The tests are divided into per tool tests and then a common set of tests across all tools in the directory 'none'.

The output per PTEST documentation looks like the following snippet:
PASS: cachegrind/tests/chdir
PASS: cachegrind/tests/clreq
PASS: cachegrind/tests/dlclose
PASS: cachegrind/tests/notpower2
FAIL: cachegrind/tests/wrap5
PASS: callgrind/tests/clreq

Below is the output summarized for two bsps. Note that this is not the output produced by this commit.

qemux86-64
----------
cachegrind:
PASS count=4 FAIL count=1 SKIP count=0
callgrind:
PASS count=13 FAIL count=0 SKIP count=0
drd:
PASS count=106 FAIL count=10 SKIP count=7
helgrind:
PASS count=20 FAIL count=31 SKIP count=1
memcheck:
PASS count=116 FAIL count=78 SKIP count=4
massif:
PASS count=9 FAIL count=26 SKIP count=0
none:
PASS count=101 FAIL count=4 SKIP count=16

qemux86
------
cachegrind:
PASS count=5 FAIL count=1 SKIP count=0
callgrind:
PASS count=13 FAIL count=0 SKIP count=0
drd:
PASS count=106 FAIL count=11 SKIP count=6
helgrind:
PASS count=20 FAIL count=31 SKIP count=1
memcheck:
PASS count=126 FAIL count=78 SKIP count=2
massif:
PASS count=9 FAIL count=26 SKIP count=0
none:
PASS count=70 FAIL count=3 SKIP count=1

qemuppc
-------
All of the tests fail for the valgrind tools cachegrind, drd, helgrind, massif, memcheck, that are not skipped due to preconditions failing. The failures of any test match the failure message for each valgrind tool run agains the program /bin/ls.

#1 lore raw

PATCH v2 1/1 valgrind: integration of regression tests to ptest

Modifies valgrind's regression test framework to be compatible with the yocto PTEST framework as follows:
* existing recipe valgrind*bb adds new methods: do_compile_ptest and do_install_ptest.
* new file run-ptest adds the wrapper interface to the valgrind regression test script vg_regtest.
* existing valgrind regression test script 'vg_regtest' changes to report the status of the valgrind component tests in the format that PTEST expects, instead of the valgrind formats, but only when vg_regtest is invoked with an optional --yocto-ptest argument
* four new patches disable building tests that don't compile with the yocto compiler and default options. See the patches for details.

---
 .../valgrind/valgrind/add-ptest.patch              |  205 ++++++++++++++++++++
 .../valgrind/valgrind/force-nostabs.patch          |   42 ++++
 .../valgrind/remove-arm-variant-specific.patch     |   67 +++++++
 .../valgrind/remove-ppc-tests-failing-build.patch  |   75 +++++++
 meta/recipes-devtools/valgrind/valgrind/run-ptest  |   11 ++
 meta/recipes-devtools/valgrind/valgrind_3.9.0.bb   |   56 +++++-
 6 files changed, 455 insertions(+), 1 deletion(-)
 create mode 100644 meta/recipes-devtools/valgrind/valgrind/add-ptest.patch
 create mode 100644 meta/recipes-devtools/valgrind/valgrind/force-nostabs.patch
 create mode 100644 meta/recipes-devtools/valgrind/valgrind/remove-arm-variant-specific.patch
 create mode 100644 meta/recipes-devtools/valgrind/valgrind/remove-ppc-tests-failing-build.patch
 create mode 100755 meta/recipes-devtools/valgrind/valgrind/run-ptest
meta/recipes-devtools/valgrind/valgrind/add-ptest.patch new file +205 -0
new file mode 100644
index 0000000..fdc9cc0
@@ -0,0 +1,205 @@
+Modify vg_test wrapper to support PTEST formats
+
+This commit changes the valgrind regression test script vg_regtest to
+support the yocto ptest stdout reporting format.  The commit adds
+'--yocto-ptest' as an optional argument to vg_regtest, which alters
+the output to use the ptest infrastructure reporting format:
+    "[PASS|SKIP|FAIL]: testname"
+instead of valgrind's internal test reporting format.  Without the added
+option, --yocto-ptest, the valgrind regression test output is unchanged.
+
+Upstream-Status: Pending
+
+Signed-off-by: Dave Lerner <[email protected]>
+
+diff --git a/tests/vg_regtest.in b/tests/vg_regtest.in
+index 224385f..dbbd23d 100755
+--- a/tests/vg_regtest.in
++++ b/tests/vg_regtest.in
+@@ -39,11 +39,11 @@
+ #               --valgrind.)
+ #   --keep-unfiltered: keep a copy of the unfiltered output/error output
+ #     of each test by adding an extension .unfiltered.out
+-#
+ #   --outer-valgrind: run this valgrind under the given outer valgrind.
+ #     This valgrind must be configured with --enable-inner.
+ #   --outer-tool: tool to use by the outer valgrind (default memcheck).
+ #   --outer-args: use this as outer tool args.
++#   --yocto-ptest: output in yocto ptest format
+ #
+ # The easiest way is to run all tests in valgrind/ with (assuming you installed
+ # in $PREFIX):
+@@ -126,7 +126,7 @@ use strict;
+ my $usage="\n"
+      . "Usage:\n"
+      . "   vg_regtest [--all, --valgrind, --valgrind-lib, --keep-unfiltered\n"
+-     . "                 --outer-valgrind, --outer-tool, --outer-args]\n"
++     . "                 --outer-valgrind, --outer-tool, --outer-args, --yocto-ptest]\n"
+      . "   Use EXTRA_REGTEST_OPTS to supply extra args for all tests\n"
+      . "\n";
+ 
+@@ -170,6 +170,7 @@ my $outer_args;
+ 
+ my $valgrind_lib = "$tests_dir/.in_place";
+ my $keepunfiltered = 0;
++my $yoctoptest = 0;
+ 
+ # default filter is the one named "filter_stderr" in the test's directory
+ my $default_stderr_filter = "filter_stderr";
+@@ -226,6 +227,8 @@ sub process_command_line()
+                 $valgrind_lib = $1;
+             } elsif ($arg =~ /^--keep-unfiltered$/) {
+                 $keepunfiltered = 1;
++            } elsif ($arg =~ /^--yocto-ptest$/) {
++                $yoctoptest = 1;
+             } else {
+                 die $usage;
+             }
+@@ -394,19 +397,21 @@ sub do_diffs($$$$)
+                 # A match;  remove .out and any previously created .diff files.
+                 unlink("$name.$mid.out");
+                 unlink(<$name.$mid.diff*>);
+-                return;
++                return 0;
+             }
+         }
+     }
+     # If we reach here, none of the .exp files matched.
+-    print "*** $name failed ($mid) ***\n";
++    print "*** $name failed ($mid) ***\n" if ($yoctoptest == 0) ;
+     push(@failures, sprintf("%-40s ($mid)", "$fullname"));
+     $num_failures{$mid}++;
++    return 1;
+ }
+ 
+ sub do_one_test($$) 
+ {
+     my ($dir, $vgtest) = @_;
++    my $diffStatus = 0;
+     $vgtest =~ /^(.*)\.vgtest/;
+     my $name = $1;
+     my $fullname = "$dir/$name"; 
+@@ -425,7 +430,11 @@ sub do_one_test($$)
+         } elsif (256 == $prereq_res) {
+             # Nb: weird Perl-ism -- exit code of '1' is seen by Perl as 256...
+             # Prereq failed, skip.
+-            printf("%-16s (skipping, prereq failed: $prereq)\n", "$name:");
++            if ($yoctoptest == 0) {
++                printf("%-16s (skipping, prereq failed: $prereq)\n", "$name:");
++            } else {
++                printf("SKIP: $fullname\n");
++            }
+             return;
+         } else {
+             # Bad prereq; abort.
+@@ -438,7 +447,7 @@ sub do_one_test($$)
+     if (defined $progB) {
+         # If there is a progB, let's start it in background:
+         printf("%-16s valgrind $extraopts $vgopts $prog $args (progB: $progB $argsB)\n",
+-               "$name:");
++               "$name:") if ($yoctoptest == 0);
+         # progB.done used to detect child has finished. See below.
+         # Note: redirection of stdout and stderr is before $progB to allow argsB
+         # to e.g. redirect stdoutB to stderrB
+@@ -452,7 +461,8 @@ sub do_one_test($$)
+                      . "touch progB.done)  &");
+         }
+     } else {
+-        printf("%-16s valgrind $extraopts $vgopts $prog $args\n", "$name:");
++        printf("%-16s valgrind $extraopts $vgopts $prog $args\n", "$name:")
++            if ($yoctoptest == 0);
+     }
+  
+     # Pass the appropriate --tool option for the directory (can be overridden
+@@ -487,7 +497,7 @@ sub do_one_test($$)
+     # Find all the .stdout.exp files.  If none, use /dev/null.
+     my @stdout_exps = <$name.stdout.exp*>;
+     @stdout_exps = ( "/dev/null" ) if (0 == scalar @stdout_exps);
+-    do_diffs($fullname, $name, "stdout", \@stdout_exps); 
++    $diffStatus |= do_diffs($fullname, $name, "stdout", \@stdout_exps);
+ 
+     # Filter stderr
+     $stderr_filter_args = $name if (! defined $stderr_filter_args);
+@@ -496,7 +506,7 @@ sub do_one_test($$)
+     # Find all the .stderr.exp files.  At least one must exist.
+     my @stderr_exps = <$name.stderr.exp*>;
+     (0 != scalar @stderr_exps) or die "Could not find `$name.stderr.exp*'\n";
+-    do_diffs($fullname, $name, "stderr", \@stderr_exps); 
++    $diffStatus |= do_diffs($fullname, $name, "stderr", \@stderr_exps);
+ 
+     if (defined $progB) {
+         # wait for the child to be finished
+@@ -520,7 +530,7 @@ sub do_one_test($$)
+         # Find all the .stdoutB.exp files.  If none, use /dev/null.
+         my @stdoutB_exps = <$name.stdoutB.exp*>;
+         @stdoutB_exps = ( "/dev/null" ) if (0 == scalar @stdoutB_exps);
+-        do_diffs($fullname, $name, "stdoutB", \@stdoutB_exps); 
++        $diffStatus |= do_diffs($fullname, $name, "stdoutB", \@stdoutB_exps);
+         
+         # Filter stderr
+         $stderrB_filter_args = $name if (! defined $stderrB_filter_args);
+@@ -529,7 +539,7 @@ sub do_one_test($$)
+         # Find all the .stderrB.exp files.  At least one must exist.
+         my @stderrB_exps = <$name.stderrB.exp*>;
+         (0 != scalar @stderrB_exps) or die "Could not find `$name.stderrB.exp*'\n";
+-        do_diffs($fullname, $name, "stderrB", \@stderrB_exps); 
++        $diffStatus |= do_diffs($fullname, $name, "stderrB", \@stderrB_exps);
+     }
+ 
+     # Maybe do post-test check
+@@ -541,7 +551,7 @@ sub do_one_test($$)
+ 	    # Find all the .post.exp files.  If none, use /dev/null.
+ 	    my @post_exps = <$name.post.exp*>;
+ 	    @post_exps = ( "/dev/null" ) if (0 == scalar @post_exps);
+-	    do_diffs($fullname, $name, "post", \@post_exps);
++	    $diffStatus |= do_diffs($fullname, $name, "post", \@post_exps);
+ 	}
+     }
+  
+@@ -550,6 +560,13 @@ sub do_one_test($$)
+             print("(cleanup operation failed: $cleanup)\n");
+     }
+ 
++    if ($yoctoptest == 1) {
++        if ($diffStatus == 0) {
++            print("PASS: $fullname\n");
++        } else {
++            print("FAIL: $fullname\n");
++        }
++    }
+     $num_tests_done++;
+ }
+ 
+@@ -589,7 +606,7 @@ sub test_one_dir($$)
+     my $found_tests = (0 != (grep { $_ =~ /\.vgtest$/ } @fs));
+ 
+     if ($found_tests) {
+-        print "-- Running  tests in $full_dir $dashes\n";
++        print "-- Running  tests in $full_dir $dashes\n" if ($yoctoptest == 0);
+     }
+     foreach my $f (@fs) {
+         if (-d $f) {
+@@ -599,7 +616,7 @@ sub test_one_dir($$)
+         }
+     }
+     if ($found_tests) {
+-        print "-- Finished tests in $full_dir $dashes\n";
++        print "-- Finished tests in $full_dir $dashes\n" if ($yoctoptest == 0);
+     }
+ 
+     chdir("..");
+@@ -625,10 +642,12 @@ sub summarise_results
+            $num_failures{"stdout"},   plural($num_failures{"stdout"}),
+            $num_failures{"stderrB"},  plural($num_failures{"stderrB"}),
+            $num_failures{"stdoutB"},  plural($num_failures{"stdoutB"}),
+-           $num_failures{"post"},     plural($num_failures{"post"}));
++           $num_failures{"post"},     plural($num_failures{"post"}))
++               if ($yoctoptest == 0);
+ 
+     foreach my $failure (@failures) {
+-        print "$failure\n";
++        print "$failure\n"
++           if ($yoctoptest == 0);
+     }
+     print "\n";
+ }
meta/recipes-devtools/valgrind/valgrind/force-nostabs.patch new file +42 -0

42 lines not on this page · raw message

meta/recipes-devtools/valgrind/valgrind/remove-arm-variant-specific.patch new file +67 -0

67 lines not on this page · raw message

meta/recipes-devtools/valgrind/valgrind/remove-ppc-tests-failing-build.patch new file +75 -0

75 lines not on this page · raw message

meta/recipes-devtools/valgrind/valgrind/run-ptest new file +11 -0
new file mode 100755
index 0000000..7d0584a
@@ -0,0 +1,11 @@
+#!/bin/bash
+# run-ptest - 'ptest' test infrastructure shell script that
+#   wraps the valgrind regression script vg_regtest. 
+#   Must be run in the /usr/lib/valgrind/ptest directory. 
+#
+# Dave Lerner <[email protected]>
+###############################################################
+VALGRINDLIB=@libdir@/valgrind
+tests/vg_regtest --all \
+    --valgrind=/usr/bin/valgrind --valgrind-lib=$VALGRINDLIB \
+	--yocto-ptest
meta/recipes-devtools/valgrind/valgrind_3.9.0.bb +55 -1

71 lines not on this page · raw message

Signature · 1 line
1.7.9.5