[LTP] Add option to skip tests

Paul Larson

author

2009-08-12 21:50 UTC #0 lore raw

One problem that LTP has long suffered is that certain tests are known to be broken, sometimes for long periods of time. In an ideal world, I think it best that these should be skipped, or at least removed from the default list of tests to run, but this isn't always practical due to a number of reasons. Of course, another option is that a 'whitelist' of tests to run can be created and passed to runltp, but this becomes hard to maintain as there are often new tests that you may want to keep up with, and with each release of LTP, the list would have to be reexamined.

So I wanted to see what people thought about a simple option, -S, to specify a list of tests that you want to blacklist from the current run.
This is handy if you have an automated testing environment with an
architecture that doesn't always get great attention and a list of tests that you know to be broken on that arch.

The skipfile is simply a list of test names, one per line.

Thanks,
Paul Larson
---

Add a -S option to runltp that allows the user to specify a SKIPFILE of tests to skip.

Signed-off-by Paul Larson <[email protected]>

---

runltp +12 -1

index c26f4ab..746a1ec 100755
@@ -141,6 +141,7 @@ usage()
     -q              Print less verbose output to screen.
     -r LTPROOT      Fully qualified path where testsuite is installed.
     -s PATTERN      Only run test cases which match PATTERN.
+    -S SKIPFILE     Skip tests specified in SKIPFILE
     -t DURATION     Execute the testsuite for given duration. Examples:
                       -t 60s = 60 seconds
                       -t 45m = 45 minutes

@@ -193,7 +194,7 @@ main()
     local DEFAULT_FILE_NAME_GENERATION_TIME=`date +"%Y_%b_%d-%Hh_%Mm_%Ss"`
     version_date=`head -n 1 $LTPROOT/ChangeLog`
 
-    while getopts a:c:C:d:D:f:ehi:g:l:m:Nno:pqr:s:t:T:vw:x:b:B: arg
+    while getopts a:c:C:d:D:f:ehi:g:l:m:Nno:pqr:s:S:t:T:vw:x:b:B: arg
     do  case $arg in
         a)  EMAIL_TO=$OPTARG
             ALT_EMAIL_OUT=1;;

@@ -353,6 +354,8 @@ main()
         r)  LTPROOT=$OPTARG;;
     
         s)  TAG_RESTRICT_STRING=$OPTARG;;
+
+	S)  SKIPFILE=$OPTARG;;
     
         t)  # In case you want to specify the time 
             # to run from the command line

@@ -630,6 +633,14 @@ main()
         mv -f ${TMP}/alltests ${TMP}/alltests.orig
 	    grep $TAG_RESTRICT_STRING ${TMP}/alltests.orig > ${TMP}/alltests #Not worth checking return codes for this case
     fi
+
+    # Blacklist or skip tests if a SKIPFILE was specified with -S
+    if [ -n "$SKIPFILE" ]
+    then
+        for file in $( cat $SKIPFILE ); do
+            sed -i "/$file/d" ${TMP}/alltests
+        done
+    fi
     
     # check for required users and groups
     ${LTPROOT}/IDcheck.sh &>/dev/null || \

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

Subrata Modak

2009-08-13 17:18 UTC #1 lore raw

Hi Paul,

Quoted 9 lines · Paul Larson

One problem that LTP has long suffered is that certain tests are known to be broken, sometimes for long periods of time. In an ideal world, I think it best that these should be skipped, or at least removed from the default list of tests to run, but this isn't always practical due to a number of reasons. Of course, another option is that a 'whitelist' of tests to run can be created and passed to runltp, but this becomes hard to maintain as there are often new tests that you may want to keep up with, and with each release of LTP, the list would have to be reexamined.

Quoted 5 lines

So I wanted to see what people thought about a simple option, -S, to specify a list of tests that you want to blacklist from the current run.
This is handy if you have an automated testing environment with an
architecture that doesn't always get great attention and a list of tests that you know to be broken on that arch.

This indeed is a good approach. Each test engineer can maintain a list of tests in afile which he is not interested to run or does not suit his/her test environment. Then doing

./runltp -S <filename>

he can just skip to run these subset of test.

However, i am not able to achieve the same results post application of this patch. I created my own version of runtest/math and runtest/math.x. I want runltp to skip all tests mentioned mentioned in math.x and run the remaining mentioned in math. However when i run the following i see runltp still runs all the tests inside runtest/math:

$ cat runtest/math.x 
abs01 abs01
atof01 atof01
fptest01 fptest01
fptest02 fptest02

$ cat runtest/math
abs01 abs01
atof01 atof01
fptest01 fptest01
fptest02 fptest02
nextafter01 nextafter01

$./runltp -S math.x -f math

<<<test_start>>>
tag=abs01 stime=1250180349
cmdline="abs01"
contacts=""
analysis=exit
<<<test_output>>>

abs01       1  TPASS  :  Test passed
abs01       2  TPASS  :  Test passed
abs01       3  TPASS  :  Test passed

<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=atof01 stime=1250180349
cmdline="atof01"
contacts=""
analysis=exit
<<<test_output>>>

atof01      1  TPASS  :  Test passed
atof01      2  TPASS  :  Test passed
atof01      3  TPASS  :  Test passed
atof01      4  TPASS  :  Test passed

<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=fptest01 stime=1250180349
cmdline="fptest01"
contacts=""
analysis=exit
<<<test_output>>>
fptest01 1 TPASS : PASS
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=fptest02 stime=1250180349
cmdline="fptest02"
contacts=""
analysis=exit
<<<test_output>>>
fptest02 1 TPASS : PASS
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no cutime=1 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=nextafter01 stime=1250180349 cmdline="nextafter01"
contacts=""
analysis=exit
<<<test_output>>>

nextafter01    1  TPASS  :  Test passed
nextafter01    2  TPASS  :  Test passed
nextafter01    3  TPASS  :  Test passed

incrementing stop
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=0
<<<test_end>>>

I thought that math.x should just have the name of the test not to be run. So, i just retained only the tag_names, and removed anything else, but still i get the following output:

$ cat runtest/math.x.2 
abs01 
atof01
fptest01
fptest02

$ ./runltp -S math.x.2 -f math

<<<test_start>>>
tag=abs01 stime=1250180546
cmdline="abs01"
contacts=""
analysis=exit
<<<test_output>>>

abs01       1  TPASS  :  Test passed
abs01       2  TPASS  :  Test passed
abs01       3  TPASS  :  Test passed

<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=atof01 stime=1250180546
cmdline="atof01"
contacts=""
analysis=exit
<<<test_output>>>

atof01      1  TPASS  :  Test passed
atof01      2  TPASS  :  Test passed
atof01      3  TPASS  :  Test passed
atof01      4  TPASS  :  Test passed

<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=fptest01 stime=1250180546
cmdline="fptest01"
contacts=""
analysis=exit
<<<test_output>>>
fptest01 1 TPASS : PASS
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=fptest02 stime=1250180546
cmdline="fptest02"
contacts=""
analysis=exit
<<<test_output>>>
fptest02 1 TPASS : PASS
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=0
<<<test_end>>>
<<<test_start>>>
tag=nextafter01 stime=1250180546 cmdline="nextafter01"
contacts=""
analysis=exit
<<<test_output>>>

nextafter01    1  TPASS  :  Test passed
nextafter01    2  TPASS  :  Test passed
nextafter01    3  TPASS  :  Test passed

incrementing stop
<<<execution_status>>>
initiation_status="ok"
duration=0 termination_type=exited termination_id=0 corefile=no cutime=0 cstime=0
<<<test_end>>>

Is there something which i am missing out ?

Regards--
Subrata

Quoted 10 lines

The skipfile is simply a list of test names, one per line.
Thanks,
Paul Larson
---
Add a -S option to runltp that allows the user to specify a SKIPFILE of tests to skip.

Quoted 8 lines

Signed-off-by Paul Larson <[email protected]>
---
------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day
trial. Simplify your report design, integration and deployment - and focus on
what you do best, core application coding. Discover what's new with
Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________ Ltp-list mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/ltp-list

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

Paul Larson

author

2009-08-13 17:54 UTC #2 lore raw

You were not providing the full path. Here's a new version that should let you either provide absolute or relative path (including no patch assuming the file is in LTPROOT). See if this works better for you.

Add a -S option to runltp that allows the user to specify a SKIPFILE of tests to skip.

Signed-off-by Paul Larson <[email protected]>

---

runltp +16 -1

index c26f4ab..15da5ab 100755
@@ -141,6 +141,7 @@ usage()
     -q              Print less verbose output to screen.
     -r LTPROOT      Fully qualified path where testsuite is installed.
     -s PATTERN      Only run test cases which match PATTERN.
+    -S SKIPFILE     Skip tests specified in SKIPFILE
     -t DURATION     Execute the testsuite for given duration. Examples:
                       -t 60s = 60 seconds
                       -t 45m = 45 minutes

@@ -193,7 +194,7 @@ main()
     local DEFAULT_FILE_NAME_GENERATION_TIME=`date +"%Y_%b_%d-%Hh_%Mm_%Ss"`
     version_date=`head -n 1 $LTPROOT/ChangeLog`
 
-    while getopts a:c:C:d:D:f:ehi:g:l:m:Nno:pqr:s:t:T:vw:x:b:B: arg
+    while getopts a:c:C:d:D:f:ehi:g:l:m:Nno:pqr:s:S:t:T:vw:x:b:B: arg
     do  case $arg in
         a)  EMAIL_TO=$OPTARG
             ALT_EMAIL_OUT=1;;

@@ -353,6 +354,12 @@ main()
         r)  LTPROOT=$OPTARG;;
     
         s)  TAG_RESTRICT_STRING=$OPTARG;;
+
+	S)  case $OPTARG in
+            /*)
+                SKIPFILE=$OPTARG;;
+            *)
+                SKIPFILE="$LTPROOT/$OPTARG";;
     
         t)  # In case you want to specify the time 
             # to run from the command line

@@ -630,6 +637,14 @@ main()
         mv -f ${TMP}/alltests ${TMP}/alltests.orig
 	    grep $TAG_RESTRICT_STRING ${TMP}/alltests.orig > ${TMP}/alltests #Not worth checking return codes for this case
     fi
+
+    # Blacklist or skip tests if a SKIPFILE was specified with -S
+    if [ -n "$SKIPFILE" ]
+    then
+        for file in $( cat $SKIPFILE ); do
+            sed -i "/$file/d" ${TMP}/alltests
+        done
+    fi
     
     # check for required users and groups
     ${LTPROOT}/IDcheck.sh &>/dev/null || \

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july

_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

Garrett Cooper

2009-08-13 22:40 UTC #3 lore raw

Quoted 9 lines · Paul Larson

You were not providing the full path. Here's a new version that should let you either provide absolute or relative path (including no patch assuming the file is in LTPROOT). See if this works better for you.
Add a -S option to runltp that allows the user to specify a SKIPFILE of tests to skip.
Signed-off-by Paul Larson <[email protected]>

Cool innovation Paul! I see potential for this providing us with
the shims necessary to automatically generate which testcases need to be skipped after running autoconf and/or the compile end :). Also, I'd say that if the basename is the same as the specified skipfile, and the skipfile isn't in the base directory, then it should be assumed to be in a predefined directory. Otherwise, it's not in the default directory. That way we don't invite any confusing defaults... Thanks,
-Garrett

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list

Subrata Modak

2009-08-14 17:24 UTC #4 lore raw

In reply to Paul Larson

Wow !! Thatś great and it works fine now.

Quoted 54 lines · Paul Larson

You were not providing the full path. Here's a new version that should
let you either provide absolute or relative path (including no patch assuming the file is in LTPROOT). See if this works better for you.
Add a -S option to runltp that allows the user to specify a SKIPFILE of
tests to skip.
Signed-off-by Paul Larson <[email protected]>
---
differences
between files
attachment
(runltp-skipfile-2.patch)
runltp +0 -0
index c26f4ab..15da5ab 100755
@@ -141,6 +141,7 @@ usage()
     -q              Print less verbose output to screen.
     -r LTPROOT      Fully qualified path where testsuite is
installed.
     -s PATTERN      Only run test cases which match PATTERN.
+    -S SKIPFILE     Skip tests specified in SKIPFILE
     -t DURATION     Execute the testsuite for given duration.
Examples:
                       -t 60s = 60 seconds
                       -t 45m = 45 minutes
runltp +0 -0
@@ -193,7 +194,7 @@ main()
     local DEFAULT_FILE_NAME_GENERATION_TIME=`date +"%Y_%b_%d-%Hh_%Mm_
%Ss"`
version_date=head -n 1 $LTPROOT/ChangeLog
runltp +1 -1
-    while getopts a:c:C:d:D:f:ehi:g:l:m:Nno:pqr:s:t:T:vw:x:b:B: arg
+    while getopts a:c:C:d:D:f:ehi:g:l:m:Nno:pqr:s:S:t:T:vw:x:b:B: arg
     do  case $arg in
         a)  EMAIL_TO=$OPTARG
             ALT_EMAIL_OUT=1;;

runltp main() · line 362

Quoted 9 lines

runltp +6 -0

@@ -353,6 +354,12 @@ main()
         r)  LTPROOT=$OPTARG;;
 
         s)  TAG_RESTRICT_STRING=$OPTARG;;
+
+       S)  case $OPTARG in
+            /*)
+                SKIPFILE=$OPTARG;;
+            *)
+                SKIPFILE="$LTPROOT/$OPTARG";;

We just needed an:
+ esac ;;
here.

Anyways this option will be used in LTP from now onwards.

Regards--
Subrata

Quoted 10 lines

runltp +0 -0

         t)  # In case you want to specify the time 
             # to run from the command line

@@ -630,6 +637,14 @@ main()
         mv -f ${TMP}/alltests ${TMP}/alltests.orig
            grep $TAG_RESTRICT_STRING ${TMP}/alltests.orig >

${TMP}/alltests #Not worth checking return codes for this case
fi

runltp +1 -0

runltp main() · line 650

Quoted 10 lines

runltp +7 -0

+    # Blacklist or skip tests if a SKIPFILE was specified with -S
+    if [ -n "$SKIPFILE" ]
+    then
+        for file in $( cat $SKIPFILE ); do
+            sed -i "/$file/d" ${TMP}/alltests
+        done
+    fi
 
     # check for required users and groups
     ${LTPROOT}/IDcheck.sh &>/dev/null || \

------------------------------------------------------------------------------
Let Crystal Reports handle the reporting - Free Crystal Reports 2008 30-Day trial. Simplify your report design, integration and deployment - and focus on what you do best, core application coding. Discover what's new with Crystal Reports now. http://p.sf.net/sfu/bobj-july
_______________________________________________
Ltp-list mailing list
[email protected]
https://lists.sourceforge.net/lists/listinfo/ltp-list