runltp: add -R option for randomize test order

PATCH Superseded Zorro Lang 2 participants 3 messages Sep 8, 2016 – Sep 21, 2016

Share & tools

Zorro Lang

author

2016-09-08 14:21 UTC #0 lore raw

PATCH v2 1/1 runltp: add -R option for randomize test order

Generally we write test entries into runtests/${testfile}, then runltp reads and runs lines sequentially everytime, nearly no one will change the test order.

Recently we find bugs by change the test orderï¼Œdifferent test combination maybe find more bugs. So I add this new option -R to randomize test order.

Signed-off-by Zorro Lang <[email protected]>

---

Hi,

V2 use 'sort -R' to instead of 'randlines.awk'.

Thanks,

 runltp | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

runltp +13 -1

index bb2e5d8..960b2c5 100755
@@ -64,6 +64,8 @@
 #               - Added capability for default Log file generation
 #		Aug 17 2009 - Modified - Subrata Modak
 #		- Added Fault Injection generation Capability through -F Option
+#		Sep 08 2016 - Modified - Zorro Lang
+#		- Added randomize test order capability through -R Option
 #
 #################################################################################

@@ -153,6 +155,7 @@ usage()
     -p              Human readable format logfiles.
     -q              Print less verbose output to screen.
     -r LTPROOT      Fully qualified path where testsuite is installed.
+    -R              Randomize test order.
     -s PATTERN      Only run test cases which match PATTERN.
     -S SKIPFILE     Skip tests specified in SKIPFILE
     -t DURATION     Execute the testsuite for given duration. Examples:

@@ -213,12 +216,13 @@ main()
     local EMAIL_TO=""
     local TAG_RESTRICT_STRING=""
     local PAN_COMMAND=""
+    local RANDOMRUN=0
     local DEFAULT_FILE_NAME_GENERATION_TIME=`date +"%Y_%m_%d-%Hh_%Mm_%Ss"`
     local scenfile=
 
     version_date=$(cat "$LTPROOT/Version")
 
-    while getopts a:c:C:T:d:D:f:F:ehi:I:K:g:l:m:M:Nno:pqr:s:S:t:T:w:x:b:B:z:Z: arg
+    while getopts a:c:C:T:d:D:f:F:ehi:I:K:g:l:m:M:Nno:pqr:Rs:S:t:T:w:x:b:B:z:Z: arg
     do  case $arg in
         a)  EMAIL_TO=$OPTARG
             ALT_EMAIL_OUT=1;;

@@ -410,6 +414,8 @@ main()
 
         r)  LTPROOT=$OPTARG;;
 
+        R)  RANDOMRUN=1;;
+
         s)  TAG_RESTRICT_STRING=$OPTARG;;
 
 	S)  case $OPTARG in

@@ -740,6 +746,12 @@ main()
          done
     fi
 
+    # randomize alltests
+    if [ "$RANDOMRUN" != "0" ]; then
+        sort -R ${TMP}/alltests > ${TMP}/alltests.temp
+        cat ${TMP}/alltests.temp > ${TMP}/alltests
+    fi
+
     [ ! -z "$QUIET_MODE" ] && { echo "INFO: Test start time: $(date)" ; }
     PAN_COMMAND="${LTPROOT}/bin/ltp-pan $QUIET_MODE -e -S $INSTANCES $DURATION -a $$ \
     -n $$ $PRETTY_PRT -f ${TMP}/alltests $LOGFILE $OUTPUTFILE $FAILCMDFILE $TCONFCMDFILE"

Signature · 1 line

2.7.4

Cyril Hrubis

2016-09-20 14:39 UTC #1 lore raw

PATCH v2 1/1 runltp: add -R option for randomize test order

Hi!

runltp line 68

Quoted 6 lines

runltp +2 -0

@@ -64,6 +64,8 @@
 #               - Added capability for default Log file generation
 #		Aug 17 2009 - Modified - Subrata Modak
 #		- Added Fault Injection generation Capability through -F Option
+#		Sep 08 2016 - Modified - Zorro Lang
+#		- Added randomize test order capability through -R Option

We do not update this changelog anymore since the information about the change is stored in the git commit messatge.

Quoted 36 lines

runltp +5 -1

 #
 #################################################################################

@@ -153,6 +155,7 @@ usage()
     -p              Human readable format logfiles.
     -q              Print less verbose output to screen.
     -r LTPROOT      Fully qualified path where testsuite is installed.
+    -R              Randomize test order.
     -s PATTERN      Only run test cases which match PATTERN.
     -S SKIPFILE     Skip tests specified in SKIPFILE
     -t DURATION     Execute the testsuite for given duration. Examples:

@@ -213,12 +216,13 @@ main()
     local EMAIL_TO=""
     local TAG_RESTRICT_STRING=""
     local PAN_COMMAND=""
+    local RANDOMRUN=0
     local DEFAULT_FILE_NAME_GENERATION_TIME=`date +"%Y_%m_%d-%Hh_%Mm_%Ss"`
     local scenfile=
 
     version_date=$(cat "$LTPROOT/Version")
 
-    while getopts a:c:C:T:d:D:f:F:ehi:I:K:g:l:m:M:Nno:pqr:s:S:t:T:w:x:b:B:z:Z: arg
+    while getopts a:c:C:T:d:D:f:F:ehi:I:K:g:l:m:M:Nno:pqr:Rs:S:t:T:w:x:b:B:z:Z: arg
     do  case $arg in
         a)  EMAIL_TO=$OPTARG
             ALT_EMAIL_OUT=1;;

@@ -410,6 +414,8 @@ main()
 
         r)  LTPROOT=$OPTARG;;
 
+        R)  RANDOMRUN=1;;
+
         s)  TAG_RESTRICT_STRING=$OPTARG;;
 
 	S)  case $OPTARG in

runltp main() · line 749

Quoted 4 lines

runltp +1 -0

@@ -740,6 +746,12 @@ main()
          done
     fi
 
+    # randomize alltests

This comment is useless, it's pretty obvious what the code does.

Quoted 14 lines

runltp +5 -0

+    if [ "$RANDOMRUN" != "0" ]; then
+        sort -R ${TMP}/alltests > ${TMP}/alltests.temp
+        cat ${TMP}/alltests.temp > ${TMP}/alltests
+    fi
+
     [ ! -z "$QUIET_MODE" ] && { echo "INFO: Test start time: $(date)" ; }
     PAN_COMMAND="${LTPROOT}/bin/ltp-pan $QUIET_MODE -e -S $INSTANCES $DURATION -a $$ \
     -n $$ $PRETTY_PRT -f ${TMP}/alltests $LOGFILE $OUTPUTFILE $FAILCMDFILE $TCONFCMDFILE"

Signature · 5 lines

2.7.4


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 2 lines

Cyril Hrubis
[email protected]

Zorro Lang

author

2016-09-21 04:56 UTC #2 lore raw

PATCH v2 1/1 runltp: add -R option for randomize test order

Quoted 10 lines · Cyril Hrubis

Hi!
runltp line 68
Quoted 6 lines
runltp +2 -0
@@ -64,6 +64,8 @@
 #               - Added capability for default Log file generation
 #		Aug 17 2009 - Modified - Subrata Modak
 #		- Added Fault Injection generation Capability through -F Option
+#		Sep 08 2016 - Modified - Zorro Lang
+#		- Added randomize test order capability through -R Option
We do not update this changelog anymore since the information about the change is stored in the git commit messatge.

OK, I saw there's changelog, so I thought maybe you hope it can be updated. V3 will remove this.

Quoted 35 lines

runltp +5 -1

 #
 #################################################################################

@@ -153,6 +155,7 @@ usage()
     -p              Human readable format logfiles.
     -q              Print less verbose output to screen.
     -r LTPROOT      Fully qualified path where testsuite is installed.
+    -R              Randomize test order.
     -s PATTERN      Only run test cases which match PATTERN.
     -S SKIPFILE     Skip tests specified in SKIPFILE
     -t DURATION     Execute the testsuite for given duration. Examples:

@@ -213,12 +216,13 @@ main()
     local EMAIL_TO=""
     local TAG_RESTRICT_STRING=""
     local PAN_COMMAND=""
+    local RANDOMRUN=0
     local DEFAULT_FILE_NAME_GENERATION_TIME=`date +"%Y_%m_%d-%Hh_%Mm_%Ss"`
     local scenfile=
 
     version_date=$(cat "$LTPROOT/Version")
 
-    while getopts a:c:C:T:d:D:f:F:ehi:I:K:g:l:m:M:Nno:pqr:s:S:t:T:w:x:b:B:z:Z: arg
+    while getopts a:c:C:T:d:D:f:F:ehi:I:K:g:l:m:M:Nno:pqr:Rs:S:t:T:w:x:b:B:z:Z: arg
     do  case $arg in
         a)  EMAIL_TO=$OPTARG
             ALT_EMAIL_OUT=1;;

@@ -410,6 +414,8 @@ main()
 
         r)  LTPROOT=$OPTARG;;
 
+        R)  RANDOMRUN=1;;
+
         s)  TAG_RESTRICT_STRING=$OPTARG;;

Quoted 8 lines

runltp main() · line 749
Quoted 6 lines
runltp +1 -0
 	S)  case $OPTARG in
@@ -740,6 +746,12 @@ main()
          done
     fi
 
+    # randomize alltests
This comment is useless, it's pretty obvious what the code does.

Sure, V3 will remove this.

Thanks,
Zorro

Quoted 19 lines

Quoted 15 lines

runltp +5 -0

+    if [ "$RANDOMRUN" != "0" ]; then
+        sort -R ${TMP}/alltests > ${TMP}/alltests.temp
+        cat ${TMP}/alltests.temp > ${TMP}/alltests
+    fi
+
     [ ! -z "$QUIET_MODE" ] && { echo "INFO: Test start time: $(date)" ; }
     PAN_COMMAND="${LTPROOT}/bin/ltp-pan $QUIET_MODE -e -S $INSTANCES $DURATION -a $$ \
     -n $$ $PRETTY_PRT -f ${TMP}/alltests $LOGFILE $OUTPUTFILE $FAILCMDFILE $TCONFCMDFILE"

Signature · 5 lines

2.7.4


-- 
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 2 lines

Cyril Hrubis
[email protected]