runltp: add -R option for randomize test order

Generally we write test entries into runtests/${testfile}, then runltp reads and runs lines sequentially everytime, nearly no one will change the test order.

Recently we find bugs by change the test order，different test combination maybe find more bugs. So I add this new option -R to randomize test order.

Signed-off-by Zorro Lang <[email protected]>

---

V2 use 'sort -R' to instead of 'randlines.awk'.
V3 remove change log and someone useless comment

Thanks,
Zorro

 runltp | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

runltp +5 -1

index bb2e5d8..33e23ab 100755
@@ -153,6 +153,7 @@ usage()
     -p              Human readable format logfiles.
     -q              Print less verbose output to screen.
     -r LTPROOT      Fully qualified path where testsuite is installed.
+    -R              Randomize test order.
     -s PATTERN      Only run test cases which match PATTERN.
     -S SKIPFILE     Skip tests specified in SKIPFILE
     -t DURATION     Execute the testsuite for given duration. Examples:

@@ -213,12 +214,13 @@ main()
     local EMAIL_TO=""
     local TAG_RESTRICT_STRING=""
     local PAN_COMMAND=""
+    local RANDOMRUN=0
     local DEFAULT_FILE_NAME_GENERATION_TIME=`date +"%Y_%m_%d-%Hh_%Mm_%Ss"`
     local scenfile=
 
     version_date=$(cat "$LTPROOT/Version")
 
-    while getopts a:c:C:T:d:D:f:F:ehi:I:K:g:l:m:M:Nno:pqr:s:S:t:T:w:x:b:B:z:Z: arg
+    while getopts a:c:C:T:d:D:f:F:ehi:I:K:g:l:m:M:Nno:pqr:Rs:S:t:T:w:x:b:B:z:Z: arg
     do  case $arg in
         a)  EMAIL_TO=$OPTARG
             ALT_EMAIL_OUT=1;;

@@ -410,6 +412,8 @@ main()
 
         r)  LTPROOT=$OPTARG;;
 
+        R)  RANDOMRUN=1;;
+
         s)  TAG_RESTRICT_STRING=$OPTARG;;
 
        S)  case $OPTARG in

runltp +2 -1

+    fi
+
     [ ! -z "$QUIET_MODE" ] && { echo "INFO: Test start time: $(date)" ; }
     PAN_COMMAND="${LTPROOT}/bin/ltp-pan $QUIET_MODE -e -S $INSTANCES $DURATION -a $$ \
     -n $$ $PRETTY_PRT -f ${TMP}/alltests $LOGFILE $OUTPUTFILE $FAILCMDFILE $TCONFCMDFILE"
--

2.7.4

--
Mailing list info: https://lists.linux.it/listinfo/ltp

Quoted 57 lines · Zorro Lang

Generally we write test entries into runtests/${testfile}, then runltp reads and runs lines sequentially everytime, nearly no one will change the test order.

Recently we find bugs by change the test order，different test combination maybe find more bugs. So I add this new option -R to randomize test order.

Signed-off-by Zorro Lang <[email protected]>

---

V2 use 'sort -R' to instead of 'randlines.awk'.
V3 remove change log and someone useless comment

Thanks,
Zorro

 runltp | 11 ++++++++++-
 1 file changed, 10 insertions(+), 1 deletion(-)

runltp +5 -1

index bb2e5d8..33e23ab 100755
@@ -153,6 +153,7 @@ usage()
     -p              Human readable format logfiles.
     -q              Print less verbose output to screen.
     -r LTPROOT      Fully qualified path where testsuite is installed.
+    -R              Randomize test order.
     -s PATTERN      Only run test cases which match PATTERN.
     -S SKIPFILE     Skip tests specified in SKIPFILE
     -t DURATION     Execute the testsuite for given duration. Examples:

@@ -213,12 +214,13 @@ main()
     local EMAIL_TO=""
     local TAG_RESTRICT_STRING=""
     local PAN_COMMAND=""
+    local RANDOMRUN=0
     local DEFAULT_FILE_NAME_GENERATION_TIME=`date +"%Y_%m_%d-%Hh_%Mm_%Ss"`
     local scenfile=
 
     version_date=$(cat "$LTPROOT/Version")
 
-    while getopts a:c:C:T:d:D:f:F:ehi:I:K:g:l:m:M:Nno:pqr:s:S:t:T:w:x:b:B:z:Z: arg
+    while getopts a:c:C:T:d:D:f:F:ehi:I:K:g:l:m:M:Nno:pqr:Rs:S:t:T:w:x:b:B:z:Z: arg
     do  case $arg in
         a)  EMAIL_TO=$OPTARG
             ALT_EMAIL_OUT=1;;

@@ -410,6 +412,8 @@ main()
 
         r)  LTPROOT=$OPTARG;;
 
+        R)  RANDOMRUN=1;;
+
         s)  TAG_RESTRICT_STRING=$OPTARG;;
 
        S)  case $OPTARG in

runltp main() · line 749

Quoted 6 lines

runltp +3 -0

@@ -740,6 +744,11 @@ main()
          done
     fi
 
+    if [ "$RANDOMRUN" != "0" ]; then
+        sort -R ${TMP}/alltests > ${TMP}/alltests.temp
+        cat ${TMP}/alltests.temp > ${TMP}/alltests

How about combine them in one line?

shuf ${TMP}/alltests -o ${TMP}/alltests
or,
sort -R ${TMP}/alltests -o ${TMP}/alltests

Quoted 12 lines

runltp +2 -1

+    fi
+
     [ ! -z "$QUIET_MODE" ] && { echo "INFO: Test start time: $(date)" ; }
     PAN_COMMAND="${LTPROOT}/bin/ltp-pan $QUIET_MODE -e -S $INSTANCES $DURATION -a $$ \
     -n $$ $PRETTY_PRT -f ${TMP}/alltests $LOGFILE $OUTPUTFILE $FAILCMDFILE $TCONFCMDFILE"
--

2.7.4

--
Mailing list info: https://lists.linux.it/listinfo/ltp

Signature · 3 lines

Regards,
Li Wang
Email: [email protected]

Hm, it works, even works for large file (>pagesize). I thought it maybe make mistake if read and rewrite same file at same time when I sent this patch. Now I just read the code of sort.c, it'll read the file into buffer and temp file at first, then close the fopen(r) file and go to fopen(w) the file:

          if (buf.eof && !nfiles && !ntemps && !buf.left)
            {
              xfclose (fp, file);
              tfp = xfopen (output_file, "w");
	      ...
	      ...

So it looks safe to read and write the same file together ... But I don't know if it worth changing this patch (the old way looks safer:). Can maintainers choose a favorite way?