Skip to contentSkip to conversation

selftests/run_kselftest.sh: make each test individually selectable

PATCH In review Hangbin Liu 4 participants 9 messages Sep 14, 2020 – Sep 29, 2020 1 message not in the archive

Subsystems (4)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

#1 lore raw

PATCH v5 1/1 selftests/run_kselftest.sh: make each test individually selectable

Currently, after generating run_kselftest.sh, there is no way to choose which test we could run. All the tests are listed together and we have to run all every time. This patch enhanced the run_kselftest.sh to make the tests individually selectable. e.g.

  $ ./run_kselftest.sh -t "bpf size timers"

Before the patch:
================

$ cat run_kselftest.sh
\#!/bin/sh
BASE_DIR=$(realpath $(dirname $0))
cd $BASE_DIR
. ./kselftest/runner.sh
ROOT=$PWD
if [ "$1" = "--summary" ]; then
  logfile=$BASE_DIR/output.log
  cat /dev/null > $logfile
fi
[ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg
cd android
run_many        \
        "run.sh"
cd $ROOT
...<snip>...
[ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg
cd zram
run_many        \
        "zram.sh"
cd $ROOT

After the patch:
===============

$ cat run_kselftest.sh
\#!/bin/sh
BASE_DIR=$(realpath $(dirname $0))
. ./kselftest/runner.sh
TESTS="android ...<snip>... filesystems/binderfs ...<snip>... zram"

run_android()

{
        [ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg
        cd android
        run_many        \
                "run.sh"
        cd $ROOT
}

...<snip>...

run_filesystems_binderfs()

{
        [ -w /dev/kmsg ] && echo "kselftest: Running tests in filesystems/binderfs" >> /dev/kmsg
        cd filesystems/binderfs
        run_many        \
                "binderfs_test"
        cd $ROOT
}

...<snip>...

run_zram()

{
        [ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg
        cd zram
        run_many        \
                "zram.sh"
        cd $ROOT
}

usage()
{
cat <<EOF
usage: ${0##*/} OPTS

        -s | --summary          Only print summary info and put detailed log in output.log
        -t | --tests            Test name you want to run specifically
        -h | --help             Show this usage info

EOF
}

while true; do

        case "$1" in
        -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;
        -t | --tests ) TESTS=$2; shift 2 ;;
	-l | --list ) echo $TESTS; exit 0 ;;
        -h | --help ) usage; exit 0 ;;
        "" ) break;;
        * ) usage; exit 1;;
        esac

done

cd $BASE_DIR
ROOT=$PWD
for folder in $TESTS; do

        folder=$(echo $folder | tr -s '/-' '_')
        run_$folder

done

---
v5:
Forgot to update commit description for the new added -l option

v4:
Add parameter -l to list available tests, suggested by Bird, Tim

v3:
1) rebase the patch to latest code
2) move tr -s "/-" "_" in for loop at the end so user could use test folder name directly. Before the fix, user need to use ./run_kselftest.sh -t 'networking_forwarding'. Now they can just run ./run_kselftest.sh -t 'networking/forwarding' directly.

v2: update document and commit description.

---
 Documentation/dev-tools/kselftest.rst |  8 +++++
 tools/testing/selftests/Makefile      | 51 +++++++++++++++++++++------
 tools/testing/selftests/lib.mk        |  2 +-
 3 files changed, 50 insertions(+), 11 deletions(-)
Documentation/dev-tools/kselftest.rst +8 -0
index 469d115a95f1..7b92f9c177f6 100644
@@ -151,6 +151,14 @@ note some tests will require root privileges::
    $ cd kselftest
    $ ./run_kselftest.sh
 
+Or you can run some specific test cases in the installed Kselftests by::
+
+   $ ./run_kselftest.sh -t "bpf size timers"
+
+You can view the available tests to run with::
+
+   $ ./run_kselftest.sh -l
+
 Packaging selftests
 ===================
 
tools/testing/selftests/Makefile +41 -10

69 lines not on this page · raw message

tools/testing/selftests/lib.mk +1 -1
index 51124b962d56..3c4c94a5d184 100644
@@ -108,7 +108,7 @@ emit_tests:
 	for TEST in $(TEST_GEN_PROGS) $(TEST_CUSTOM_PROGS) $(TEST_PROGS); do \
 		BASENAME_TEST=`basename $$TEST`;	\
 		echo "	\\";				\
-		echo -n "	\"$$BASENAME_TEST\"";	\
+		echo -ne "\t\t\"$$BASENAME_TEST\"";	\
 	done;						\
 
 # define if isn't already. It is undefined in make O= case.
Signature · 1 line
2.25.4
#2 lore raw
Quoted 7 lines · Hangbin Liu

Currently, after generating run_kselftest.sh, there is no way to choose which test we could run. All the tests are listed together and we have to run all every time. This patch enhanced the run_kselftest.sh to make the tests individually selectable. e.g.

  $ ./run_kselftest.sh -t "bpf size timers"

My test run break on linux next

./run_kselftest.sh: line 1331: syntax error near unexpected token )' ./run_kselftest.sh: line 1331: -e -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'

steps do run:

# run_kselftest.sh file generated by kselftest Makefile and included in tarball
./run_kselftest.sh 2>&1 | tee "${LOGFILE}"

ref:
https://github.com/nareshkamboju/test-definitions/blob/master/automated/linux/kselftest/kselftest.sh#L222

full test run log:
https://lkft.validation.linaro.org/scheduler/job/1786826#L1391

Quoted 87 lines

Before the patch:
================

$ cat run_kselftest.sh
\#!/bin/sh
BASE_DIR=$(realpath $(dirname $0))
cd $BASE_DIR
. ./kselftest/runner.sh
ROOT=$PWD
if [ "$1" = "--summary" ]; then
  logfile=$BASE_DIR/output.log
  cat /dev/null > $logfile
fi
[ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg
cd android
run_many        \
        "run.sh"
cd $ROOT
...<snip>...
[ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg
cd zram
run_many        \
        "zram.sh"
cd $ROOT

After the patch:
===============

$ cat run_kselftest.sh
\#!/bin/sh
BASE_DIR=$(realpath $(dirname $0))
. ./kselftest/runner.sh
TESTS="android ...<snip>... filesystems/binderfs ...<snip>... zram"

run_android()

{
        [ -w /dev/kmsg ] && echo "kselftest: Running tests in android" >> /dev/kmsg
        cd android
        run_many        \
                "run.sh"
        cd $ROOT
}

...<snip>...

run_filesystems_binderfs()

{
        [ -w /dev/kmsg ] && echo "kselftest: Running tests in filesystems/binderfs" >> /dev/kmsg
        cd filesystems/binderfs
        run_many        \
                "binderfs_test"
        cd $ROOT
}

...<snip>...

run_zram()

{
        [ -w /dev/kmsg ] && echo "kselftest: Running tests in zram" >> /dev/kmsg
        cd zram
        run_many        \
                "zram.sh"
        cd $ROOT
}

usage()
{
cat <<EOF
usage: ${0##*/} OPTS

        -s | --summary          Only print summary info and put detailed log in output.log
        -t | --tests            Test name you want to run specifically
        -h | --help             Show this usage info

EOF
}

while true; do

        case "$1" in
        -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;
        -t | --tests ) TESTS=$2; shift 2 ;;
        -l | --list ) echo $TESTS; exit 0 ;;
        -h | --help ) usage; exit 0 ;;
        "" ) break;;
        * ) usage; exit 1;;
        esac

done

Quoted 8 lines

cd $BASE_DIR
ROOT=$PWD
for folder in $TESTS; do

        folder=$(echo $folder | tr -s '/-' '_')
        run_$folder

done

Signature · 2 lines
Linaro LKFT
https://lkft.linaro.org
#3 lore raw
Quoted 7 lines · Naresh Kamboju
Quoted 6 lines · Hangbin Liu

Currently, after generating run_kselftest.sh, there is no way to choose which test we could run. All the tests are listed together and we have to run all every time. This patch enhanced the run_kselftest.sh to make the tests individually selectable. e.g.

Quoted 7 lines
Quoted 1 line
  $ ./run_kselftest.sh -t "bpf size timers"

My test run break on linux next

./run_kselftest.sh: line 1331: syntax error near unexpected token )' ./run_kselftest.sh: line 1331: -e -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'

Yes, please revert this patch. The resulting script is completely trashed:

BASE_DIR=$(realpath $(dirname $0))
. ./kselftest/runner.sh
TESTS="seccomp"

run_seccomp()
{

-e      [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
-e      cd seccomp
-en     run_many
        \
-ne             "seccomp_bpf"
        \

-ne "seccomp_benchmark"

-e cd $ROOT
}

Signature · 1 line
Kees Cook
maintainer
#4 lore raw
Quoted 34 lines · Kees Cook
Quoted 7 lines · Naresh Kamboju
Quoted 6 lines · Hangbin Liu

Currently, after generating run_kselftest.sh, there is no way to choose which test we could run. All the tests are listed together and we have to run all every time. This patch enhanced the run_kselftest.sh to make the tests individually selectable. e.g.

Quoted 7 lines
Quoted 1 line
   $ ./run_kselftest.sh -t "bpf size timers"

My test run break on linux next

./run_kselftest.sh: line 1331: syntax error near unexpected token )' ./run_kselftest.sh: line 1331: -e -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'

Yes, please revert this patch. The resulting script is completely trashed:

BASE_DIR=$(realpath $(dirname $0))
. ./kselftest/runner.sh
TESTS="seccomp"

run_seccomp()
{

(quoted diff) +0 -6
-e      [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
-e      cd seccomp
-en     run_many
         \
-ne             "seccomp_bpf"
         \
-ne             "seccomp_benchmark"
 
-e      cd $ROOT
Quoted 4 lines

}

Will do.

thanks,
-- Shuah

maintainer
#5 lore raw
In reply to Kees Cook
Quoted 10 lines · Kees Cook
Quoted 9 lines · Naresh Kamboju
Quoted 7 lines · Hangbin Liu

Currently, after generating run_kselftest.sh, there is no way to choose which test we could run. All the tests are listed together and we have to run all every time. This patch enhanced the run_kselftest.sh to make the tests individually selectable. e.g.

   $ ./run_kselftest.sh -t "bpf size timers"
Quoted 9 lines
Quoted 5 lines

My test run break on linux next

./run_kselftest.sh: line 1331: syntax error near unexpected token )' ./run_kselftest.sh: line 1331: -e -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'

Yes, please revert this patch. The resulting script is completely trashed:

Thank you both. Now reverted.

thanks,
-- Shuah

#6 lore raw
Quoted 17 lines · Shuah Khan
Quoted 16 lines · Kees Cook
Quoted 14 lines · Naresh Kamboju
Quoted 7 lines · Hangbin Liu

Currently, after generating run_kselftest.sh, there is no way to choose which test we could run. All the tests are listed together and we have to run all every time. This patch enhanced the run_kselftest.sh to make the tests individually selectable. e.g.

   $ ./run_kselftest.sh -t "bpf size timers"

My test run break on linux next

./run_kselftest.sh: line 1331: syntax error near unexpected token )' ./run_kselftest.sh: line 1331: -e -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'

Quoted 5 lines
Quoted 3 lines

Yes, please revert this patch. The resulting script is completely trashed:

Thank you both. Now reverted.

I've sent an alternative that I think should do nicely. It will work well with LAVA as well.

Signature · 1 line
Kees Cook
#7 lore raw
In reply to Kees Cook
Quoted 34 lines · Kees Cook
Quoted 7 lines · Naresh Kamboju
Quoted 6 lines · Hangbin Liu

Currently, after generating run_kselftest.sh, there is no way to choose which test we could run. All the tests are listed together and we have to run all every time. This patch enhanced the run_kselftest.sh to make the tests individually selectable. e.g.

Quoted 7 lines
Quoted 1 line
  $ ./run_kselftest.sh -t "bpf size timers"

My test run break on linux next

./run_kselftest.sh: line 1331: syntax error near unexpected token )' ./run_kselftest.sh: line 1331: -e -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'

Yes, please revert this patch. The resulting script is completely trashed:

BASE_DIR=$(realpath $(dirname $0))
. ./kselftest/runner.sh
TESTS="seccomp"

run_seccomp()
{

(quoted diff) +0 -6
-e      [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
-e      cd seccomp
-en     run_many
        \
-ne             "seccomp_bpf"
        \
-ne             "seccomp_benchmark"
 
-e      cd $ROOT
Quoted 1 line

}

I'm really sorry to make this trouble. And I'm OK to revert the patch. I just a little wondering how do you generate this script.
I tested with 'make -C tools/testing/selftests gen_tar FORMAT=.xz' before post the patch and all looks good to me.

```

run_seccomp()
{
        [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
        cd seccomp
        run_many        \
                "seccomp_bpf"   \
                "seccomp_benchmark"
        cd $ROOT
}

```

Thanks
Hangbin

#8 lore raw
Quoted 34 lines · Hangbin Liu
Quoted 33 lines · Kees Cook
Quoted 7 lines · Naresh Kamboju
Quoted 6 lines · Hangbin Liu

Currently, after generating run_kselftest.sh, there is no way to choose which test we could run. All the tests are listed together and we have to run all every time. This patch enhanced the run_kselftest.sh to make the tests individually selectable. e.g.

Quoted 7 lines
Quoted 1 line
  $ ./run_kselftest.sh -t "bpf size timers"

My test run break on linux next

./run_kselftest.sh: line 1331: syntax error near unexpected token )' ./run_kselftest.sh: line 1331: -e -s | --summary ) logfile=$BASE_DIR/output.log; cat /dev/null > $logfile; shift ;;'

Yes, please revert this patch. The resulting script is completely trashed:

BASE_DIR=$(realpath $(dirname $0))
. ./kselftest/runner.sh
TESTS="seccomp"

run_seccomp()
{

(quoted diff) +0 -5
-e      [ -w /dev/kmsg ] && echo "kselftest: Running tests in seccomp" >> /dev/kmsg
-e      cd seccomp
-en     run_many
        \
-ne             "seccomp_bpf"
        \
-ne             "seccomp_benchmark"
Quoted 5 lines
Quoted 2 lines

-e cd $ROOT
}

I'm really sorry to make this trouble. And I'm OK to revert the patch. I just a little wondering how do you generate this script.

This issue is with which shell is used. I suspect your /bin/sh is full /bin/bash, where as Naresh's, the CI's, and mine are /bin/dash (which lacks "-e" support for the built-in "echo").

Signature · 1 line
Kees Cook
#9 lore raw
Quoted 6 lines · Kees Cook
Quoted 2 lines

I'm really sorry to make this trouble. And I'm OK to revert the patch. I just a little wondering how do you generate this script.

This issue is with which shell is used. I suspect your /bin/sh is full /bin/bash, where as Naresh's, the CI's, and mine are /bin/dash (which lacks "-e" support for the built-in "echo").

Ah, got it. Thanks for your explanation.

Regards
Hangbin