Skip to contentSkip to conversation

Shell API: TST_TESTFUNC_DATA

RFC PATCH New Petr Vorel 2 participants 6 messages May 16, 2018

lore
Share & tools
lore

Petr Vorel

author
#0 lore raw

cover Shell API: TST_TESTFUNC_DATA

Hi,

this patch is maybe controversial as 1) it adds yet another way how to run tests in new shell API 2) it does not support new C API. It simply workaround missing POSIX support for arrays (in C we do similar things with array struct).

Some network functions which use old API run several times with for loop. Here, in dccp_ipsec.sh, $IPSEC_SIZE_ARRAY gets changed with getopts in ipsec_lib.sh.

runtest/net_stress.ipsec_dccp
dccp4_ipsec01 dccp_ipsec.sh -s "100 500 1000"
...

testcases/network/stress/dccp/dccp_ipsec.sh (simplified) TCID=dccp_ipsec
TST_TOTAL=3
. ipsec_lib.sh
do_test()

{
	for p in $IPSEC_SIZE_ARRAY; do
		tst_netload -H $(tst_ipaddr rhost) -T dccp -n $p -N $p \
			-r $IPSEC_REQUESTS
	done
}

do_test
tst_exit
---
Running it:
/opt/ltp/testscripts/network.sh -D
...
<<<test_start>>>
tag=dccp4_ipsec01 stime=1526471999
cmdline="dccp_ipsec.sh -s 100:500:1000"
contacts=""
analysis=exit
<<<test_output>>>
dccp_ipsec 1 TINFO: run server 'netstress -T dccp -R 500000 -B /tmp/netpan-24771/LTP_dccp_ipsec.uHpwIo4sw4' dccp_ipsec 1 TINFO: run client 'netstress -l -T dccp -H 10.0.0.1 -n 100 -N 100 -a 2 -r 500 -d tst_netload.res -g 43799' dccp_ipsec 1 TPASS: netstress passed, time spent '116' ms
dccp_ipsec 2 TINFO: run server 'netstress -T dccp -R 500000 -B /tmp/netpan-24771/LTP_dccp_ipsec.uHpwIo4sw4' dccp_ipsec 2 TINFO: run client 'netstress -l -T dccp -H 10.0.0.1 -n 500 -N 500 -a 2 -r 500 -d tst_netload.res -g 39537' dccp_ipsec 2 TPASS: netstress passed, time spent '11' ms
dccp_ipsec 3 TINFO: run server 'netstress -T dccp -R 500000 -B /tmp/netpan-24771/LTP_dccp_ipsec.uHpwIo4sw4' dccp_ipsec 3 TINFO: run client 'netstress -l -T dccp -H 10.0.0.1 -n 1000 -N 1000 -a 2 -r 500 -d tst_netload.res -g 34207' dccp_ipsec 3 TPASS: netstress passed, time spent '9' ms
...

When rewritten into new API (simplified):
TST_TESTFUNC=do_test
. ipsec_lib.sh
do_test()

{
	for p in $IPSEC_SIZE_ARRAY; do
		tst_netload -H $(tst_ipaddr rhost) -T dccp -n $p -N $p \
			-r $IPSEC_REQUESTS
	done
}

tst_run
---
The problem is we lost counters:
...
<<<test_output>>>
dccp_ipsec 1 TINFO: run server 'netstress -T dccp -R 500000 -B /tmp/netpan-20326/LTP_dccp_ipsec.6mH0vypvZC' dccp_ipsec 1 TINFO: run client 'netstress -l -T dccp -H 10.0.0.1 -n 100 -N 100 -a 2 -r 500 -d tst_netload.res -g 44791' dccp_ipsec 1 TPASS: netstress passed, time spent '18' ms
dccp_ipsec 1 TINFO: run server 'netstress -T dccp -R 500000 -B /tmp/netpan-20326/LTP_dccp_ipsec.6mH0vypvZC' dccp_ipsec 1 TINFO: run client 'netstress -l -T dccp -H 10.0.0.1 -n 500 -N 500 -a 2 -r 500 -d tst_netload.res -g 39661' dccp_ipsec 1 TPASS: netstress passed, time spent '16' ms
dccp_ipsec 1 TINFO: run server 'netstress -T dccp -R 500000 -B /tmp/netpan-20326/LTP_dccp_ipsec.6mH0vypvZC' dccp_ipsec 1 TINFO: run client 'netstress -l -T dccp -H 10.0.0.1 -n 1000 -N 1000 -a 2 -r 500 -d tst_netload.res -g 41279' dccp_ipsec 1 TPASS: netstress passed, time spent '12' ms

UDP IPsec tests (udp_ipsec.sh, udp_ipsec_vti.sh) have TST_TOTAL=6, so it gets a bit unclear with just '1'.

With TST_TESTFUNC_DATA and TST_TESTFUNC_DATA_IFS we can 1) simplify 7 tests now without creating special helper 2) keep ids.

Second patch isn't applicable, it's just for demonstrating how it works as it expects further changes, I'll post with bigger patch for rewriting net/{stress,virt} tests. What is missing here is rewritten dccp/dccp_ipsec.sh and ipsec_lib.sh into new shell API and changes to runtest/net_stress.ipsec_dccp:
-dccp4_ipsec01 dccp_ipsec.sh -s "100 500 1000" +dccp4_ipsec01 dccp_ipsec.sh -s 100:500:1000

Kind regards,
Petr

Petr Vorel (2):
lib/tst_test.sh: TST_TESTFUNC_DATA and TST_TESTFUNC_DATA_IFS net/{stress,virt}: Use TST_TESTFUNC_DATA and TST_TESTFUNC_DATA_IFS

 doc/test-writing-guidelines.txt             | 23 +++++++++++++++++++++++
 testcases/lib/tst_test.sh                   | 20 ++++++++++++++++++--
 testcases/network/stress/dccp/dccp_ipsec.sh |  6 ++----
 testcases/network/stress/ipsec/ipsec_lib.sh |  3 ++-
 4 files changed, 45 insertions(+), 7 deletions(-)
Signature · 1 line
2.16.3
author
#1 lore raw

RFC PATCH 1/2 lib/tst_test.sh: TST_TESTFUNC_DATA and TST_TESTFUNC_DATA_IFS

This is specific only for shell.

Signed-off-by Petr Vorel <[email protected]>

---
If we accept it, it's a question if pass iteration number as well (and if yes, maybe it should be in $1 and data in $2).

Kind regards,
Petr

---
 doc/test-writing-guidelines.txt | 23 +++++++++++++++++++++++
 testcases/lib/tst_test.sh       | 20 ++++++++++++++++++--
 2 files changed, 41 insertions(+), 2 deletions(-)
doc/test-writing-guidelines.txt +23 -0
index 8e405a034..2184f6365 100644
@@ -1485,6 +1485,29 @@ Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc.,
 the '$TST_TESTFUNC' is executed '$TST_CNT' times and the test number is passed
 to it in the '$1'.
 
+[source,sh]
+-------------------------------------------------------------------------------
+#!/bin/sh
+#
+# Example test with tests using TST_TESTFUNC_DATA
+#
+
+TST_TESTFUNC=do_test
+TST_TESTFUNC_DATA="foo bar"
+. tst_test.sh
+
+do_test()
+{
+	1) tst_res TPASS "Test passed with data '$1'";;
+}
+
+tst_run
+-------------------------------------------------------------------------------
+
+It's possible to pass data for function with '$TST_TESTFUNC_DATA'. Optional
+'$TST_TESTFUNC_DATA_IFS' variable is supported to set '$IFS' for splitting.
+'$TST_TESTFUNC_DATA' cannot be mixed with '$TST_CNT'.
+
 2.3.2 Library variables
 ^^^^^^^^^^^^^^^^^^^^^^^
testcases/lib/tst_test.sh +18 -2
index 8d49d34b6..31bf51e10 100644
@@ -246,7 +246,7 @@ tst_rescmp()
 
 tst_run()
 {
-	local tst_i
+	local tst_i ifs
 
 	if [ -n "$TST_TEST_PATH" ]; then
 		for tst_i in $(grep TST_ "$TST_TEST_PATH" | sed 's/.*TST_//; s/[="} \t\/:`].*//'); do

@@ -255,7 +255,7 @@ tst_run()
 			OPTS|USAGE|PARSE_ARGS|POS_ARGS);;
 			NEEDS_ROOT|NEEDS_TMPDIR|NEEDS_DEVICE|DEVICE);;
 			NEEDS_CMDS|NEEDS_MODULE|MODPATH|DATAROOT);;
-			IPV6);;
+			IPV6|TESTFUNC_DATA|TESTFUNC_DATA_IFS);;
 			*) tst_res TWARN "Reserved variable TST_$tst_i used!";;
 			esac
 		done

@@ -364,6 +364,18 @@ tst_run()
 					TST_COUNT=$((TST_COUNT+1))
 				done
 			fi
+		elif [ -n "$TST_TESTFUNC_DATA" ]; then
+			if [ -n "$TST_TESTFUNC_DATA_IFS" ]; then
+				ifs="$IFS"
+				IFS="$TST_TESTFUNC_DATA_IFS"
+			fi
+			for tst_i in $TST_TESTFUNC_DATA; do
+				[ -n "$ifs" ] && IFS="$ifs"
+				local res=$(tst_resstr)
+				$TST_TESTFUNC $tst_i
+				tst_rescmp "$res"
+				TST_COUNT=$((TST_COUNT+1))
+			done
 		else
 			local res=$(tst_resstr)
 			$TST_TESTFUNC

@@ -400,6 +412,10 @@ if [ -z "$TST_NO_DEFAULT_RUN" ]; then
 		tst_brk TBROK "TST_TESTFUNC is not defined"
 	fi
 
+	if [ -n "$TST_CNT" -a -n "$TST_TESTFUNC_DATA" ]; then
+		tst_brk TBROK "TST_CNT cannot be mixed with TST_TESTFUNC_DATA"
+	fi
+
 	if [ -n "$TST_CNT" ]; then
 		if ! tst_is_int "$TST_CNT"; then
 			tst_brk TBROK "TST_CNT must be integer"
Signature · 1 line
2.16.3
#2 lore raw

RFC PATCH 1/2 lib/tst_test.sh: TST_TESTFUNC_DATA and TST_TESTFUNC_DATA_IFS

Hi!

Quoted 21 lines

This is specific only for shell.

Signed-off-by Petr Vorel <[email protected]>

---
If we accept it, it's a question if pass iteration number as well (and if yes, maybe it should be in $1 and data in $2).

Kind regards,
Petr

---
 doc/test-writing-guidelines.txt | 23 +++++++++++++++++++++++
 testcases/lib/tst_test.sh       | 20 ++++++++++++++++++--
 2 files changed, 41 insertions(+), 2 deletions(-)
doc/test-writing-guidelines.txt +0 -0
index 8e405a034..2184f6365 100644
@@ -1485,6 +1485,29 @@ Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc.,
 the '$TST_TESTFUNC' is executed '$TST_CNT' times and the test number is passed
 to it in the '$1'.

doc/test-writing-guidelines.txt Otherwise, if '$TST_CNT' is set but there is no '$\{TST_TESTFUNC\}1', etc., · line 1496

Quoted 10 lines
doc/test-writing-guidelines.txt +9 -0
 
+[source,sh]
+-------------------------------------------------------------------------------
+#!/bin/sh
+#
+# Example test with tests using TST_TESTFUNC_DATA
+#
+
+TST_TESTFUNC=do_test
+TST_TESTFUNC_DATA="foo bar"

Why not just TST_TEST_DATA and TST_TEST_DATA_IFS?

Quoted 22 lines
doc/test-writing-guidelines.txt +14 -0
+. tst_test.sh
+
+do_test()
+{
+	1) tst_res TPASS "Test passed with data '$1'";;
+}
+
+tst_run
+-------------------------------------------------------------------------------
+
+It's possible to pass data for function with '$TST_TESTFUNC_DATA'. Optional
+'$TST_TESTFUNC_DATA_IFS' variable is supported to set '$IFS' for splitting.
+'$TST_TESTFUNC_DATA' cannot be mixed with '$TST_CNT'.
+
 2.3.2 Library variables
 ^^^^^^^^^^^^^^^^^^^^^^^
testcases/lib/tst_test.sh +0 -0
index 8d49d34b6..31bf51e10 100644

testcases/lib/tst_test.sh tst_rescmp() · line 249

Quoted 5 lines
testcases/lib/tst_test.sh +1 -1
@@ -246,7 +246,7 @@ tst_rescmp()
 
 tst_run()
 {
-	local tst_i
+	local tst_i ifs

The variables here has to be still prefixed with tst_, even though they are local these are still defined in the $TST_TESTFUNC.

Quoted 17 lines
testcases/lib/tst_test.sh +3 -1
 	if [ -n "$TST_TEST_PATH" ]; then
 		for tst_i in $(grep TST_ "$TST_TEST_PATH" | sed 's/.*TST_//; s/[="} \t\/:`].*//'); do

@@ -255,7 +255,7 @@ tst_run()
 			OPTS|USAGE|PARSE_ARGS|POS_ARGS);;
 			NEEDS_ROOT|NEEDS_TMPDIR|NEEDS_DEVICE|DEVICE);;
 			NEEDS_CMDS|NEEDS_MODULE|MODPATH|DATAROOT);;
-			IPV6);;
+			IPV6|TESTFUNC_DATA|TESTFUNC_DATA_IFS);;
 			*) tst_res TWARN "Reserved variable TST_$tst_i used!";;
 			esac
 		done

@@ -364,6 +364,18 @@ tst_run()
 					TST_COUNT=$((TST_COUNT+1))
 				done
 			fi
+		elif [ -n "$TST_TESTFUNC_DATA" ]; then
+			if [ -n "$TST_TESTFUNC_DATA_IFS" ]; then

testcases/lib/tst_test.sh tst_run() · line 378

Quoted 10 lines
testcases/lib/tst_test.sh +10 -0
+				ifs="$IFS"
+				IFS="$TST_TESTFUNC_DATA_IFS"
+			fi
+			for tst_i in $TST_TESTFUNC_DATA; do
+				[ -n "$ifs" ] && IFS="$ifs"
+				local res=$(tst_resstr)
+				$TST_TESTFUNC $tst_i
+				tst_rescmp "$res"
+				TST_COUNT=$((TST_COUNT+1))
+			done

I do not like redefing IFS, it's really ugly.

What about something as:

DATA="foo:bar:d dd"
DELIM=":"
i=1
while true; do

        param="$(echo "$DATA" | cut -d$DELIM -f$i)"
        [ -z "$param" ] && break;
        echo "$param"
        i=$((i+1))

done

testcases/lib/tst_test.sh if [ -z "$TST_NO_DEFAULT_RUN" ]; then · line 417

Quoted 10 lines
testcases/lib/tst_test.sh +3 -0
 		else
 			local res=$(tst_resstr)
 			$TST_TESTFUNC

@@ -400,6 +412,10 @@ if [ -z "$TST_NO_DEFAULT_RUN" ]; then
 		tst_brk TBROK "TST_TESTFUNC is not defined"
 	fi
 
+	if [ -n "$TST_CNT" -a -n "$TST_TESTFUNC_DATA" ]; then
+		tst_brk TBROK "TST_CNT cannot be mixed with TST_TESTFUNC_DATA"
+	fi

I'm not sure about this, why cannot each of the functions take TST_TESTFUNC_DATA parameters?

I.e. we may have two test functions that take exaclty the same array of parameters, setting TST_CNT=2 and TST_TESTFUNC_DATA may make sense...

Quoted 3 lines
testcases/lib/tst_test.sh +0 -0
 	if [ -n "$TST_CNT" ]; then
 		if ! tst_is_int "$TST_CNT"; then
 			tst_brk TBROK "TST_CNT must be integer"
Signature · 2 lines
Cyril Hrubis
[email protected]
author
#3 lore raw

RFC PATCH 1/2 lib/tst_test.sh: TST_TESTFUNC_DATA and TST_TESTFUNC_DATA_IFS

Hi Cyril,

Thanks for your comments, they make sense, I'll send v2.

+ more comments bellow:

Quoted 8 lines

I do not like redefing IFS, it's really ugly.

What about something as:

DATA="foo:bar:d dd"
DELIM=":"
i=1
while true; do

Quoted 5 lines
        param="$(echo "$DATA" | cut -d$DELIM -f$i)"
        [ -z "$param" ] && break;
        echo "$param"
        i=$((i+1))

done

Playing with IFS is ugly, but it doesn't require any external dependency. But cut is common, let's make it.

Quoted 14 lines

testcases/lib/tst_test.sh if [ -z "$TST_NO_DEFAULT_RUN" ]; then · line 417

Quoted 10 lines
testcases/lib/tst_test.sh +3 -0
 		else
 			local res=$(tst_resstr)
 			$TST_TESTFUNC

@@ -400,6 +412,10 @@ if [ -z "$TST_NO_DEFAULT_RUN" ]; then
 		tst_brk TBROK "TST_TESTFUNC is not defined"
 	fi
 
+	if [ -n "$TST_CNT" -a -n "$TST_TESTFUNC_DATA" ]; then
+		tst_brk TBROK "TST_CNT cannot be mixed with TST_TESTFUNC_DATA"
+	fi

I'm not sure about this, why cannot each of the functions take TST_TESTFUNC_DATA parameters?

Quoted 2 lines

I.e. we may have two test functions that take exaclty the same array of parameters, setting TST_CNT=2 and TST_TESTFUNC_DATA may make sense...

Well, I'll try to add meaningful support for TST_TESTFUNC_DATA :). Let's pass them as whole for cases where $TST_CNT is set:
+++ testcases/lib/tst_test.sh
@@ -352,14 +352,14 @@ tst_run()
if type test1 > /dev/null 2>&1; then
for tst_i in $(seq $TST_CNT); do
local res=$(tst_resstr)
- $TST_TESTFUNC$tst_i
+ $TST_TESTFUNC$tst_i $TST_TEST_DATA
tst_rescmp "$res" TST_COUNT=$((TST_COUNT+1))
done
else
for tst_i in $(seq $TST_CNT); do
local res=$(tst_resstr)
- $TST_TESTFUNC $tst_i
+ $TST_TESTFUNC $tst_i $TST_TEST_DATA
tst_rescmp "$res" TST_COUNT=$((TST_COUNT+1))
done

Quoted 3 lines

testcases/lib/tst_test.sh if [ -z "$TST_NO_DEFAULT_RUN" ]; then · line 421

Quoted 3 lines
testcases/lib/tst_test.sh +0 -0
 	if [ -n "$TST_CNT" ]; then
 		if ! tst_is_int "$TST_CNT"; then
 			tst_brk TBROK "TST_CNT must be integer"

And know tst_i (sequence number of a test) for test is needed for some tests which use only one function - i.e. TST_TESTFUNC_DATA is passed (no TST_CNT). Let's keep $1 as sequence number and $2 for data itself.
I wonder if $1 shouldn't always have tst_i, even for tests without $TST_CNT. Something like:
+++ testcases/lib/tst_test.sh
@@ -352,14 +352,14 @@ tst_run()
if type test1 > /dev/null 2>&1; then
for tst_i in $(seq $TST_CNT); do
local res=$(tst_resstr)
- $TST_TESTFUNC$tst_i
+ $TST_TESTFUNC$tst_i $tst_i $TST_TEST_DATA
tst_rescmp "$res" TST_COUNT=$((TST_COUNT+1))
done
else
for tst_i in $(seq $TST_CNT); do
local res=$(tst_resstr)
- $TST_TESTFUNC $tst_i
+ $TST_TESTFUNC $tst_i $TST_TEST_DATA
tst_rescmp "$res" TST_COUNT=$((TST_COUNT+1))
done
@@ -378,7 +378,7 @@ tst_run()
done
else
local res=$(tst_resstr)
- $TST_TESTFUNC
+ $TST_TESTFUNC 1
tst_rescmp "$res"
TST_COUNT=$((TST_COUNT+1))
fi

Kind regards,
Petr

author
#4 lore raw

RFC PATCH 1/2 lib/tst_test.sh: TST_TESTFUNC_DATA and TST_TESTFUNC_DATA_IFS

Hi Cyril,

Quoted 24 lines

And know tst_i (sequence number of a test) for test is needed for some tests which use only one function - i.e. TST_TESTFUNC_DATA is passed (no TST_CNT). Let's keep $1 as sequence number and $2 for data itself.
I wonder if $1 shouldn't always have tst_i, even for tests without $TST_CNT. Something like:
+++ testcases/lib/tst_test.sh

testcases/lib/tst_test.sh +2 -2
@@ -352,14 +352,14 @@ tst_run()
 			if type test1 > /dev/null 2>&1; then
 				for tst_i in $(seq $TST_CNT); do
 					local res=$(tst_resstr)
-					$TST_TESTFUNC$tst_i
+					$TST_TESTFUNC$tst_i $tst_i $TST_TEST_DATA
 					tst_rescmp "$res"
 					TST_COUNT=$((TST_COUNT+1))
 				done
 			else
 				for tst_i in $(seq $TST_CNT); do
 					local res=$(tst_resstr)
-					$TST_TESTFUNC $tst_i
+					$TST_TESTFUNC $tst_i $TST_TEST_DATA
 					tst_rescmp "$res"
 					TST_COUNT=$((TST_COUNT+1))
 				done

testcases/lib/tst_test.sh tst_run() · line 381

Quoted 5 lines
testcases/lib/tst_test.sh +1 -1
@@ -378,7 +378,7 @@ tst_run()
 			done
 		else
 			local res=$(tst_resstr)
-			$TST_TESTFUNC
+			$TST_TESTFUNC 1

Actually, I'd avoid this '1', not useful much.

testcases/lib/tst_test.sh tst_run() · line 366

Quoted 3 lines
testcases/lib/tst_test.sh +0 -0
 			tst_rescmp "$res"
 			TST_COUNT=$((TST_COUNT+1))
 		fi

Kind regards,
Petr

author
#5 lore raw

RFC PATCH 2/2 net/{stress, virt}: Use TST_TESTFUNC_DATA and TST_TESTFUNC_DATA_IFS

In reply to Petr Vorel
Not-yet-signed-off-by Petr Vorel <[email protected]>

---
This is just for demonstration purposes as
1) It's not applicable as it expect more changes to be posted later in patch rewriting net/{stress,virt} into new shell API (see cover letter).
2) Changes are going to be done in 6 more files.

---
 testcases/network/stress/dccp/dccp_ipsec.sh | 6 ++----
 testcases/network/stress/ipsec/ipsec_lib.sh | 3 ++-
 2 files changed, 4 insertions(+), 5 deletions(-)
testcases/network/stress/dccp/dccp_ipsec.sh +2 -4
index dae530fa3..3b51e4587 100755
@@ -21,10 +21,8 @@ do_setup()
 
 do_test()
 {
-	for p in $IPSEC_SIZE_ARRAY; do
-		tst_netload -H $(tst_ipaddr rhost) -T dccp -n $p -N $p \
-			-r $IPSEC_REQUESTS
-	done
+	tst_netload -H $(tst_ipaddr rhost) -T dccp -n $1 -N $1 \
+		-r $IPSEC_REQUESTS
 }
 
 tst_run
testcases/network/stress/ipsec/ipsec_lib.sh +2 -1
index bfb022bd0..782c3fb89 100644
@@ -48,7 +48,8 @@ ipsec_lib_parse_args()
 	r) IPSEC_REQUESTS="$2" ;;
 	esac
 
-	IPSEC_SIZE_ARRAY="$(echo $IPSEC_SIZE_ARRAY | sed 's/:/ /g')"
+	TST_TESTFUNC_DATA="$IPSEC_SIZE_ARRAY"
+	TST_TESTFUNC_DATA_IFS=":"
 }
 
 TST_OPTS="l:m:p:s:S:k:A:e:a:c:r:"
Signature · 1 line
2.16.3