tests: add feature probing
Florian Westphal
maintainerRFC PATCH 1/1 tests: add feature probing
Running selftests on older kernels makes some of them fail very early because some tests use features that are not available on older kernels, e.g. -stable releases.
Known examples:
- inner header matching
- anonymous chains
- elem delete from packet path
Also, some test cases might fail because a feature isn't compiled in, such as netdev chains for example.
This adds a feature-probing to the shell tests.
Simply drop a 'nft -f' compatible file with a .nft suffix into tests/shell/features.
run-tests.sh will load it via --check and will add
NFT_TESTS_HAVE_${filename}=$?
to the environment.
The test script can then either elide a part of the test or replace it with a supported feature subset.
This adds the probing skeleton, a probe file for chain_binding and alters 30s-stress to suppress anonon chains when its unuspported.
Note that 30s-stress is optionally be run standalone, so this adds more code than needed, for tests that always run via run-tests.sh its enough to do
[ $NFT_HAVE_chain_binding -eq 1 ] && test_chain_binding
---
Not yet fully tested, so RFC tag.
tests/shell/features/chain_binding.nft | 5 ++ tests/shell/run-tests.sh | 23 ++++++++ tests/shell/testcases/transactions/30s-stress | 52 ++++++++++++++++--- 3 files changed, 73 insertions(+), 7 deletions(-) create mode 100644 tests/shell/features/chain_binding.nft
tests/shell/features/chain_binding.nft new file +5 -0
@@ -0,0 +1,5 @@ +table ip t { + chain c { + jump { counter; } + } +}
tests/shell/run-tests.sh +23 -0
@@ -163,6 +163,23 @@ ok=0 failed=0 taint=0 +check_features() +{ + for ffilename in features/*.nft; do + feature=${ffilename#*/} + feature=${feature%*.nft} + eval NFT_HAVE_${feature}=0 + $NFT -f "$ffilename" 2>/dev/null + if [ $? -eq 0 ]; then + eval NFT_HAVE_${feature}=1 + else + echo "WARNING: Disabling feature $feature" 1>&2 + fi + + export NFT_HAVE_${feature} + done +} + check_taint() { read taint_now < /proc/sys/kernel/tainted
@@ -211,6 +228,7 @@ check_kmemleak() fi } +check_features check_taint for testfile in $(find_tests)
@@ -277,5 +295,10 @@ check_kmemleak_force msg_info "results: [OK] $ok [FAILED] $failed [TOTAL] $((ok+failed))" +if [ "$VERBOSE" == "y" ] ; then + echo "Used Features:" + env | grep NFT_HAVE_ +fi + kernel_cleanup [ "$failed" -eq 0 ]
tests/shell/testcases/transactions/30s-stress +45 -7
102 lines not on this page · raw message
Signature · 1 line
2.41.0
Florian Westphal
maintainertests/shell/run-tests.sh ok=0 · line 172
Quoted 7 lines · Florian Westphal
tests/shell/run-tests.sh +7
-0+check_features() +{ + for ffilename in features/*.nft; do + feature=${ffilename#*/} + feature=${feature%*.nft} + eval NFT_HAVE_${feature}=0 + $NFT -f "$ffilename" 2>/dev/null
This is broken if run-tests.sh is not invoked inside tests/shell, I've fixed this up.
Meanwhile I added probe for netdev-chains-without-device and SKIPPED support:
I: [OK] ././testcases/chains/0044chain_destroy_0
I: [SKIPPED] ././testcases/chains/netdev_chain_0
I: [OK] ././testcases/comments/comments_0
My plan is to keep adding more probe files locally until ./run-tests.sh passes on 5.14-patched-kernel and then send the batch.
I've also changed the probe files to contain the upstream patch and its upstream kernel version as a comment.
Quoted 66 lines · Florian Westphal
Running selftests on older kernels makes some of them fail very early because some tests use features that are not available on older kernels, e.g. -stable releases.
Known examples:
- inner header matching
- anonymous chains
- elem delete from packet pathAlso, some test cases might fail because a feature isn't compiled in, such as netdev chains for example.
This adds a feature-probing to the shell tests.
Simply drop a 'nft -f' compatible file with a .nft suffix into tests/shell/features.
run-tests.shwill load it via --check and will addNFT_TESTS_HAVE_${filename}=$?
to the environment.
The test script can then either elide a part of the test or replace it with a supported feature subset.
This adds the probing skeleton, a probe file for
chain_bindingand alters 30s-stress to suppress anonon chains when its unuspported.Note that 30s-stress is optionally be run standalone, so this adds more code than needed, for tests that always run via
run-tests.shits enough to do[ $NFT_HAVE_chain_binding -eq 1 ] &&
test_chain_bindingSigned-off-by Florian Westphal <[email protected]>---
Not yet fully tested, so RFC tag.tests/shell/features/chain_binding.nft | 5 ++ tests/shell/run-tests.sh | 23 ++++++++ tests/shell/testcases/transactions/30s-stress | 52 ++++++++++++++++- --
3 files changed, 73 insertions(+), 7 deletions(-)create mode 100644 tests/shell/features/chain_binding.nft
tests/shell/features/chain_binding.nft new file +5
-0@@ -0,0 +1,5 @@ +table ip t { + chain c { + jump { counter; } + } +}tests/shell/run-tests.sh +0
-0@@ -163,6 +163,23 @@ ok=0failed=0
taint=0
tests/shell/run-tests.sh ok=0 · line 172
Quoted 7 lines
tests/shell/run-tests.sh +7
-0+check_features() +{ + for ffilename in features/*.nft; do + feature=${ffilename#*/} + feature=${feature%*.nft} + eval NFT_HAVE_${feature}=0 + $NFT -f "$ffilename" 2>/dev/null
is the "--check" option here missing? At least, the commit message says
"run-tests.sh will load it via --check and will add"
Quoted 8 lines
+ if [ $? -eq 0 ]; then
+ eval NFT_HAVE_${feature}=1
+ else
+ echo "WARNING: Disabling feature $feature" 1>&2
+ fi
+
+ export NFT_HAVE_${feature}
I think it should run in a separate netns too.
With "[PATCH nft v2 0/3] tests/shell: allow running tests as non-root" patch, that would be "${UNSHARE[@]}" $NFT ...
Quoted 18 lines
+ done
tests/shell/run-tests.sh +2
-0+} +check_taint()
{
readtaint_now</proc/sys/kernel/taintedtests/shell/run-tests.sh +0
-0@@ -211,6 +228,7 @@ check_kmemleak()fi
}
+check_features
check_taint
for testfile in $(find_tests)tests/shell/run-tests.sh +0
-0@@ -277,5 +295,10 @@ check_kmemleak_force
msg_info "results: [OK] $ok [FAILED] $failed [TOTAL] $((ok+failed))"
tests/shell/run-tests.sh check_kmemleak_force · line 301
Quoted 4 lines
tests/shell/run-tests.sh +4
-0+if [ "$VERBOSE" == "y" ] ; then + echo "Used Features:" + env | grep NFT_HAVE_ +fi
OK, that's nice, to see in the output.
But why this "nft -f" specific detection? Why not just executable scripts?
Also, why should "run-tests.sh" pre-evaluate those NFT_HAVE_* features? Just let each tests:
if ! "$BASEDIR/features/chain-binding" ; then
echo " defaultchain"
return
fi
then the checks are more flexible (arbitrary executables).
Downside, if the check is time consuming (which it shouldn't), then tests might call it over and over. Workaround for that: have "run- tests.sh" prepare a temporary directory and export it as $NFT_TEST_TMPDIR". Then "features/chain-binding" can cache the result there. "run-tests.sh" would delete the directory afterwards.
If you want to print those features, you can still let run-tests.sh iterate over "$BASEDIR"/features/* and print the result.
Quoted 9 lines
+
kernel_cleanup
[ "$failed" -eq 0 ]tests/shell/testcases/transactions/30s-stress +0
-0@@ -16,6 +16,18 @@ if [ x = x"$NFT" ] ; then
Quoted 6 lines
NFT=nft
fi
+if [ -z $NFT_HAVE_chain_binding ];then
+ NFT_HAVE_chain_binding=0
+ mydir=$(dirname $0)
I think run-tests.sh should export the base directory, like "$BASEDIR", or "$NFT_TEST_BASEDIR". Tests should use it (and rely to have it).
tests/shell/testcases/transactions/30s-stress
Quoted 8 lines
+ $NFT --check -f $mydir/../../features/chain_binding.nft + if [ $? -eq 0 ];then
+ NFT_HAVE_chain_binding=1
+ else
+ echo "Assuming anonymous chains are not supported" + fitests/shell/testcases/transactions/30s-stress +2
-0+ +fi
ah, you'd have each tests re-implement the check? So that they can run without the "run-tests.sh" wrapper?
I think that use-case should be dropped. The "run-tests.sh" wrapper can provide very useful features, and every test reimplementing that is wrong. Just accept that test scripts should not be run directly, then drop this [ -z $NFT_HAVE_chain_binding ] check.
Well, above I argue that "run-tests.sh" should not prepare "$NFT_HAVE_" variables, instead have features-detections plain shell scripts, that each test runs as needed.
The point here is, that if the "run-tests.sh" wrapper can provide something useful, then tests should be able to rely on it (and don't implement a fallback path).
Quoted 120 lines
+
testns=testns-$(mktemp -u "XXXXXXXX")
tmp=""tests/shell/testcases/transactions/30s-stress +0
-0@@ -31,8 +43,8 @@ failslab_defaults() { # allow all slabs to fail (if process is tagged).find
/sys/kernel/slab/-wholename '*/kmalloc-[0-9]*/failslab' -type f -exec sh -c 'echo 1 > {}' \;
- # no limit on the number of failures
- echo -1 >/sys/kernel/debug/failslab/times
+ # no limit on the number of failures, or clause works around old kernels that reject negative integer.
+ echo -1 >/sys/kernel/debug/failslab/times2>/dev/null || printf '%#x -1' >/sys/kernel/debug/failslab/times
# Set to 2 for full dmesg traces for each injected error echo 0 >/sys/kernel/debug/failslab/verbosetests/shell/testcases/transactions/30s-stress +0
-0@@ -91,6 +103,15 @@ nft_with_fault_inject()trap cleanup EXIT
tmp=$(mktemp)tests/shell/testcases/transactions/30s-stress +9
-0+jump_or_goto() +{ + if [ $((RANDOM & 1)) -eq 0 ] ;then + echo -n "jump" + else + echo -n "goto" + fi +} +random_verdict()
{
max="$1"tests/shell/testcases/transactions/30s-stress +0
-0@@ -102,7 +123,8 @@ random_verdict() rnd=$((RANDOM%max))
if [ $rnd -gt 0 ];then
- printf "jump chain%03u" "$((rnd+1))" + jump_or_goto
+ printf " chain%03u" "$((rnd+1))" return
fitests/shell/testcases/transactions/30s-stress +0
-0@@ -411,6 +433,21 @@ stress_all()randmonitor &
}tests/shell/testcases/transactions/30s-stress +15
-0+gen_anon_chain_jump() +{ + echo -n "insert rule inet $@ " + jump_or_goto + + if [ $NFT_HAVE_chain_binding -ne 1 ];then + echo " defaultchain" + return + fi + + echo -n " { " + jump_or_goto + echo " defaultchain; counter; }" +} +gen_ruleset() {
echo > "$tmp"
for table in $tables; dotests/shell/testcases/transactions/30s-stress +0
-0@@ -452,12 +489,13 @@ for table in $tables; do echo "insert rule inet $table $chain ip6 saddr { ::1,
dead::beef} counter" comment hash >> "$tmp"
echo "insert rule inet $table $chain ip saddr { 1.2.3.4 - 5.6.7.8, 127.0.0.1 } comment rbtree" >> "$tmp"# bitmap 1byte, with anon chain jump - echo "insert rule inet $table $chain ip protocol { 6, 17 } jump { jump defaultchain; counter; }" >> "$tmp" + gen_anon_chain_jump "$table $chain ip protocol { 6, 17 }" >> "$tmp" + # bitmap 2byte echo "insert rule inet $table $chain tcp dport != { 22, 23, 80 } goto defaultchain" >> "$tmp" echo "insert rule inet $table $chain tcp dport { 1-1024, 8000-8080 } jump defaultchain comment rbtree" >> "$tmp" # pipapo (concat + set), with goto anonymous chain. - echo "insert rule inet $table $chain ip saddr . tcp dport { 1.2.3.4 . 1-1024, 1.2.3.6 - 1.2.3.10 . 8000-8080, 1.2.3.4 . 8080, 1.2.3.6 - 1.2.3.10 . 22 } goto { jump defaultchain; counter; }" >> "$tmp" + gen_anon_chain_jump "$table $chain ip saddr . tcp dport { 1.2.3.4 . 1-1024, 1.2.3.6 - 1.2.3.10 . 8000-8080, 1.2.3.4 . 8080, 1.2.3.6 - 1.2.3.10 . 22 }" >> "$tmp" # add a few anonymous sets. rhashtable is convered by named sets below. c=$((RANDOM%$count))tests/shell/testcases/transactions/30s-stress +0
-0@@ -466,12 +504,12 @@ for table in $tables; do echo "insert rule inet $table $chain ip6 saddr { ::1,
dead::beef} counter" comment hash >> "$tmp"
echo "insert rule inet $table $chain ip saddr { 1.2.3.4 - 5.6.7.8, 127.0.0.1 } comment rbtree" >> "$tmp"# bitmap 1byte, with anon chain jump - echo "insert rule inet $table $chain ip protocol { 6, 17 } jump { jump defaultchain; counter; }" >> "$tmp" + gen_anon_chain_jump "$table $chain ip protocol { 6, 17 }" >> "$tmp" # bitmap 2byte echo "insert rule inet $table $chain tcp dport != { 22, 23, 80 } goto defaultchain" >> "$tmp" echo "insert rule inet $table $chain tcp dport { 1-1024, 8000-8080 } jump defaultchain comment rbtree" >> "$tmp" # pipapo (concat + set), with goto anonymous chain. - echo "insert rule inet $table $chain ip saddr . tcp dport { 1.2.3.4 . 1-1024, 1.2.3.6 - 1.2.3.10 . 8000-8080, 1.2.3.4 . 8080, 1.2.3.6 - 1.2.3.10 . 22 } goto { jump defaultchain; counter; }" >> "$tmp" + gen_anon_chain_jump "$table $chain ip saddr . tcp dport { 1.2.3.4 . 1-1024, 1.2.3.6 - 1.2.3.10 . 8000-8080, 1.2.3.4 . 8080, 1.2.3.6 - 1.2.3.10 . 22 }" >> "$tmp" # add constant/immutable sets size=$((RANDOM%5120000))
Florian Westphal
maintainerQuoted 9 lines · Thomas Haller
tests/shell/run-tests.sh ok=0 · line 172
Quoted 7 lines
tests/shell/run-tests.sh +7
-0+check_features() +{ + for ffilename in features/*.nft; do + feature=${ffilename#*/} + feature=${feature%*.nft} + eval NFT_HAVE_${feature}=0 + $NFT -f "$ffilename" 2>/dev/nullis the "--check" option here missing? At least, the commit message says
Yes, added.
Quoted 1 line
I think it should run in a separate netns too.
Should not make a difference due to --check, nothing is altered.
Quoted 4 lines
OK, that's nice, to see in the output.
But why this "nft -f" specific detection? Why not just executable scripts?
Because I want it to be simple, it will be enough to drop a ruleset with the feature into features/ durectory and run-tests.sh would do the reset.
While its possible to launch nft as a script via
#!/usr/bin/env nft
This would use the system nft, not the newly built one. And fudging with $PATH seems ugly...
Quoted 9 lines
Also, why should "
run-tests.sh" pre-evaluate those NFT_HAVE_* features? Just let each tests:if ! "$BASEDIR/features/chain-binding" ; then
echo " defaultchain"
return
fithen the checks are more flexible (arbitrary executables).
I could do that, but I don't see the need for arbitrary scripts so far.
Quoted 1 line
Downside, if the check is time consuming (which it shouldn't),
Agree, it should not.
Quoted 8 lines
then
tests might call it over and over. Workaround for that: have "run-tests.sh" prepare a temporary directory and export it as $NFT_TEST_TMPDIR". Then "features/chain-binding" can cache the result there. "run-tests.sh" would delete the directory afterwards.If you want to print those features, you can still let
run-tests.shiterate over "$BASEDIR"/features/* and print the result.
Yes, thats a working alternative but I don't see the need for this.
Quoted 7 lines
Quoted 7 lines
tests/shell/testcases/transactions/30s-stress +0
-0@@ -16,6 +16,18 @@ if [ x = x"$NFT" ] ; thenNFT=nft
Quoted 8 lines
Quoted 5 lines
fi
+if [ -z $NFT_HAVE_chain_binding ];then
+ NFT_HAVE_chain_binding=0
+ mydir=$(dirname $0)I think
run-tests.shshould export the base directory, like "$BASEDIR", or "$NFT_TEST_BASEDIR". Tests should use it (and rely to have it).
Can do that.
Quoted 2 lines
ah, you'd have each tests re-implement the check? So that they can run without the "
run-tests.sh" wrapper?
No, just 30s-stress. This test is special because its random by nature and it makes sense to run it for multiple hours once in a while.
Quoted 3 lines
The point here is, that if the "
run-tests.sh" wrapper can provide something useful, then tests should be able to rely on it (and don't implement a fallback path).
Yes, no other script will do this.
I'll send an updated batch soon to see how this looks like with more feature tests in place.
Quoted 7 lines · Florian Westphal
Quoted 4 lines · Thomas Haller
But why this "nft -f" specific detection? Why not just executable scripts?
Because I want it to be simple,
It does not seem "simple[r]" to me. The approach requires extra infrastructure in run-test.sh, while being less flexible.
Quoted 2 lines
I could do that, but I don't see the need for arbitrary scripts so far.
When building without JSON support, various tests fail, but should be skipped.
Could we detect JSON support via .nft files? Would we drop then a JSON .nft file and change the check call to nft --check -j?).
Or maybe detection of JSON support needs to be a shell script (doing ldd "$NFT_REAL" | greq libjansson)? In that case, we would have features-as-shell-scripts very soon.
Thomas
Florian Westphal
maintainerQuoted 11 lines · Thomas Haller
Quoted 7 lines · Florian Westphal
Quoted 4 lines · Thomas Haller
But why this "nft -f" specific detection? Why not just executable scripts?
Because I want it to be simple,
It does not seem "simple[r]" to me. The approach requires extra infrastructure in
run-test.sh, while being less flexible.
I can add bla.nft and use nft --check -f bla.nft.
Or, I can add bla.sh, which does
exec $NFT -f - <<EOF
table ...
EOF
I see zero reason why we can't add scripts later on if there are cases where flat-files don't work.
At this point, its just more boilerplate to add a script wrapper around the .nft file.
Quoted 8 lines
Quoted 2 lines
I could do that, but I don't see the need for arbitrary scripts so far.
When building without JSON support, various tests fail, but should be skipped.
Could we detect JSON support via .nft files? Would we drop then a JSON .nft file and change the check call to
nft --check -j?).
No, but the test that should be skipped can do
$NFT -j list ruleset || exit 77
as first line of the script, no need to load any files, nft will fail with error in case its not built with json support.
Quoted 3 lines
Or maybe detection of JSON support needs to be a shell script (doing
ldd "$NFT_REAL" | greq libjansson)? In that case, we would have features-as-shell-scripts very soon.
Sure, I see no reason why to not have both. The flat files have the '*nft' suffix for a reason...
I'll no longer work on this for the remainder of the month due to time constraints.
Pablo Neira Ayuso
maintainerQuoted 15 lines · Florian Westphal
Quoted 11 lines · Thomas Haller
Quoted 7 lines · Florian Westphal
Quoted 4 lines · Thomas Haller
But why this "nft -f" specific detection? Why not just executable scripts?
Because I want it to be simple,
It does not seem "simple[r]" to me. The approach requires extra infrastructure in
run-test.sh, while being less flexible.I can add bla.nft and use nft --check -f bla.nft.
Quoted 8 lines
Or, I can add
bla.sh, which doesexec $NFT -f - <<EOF
table ...
EOFI see zero reason why we can't add scripts later on if there are cases where flat-files don't work.
Agreed, we need this flexibility.
Quoted 12 lines
At this point, its just more boilerplate to add a script wrapper around the .nft file.
Quoted 8 lines
Quoted 2 lines
I could do that, but I don't see the need for arbitrary scripts so far.
When building without JSON support, various tests fail, but should be skipped.
Could we detect JSON support via .nft files? Would we drop then a JSON .nft file and change the check call to
nft --check -j?).
Quoted 6 lines
No, but the test that should be skipped can do
$NFT -j list ruleset || exit 77
as first line of the script, no need to load any files, nft will fail with error in case its not built with json support.
This is fine to start with.
Quoted 6 lines
Quoted 3 lines
Or maybe detection of JSON support needs to be a shell script (doing
ldd "$NFT_REAL" | greq libjansson)? In that case, we would have features-as-shell-scripts very soon.Sure, I see no reason why to not have both. The flat files have the '*nft' suffix for a reason...
I think this feature approach you propose is good enough and it is rather incremental and small.
Quoted 52 lines · Florian Westphal
Quoted 4 lines · Thomas Haller
Quoted 3 lines · Florian Westphal
Quoted 2 lines · Thomas Haller
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
But why this "nft -f" specific detection? Why not just executable
scripts?Because I want it to be simple,
It does not seem "simple[r]" to me. The approach requires extra infrastructure in
run-test.sh, while being less flexible.I can add bla.nft and use nft --check -f bla.nft.
Or, I can add
bla.sh, which doesexec $NFT -f - <<EOF
table ...
EOFI see zero reason why we can't add scripts later on if there are cases where flat-files don't work.
At this point, its just more boilerplate to add a script wrapper around the .nft file.
Quoted 11 lines
Quoted 3 lines
I could do that, but I don't see the need for arbitrary scripts so
far.When building without JSON support, various tests fail, but should be
skipped.Could we detect JSON support via .nft files? Would we drop then a JSON
.nft file and change the check call tonft --check -j?).No, but the test that should be skipped can do
$NFT -j list ruleset || exit 77
as first line of the script, no need to load any files, nft will fail with error in case its not built with json support.
Quoted 4 lines
Or maybe detection of JSON support needs to be a shell script (doing
ldd "$NFT_REAL" | greq libjansson)? In that case, we would have features-as-shell-scripts very soon.
Quoted 6 lines
Sure, I see no reason why to not have both. The flat files have the '*nft' suffix for a reason...
I'll no longer work on this for the remainder of the month due to time constraints.
Sounds all good! Thanks.
I go ahead and implement an early version of the "NFT_HAVE_json" feature. It can later be reconciled with your feature probing patch.
Thomas