selftests: openvswitch: Address some flakes in the CI environment
Subsystems (4)
Aaron Conole
maintainercover selftests: openvswitch: Address some flakes in the CI environment
These patches aim to make using the openvswitch testsuite more reliable. These should address the major sources of flakiness in the openvswitch test suite allowing the CI infrastructure to exercise the openvswitch module for patch series. There should be no change for users who simply run the tests (except that patch 3/3 does make some of the debugging a bit easier by making some output more verbose).
Aaron Conole (3):
selftests: openvswitch: Bump timeout to 15 minutes.
selftests: openvswitch: Attempt to autoload module.
selftests: openvswitch: Be more verbose with selftest debugging.
.../selftests/net/openvswitch/openvswitch.sh | 23 ++++++++++++------- .../selftests/net/openvswitch/settings | 1 + 2 files changed, 16 insertions(+), 8 deletions(-) create mode 100644 tools/testing/selftests/net/openvswitch/settings
Signature · 1 line
2.45.1
Aaron Conole
maintainerPATCH 1/3 selftests: openvswitch: Bump timeout to 15 minutes.
We found that since some tests rely on the TCP SYN timeouts to cause flow misses, the default test suite timeout of 45 seconds is quick to be exceeded. Bump the timeout to 15 minutes.
--- tools/testing/selftests/net/openvswitch/settings | 1 + 1 file changed, 1 insertion(+) create mode 100644 tools/testing/selftests/net/openvswitch/settings
tools/testing/selftests/net/openvswitch/settings new file +1 -0
@@ -0,0 +1 @@ +timeout=900
Signature · 1 line
2.45.1
Re: [PATCH net-next 1/3] selftests: openvswitch: Bump timeout to 15 minutes.
Quoted 5 lines · Aaron Conole
We found that since some tests rely on the TCP SYN timeouts to cause flow misses, the default test suite timeout of 45 seconds is quick to be exceeded. Bump the timeout to 15 minutes.
Signed-off-by Aaron Conole <[email protected]>
FWIIW, locally I had been using a timeout of 720s. So 900 seems entirely reasonable to me.
Aaron Conole
maintainerPATCH 2/3 selftests: openvswitch: Attempt to autoload module.
In reply to Aaron ConolePreviously, the openvswitch.sh test suites would not attempt to autoload the openvswitch module. The idea was that a user who is manually running tests might not even have the OVS module loaded or configured for their own development. However, if the kernel module is configured, and the module can be autoloaded then we should just attempt to load it and run the tests. This is especially true in the CI environments, where the CI tests should be able to rely on auto loading to get the test suite running.
--- .../selftests/net/openvswitch/openvswitch.sh | 14 +++++++++----- 1 file changed, 9 insertions(+), 5 deletions(-)
tools/testing/selftests/net/openvswitch/openvswitch.sh +9 -5
@@ -613,16 +613,20 @@ run_test() { tname="$1" tdesc="$2" - if ! lsmod | grep openvswitch >/dev/null 2>&1; then - stdbuf -o0 printf "TEST: %-60s [NOMOD]\n" "${tdesc}" - return $ksft_skip - fi - if python3 ovs-dpctl.py -h 2>&1 | \ grep -E "Need to (install|upgrade) the python" >/dev/null 2>&1; then stdbuf -o0 printf "TEST: %-60s [PYLIB]\n" "${tdesc}" return $ksft_skip fi + + python3 ovs-dpctl.py show >/dev/null 2>&1 || \ + echo "[DPCTL] show exception." + + if ! lsmod | grep openvswitch >/dev/null 2>&1; then + stdbuf -o0 printf "TEST: %-60s [NOMOD]\n" "${tdesc}" + return $ksft_skip + fi + printf "TEST: %-60s [START]\n" "${tname}" unset IFS
Signature · 1 line
2.45.1
Re: [PATCH net-next 2/3] selftests: openvswitch: Attempt to autoload module.
Quoted 9 lines · Aaron Conole
Previously, the
openvswitch.shtest suites would not attempt to autoload the openvswitch module. The idea was that a user who is manually running tests might not even have the OVS module loaded or configured for their own development. However, if the kernel module is configured, and the module can be autoloaded then we should just attempt to load it and run the tests. This is especially true in the CI environments, where the CI tests should be able to rely on auto loading to get the test suite running.Signed-off-by Aaron Conole <[email protected]>
Aaron Conole
maintainerPATCH 3/3 selftests: openvswitch: Be more verbose with selftest debugging.
In reply to Aaron ConoleThe openvswitch selftest is difficult to debug for anyone that isn't directly familiar with the openvswitch module and the specifics of the test cases. Many times when something fails, the debug log will be sparsely populated and it takes some time to understand where a failure occured.
Increase the amount of details logged to the debug log by trapping all 'info' logs, and all 'ovs_sbx' commands.
---
NOTE: There is a conflict here with a patch on list that adds psample
support, but it should be simple to resolve, since the conflict would be due to a context change in tests="". I can also respin if the patches collide.
tools/testing/selftests/net/openvswitch/openvswitch.sh | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-)
tools/testing/selftests/net/openvswitch/openvswitch.sh +6 -3
@@ -23,7 +23,9 @@ tests=" drop_reason drop: test drop reasons are emitted" info() { - [ $VERBOSE = 0 ] || echo $* + [ "${ovs_dir}" != "" ] && + echo "`date +"[%m-%d %H:%M:%S]"` $*" >> ${ovs_dir}/debug.log + [ $VERBOSE = 0 ] || echo $* } ovs_base=`pwd`
@@ -65,7 +67,8 @@ ovs_setenv() { ovs_sbx() { if test "X$2" != X; then - (ovs_setenv $1; shift; "$@" >> ${ovs_dir}/debug.log) + (ovs_setenv $1; shift; + info "run cmd: $@"; "$@" >> ${ovs_dir}/debug.log) else ovs_setenv $1 fi
@@ -139,7 +142,7 @@ ovs_add_flow () { info "Adding flow to DP: sbx:$1 br:$2 flow:$3 act:$4" ovs_sbx "$1" python3 $ovs_base/ovs-dpctl.py add-flow "$2" "$3" "$4" if [ $? -ne 0 ]; then - echo "Flow [ $3 : $4 ] failed" >> ${ovs_dir}/debug.log + info "Flow [ $3 : $4 ] failed" return 1 fi return 0
Signature · 1 line
2.45.1
Re: [PATCH net-next 3/3] selftests: openvswitch: Be more verbose with selftest debugging.
Quoted 10 lines · Aaron Conole
The openvswitch selftest is difficult to debug for anyone that isn't directly familiar with the openvswitch module and the specifics of the test cases. Many times when something fails, the debug log will be sparsely populated and it takes some time to understand where a failure occured.
Increase the amount of details logged to the debug log by trapping all 'info' logs, and all '
ovs_sbx' commands.Signed-off-by Aaron Conole <[email protected]>
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by Jakub Kicinski <[email protected]>:
Quoted 8 lines · On Tue, 2 Jul 2024 09:28:27 -0400 you wrote:
These patches aim to make using the openvswitch testsuite more reliable. These should address the major sources of flakiness in the openvswitch test suite allowing the CI infrastructure to exercise the openvswitch module for patch series. There should be no change for users who simply run the tests (except that patch 3/3 does make some of the debugging a bit easier by making some output more verbose).
[...]
Here is the summary with links:
- [net-next,1/3] selftests: openvswitch: Bump timeout to 15 minutes.
https://git.kernel.org/netdev/net-next/c/ff015706fc73 - [net-next,2/3] selftests: openvswitch: Attempt to autoload module. https://git.kernel.org/netdev/net-next/c/818481db3df4 - [net-next,3/3] selftests: openvswitch: Be more verbose with selftest debugging. https://git.kernel.org/netdev/net-next/c/7abfd8ecb785
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Jakub Kicinski
maintainerQuoted 6 lines · On Tue, 2 Jul 2024 09:28:27 -0400 Aaron Conole wrote:
These patches aim to make using the openvswitch testsuite more reliable. These should address the major sources of flakiness in the openvswitch test suite allowing the CI infrastructure to exercise the openvswitch module for patch series. There should be no change for users who simply run the tests (except that patch 3/3 does make some of the debugging a bit easier by making some output more verbose).
Hi Aaron!
The results look solid on normal builds now, but with a debug kernel the test is failing consistently:
https://netdev.bots.linux.dev/contest.html?executor=vmksft-net-dbg&test=openvswitch-sh
Aaron Conole
maintainerQuoted 8 lines · Jakub Kicinski
Quoted 6 lines · On Tue, 2 Jul 2024 09:28:27 -0400 Aaron Conole wrote:
These patches aim to make using the openvswitch testsuite more reliable. These should address the major sources of flakiness in the openvswitch test suite allowing the CI infrastructure to exercise the openvswitch module for patch series. There should be no change for users who simply run the tests (except that patch 3/3 does make some of the debugging a bit easier by making some output more verbose).
Quoted 6 lines
Hi Aaron!
The results look solid on normal builds now, but with a debug kernel the test is failing consistently:
https://netdev.bots.linux.dev/contest.html?executor=vmksft-net-dbg&test=openvswitch-sh
Yes - it shows a test case issue with the upcall and psample tests.
Adrian and I discussed the correct approach would be using a wait_for instead of just sleeping, because it seems the dbg environment might be too racy. I think he is working on a follow up to submit after the psample work gets merged - we were hoping not to hold that patch series up with more potential conflicts or merge issues if that's okay.
Jakub Kicinski
maintainerQuoted 5 lines · On Fri, 05 Jul 2024 09:49:12 -0400 Aaron Conole wrote:
Quoted 4 lines
The results look solid on normal builds now, but with a debug kernel the test is failing consistently:
https://netdev.bots.linux.dev/contest.html?executor=vmksft-net-dbg&test=openvswitch-sh
Quoted 7 lines
Yes - it shows a test case issue with the upcall and psample tests.
Adrian and I discussed the correct approach would be using a
wait_forinstead of just sleeping, because it seems the dbg environment might be too racy. I think he is working on a follow up to submit after the psample work gets merged - we were hoping not to hold that patch series up with more potential conflicts or merge issues if that's okay.
Makes sense, thanks!
Quoted 17 lines · Aaron Conole
Quoted 8 lines · Jakub Kicinski
Quoted 6 lines · On Tue, 2 Jul 2024 09:28:27 -0400 Aaron Conole wrote:
These patches aim to make using the openvswitch testsuite more reliable. These should address the major sources of flakiness in the openvswitch test suite allowing the CI infrastructure to exercise the openvswitch module for patch series. There should be no change for users who simply run the tests (except that patch 3/3 does make some of the debugging a bit easier by making some output more verbose).
Quoted 6 lines
Hi Aaron!
The results look solid on normal builds now, but with a debug kernel the test is failing consistently:
https://netdev.bots.linux.dev/contest.html?executor=vmksft-net-dbg&test=openvswitch-sh
Quoted 8 lines
Yes - it shows a test case issue with the upcall and psample tests.
Adrian and I discussed the correct approach would be using a
wait_forinstead of just sleeping, because it seems the dbg environment might be too racy. I think he is working on a follow up to submit after the psample work gets merged - we were hoping not to hold that patch series up with more potential conflicts or merge issues if that's okay.
Yes. I am working on a patch to solve the failures in slow systems.
Thanks.
Adrián