conntrack_reverse_clash.sh flakes

Hi Florian!

We have a new faster NIPA setup, and now on non-debug builds we see a few (4 a week to be exact) flakes in conntrack_reverse_clash.sh

List of flakes from the last 100 runs:
https://netdev.bots.linux.dev/contest.html?pass=0&test=conntrack-reverse-clash-sh

Example:

# selftests: net/netfilter: conntrack_reverse_clash.sh
# Port number changed, wanted 56789 got 5950
# ERROR: SNAT performed without any matching snat rule
# kill: sending signal to 16051 failed: No such process
not ok 1 selftests: net/netfilter: conntrack_reverse_clash.sh # exit=1

Looks like the test also occasionally flaked on the old setup ("remote" column with "metal" instead of "virt") which is now shut down:

# selftests: net/netfilter: conntrack_reverse_clash.sh
# Port number changed, wanted 56789 got 54630
# Port number changed, wanted 56790 got 25814
# ERROR: SNAT performed without any matching snat rule
not ok 1 selftests: net/netfilter: conntrack_reverse_clash.sh # exit=1

so this isn't new, just more likely now..

Could you TAL when you have spare cycles? (BTW the new setup is owned by netdev foundation so I can give you access if that helps).

Ah, one more, the non-reverse conntrack-clash is SKIPping, occasionally:
https://netdev.bots.linux.dev/contest.html?pass=0&test=conntrack-clash-sh

If the event it's testing is probabilistic could we make it return XFAIL when it doesn't trigger? We try to reserve SKIP for tests skipped because tool is missing in env, something isn't built into the kernel etc.

Sure, I can switch this to xfail. Its trying to trigger a conntrack race but there is no way to force this and I did not want the script to spin for long when trying to trigger the nat clash undo logic.

No need, I can reproduce this:
# selftests: conntrack_reverse_clash.sh
# Port number changed, wanted 56790 got 64562 from 127.0.0.12 # ERROR: SNAT performed without any matching snat rule

# udp      17 30 src=127.0.0.11 dst=127.0.0.12 sport=56789 dport=56790 [UNREPLIED] src=127.0.0.12 dst=127.0.0.11 sport=56790 dport=56789 mark=0 use=1
# conntrack v1.4.8 (conntrack-tools): 1 flow entries have been shown.
# cpu=0         found=0 invalid=0 insert=0 insert_failed=0 drop=0 early_drop=0 error=0 search_restart=0 clash_resolve=0 chaintoolong=0
...

Looks like an actual bug to me, will need some time to investigate this. If its too annoying consider disabling this particular test for now.

Thanks for reporting.