nft-flowtable-sh flaking after pulling first chunk of the merge window
Subsystems (3)
Jakub Kicinski
maintainerHi!
Could be very bad luck but after we fast forwarded net-next yesterday we have 3 failures in less than 24h in nft_flowtabl.sh:
https://netdev.bots.linux.dev/contest.html?test=nft-flowtable-sh
# FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter 2113852 exceeds expected value 2097152, reply counter 60
https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh/stdout
# FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter 3530493 exceeds expected value 3478585, reply counter 60
https://netdev-3.bots.linux.dev/vmksft-nf/results/960022/10-nft-flowtable-sh/stdout
# FAIL: dscp counters do not match, expected dscp3 and dscp0 > 0 but got 1431 , 0
https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh-retry/stdout
Pablo Neira Ayuso
maintainerHi Jakub,
Quoted 7 lines · Jakub Kicinski
Hi!
Could be very bad luck but after we fast forwarded net-next yesterday we have 3 failures in less than 24h in
nft_flowtabl.sh:https://netdev.bots.linux.dev/contest.html?test=nft-flowtable-sh
Quoted 8 lines
# FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter 2113852 exceeds expected value 2097152, reply counter 60
https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh/stdout# FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter 3530493 exceeds expected value 3478585, reply counter 60
https://netdev-3.bots.linux.dev/vmksft-nf/results/960022/10-nft-flowtable-sh/stdout# FAIL: dscp counters do not match, expected dscp3 and dscp0 > 0 but got 1431 , 0
https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh-retry/stdout
Thanks for your report, let me take a look.
Pablo Neira Ayuso
maintainerHi Jakub,
Quoted 5 lines · Jakub Kicinski
Hi!
Could be very bad luck but after we fast forwarded net-next yesterday we have 3 failures in less than 24h in
nft_flowtabl.sh:
Quoted 7 lines
https://netdev.bots.linux.dev/contest.html?test=nft-flowtable-sh
# FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter 2113852 exceeds expected value 2097152, reply counter 60
https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh/stdout# FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter 3530493 exceeds expected value 3478585, reply counter 60
https://netdev-3.bots.linux.dev/vmksft-nf/results/960022/10-nft-flowtable-sh/stdout
this is reporting a flow in forward chain going over the size of the file, this is a flow that is not follow flowtable path.
Quoted 2 lines
# FAIL: dscp counters do not match, expected dscp3 and dscp0 > 0 but got 1431 , 0
https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh-retry/stdout
this is reporting that occasionally a flow does not follow flowtable path, dscp3 gets bumped from the forward chain.
I can rarely see this last dscp tests FAIL when running this test in a loop here.
Just a follow up, I am still diagnosing.
Jakub Kicinski
maintainerQuoted 17 lines · On Wed, 29 Jan 2025 12:21:24 +0100 Pablo Neira Ayuso wrote:
Quoted 10 lines
Could be very bad luck but after we fast forwarded net-next yesterday we have 3 failures in less than 24h in
nft_flowtabl.sh:https://netdev.bots.linux.dev/contest.html?test=nft-flowtable-sh
# FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter 2113852 exceeds expected value 2097152, reply counter 60
https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh/stdout# FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter 3530493 exceeds expected value 3478585, reply counter 60
https://netdev-3.bots.linux.dev/vmksft-nf/results/960022/10-nft-flowtable-sh/stdoutthis is reporting a flow in forward chain going over the size of the file, this is a flow that is not follow flowtable path.
Quoted 2 lines
# FAIL: dscp counters do not match, expected dscp3 and dscp0 > 0 but got 1431 , 0
https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh-retry/stdout
Quoted 7 lines
this is reporting that occasionally a flow does not follow flowtable path, dscp3 gets bumped from the forward chain.
I can rarely see this last dscp tests FAIL when running this test in a loop here.
Just a follow up, I am still diagnosing.
Thanks for the update!
FWIW we hit 4 more flakes since I reported it to you last week (first link from previous message will take you to them). All four in dscp_fwd
Pablo Neira Ayuso
maintainerHi Jakub,
Quoted 24 lines · Jakub Kicinski
Quoted 23 lines · On Wed, 29 Jan 2025 12:21:24 +0100 Pablo Neira Ayuso wrote:
Quoted 10 lines
Could be very bad luck but after we fast forwarded net-next yesterday we have 3 failures in less than 24h in
nft_flowtabl.sh:https://netdev.bots.linux.dev/contest.html?test=nft-flowtable-sh
# FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter 2113852 exceeds expected value 2097152, reply counter 60
https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh/stdout# FAIL: flow offload for ns1/ns2 with masquerade and pmtu discovery : original counter 3530493 exceeds expected value 3478585, reply counter 60
https://netdev-3.bots.linux.dev/vmksft-nf/results/960022/10-nft-flowtable-sh/stdoutthis is reporting a flow in forward chain going over the size of the file, this is a flow that is not follow flowtable path.
Quoted 2 lines
# FAIL: dscp counters do not match, expected dscp3 and dscp0 > 0 but got 1431 , 0
https://netdev-3.bots.linux.dev/vmksft-nf/results/960740/11-nft-flowtable-sh-retry/stdoutthis is reporting that occasionally a flow does not follow flowtable path, dscp3 gets bumped from the forward chain.
I can rarely see this last dscp tests FAIL when running this test in a loop here.
Quoted 7 lines
Quoted 1 line
Just a follow up, I am still diagnosing.
Thanks for the update!
FWIW we hit 4 more flakes since I reported it to you last week (first link from previous message will take you to them). All four in
dscp_fwd
Just another follow up on this. I am testing here a revert of:
b8baac3b9c5c ("netfilter: flowtable: teardown flow if cached mtu is stale")
nft_flowtable.sh shows too frequent re-offloads (create/teardown cycles) with fragments that can lead no packets following the flowtable path as dscp_fwd reports.
Let me give it more testing then, if results are positive, I will formally propose this revert.
Thanks.