selftests: mlxsw: Stabilize RED tests

Tweak the mlxsw-specific RED selftests to increase stability on Spectrum-3 and Spectrum-4 machines.

Petr Machata (5):
selftests: mlxsw: sch_red_ets: Increase required backlog
selftests: mlxsw: sch_red_core: Increase backlog size tolerance
selftests: mlxsw: sch_red_core: Sleep before querying queue depth
selftests: mlxsw: sch_red_core: Send more packets for drop tests
selftests: mlxsw: sch_red_core: Lower TBF rate

 .../drivers/net/mlxsw/sch_red_core.sh         | 28 +++++++++++--------
 .../drivers/net/mlxsw/sch_red_ets.sh          |  8 +++---
 2 files changed, 20 insertions(+), 16 deletions(-)

2.45.0

Backlog fluctuates on Spectrum-4 much more than on <4. Increasing the desired backlog seems to help, as the constant fluctuations do not overlap into the territory where packets are marked.

---
 tools/testing/selftests/drivers/net/mlxsw/sch_red_ets.sh | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

index 8ecddafa79b3..576067b207a8 100755
@@ -20,8 +20,8 @@ source sch_red_core.sh
 # $BACKLOG2 are far enough not to overlap, so that we can assume that if we do
 # see (do not see) marking, it is actually due to the configuration of that one
 # TC, and not due to configuration of the other TC leaking over.
-BACKLOG1=200000
-BACKLOG2=500000
+BACKLOG1=400000
+BACKLOG2=1000000
 
 install_root_qdisc()
 {

@@ -35,7 +35,7 @@ install_qdisc_tc0()
 
 	tc qdisc add dev $swp3 parent 10:8 handle 108: red \
 	   limit 1000000 min $BACKLOG1 max $((BACKLOG1 + 1)) \
-	   probability 1.0 avpkt 8000 burst 38 "${args[@]}"
+	   probability 1.0 avpkt 8000 burst 51 "${args[@]}"
 }
 
 install_qdisc_tc1()

@@ -44,7 +44,7 @@ install_qdisc_tc1()
 
 	tc qdisc add dev $swp3 parent 10:7 handle 107: red \
 	   limit 1000000 min $BACKLOG2 max $((BACKLOG2 + 1)) \
-	   probability 1.0 avpkt 8000 burst 63 "${args[@]}"
+	   probability 1.0 avpkt 8000 burst 126 "${args[@]}"
 }
 
 install_qdisc()

2.45.0

Backlog fluctuates on Spectrum-4 much more than on <4. In practice we can sample queue depth values going from about -12% to about +7% of the configured RED limit. The test which checks the queue size has a limit of +-10%, and as a result often fails. We attempted to fix the issue by busywaiting for several seconds hoping to get within the bounds, but that still proved to be too noisy (or the wait time would be impractically long). Unfortunately we have to bump the value tolerance from 10% to 15%, which in this patch do.

---
 tools/testing/selftests/drivers/net/mlxsw/sch_red_core.sh | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

index 299e06a5808c..a25a15eb6d31 100644
@@ -532,10 +532,11 @@ do_red_test()
 	check_fail $? "Traffic went into backlog instead of being early-dropped"
 	pct=$(check_marking get_nmarked $vlan "== 0")
 	check_err $? "backlog $backlog / $limit Got $pct% marked packets, expected == 0."
+	backlog=$(get_qdisc_backlog $vlan)
 	local diff=$((limit - backlog))
 	pct=$((100 * diff / limit))
-	((-10 <= pct && pct <= 10))
-	check_err $? "backlog $backlog / $limit expected <= 10% distance"
+	((-15 <= pct && pct <= 15))
+	check_err $? "backlog $backlog / $limit expected <= 15% distance"
 	log_test "TC $((vlan - 10)): RED backlog > limit"
 
 	stop_traffic

2.45.0

The qdisc stats are taken from the port's periodic HW stats, which are updated once a second. We try to accommodate the latency by using busywait in build_backlog().

The issue in that seems to be that when do_mark_test() builds the backlog, it makes the decision whether to send more packets based on the first instance of the queue depth stat exceeding the current value, when in fact more traffic is on the way and the queue depth would increase further. This leads to failures in TC 1 of mark-mirror test, where we see the following failure:

TEST: TC 0: marked packets mirror'd [ OK ]
TEST: TC 1: marked packets mirror'd [FAIL]

        Spurious packets (1680 -> 2290) observed without buffer pressure

Fix by waiting for the full second before reading the queue depth for the first time, to make sure it reflects all in-flight traffic.

---
 tools/testing/selftests/drivers/net/mlxsw/sch_red_core.sh | 1 +
 1 file changed, 1 insertion(+)

index a25a15eb6d31..b1ff395b3880 100644
@@ -372,6 +372,7 @@ build_backlog()
 	local i=0
 
 	while :; do
+		sleep 1
 		local cur=$(busywait 1100 until_counter_is "> $cur" \
 					    get_qdisc_backlog $vlan)
 		local diff=$((size - cur))

2.45.0

This test works by injecting into a port with a maxed-out queue a couple packets and checks if a corresponding number of packets were dropped. This has worked well on Spectrum<4, but on Spectrum-4 it has been noisy. This is in line with the observation that on Spectrum-4, queue size tends to fluctuate more. A handful of packets could then still be accepted to the queue even though it was nominally full just recently.

In order to accommodate this behavior, send many more packets. The buffer can fit N extra packets, but not N% packets. This therefore allows us to set wider absolute margins, while actually narrowing them relatively.

---
 .../selftests/drivers/net/mlxsw/sch_red_core.sh  | 16 +++++++++-------
 1 file changed, 9 insertions(+), 7 deletions(-)

index b1ff395b3880..316444389c4e 100644
@@ -653,20 +653,22 @@ do_drop_test()
 	build_backlog $vlan $((3 * limit / 2)) udp >/dev/null
 
 	base=$($fetch_counter)
-	send_packets $vlan udp 11
+	send_packets $vlan udp 100
 
-	now=$(busywait 1100 until_counter_is ">= $((base + 10))" $fetch_counter)
-	check_err $? "Dropped packets not observed: 11 expected, $((now - base)) seen"
+	now=$(busywait 1100 until_counter_is ">= $((base + 95))" $fetch_counter)
+	check_err $? "${trigger}ped packets not observed: 100 expected, $((now - base)) seen"
 
 	# When no extra traffic is injected, there should be no mirroring.
-	busywait 1100 until_counter_is ">= $((base + 20))" $fetch_counter >/dev/null
+	busywait 1100 until_counter_is ">= $((base + 110))" \
+		 $fetch_counter >/dev/null
 	check_fail $? "Spurious packets observed"
 
 	# When the rule is uninstalled, there should be no mirroring.
 	qevent_rule_uninstall_$subtest
-	send_packets $vlan udp 11
-	busywait 1100 until_counter_is ">= $((base + 20))" $fetch_counter >/dev/null
-	check_fail $? "Spurious packets observed after uninstall"
+	send_packets $vlan udp 100
+	now=$(busywait 1100 until_counter_is ">= $((base + 110))" \
+		       $fetch_counter)
+	check_fail $? "$((now - base)) spurious packets observed after uninstall"
 
 	log_test "TC $((vlan - 10)): ${trigger}ped packets $subtest'd"
 

2.45.0

The RED test uses a pair of TBF shapers. The first to get predictably-sized stream of traffic, and second to get a 100% saturated chokepoint. To this chokepoint it injects individual packets. Because the chokepoint is saturated, these additional packets go straight to the backlog. This allows the test to check RED behavior across various queue sizes.

The shapers are rated at 1Gbps, for historical reasons (before mlxsw supported TBF offload, the test used port speed to create the chokepoints). Machines with a low-power CPU may have trouble consistently generating 1Gbps of traffic, and the test then spuriously fails.

Instead, drop the rate to 200Mbps (Spectrum has a guaranteed shaper rate granularity of 200Mbps, so anything lower is not guaranteed to work well). Because that means fewer packets will be mirrored in the ECN-mark test, adjust the passing condition accordingly.

---
 tools/testing/selftests/drivers/net/mlxsw/sch_red_core.sh | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

index 316444389c4e..f4c324957dcc 100644
@@ -137,7 +137,7 @@ h2_create()
 	# Prevent this by adding a shaper which limits the traffic in $h2 to
 	# 1Gbps.
 
-	tc qdisc replace dev $h2 root handle 10: tbf rate 1gbit \
+	tc qdisc replace dev $h2 root handle 10: tbf rate 200mbit \
 		burst 128K limit 1G
 }
 

@@ -199,7 +199,7 @@ switch_create()
 	done
 
 	for intf in $swp3 $swp4; do
-		tc qdisc replace dev $intf root handle 1: tbf rate 1gbit \
+		tc qdisc replace dev $intf root handle 1: tbf rate 200mbit \
 			burst 128K limit 1G
 	done
 

@@ -602,7 +602,7 @@ do_mark_test()
 	# Above limit, everything should be mirrored, we should see lots of
 	# packets.
 	build_backlog $vlan $((3 * limit / 2)) tcp tos=0x01 >/dev/null
-	busywait_for_counter 1100 +10000 \
+	busywait_for_counter 1100 +2500 \
 		 $fetch_counter > /dev/null
 	check_err_fail "$should_fail" $? "ECN-marked packets $subtest'd"
 

2.45.0