Skip to contentSkip to conversation

netfilter: nft_set_pipapo_avx2: restore performance optimization

PATCH Superseded Florian Westphal 2 participants 4 messages Apr 1 – Apr 12

Subsystems (2)

v1 of 2 Superseded Latest is v2 →

lore
Share & tools
lore

Florian Westphal

author maintainer
#0 lore raw

PATCH 1/1 netfilter: nft_set_pipapo_avx2: restore performance optimization

The avx2 lookup routines get the next map index to process passes as a function argument, but this isn't bery obvious because its hidden in the lookup macro.

In commit 17a20e09f086 ("netfilter: nft_set: remove one argument from lookup and update functions") I incorrectly moved the "ret" scope into the loop.

This has no effect on the correctness, but it can (depending on map sizes) cause a redundant repeat of an earlier processing step.

Restore the intended 'pass map index' instead of always-0. Note that I did not see any change in performance numbers, so an alternative would be to axe this optimization and go with slightly simpler code instead.

Additionally, a recent LLM review pointed out following "bug":
-------------------------------------------------------------

Review

Quoted 7 lines
              b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
              if (last)
(quoted diff) +1 -1
-                     return b;
+                     ret = b;
 
              if (unlikely(ret == -1))
                      ret = b / XSAVE_YMM_SIZE;

Does this change introduce a logic error when last=true and no match is found? The old code used 'return b;' which immediately exited the loop. The new code changes this to 'ret = b;' to allow loop continuation, but when last=true and nft_pipapo_avx2_refill() returns -1 (no match found), the execution flow becomes:

1. ret = -1 (from 'if (last) ret = b;')
2. The condition 'if (unlikely(ret == -1))' evaluates to TRUE
3. ret = -1 / XSAVE_YMM_SIZE = -1 / 32 = 0 (integer division)
4. Loop continues with ret=0

[..]

Should this be changed to an else-if structure instead?
-------------------------------------------------------------

All call sites invoke nft_pipapo_avx2_refill() only when at least one bit in the map is set, i.e. nft_pipapo_avx2_refill() never returns -1.

Add a runtime debug check that fires if we'd return -1 as additional documentation and also make the suggested change, code might be easier to understand this way.

---
 net/netfilter/nft_set_pipapo_avx2.c | 35 ++++++++++++-----------------
 1 file changed, 14 insertions(+), 21 deletions(-)
net/netfilter/nft_set_pipapo_avx2.c +14 -21
index dad265807b8b..b3f105520a85 100644
@@ -144,6 +144,7 @@ static void nft_pipapo_avx2_fill(unsigned long *data, int start, int len)
  * This is an alternative implementation of pipapo_refill() suitable for usage
  * with AVX2 lookup routines: we know there are four words to be scanned, at
  * a given offset inside the map, for each matching iteration.
+ * The caller must ensure at least one bit in the four words is set.
  *
  * This function doesn't actually use any AVX2 instruction.
  *
@@ -179,6 +180,7 @@ static int nft_pipapo_avx2_refill(int offset, unsigned long *map,
 	NFT_PIPAPO_AVX2_REFILL_ONE_WORD(3);
 #undef NFT_PIPAPO_AVX2_REFILL_ONE_WORD
 
+	DEBUG_NET_WARN_ON_ONCE(ret < 0);
 	return ret;
 }
 
@@ -243,8 +245,7 @@ static int nft_pipapo_avx2_lookup_4b_2(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -320,8 +321,7 @@ static int nft_pipapo_avx2_lookup_4b_4(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -415,8 +415,7 @@ static int nft_pipapo_avx2_lookup_4b_8(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -506,8 +505,7 @@ static int nft_pipapo_avx2_lookup_4b_12(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -642,8 +640,7 @@ static int nft_pipapo_avx2_lookup_4b_32(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -700,8 +697,7 @@ static int nft_pipapo_avx2_lookup_8b_1(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -765,8 +761,7 @@ static int nft_pipapo_avx2_lookup_8b_2(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -840,8 +835,7 @@ static int nft_pipapo_avx2_lookup_8b_4(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -926,8 +920,7 @@ static int nft_pipapo_avx2_lookup_8b_6(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -1020,8 +1013,7 @@ static int nft_pipapo_avx2_lookup_8b_16(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -1143,6 +1135,7 @@ struct nft_pipapo_elem *pipapo_get_avx2(const struct nft_pipapo_match *m,
 	const struct nft_pipapo_field *f;
 	unsigned long *res, *fill, *map;
 	bool map_index;
+	int ret = 0;
 	int i;
 
 	scratch = *raw_cpu_ptr(m->scratch);
@@ -1167,8 +1160,8 @@ struct nft_pipapo_elem *pipapo_get_avx2(const struct nft_pipapo_match *m,
 
 	nft_pipapo_for_each_field(f, i, m) {
 		bool last = i == m->field_count - 1, first = !i;
-		int ret = 0;
 
+		/* NB: previous round @ret is passed to avx2 lookup fn */
 #define NFT_SET_PIPAPO_AVX2_LOOKUP(b, n)				\
 		(ret = nft_pipapo_avx2_lookup_##b##b_##n(res, fill, f,	\
 							 ret, data,	\
Signature · 1 line
2.52.0
Reviewed-by on v1 1/1
#1 lore raw

Sorry for the late review.

Quoted 2 lines · On Wed, 1 Apr 2026 13:02:27 +0200 Florian Westphal <[email protected]> wrote:

The avx2 lookup routines get the next map index to process passes as a function argument, but this isn't bery obvious because its hidden in the

Nits: very, it's

Quoted 1 line

lookup macro.

...right, I'm not exactly proud of it, but if we don't hide that stuff the amount of visual noise explodes. I couldn't think of any better solution.

Quoted 3 lines

In commit 17a20e09f086 ("netfilter: nft_set: remove one argument from lookup and update functions") I incorrectly moved the "ret" scope into the loop.

This took me quite a bit to review because I stupidly assumed that that commit was the same as the "return b;" -> "ret = b;" recent change :) ...that part is my fault of course, I didn't read the reference.

I really couldn't understand how turning those into "else if" would have anything to do with the scope "ret"... until I realised and actually read your reference. Oops.

But I have to say this was triggered by reading the next part which had little to do with this...

Quoted 7 lines

This has no effect on the correctness, but it can (depending on map sizes) cause a redundant repeat of an earlier processing step.

Restore the intended 'pass map index' instead of always-0. Note that I did not see any change in performance numbers, so an alternative would be to axe this optimization and go with slightly simpler code instead.

This probably depends a lot on the data set we're using. I'm not sure if the current performance tests would consistently trigger a sparse intermediate matching bitmap (between fields) with a lot of leading zeroes.

By the way, I don't think a branch predictor could figure that out. So all in all I would keep the optimisation you're restoring here.

Quoted 21 lines

Additionally, a recent LLM review pointed out following "bug":
-------------------------------------------------------------

Review

Quoted 7 lines
              b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
              if (last)
(quoted diff) +1 -1
-                     return b;
+                     ret = b;
 
              if (unlikely(ret == -1))
                      ret = b / XSAVE_YMM_SIZE;  

Does this change introduce a logic error when last=true and no match is found? The old code used 'return b;' which immediately exited the loop. The new code changes this to 'ret = b;' to allow loop continuation, but when last=true and nft_pipapo_avx2_refill() returns -1 (no match found), the execution flow becomes:

1. ret = -1 (from 'if (last) ret = b;')
2. The condition 'if (unlikely(ret == -1))' evaluates to TRUE
3. ret = -1 / XSAVE_YMM_SIZE = -1 / 32 = 0 (integer division)
4. Loop continues with ret=0

Quoted 7 lines

[..]

Should this be changed to an else-if structure instead?
-------------------------------------------------------------

All call sites invoke nft_pipapo_avx2_refill() only when at least one bit in the map is set, i.e. nft_pipapo_avx2_refill() never returns -1.

It also took me a while to understand the actual reason behind this LLM report.

It simply seems to be inferring that your patch introduced a control flow change in a case that would never happen anyway, so that wasn't expected, and even though it doesn't happen now either, it reports that it was unexpected... not exactly useful.

But anyway:

Quoted 3 lines

Add a runtime debug check that fires if we'd return -1 as additional documentation and also make the suggested change, code might be easier to understand this way.

...for me it's exactly as readable as before. If you think that it might be easier to understand this way, though, I guess that's a good indication that it's better to change it. So, either way:

Quoted 1 line
Quoted 142 lines
---
 net/netfilter/nft_set_pipapo_avx2.c | 35 ++++++++++++-----------------
 1 file changed, 14 insertions(+), 21 deletions(-)
net/netfilter/nft_set_pipapo_avx2.c +14 -21
index dad265807b8b..b3f105520a85 100644
@@ -144,6 +144,7 @@ static void nft_pipapo_avx2_fill(unsigned long *data, int start, int len)
  * This is an alternative implementation of pipapo_refill() suitable for usage
  * with AVX2 lookup routines: we know there are four words to be scanned, at
  * a given offset inside the map, for each matching iteration.
+ * The caller must ensure at least one bit in the four words is set.
  *
  * This function doesn't actually use any AVX2 instruction.
  *
@@ -179,6 +180,7 @@ static int nft_pipapo_avx2_refill(int offset, unsigned long *map,
 	NFT_PIPAPO_AVX2_REFILL_ONE_WORD(3);
 #undef NFT_PIPAPO_AVX2_REFILL_ONE_WORD
 
+	DEBUG_NET_WARN_ON_ONCE(ret < 0);
 	return ret;
 }
 
@@ -243,8 +245,7 @@ static int nft_pipapo_avx2_lookup_4b_2(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -320,8 +321,7 @@ static int nft_pipapo_avx2_lookup_4b_4(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -415,8 +415,7 @@ static int nft_pipapo_avx2_lookup_4b_8(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -506,8 +505,7 @@ static int nft_pipapo_avx2_lookup_4b_12(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -642,8 +640,7 @@ static int nft_pipapo_avx2_lookup_4b_32(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -700,8 +697,7 @@ static int nft_pipapo_avx2_lookup_8b_1(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -765,8 +761,7 @@ static int nft_pipapo_avx2_lookup_8b_2(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -840,8 +835,7 @@ static int nft_pipapo_avx2_lookup_8b_4(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -926,8 +920,7 @@ static int nft_pipapo_avx2_lookup_8b_6(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -1020,8 +1013,7 @@ static int nft_pipapo_avx2_lookup_8b_16(unsigned long *map, unsigned long *fill,
 		b = nft_pipapo_avx2_refill(i_ul, &map[i_ul], fill, f->mt, last);
 		if (last)
 			ret = b;
-
-		if (unlikely(ret == -1))
+		else if (unlikely(ret == -1))
 			ret = b / XSAVE_YMM_SIZE;
 
 		continue;
@@ -1143,6 +1135,7 @@ struct nft_pipapo_elem *pipapo_get_avx2(const struct nft_pipapo_match *m,
 	const struct nft_pipapo_field *f;
 	unsigned long *res, *fill, *map;
 	bool map_index;
+	int ret = 0;
 	int i;
 
 	scratch = *raw_cpu_ptr(m->scratch);
@@ -1167,8 +1160,8 @@ struct nft_pipapo_elem *pipapo_get_avx2(const struct nft_pipapo_match *m,
 
 	nft_pipapo_for_each_field(f, i, m) {
 		bool last = i == m->field_count - 1, first = !i;
-		int ret = 0;
 
+		/* NB: previous round @ret is passed to avx2 lookup fn */
 #define NFT_SET_PIPAPO_AVX2_LOOKUP(b, n)				\
 		(ret = nft_pipapo_avx2_lookup_##b##b_##n(res, fill, f,	\
 							 ret, data,	\
Signature · 1 line
Stefano
author maintainer
#2 lore raw
Quoted 3 lines · Stefano Brivio

...right, I'm not exactly proud of it, but if we don't hide that stuff the amount of visual noise explodes. I couldn't think of any better solution.

It's fine as-is, I just missed that "ret" thing in the first place.

Quoted 3 lines

I really couldn't understand how turning those into "else if" would have anything to do with the scope "ret"... until I realised and actually read your reference. Oops.

I've sent a v2 that moves the LLM report ahead.
I hope its a bit better this way. Actual patch is not changed so I copied your RvB tag.

Quoted 2 lines

intermediate matching bitmap (between fields) with a lot of leading zeroes.

Right. Thanks for reviewing!

#3 lore raw
Quoted 3 lines · On Sat, 11 Apr 2026 14:14:45 +0200 Florian Westphal <[email protected]> wrote:

I've sent a v2 that moves the LLM report ahead.
I hope its a bit better this way. Actual patch is not changed so I copied your RvB tag.

Yes, I think it's better, thanks!

Signature · 1 line
Stefano