nft_set_pipapo: Performance improvements: Season 1
Subsystems (2)
v2 cover nft_set_pipapo: Performance improvements: Season 1
Patches 1/6 and 2/6, as discussed with Pablo, introduce support and switching mechanisms for 8-bit packet matching groups. I opted to pick the fitting implementation with conditionals, instead of replacing the set lookup operation on the fly, as this allows for fields with different group sizes. The cost of these conditionals actually appears negligible.
For the non-vectorised case, the two implementations are almost identical and mostly remain as a single function, while, at least for AVX2, operation sequences turned out to be fairly different, so the new matching functions for 8-bit groups are all separated.
As a side note, I also tried out Pablo's suggestion to use the stack for scratch maps, instead of per-CPU pre-allocated ones, if bucket sizes are small enough. The outcome was rather surprising: it looks cheaper, at least on x86_64, to access pre-allocated data compared to initialise the room we need on the stack.
Patches 3/6 and 4/6 are similar to what I posted earlier, and they are preparation work for vectorised implementations: we need to support arbitrary requirements about data alignment and we also need to share some helper functions.
Patch 5/6 implements the AVX2 lookup routines, now supporting 4-bit and 8-bit as group sizes.
Patch 6/6 adjusts the set implementations to also support sets with a single, ranged field. This can be now conveniently enabled with a define, and allows us to have a fair comparison with the rbtree set back-end.
The matching rate figures below were obtained with the usual kselftests cases, averaged over five runs, on a single thread of an AMD Epyc 7402 CPU for x86_64 and on a single BCM2711 thread (Raspberry Pi 4 Model B clocked at 2147MHz) for a comparison with ARM 64-bit.
Note that I disabled retpolines (on x86_64) and SSBD (on aarch64), so these matching rates can't be directly compared to figures I shared previously -- hence the new baselines (also repeated in single patch messages). For some reason, I'm getting more repeatable numbers this way, and we're probably going to get rid of a number of indirect calls in the future anyway. By hardcoding calls to set lookup functions, I'm getting numbers rather close to these baselines even with CONFIG_RETPOLINE set.
Also note, as it was the case earlier, that this is not a fair comparison with hash types, because hash types don't support ranges.
Matching rates for concatenated ranges:
---------------.-----------------------------------.-------------------------.
AMD Epyc 7402 | baselines, Mpps | this series, Mpps |
1 thread |___________________________________|_________________________|
3.35GHz | | | | | | |
768KiB L1D$ | netdev | hash | rbtree | | | |
---------------| hook | no | single | pipapo | pipapo | pipapo |
type entries | drop | ranges | field | 4 bits | bit switch | AVX2 |
---------------|--------|--------|--------|--------|------------|------------|
net,port | | | | | | |
1000 | 19.0 | 10.4 | 3.8 | 2.8 | 4.0 +43% | 7.5 +168% |
---------------|--------|--------|--------|--------|------------|------------|
port,net | | | | | | |
100 | 18.8 | 10.3 | 5.8 | 5.5 | 6.3 +14% | 8.1 +47% |
---------------|--------|--------|--------|--------|------------|------------|
net6,port | | | | | | |
1000 | 16.4 | 7.6 | 1.8 | 1.3 | 2.1 +61% | 4.8 +269% |
---------------|--------|--------|--------|--------|------------|------------|
port,proto | | | | | [1] | [1] |
30000 | 19.6 | 11.6 | 3.9 | 0.3 | 0.5 +66% | 2.6 +766% |
---------------|--------|--------|--------|--------|------------|------------|
net6,port,mac | | | | | | |
10 | 16.5 | 5.4 | 4.3 | 2.6 | 3.4 +31% | 4.7 +81% |
---------------|--------|--------|--------|--------|------------|------------|
net6,port,mac, | | | | | | |
proto 1000 | 16.5 | 5.7 | 1.9 | 1.0 | 1.4 +40% | 3.6 +260% |
---------------|--------|--------|--------|--------|------------|------------|
net,mac | | | | | | |
1000 | 19.0 | 8.4 | 3.9 | 1.7 | 2.5 +47% | 6.4 +276% | ---------------'--------'--------'--------'--------'------------'------------'
[1] Causes switch of lookup table buckets for 'port' to 4-bit groups
---------------.-----------------------------------.------------.
BCM2711 | baselines, Mpps | patch 2/6 |
1 thread |___________________________________|____________|
2147MHz | | | | | |
32KiB L1D$ | netdev | hash | rbtree | | |
---------------| hook | no | single | pipapo | pipapo |
type entries | drop | ranges | field | 4 bits | bit switch |
---------------|--------|--------|--------|--------|------------|
net,port | | | | | |
1000 | 1.63 | 1.37 | 0.87 | 0.61 | 0.70 +17% |
---------------|--------|--------|--------|--------|------------|
port,net | | | | | |
100 | 1.64 | 1.36 | 1.02 | 0.78 | 0.81 +4% |
---------------|--------|--------|--------|--------|------------|
net6,port | | | | | |
1000 | 1.56 | 1.27 | 0.65 | 0.34 | 0.50 +47% |
---------------|--------|--------|--------|--------|------------|
port,proto [1] | | | | | |
10000 | 1.68 | 1.43 | 0.84 | 0.30 | 0.40 +13% |
---------------|--------|--------|--------|--------|------------|
net6,port,mac | | | | | |
10 | 1.56 | 1.14 | 1.02 | 0.62 | 0.66 +6% |
---------------|--------|--------|--------|--------|------------|
net6,port,mac, | | | | | |
proto 1000 | 1.56 | 1.12 | 0.64 | 0.27 | 0.40 +48% |
---------------|--------|--------|--------|--------|------------|
net,mac | | | | | |
1000 | 1.63 | 1.26 | 0.87 | 0.41 | 0.53 +29% | ---------------'--------'--------'--------'--------'------------'
[1] Using 10000 entries instead of 30000 as it would take way too long for the test script to generate all of them
Matching rates for non-concatenated ranges (first field):
---------------.--------------------------.-------------------------.
AMD Epyc 7402 | baselines, Mpps | Mpps, % over rbtree |
1 thread |__________________________|_________________________|
3.35GHz | | | | | |
768KiB L1D$ | netdev | hash | rbtree | | pipapo |
---------------| hook | no | single | pipapo |single field|
type entries | drop | ranges | field |single field| AVX2 |
---------------|--------|--------|--------|------------|------------|
net,port | | | | | |
1000 | 19.0 | 10.4 | 3.8 | 6.0 +58% | 9.6 +153% |
---------------|--------|--------|--------|------------|------------|
port,net | | | | | |
100 | 18.8 | 10.3 | 5.8 | 9.1 +57% |11.6 +100% |
---------------|--------|--------|--------|------------|------------|
net6,port | | | | | |
1000 | 16.4 | 7.6 | 1.8 | 2.8 +55% | 6.5 +261% |
---------------|--------|--------|--------|------------|------------|
port,proto | | | | [1] | [1] |
30000 | 19.6 | 11.6 | 3.9 | 0.9 -77% | 2.7 -31% |
---------------|--------|--------|--------|------------|------------|
port,proto | | | | | |
10000 | 19.6 | 11.6 | 4.4 | 2.1 -52% | 5.6 +27% |
---------------|--------|--------|--------|------------|------------|
port,proto | | | | | |
4 threads 10000| 77.9 | 45.1 | 17.4 | 8.3 -52% |22.4 +29% |
---------------|--------|--------|--------|------------|------------|
net6,port,mac | | | | | |
10 | 16.5 | 5.4 | 4.3 | 4.5 +5% | 8.2 +91% |
---------------|--------|--------|--------|------------|------------|
net6,port,mac, | | | | | |
proto 1000 | 16.5 | 5.7 | 1.9 | 2.8 +47% | 6.6 +247% |
---------------|--------|--------|--------|------------|------------|
net,mac | | | | | |
1000 | 19.0 | 8.4 | 3.9 | 6.0 +54% | 9.9 +154% | ---------------'--------'--------'--------'------------'------------'
[1] Causes switch of lookup table buckets for 'port' to 4-bit groups
v2: Rebase, especially as series "netfilter: nf_tables: make sets
built-in" was merged, add 6/6 as new patch and single-field comparison with rbtree.
Stefano Brivio (6):
nft_set_pipapo: Generalise group size for buckets
nft_set_pipapo: Add support for 8-bit lookup groups and dynamic switch
nft_set_pipapo: Prepare for vectorised implementation: alignment
nft_set_pipapo: Prepare for vectorised implementation: helpers
nft_set_pipapo: Introduce AVX2-based lookup implementation
nft_set_pipapo: Prepare for single ranged field usage
include/net/netfilter/nf_tables_core.h | 1 + net/netfilter/Makefile | 6 + net/netfilter/nf_tables_api.c | 3 + net/netfilter/nft_set_pipapo.c | 630 +++++++----- net/netfilter/nft_set_pipapo.h | 280 ++++++ net/netfilter/nft_set_pipapo_avx2.c | 1223 ++++++++++++++++++++++++ net/netfilter/nft_set_pipapo_avx2.h | 14 + 7 files changed, 1893 insertions(+), 264 deletions(-) create mode 100644 net/netfilter/nft_set_pipapo.h create mode 100644 net/netfilter/nft_set_pipapo_avx2.c create mode 100644 net/netfilter/nft_set_pipapo_avx2.h
Signature · 1 line
2.25.1
PATCH v2 1/6 nft_set_pipapo: Generalise group size for buckets
Get rid of all hardcoded assumptions that buckets in lookup tables correspond to four-bit groups, and replace them with appropriate calculations based on a variable group size, now stored in struct field.
The group size could now be in principle any divisor of eight. Note, though, that lookup and get functions need an implementation intimately depending on the group size, and the only supported size there, currently, is four bits, which is also the initial and only used size at the moment.
While at it, drop 'groups' from struct nft_pipapo: it was never used.
---
v2: Rebase, no changes
net/netfilter/nft_set_pipapo.c | 208 ++++++++++++++++++--------------- 1 file changed, 112 insertions(+), 96 deletions(-)
net/netfilter/nft_set_pipapo.c +112 -96
@@ -350,16 +350,18 @@/* Number of bits to be grouped together in lookup table buckets, arbitrary */ #define NFT_PIPAPO_GROUP_BITS 4 -#define NFT_PIPAPO_GROUPS_PER_BYTE (BITS_PER_BYTE / NFT_PIPAPO_GROUP_BITS) + +#define NFT_PIPAPO_GROUPS_PER_BYTE (f) (BITS_PER_BYTE / (f)->bb)/* Fields are padded to 32 bits in input registers */ -#define NFT_PIPAPO_GROUPS_PADDED_SIZE (x) \ - (round_up((x) / NFT_PIPAPO_GROUPS_PER_BYTE, sizeof(u32))) -#define NFT_PIPAPO_GROUPS_PADDING (x) \ - (NFT_PIPAPO_GROUPS_PADDED_SIZE((x)) - (x) / NFT_PIPAPO_GROUPS_PER_BYTE) +#define NFT_PIPAPO_GROUPS_PADDED_SIZE (f) \ + (round_up((f)->groups / NFT_PIPAPO_GROUPS_PER_BYTE(f), sizeof(u32))) +#define NFT_PIPAPO_GROUPS_PADDING (f) \ + (NFT_PIPAPO_GROUPS_PADDED_SIZE(f) - (f)->groups / \ + NFT_PIPAPO_GROUPS_PER_BYTE(f)) -/* Number of buckets, given by 2 ^ n, with n grouped bits */ -#define NFT_PIPAPO_BUCKETS (1 << NFT_PIPAPO_GROUP_BITS) +/* Number of buckets given by 2 ^ n, with n bucket bits */ +#define NFT_PIPAPO_BUCKETS (bb) (1 << (bb))/* Each n-bit range maps to up to n * 2 rules */ #define NFT_PIPAPO_MAP_NBITS (const_ilog2(NFT_PIPAPO_MAX_BITS * 2))
@@ -406,16 +408,18 @@ union nft_pipapo_map_bucket {/** * struct nft_pipapo_field - Lookup, mapping tables and related data for a field -* @groups: Amount of 4-bit groups +* @groups: Amount of bit groups * @rules: Number of inserted rules * @bsize: Size of each bucket in lookup table, in longs -* @lt: Lookup table: 'groups' rows of NFT_PIPAPO_BUCKETS buckets +* @bb: Number of bits grouped together in lookup table buckets +* @lt: Lookup table: 'groups' rows of buckets * @mt: Mapping table: one bucket per rule */ struct nft_pipapo_field {int groups;unsigned long rules;size_t bsize; +int bb;unsigned long *lt;union nft_pipapo_map_bucket *mt; };
@@ -443,7 +447,6 @@ static DEFINE_PER_CPU(bool, nft_pipapo_scratch_index);* struct nft_pipapo - Representation of a set * @match: Currently in-use matching data * @clone: Copy where pending insertions and deletions are kept -* @groups: Total amount of 4-bit groups for fields in this set * @width: Total bytes to be matched for one packet, including padding * @dirty: Working copy has pending insertions or deletions * @last_gc: Timestamp of last garbage collection run, jiffies
@@ -451,7 +454,6 @@ static DEFINE_PER_CPU(bool, nft_pipapo_scratch_index);struct nft_pipapo {struct nft_pipapo_match __rcu *match;struct nft_pipapo_match *clone; -int groups;int width;bool dirty;unsigned long last_gc;
@@ -520,6 +522,34 @@ static int pipapo_refill(unsigned long *map, int len, int rules,return ret; } +/** +* pipapo_and_field_buckets_4bit() - Intersect buckets for 4-bit groups +* @f: Field including lookup table +* @dst: Area to store result +* @data: Input data selecting table buckets +*/ +static void pipapo_and_field_buckets_4bit (struct nft_pipapo_field *f, +unsigned long *dst, +const u8 *data) +{ +unsigned long *lt = f->lt; +int group; + +for (group =0 ; group < f->groups; group +=BITS_PER_BYTE /4 , data++) { +u8 v; + + v = *data >>4 ; +__bitmap_and (dst, dst, lt + v * f->bsize, + f->bsize *BITS_PER_LONG ); + lt += f->bsize *NFT_PIPAPO_BUCKETS (4 ); + + v = *data &0x0f ; +__bitmap_and (dst, dst, lt + v * f->bsize, + f->bsize *BITS_PER_LONG ); + lt += f->bsize *NFT_PIPAPO_BUCKETS (4 ); + } +} +/** * nft_pipapo_lookup() - Lookup function * @net: Network namespace
@@ -559,26 +589,15 @@ static bool nft_pipapo_lookup(const struct net *net, const struct nft_set *set,nft_pipapo_for_each_field (f, i, m) {bool last = i == m->field_count -1 ; -unsigned long *lt = f->lt; -int b, group; +int b; -/* For each 4-bit group: select lookup table bucket depending on +/* For each bit group: select lookup table bucket depending on * packet bytes value, then AND bucket value */ -for (group =0 ; group < f->groups; group +=2 ) { -u8 v; - - v = *rp >>4 ; -__bitmap_and (res_map, res_map, lt + v * f->bsize, - f->bsize *BITS_PER_LONG ); - lt += f->bsize *NFT_PIPAPO_BUCKETS ; - - v = *rp &0x0f ; - rp++; -__bitmap_and (res_map, res_map, lt + v * f->bsize, - f->bsize *BITS_PER_LONG ); - lt += f->bsize *NFT_PIPAPO_BUCKETS ; - } +pipapo_and_field_buckets_4bit (f, res_map, rp); +BUILD_BUG_ON (NFT_PIPAPO_GROUP_BITS !=4 ); + + rp += f->groups /NFT_PIPAPO_GROUPS_PER_BYTE (f);/* Now populate the bitmap for the next field, unless this is * the last field, in which case return the matched 'ext'
@@ -621,7 +640,7 @@ static bool nft_pipapo_lookup(const struct net *net, const struct nft_set *set, map_index = !map_index;swap (res_map, fill_map); - rp +=NFT_PIPAPO_GROUPS_PADDING (f->groups); + rp +=NFT_PIPAPO_GROUPS_PADDING (f); } out:
@@ -669,26 +688,17 @@ static struct nft_pipapo_elem *pipapo_get(const struct net *net,nft_pipapo_for_each_field (f, i, m) {bool last = i == m->field_count -1 ; -unsigned long *lt = f->lt; -int b, group; +int b; -/* For each 4-bit group: select lookup table bucket depending on +/* For each bit group: select lookup table bucket depending on * packet bytes value, then AND bucket value */ -for (group =0 ; group < f->groups; group++) { -u8 v; - -if (group %2 ) { - v = *data &0x0f ; - data++; - }else { - v = *data >>4 ; - } -__bitmap_and (res_map, res_map, lt + v * f->bsize, - f->bsize *BITS_PER_LONG ); +if (f->bb ==4 ) +pipapo_and_field_buckets_4bit (f, res_map, data); +else +BUG (); - lt += f->bsize *NFT_PIPAPO_BUCKETS ; - } + data += f->groups /NFT_PIPAPO_GROUPS_PER_BYTE (f);/* Now populate the bitmap for the next field, unless this is * the last field, in which case return the matched 'ext'
@@ -713,7 +723,7 @@ static struct nft_pipapo_elem *pipapo_get(const struct net *net, goto out; } - data +=NFT_PIPAPO_GROUPS_PADDING (f->groups); + data +=NFT_PIPAPO_GROUPS_PADDING (f);/* Swap bitmap indices: fill_map will be the initial bitmap for * the next field (i.e. the new res_map), and res_map is
@@ -772,15 +782,15 @@ static int pipapo_resize(struct nft_pipapo_field *f, int old_rules, int rules)else copy = new_bucket_size; - new_lt =kvzalloc (f->groups *NFT_PIPAPO_BUCKETS * new_bucket_size * -sizeof (*new_lt),GFP_KERNEL ); + new_lt =kvzalloc (f->groups *NFT_PIPAPO_BUCKETS (f->bb) * + new_bucket_size *sizeof (*new_lt),GFP_KERNEL );if (!new_lt)return -ENOMEM ; new_p = new_lt; old_p = old_lt;for (group =0 ; group < f->groups; group++) { -for (bucket =0 ; bucket <NFT_PIPAPO_BUCKETS ; bucket++) { +for (bucket =0 ; bucket <NFT_PIPAPO_BUCKETS (f->bb); bucket++) {memcpy (new_p, old_p, copy *sizeof (*new_p)); new_p += copy; old_p += copy;
@@ -829,7 +839,7 @@ static void pipapo_bucket_set(struct nft_pipapo_field *f, int rule, int group, {unsigned long *pos; - pos = f->lt + f->bsize *NFT_PIPAPO_BUCKETS * group; + pos = f->lt + f->bsize *NFT_PIPAPO_BUCKETS (f->bb) * group; pos += f->bsize * v;__set_bit (rule, pos);
@@ -849,7 +859,7 @@ static void pipapo_bucket_set(struct nft_pipapo_field *f, int rule, int group,static int pipapo_insert (struct nft_pipapo_field *f,const uint8_t *k,int mask_bits) { -int rule = f->rules++, group, ret; +int rule = f->rules++, group, ret, bit_offset =0 ; ret =pipapo_resize (f, f->rules -1 , f->rules);if (ret)
@@ -859,22 +869,25 @@ static int pipapo_insert(struct nft_pipapo_field *f, const uint8_t *k,int i, v;u8 mask; -if (group %2 ) - v = k[group /2 ] &0x0f ; -else - v = k[group /2 ] >>4 ; + v = k[group / (BITS_PER_BYTE / f->bb)]; + v &=GENMASK (BITS_PER_BYTE - bit_offset -1 ,0 ); + v >>= (BITS_PER_BYTE - bit_offset) - f->bb; -if (mask_bits >= (group +1 ) *4 ) { + bit_offset += f->bb; + bit_offset %=BITS_PER_BYTE ; + +if (mask_bits >= (group +1 ) * f->bb) {/* Not masked */ pipapo_bucket_set (f, rule, group, v); - }else if (mask_bits <= group *4 ) { + }else if (mask_bits <= group * f->bb) {/* Completely masked */ -for (i =0 ; i <NFT_PIPAPO_BUCKETS ; i++) +for (i =0 ; i <NFT_PIPAPO_BUCKETS (f->bb); i++)pipapo_bucket_set (f, rule, group, i); }else {/* The mask limit falls on this group */ - mask =0x0f >> (mask_bits - group *4 ); -for (i =0 ; i <NFT_PIPAPO_BUCKETS ; i++) { + mask =GENMASK (f->bb -1 ,0 ); + mask >>= mask_bits - group * f->bb; +for (i =0 ; i <NFT_PIPAPO_BUCKETS (f->bb); i++) {if ((i & ~mask) == (v & ~mask))pipapo_bucket_set (f, rule, group, i); }
@@ -1123,11 +1136,11 @@ static int nft_pipapo_insert(const struct net *net, const struct nft_set *set,return -ENOSPC ;if (memcmp (start_p, end_p, - f->groups /NFT_PIPAPO_GROUPS_PER_BYTE ) >0 ) + f->groups /NFT_PIPAPO_GROUPS_PER_BYTE (f)) >0 )return -EINVAL ; - start_p +=NFT_PIPAPO_GROUPS_PADDED_SIZE (f->groups); - end_p +=NFT_PIPAPO_GROUPS_PADDED_SIZE (f->groups); + start_p +=NFT_PIPAPO_GROUPS_PADDED_SIZE (f); + end_p +=NFT_PIPAPO_GROUPS_PADDED_SIZE (f); }/* Insert */
@@ -1141,22 +1154,19 @@ static int nft_pipapo_insert(const struct net *net, const struct nft_set *set, rulemap[i].to = f->rules; ret =memcmp (start, end, - f->groups /NFT_PIPAPO_GROUPS_PER_BYTE ); -if (!ret) { - ret =pipapo_insert (f, start, - f->groups *NFT_PIPAPO_GROUP_BITS ); - }else { - ret =pipapo_expand (f, start, end, - f->groups *NFT_PIPAPO_GROUP_BITS ); - } + f->groups /NFT_PIPAPO_GROUPS_PER_BYTE (f)); +if (!ret) + ret =pipapo_insert (f, start, f->groups * f->bb); +else + ret =pipapo_expand (f, start, end, f->groups * f->bb);if (f->bsize > bsize_max) bsize_max = f->bsize; rulemap[i].n = ret; - start +=NFT_PIPAPO_GROUPS_PADDED_SIZE (f->groups); - end +=NFT_PIPAPO_GROUPS_PADDED_SIZE (f->groups); + start +=NFT_PIPAPO_GROUPS_PADDED_SIZE (f); + end +=NFT_PIPAPO_GROUPS_PADDED_SIZE (f); }if (!*this_cpu_ptr (m->scratch) || bsize_max > m->bsize_max) {
@@ -1208,7 +1218,7 @@ static struct nft_pipapo_match *pipapo_clone(struct nft_pipapo_match *old)for (i =0 ; i < old->field_count; i++) {memcpy (dst, src, offsetof(struct nft_pipapo_field , lt)); - dst->lt =kvzalloc (src->groups *NFT_PIPAPO_BUCKETS * + dst->lt =kvzalloc (src->groups *NFT_PIPAPO_BUCKETS (src->bb) * src->bsize *sizeof (*dst->lt),GFP_KERNEL );if (!dst->lt)
@@ -1216,7 +1226,7 @@ static struct nft_pipapo_match *pipapo_clone(struct nft_pipapo_match *old)memcpy (dst->lt, src->lt, src->bsize *sizeof (*dst->lt) * - src->groups *NFT_PIPAPO_BUCKETS ); + src->groups *NFT_PIPAPO_BUCKETS (src->bb)); dst->mt =kvmalloc (src->rules *sizeof (*src->mt),GFP_KERNEL );if (!dst->mt)
@@ -1394,9 +1404,9 @@ static void pipapo_drop(struct nft_pipapo_match *m,unsigned long *pos;int b; - pos = f->lt + g *NFT_PIPAPO_BUCKETS * f->bsize; + pos = f->lt + g *NFT_PIPAPO_BUCKETS (f->bb) * f->bsize; -for (b =0 ; b <NFT_PIPAPO_BUCKETS ; b++) { +for (b =0 ; b <NFT_PIPAPO_BUCKETS (f->bb); b++) {bitmap_cut (pos, pos, rulemap[i].to, rulemap[i].n, f->bsize *BITS_PER_LONG );
@@ -1690,30 +1700,33 @@ static bool nft_pipapo_flush(const struct net *net, const struct nft_set *set,static int pipapo_get_boundaries (struct nft_pipapo_field *f,int first_rule,int rule_count,u8 *left,u8 *right) { +int g, mask_len =0 , bit_offset =0 ;u8 *l = left, *r = right; -int g, mask_len =0 ;for (g =0 ; g < f->groups; g++) {int b, x0, x1; x0 =-1 ; x1 =-1 ; -for (b =0 ; b <NFT_PIPAPO_BUCKETS ; b++) { +for (b =0 ; b <NFT_PIPAPO_BUCKETS (f->bb); b++) {unsigned long *pos; - pos = f->lt + (g *NFT_PIPAPO_BUCKETS + b) * f->bsize; + pos = f->lt + (g *NFT_PIPAPO_BUCKETS (f->bb) + b) * + f->bsize;if (test_bit (first_rule, pos) && x0 ==-1 ) x0 = b;if (test_bit (first_rule + rule_count -1 , pos)) x1 = b; } -if (g %2 ) { - *(l++) |= x0 &0x0f ; - *(r++) |= x1 &0x0f ; - }else { - *l |= x0 <<4 ; - *r |= x1 <<4 ; + *l |= x0 << (BITS_PER_BYTE - f->bb - bit_offset); + *r |= x1 << (BITS_PER_BYTE - f->bb - bit_offset); + + bit_offset += f->bb; +if (bit_offset >=BITS_PER_BYTE ) { + bit_offset %=BITS_PER_BYTE ; + l++; + r++; }if (x1 - x0 ==0 )
@@ -1748,8 +1761,9 @@ static bool pipapo_match_field(struct nft_pipapo_field *f,pipapo_get_boundaries (f, first_rule, rule_count, left, right); -return !memcmp (start, left, f->groups /NFT_PIPAPO_GROUPS_PER_BYTE ) && - !memcmp (end, right, f->groups /NFT_PIPAPO_GROUPS_PER_BYTE ); +return !memcmp (start, left, + f->groups /NFT_PIPAPO_GROUPS_PER_BYTE (f)) && + !memcmp (end, right, f->groups /NFT_PIPAPO_GROUPS_PER_BYTE (f)); }/**
@@ -1801,8 +1815,8 @@ static void nft_pipapo_remove(const struct net *net, const struct nft_set *set, rules_fx = f->mt[start].n; start = f->mt[start].to; - match_start +=NFT_PIPAPO_GROUPS_PADDED_SIZE (f->groups); - match_end +=NFT_PIPAPO_GROUPS_PADDED_SIZE (f->groups); + match_start +=NFT_PIPAPO_GROUPS_PADDED_SIZE (f); + match_end +=NFT_PIPAPO_GROUPS_PADDED_SIZE (f); }if (i == m->field_count) {
@@ -1895,9 +1909,9 @@ static u64 nft_pipapo_privsize(const struct nlattr * const nla[],* case here. * * In general, for a non-ranged entry or a single composing netmask, we need -* one bit in each of the sixteen NFT_PIPAPO_BUCKETS, for each 4-bit group (that -* is, each input bit needs four bits of matching data), plus a bucket in the -* mapping table for each field. +* one bit in each of the sixteen buckets, for each 4-bit group (that is, each +* input bit needs four bits of matching data), plus a bucket in the mapping +* table for each field. * * Return: true only for compatible range concatenations */
@@ -1921,7 +1935,9 @@ static bool nft_pipapo_estimate(const struct nft_set_desc *desc, u32 features,* each rule also needs a mapping bucket. */ rules =ilog2 (desc->field_len[i] *BITS_PER_BYTE ) *2 ; - entry_size += rules *NFT_PIPAPO_BUCKETS /BITS_PER_BYTE ; + entry_size += rules * +NFT_PIPAPO_BUCKETS (NFT_PIPAPO_GROUP_BITS ) / +BITS_PER_BYTE ; entry_size += rules *sizeof (union nft_pipapo_map_bucket ); }
@@ -1985,8 +2001,8 @@ static int nft_pipapo_init(const struct nft_set *set,rcu_head_init (&m->rcu);nft_pipapo_for_each_field (f, i, m) { - f->groups = desc->field_len[i] *NFT_PIPAPO_GROUPS_PER_BYTE ; - priv->groups += f->groups; + f->bb =NFT_PIPAPO_GROUP_BITS ; + f->groups = desc->field_len[i] *NFT_PIPAPO_GROUPS_PER_BYTE (f); priv->width +=round_up (desc->field_len[i],sizeof (u32));
Signature · 1 line
2.25.1
PATCH v2 2/6 nft_set_pipapo: Add support for 8-bit lookup groups and dynamic switch
In reply to Stefano BrivioWhile grouping matching bits in groups of four saves memory compared to the more natural choice of 8-bit words (lookup table size is one eighth), it comes at a performance cost, as the number of lookup comparisons is doubled, and those also needs bitshifts and masking.
Introduce support for 8-bit lookup groups, together with a mapping mechanism to dynamically switch, based on defined per-table size thresholds and hysteresis, between 8-bit and 4-bit groups, as tables grow and shrink. Empty sets start with 8-bit groups, and per-field tables are converted to 4-bit groups if they get too big.
An alternative approach would have been to swap per-set lookup operation functions as needed, but this doesn't allow for different group sizes in the same set, which looks desirable if some fields need significantly more matching data compared to others due to heavier impact of ranges (e.g. a big number of subnets with relatively simple port specifications).
Allowing different group sizes for the same lookup functions implies the need for further conditional clauses, whose cost, however, appears to be negligible in tests.
The matching rate figures below were obtained for x86_64 running the nft_concat_range.sh "performance" cases, averaged over five runs, on a single thread of an AMD Epyc 7402 CPU, and for aarch64 on a single thread of a BCM2711 (Raspberry Pi 4 Model B 4GB), clocked at a stable 2147MHz frequency:
---------------.-----------------------------------.------------.
AMD Epyc 7402 | baselines, Mpps | this patch |
1 thread |___________________________________|____________|
3.35GHz | | | | | |
768KiB L1D$ | netdev | hash | rbtree | | |
---------------| hook | no | single | pipapo | pipapo |
type entries | drop | ranges | field | 4 bits | bit switch |
---------------|--------|--------|--------|--------|------------|
net,port | | | | | |
1000 | 19.0 | 10.4 | 3.8 | 2.8 | 4.0 +43% |
---------------|--------|--------|--------|--------|------------|
port,net | | | | | |
100 | 18.8 | 10.3 | 5.8 | 5.5 | 6.3 +14% |
---------------|--------|--------|--------|--------|------------|
net6,port | | | | | |
1000 | 16.4 | 7.6 | 1.8 | 1.3 | 2.1 +61% |
---------------|--------|--------|--------|--------|------------|
port,proto | | | | | [1] |
30000 | 19.6 | 11.6 | 3.9 | 0.3 | 0.5 +66% |
---------------|--------|--------|--------|--------|------------|
net6,port,mac | | | | | |
10 | 16.5 | 5.4 | 4.3 | 2.6 | 3.4 +31% |
---------------|--------|--------|--------|--------|------------|
net6,port,mac, | | | | | |
proto 1000 | 16.5 | 5.7 | 1.9 | 1.0 | 1.4 +40% |
---------------|--------|--------|--------|--------|------------|
net,mac | | | | | |
1000 | 19.0 | 8.4 | 3.9 | 1.7 | 2.5 +47% |---------------'--------'--------'--------'--------'------------'
[1] Causes switch of lookup table buckets for 'port', not 'proto', to 4-bit groups
---------------.-----------------------------------.------------.
BCM2711 | baselines, Mpps | this patch |
1 thread |___________________________________|____________|
2147MHz | | | | | |
32KiB L1D$ | netdev | hash | rbtree | | |
---------------| hook | no | single | pipapo | pipapo |
type entries | drop | ranges | field | 4 bits | bit switch |
---------------|--------|--------|--------|--------|------------|
net,port | | | | | |
1000 | 1.63 | 1.37 | 0.87 | 0.61 | 0.70 +17% |
---------------|--------|--------|--------|--------|------------|
port,net | | | | | |
100 | 1.64 | 1.36 | 1.02 | 0.78 | 0.81 +4% |
---------------|--------|--------|--------|--------|------------|
net6,port | | | | | |
1000 | 1.56 | 1.27 | 0.65 | 0.34 | 0.50 +47% |
---------------|--------|--------|--------|--------|------------|
port,proto [2] | | | | | |
10000 | 1.68 | 1.43 | 0.84 | 0.30 | 0.40 +13% |
---------------|--------|--------|--------|--------|------------|
net6,port,mac | | | | | |
10 | 1.56 | 1.14 | 1.02 | 0.62 | 0.66 +6% |
---------------|--------|--------|--------|--------|------------|
net6,port,mac, | | | | | |
proto 1000 | 1.56 | 1.12 | 0.64 | 0.27 | 0.40 +48% |
---------------|--------|--------|--------|--------|------------|
net,mac | | | | | |
1000 | 1.63 | 1.26 | 0.87 | 0.41 | 0.53 +29% | ---------------'--------'--------'--------'--------'------------'
[2] Using 10000 entries instead of 30000 as it would take way too long for the test script to generate all of them
---
v2: Rebase, no changes
net/netfilter/nft_set_pipapo.c | 241 +++++++++++++++++++++++++++++++-- 1 file changed, 233 insertions(+), 8 deletions(-)
net/netfilter/nft_set_pipapo.c +233 -8
@@ -348,11 +348,30 @@#define NFT_PIPAPO_MAX_BYTES (sizeof(struct in6_addr))#define NFT_PIPAPO_MAX_BITS (NFT_PIPAPO_MAX_BYTES * BITS_PER_BYTE) -/* Number of bits to be grouped together in lookup table buckets, arbitrary */ -#define NFT_PIPAPO_GROUP_BITS 4 - +/* Bits to be grouped together in table buckets depending on set size */ +#define NFT_PIPAPO_GROUP_BITS_INIT NFT_PIPAPO_GROUP_BITS_SMALL_SET +#define NFT_PIPAPO_GROUP_BITS_SMALL_SET 8 +#define NFT_PIPAPO_GROUP_BITS_LARGE_SET 4 +#define NFT_PIPAPO_GROUP_BITS_ARE_8_OR_4 \ + BUILD_BUG_ON((NFT_PIPAPO_GROUP_BITS_SMALL_SET != 8) || \ + (NFT_PIPAPO_GROUP_BITS_LARGE_SET != 4))#define NFT_PIPAPO_GROUPS_PER_BYTE (f) (BITS_PER_BYTE / (f)->bb) +/* If a lookup table gets bigger than NFT_PIPAPO_LT_SIZE_HIGH, switch to the +* small group width, and switch to the big group width if the table gets +* smaller than NFT_PIPAPO_LT_SIZE_LOW. +* +* Picking 2MiB as threshold (for a single table) avoids as much as possible +* crossing page boundaries on most architectures (x86-64 and MIPS huge pages, +* ARMv7 supersections, POWER "large" pages, SPARC Level 1 regions, etc.), which +* keeps performance nice in case kvmalloc() gives us non-contiguous areas. +*/ +#define NFT_PIPAPO_LT_SIZE_THRESHOLD (1 << 21) +#define NFT_PIPAPO_LT_SIZE_HYSTERESIS (1 << 16) +#define NFT_PIPAPO_LT_SIZE_HIGH NFT_PIPAPO_LT_SIZE_THRESHOLD +#define NFT_PIPAPO_LT_SIZE_LOW NFT_PIPAPO_LT_SIZE_THRESHOLD - \ + NFT_PIPAPO_LT_SIZE_HYSTERESIS +/* Fields are padded to 32 bits in input registers */ #define NFT_PIPAPO_GROUPS_PADDED_SIZE (f) \ (round_up((f)->groups / NFT_PIPAPO_GROUPS_PER_BYTE(f), sizeof(u32)))
@@ -550,6 +569,26 @@ static void pipapo_and_field_buckets_4bit(struct nft_pipapo_field *f, } } +/** +* pipapo_and_field_buckets_8bit() - Intersect buckets for 8-bit groups +* @f: Field including lookup table +* @dst: Area to store result +* @data: Input data selecting table buckets +*/ +static void pipapo_and_field_buckets_8bit (struct nft_pipapo_field *f, +unsigned long *dst, +const u8 *data) +{ +unsigned long *lt = f->lt; +int group; + +for (group =0 ; group < f->groups; group++, data++) { +__bitmap_and (dst, dst, lt + *data * f->bsize, + f->bsize *BITS_PER_LONG ); + lt += f->bsize *NFT_PIPAPO_BUCKETS (8 ); + } +} +/** * nft_pipapo_lookup() - Lookup function * @net: Network namespace
@@ -594,8 +633,11 @@ static bool nft_pipapo_lookup(const struct net *net, const struct nft_set *set,/* For each bit group: select lookup table bucket depending on * packet bytes value, then AND bucket value */ -pipapo_and_field_buckets_4bit (f, res_map, rp); -BUILD_BUG_ON (NFT_PIPAPO_GROUP_BITS !=4 ); +if (likely (f->bb ==8 )) +pipapo_and_field_buckets_8bit (f, res_map, rp); +else +pipapo_and_field_buckets_4bit (f, res_map, rp); +NFT_PIPAPO_GROUP_BITS_ARE_8_OR_4 ; rp += f->groups /NFT_PIPAPO_GROUPS_PER_BYTE (f);
@@ -693,7 +735,9 @@ static struct nft_pipapo_elem *pipapo_get(const struct net *net,/* For each bit group: select lookup table bucket depending on * packet bytes value, then AND bucket value */ -if (f->bb ==4 ) +if (f->bb ==8 ) +pipapo_and_field_buckets_8bit (f, res_map, data); +else if (f->bb ==4 )pipapo_and_field_buckets_4bit (f, res_map, data);else BUG ();
@@ -845,6 +889,183 @@ static void pipapo_bucket_set(struct nft_pipapo_field *f, int rule, int group,__set_bit (rule, pos); } +/** +* pipapo_lt_4b_to_8b() - Switch lookup table group width from 4 bits to 8 bits +* @old_groups: Number of current groups +* @bsize: Size of one bucket, in longs +* @old_lt: Pointer to the current lookup table +* @new_lt: Pointer to the new, pre-allocated lookup table +* +* Each bucket with index b in the new lookup table, belonging to group g, is +* filled with the bit intersection between: +* - bucket with index given by the upper 4 bits of b, from group g, and +* - bucket with index given by the lower 4 bits of b, from group g + 1 +* +* That is, given buckets from the new lookup table N(x, y) and the old lookup +* table O(x, y), with x bucket index, and y group index: +* +* N(b, g) := O(b / 16, g) & O(b % 16, g + 1) +* +* This ensures equivalence of the matching results on lookup. Two examples in +* pictures: +* +* bucket +* group 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 ... 254 255 +* 0 ^ +* 1 | ^ +* ... ( & ) | +* / \ | +* / \ .-( & )-. +* / bucket \ | | +* group 0 / 1 2 3 \ 4 5 6 7 8 9 10 11 12 13 |14 15 | +* 0 / \ | | +* 1 \ | | +* 2 | --' +* 3 '- +* ... +*/ +static void pipapo_lt_4b_to_8b (int old_groups,int bsize, +unsigned long *old_lt,unsigned long *new_lt) +{ +int g, b, i; + +for (g =0 ; g < old_groups /2 ; g++) { +int src_g0 = g *2 , src_g1 = g *2 +1 ; + +for (b =0 ; b <NFT_PIPAPO_BUCKETS (8 ); b++) { +int src_b0 = b /NFT_PIPAPO_BUCKETS (4 ); +int src_b1 = b %NFT_PIPAPO_BUCKETS (4 ); +int src_i0 = src_g0 *NFT_PIPAPO_BUCKETS (4 ) + src_b0; +int src_i1 = src_g1 *NFT_PIPAPO_BUCKETS (4 ) + src_b1; + +for (i =0 ; i < bsize; i++) { + *new_lt = old_lt[src_i0 * bsize + i] & + old_lt[src_i1 * bsize + i]; + new_lt++; + } + } + } +} + +/** +* pipapo_lt_8b_to_4b() - Switch lookup table group width from 8 bits to 4 bits +* @old_groups: Number of current groups +* @bsize: Size of one bucket, in longs +* @old_lt: Pointer to the current lookup table +* @new_lt: Pointer to the new, pre-allocated lookup table +* +* Each bucket with index b in the new lookup table, belonging to group g, is +* filled with the bit union of: +* - all the buckets with index such that the upper four bits of the lower byte +* equal b, from group g, with g odd +* - all the buckets with index such that the lower four bits equal b, from +* group g, with g even +* +* That is, given buckets from the new lookup table N(x, y) and the old lookup +* table O(x, y), with x bucket index, and y group index: +* +* - with g odd: N(b, g) := U(O(x, g) for each x : x = (b & 0xf0) >> 4) +* - with g even: N(b, g) := U(O(x, g) for each x : x = b & 0x0f) +* +* where U() denotes the arbitrary union operation (binary OR of n terms). This +* ensures equivalence of the matching results on lookup. +*/ +static void pipapo_lt_8b_to_4b (int old_groups,int bsize, +unsigned long *old_lt,unsigned long *new_lt) +{ +int g, b, bsrc, i; + +memset (new_lt,0 , old_groups *2 *NFT_PIPAPO_BUCKETS (4 ) * bsize * +sizeof (unsigned long )); + +for (g =0 ; g < old_groups *2 ; g +=2 ) { +int src_g = g /2 ; + +for (b =0 ; b <NFT_PIPAPO_BUCKETS (4 ); b++) { +for (bsrc =NFT_PIPAPO_BUCKETS (8 ) * src_g; + bsrc <NFT_PIPAPO_BUCKETS (8 ) * (src_g +1 ); + bsrc++) { +if (((bsrc &0xf0 ) >>4 ) != b) +continue ; + +for (i =0 ; i < bsize; i++) + new_lt[i] |= old_lt[bsrc * bsize + i]; + } + + new_lt += bsize; + } + +for (b =0 ; b <NFT_PIPAPO_BUCKETS (4 ); b++) { +for (bsrc =NFT_PIPAPO_BUCKETS (8 ) * src_g; + bsrc <NFT_PIPAPO_BUCKETS (8 ) * (src_g +1 ); + bsrc++) { +if ((bsrc &0x0f ) != b) +continue ; + +for (i =0 ; i < bsize; i++) + new_lt[i] |= old_lt[bsrc * bsize + i]; + } + + new_lt += bsize; + } + } +} + +/** +* pipapo_lt_bits_adjust() - Adjust group size for lookup table if needed +* @f: Field containing lookup table +*/ +static void pipapo_lt_bits_adjust (struct nft_pipapo_field *f) +{ +unsigned long *new_lt; +int groups, bb; +size_t lt_size; + + lt_size = f->groups *NFT_PIPAPO_BUCKETS (f->bb) * f->bsize * +sizeof (*f->lt); + +if (f->bb ==NFT_PIPAPO_GROUP_BITS_SMALL_SET && + lt_size >NFT_PIPAPO_LT_SIZE_HIGH ) { + groups = f->groups *2 ; + bb =NFT_PIPAPO_GROUP_BITS_LARGE_SET ; + + lt_size = groups *NFT_PIPAPO_BUCKETS (bb) * f->bsize * +sizeof (*f->lt); + }else if (f->bb ==NFT_PIPAPO_GROUP_BITS_LARGE_SET && + lt_size <NFT_PIPAPO_LT_SIZE_LOW ) { + groups = f->groups /2 ; + bb =NFT_PIPAPO_GROUP_BITS_SMALL_SET ; + + lt_size = groups *NFT_PIPAPO_BUCKETS (bb) * f->bsize * +sizeof (*f->lt); + +/* Don't increase group width if the resulting lookup table size +* would exceed the upper size threshold for a "small" set. +*/ +if (lt_size >NFT_PIPAPO_LT_SIZE_HIGH ) +return ; + }else { +return ; + } + + new_lt =kvzalloc (lt_size,GFP_KERNEL ); +if (!new_lt) +return ; + +NFT_PIPAPO_GROUP_BITS_ARE_8_OR_4 ; +if (f->bb ==4 && bb ==8 ) +pipapo_lt_4b_to_8b (f->groups, f->bsize, f->lt, new_lt); +else if (f->bb ==8 && bb ==4 ) +pipapo_lt_8b_to_4b (f->groups, f->bsize, f->lt, new_lt); +else +BUG (); + + f->groups = groups; + f->bb = bb; +kvfree (f->lt); + f->lt = new_lt; +} +/** * pipapo_insert() - Insert new rule in field given input key and mask length * @f: Field containing lookup table
@@ -894,6 +1115,8 @@ static int pipapo_insert(struct nft_pipapo_field *f, const uint8_t *k, } } +pipapo_lt_bits_adjust (f); +return 1 ; }
@@ -1424,6 +1647,8 @@ static void pipapo_drop(struct nft_pipapo_match *m, ; } f->rules -= rulemap[i].n; + +pipapo_lt_bits_adjust (f); } }
@@ -1936,7 +2161,7 @@ static bool nft_pipapo_estimate(const struct nft_set_desc *desc, u32 features,*/ rules =ilog2 (desc->field_len[i] *BITS_PER_BYTE ) *2 ; entry_size += rules * -NFT_PIPAPO_BUCKETS (NFT_PIPAPO_GROUP_BITS ) / +NFT_PIPAPO_BUCKETS (NFT_PIPAPO_GROUP_BITS_INIT ) /BITS_PER_BYTE ; entry_size += rules *sizeof (union nft_pipapo_map_bucket ); }
@@ -2001,7 +2226,7 @@ static int nft_pipapo_init(const struct nft_set *set,rcu_head_init (&m->rcu);nft_pipapo_for_each_field (f, i, m) { - f->bb =NFT_PIPAPO_GROUP_BITS ; + f->bb =NFT_PIPAPO_GROUP_BITS_INIT ; f->groups = desc->field_len[i] *NFT_PIPAPO_GROUPS_PER_BYTE (f); priv->width +=round_up (desc->field_len[i],sizeof (u32));
Signature · 1 line
2.25.1
PATCH v2 3/6 nft_set_pipapo: Prepare for vectorised implementation: alignment
In reply to Stefano BrivioSIMD vector extension sets require stricter alignment than native instruction sets to operate efficiently (AVX, NEON) or for some instructions to work at all (AltiVec).
Provide facilities to define arbitrary alignment for lookup tables and scratch maps. By defining byte alignment with NFT_PIPAPO_ALIGN, lt_aligned and scratch_aligned pointers become available.
Additional headroom is allocated, and pointers to the possibly unaligned, originally allocated areas are kept so that they can be freed.
---
v2: Rebase, no changes
net/netfilter/nft_set_pipapo.c | 135 +++++++++++++++++++++++++++------ 1 file changed, 110 insertions(+), 25 deletions(-)
net/netfilter/nft_set_pipapo.c +110 -25
@@ -398,6 +398,22 @@#define NFT_PIPAPO_RULE0_MAX ((1UL << (NFT_PIPAPO_MAP_TOBITS - 1)) \ - (1UL << NFT_PIPAPO_MAP_NBITS)) +/* Definitions for vectorised implementations */ +#ifdef NFT_PIPAPO_ALIGN +#define NFT_PIPAPO_ALIGN_HEADROOM \ + (NFT_PIPAPO_ALIGN - ARCH_KMALLOC_MINALIGN) +#define NFT_PIPAPO_LT_ALIGN (lt) (PTR_ALIGN((lt), NFT_PIPAPO_ALIGN)) +#define NFT_PIPAPO_LT_ASSIGN (field, x) \ + do { \ + (field)->lt_aligned = NFT_PIPAPO_LT_ALIGN(x); \ + (field)->lt = (x); \ + } while (0) +#else +#define NFT_PIPAPO_ALIGN_HEADROOM 0 +#define NFT_PIPAPO_LT_ALIGN (lt) (lt) +#define NFT_PIPAPO_LT_ASSIGN (field, x) ((field)->lt = (x)) +#endif /* NFT_PIPAPO_ALIGN */ +#define nft_pipapo_for_each_field (field, index, match) \ for ((field) = (match)->f, (index) = 0; \ (index) < (match)->field_count; \
@@ -432,6 +448,7 @@ union nft_pipapo_map_bucket {* @bsize: Size of each bucket in lookup table, in longs * @bb: Number of bits grouped together in lookup table buckets * @lt: Lookup table: 'groups' rows of buckets +* @lt_aligned: Version of @lt aligned to NFT_PIPAPO_ALIGN bytes * @mt: Mapping table: one bucket per rule */ struct nft_pipapo_field {
@@ -439,6 +456,9 @@ struct nft_pipapo_field {unsigned long rules;size_t bsize;int bb; +#ifdef NFT_PIPAPO_ALIGN +unsigned long *lt_aligned; +#endif unsigned long *lt;union nft_pipapo_map_bucket *mt; };
@@ -447,12 +467,16 @@ struct nft_pipapo_field {* struct nft_pipapo_match - Data used for lookup and matching * @field_count Amount of fields in set * @scratch: Preallocated per-CPU maps for partial matching results +* @scratch_aligned: Version of @scratch aligned to NFT_PIPAPO_ALIGN bytes * @bsize_max: Maximum lookup table bucket size of all fields, in longs * @rcu Matching data is swapped on commits * @f: Fields, with lookup and mapping tables */ struct nft_pipapo_match {int field_count; +#ifdef NFT_PIPAPO_ALIGN +unsigned long *__percpu *scratch_aligned; +#endif unsigned long *__percpu *scratch;size_t bsize_max;struct rcu_head rcu;
@@ -729,6 +753,7 @@ static struct nft_pipapo_elem *pipapo_get(const struct net *net,memset (res_map,0xff , m->bsize_max *sizeof (*res_map));nft_pipapo_for_each_field (f, i, m) { +unsigned long *lt =NFT_PIPAPO_LT_ALIGN (f->lt);bool last = i == m->field_count -1 ;int b;
@@ -817,6 +842,10 @@ static int pipapo_resize(struct nft_pipapo_field *f, int old_rules, int rules)int group, bucket; new_bucket_size =DIV_ROUND_UP (rules,BITS_PER_LONG ); +#ifdef NFT_PIPAPO_ALIGN + new_bucket_size =roundup (new_bucket_size, +NFT_PIPAPO_ALIGN /sizeof (*new_lt)); +#endif if (new_bucket_size == f->bsize) goto mt;
@@ -827,12 +856,15 @@ static int pipapo_resize(struct nft_pipapo_field *f, int old_rules, int rules) copy = new_bucket_size; new_lt =kvzalloc (f->groups *NFT_PIPAPO_BUCKETS (f->bb) * - new_bucket_size *sizeof (*new_lt),GFP_KERNEL ); + new_bucket_size *sizeof (*new_lt) + +NFT_PIPAPO_ALIGN_HEADROOM , +GFP_KERNEL );if (!new_lt)return -ENOMEM ; - new_p = new_lt; - old_p = old_lt; + new_p =NFT_PIPAPO_LT_ALIGN (new_lt); + old_p =NFT_PIPAPO_LT_ALIGN (old_lt); +for (group =0 ; group < f->groups; group++) {for (bucket =0 ; bucket <NFT_PIPAPO_BUCKETS (f->bb); bucket++) {memcpy (new_p, old_p, copy *sizeof (*new_p));
@@ -861,7 +893,7 @@ static int pipapo_resize(struct nft_pipapo_field *f, int old_rules, int rules)if (new_lt) { f->bsize = new_bucket_size; - f->lt = new_lt; +NFT_PIPAPO_LT_ASSIGN (f, new_lt);kvfree (old_lt); }
@@ -883,7 +915,8 @@ static void pipapo_bucket_set(struct nft_pipapo_field *f, int rule, int group, {unsigned long *pos; - pos = f->lt + f->bsize *NFT_PIPAPO_BUCKETS (f->bb) * group; + pos =NFT_PIPAPO_LT_ALIGN (f->lt); + pos += f->bsize *NFT_PIPAPO_BUCKETS (f->bb) * group; pos += f->bsize * v;__set_bit (rule, pos);
@@ -1048,22 +1081,27 @@ static void pipapo_lt_bits_adjust(struct nft_pipapo_field *f)return ; } - new_lt =kvzalloc (lt_size,GFP_KERNEL ); + new_lt =kvzalloc (lt_size +NFT_PIPAPO_ALIGN_HEADROOM ,GFP_KERNEL );if (!new_lt)return ;NFT_PIPAPO_GROUP_BITS_ARE_8_OR_4 ; -if (f->bb ==4 && bb ==8 ) -pipapo_lt_4b_to_8b (f->groups, f->bsize, f->lt, new_lt); -else if (f->bb ==8 && bb ==4 ) -pipapo_lt_8b_to_4b (f->groups, f->bsize, f->lt, new_lt); -else +if (f->bb ==4 && bb ==8 ) { +pipapo_lt_4b_to_8b (f->groups, f->bsize, +NFT_PIPAPO_LT_ALIGN (f->lt), +NFT_PIPAPO_LT_ALIGN (new_lt)); + }else if (f->bb ==8 && bb ==4 ) { +pipapo_lt_8b_to_4b (f->groups, f->bsize, +NFT_PIPAPO_LT_ALIGN (f->lt), +NFT_PIPAPO_LT_ALIGN (new_lt)); + }else {BUG (); + } f->groups = groups; f->bb = bb;kvfree (f->lt); - f->lt = new_lt; +NFT_PIPAPO_LT_ASSIGN (f, new_lt); }/**
@@ -1289,8 +1327,12 @@ static int pipapo_realloc_scratch(struct nft_pipapo_match *clone,for_each_possible_cpu (i) {unsigned long *scratch; +#ifdef NFT_PIPAPO_ALIGN +unsigned long *scratch_aligned; +#endif - scratch =kzalloc_node (bsize_max *sizeof (*scratch) *2 , + scratch =kzalloc_node (bsize_max *sizeof (*scratch) *2 + +NFT_PIPAPO_ALIGN_HEADROOM ,GFP_KERNEL ,cpu_to_node (i));if (!scratch) {/* On failure, there's no need to undo previous
@@ -1306,6 +1348,11 @@ static int pipapo_realloc_scratch(struct nft_pipapo_match *clone,kfree (*per_cpu_ptr (clone->scratch, i)); *per_cpu_ptr (clone->scratch, i) = scratch; + +#ifdef NFT_PIPAPO_ALIGN + scratch_aligned =NFT_PIPAPO_LT_ALIGN (scratch); + *per_cpu_ptr (clone->scratch_aligned, i) = scratch_aligned; +#endif }return 0 ;
@@ -1433,21 +1480,33 @@ static struct nft_pipapo_match *pipapo_clone(struct nft_pipapo_match *old)if (!new->scratch) goto out_scratch; +#ifdef NFT_PIPAPO_ALIGN + new->scratch_aligned =alloc_percpu (*new->scratch_aligned); +if (!new->scratch_aligned) + goto out_scratch; +#endif +rcu_head_init (&new->rcu); src = old->f; dst = new->f;for (i =0 ; i < old->field_count; i++) { +unsigned long *new_lt; +memcpy (dst, src, offsetof(struct nft_pipapo_field , lt)); - dst->lt =kvzalloc (src->groups *NFT_PIPAPO_BUCKETS (src->bb) * - src->bsize *sizeof (*dst->lt), -GFP_KERNEL ); -if (!dst->lt) + new_lt =kvzalloc (src->groups *NFT_PIPAPO_BUCKETS (src->bb) * + src->bsize *sizeof (*dst->lt) + +NFT_PIPAPO_ALIGN_HEADROOM , +GFP_KERNEL ); +if (!new_lt) goto out_lt; -memcpy (dst->lt, src->lt, +NFT_PIPAPO_LT_ASSIGN (dst, new_lt); + +memcpy (NFT_PIPAPO_LT_ALIGN (new_lt), +NFT_PIPAPO_LT_ALIGN (src->lt), src->bsize *sizeof (*dst->lt) * src->groups *NFT_PIPAPO_BUCKETS (src->bb));
@@ -1470,8 +1529,11 @@ static struct nft_pipapo_match *pipapo_clone(struct nft_pipapo_match *old)kvfree (dst->lt); dst--; } -free_percpu (new->scratch); +#ifdef NFT_PIPAPO_ALIGN +free_percpu (new->scratch_aligned); +#endif out_scratch: +free_percpu (new->scratch);kfree (new);return ERR_PTR (-ENOMEM );
@@ -1627,7 +1689,8 @@ static void pipapo_drop(struct nft_pipapo_match *m,unsigned long *pos;int b; - pos = f->lt + g *NFT_PIPAPO_BUCKETS (f->bb) * f->bsize; + pos =NFT_PIPAPO_LT_ALIGN (f->lt) + g * +NFT_PIPAPO_BUCKETS (f->bb) * f->bsize;for (b =0 ; b <NFT_PIPAPO_BUCKETS (f->bb); b++) { bitmap_cut(pos, pos, rulemap[i].to,
@@ -1733,6 +1796,9 @@ static void pipapo_reclaim_match(struct rcu_head *rcu)for_each_possible_cpu (i)kfree (*per_cpu_ptr (m->scratch, i)); +#ifdef NFT_PIPAPO_ALIGN +free_percpu (m->scratch_aligned); +#endif free_percpu (m->scratch);pipapo_free_fields (m);
@@ -1936,8 +2002,8 @@ static int pipapo_get_boundaries(struct nft_pipapo_field *f, int first_rule,for (b =0 ; b <NFT_PIPAPO_BUCKETS (f->bb); b++) {unsigned long *pos; - pos = f->lt + (g *NFT_PIPAPO_BUCKETS (f->bb) + b) * - f->bsize; + pos =NFT_PIPAPO_LT_ALIGN (f->lt) + + (g *NFT_PIPAPO_BUCKETS (f->bb) + b) * f->bsize;if (test_bit (first_rule, pos) && x0 ==-1 ) x0 = b;if (test_bit (first_rule + rule_count -1 , pos))
@@ -2218,11 +2284,21 @@ static int nft_pipapo_init(const struct nft_set *set, m->scratch = alloc_percpu(unsigned long *);if (!m->scratch) { err = -ENOMEM ; - goto out_free; + goto out_scratch; }for_each_possible_cpu (i) *per_cpu_ptr (m->scratch, i) =NULL ; +#ifdef NFT_PIPAPO_ALIGN + m->scratch_aligned = alloc_percpu(unsigned long *); +if (!m->scratch_aligned) { + err = -ENOMEM ; + goto out_free; + } +for_each_possible_cpu (i) + *per_cpu_ptr (m->scratch_aligned, i) =NULL ; +#endif +rcu_head_init (&m->rcu);nft_pipapo_for_each_field (f, i, m) {
@@ -2233,7 +2309,7 @@ static int nft_pipapo_init(const struct nft_set *set, f->bsize =0 ; f->rules =0 ; - f->lt =NULL ; +NFT_PIPAPO_LT_ASSIGN (f,NULL ); f->mt =NULL ; }
@@ -2251,7 +2327,11 @@ static int nft_pipapo_init(const struct nft_set *set,return 0 ; out_free: +#ifdef NFT_PIPAPO_ALIGN +free_percpu (m->scratch_aligned); +#endif free_percpu (m->scratch); +out_scratch:kfree (m);return err;
@@ -2286,16 +2366,21 @@ static void nft_pipapo_destroy(const struct nft_set *set)nft_set_elem_destroy (set, e, true); } +#ifdef NFT_PIPAPO_ALIGN +free_percpu (m->scratch_aligned); +#endif for_each_possible_cpu (cpu)kfree (*per_cpu_ptr (m->scratch, cpu));free_percpu (m->scratch); -pipapo_free_fields (m);kfree (m); priv->match =NULL ; }if (priv->clone) { +#ifdef NFT_PIPAPO_ALIGN +free_percpu (priv->clone->scratch_aligned); +#endif for_each_possible_cpu (cpu)kfree (*per_cpu_ptr (priv->clone->scratch, cpu));free_percpu (priv->clone->scratch);
Signature · 1 line
2.25.1
PATCH v2 4/6 nft_set_pipapo: Prepare for vectorised implementation: helpers
In reply to Stefano BrivioMove most macros and helpers to a header file, so that they can be conveniently used by related implementations.
No functional changes are intended here.
---
v2: Rebase, no significant changes
net/netfilter/nft_set_pipapo.c | 269 +------------------------------- net/netfilter/nft_set_pipapo.h | 277 +++++++++++++++++++++++++++++++++ 2 files changed, 285 insertions(+), 261 deletions(-) create mode 100644 net/netfilter/nft_set_pipapo.h
net/netfilter/nft_set_pipapo.c +8 -261
@@ -330,188 +330,20 @@#include <linux/kernel.h> #include <linux/init.h> -#include <linux/log2.h> #include <linux/module.h> #include <linux/netlink.h> #include <linux/netfilter.h> #include <linux/netfilter/nf_tables.h> #include <net/netfilter/nf_tables_core.h> #include <uapi/linux/netfilter/nf_tables.h> -#include <net/ipv6.h> /* For the maximum length of a field */ #include <linux/bitmap.h> #include <linux/bitops.h> -/* Count of concatenated fields depends on count of 32-bit nftables registers */ -#define NFT_PIPAPO_MAX_FIELDS NFT_REG32_COUNT - -/* Largest supported field size */ -#define NFT_PIPAPO_MAX_BYTES (sizeof(struct in6_addr)) -#define NFT_PIPAPO_MAX_BITS (NFT_PIPAPO_MAX_BYTES * BITS_PER_BYTE) - -/* Bits to be grouped together in table buckets depending on set size */ -#define NFT_PIPAPO_GROUP_BITS_INIT NFT_PIPAPO_GROUP_BITS_SMALL_SET -#define NFT_PIPAPO_GROUP_BITS_SMALL_SET 8 -#define NFT_PIPAPO_GROUP_BITS_LARGE_SET 4 -#define NFT_PIPAPO_GROUP_BITS_ARE_8_OR_4 \ - BUILD_BUG_ON((NFT_PIPAPO_GROUP_BITS_SMALL_SET != 8) || \ - (NFT_PIPAPO_GROUP_BITS_LARGE_SET != 4)) -#define NFT_PIPAPO_GROUPS_PER_BYTE (f) (BITS_PER_BYTE / (f)->bb) - -/* If a lookup table gets bigger than NFT_PIPAPO_LT_SIZE_HIGH, switch to the -* small group width, and switch to the big group width if the table gets -* smaller than NFT_PIPAPO_LT_SIZE_LOW. -* -* Picking 2MiB as threshold (for a single table) avoids as much as possible -* crossing page boundaries on most architectures (x86-64 and MIPS huge pages, -* ARMv7 supersections, POWER "large" pages, SPARC Level 1 regions, etc.), which -* keeps performance nice in case kvmalloc() gives us non-contiguous areas. -*/ -#define NFT_PIPAPO_LT_SIZE_THRESHOLD (1 << 21) -#define NFT_PIPAPO_LT_SIZE_HYSTERESIS (1 << 16) -#define NFT_PIPAPO_LT_SIZE_HIGH NFT_PIPAPO_LT_SIZE_THRESHOLD -#define NFT_PIPAPO_LT_SIZE_LOW NFT_PIPAPO_LT_SIZE_THRESHOLD - \ - NFT_PIPAPO_LT_SIZE_HYSTERESIS - -/* Fields are padded to 32 bits in input registers */ -#define NFT_PIPAPO_GROUPS_PADDED_SIZE (f) \ - (round_up((f)->groups / NFT_PIPAPO_GROUPS_PER_BYTE(f), sizeof(u32))) -#define NFT_PIPAPO_GROUPS_PADDING (f) \ - (NFT_PIPAPO_GROUPS_PADDED_SIZE(f) - (f)->groups / \ - NFT_PIPAPO_GROUPS_PER_BYTE(f)) - -/* Number of buckets given by 2 ^ n, with n bucket bits */ -#define NFT_PIPAPO_BUCKETS (bb) (1 << (bb)) - -/* Each n-bit range maps to up to n * 2 rules */ -#define NFT_PIPAPO_MAP_NBITS (const_ilog2(NFT_PIPAPO_MAX_BITS * 2)) - -/* Use the rest of mapping table buckets for rule indices, but it makes no sense -* to exceed 32 bits -*/ -#if BITS_PER_LONG ==64 -#define NFT_PIPAPO_MAP_TOBITS 32 -#else -#define NFT_PIPAPO_MAP_TOBITS (BITS_PER_LONG - NFT_PIPAPO_MAP_NBITS) -#endif - -/* ...which gives us the highest allowed index for a rule */ -#define NFT_PIPAPO_RULE0_MAX ((1UL << (NFT_PIPAPO_MAP_TOBITS - 1)) \ - - (1UL << NFT_PIPAPO_MAP_NBITS)) - -/* Definitions for vectorised implementations */ -#ifdef NFT_PIPAPO_ALIGN -#define NFT_PIPAPO_ALIGN_HEADROOM \ - (NFT_PIPAPO_ALIGN - ARCH_KMALLOC_MINALIGN) -#define NFT_PIPAPO_LT_ALIGN (lt) (PTR_ALIGN((lt), NFT_PIPAPO_ALIGN)) -#define NFT_PIPAPO_LT_ASSIGN (field, x) \ - do { \ - (field)->lt_aligned = NFT_PIPAPO_LT_ALIGN(x); \ - (field)->lt = (x); \ - } while (0) -#else -#define NFT_PIPAPO_ALIGN_HEADROOM 0 -#define NFT_PIPAPO_LT_ALIGN (lt) (lt) -#define NFT_PIPAPO_LT_ASSIGN (field, x) ((field)->lt = (x)) -#endif /* NFT_PIPAPO_ALIGN */ - -#define nft_pipapo_for_each_field (field, index, match) \ - for ((field) = (match)->f, (index) = 0; \ - (index) < (match)->field_count; \ - (index)++, (field)++) - -/** -* union nft_pipapo_map_bucket - Bucket of mapping table -* @to: First rule number (in next field) this rule maps to -* @n: Number of rules (in next field) this rule maps to -* @e: If there's no next field, pointer to element this rule maps to -*/ -union nft_pipapo_map_bucket { -struct { -#if BITS_PER_LONG ==64 - static_assert(NFT_PIPAPO_MAP_TOBITS <= 32); -u32 to; - - static_assert(NFT_PIPAPO_MAP_NBITS <= 32); -u32 n; -#else -unsigned long to:NFT_PIPAPO_MAP_TOBITS ; -unsigned long n:NFT_PIPAPO_MAP_NBITS ; -#endif - }; -struct nft_pipapo_elem *e; -}; - -/** -* struct nft_pipapo_field - Lookup, mapping tables and related data for a field -* @groups: Amount of bit groups -* @rules: Number of inserted rules -* @bsize: Size of each bucket in lookup table, in longs -* @bb: Number of bits grouped together in lookup table buckets -* @lt: Lookup table: 'groups' rows of buckets -* @lt_aligned: Version of @lt aligned to NFT_PIPAPO_ALIGN bytes -* @mt: Mapping table: one bucket per rule -*/ -struct nft_pipapo_field { -int groups; -unsigned long rules; -size_t bsize; -int bb; -#ifdef NFT_PIPAPO_ALIGN -unsigned long *lt_aligned; -#endif -unsigned long *lt; -union nft_pipapo_map_bucket *mt; -}; - -/** -* struct nft_pipapo_match - Data used for lookup and matching -* @field_count Amount of fields in set -* @scratch: Preallocated per-CPU maps for partial matching results -* @scratch_aligned: Version of @scratch aligned to NFT_PIPAPO_ALIGN bytes -* @bsize_max: Maximum lookup table bucket size of all fields, in longs -* @rcu Matching data is swapped on commits -* @f: Fields, with lookup and mapping tables -*/ -struct nft_pipapo_match { -int field_count; -#ifdef NFT_PIPAPO_ALIGN -unsigned long *__percpu *scratch_aligned; -#endif -unsigned long *__percpu *scratch; -size_t bsize_max; -struct rcu_head rcu; -struct nft_pipapo_field f[]; -}; +#include "nft_set_pipapo.h" /* Current working bitmap index, toggled between field matches */ static DEFINE_PER_CPU(bool, nft_pipapo_scratch_index); -/** -* struct nft_pipapo - Representation of a set -* @match: Currently in-use matching data -* @clone: Copy where pending insertions and deletions are kept -* @width: Total bytes to be matched for one packet, including padding -* @dirty: Working copy has pending insertions or deletions -* @last_gc: Timestamp of last garbage collection run, jiffies -*/ -struct nft_pipapo { -struct nft_pipapo_match __rcu *match; -struct nft_pipapo_match *clone; -int width; -bool dirty; -unsigned long last_gc; -}; - -struct nft_pipapo_elem ; - -/** -* struct nft_pipapo_elem - API-facing representation of single set element -* @ext: nftables API extensions -*/ -struct nft_pipapo_elem { -struct nft_set_ext ext; -}; -/** * pipapo_refill() - For each set bit, set bits from selected mapping table item * @map: Bitmap to be scanned for set bits
@@ -529,9 +361,8 @@ struct nft_pipapo_elem {* * Return: -1 on no match, bit position on 'match_only', 0 otherwise. */ -static int pipapo_refill (unsigned long *map,int len,int rules, -unsigned long *dst,union nft_pipapo_map_bucket *mt, -bool match_only) +int pipapo_refill (unsigned long *map,int len,int rules,unsigned long *dst, +union nft_pipapo_map_bucket *mt,bool match_only) {unsigned long bitset;int k, ret =-1 ;
@@ -565,54 +396,6 @@ static int pipapo_refill(unsigned long *map, int len, int rules,return ret; } -/** -* pipapo_and_field_buckets_4bit() - Intersect buckets for 4-bit groups -* @f: Field including lookup table -* @dst: Area to store result -* @data: Input data selecting table buckets -*/ -static void pipapo_and_field_buckets_4bit (struct nft_pipapo_field *f, -unsigned long *dst, -const u8 *data) -{ -unsigned long *lt = f->lt; -int group; - -for (group =0 ; group < f->groups; group +=BITS_PER_BYTE /4 , data++) { -u8 v; - - v = *data >>4 ; -__bitmap_and (dst, dst, lt + v * f->bsize, - f->bsize *BITS_PER_LONG ); - lt += f->bsize *NFT_PIPAPO_BUCKETS (4 ); - - v = *data &0x0f ; -__bitmap_and (dst, dst, lt + v * f->bsize, - f->bsize *BITS_PER_LONG ); - lt += f->bsize *NFT_PIPAPO_BUCKETS (4 ); - } -} - -/** -* pipapo_and_field_buckets_8bit() - Intersect buckets for 8-bit groups -* @f: Field including lookup table -* @dst: Area to store result -* @data: Input data selecting table buckets -*/ -static void pipapo_and_field_buckets_8bit (struct nft_pipapo_field *f, -unsigned long *dst, -const u8 *data) -{ -unsigned long *lt = f->lt; -int group; - -for (group =0 ; group < f->groups; group++, data++) { -__bitmap_and (dst, dst, lt + *data * f->bsize, - f->bsize *BITS_PER_LONG ); - lt += f->bsize *NFT_PIPAPO_BUCKETS (8 ); - } -} -/** * nft_pipapo_lookup() - Lookup function * @net: Network namespace
@@ -753,7 +536,6 @@ static struct nft_pipapo_elem *pipapo_get(const struct net *net,memset (res_map,0xff , m->bsize_max *sizeof (*res_map));nft_pipapo_for_each_field (f, i, m) { -unsigned long *lt =NFT_PIPAPO_LT_ALIGN (f->lt);bool last = i == m->field_count -1 ;int b;
@@ -2190,58 +1972,23 @@ static u64 nft_pipapo_privsize(const struct nlattr * const nla[], }/** -* nft_pipapo_estimate() - Estimate set size, space and lookup complexity -* @desc: Set description, element count and field description used here +* nft_pipapo_estimate() - Set size, space and lookup complexity +* @desc: Set description, element count and field description used * @features: Flags: NFT_SET_INTERVAL needs to be there * @est: Storage for estimation data * -* The size for this set type can vary dramatically, as it depends on the number -* of rules (composing netmasks) the entries expand to. We compute the worst -* case here. -* -* In general, for a non-ranged entry or a single composing netmask, we need -* one bit in each of the sixteen buckets, for each 4-bit group (that is, each -* input bit needs four bits of matching data), plus a bucket in the mapping -* table for each field. -* -* Return: true only for compatible range concatenations +* Return: true if set description is compatible, false otherwise */ static bool nft_pipapo_estimate (const struct nft_set_desc *desc,u32 features,struct nft_set_estimate *est) { -unsigned long entry_size; -int i; -if (!(features &NFT_SET_INTERVAL ) || desc->field_count <=1 )return false; -for (i =0 , entry_size =0 ; i < desc->field_count; i++) { -unsigned long rules; - -if (desc->field_len[i] >NFT_PIPAPO_MAX_BYTES ) -return false; - -/* Worst-case ranges for each concatenated field: each n-bit -* field can expand to up to n * 2 rules in each bucket, and -* each rule also needs a mapping bucket. -*/ - rules =ilog2 (desc->field_len[i] *BITS_PER_BYTE ) *2 ; - entry_size += rules * -NFT_PIPAPO_BUCKETS (NFT_PIPAPO_GROUP_BITS_INIT ) / -BITS_PER_BYTE ; - entry_size += rules *sizeof (union nft_pipapo_map_bucket ); - } - -/* Rules in lookup and mapping tables are needed for each entry */ - est->size = desc->size * entry_size; -if (est->size &&div_u64 (est->size, desc->size) != entry_size) + est->size =pipapo_estimate_size (desc); +if (!est->size)return false; - est->size +=sizeof (struct nft_pipapo ) + -sizeof (struct nft_pipapo_match ) *2 ; - - est->size +=sizeof (struct nft_pipapo_field ) * desc->field_count; - est->lookup =NFT_SET_CLASS_O_LOG_N ; est->space =NFT_SET_CLASS_O_N ;
net/netfilter/nft_set_pipapo.h new file +277 -0
277 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 6/6 nft_set_pipapo: Prepare for single ranged field usage
In reply to Stefano BrivioA few adjustments in nft_pipapo_init() are needed to allow usage of this set back-end for a single, ranged field.
Provide a convenient NFT_PIPAPO_MIN_FIELDS definition that currently makes sure that the rbtree back-end is selected instead, for sets with a single field.
This finally allows a fair comparison with rbtree sets, by defining NFT_PIPAPO_MIN_FIELDS as 0 and skipping rbtree back-end initialisation:
---------------.--------------------------.-------------------------.
AMD Epyc 7402 | baselines, Mpps | Mpps, % over rbtree |
1 thread |__________________________|_________________________|
3.35GHz | | | | | |
768KiB L1D$ | netdev | hash | rbtree | | pipapo |
---------------| hook | no | single | pipapo |single field|
type entries | drop | ranges | field |single field| AVX2 |
---------------|--------|--------|--------|------------|------------|
net,port | | | | | |
1000 | 19.0 | 10.4 | 3.8 | 6.0 +58% | 9.6 +153% |
---------------|--------|--------|--------|------------|------------|
port,net | | | | | |
100 | 18.8 | 10.3 | 5.8 | 9.1 +57% |11.6 +100% |
---------------|--------|--------|--------|------------|------------|
net6,port | | | | | |
1000 | 16.4 | 7.6 | 1.8 | 2.8 +55% | 6.5 +261% |
---------------|--------|--------|--------|------------|------------|
port,proto | | | | [1] | [1] |
30000 | 19.6 | 11.6 | 3.9 | 0.9 -77% | 2.7 -31% |
---------------|--------|--------|--------|------------|------------|
port,proto | | | | | |
10000 | 19.6 | 11.6 | 4.4 | 2.1 -52% | 5.6 +27% |
---------------|--------|--------|--------|------------|------------|
port,proto | | | | | |
4 threads 10000| 77.9 | 45.1 | 17.4 | 8.3 -52% |22.4 +29% |
---------------|--------|--------|--------|------------|------------|
net6,port,mac | | | | | |
10 | 16.5 | 5.4 | 4.3 | 4.5 +5% | 8.2 +91% |
---------------|--------|--------|--------|------------|------------|
net6,port,mac, | | | | | |
proto 1000 | 16.5 | 5.7 | 1.9 | 2.8 +47% | 6.6 +247% |
---------------|--------|--------|--------|------------|------------|
net,mac | | | | | |
1000 | 19.0 | 8.4 | 3.9 | 6.0 +54% | 9.9 +154% | ---------------'--------'--------'--------'------------'------------'
[1] Causes switch of lookup table buckets for 'port' to 4-bit groups
---
v2: New patch
net/netfilter/nft_set_pipapo.c | 19 ++++++++++++------- net/netfilter/nft_set_pipapo.h | 3 +++ net/netfilter/nft_set_pipapo_avx2.c | 3 ++- 3 files changed, 17 insertions(+), 8 deletions(-)
net/netfilter/nft_set_pipapo.c +12 -7
@@ -1983,7 +1983,8 @@ static u64 nft_pipapo_privsize(const struct nlattr * const nla[],static bool nft_pipapo_estimate (const struct nft_set_desc *desc,u32 features,struct nft_set_estimate *est) { -if (!(features &NFT_SET_INTERVAL ) || desc->field_count <=1 ) +if (!(features &NFT_SET_INTERVAL ) || + desc->field_count <NFT_PIPAPO_MIN_FIELDS )return false; est->size =pipapo_estimate_size (desc);
@@ -2016,17 +2017,19 @@ static int nft_pipapo_init(const struct nft_set *set,struct nft_pipapo *priv =nft_set_priv (set);struct nft_pipapo_match *m;struct nft_pipapo_field *f; -int err, i; +int err, i, field_count; -if (desc->field_count >NFT_PIPAPO_MAX_FIELDS ) + field_count = desc->field_count ? :1 ; + +if (field_count >NFT_PIPAPO_MAX_FIELDS )return -EINVAL ; - m =kmalloc (sizeof (*priv->match) +sizeof (*f) * desc->field_count, + m =kmalloc (sizeof (*priv->match) +sizeof (*f) * field_count,GFP_KERNEL );if (!m)return -ENOMEM ; - m->field_count = desc->field_count; + m->field_count = field_count; m->bsize_max =0 ; m->scratch = alloc_percpu(unsigned long *);
@@ -2050,10 +2053,12 @@ static int nft_pipapo_init(const struct nft_set *set,rcu_head_init (&m->rcu);nft_pipapo_for_each_field (f, i, m) { +int len = desc->field_len[i] ? : set->klen; + f->bb =NFT_PIPAPO_GROUP_BITS_INIT ; - f->groups = desc->field_len[i] *NFT_PIPAPO_GROUPS_PER_BYTE (f); + f->groups = len *NFT_PIPAPO_GROUPS_PER_BYTE (f); - priv->width +=round_up (desc->field_len[i],sizeof (u32)); + priv->width +=round_up (len,sizeof (u32)); f->bsize =0 ; f->rules =0 ;
net/netfilter/nft_set_pipapo.h +3 -0
@@ -8,6 +8,9 @@/* Count of concatenated fields depends on count of 32-bit nftables registers */ #define NFT_PIPAPO_MAX_FIELDS NFT_REG32_COUNT +/* Restrict usage to multiple fields, make sure rbtree is used otherwise */ +#define NFT_PIPAPO_MIN_FIELDS 2 +/* Largest supported field size */ #define NFT_PIPAPO_MAX_BYTES (sizeof(struct in6_addr))#define NFT_PIPAPO_MAX_BITS (NFT_PIPAPO_MAX_BYTES * BITS_PER_BYTE)
net/netfilter/nft_set_pipapo_avx2.c +2 -1
@@ -1087,7 +1087,8 @@ static int nft_pipapo_avx2_lookup_slow(unsigned long *map, unsigned long *fill,bool nft_pipapo_avx2_estimate (const struct nft_set_desc *desc,u32 features,struct nft_set_estimate *est) { -if (!(features &NFT_SET_INTERVAL ) || desc->field_count <=1 ) +if (!(features &NFT_SET_INTERVAL ) || + desc->field_count <NFT_PIPAPO_MIN_FIELDS )return false;if (!boot_cpu_has (X86_FEATURE_AVX2 ) || !boot_cpu_has (X86_FEATURE_AVX ))
Signature · 1 line
2.25.1
PATCH v2 5/6 nft_set_pipapo: Introduce AVX2-based lookup implementation
In reply to Stefano BrivioIf the AVX2 set is available, we can exploit the repetitive characteristic of this algorithm to provide a fast, vectorised version by using 256-bit wide AVX2 operations for bucket loads and bitwise intersections.
In most cases, this implementation consistently outperforms rbtree set instances despite the fact they are configured to use a given, single, ranged data type out of the ones used for performance measurements by the nft_concat_range.sh kselftest.
That script, injecting packets directly on the ingoing device path with pktgen, reports, averaged over five runs on a single AMD Epyc 7402 thread (3.35GHz, 768 KiB L1D$, 12 MiB L2$), the figures below. CONFIG_RETPOLINE was not set here.
Note that this is not a fair comparison over hash and rbtree set
types: non-ranged entries (used to have a reference for hash types) would be matched faster than this, and matching on a single field only (which is the case for rbtree) is also significantly faster.
However, it's not possible at the moment to choose this set type for non-ranged entries, and the current implementation also needs a few minor adjustments in order to match on less than two fields.
---------------.-----------------------------------.------------.
AMD Epyc 7402 | baselines, Mpps | this patch |
1 thread |___________________________________|____________|
3.35GHz | | | | | |
768KiB L1D$ | netdev | hash | rbtree | | |
---------------| hook | no | single | | pipapo |
type entries | drop | ranges | field | pipapo | AVX2 |
---------------|--------|--------|--------|--------|------------|
net,port | | | | | |
1000 | 19.0 | 10.4 | 3.8 | 4.0 | 7.5 +87% |
---------------|--------|--------|--------|--------|------------|
port,net | | | | | |
100 | 18.8 | 10.3 | 5.8 | 6.3 | 8.1 +29% |
---------------|--------|--------|--------|--------|------------|
net6,port | | | | | |
1000 | 16.4 | 7.6 | 1.8 | 2.1 | 4.8 +128% |
---------------|--------|--------|--------|--------|------------|
port,proto | | | | | |
30000 | 19.6 | 11.6 | 3.9 | 0.5 | 2.6 +420% |
---------------|--------|--------|--------|--------|------------|
net6,port,mac | | | | | |
10 | 16.5 | 5.4 | 4.3 | 3.4 | 4.7 +38% |
---------------|--------|--------|--------|--------|------------|
net6,port,mac, | | | | | |
proto 1000 | 16.5 | 5.7 | 1.9 | 1.4 | 3.6 +26% |
---------------|--------|--------|--------|--------|------------|
net,mac | | | | | |
1000 | 19.0 | 8.4 | 3.9 | 2.5 | 6.4 +156% |---------------'--------'--------'--------'--------'------------'
A similar strategy could be easily reused to implement specialised versions for other SIMD sets, and I plan to post at least a NEON version at a later time.
---
v2: Rebase, no significant changes
include/net/netfilter/nf_tables_core.h | 1 + net/netfilter/Makefile | 6 + net/netfilter/nf_tables_api.c | 3 + net/netfilter/nft_set_pipapo.c | 24 + net/netfilter/nft_set_pipapo_avx2.c | 1222 ++++++++++++++++++++++++ net/netfilter/nft_set_pipapo_avx2.h | 14 + 6 files changed, 1270 insertions(+) create mode 100644 net/netfilter/nft_set_pipapo_avx2.c create mode 100644 net/netfilter/nft_set_pipapo_avx2.h
include/net/netfilter/nf_tables_core.h +1 -0
@@ -75,6 +75,7 @@ extern const struct nft_set_type nft_set_hash_fast_type;extern const struct nft_set_type nft_set_rbtree_type;extern const struct nft_set_type nft_set_bitmap_type;extern const struct nft_set_type nft_set_pipapo_type; +extern const struct nft_set_type nft_set_pipapo_avx2_type;struct nft_expr ;struct nft_regs ;
net/netfilter/Makefile +6 -0
@@ -82,6 +82,12 @@ nf_tables-objs := nf_tables_core.o nf_tables_api.o nft_chain_filter.o \ nft_set_hash.o nft_set_bitmap.o nft_set_rbtree.o \ nft_set_pipapo.o +ifdef CONFIG_X86_64 +ifneq (,$(findstring -DCONFIG_AS_AVX2=1,$(KBUILD_CFLAGS))) +nf_tables-objs += nft_set_pipapo_avx2.o +endif +endif + obj-$(CONFIG_NF_TABLES) += nf_tables.o obj-$(CONFIG_NFT_COMPAT) += nft_compat.o obj-$(CONFIG_NFT_CONNLIMIT) += nft_connlimit.o
net/netfilter/nf_tables_api.c +3 -0
@@ -3267,6 +3267,9 @@ static const struct nft_set_type *nft_set_types[] = { &nft_set_rhash_type, &nft_set_bitmap_type, &nft_set_rbtree_type, +#if defined(CONFIG_X86_64) && defined(CONFIG_AS_AVX2) + &nft_set_pipapo_avx2_type, +#endif &nft_set_pipapo_type, };
net/netfilter/nft_set_pipapo.c +24 -0
@@ -339,6 +339,7 @@#include <linux/bitmap.h> #include <linux/bitops.h> +#include "nft_set_pipapo_avx2.h" #include "nft_set_pipapo.h" /* Current working bitmap index, toggled between field matches */
@@ -2174,3 +2175,26 @@ const struct nft_set_type nft_set_pipapo_type = { .elemsize = offsetof(struct nft_pipapo_elem, ext), }, }; + +#if defined(CONFIG_X86_64 ) && defined(CONFIG_AS_AVX2 ) +const struct nft_set_type nft_set_pipapo_avx2_type = { + .features =NFT_SET_INTERVAL |NFT_SET_MAP |NFT_SET_OBJECT | +NFT_SET_TIMEOUT , + .ops = { + .lookup = nft_pipapo_avx2_lookup, + .insert = nft_pipapo_insert, + .activate = nft_pipapo_activate, + .deactivate = nft_pipapo_deactivate, + .flush = nft_pipapo_flush, + .remove = nft_pipapo_remove, + .walk = nft_pipapo_walk, + .get = nft_pipapo_get, + .privsize = nft_pipapo_privsize, + .estimate = nft_pipapo_avx2_estimate, + .init = nft_pipapo_init, + .destroy = nft_pipapo_destroy, + .gc_init = nft_pipapo_gc_init, + .elemsize = offsetof(struct nft_pipapo_elem , ext), + }, +}; +#endif
net/netfilter/nft_set_pipapo_avx2.c new file +1222 -0
1,222 lines not on this page · raw message
net/netfilter/nft_set_pipapo_avx2.h new file +14 -0
@@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +#ifndef _NFT_SET_PIPAPO_AVX2_H + +#ifdef CONFIG_AS_AVX2 +#include <asm/fpu/xstate.h> +#define NFT_PIPAPO_ALIGN (XSAVE_YMM_SIZE / BITS_PER_BYTE) + +bool nft_pipapo_avx2_lookup (const struct net *net,const struct nft_set *set, +const u32 *key,const struct nft_set_ext **ext); +bool nft_pipapo_avx2_estimate (const struct nft_set_desc *desc,u32 features, +struct nft_set_estimate *est); +#endif /* CONFIG_AS_AVX2 */ + +#endif /* _NFT_SET_PIPAPO_AVX2_H */
Signature · 1 line
2.25.1
Pablo Neira Ayuso
maintainerSeries applied, thanks.