Skip to content

nftables: Set implementation for arbitrary concatenation of ranges

Stefano Brivio netfilter-devel oe-kbuild-all 4 versions – In review

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v4 Latest9 patches 9 repliesIn review
  2. v39 patches 1 replySuperseded
  3. v28 patches 14 repliesSuperseded
Earlier versions · 1
VersionPostedPatchesRepliesStatus
v18 patches13 repliesSuperseded

Changes in v4 from the cover letter

- fix build for 32-bit architectures: 64-bit division needs div_u64() in 5/9 (kbuild test robot <[email protected]>)

Patches in v4 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/9netfilter: nf_tables: add nft_setelem_parse_key()net/netfilter/nf_tables_api.c+45 -460 replies
2/9netfilter: nf_tables: add NFTA_SET_ELEM_KEY_END attributeinclude/net/netfilter/nf_tables.h include/uapi/linux/netfilter/nf_tables.h net/netfilter/nf_tables_api.c +1 more+79 -240 replies
3/9netfilter: nf_tables: Support for sets with multiple ranged fieldsinclude/net/netfilter/nf_tables.h include/uapi/linux/netfilter/nf_tables.h net/netfilter/nf_tables_api.c +1 more+115 -10 replies
4/9bitmap: Introduce bitmap_cut(): cut bits and shift remaininginclude/linux/bitmap.h lib/bitmap.c+70 -00 replies
5/9nf_tables: Add set type for arbitrary concatenation of rangesinclude/net/netfilter/nf_tables_core.h net/netfilter/Makefile net/netfilter/nf_tables_set_core.c +1 more+2107 -16 replies
6/9selftests: netfilter: Introduce tests for sets with range concatenationtools/testing/selftests/netfilter/Makefile tools/testing/selftests/netfilter/nft_concat_range.sh+1483 -10 replies
7/9nft_set_pipapo: Prepare for vectorised implementation: alignmentnet/netfilter/nft_set_pipapo.c+97 -180 replies
8/9nft_set_pipapo: Prepare for vectorised implementation: helpersnet/netfilter/nft_set_pipapo.c net/netfilter/nft_set_pipapo.h+248 -2050 replies
9/9nft_set_pipapo: Introduce AVX2-based lookup implementationinclude/net/netfilter/nf_tables_core.h net/netfilter/Makefile net/netfilter/nf_tables_set_core.c +3 more+893 -00 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v4 yet.

Read the v4 thread

Cover letter [PATCH nf-next v4 0/9] nftables: Set implementation for arbitrary concatenation of ranges

Existing nftables set implementations allow matching entries with interval expressions (rbtree), e.g. 192.0.2.1-192.0.2.4, entries specifying field concatenation (hash, rhash), e.g. 192.0.2.1:22, but not both.

In other words, none of the set types allows matching on range expressions for more than one packet field at a time, such as ipset does with types bitmap:ip,mac, and, to a more limited extent (netmasks, not arbitrary ranges), with types hash:net,net, hash:net,port, hash:ip,port,net, and hash:net,port,net.

As a pure hash-based approach is unsuitable for matching on ranges, and "proxying" the existing red-black tree type looks impractical as elements would need to be shared and managed across all employed trees, this new set implementation intends to fill the functionality gap by employing a relatively novel approach.

The fundamental idea, illustrated in deeper detail in patch 5/9, is to use lookup tables classifying a small number of grouped bits from each field, and map the lookup results in a way that yields a verdict for the full set of specified fields.

The grouping bit aspect is loosely inspired by the Grouper algorithm, by Jay Ligatti, Josh Kuhn, and Chris Gage (see patch 5/9 for the full reference).

A reference, stand-alone implementation of the algorithm itself is available at:
https://pipapo.lameexcu.se

Some notes about possible future optimisations are also mentioned there. This algorithm reduces the matching problem to, essentially, a repetitive sequence of simple bitwise operations, and is particularly suitable to be optimised by leveraging SIMD instruction sets. An AVX2-based implementation is also presented in this series.

I plan to post the adaptation of the existing AVX2 vectorised implementation for (at least) NEON at a later time.

Patches 1/9 to 3/9 implement the needed infrastructure: new attributes are used to describe length of single ranged fields in concatenations and to denote the upper bound for ranges.

Patch 4/9 adds a new bitmap operation that copies the source bitmap onto the destination while removing a given region, and is needed to delete regions of arrays mapping between lookup tables.

Patch 5/9 is the actual set implementation.

Patch 6/9 introduces selftests for the new implementation.

Patches 7/9 and 8/9 are preparatory work to add an alternative, vectorised lookup implementation.

Patch 9/9 contains the AVX2-based implementation of the lookup routines.

The nftables and libnftnl counterparts depend on changes to the UAPI header file included in patches 2/9 and 3/9.

Credits go to Jay Ligatti, Josh Kuhn, and Chris Gage for their original Grouper implementation and article from ICCCN proceedings (see reference in patch 5/9), and to Daniel Lemire for his public domain implementation of a fast iterator on set bits using built-in implementations of the CTZL operation, also included in patch 5/9.

Special thanks go to Florian Westphal for all the nftables consulting and the original interface idea, to Sabrina Dubroca for support with RCU and bit manipulation topics, to Eric Garver for an early review, to Phil Sutter for reaffirming the need for the use case covered here, and to Pablo Neira Ayuso for proposing a dramatic simplification of the infrastructure.

v4:
- fix build for 32-bit architectures: 64-bit division needs div_u64() in 5/9 (kbuild test robot <[email protected]>)
v3:
- add patches 1/9 and 2/9
- drop patch 5/8 (unrolled lookup loops), as it actually decreases matching rates in some cases
- other changes listed in single patches
v2: Changes listed in messages for 3/8 and 8/8

Pablo Neira Ayuso (2):
netfilter: nf_tables: add nft_setelem_parse_key()
netfilter: nf_tables: add NFTA_SET_ELEM_KEY_END attribute

Stefano Brivio (7):
netfilter: nf_tables: Support for sets with multiple ranged fields
bitmap: Introduce bitmap_cut(): cut bits and shift remaining
nf_tables: Add set type for arbitrary concatenation of ranges
selftests: netfilter: Introduce tests for sets with range
concatenation
nft_set_pipapo: Prepare for vectorised implementation: alignment
nft_set_pipapo: Prepare for vectorised implementation: helpers
nft_set_pipapo: Introduce AVX2-based lookup implementation

 include/linux/bitmap.h                        |    4 +
 include/net/netfilter/nf_tables.h             |   22 +-
 include/net/netfilter/nf_tables_core.h        |    2 +
 include/uapi/linux/netfilter/nf_tables.h      |   17 +
 lib/bitmap.c                                  |   66 +
 net/netfilter/Makefile                        |    8 +-
 net/netfilter/nf_tables_api.c                 |  260 ++-
 net/netfilter/nf_tables_set_core.c            |    8 +
 net/netfilter/nft_dynset.c                    |    2 +-
 net/netfilter/nft_set_pipapo.c                | 2012 +++++++++++++++++
 net/netfilter/nft_set_pipapo.h                |  237 ++
 net/netfilter/nft_set_pipapo_avx2.c           |  842 +++++++
 net/netfilter/nft_set_pipapo_avx2.h           |   14 +
 net/netfilter/nft_set_rbtree.c                |    3 +
 tools/testing/selftests/netfilter/Makefile    |    3 +-
 .../selftests/netfilter/nft_concat_range.sh   | 1481 ++++++++++++
 16 files changed, 4911 insertions(+), 70 deletions(-)
 create mode 100644 net/netfilter/nft_set_pipapo.c
 create mode 100644 net/netfilter/nft_set_pipapo.h
 create mode 100644 net/netfilter/nft_set_pipapo_avx2.c
 create mode 100644 net/netfilter/nft_set_pipapo_avx2.h
 create mode 100755 tools/testing/selftests/netfilter/nft_concat_range.sh
Signature · 1 line
2.24.1