netfilter: nf_tables: coalesce multiple notifications into one skbuff
Subsystems (2)
Pablo Neira Ayuso
maintainerPATCH 1/1 netfilter: nf_tables: coalesce multiple notifications into one skbuff
On x86_64, each notification results in one skbuff allocation which consumes at least 768 bytes due to the skbuff overhead.
This patch coalesces several notifications into one single skbuff, so each notification consumes at least ~211 bytes, that ~3.5 times less memory consumption. As a result, this is reducing the chances to exhaust the netlink socket receive buffer.
Rule of thumb is that each notification batch only contains netlink messages whose report flag is the same, nfnetlink_send() requires this to do appropriately delivery to userspace, either via unicast (echo mode) or multicast (monitor mode).
The skbuff control buffer is used to annotate the report flag for later handling at the new coalescing routine.
The batch skbuff notification size is NLMSG_GOODSIZE, using a larger skbuff would allow for more socket receiver buffer savings (to amortize the cost of the skbuff even more), however, going over that size might break userspace applications, so let's be conservative and stick to NLMSG_GOODSIZE.
---
@Phil: I don't know yet where to target this patch to, if this helps
firewalld's issue, it might be a good option to enqueue this for nf.git
As a side note, it should be possible to skip alloc_skb() from _notify() then memcpy() and kfree_skb(), ie. start batching a bit earlier, but I leave this for the future in favour of this simplistic approach.
include/net/netns/nftables.h | 1 + net/netfilter/nf_tables_api.c | 71 ++++++++++++++++++++++++++++------- 2 files changed, 59 insertions(+), 13 deletions(-)
include/net/netns/nftables.h +1 -0
@@ -8,6 +8,7 @@ struct netns_nftables {struct list_head tables;struct list_head commit_list;struct list_head module_list; +struct list_head notify_list;struct mutex commit_mutex;unsigned int base_seq;u8 gencursor;
net/netfilter/nf_tables_api.c +58 -13
143 lines not on this page · raw message
Signature · 1 line
2.20.1
Phil Sutter
reviewerHi Pablo,
Quoted 13 lines · Pablo Neira Ayuso
On
x86_64, each notification results in one skbuff allocation which consumes at least 768 bytes due to the skbuff overhead.This patch coalesces several notifications into one single skbuff, so each notification consumes at least ~211 bytes, that ~3.5 times less memory consumption. As a result, this is reducing the chances to exhaust the netlink socket receive buffer.
Rule of thumb is that each notification batch only contains netlink messages whose report flag is the same,
nfnetlink_send()requires this to do appropriately delivery to userspace, either via unicast (echo mode) or multicast (monitor mode).
Quoted 8 lines
The skbuff control buffer is used to annotate the report flag for later handling at the new coalescing routine.
The batch skbuff notification size is
NLMSG_GOODSIZE, using a larger skbuff would allow for more socket receiver buffer savings (to amortize the cost of the skbuff even more), however, going over that size might break userspace applications, so let's be conservative and stick toNLMSG_GOODSIZE.
With this patch in place on top of your other one ("netfilter:
nfnetlink: nfnetlink_unicast() reports EAGAIN instead of ENOBUFS"), firewalld (as well as nft with same input) now report:
| netlink: Error: Could not process rule: No space left on device
The JSON snippet causing the problem is indeed quite big though:
| % grep "\"\(add\|insert\)\"" fail.pp.json | wc -l | 462
Eric told me he plans to split initial ruleset creation into several chunks though, so this should at least not be a blocker for firewalld.
something seems to be fishy, though. Here's a reproducer:
| #!/bin/bash | | numrules="$1" | | nft flush ruleset | ( | echo "add table t" | echo "add chain t c" | for ((i = 0; i < $numrules; i++)); do | echo "add rule t c ip saddr 10.0.0.1 ip daddr 10.0.0.2 tcp dport 27374 mark 0x23 counter accept" | done | ) | nft -ef -
It starts failing at 13, which is not much. Interestingly, it fails outside of the container, too. And it even echoes part of the commands:
| add table ip t | add chain ip t c | add rule ip t c ip saddr 10.0.0.1 ip daddr 10.0.0.2 tcp dport 27374 meta mark 0x00000023 counter packets 0 bytes 0 accept | add rule ip t c ip saddr 10.0.0.1 ip daddr 10.0.0.2 tcp dport 27374 meta mark 0x00000023 counter packets 0 bytes 0 accept | add rule ip t c ip saddr 10.0.0.1 ip daddr 10.0.0.2 tcp dport 27374 meta mark 0x00000023 counter packets 0 bytes 0 accept | add rule ip t c ip saddr 10.0.0.1 ip daddr 10.0.0.2 tcp dport 27374 meta mark 0x00000023 counter packets 0 bytes 0 accept | add rule ip t c ip saddr 10.0.0.1 ip daddr 10.0.0.2 tcp dport 27374 meta mark 0x00000023 counter packets 0 bytes 0 accept | add rule ip t c ip saddr 10.0.0.1 ip daddr 10.0.0.2 tcp dport 27374 meta mark 0x00000023 counter packets 0 bytes 0 accept | netlink: Error: Could not process rule: No space left on device
Is this a bug in your patch?
Cheers, Phil
Phil Sutter
reviewerHi Pablo,
On Thu, Aug 27, 2020 at 07:28:42PM +0200, Pablo Neira Ayuso wrote:
[...]
net/netfilter/nf_tables_api.c static void nf_tables_commit_release(struct net *net) · line 7715
Quoted 12 lines
net/netfilter/nf_tables_api.c +12
-0+static void nft_commit_notify (struct net *net,u32 portid) +{ +struct sk_buff *batch_skb =NULL , *nskb, *skb; +unsigned char *data; +int len; + +list_for_each_entry_safe (skb, nskb, &net->nft.notify_list, list) { +if (!batch_skb) { +new_batch: + batch_skb = skb; +NFT_CB (batch_skb).report =NFT_CB (skb).report; + len =NLMSG_GOODSIZE ;
This doesn't account for the data in the first skb. After changing the line into 'len = NLMSG_GOODSIZE - skb->len;', the reported problem disappears and the patch works as expected.
Cheers, Phil
Pablo Neira Ayuso
maintainerQuoted 4 lines · Phil Sutter
Hi Pablo,
On Thu, Aug 27, 2020 at 07:28:42PM +0200, Pablo Neira Ayuso wrote:
[...]
Quoted 16 lines
net/netfilter/nf_tables_api.c static void nf_tables_commit_release(struct net *net) · line 7715
Quoted 12 lines
net/netfilter/nf_tables_api.c +12
-0+static void nft_commit_notify (struct net *net,u32 portid) +{ +struct sk_buff *batch_skb =NULL , *nskb, *skb; +unsigned char *data; +int len; + +list_for_each_entry_safe (skb, nskb, &net->nft.notify_list, list) { +if (!batch_skb) { +new_batch: + batch_skb = skb; +NFT_CB (batch_skb).report =NFT_CB (skb).report; + len =NLMSG_GOODSIZE ;This doesn't account for the data in the first skb. After changing the line into 'len =
NLMSG_GOODSIZE-skb->len;', the reported problem disappears and the patch works as expected.
Thanks for narrowing down the problem.
I'll send a v2 including this update.