Skip to contentSkip to conversation

netfilter: nf_tables: coalesce multiple notifications into one skbuff

PATCH In review Pablo Neira Ayuso 2 participants 4 messages Sep 2, 2020 – Sep 3, 2020

Subsystems (2)
lore
Share & tools
lore

Pablo Neira Ayuso

author maintainer
#0 lore raw

PATCH v3 1/1 netfilter: nf_tables: coalesce multiple notifications into one skbuff

On x86_64, each notification results in one skbuff allocation which consumes at least 768 bytes due to the skbuff overhead.

This patch coalesces several notifications into one single skbuff, so each notification consumes at least ~211 bytes, that ~3.5 times less memory consumption. As a result, this is reducing the chances to exhaust the netlink socket receive buffer.

Rule of thumb is that each notification batch only contains netlink messages whose report flag is the same, nfnetlink_send() requires this to do appropriately delivery to userspace, either via unicast (echo mode) or multicast (monitor mode).

The skbuff control buffer is used to annotate the report flag for later handling at the new coalescing routine.

The batch skbuff notification size is NLMSG_GOODSIZE, using a larger skbuff would allow for more socket receiver buffer savings (to amortize the cost of the skbuff even more), however, going over that size might break userspace applications, so let's be conservative and stick to NLMSG_GOODSIZE.

---
v3: remove unneeded assignment, per Phil Sutter.

 include/net/netns/nftables.h  |  1 +
 net/netfilter/nf_tables_api.c | 70 ++++++++++++++++++++++++++++-------
 2 files changed, 58 insertions(+), 13 deletions(-)
include/net/netns/nftables.h +1 -0
index a1a8d45adb42..6c0806bd8d1e 100644
@@ -8,6 +8,7 @@ struct netns_nftables {
 	struct list_head	tables;
 	struct list_head	commit_list;
 	struct list_head	module_list;
+	struct list_head	notify_list;
 	struct mutex		commit_mutex;
 	unsigned int		base_seq;
 	u8			gencursor;
net/netfilter/nf_tables_api.c +57 -13

142 lines not on this page · raw message

Signature · 1 line
2.20.1
reviewer Acked-by on v3 1/1
#1 lore raw
Quoted 16 lines · Pablo Neira Ayuso

On x86_64, each notification results in one skbuff allocation which consumes at least 768 bytes due to the skbuff overhead.

This patch coalesces several notifications into one single skbuff, so each notification consumes at least ~211 bytes, that ~3.5 times less memory consumption. As a result, this is reducing the chances to exhaust the netlink socket receive buffer.

Rule of thumb is that each notification batch only contains netlink messages whose report flag is the same, nfnetlink_send() requires this to do appropriately delivery to userspace, either via unicast (echo mode) or multicast (monitor mode).

The skbuff control buffer is used to annotate the report flag for later handling at the new coalescing routine.

Quoted 8 lines

The batch skbuff notification size is NLMSG_GOODSIZE, using a larger skbuff would allow for more socket receiver buffer savings (to amortize the cost of the skbuff even more), however, going over that size might break userspace applications, so let's be conservative and stick to NLMSG_GOODSIZE.

Thanks, Phil

author maintainer
#2 lore raw
Quoted 23 lines · Phil Sutter
Quoted 22 lines · Pablo Neira Ayuso

On x86_64, each notification results in one skbuff allocation which consumes at least 768 bytes due to the skbuff overhead.

This patch coalesces several notifications into one single skbuff, so each notification consumes at least ~211 bytes, that ~3.5 times less memory consumption. As a result, this is reducing the chances to exhaust the netlink socket receive buffer.

Rule of thumb is that each notification batch only contains netlink messages whose report flag is the same, nfnetlink_send() requires this to do appropriately delivery to userspace, either via unicast (echo mode) or multicast (monitor mode).

The skbuff control buffer is used to annotate the report flag for later handling at the new coalescing routine.

The batch skbuff notification size is NLMSG_GOODSIZE, using a larger skbuff would allow for more socket receiver buffer savings (to amortize the cost of the skbuff even more), however, going over that size might break userspace applications, so let's be conservative and stick to NLMSG_GOODSIZE.

Quoted 4 lines
Quoted 2 lines

Thanks, I'll place this into nf.git

BTW, I assume this mitigates the problem that Eric reported? Is it not so easy to trigger the problem after this patch?

I forgot to say, probably it would be good to monitor /proc/net/netlink to catch how busy the socket receive buffer is getting with your firewalld ruleset.

Thanks.

reviewer
#3 lore raw

Hi Pablo,

Quoted 24 lines · Pablo Neira Ayuso
Quoted 23 lines · Phil Sutter
Quoted 22 lines · Pablo Neira Ayuso

On x86_64, each notification results in one skbuff allocation which consumes at least 768 bytes due to the skbuff overhead.

This patch coalesces several notifications into one single skbuff, so each notification consumes at least ~211 bytes, that ~3.5 times less memory consumption. As a result, this is reducing the chances to exhaust the netlink socket receive buffer.

Rule of thumb is that each notification batch only contains netlink messages whose report flag is the same, nfnetlink_send() requires this to do appropriately delivery to userspace, either via unicast (echo mode) or multicast (monitor mode).

The skbuff control buffer is used to annotate the report flag for later handling at the new coalescing routine.

The batch skbuff notification size is NLMSG_GOODSIZE, using a larger skbuff would allow for more socket receiver buffer savings (to amortize the cost of the skbuff even more), however, going over that size might break userspace applications, so let's be conservative and stick to NLMSG_GOODSIZE.

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

Thanks, I'll place this into nf.git

Thanks!

Quoted 2 lines

BTW, I assume this mitigates the problem that Eric reported? Is it not so easy to trigger the problem after this patch?

Eric plans to push zones individually into the kernel from firewalld so the problem shouldn't occur anymore unless one uses a ridiculously large zone.

Quoted 3 lines

I forgot to say, probably it would be good to monitor /proc/net/netlink to catch how busy the socket receive buffer is getting with your firewalld ruleset.

The socket doesn't live long enough to monitor it this way, but I tested at which point things start failing again:

In firewalld, I see startup errors when having more than eight zones configured. This is not too much, but given that we're talking about a restrictive environment and the above change is planned anyway, it's not a real problem.

The simple reproducer script I pasted earlier fails if the number of rules exceeds 382. The error message is:

| netlink: Error: Could not process rule: Message too long

So I assume it is simply exhausting netlink send buffer space.

BTW: Outside of lxc, my script still succeeds for 100k rules and 1M rules triggers OOM killer. :)

Cheers, Phil