nfnetlink: Busy-loop in nfnetlink_rcv_msg()
Subsystems (2)
Phil Sutter
reviewerHi,
Starting firewalld with two active zones in an lxc container provokes a situation in which nfnetlink_rcv_msg() loops indefinitely, because nc->call_rcu() (nf_tables_getgen() in this case) returns -EAGAIN every time.
I identified netlink_attachskb() as the originator for the above error code. The conditional leading to it looks like this:
| if ((atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf ||
| test_bit(NETLINK_S_CONGESTED, &nlk->state))) {
| [...]
| if (!*timeo) {*timeo is zero, so this seems to be a non-blocking socket. Both NETLINK_S_CONGESTED bit is set and sk->sk_rmem_alloc exceeds sk->sk_rcvbuf.
From user space side, firewalld seems to simply call sendto() and the call never returns.
How to solve that? I tried to find other code which does the same, but I haven't found one that does any looping. Should nfnetlink_rcv_msg() maybe just return -EAGAIN to the caller if it comes from call_rcu backend?
This happening only in an lxc container may be due to some setsockopt() calls not being allowed. In particular, setsockopt(SO_RCVBUFFORCE) returns EPERM.
The value of sk_rcvbuf is 425984, BTW. sk_rmem_alloc is 426240. In user space, I see a call to setsockopt(SO_RCVBUF) with value 4194304. No idea if this is related and how.
Cheers, Phil
Florian Westphal
maintainerQuoted 17 lines · Phil Sutter
Starting firewalld with two active zones in an lxc container provokes a situation in which
nfnetlink_rcv_msg()loops indefinitely, becausenc->call_rcu()(nf_tables_getgen()in this case) returns-EAGAINevery time.I identified
netlink_attachskb()as the originator for the above error code. The conditional leading to it looks like this:| if ((atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf || | test_bit(NETLINK_S_CONGESTED, &nlk->state))) { | [...] | if (!*timeo) {*timeo is zero, so this seems to be a non-blocking socket. Both
NETLINK_S_CONGESTEDbit is set andsk->sk_rmem_allocexceedssk->sk_rcvbuf.
Quoted 7 lines
From user space side, firewalld seems to simply call
sendto()and the call never returns.How to solve that? I tried to find other code which does the same, but I haven't found one that does any looping. Should
nfnetlink_rcv_msg()maybe just return-EAGAINto the caller if it comes fromcall_rcubackend?
Yes, I think thats the most straightforward solution.
We can of course also intercept -EAGAIN in nf_tables_api.c and translate it to -ENOBUFS like in nft_get_set_elem().
But I think a generic solution it better. The call_rcu backends should not result in changes to nf_tables internal state so they do not load modules and therefore don't need a restart.
Quoted 3 lines
This happening only in an lxc container may be due to some
setsockopt()calls not being allowed. In particular,setsockopt(SO_RCVBUFFORCE)returns EPERM.
Right.
Quoted 3 lines
The value of
sk_rcvbufis 425984, BTW.sk_rmem_allocis 426240. In user space, I see a call tosetsockopt(SO_RCVBUF)with value 4194304. No idea if this is related and how.
Does that SO_RCVBUF succeed? How large is the recvbuf?
We should try to investigate and see that nft works rather than just fix the loop and claim that fixes the bug (but just changes 'nft loops' to 'nft exits with an error').
Pablo Neira Ayuso
maintainerHi Florian,
Sorry, I overlook your reply.
Quoted 26 lines · Florian Westphal
Quoted 17 lines · Phil Sutter
Starting firewalld with two active zones in an lxc container provokes a situation in which
nfnetlink_rcv_msg()loops indefinitely, becausenc->call_rcu()(nf_tables_getgen()in this case) returns-EAGAINevery time.I identified
netlink_attachskb()as the originator for the above error code. The conditional leading to it looks like this:| if ((atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf || | test_bit(NETLINK_S_CONGESTED, &nlk->state))) { | [...] | if (!*timeo) {*timeo is zero, so this seems to be a non-blocking socket. Both
NETLINK_S_CONGESTEDbit is set andsk->sk_rmem_allocexceedssk->sk_rcvbuf.Quoted 7 lines
From user space side, firewalld seems to simply call
sendto()and the call never returns.How to solve that? I tried to find other code which does the same, but I haven't found one that does any looping. Should
nfnetlink_rcv_msg()maybe just return-EAGAINto the caller if it comes fromcall_rcubackend?
Quoted 8 lines
Yes, I think thats the most straightforward solution.
We can of course also intercept
-EAGAINinnf_tables_api.cand translate it to-ENOBUFSlike innft_get_set_elem().But I think a generic solution it better. The
call_rcubackends should not result in changes tonf_tablesinternal state so they do not load modules and therefore don't need a restart.
Handling this from the core would be better, so people don't have to remember to use the nfnetlink_unicast() that I'm proposing.
Looking at the tree, call_rcu is not enough to assume this: there are several nfnetlink subsystems calling netlink_unicast() that translate EAGAIN to ENOBUFS, from .call and .call_rcu. The way to identify this would be to decorate callbacks to know what are specifically GET commands.
So either do this or just extend my patch to use nfnetlink_send() everywhere to remove all existing translations from EAGAIN to ENOBUFS.
Florian Westphal
maintainerQuoted 9 lines · Pablo Neira Ayuso
Quoted 6 lines
We can of course also intercept
-EAGAINinnf_tables_api.cand translate it to-ENOBUFSlike innft_get_set_elem().But I think a generic solution it better. The
call_rcubackends should not result in changes tonf_tablesinternal state so they do not load modules and therefore don't need a restart.Handling this from the core would be better, so people don't have to remember to use the
nfnetlink_unicast()that I'm proposing.
Your patch looks good to me.
Quoted 5 lines
Looking at the tree,
call_rcuis not enough to assume this: there are several nfnetlink subsystems callingnetlink_unicast()that translate EAGAIN to ENOBUFS, from .call and .call_rcu. The way to identify this would be to decorate callbacks to know what are specifically GET commands.
? Which .call_rcu is NOT "passive" (does not just peek at things) and, more specifically, which .call_rcu depends on -EAGAIN requiring a tansaction replay?
Pablo Neira Ayuso
maintainerHi Phil,
Quoted 19 lines · Phil Sutter
Hi,
Starting firewalld with two active zones in an lxc container provokes a situation in which
nfnetlink_rcv_msg()loops indefinitely, becausenc->call_rcu()(nf_tables_getgen()in this case) returns-EAGAINevery time.I identified
netlink_attachskb()as the originator for the above error code. The conditional leading to it looks like this:| if ((atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf || | test_bit(NETLINK_S_CONGESTED, &nlk->state))) { | [...] | if (!*timeo) {*timeo is zero, so this seems to be a non-blocking socket. Both
NETLINK_S_CONGESTEDbit is set andsk->sk_rmem_allocexceedssk->sk_rcvbuf.
Quoted 7 lines
From user space side, firewalld seems to simply call
sendto()and the call never returns.How to solve that? I tried to find other code which does the same, but I haven't found one that does any looping. Should
nfnetlink_rcv_msg()maybe just return-EAGAINto the caller if it comes fromcall_rcubackend?
It's a bug in the netlink frontend, which erroneously reports -EAGAIN to the nfnetlink when the socket buffer is full, see:
https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/
Quoted 3 lines
This happening only in an lxc container may be due to some
setsockopt()calls not being allowed. In particular,setsockopt(SO_RCVBUFFORCE)returns EPERM.
SO_RCVBUFFORCE fails with EPERM if CAP_NET_ADMIN is not available.
Quoted 3 lines
The value of
sk_rcvbufis 425984, BTW.sk_rmem_allocis 426240. In user space, I see a call tosetsockopt(SO_RCVBUF)with value 4194304. No idea if this is related and how.
Next problem is to track why socket buffer is getting full with GET_GENID.
firewalld heavily uses NLM_F_ECHO, there I can see how it can easily reach the default socket buffer size, but with GET_GENID I'm not sure yet, probably the problem is elsewhere but it manifests in GET_GENID because it's the first thing that is done when sending a batch (maybe there are unread messages in the socket buffer, you might check /proc/net/netlink to see if the socket buffer keeps growing as firewalld moves on).
Is this easy to reproduce? Or does this happens after some time of firewalld execution?
Pablo Neira Ayuso
maintainerPATCH 1/1 netfilter: nfnetlink: nfnetlink_unicast() reports EAGAIN instead of ENOBUFS
Frontend callback reports EAGAIN to nfnetlink to retry a command, this is used to signal that module autoloading is required. Unfortunately, nlmsg_unicast() reports EAGAIN in case the receiver socket buffer gets full, so it enters a busy-loop.
This patch updates nfnetlink_unicast() to turn EAGAIN into ENOBUFS and to use nlmsg_unicast(). Remove the flags field in nfnetlink_unicast() since this is always MSG_DONTWAIT in the existing code which is exactly what nlmsg_unicast() passes to netlink_unicast() as parameter.
--- include/linux/netfilter/nfnetlink.h | 3 +- net/netfilter/nf_tables_api.c | 61 ++++++++++++++--------------- net/netfilter/nfnetlink.c | 11 ++++-- net/netfilter/nfnetlink_log.c | 3 +- net/netfilter/nfnetlink_queue.c | 2 +- 5 files changed, 40 insertions(+), 40 deletions(-)
include/linux/netfilter/nfnetlink.h +1 -2
@@ -43,8 +43,7 @@ int nfnetlink_has_listeners(struct net *net, unsigned int group);int nfnetlink_send (struct sk_buff *skb,struct net *net,u32 portid,unsigned int group,int echo,gfp_t flags);int nfnetlink_set_err (struct net *net,u32 portid,u32 group,int error); -int nfnetlink_unicast (struct sk_buff *skb,struct net *net,u32 portid, -int flags); +int nfnetlink_unicast (struct sk_buff *skb,struct net *net,u32 portid);static inline u16 nfnl_msg_type (u8 subsys,u8 msg_type) {
net/netfilter/nf_tables_api.c +29 -32
127 lines not on this page · raw message
net/netfilter/nfnetlink.c +8 -3
@@ -149,10 +149,15 @@ int nfnetlink_set_err(struct net *net, u32 portid, u32 group, int error) }EXPORT_SYMBOL_GPL (nfnetlink_set_err); -int nfnetlink_unicast (struct sk_buff *skb,struct net *net,u32 portid, -int flags) +int nfnetlink_unicast (struct sk_buff *skb,struct net *net,u32 portid) { -return netlink_unicast (net->nfnl, skb, portid, flags); +int err; + + err =nlmsg_unicast (net->nfnl, skb, portid); +if (err == -EAGAIN ) + err = -ENOBUFS ; + +return err; }EXPORT_SYMBOL_GPL (nfnetlink_unicast);
net/netfilter/nfnetlink_log.c +1 -2
@@ -356,8 +356,7 @@ __nfulnl_send(struct nfulnl_instance *inst) goto out; } } -nfnetlink_unicast (inst->skb, inst->net, inst->peer_portid, -MSG_DONTWAIT ); +nfnetlink_unicast (inst->skb, inst->net, inst->peer_portid); out: inst->qlen =0 ; inst->skb =NULL ;
net/netfilter/nfnetlink_queue.c +1 -1
@@ -681,7 +681,7 @@ __nfqnl_enqueue_packet(struct net *net, struct nfqnl_instance *queue, *packet_id_ptr =htonl (entry->id);/* nfnetlink_unicast will either free the nskb or add it to a socket */ - err =nfnetlink_unicast (nskb, net, queue->peer_portid,MSG_DONTWAIT ); + err =nfnetlink_unicast (nskb, net, queue->peer_portid);if (err <0 ) {if (queue->flags &NFQA_CFG_F_FAIL_OPEN ) { failopen =1 ;
Signature · 1 line
2.20.1
Phil Sutter
reviewerHi,
Quoted 1 line · Florian Westphal
Phil Sutter <[email protected]> wrote:
[...]
Quoted 8 lines
Quoted 3 lines
The value of
sk_rcvbufis 425984, BTW.sk_rmem_allocis 426240. In user space, I see a call tosetsockopt(SO_RCVBUF)with value 4194304. No idea if this is related and how.Does that
SO_RCVBUFsucceed? How large is the recvbuf?
We should try to investigate and see that nft works rather than just fix the loop and claim that fixes the bug (but just changes 'nft loops' to 'nft exits with an error').
Yes, that call succeeds. A following getsockopt() reveals a bufsize of 425984. Should we print a warning if the kernel accepts but restricts the value? Not sure how well warnings work when emitted by libnftables, though.
In mnl_nft_event_listener(), we check the return value of mnl_set_rcvbuffer() and print an error message:
| nft_print(octx, "# Cannot set up netlink receive socket buffer size to %u bytes, falling back to %u bytes\n",
| NFTABLES_NLEVENT_BUFSIZ, bufsiz);
When called from mnl_batch_talk(), the return code is ignored.
Quoted 30 lines · Pablo Neira Ayuso
Hi Phil,
Quoted 19 lines · Phil Sutter
Hi,
Starting firewalld with two active zones in an lxc container provokes a situation in which
nfnetlink_rcv_msg()loops indefinitely, becausenc->call_rcu()(nf_tables_getgen()in this case) returns-EAGAINevery time.I identified
netlink_attachskb()as the originator for the above error code. The conditional leading to it looks like this:| if ((atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf || | test_bit(NETLINK_S_CONGESTED, &nlk->state))) { | [...] | if (!*timeo) {*timeo is zero, so this seems to be a non-blocking socket. Both
NETLINK_S_CONGESTEDbit is set andsk->sk_rmem_allocexceedssk->sk_rcvbuf.Quoted 7 lines
From user space side, firewalld seems to simply call
sendto()and the call never returns.How to solve that? I tried to find other code which does the same, but I haven't found one that does any looping. Should
nfnetlink_rcv_msg()maybe just return-EAGAINto the caller if it comes fromcall_rcubackend?
Quoted 4 lines
It's a bug in the netlink frontend, which erroneously reports
-EAGAINto the nfnetlink when the socket buffer is full, see:https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/
Obviously this avoids the lockup. As correctly assumed by Florian, firewalld startup fails instead. (The daemon keeps running, but an error message is printed indicating that initial ruleset setup failed.)
[...]
Quoted 7 lines
Quoted 3 lines
The value of
sk_rcvbufis 425984, BTW.sk_rmem_allocis 426240. In user space, I see a call tosetsockopt(SO_RCVBUF)with value 4194304. No idea if this is related and how.Next problem is to track why socket buffer is getting full with
GET_GENID.
Quoted 7 lines
firewalld heavily uses
NLM_F_ECHO, there I can see how it can easily reach the default socket buffer size, but withGET_GENIDI'm not sure yet, probably the problem is elsewhere but it manifests inGET_GENIDbecause it's the first thing that is done when sending a batch (maybe there are unread messages in the socket buffer, you might check/proc/net/netlinkto see if the socket buffer keeps growing as firewalld moves on).
Yes, it happens only for echo mode. With your fix in place, I also see what firewalld is trying to do: The JSON input leading to the error is huge (~72k characters). I suspect that GET_GENID just happens to be the last straw. Or my debugging was faulty somehow and netlink_attachskb() really got called via a different code-path.
Quoted 2 lines
Is this easy to reproduce? Or does this happens after some time of firewalld execution?
The necessary lxd setup aside, it's pretty trivial: launch an instance of images:centos/8/amd64, install firewalld therein, add two zone files and start firewalld. It happens immediately, so two active zones already make firewalld generate enough rules to exceed the buffer space.
Quoted 5 lines · Pablo Neira Ayuso
Frontend callback reports EAGAIN to nfnetlink to retry a command, this is used to signal that module autoloading is required. Unfortunately,
nlmsg_unicast()reports EAGAIN in case the receiver socket buffer gets full, so it enters a busy-loop.
Quoted 8 lines
This patch updates
nfnetlink_unicast()to turn EAGAIN into ENOBUFS and to usenlmsg_unicast(). Remove the flags field innfnetlink_unicast()since this is alwaysMSG_DONTWAITin the existing code which is exactly whatnlmsg_unicast()passes tonetlink_unicast()as parameter.Fixes 96518518cc41 ("netfilter: add nftables")Reported-by Phil Sutter <[email protected]>Signed-off-by Pablo Neira Ayuso <[email protected]>
This indeed "fixes" the problem. Or rather, exposes the actual problem in echo-related code, namely the tendency to exhaust socket buffers.
So the problem we're facing is that while user space still waits for sendmsg() to complete, receive buffer fills up. Is it possible to buffer the data in kernel somewhere else so user space has a chance to call recvmsg()?
Thanks, Phil
Pablo Neira Ayuso
maintainerHi Phil,
On Mon, Aug 24, 2020 at 03:11:04PM +0200, Phil Sutter wrote:
[...]
Quoted 31 lines · Pablo Neira Ayuso
Quoted 30 lines
Hi Phil,
Quoted 26 lines · Phil Sutter
Hi,
Starting firewalld with two active zones in an lxc container provokes a situation in which
nfnetlink_rcv_msg()loops indefinitely, becausenc->call_rcu()(nf_tables_getgen()in this case) returns-EAGAINevery time.I identified
netlink_attachskb()as the originator for the above error code. The conditional leading to it looks like this:| if ((atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf || | test_bit(NETLINK_S_CONGESTED, &nlk->state))) { | [...] | if (!*timeo) {*timeo is zero, so this seems to be a non-blocking socket. Both
NETLINK_S_CONGESTEDbit is set andsk->sk_rmem_allocexceedssk->sk_rcvbuf.From user space side, firewalld seems to simply call
sendto()and the call never returns.How to solve that? I tried to find other code which does the same, but I haven't found one that does any looping. Should
nfnetlink_rcv_msg()maybe just return-EAGAINto the caller if it comes fromcall_rcubackend?
Quoted 8 lines
Quoted 4 lines
It's a bug in the netlink frontend, which erroneously reports
-EAGAINto the nfnetlink when the socket buffer is full, see:https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/
Obviously this avoids the lockup. As correctly assumed by Florian, firewalld startup fails instead. (The daemon keeps running, but an error message is printed indicating that initial ruleset setup failed.)
Thanks for confirming, I'll apply this patch to nf.git.
Quoted 45 lines
[...]
Quoted 7 lines
Quoted 3 lines
The value of
sk_rcvbufis 425984, BTW.sk_rmem_allocis 426240. In user space, I see a call tosetsockopt(SO_RCVBUF)with value 4194304. No idea if this is related and how.Next problem is to track why socket buffer is getting full with
GET_GENID.Quoted 7 lines
firewalld heavily uses
NLM_F_ECHO, there I can see how it can easily reach the default socket buffer size, but withGET_GENIDI'm not sure yet, probably the problem is elsewhere but it manifests inGET_GENIDbecause it's the first thing that is done when sending a batch (maybe there are unread messages in the socket buffer, you might check/proc/net/netlinkto see if the socket buffer keeps growing as firewalld moves on).Yes, it happens only for echo mode. With your fix in place, I also see what firewalld is trying to do: The JSON input leading to the error is huge (~72k characters). I suspect that
GET_GENIDjust happens to be the last straw. Or my debugging was faulty somehow andnetlink_attachskb()really got called via a different code-path.Quoted 2 lines
Is this easy to reproduce? Or does this happens after some time of firewalld execution?
The necessary lxd setup aside, it's pretty trivial: launch an instance of images:centos/8/amd64, install firewalld therein, add two zone files and start firewalld. It happens immediately, so two active zones already make firewalld generate enough rules to exceed the buffer space.
Quoted 5 lines · Pablo Neira Ayuso
Frontend callback reports EAGAIN to nfnetlink to retry a command, this is used to signal that module autoloading is required. Unfortunately,
nlmsg_unicast()reports EAGAIN in case the receiver socket buffer gets full, so it enters a busy-loop.Quoted 8 lines
This patch updates
nfnetlink_unicast()to turn EAGAIN into ENOBUFS and to usenlmsg_unicast(). Remove the flags field innfnetlink_unicast()since this is alwaysMSG_DONTWAITin the existing code which is exactly whatnlmsg_unicast()passes tonetlink_unicast()as parameter.Fixes 96518518cc41 ("netfilter: add nftables")Reported-by Phil Sutter <[email protected]>Signed-off-by Pablo Neira Ayuso <[email protected]>
Quoted 7 lines
This indeed "fixes" the problem. Or rather, exposes the actual problem in echo-related code, namely the tendency to exhaust socket buffers.
So the problem we're facing is that while user space still waits for
sendmsg()to complete, receive buffer fills up. Is it possible to buffer the data in kernel somewhere else so user space has a chance to callrecvmsg()?
There several possibilities, just a few that can be explored:
* I made a quick patch to batch several netlink messages coming as reply to the NLM_F_ECHO request into one single skbuff. If you look at the _notify() functions in the kernel, this is currently taking one single skbuff for one message which adds a bit of overhead (the skbuff truesize is used for the socket buffer accounting). I'm measuring here on x86_64 that each command takes 768 bytes. With a quick patch I'm batching several reply netlink messages into one single skbuff, now each commands takes ~120 bytes (well, size depends on how many expressions you use actually). This means this can handle ~3550 commands vs. the existing ~555 commands (assuming very small sk_rmem_alloc 426240 as the one you're reporting).
Even if this does not fix your problem (you refer to 72k chars, not sure how many commands this is), this is probably good to have anyway, this decreasing memory consumption by 85%. This will also make event reporting (monitor mode) more reliable through netlink (it's the same codepath).
Note that userspace requires no changes to support batching mode:
libmnl's mnl_cb_run() keeps iterating over the buffer that was received until all netlink messages are consumed.
The quick patch is incomplete, I just want to prove the saving in terms of memory. I'll give it another spin and submit this for review.
* Probably recover the cookie idea: firewalld specifies a cookie that identifies the rule from userspace, so there is no need for NLM_F_ECHO. Eric mentioned he would like to delete rules by cookie, this should be possible by looking up for the handle from the cookie to delete rules. The cookie is stored in NFTA_RULE_USERDATA so this is only meaningful to userspace. No kernel changes are required (this is supported for a bit of time already).
Note: The cookie could be duplicated. Since the cookie is allocated by userspace, it's up to userspace to ensure uniqueness. In case cookies are duplicated, if you delete a rule by cookie then
Rule deletion by cookie requires dumping the whole ruleset though (slow). However, I think firewalld keeps a rule cache, so it uses the rule handle to delete the rule instead (no need to dump the cache then). Therefore, the cookie is only used to fetch the rule handle.
With the rule cookie in place, firewalld can send the batch, then make a NLM_F_DUMP request after sending the batch to retrieve the handle from the cookie instead of using NLM_F_ECHO. The batch send + NLM_F_DUMP barely consumes 16KBytes per recvmsg() call. The send + dump is not atomic though.
Because NLM_F_ECHO is only used to get back the rule handle, right?
Quoted 127 lines · Pablo Neira Ayuso
Hi Phil,
On Mon, Aug 24, 2020 at 03:11:04PM +0200, Phil Sutter wrote:
[...]Quoted 31 lines · Pablo Neira Ayuso
Quoted 30 lines
Hi Phil,
Quoted 26 lines · Phil Sutter
Hi,
Starting firewalld with two active zones in an lxc container provokes a situation in which
nfnetlink_rcv_msg()loops indefinitely, becausenc->call_rcu()(nf_tables_getgen()in this case) returns-EAGAINevery time.I identified
netlink_attachskb()as the originator for the above error code. The conditional leading to it looks like this:| if ((atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf || | test_bit(NETLINK_S_CONGESTED, &nlk->state))) { | [...] | if (!*timeo) {*timeo is zero, so this seems to be a non-blocking socket. Both
NETLINK_S_CONGESTEDbit is set andsk->sk_rmem_allocexceedssk->sk_rcvbuf.From user space side, firewalld seems to simply call
sendto()and the call never returns.How to solve that? I tried to find other code which does the same, but I haven't found one that does any looping. Should
nfnetlink_rcv_msg()maybe just return-EAGAINto the caller if it comes fromcall_rcubackend?Quoted 8 lines
Quoted 4 lines
It's a bug in the netlink frontend, which erroneously reports
-EAGAINto the nfnetlink when the socket buffer is full, see:https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/
Obviously this avoids the lockup. As correctly assumed by Florian, firewalld startup fails instead. (The daemon keeps running, but an error message is printed indicating that initial ruleset setup failed.)
Thanks for confirming, I'll apply this patch to nf.git.
Quoted 45 lines
[...]
Quoted 7 lines
Quoted 3 lines
The value of
sk_rcvbufis 425984, BTW.sk_rmem_allocis 426240. In user space, I see a call tosetsockopt(SO_RCVBUF)with value 4194304. No idea if this is related and how.Next problem is to track why socket buffer is getting full with
GET_GENID.Quoted 7 lines
firewalld heavily uses
NLM_F_ECHO, there I can see how it can easily reach the default socket buffer size, but withGET_GENIDI'm not sure yet, probably the problem is elsewhere but it manifests inGET_GENIDbecause it's the first thing that is done when sending a batch (maybe there are unread messages in the socket buffer, you might check/proc/net/netlinkto see if the socket buffer keeps growing as firewalld moves on).Yes, it happens only for echo mode. With your fix in place, I also see what firewalld is trying to do: The JSON input leading to the error is huge (~72k characters). I suspect that
GET_GENIDjust happens to be the last straw. Or my debugging was faulty somehow andnetlink_attachskb()really got called via a different code-path.Quoted 2 lines
Is this easy to reproduce? Or does this happens after some time of firewalld execution?
The necessary lxd setup aside, it's pretty trivial: launch an instance of images:centos/8/amd64, install firewalld therein, add two zone files and start firewalld. It happens immediately, so two active zones already make firewalld generate enough rules to exceed the buffer space.
Quoted 5 lines · Pablo Neira Ayuso
Frontend callback reports EAGAIN to nfnetlink to retry a command, this is used to signal that module autoloading is required. Unfortunately,
nlmsg_unicast()reports EAGAIN in case the receiver socket buffer gets full, so it enters a busy-loop.Quoted 8 lines
This patch updates
nfnetlink_unicast()to turn EAGAIN into ENOBUFS and to usenlmsg_unicast(). Remove the flags field innfnetlink_unicast()since this is alwaysMSG_DONTWAITin the existing code which is exactly whatnlmsg_unicast()passes tonetlink_unicast()as parameter.Fixes 96518518cc41 ("netfilter: add nftables")Reported-by Phil Sutter <[email protected]>Signed-off-by Pablo Neira Ayuso <[email protected]>Quoted 7 lines
This indeed "fixes" the problem. Or rather, exposes the actual problem in echo-related code, namely the tendency to exhaust socket buffers.
So the problem we're facing is that while user space still waits for
sendmsg()to complete, receive buffer fills up. Is it possible to buffer the data in kernel somewhere else so user space has a chance to callrecvmsg()?There several possibilities, just a few that can be explored:
* I made a quick patch to batch several netlink messages coming as reply to the
NLM_F_ECHOrequest into one single skbuff. If you look at the_notify()functions in the kernel, this is currently taking one single skbuff for one message which adds a bit of overhead (the skbuff truesize is used for the socket buffer accounting). I'm measuring here onx86_64that each command takes 768 bytes. With a quick patch I'm batching several reply netlink messages into one single skbuff, now each commands takes ~120 bytes (well, size depends on how many expressions you use actually). This means this can handle ~3550 commands vs. the existing ~555 commands (assuming very smallsk_rmem_alloc426240 as the one you're reporting).Even if this does not fix your problem (you refer to 72k chars, not sure how many commands this is), this is probably good to have anyway, this decreasing memory consumption by 85%. This will also make event reporting (monitor mode) more reliable through netlink (it's the same codepath).
Note that userspace requires no changes to support batching mode:
libmnl'smnl_cb_run()keeps iterating over the buffer that was received until all netlink messages are consumed.The quick patch is incomplete, I just want to prove the saving in terms of memory. I'll give it another spin and submit this for review.
Quoted 7 lines
* Probably recover the cookie idea: firewalld specifies a cookie that identifies the rule from userspace, so there is no need for
NLM_F_ECHO. Eric mentioned he would like to delete rules by cookie, this should be possible by looking up for the handle from the cookie to delete rules. The cookie is stored inNFTA_RULE_USERDATAso this is only meaningful to userspace. No kernel changes are required (this is supported for a bit of time already).
Good that it doesn't require a new kernel.
Quoted 3 lines
Note: The cookie could be duplicated. Since the cookie is allocated by userspace, it's up to userspace to ensure uniqueness. In case cookies are duplicated, if you delete a rule by cookie then
I think this is advantageous. It means a user/program can group rules with a given cookie value.
Quoted 11 lines
Rule deletion by cookie requires dumping the whole ruleset though (slow). However, I think firewalld keeps a rule cache, so it uses the rule handle to delete the rule instead (no need to dump the cache then). Therefore, the cookie is only used to fetch the rule handle.
With the rule cookie in place, firewalld can send the batch, then make a
NLM_F_DUMPrequest after sending the batch to retrieve the handle from the cookie instead of usingNLM_F_ECHO. The batch send +NLM_F_DUMPbarely consumes 16KBytes perrecvmsg()call. The send + dump is not atomic though.
NLM_F_DUMP has the downside that a single rule update means fetching the whole rule set.
firewalld has a large batch when it starts up. After the initial startup rule updates are generally small. I don't think NLM_F_DUMP is good for firewalld's use case. The simpler solution would be for firewalld to step the initial large batch, i.e. break it up into smaller batches.
Quoted 1 line
Because
NLM_F_ECHOis only used to get back the rule handle, right?
Right.
Phil Sutter
reviewerHi Pablo,
On Wed, Aug 26, 2020 at 05:32:19PM +0200, Pablo Neira Ayuso wrote:
[...]
Quoted 7 lines
Quoted 5 lines
Obviously this avoids the lockup. As correctly assumed by Florian, firewalld startup fails instead. (The daemon keeps running, but an error message is printed indicating that initial ruleset setup failed.)
Thanks for confirming, I'll apply this patch to nf.git.
Thanks!
[...]
Quoted 14 lines
There several possibilities, just a few that can be explored:
* I made a quick patch to batch several netlink messages coming as reply to the
NLM_F_ECHOrequest into one single skbuff. If you look at the_notify()functions in the kernel, this is currently taking one single skbuff for one message which adds a bit of overhead (the skbuff truesize is used for the socket buffer accounting). I'm measuring here onx86_64that each command takes 768 bytes. With a quick patch I'm batching several reply netlink messages into one single skbuff, now each commands takes ~120 bytes (well, size depends on how many expressions you use actually). This means this can handle ~3550 commands vs. the existing ~555 commands (assuming very smallsk_rmem_alloc426240 as the one you're reporting).
Quoted 5 lines
Even if this does not fix your problem (you refer to 72k chars, not sure how many commands this is), this is probably good to have anyway, this decreasing memory consumption by 85%. This will also make event reporting (monitor mode) more reliable through netlink (it's the same codepath).
I didn't count the number of commands, but highly doubt it was even close to 3000.
Quoted 7 lines
Note that userspace requires no changes to support batching mode:
libmnl'smnl_cb_run()keeps iterating over the buffer that was received until all netlink messages are consumed.The quick patch is incomplete, I just want to prove the saving in terms of memory. I'll give it another spin and submit this for review.
Highly appreciated. Put me in Cc and I'll stress-test it a bit.
Quoted 18 lines
* Probably recover the cookie idea: firewalld specifies a cookie that identifies the rule from userspace, so there is no need for
NLM_F_ECHO. Eric mentioned he would like to delete rules by cookie, this should be possible by looking up for the handle from the cookie to delete rules. The cookie is stored inNFTA_RULE_USERDATAso this is only meaningful to userspace. No kernel changes are required (this is supported for a bit of time already).Note: The cookie could be duplicated. Since the cookie is allocated by userspace, it's up to userspace to ensure uniqueness. In case cookies are duplicated, if you delete a rule by cookie then
Rule deletion by cookie requires dumping the whole ruleset though (slow). However, I think firewalld keeps a rule cache, so it uses the rule handle to delete the rule instead (no need to dump the cache then). Therefore, the cookie is only used to fetch the rule handle.
Quoted 7 lines
With the rule cookie in place, firewalld can send the batch, then make a
NLM_F_DUMPrequest after sending the batch to retrieve the handle from the cookie instead of usingNLM_F_ECHO. The batch send +NLM_F_DUMPbarely consumes 16KBytes perrecvmsg()call. The send + dump is not atomic though.Because
NLM_F_ECHOis only used to get back the rule handle, right?
The discussion that led to NLM_F_ECHO was to support atomic rule handle retrieval. A user-defined "pseudo-handle" obviously can't uphold that promise and therefore won't be a full replacement.
Apart from that, JSON echo output in it's current form is useful for scripts to retrieve the handle. We've had that discussion already, I pointed out they could just do 'input = output' and know 'input["nftables"][5]["add"]["rule"]["expr"][0]["match"]["right"]' is still present and has the expected value.
I recently found out that firewalld (e.g.) doesn't even do that, but instead manually iterates over the list of commands it got back and extracts handle values. Doing that without NLM_F_ECHO but with cookies instead means to add some rules, then list ruleset and search for one's cookies.
That aside, if nftables doesn't support cookies beyond keeping them in place, why not just use a custom comment instead? That's even backwards-compatible.
Cheers, Phil
Pablo Neira Ayuso
maintainerHi Phil,
Quoted 5 lines · Phil Sutter
Hi Pablo,
On Wed, Aug 26, 2020 at 05:32:19PM +0200, Pablo Neira Ayuso wrote:
[...]Quoted 1 line
There several possibilities, just a few that can be explored:
[...]
Quoted 9 lines
Quoted 7 lines
Note that userspace requires no changes to support batching mode:
libmnl'smnl_cb_run()keeps iterating over the buffer that was received until all netlink messages are consumed.The quick patch is incomplete, I just want to prove the saving in terms of memory. I'll give it another spin and submit this for review.
Highly appreciated. Put me in Cc and I'll stress-test it a bit.
Let's give a try to this patch:
https://patchwork.ozlabs.org/project/netfilter-devel/patch/[email protected]/
Quoted 40 lines
Quoted 18 lines
* Probably recover the cookie idea: firewalld specifies a cookie that identifies the rule from userspace, so there is no need for
NLM_F_ECHO. Eric mentioned he would like to delete rules by cookie, this should be possible by looking up for the handle from the cookie to delete rules. The cookie is stored inNFTA_RULE_USERDATAso this is only meaningful to userspace. No kernel changes are required (this is supported for a bit of time already).Note: The cookie could be duplicated. Since the cookie is allocated by userspace, it's up to userspace to ensure uniqueness. In case cookies are duplicated, if you delete a rule by cookie then
Rule deletion by cookie requires dumping the whole ruleset though (slow). However, I think firewalld keeps a rule cache, so it uses the rule handle to delete the rule instead (no need to dump the cache then). Therefore, the cookie is only used to fetch the rule handle.
Quoted 7 lines
With the rule cookie in place, firewalld can send the batch, then make a
NLM_F_DUMPrequest after sending the batch to retrieve the handle from the cookie instead of usingNLM_F_ECHO. The batch send +NLM_F_DUMPbarely consumes 16KBytes perrecvmsg()call. The send + dump is not atomic though.Because
NLM_F_ECHOis only used to get back the rule handle, right?The discussion that led to
NLM_F_ECHOwas to support atomic rule handle retrieval. A user-defined "pseudo-handle" obviously can't uphold that promise and therefore won't be a full replacement.Apart from that, JSON echo output in it's current form is useful for scripts to retrieve the handle. We've had that discussion already, I pointed out they could just do 'input = output' and know 'input["nftables"][5]["add"]["rule"]["expr"][0]["match"]["right"]' is still present and has the expected value.
I recently found out that firewalld (e.g.) doesn't even do that, but instead manually iterates over the list of commands it got back and extracts handle values.
Quoted 6 lines
Doing that without
NLM_F_ECHObut with cookies instead means to add some rules, then list ruleset and search for one's cookies.That aside, if nftables doesn't support cookies beyond keeping them in place, why not just use a custom comment instead? That's even backwards-compatible.
Something else to improve netlink socket receiver usage: Userspace might also set on NLM_F_ECHO for rules only, so the kernel only consumes the netlink receive socket buffer for these reports. Although not sure how to expose this yet through the library in a nice way...
Probably it should be possible to extend netlink to filter out attributes that do not need to be reported back to userspace via NLM_F_ECHO...
The main gain is to amortize skbuff cost, ie. increasing the batching factor, but going over NLMSG_GOODSIZE is tricky. We have to update userspace (provide larger buffer) and revisit if this is possible without breaking backward compatibility.
BTW, if NLM_F_ECHO results in ENOSPC, the ruleset has been applied, it's just that the socket buffer got full, a fallback to NLM_F_DUMP is probably an option in that case? But I understand the goal is to not hit ENOSPC.
Let me know if my patch helps mitigate the problem, thanks.