netfilter: "fail-open" feature support for NFQUEUE
Subsystems (3)
v4 cover netfilter: "fail-open" feature support for NFQUEUE
Many users of an IBM security product, which uses netfilter's NFQUEUE target to process packets in userspace, face a problem of dropped connections during heavy load. Incoming packets are queued and processed by the security module, which does deep packet analysis to decide whether to accept or reject them. However during heavy load, the queue fills up and connections fail when large number of packets get dropped.
This patch implements a "failopen" support for NFQUEUE to help keep connections open during such failures. This is achieved by allowing acceptance of packets temporarily when the queue is full, which enables existing connections to be kept open.
Failopen is enabled/disabled using a new call - nfq_set_flags(qh, mask, flags), which makes use of two new netlink attributes:
NFQA_CFG_MASK - Specifies which flags are being modified. NFQA_CFG_FLAGS - Set/reset the bits for each of those flags.
Tests done:
------------
- netperf TCP_STREAM
- 64 netperf stress testing to ensure there are no memory leaks
- icmp ping
- enabling/disabling failopen in the middle of existing connections
- checksum verification of transferred files using scp
- different flag/mask values to check that code handling NFQA_CFG_MASK works as expected
Test results: ------------- Server: ------- # iptables -A INPUT -p tcp -m mac --mac-source 00:00:C9:C6:4F:22 \ -j NFQUEUE --queue-num 0 # Run interceptor program with 50ms delay between packet processing, and also sets qlen to 16. After every read system call, this program tests and read's a config file's contents and calls nfq_set_flags(qh, mask, flags).
Client:
-------
---> failopen is disabled on server at this time # netperf -v0 -H 10.0.4.1 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.4.1 (10.0.4.1) port 0 AF_INET 0.16 ---> failopen is enabled on server at this time # netperf -v0 -H 10.0.4.1 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.4.1 (10.0.4.1) port 0 AF_INET 2292.82
---> failopen is disabled on server at this time # scp FILE 10.0.4.1:/tmp FILE 0% 2960KB 88.4KB/s 12:19:37 ETA ---> Enable failopen on server at this time FILE 21% 809MB 44.2MB/s 01:08 ETA ---> Disable failopen on server at this time FILE 23% 903MB 157.4KB/s 5:18:01 ETA ---> Enable failopen on server at this time FILE 100% 3835MB 24.1MB/s 02:39
Changes from rev3:
------------------
1. Changed flags/mask to big-endian.
2. Use nla_get_be32 instead of nla_data to access flags/masks.
3. Cleaned up some comments.
Changes from rev2:
------------------
1. Changed NFQA_CFG_FAIL_OPEN to generic NFQA_CFG_FLAGS and NFQA_CFG_MASK to support new flags/options in future.
2. Enqueue handler changed to return -ENOSPC on queue-full condition.
3. Do not invoke okfn on -ENOSPC, but process all hooks first. nf_hook_slow has code to handle failopen.
Please review.
---
PATCH v4 1/1 netfilter: Add fail-open support
Implement a new "fail-open" mode where packets are not dropped upon queue-full condition. This mode can be enabled/disabled per queue using netlink NFAQ_CFG_FLAGS & NFAQ_CFG_MASK attributes.
--- include/linux/netfilter/nfnetlink_queue.h | 5 ++ net/netfilter/core.c | 37 +++++++++++++++++++- net/netfilter/nf_queue.c | 15 ++++++-- net/netfilter/nfnetlink_queue.c | 36 +++++++++++++++++-- 4 files changed, 86 insertions(+), 7 deletions(-)
include/linux/netfilter/nfnetlink_queue.h +5 -0
@@ -84,8 +84,13 @@ enum nfqnl_attr_config {NFQA_CFG_CMD ,/* nfqnl_msg_config_cmd */ NFQA_CFG_PARAMS ,/* nfqnl_msg_config_params */ NFQA_CFG_QUEUE_MAXLEN ,/* __u32 */ +NFQA_CFG_MASK ,/* identify which flags to change */ +NFQA_CFG_FLAGS ,/* value of these flags (__be32) */ __NFQA_CFG_MAX };#define NFQA_CFG_MAX (__NFQA_CFG_MAX-1) +/* Flags for NFQA_CFG_FLAGS */ +#define NFQA_CFG_F_FAIL_OPEN (1 << 0) +#endif /* _NFNETLINK_QUEUE_H */
net/netfilter/core.c +36 -1
@@ -163,6 +163,31 @@ repeat:return NF_ACCEPT ; } +/* +* Handler was not able to enqueue the packet, and returned ENOSPC +* as "fail-open" was enabled. We temporarily accept the skb; or +* each segment for a GSO skb and free the header. +*/ +static void handle_fail_open (struct sk_buff *skb, +int (*okfn)(struct sk_buff *)) +{ +struct sk_buff *segs; +bool gso; + + segs = skb->next ? : skb; + gso = skb->next !=NULL ; + +do { +struct sk_buff *nskb = segs->next; + + segs->next =NULL ; +okfn (segs); + segs = nskb; + }while (segs); + +if (gso) +kfree_skb (skb); +}/* Returns 1 if okfn() needs to be executed by the caller, * -EPERM for NF_DROP, 0 otherwise. */
@@ -174,6 +199,7 @@ int nf_hook_slow(u_int8_t pf, unsigned i {struct list_head *elem;unsigned int verdict; +int failopen =0 ;int ret =0 ;/* We may already have this, but read-locks nest anyway */
@@ -184,7 +210,8 @@ next_hook: verdict =nf_iterate (&nf_hooks[pf][hook], skb, hook, indev, outdev, &elem, okfn, hook_thresh);if (verdict ==NF_ACCEPT || verdict ==NF_STOP ) { - ret =1 ; +if (!failopen)/* don't use the default verdict if 'failopen' */ + ret =1 ; }else if ((verdict &NF_VERDICT_MASK ) ==NF_DROP ) {kfree_skb (skb); ret =NF_DROP_GETERR (verdict);
@@ -199,10 +226,18 @@ next_hook:if (err == -ESRCH && (verdict &NF_VERDICT_FLAG_QUEUE_BYPASS )) goto next_hook; +if (err == -ENOSPC ) { + failopen =1 ; + goto next_hook; + }kfree_skb (skb); } }rcu_read_unlock (); + +if (!ret && failopen) +handle_fail_open (skb, okfn); +return ret; }EXPORT_SYMBOL (nf_hook_slow);
net/netfilter/nfnetlink_queue.c +33 -3
@@ -52,6 +52,7 @@ struct nfqnl_instance {u_int16_t queue_num;/* number of this queue */ u_int8_t copy_mode; +u_int32_t flags;/* Set using NFQA_CFG_FLAGS */ /* * Following fields are dirtied for each queued packet, * keep them in same cache line if possible.
@@ -431,9 +432,13 @@ nfqnl_enqueue_packet(struct nf_queue_ent goto err_out_free_nskb; }if (queue->queue_total >= queue->queue_maxlen) { - queue->queue_dropped++; -net_warn_ratelimited ("nf_queue: full at %d entries, dropping packets(s)\n" , - queue->queue_total); +if (queue->flags &NFQA_CFG_F_FAIL_OPEN ) { + err = -ENOSPC ; + }else { + queue->queue_dropped++; +net_warn_ratelimited ("nf_queue: full at %d entries, dropping packets(s)\n" , + queue->queue_total); + } goto err_out_free_nskb; } entry->id = ++queue->id_sequence;
@@ -858,6 +863,31 @@ nfqnl_recv_config(struct sock *ctnl, strspin_unlock_bh (&queue->lock); } +if (nfqa[NFQA_CFG_FLAGS ]) { +__be32 flags, mask; + +if (!queue) { + ret = -ENODEV ; + goto err_out_unlock; + } + +if (!nfqa[NFQA_CFG_MASK ]) { +/* A mask is needed to specify which flags are being +* changed. +*/ + ret = -EINVAL ; + goto err_out_unlock; + } + + flags =ntohl (nla_get_be32 (nfqa[NFQA_CFG_FLAGS ])); + mask =ntohl (nla_get_be32 (nfqa[NFQA_CFG_MASK ])); + +spin_lock_bh (&queue->lock); + queue->flags &= ~mask; + queue->flags |= flags & mask; +spin_unlock_bh (&queue->lock); + } + err_out_unlock:rcu_read_unlock ();return ret;
net/netfilter/nf_queue.c +12 -3
@@ -268,14 +268,23 @@ int nf_queue(struct sk_buff *skb, err =__nf_queue (segs, elem, pf, hook, indev, outdev, okfn, queuenum); } -if (err ==0 ) + +if (err ==0 ) { queued++; -else + }else if (err == -ENOSPC ) { +/* Enqueue failed due to queue-full and handler is +* in "fail-open" mode. +*/ + segs->next = nskb; + skb->next = segs; +break ; + }else {kfree_skb (segs); + } segs = nskb; }while (segs); -if (queued) { +if (queued && err != -ENOSPC ) {kfree_skb (skb);return 0 ; }
Florian Westphal
maintainerRe: [v4 PATCH 1/1] netfilter: Add fail-open support
Quoted 16 lines · Krishna Kumar
diff -ruNp
org/net/netfilter/nfnetlink_queue.cnew/net/netfilter/nfnetlink_queue.cnet/netfilter/nfnetlink_queue.c +1
-1@@ -52,6 +52,7 @@ struct nfqnl_instance {u_int16_t queue_num;/* number of this queue */ u_int8_t copy_mode; +u_int32_t flags;/* Set using NFQA_CFG_FLAGS */ /* * Following fields are dirtied for each queued packet, * keep them in same cache line if possible. @@ -431,9 +432,13 @@ nfqnl_enqueue_packet(struct nf_queue_ent goto err_out_free_nskb; }if (queue->queue_total >= queue->queue_maxlen) { - queue->queue_dropped++;
net/netfilter/nfnetlink_queue.c nfqnl_enqueue_packet(struct nf_queue_ent · line 442
Quoted 10 lines
net/netfilter/nfnetlink_queue.c +7
-2-net_warn_ratelimited ("nf_queue: full at %d entries, dropping packets(s)\n" , - queue->queue_total); +if (queue->flags &NFQA_CFG_F_FAIL_OPEN ) { + err = -ENOSPC ; + }else { + queue->queue_dropped++; +net_warn_ratelimited ("nf_queue: full at %d entries, dropping packets(s)\n" , + queue->queue_total); + } goto err_out_free_nskb;
What about this:
if (queue->queue_total >= queue->queue_maxlen) {
if (queue->flags & NFQA_CFG_F_FAIL_OPEN) {
nf_reinject(entry, NF_ACCEPT);
err = 0;
goto err_out_free_nskb;
}
queue->queue_dropped++;
net_warn_ratelimited("nf_queue: full at %d entries, dropping packets(s)\n",[..]
Do you see any problems with that?
It should do the same as the nf_hook_slow/nf_queue ENOSPC changes while avoiding modifications outside the queueing backend.
Quoted 2 lines
net/netfilter/nfnetlink_queue.c +2
-0+if (nfqa[NFQA_CFG_FLAGS ]) { +__be32 flags, mask;
[..]
net/netfilter/nfnetlink_queue.c nfqnl_recv_config(struct sock *ctnl, str · line 883
Quoted 2 lines
net/netfilter/nfnetlink_queue.c +2
-0+ flags =ntohl (nla_get_be32 (nfqa[NFQA_CFG_FLAGS ])); + mask =ntohl (nla_get_be32 (nfqa[NFQA_CFG_MASK ]));
ntohl returns __u32 type.
Thanks,
Florian
Re: [v4 PATCH 1/1] netfilter: Add fail-open support
Florian Westphal <[email protected]> wrote on 05/24/2012 03:00:43 PM:
Quoted 10 lines
What about this:
if (queue->queue_total >= queue->queue_maxlen) { if (queue->flags & NFQA_CFG_F_FAIL_OPEN) { nf_reinject(entry, NF_ACCEPT); err = 0; goto err_out_free_nskb; } queue->queue_dropped++; net_warn_ratelimited("nf_queue: full at %d entries, dropping packets
(s)\n",
Quoted 4 lines
[..]
Do you see any problems with that?
No, I think it should work fine. Do this after dropping the lock?
Quoted 2 lines
It should do the same as the nf_hook_slow/nf_queue ENOSPC changes while avoiding modifications outside the queueing backend.
Yes, that will help remove quite some code from front-end.
Quoted 9 lines
Quoted 2 lines
net/netfilter/nfnetlink_queue.c +2
-0+if (nfqa[NFQA_CFG_FLAGS ]) { +__be32 flags, mask;[..]
net/netfilter/nfnetlink_queue.c nfqnl_recv_config(struct sock *ctnl, str · line 883
Quoted 2 lines
net/netfilter/nfnetlink_queue.c +2
-0+ flags =ntohl (nla_get_be32 (nfqa[NFQA_CFG_FLAGS ])); + mask =ntohl (nla_get_be32 (nfqa[NFQA_CFG_MASK ]));ntohl returns
__u32type.
Maybe just use __u32 type for flags/mask?
Thanks,
- KK
Florian Westphal
maintainerRe: [v4 PATCH 1/1] netfilter: Add fail-open support
Quoted 8 lines · Krishna Kumar2 <[email protected]> wrote:
Quoted 8 lines
if (queue->queue_total >= queue->queue_maxlen) { if (queue->flags & NFQA_CFG_F_FAIL_OPEN) { nf_reinject(entry, NF_ACCEPT); err = 0; goto err_out_free_nskb; } queue->queue_dropped++; net_warn_ratelimited("nf_queue: full at %d entries, dropping packets
Quoted 7 lines
(s)\n",
Quoted 4 lines
[..]
Do you see any problems with that?
No, I think it should work fine. Do this after dropping the lock?
Good catch. Yes, the lock should be dropped, else we deadlock when same queue is hit in next table/chain.
Quoted 11 lines
Quoted 9 lines
Quoted 2 lines
net/netfilter/nfnetlink_queue.c +2
-0+if (nfqa[NFQA_CFG_FLAGS ]) { +__be32 flags, mask;[..]
net/netfilter/nfnetlink_queue.c nfqnl_recv_config(struct sock *ctnl, str · line 883
Quoted 2 lines
net/netfilter/nfnetlink_queue.c +2
-0+ flags =ntohl (nla_get_be32 (nfqa[NFQA_CFG_FLAGS ])); + mask =ntohl (nla_get_be32 (nfqa[NFQA_CFG_MASK ]));ntohl returns
__u32type.Maybe just use
__u32type for flags/mask?
Yes.
Thanks,
Florian
Pablo Neira Ayuso
maintainerRe: [v4 PATCH 1/1] netfilter: Add fail-open support
In reply to Krishna KumarMy main objection with this patch is that it adds more code out of the scope of the nf_queue handling to nf_hook_slow. And this is done for very specific purpose.
@David, @Eric: Krishna aims to provide a mechanism that can be enabled to accept packets if the nfqueue becomes full, ie. it changes the default behaviour under congestion from drop to accept. It seems some users prefer not to block traffic under nfqueue congestion.
The problem is the GSO handling: If we start enqueueing segments and the queue gets full, we've got a list with the remaining segments that need to be accepted. The current approach to handle this situation does not look very nice. Do you have any suggestion for this?
Thanks!
Patch is below, in case you want to have a look at it.
Quoted 194 lines · Krishna Kumar
Implement a new "fail-open" mode where packets are not dropped upon queue-full condition. This mode can be enabled/disabled per queue using netlink
NFAQ_CFG_FLAGS&NFAQ_CFG_MASKattributes.Signed-off-by Krishna Kumar <[email protected]>Signed-off-by Vivek Kashyap <[email protected]>Signed-off-by Sridhar Samudrala <[email protected]>--- include/linux/netfilter/nfnetlink_queue.h | 5 ++ net/netfilter/core.c | 37 +++++++++++++++++++- net/netfilter/nf_queue.c | 15 ++++++-- net/netfilter/nfnetlink_queue.c | 36 +++++++++++++++++-- 4 files changed, 86 insertions(+), 7 deletions(-)diff -ruNp
org/include/linux/netfilter/nfnetlink_queue.hnew/include/linux/netfilter/nfnetlink_queue.hinclude/linux/netfilter/nfnetlink_queue.h +5
-0@@ -84,8 +84,13 @@ enum nfqnl_attr_config {NFQA_CFG_CMD ,/* nfqnl_msg_config_cmd */ NFQA_CFG_PARAMS ,/* nfqnl_msg_config_params */ NFQA_CFG_QUEUE_MAXLEN ,/* __u32 */ +NFQA_CFG_MASK ,/* identify which flags to change */ +NFQA_CFG_FLAGS ,/* value of these flags (__be32) */ __NFQA_CFG_MAX };#define NFQA_CFG_MAX (__NFQA_CFG_MAX-1) +/* Flags for NFQA_CFG_FLAGS */ +#define NFQA_CFG_F_FAIL_OPEN (1 << 0) +#endif /* _NFNETLINK_QUEUE_H */ diff -ruNp
org/net/netfilter/core.cnew/net/netfilter/core.cnet/netfilter/core.c +36
-1@@ -163,6 +163,31 @@ repeat:return NF_ACCEPT ; } +/* +* Handler was not able to enqueue the packet, and returned ENOSPC +* as "fail-open" was enabled. We temporarily accept the skb; or +* each segment for a GSO skb and free the header. +*/ +static void handle_fail_open (struct sk_buff *skb, +int (*okfn)(struct sk_buff *)) +{ +struct sk_buff *segs; +bool gso; + + segs = skb->next ? : skb; + gso = skb->next !=NULL ; + +do { +struct sk_buff *nskb = segs->next; + + segs->next =NULL ; +okfn (segs); + segs = nskb; + }while (segs); + +if (gso) +kfree_skb (skb); +}/* Returns 1 if okfn() needs to be executed by the caller, * -EPERM for NF_DROP, 0 otherwise. */ @@ -174,6 +199,7 @@ int nf_hook_slow(u_int8_t pf, unsigned i {struct list_head *elem;unsigned int verdict; +int failopen =0 ;int ret =0 ;/* We may already have this, but read-locks nest anyway */ @@ -184,7 +210,8 @@ next_hook: verdict =nf_iterate (&nf_hooks[pf][hook], skb, hook, indev, outdev, &elem, okfn, hook_thresh);if (verdict ==NF_ACCEPT || verdict ==NF_STOP ) { - ret =1 ; +if (!failopen)/* don't use the default verdict if 'failopen' */ + ret =1 ; }else if ((verdict &NF_VERDICT_MASK ) ==NF_DROP ) {kfree_skb (skb); ret =NF_DROP_GETERR (verdict);@@ -199,10 +226,18 @@ next_hook:if (err == -ESRCH && (verdict &NF_VERDICT_FLAG_QUEUE_BYPASS )) goto next_hook; +if (err == -ENOSPC ) { + failopen =1 ; + goto next_hook; + }kfree_skb (skb); } }rcu_read_unlock (); + +if (!ret && failopen) +handle_fail_open (skb, okfn); +return ret; }EXPORT_SYMBOL (nf_hook_slow);diff -ruNp
org/net/netfilter/nfnetlink_queue.cnew/net/netfilter/nfnetlink_queue.cnet/netfilter/nfnetlink_queue.c +33
-3@@ -52,6 +52,7 @@ struct nfqnl_instance {u_int16_t queue_num;/* number of this queue */ u_int8_t copy_mode; +u_int32_t flags;/* Set using NFQA_CFG_FLAGS */ /* * Following fields are dirtied for each queued packet, * keep them in same cache line if possible. @@ -431,9 +432,13 @@ nfqnl_enqueue_packet(struct nf_queue_ent goto err_out_free_nskb; }if (queue->queue_total >= queue->queue_maxlen) { - queue->queue_dropped++; -net_warn_ratelimited ("nf_queue: full at %d entries, dropping packets(s)\n" , - queue->queue_total); +if (queue->flags &NFQA_CFG_F_FAIL_OPEN ) { + err = -ENOSPC ; + }else { + queue->queue_dropped++; +net_warn_ratelimited ("nf_queue: full at %d entries, dropping packets(s)\n" , + queue->queue_total); + } goto err_out_free_nskb; } entry->id = ++queue->id_sequence;@@ -858,6 +863,31 @@ nfqnl_recv_config(struct sock *ctnl, strspin_unlock_bh (&queue->lock); } +if (nfqa[NFQA_CFG_FLAGS ]) { +__be32 flags, mask; + +if (!queue) { + ret = -ENODEV ; + goto err_out_unlock; + } + +if (!nfqa[NFQA_CFG_MASK ]) { +/* A mask is needed to specify which flags are being +* changed. +*/ + ret = -EINVAL ; + goto err_out_unlock; + } + + flags =ntohl (nla_get_be32 (nfqa[NFQA_CFG_FLAGS ])); + mask =ntohl (nla_get_be32 (nfqa[NFQA_CFG_MASK ])); + +spin_lock_bh (&queue->lock); + queue->flags &= ~mask; + queue->flags |= flags & mask; +spin_unlock_bh (&queue->lock); + } + err_out_unlock:rcu_read_unlock ();return ret;diff -ruNp
org/net/netfilter/nf_queue.cnew/net/netfilter/nf_queue.cnet/netfilter/nf_queue.c +12
-3@@ -268,14 +268,23 @@ int nf_queue(struct sk_buff *skb, err =__nf_queue (segs, elem, pf, hook, indev, outdev, okfn, queuenum); } -if (err ==0 ) + +if (err ==0 ) { queued++; -else + }else if (err == -ENOSPC ) { +/* Enqueue failed due to queue-full and handler is +* in "fail-open" mode. +*/ + segs->next = nskb; + skb->next = segs; +break ; + }else {kfree_skb (segs); + } segs = nskb; }while (segs); -if (queued) { +if (queued && err != -ENOSPC ) {kfree_skb (skb);return 0 ; }
Pablo Neira Ayuso
maintainerRe: [v4 PATCH 1/1] netfilter: Add fail-open support
Quoted 8 lines · Pablo Neira Ayuso
My main objection with this patch is that it adds more code out of the scope of the
nf_queuehandling tonf_hook_slow. And this is done for very specific purpose.@David, @Eric: Krishna aims to provide a mechanism that can be enabled to accept packets if the nfqueue becomes full, ie. it changes the default behaviour under congestion from drop to accept. It seems some users prefer not to block traffic under nfqueue congestion.
Florian Westphal just proposed some possible interesting solution for this.
Re: [v4 PATCH 1/1] netfilter: Add fail-open support
Pablo Neira Ayuso <[email protected]> wrote on 05/24/2012 04:11:56 PM:
Quoted 5 lines · Pablo Neira Ayuso
Quoted 4 lines
My main objection with this patch is that it adds more code out of the scope of the
nf_queuehandling tonf_hook_slow. And this is done for very specific purpose.
Quoted 7 lines
Quoted 4 lines
@David, @Eric: Krishna aims to provide a mechanism that can be enabled to accept packets if the nfqueue becomes full, ie. it changes the default behaviour under congestion from drop to accept. It seems some users prefer not to block traffic under nfqueue congestion.
Florian Westphal just proposed some possible interesting solution for this.
Yes, and I have just finished testing this and it works fine. With this, all the changes are localized to nfnetlink_queue.c. I am doing some more tests before resubmitting this.
thanks,
- KK