netfilter: "fail-open" feature support for NFQUEUE
Subsystems (2)
PATCH v2 1/6 netfilter: Add new netlink NFQA_CFG_FAIL_OPEN
In reply to Krishna KumarDefine a new netlink message: NFQA_CFG_FAIL_OPEN
--- include/linux/netfilter/nfnetlink_queue.h | 1 + 1 file changed, 1 insertion(+)
include/linux/netfilter/nfnetlink_queue.h +1 -0
@@ -84,6 +84,7 @@ enum nfqnl_attr_config {NFQA_CFG_CMD ,/* nfqnl_msg_config_cmd */ NFQA_CFG_PARAMS ,/* nfqnl_msg_config_params */ NFQA_CFG_QUEUE_MAXLEN ,/* __u32 */ +NFQA_CFG_FAIL_OPEN ,/* __u8 */ __NFQA_CFG_MAX };#define NFQA_CFG_MAX (__NFQA_CFG_MAX-1)
PATCH v2 2/6 netfilter: Change enqueue handlers return values
In reply to Krishna KumarSince fail-open is designed to return >0 to upper layer, change ipqueue to return 0 as success, instead of skb->len.
--- net/ipv4/netfilter/ip_queue.c | 2 +- net/ipv6/netfilter/ip6_queue.c | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
net/ipv4/netfilter/ip_queue.c +1 -1
@@ -262,7 +262,7 @@ ipq_enqueue_packet(struct nf_queue_entry__ipq_enqueue_entry (entry);spin_unlock_bh (&queue_lock); -return status; +return 0 ; err_out_free_nskb:kfree_skb (nskb);
net/ipv6/netfilter/ip6_queue.c +1 -1
@@ -262,7 +262,7 @@ ipq_enqueue_packet(struct nf_queue_entry__ipq_enqueue_entry (entry);spin_unlock_bh (&queue_lock); -return status; +return 0 ; err_out_free_nskb:kfree_skb (nskb);
PATCH v2 4/6 netfilter: Add fail-open support to handler
In reply to Krishna KumarChange NFQUEUE handler to return >0 value on queue full to signify "fail-open".
--- net/netfilter/nfnetlink_queue.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)
net/netfilter/nfnetlink_queue.c +10 -5
@@ -433,11 +433,16 @@ nfqnl_enqueue_packet(struct nf_queue_ent goto err_out_free_nskb; }if (queue->queue_total >= queue->queue_maxlen) { - queue->queue_dropped++; -if (net_ratelimit ()) -printk (KERN_WARNING "nf_queue: full at %d entries, " -"dropping packets(s).\n" , - queue->queue_total); +if (queue->fail_open) { +/* Accept the packet temporarily skipping rules */ + err =1 ; + }else { + queue->queue_dropped++; +if (net_ratelimit ()) +printk (KERN_WARNING "nf_queue: full at %d " +"entries, dropping packets(s).\n" , + queue->queue_total); + } goto err_out_free_nskb; } entry->id = ++queue->id_sequence;
PATCH v2 6/6 netfilter: Enable fail-open support.
In reply to Krishna KumarProcess NFQA_CFG_FAIL_OPEN and enable fail-open support.
--- net/netfilter/nfnetlink_queue.c | 13 +++++++++++++ 1 file changed, 13 insertions(+)
net/netfilter/nfnetlink_queue.c +13 -0
@@ -867,6 +867,19 @@ nfqnl_recv_config(struct sock *ctnl, strspin_unlock_bh (&queue->lock); } +if (nfqa[NFQA_CFG_FAIL_OPEN ]) { +u_int8_t *fail_open; + +if (!queue) { + ret = -ENODEV ; + goto err_out_unlock; + } + fail_open =nla_data (nfqa[NFQA_CFG_FAIL_OPEN ]); +spin_lock_bh (&queue->lock); + queue->fail_open = *fail_open; +spin_unlock_bh (&queue->lock); + } + err_out_unlock:rcu_read_unlock ();return ret;
v2 cover netfilter: "fail-open" feature support for NFQUEUE
Many users of an IBM security product, which uses netfilter's NFQUEUE target to process packets in userspace, face a problem of dropped connections during heavy load. Incoming packets are queued and processed by the security module, which does deep packet analysis to decide whether to accept or reject them. However during heavy load, NFQUEUE queue (default 1024 entries) fills up and connections fail after large number of packets drop during enqueue. Increasing the queue size delays the problem and also worsens latency.
This patch set implements a "failopen" support for NFQUEUE to help keep connections open during such failures. This is achieved by allowing acceptance of packets temporarily when the queue is full, which enables existing connections to be kept alive. Customers have expressed preference for this option as similar feature is available on other systems.
Failopen is enabled/disabled using nfq_set_failopen() call, which uses netlink's NFQA_CFG_FAIL_OPEN attribute (will submit the library code to turn fail-open on/off later). During a stalled scp session, application can call this new API with fail-open set to continue the session, as can be seen below:
Server:
# iptables -A INPUT -p tcp -m mac --mac-source 00:00:C9:C6:4F:22 \ -j NFQUEUE --queue-num 0 # Run interceptor program with 50ms delay between packet processing, and also sets qlen to 16. After every read system call, this program tests and read's a config file's contents (0 or 1) and dynamically calls nfq_set_failopen(qh, val).
Client:
# scp LARGE_FILE 10.0.4.1:/tmp LARGE_FILE 2% 2832KB 183.7KB/s 09:38 ETA LARGE_FILE 2% 3088KB 67.3KB/s 26:14 ETA <Set failopen=1 on server at this time> LARGE_FILE 11% 12MB 1.0MB/s 01:35 ETA LARGE_FILE 70% 75MB 7.2MB/s 00:04 ETA LARGE_FILE 100% 107MB 2.5MB/s 00:42
Please review.
---
PATCH v2 3/6 netfilter: Add support for per-queue fail-open
Add support for per-queue fail-open
--- net/netfilter/nfnetlink_queue.c | 1 + 1 file changed, 1 insertion(+)
net/netfilter/nfnetlink_queue.c +1 -0
@@ -52,6 +52,7 @@ struct nfqnl_instance {u_int16_t queue_num;/* number of this queue */ u_int8_t copy_mode; +u_int8_t fail_open;/* whether fail-open enabled */ /* * Following fields are dirtied for each queued packet, * keep them in same cache line if possible.
PATCH v2 5/6 netfilter: GSO packet handling
In reply to Krishna KumarHandle >0 return value from outfn in __nf_queue(). This value is not passed up the stack but intercepted by nf_queue(), which returns 0 to upper layers.
Also add support for GSO skb. If __nf_queue() returns >0 to indicate fail-open, we call okfn() immediately.
--- net/netfilter/nf_queue.c | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-)
net/netfilter/nf_queue.c +27 -6
@@ -189,7 +189,7 @@ static int __nf_queue(struct sk_buff *skrcu_read_unlock (); -if (status <0 ) { +if (status) {nf_queue_entry_release_refs (entry); goto err; }
@@ -236,9 +236,18 @@ int nf_queue(struct sk_buff *skb,int err = -EINVAL ;unsigned int queued; -if (!skb_is_gso (skb)) -return __nf_queue (skb, elem, pf, hook, indev, outdev, okfn, +if (!skb_is_gso (skb)) { + err =__nf_queue (skb, elem, pf, hook, indev, outdev, okfn, queuenum); +if (err >0 ) { +/* Queue failed due to queue-full and handler +* returned >0 indicating fail-open - temporarily +* accept packets. +*/ + err =okfn (skb); + } +return err; + }switch (pf) {case NFPROTO_IPV4 :
@@ -268,14 +277,26 @@ int nf_queue(struct sk_buff *skb, err =__nf_queue (segs, elem, pf, hook, indev, outdev, okfn, queuenum); } -if (err ==0 ) + +if (err ==0 ) { queued++; -else + }else if (err >0 ) { +/* Queue failed due to queue-full and handler +* returned >0 indicating fail-open - accept +* this and remaining segments. +*/ +okfn (segs); + }else { +/* Queue failed due to queue-full and handler +* returned <0 - free this and remaining skb +* segments. +*/ kfree_skb (segs); + } segs = nskb; }while (segs); -if (queued) { +if (queued || err >0 ) {kfree_skb (skb);return 0 ; }
Pablo Neira Ayuso
maintainerRe: [v2 PATCH 1/6] netfilter: Add new netlink NFQA_CFG_FAIL_OPEN
In reply to Krishna KumarQuoted 13 lines · Krishna Kumar
Define a new netlink message:
NFQA_CFG_FAIL_OPENSigned-off-by Krishna Kumar <[email protected]>Signed-off-by Vivek Kashyap <[email protected]>Signed-off-by Sridhar Samudrala <[email protected]>--- include/linux/netfilter/nfnetlink_queue.h | 1 + 1 file changed, 1 insertion(+)diff -ruNp
org/include/linux/netfilter/nfnetlink_queue.hnew/include/linux/netfilter/nfnetlink_queue.h
include/linux/netfilter/nfnetlink_queue.h enum nfqnl_attr_config { · line 90
Quoted 7 lines
include/linux/netfilter/nfnetlink_queue.h +1
-0@@ -84,6 +84,7 @@ enum nfqnl_attr_config {NFQA_CFG_CMD ,/* nfqnl_msg_config_cmd */ NFQA_CFG_PARAMS ,/* nfqnl_msg_config_params */ NFQA_CFG_QUEUE_MAXLEN ,/* __u32 */ +NFQA_CFG_FAIL_OPEN ,/* __u8 */ __NFQA_CFG_MAX };#define NFQA_CFG_MAX (__NFQA_CFG_MAX-1)
The patch logic that you use is not correct. One new feature, one patch, please.
In case it gets big, then you can split changes following this logic:
patches that prepare the new feature first, then the patches that contain the feature.
In this case, you have to put everything into one single patch.
Please, send one single patch for this new feature.
Moreover, rename NFQA_CFG_FAIL_OPEN to NFAQ_CFG_FLAGS and declare:
#define NFAQ_CFG_F_FAIL_OPEN (1 << 0)
Then, check for that flag in the code to enable the fail open behaviour.
I have another feature here that will use those flags for another option.
Pablo Neira Ayuso
maintainerRe: [v2 PATCH 4/6] netfilter: Add fail-open support to handler
In reply to Krishna KumarQuoted 15 lines · Krishna Kumar
Change NFQUEUE handler to return >0 value on queue full to signify "fail-open".
Signed-off-by Krishna Kumar <[email protected]>Signed-off-by Vivek Kashyap <[email protected]>Signed-off-by Sridhar Samudrala <[email protected]>--- net/netfilter/nfnetlink_queue.c | 15 ++++++++++----- 1 file changed, 10 insertions(+), 5 deletions(-)diff -ruNp
org/net/netfilter/nfnetlink_queue.cnew/net/netfilter/nfnetlink_queue.cnet/netfilter/nfnetlink_queue.c +0
-0@@ -433,11 +433,16 @@ nfqnl_enqueue_packet(struct nf_queue_ent goto err_out_free_nskb;
net/netfilter/nfnetlink_queue.c nfqnl_enqueue_packet(struct nf_queue_ent · line 438
Quoted 10 lines
net/netfilter/nfnetlink_queue.c +3
-5}if (queue->queue_total >= queue->queue_maxlen) { - queue->queue_dropped++; -if (net_ratelimit ()) -printk (KERN_WARNING "nf_queue: full at %d entries, " -"dropping packets(s).\n" , - queue->queue_total); +if (queue->fail_open) { +/* Accept the packet temporarily skipping rules */ + err =1 ;
Please, return -ENOSPC and handle this special case in nf_queue.
Quoted 15 lines
net/netfilter/nfnetlink_queue.c +7
-1+ }else { + queue->queue_dropped++; +if (net_ratelimit ()) +printk (KERN_WARNING "nf_queue: full at %d " +"entries, dropping packets(s).\n" , + queue->queue_total); + } goto err_out_free_nskb; } entry->id = ++queue->id_sequence; --To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Pablo Neira Ayuso
maintainerRe: [v2 PATCH 5/6] netfilter: GSO packet handling
In reply to Krishna KumarQuoted 31 lines · Krishna Kumar
Handle >0 return value from outfn in
__nf_queue(). This value is not passed up the stack but intercepted bynf_queue(), which returns 0 to upper layers.Also add support for GSO skb. If
__nf_queue()returns >0 to indicate fail-open, we callokfn()immediately.Signed-off-by Krishna Kumar <[email protected]>Signed-off-by Vivek Kashyap <[email protected]>Signed-off-by Sridhar Samudrala <[email protected]>--- net/netfilter/nf_queue.c | 33 +++++++++++++++++++++++++++------ 1 file changed, 27 insertions(+), 6 deletions(-)diff -ruNp
org/net/netfilter/nf_queue.cnew/net/netfilter/nf_queue.cnet/netfilter/nf_queue.c +1
-2@@ -189,7 +189,7 @@ static int __nf_queue(struct sk_buff *skrcu_read_unlock (); -if (status <0 ) { +if (status) {nf_queue_entry_release_refs (entry); goto err; }@@ -236,9 +236,18 @@ int nf_queue(struct sk_buff *skb,int err = -EINVAL ;unsigned int queued; -if (!skb_is_gso (skb))
net/netfilter/nf_queue.c int nf_queue(struct sk_buff *skb, · line 247
Quoted 10 lines
net/netfilter/nf_queue.c +8
-1-return __nf_queue (skb, elem, pf, hook, indev, outdev, okfn, +if (!skb_is_gso (skb)) { + err =__nf_queue (skb, elem, pf, hook, indev, outdev, okfn, queuenum); +if (err >0 ) { +/* Queue failed due to queue-full and handler +* returned >0 indicating fail-open - temporarily +* accept packets. +*/ + err =okfn (skb);
You cannot invoke okfn here. If you do so, you'll skip the remaining hooks.
Return the error and handle the -ENOSPC in nf_hook_slow.
(Note that the -ENOSPC is the error value I'm proposing you to use for this special case).
Quoted 41 lines
net/netfilter/nf_queue.c +18
-4+ } +return err; + }switch (pf) {case NFPROTO_IPV4 :@@ -268,14 +277,26 @@ int nf_queue(struct sk_buff *skb, err =__nf_queue (segs, elem, pf, hook, indev, outdev, okfn, queuenum); } -if (err ==0 ) + +if (err ==0 ) { queued++; -else + }else if (err >0 ) { +/* Queue failed due to queue-full and handler +* returned >0 indicating fail-open - accept +* this and remaining segments. +*/ +okfn (segs); + }else { +/* Queue failed due to queue-full and handler +* returned <0 - free this and remaining skb +* segments. +*/ kfree_skb (segs); + } segs = nskb; }while (segs); -if (queued) { +if (queued || err >0 ) {kfree_skb (skb);return 0 ; } --To unsubscribe from this list: send the line "unsubscribe netfilter-devel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: [v2 PATCH 5/6] netfilter: GSO packet handling
Pablo Neira Ayuso <[email protected]> wrote on 05/08/2012 05:58:28 PM:
Quoted 10 lines
net/netfilter/nf_queue.c int nf_queue(struct sk_buff *skb, · line 247
Quoted 9 lines
net/netfilter/nf_queue.c +8
-0+if (!skb_is_gso (skb)) { + err =__nf_queue (skb, elem, pf, hook, indev, outdev, okfn, queuenum); +if (err >0 ) { +/* Queue failed due to queue-full and handler +* returned >0 indicating fail-open - temporarily +* accept packets. +*/ + err =okfn (skb);
Quoted 7 lines
You cannot invoke okfn here. If you do so, you'll skip the remaining hooks.
Return the error and handle the
-ENOSPCinnf_hook_slow.(Note that the
-ENOSPCis the error value I'm proposing you to use for this special case).
Are you suggesting to change nf_hook_slow to:
if (err == -ENOSPC || err == -ECANCELED) goto next_hook;
In that case, how should the GSO case be handled (code below)?
Thanks for your feedback,
- KK
Quoted 30 lines
Quoted 30 lines
net/netfilter/nf_queue.c +15
-3@@ -268,14 +277,26 @@ int nf_queue(struct sk_buff *skb, err =__nf_queue (segs, elem, pf, hook, indev, outdev, okfn, queuenum); } -if (err ==0 ) + +if (err ==0 ) { queued++; -else + }else if (err >0 ) { +/* Queue failed due to queue-full and handler +* returned >0 indicating fail-open - accept +* this and remaining segments. +*/ +okfn (segs); + }else { +/* Queue failed due to queue-full and handler +* returned <0 - free this and remaining skb +* segments. +*/ kfree_skb (segs); + } segs = nskb; }while (segs); -if (queued) { +if (queued || err >0 ) {kfree_skb (skb);return 0 ; }