netfilter: "fail-open" feature support for NFQUEUE
Subsystems (2)
cover netfilter: "fail-open" feature support for NFQUEUE
Many users of an IBM security product, which uses netfilter's NFQUEUE target to process packets in userspace, face a problem of dropped connections during heavy load. Incoming packets are queued and processed by the security module, which does deep packet analysis to decide whether to accept or reject them. However during heavy load, NFQUEUE queue (default 1024 entries) fills up and connections fail after large number of packets drop during enqueue. Increasing the queue size delays the problem and also worsens latency.
This patch set implements a "failopen" support to help keep connections open during such failures. This is achieved by allowing acceptance of packets temporarily when the queue is full, which enables existing connections to be kept alive. Customers prefer this option as similar feature is available on other systems.
This patch set implements failopen for NFQUEUE (though a similar patch for IPQUEUE is also implemented but not submitted at this time). I will submit the iptables changes which controls turning failopen mode on/off later. The original requirement for sysctl option is not implemented - please let me know whether that is acceptable/preferable.
-------------------------- Results -----------------------------
Server:
# iptables -A INPUT -p tcp -m mac --mac-source 00:00:C9:C6:4F:22 -j NFQUEUE --queue-num 0 # Run interceptor program with 50ms delay between packet processing, and also sets qlen to 8
Client:
# netperf -v0 -H 10.0.4.1 -l 10 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.4.1 (10.0.4.1) port 0 AF_INET 0.22
# scp /tmp/LARGE_FILE_1 10.0.4.1:/tmp /tmp/LARGE_FILE_1 8% 8848KB 24.0KB/s 1:09:41 ETA --------------------------------------------------------- Server: # iptables -A INPUT -p tcp -m mac --mac-source 00:00:C9:C6:4F:22 -j NFQUEUE --queue-num 0 --fail-open # Run interceptor program with 50ms delay between packet processing, and also sets qlen to 8
Client:
# netperf -v0 -H 10.0.4.1 -l 10 TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.4.1 (10.0.4.1) port 0 AF_INET 4184.48
# scp /tmp/LARGE_FILE_2 10.0.4.1:/tmp /tmp/LARGE_FILE_2 100% 107MB 106.5MB/s 00:01 ---------------------------------------------------------
Please review and provide feedback/comments.
---
RFC PATCH 1/4 netfilter: Define FAILOPEN flag
Define a new verdict: FAIL_OPEN
--- include/linux/netfilter.h | 1 + 1 file changed, 1 insertion(+)
include/linux/netfilter.h +1 -0
@@ -30,6 +30,7 @@#define NF_VERDICT_MASK 0x000000ff/* extra verdict flags have mask 0x0000ff00 */ +#define NF_VERDICT_FLAG_FAIL_OPEN 0x00004000#define NF_VERDICT_FLAG_QUEUE_BYPASS 0x00008000/* queue number (NF_QUEUE) or errno (NF_DROP) */
RFC PATCH 2/4 netfilter: Add new argument to enqueue handlers
In reply to Krishna KumarAdd a new argument to enqueue handlers. Change handlers to return
Quoted 1 line
0 value to signify "failopen". This value is not passed up the
stack but intercepted by nf_queue() which calls okfn() and returns 0 to upper layers. This also means ipqueue should return 0 and not skb->len on success.
--- include/net/netfilter/nf_queue.h | 3 ++- net/ipv4/netfilter/ip_queue.c | 5 +++-- net/ipv6/netfilter/ip6_queue.c | 5 +++-- net/netfilter/nf_queue.c | 2 +- net/netfilter/nfnetlink_queue.c | 18 ++++++++++++------ 5 files changed, 21 insertions(+), 12 deletions(-)
include/net/netfilter/nf_queue.h +2 -1
@@ -20,7 +20,8 @@ struct nf_queue_entry {/* Packet queuing */ struct nf_queue_handler {int (*outfn)(struct nf_queue_entry *entry, -unsigned int queuenum); +unsigned int queuenum, +int failopen);char *name; };
net/ipv4/netfilter/ip_queue.c +3 -2
@@ -225,7 +225,8 @@ nlmsg_failure: }static int -ipq_enqueue_packet (struct nf_queue_entry *entry,unsigned int queuenum) +ipq_enqueue_packet (struct nf_queue_entry *entry,unsigned int queuenum, +int failopen) {int status = -EINVAL ;struct sk_buff *nskb;
@@ -262,7 +263,7 @@ ipq_enqueue_packet(struct nf_queue_entry__ipq_enqueue_entry (entry);spin_unlock_bh (&queue_lock); -return status; +return 0 ; err_out_free_nskb:kfree_skb (nskb);
net/ipv6/netfilter/ip6_queue.c +3 -2
@@ -225,7 +225,8 @@ nlmsg_failure: }static int -ipq_enqueue_packet (struct nf_queue_entry *entry,unsigned int queuenum) +ipq_enqueue_packet (struct nf_queue_entry *entry,unsigned int queuenum, +int failopen) {int status = -EINVAL ;struct sk_buff *nskb;
@@ -262,7 +263,7 @@ ipq_enqueue_packet(struct nf_queue_entry__ipq_enqueue_entry (entry);spin_unlock_bh (&queue_lock); -return status; +return 0 ; err_out_free_nskb:kfree_skb (nskb);
net/netfilter/nfnetlink_queue.c +12 -6
@@ -401,7 +401,8 @@ nla_put_failure: }static int -nfqnl_enqueue_packet (struct nf_queue_entry *entry,unsigned int queuenum) +nfqnl_enqueue_packet (struct nf_queue_entry *entry,unsigned int queuenum, +int failopen) {struct sk_buff *nskb;struct nfqnl_instance *queue;
@@ -432,11 +433,16 @@ nfqnl_enqueue_packet(struct nf_queue_ent goto err_out_free_nskb; }if (queue->queue_total >= queue->queue_maxlen) { - queue->queue_dropped++; -if (net_ratelimit ()) -printk (KERN_WARNING "nf_queue: full at %d entries, " -"dropping packets(s).\n" , - queue->queue_total); +if (failopen) { +/* Accept the packet temporarily skipping rules */ + err =1 ; + }else { + queue->queue_dropped++; +if (net_ratelimit ()) +printk (KERN_WARNING "nf_queue: full at %d " +"entries, dropping packets(s).\n" , + queue->queue_total); + } goto err_out_free_nskb; } entry->id = ++queue->id_sequence;
net/netfilter/nf_queue.c +1 -1
@@ -185,7 +185,7 @@ static int __nf_queue(struct sk_buff *sk#endif skb_dst_force (skb); afinfo->saveroute (skb, entry); - status = qh->outfn (entry, queuenum); + status = qh->outfn (entry, queuenum,0 );rcu_read_unlock ();
RFC PATCH 3/4 netfilter: Add support for failopen in nf_queue()
In reply to Krishna KumarPass FAILOPEN flags, add support for fail-open, add support for GSO skb. If __nf_queue() returns >0 to indicate fail-open, we call okfn() immediately and return 0 to caller.
--- net/netfilter/core.c | 4 ++ net/netfilter/nf_internals.h | 3 +- net/netfilter/nf_queue.c | 47 ++++++++++++++++++++++++--------- 3 files changed, 40 insertions(+), 14 deletions(-)
net/netfilter/core.c +3 -1
@@ -192,7 +192,9 @@ next_hook: ret = -EPERM ; }else if ((verdict &NF_VERDICT_MASK ) ==NF_QUEUE ) {int err =nf_queue (skb, elem, pf, hook, indev, outdev, okfn, - verdict >>NF_VERDICT_QBITS ); + verdict >>NF_VERDICT_QBITS , + verdict &NF_VERDICT_FLAG_FAIL_OPEN ); +if (err <0 ) {if (err == -ECANCELED ) goto next_hook;
net/netfilter/nf_internals.h +2 -1
@@ -29,7 +29,8 @@ extern int nf_queue(struct sk_buff *skb,struct net_device *indev, struct net_device *outdev, int (*okfn)(struct sk_buff *), - unsigned int queuenum); + unsigned int queuenum, + int flags);extern int __init netfilter_queue_init (void );/* nf_log.c */
net/netfilter/nf_queue.c +35 -12
91 lines not on this page · raw message
RFC PATCH 4/4 netfilter: Enable fail-open
In reply to Krishna KumarDefine xt_NFQ_info_v3 to get fail-open argument from iptables. Also enable FAIL_OPEN.
--- include/linux/netfilter/xt_NFQUEUE.h | 7 +++++++ net/netfilter/xt_NFQUEUE.c | 19 +++++++++++++++++++ 2 files changed, 26 insertions(+)
include/linux/netfilter/xt_NFQUEUE.h +7 -0
@@ -26,4 +26,11 @@ struct xt_NFQ_info_v2 { __u16 bypass; }; +struct xt_NFQ_info_v3 { +__u16 queuenum; +__u16 queues_total; +__u16 bypass; +__u16 fail_open; +}; +#endif /* _XT_NFQ_TARGET_H */
net/netfilter/xt_NFQUEUE.c +19 -0
@@ -94,6 +94,17 @@ nfqueue_tg_v2(struct sk_buff *skb, constreturn ret; } +static unsigned int +nfqueue_tg_v3 (struct sk_buff *skb,const struct xt_action_param *par) +{ +const struct xt_NFQ_info_v3 *info = par->targinfo; +unsigned int ret =nfqueue_tg_v1 (skb, par); + +if (info->fail_open) + ret |=NF_VERDICT_FLAG_FAIL_OPEN ; +return ret; +} +static int nfqueue_tg_check (const struct xt_tgchk_param *par) {const struct xt_NFQ_info_v2 *info = par->targinfo;
@@ -144,6 +155,14 @@ static struct xt_target nfqueue_tg_reg[] .targetsize = sizeof(struct xt_NFQ_info_v2), .me = THIS_MODULE, }, + { + .name = "NFQUEUE", + .revision = 3, + .family = NFPROTO_UNSPEC, + .target = nfqueue_tg_v3, + .targetsize = sizeof(struct xt_NFQ_info_v3), + .me = THIS_MODULE, + }, };static int __init nfqueue_tg_init (void )
Florian Westphal
maintainerRe: [RFC] [PATCH 4/4] netfilter: Enable fail-open
Quoted 8 lines · Krishna Kumar
Define
xt_NFQ_info_v3to get fail-open argument from iptables. Also enableFAIL_OPEN.Signed-off-by Krishna Kumar <[email protected]>diff -ruNp
org/include/linux/netfilter/xt_NFQUEUE.hnew/include/linux/netfilter/xt_NFQUEUE.h
include/linux/netfilter/xt_NFQUEUE.h struct xt_NFQ_info_v2 { · line 34
Quoted 9 lines
include/linux/netfilter/xt_NFQUEUE.h +6
-0@@ -26,4 +26,11 @@ struct xt_NFQ_info_v2 {__u16 bypass; }; +struct xt_NFQ_info_v3 { +__u16 queuenum; +__u16 queues_total; +__u16 bypass; +__u16 fail_open; +};
Minor nit:
This shouldn't be necessary; bypass is always 0 or 1.
You could just rename it to "options" or something
like that. Would also mean that you could have the v2 target revision use the same target callback as v3 (since struct layout would be the same).
Pablo Neira Ayuso
maintainerRe: [RFC] [PATCH 4/4] netfilter: Enable fail-open
Quoted 21 lines · Florian Westphal
Quoted 8 lines · Krishna Kumar
Define
xt_NFQ_info_v3to get fail-open argument from iptables. Also enableFAIL_OPEN.Signed-off-by Krishna Kumar <[email protected]>diff -ruNp
org/include/linux/netfilter/xt_NFQUEUE.hnew/include/linux/netfilter/xt_NFQUEUE.hinclude/linux/netfilter/xt_NFQUEUE.h struct xt_NFQ_info_v2 { · line 34
Quoted 9 lines
include/linux/netfilter/xt_NFQUEUE.h +6
-0@@ -26,4 +26,11 @@ struct xt_NFQ_info_v2 {__u16 bypass; }; +struct xt_NFQ_info_v3 { +__u16 queuenum; +__u16 queues_total; +__u16 bypass; +__u16 fail_open; +};Minor nit:
Quoted 5 lines
This shouldn't be necessary; bypass is always 0 or 1.
You could just rename it to "options" or something
like that. Would also mean that you could have the v2 target revision use the same target callback as v3 (since struct layout would be the same).
Yes, something like "flags" can make it.
Where flag (1 << 0) is bypass to ensure backward compatibility.
Florian Westphal
maintainerQuoted 15 lines · Krishna Kumar
Many users of an IBM security product, which uses netfilter's NFQUEUE target to process packets in userspace, face a problem of dropped connections during heavy load. Incoming packets are queued and processed by the security module, which does deep packet analysis to decide whether to accept or reject them. However during heavy load, NFQUEUE queue (default 1024 entries) fills up and connections fail after large number of packets drop during enqueue. Increasing the queue size delays the problem and also worsens latency.
This patch set implements a "failopen" support to help keep connections open during such failures. This is achieved by allowing acceptance of packets temporarily when the queue is full, which enables existing connections to be kept alive. Customers prefer this option as similar feature is available on other systems.
Quoted 5 lines
This patch set implements failopen for NFQUEUE (though a similar patch for IPQUEUE is also implemented but not submitted at this time). I will submit the iptables changes which controls turning failopen mode on/off later. The original requirement for sysctl option is not implemented - please let me know whether that is acceptable/preferable.
I think that exposing this feature as userspace-changeable via netlink (eg. by adding "NFQA_CFG_FAILOPEN" attribute) rather than via ruleset would make most sense, as only the application can know wheter it can cope with missing packets.
Pablo Neira Ayuso
maintainerQuoted 22 lines · Florian Westphal
Quoted 15 lines · Krishna Kumar
Many users of an IBM security product, which uses netfilter's NFQUEUE target to process packets in userspace, face a problem of dropped connections during heavy load. Incoming packets are queued and processed by the security module, which does deep packet analysis to decide whether to accept or reject them. However during heavy load, NFQUEUE queue (default 1024 entries) fills up and connections fail after large number of packets drop during enqueue. Increasing the queue size delays the problem and also worsens latency.
This patch set implements a "failopen" support to help keep connections open during such failures. This is achieved by allowing acceptance of packets temporarily when the queue is full, which enables existing connections to be kept alive. Customers prefer this option as similar feature is available on other systems.
Quoted 5 lines
This patch set implements failopen for NFQUEUE (though a similar patch for IPQUEUE is also implemented but not submitted at this time). I will submit the iptables changes which controls turning failopen mode on/off later. The original requirement for sysctl option is not implemented - please let me know whether that is acceptable/preferable.
Quoted 4 lines
I think that exposing this feature as userspace-changeable via netlink (eg. by adding "
NFQA_CFG_FAILOPEN" attribute) rather than via ruleset would make most sense, as only the application can know wheter it can cope with missing packets.
Agreed.
I have a patch here to add NFAQ_CFG_FLAGS attribute, we can add some new flag to specify this behaviour.
I'm using that with NFQNL_F_CONNTRACK flag to integrate nfnetlink_queue with conntrack so it sends the conntrack together with the packet via nfnetlink_queue.
Florian Westphal <[email protected]> wrote on 05/07/2012 01:40:29 PM:
Quoted 24 lines
Re: [RFC] [PATCH 0/4] netfilter: "fail-open" feature support for NFQUEUE
Quoted 15 lines · Krishna Kumar
Many users of an IBM security product, which uses netfilter's NFQUEUE target to process packets in userspace, face a problem of dropped connections during heavy load. Incoming packets are queued and processed by the security module, which does deep packet analysis to decide whether to accept or reject them. However during heavy load, NFQUEUE queue (default 1024 entries) fills up and connections fail after large number of packets drop during enqueue. Increasing the queue size delays the problem and also worsens latency.
This patch set implements a "failopen" support to help keep connections open during such failures. This is achieved by allowing acceptance of packets temporarily when the queue is full, which enables existing connections to be kept alive. Customers prefer this option as similar feature is available on other systems.
Quoted 5 lines
This patch set implements failopen for NFQUEUE (though a similar patch for IPQUEUE is also implemented but not submitted at this time). I will submit the iptables changes which controls turning failopen mode on/off later. The original requirement for sysctl option is not implemented - please let me know whether that is acceptable/preferable.
Quoted 4 lines
I think that exposing this feature as userspace-changeable via netlink (eg. by adding "
NFQA_CFG_FAILOPEN" attribute) rather than via ruleset would make most sense, as only the application can know wheter it can cope with missing packets.
Thanks for your review. With this change, is there any reason to modify xt_NFQ_info_v2's bypass field, since app can specify this option directly? I tested without this for now and it works.
Thanks,
- KK
Florian Westphal
maintainerQuoted 9 lines · Krishna Kumar2 <[email protected]> wrote:
Florian Westphal <[email protected]> wrote on 05/07/2012 01:40:29 PM:
Quoted 4 lines
I think that exposing this feature as userspace-changeable via netlink (eg. by adding "
NFQA_CFG_FAILOPEN" attribute) rather than via ruleset would make most sense, as only the application can know wheter it can cope with missing packets.Thanks for your review. With this change, is there any reason to modify xt_NFQ_info_v2's bypass field, since app can specify this option directly? I tested without this for now and it works.
I don't think so. If the netlink attribute works for you we should leave xt_NFQUEUE as-is.
Regards,
Florian