Skip to contentSkip to conversation

netfilter: "fail-open" feature support for NFQUEUE

RFC PATCH Superseded Krishna Kumar 3 participants 11 messages May 7, 2012

Subsystems (2)

v1 of 5 Superseded Latest is v5 →

lore
Share & tools
lore
#0 lore raw

cover netfilter: "fail-open" feature support for NFQUEUE

Many users of an IBM security product, which uses netfilter's NFQUEUE target to process packets in userspace, face a problem of dropped connections during heavy load. Incoming packets are queued and processed by the security module, which does deep packet analysis to decide whether to accept or reject them. However during heavy load, NFQUEUE queue (default 1024 entries) fills up and connections fail after large number of packets drop during enqueue. Increasing the queue size delays the problem and also worsens latency.

This patch set implements a "failopen" support to help keep connections open during such failures. This is achieved by allowing acceptance of packets temporarily when the queue is full, which enables existing connections to be kept alive. Customers prefer this option as similar feature is available on other systems.

This patch set implements failopen for NFQUEUE (though a similar patch for IPQUEUE is also implemented but not submitted at this time). I will submit the iptables changes which controls turning failopen mode on/off later. The original requirement for sysctl option is not implemented - please let me know whether that is acceptable/preferable.

-------------------------- Results -----------------------------
Server:

# iptables -A INPUT -p tcp -m mac --mac-source 00:00:C9:C6:4F:22 -j NFQUEUE --queue-num 0
# Run interceptor program with 50ms delay between packet processing, and
	also sets qlen to 8

Client:

# netperf -v0 -H 10.0.4.1 -l 10
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.4.1 (10.0.4.1) port 0 AF_INET
0.22 
# scp /tmp/LARGE_FILE_1 10.0.4.1:/tmp
/tmp/LARGE_FILE_1                          8% 8848KB  24.0KB/s 1:09:41 ETA
---------------------------------------------------------
		Server:
# iptables -A INPUT -p tcp -m mac --mac-source 00:00:C9:C6:4F:22 -j NFQUEUE --queue-num 0 --fail-open
# Run interceptor program with 50ms delay between packet processing, and
	also sets qlen to 8

Client:

# netperf -v0 -H 10.0.4.1 -l 10
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 10.0.4.1 (10.0.4.1) port 0 AF_INET
4184.48 
# scp /tmp/LARGE_FILE_2 10.0.4.1:/tmp
/tmp/LARGE_FILE_2                          100%  107MB 106.5MB/s   00:01    
---------------------------------------------------------

Please review and provide feedback/comments.

---

#1 lore raw

RFC PATCH 1/4 netfilter: Define FAILOPEN flag

Define a new verdict: FAIL_OPEN

---
 include/linux/netfilter.h |    1 +
 1 file changed, 1 insertion(+)
include/linux/netfilter.h +1 -0
diff -ruNp org/include/linux/netfilter.h new/include/linux/netfilter.h
@@ -30,6 +30,7 @@
 #define NF_VERDICT_MASK 0x000000ff
 
 /* extra verdict flags have mask 0x0000ff00 */
+#define NF_VERDICT_FLAG_FAIL_OPEN	0x00004000
 #define NF_VERDICT_FLAG_QUEUE_BYPASS	0x00008000
 
 /* queue number (NF_QUEUE) or errno (NF_DROP) */
#2 lore raw

RFC PATCH 2/4 netfilter: Add new argument to enqueue handlers

In reply to Krishna Kumar

Add a new argument to enqueue handlers. Change handlers to return

Quoted 1 line

0 value to signify "failopen". This value is not passed up the

stack but intercepted by nf_queue() which calls okfn() and returns 0 to upper layers. This also means ipqueue should return 0 and not skb->len on success.

---
 include/net/netfilter/nf_queue.h |    3 ++-
 net/ipv4/netfilter/ip_queue.c    |    5 +++--
 net/ipv6/netfilter/ip6_queue.c   |    5 +++--
 net/netfilter/nf_queue.c         |    2 +-
 net/netfilter/nfnetlink_queue.c  |   18 ++++++++++++------
 5 files changed, 21 insertions(+), 12 deletions(-)
include/net/netfilter/nf_queue.h +2 -1
diff -ruNp org/include/net/netfilter/nf_queue.h new/include/net/netfilter/nf_queue.h
@@ -20,7 +20,8 @@ struct nf_queue_entry {
 /* Packet queuing */
 struct nf_queue_handler {
 	int			(*outfn)(struct nf_queue_entry *entry,
-					 unsigned int queuenum);
+					 unsigned int queuenum,
+					 int failopen);
 	char			*name;
 };
 
net/ipv4/netfilter/ip_queue.c +3 -2
diff -ruNp org/net/ipv4/netfilter/ip_queue.c new/net/ipv4/netfilter/ip_queue.c
@@ -225,7 +225,8 @@ nlmsg_failure:
 }
 
 static int
-ipq_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum)
+ipq_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum,
+		   int failopen)
 {
 	int status = -EINVAL;
 	struct sk_buff *nskb;
@@ -262,7 +263,7 @@ ipq_enqueue_packet(struct nf_queue_entry
 	__ipq_enqueue_entry(entry);
 
 	spin_unlock_bh(&queue_lock);
-	return status;
+	return 0;
 
 err_out_free_nskb:
 	kfree_skb(nskb);
net/ipv6/netfilter/ip6_queue.c +3 -2
diff -ruNp org/net/ipv6/netfilter/ip6_queue.c new/net/ipv6/netfilter/ip6_queue.c
@@ -225,7 +225,8 @@ nlmsg_failure:
 }
 
 static int
-ipq_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum)
+ipq_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum,
+		   int failopen)
 {
 	int status = -EINVAL;
 	struct sk_buff *nskb;
@@ -262,7 +263,7 @@ ipq_enqueue_packet(struct nf_queue_entry
 	__ipq_enqueue_entry(entry);
 
 	spin_unlock_bh(&queue_lock);
-	return status;
+	return 0;
 
 err_out_free_nskb:
 	kfree_skb(nskb);
net/netfilter/nfnetlink_queue.c +12 -6
diff -ruNp org/net/netfilter/nfnetlink_queue.c new/net/netfilter/nfnetlink_queue.c
@@ -401,7 +401,8 @@ nla_put_failure:
 }
 
 static int
-nfqnl_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum)
+nfqnl_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum,
+		     int failopen)
 {
 	struct sk_buff *nskb;
 	struct nfqnl_instance *queue;
@@ -432,11 +433,16 @@ nfqnl_enqueue_packet(struct nf_queue_ent
 		goto err_out_free_nskb;
 	}
 	if (queue->queue_total >= queue->queue_maxlen) {
-		queue->queue_dropped++;
-		if (net_ratelimit())
-			  printk(KERN_WARNING "nf_queue: full at %d entries, "
-				 "dropping packets(s).\n",
-				 queue->queue_total);
+		if (failopen) {
+			/* Accept the packet temporarily skipping rules */
+			err = 1;
+		} else {
+			queue->queue_dropped++;
+			if (net_ratelimit())
+				  printk(KERN_WARNING "nf_queue: full at %d "
+					 "entries, dropping packets(s).\n",
+					 queue->queue_total);
+		}
 		goto err_out_free_nskb;
 	}
 	entry->id = ++queue->id_sequence;
net/netfilter/nf_queue.c +1 -1
diff -ruNp org/net/netfilter/nf_queue.c new/net/netfilter/nf_queue.c
@@ -185,7 +185,7 @@ static int __nf_queue(struct sk_buff *sk
 #endif
 	skb_dst_force(skb);
 	afinfo->saveroute(skb, entry);
-	status = qh->outfn(entry, queuenum);
+	status = qh->outfn(entry, queuenum, 0);
 
 	rcu_read_unlock();
 
#3 lore raw

RFC PATCH 3/4 netfilter: Add support for failopen in nf_queue()

In reply to Krishna Kumar

Pass FAILOPEN flags, add support for fail-open, add support for GSO skb. If __nf_queue() returns >0 to indicate fail-open, we call okfn() immediately and return 0 to caller.

---
 net/netfilter/core.c         |    4 ++
 net/netfilter/nf_internals.h |    3 +-
 net/netfilter/nf_queue.c     |   47 ++++++++++++++++++++++++---------
 3 files changed, 40 insertions(+), 14 deletions(-)
net/netfilter/core.c +3 -1
diff -ruNp org/net/netfilter/core.c new/net/netfilter/core.c
@@ -192,7 +192,9 @@ next_hook:
 			ret = -EPERM;
 	} else if ((verdict & NF_VERDICT_MASK) == NF_QUEUE) {
 		int err = nf_queue(skb, elem, pf, hook, indev, outdev, okfn,
-						verdict >> NF_VERDICT_QBITS);
+				   verdict >> NF_VERDICT_QBITS,
+				   verdict & NF_VERDICT_FLAG_FAIL_OPEN);
+
 		if (err < 0) {
 			if (err == -ECANCELED)
 				goto next_hook;
net/netfilter/nf_internals.h +2 -1
diff -ruNp org/net/netfilter/nf_internals.h new/net/netfilter/nf_internals.h
@@ -29,7 +29,8 @@ extern int nf_queue(struct sk_buff *skb,
 		    struct net_device *indev,
 		    struct net_device *outdev,
 		    int (*okfn)(struct sk_buff *),
-		    unsigned int queuenum);
+		    unsigned int queuenum,
+		    int flags);
 extern int __init netfilter_queue_init(void);
 
 /* nf_log.c */
net/netfilter/nf_queue.c +35 -12

91 lines not on this page · raw message

#4 lore raw

RFC PATCH 4/4 netfilter: Enable fail-open

In reply to Krishna Kumar

Define xt_NFQ_info_v3 to get fail-open argument from iptables. Also enable FAIL_OPEN.

---
 include/linux/netfilter/xt_NFQUEUE.h |    7 +++++++
 net/netfilter/xt_NFQUEUE.c           |   19 +++++++++++++++++++
 2 files changed, 26 insertions(+)
include/linux/netfilter/xt_NFQUEUE.h +7 -0
diff -ruNp org/include/linux/netfilter/xt_NFQUEUE.h new/include/linux/netfilter/xt_NFQUEUE.h
@@ -26,4 +26,11 @@ struct xt_NFQ_info_v2 {
 	__u16 bypass;
 };
 
+struct xt_NFQ_info_v3 {
+	__u16 queuenum;
+	__u16 queues_total;
+	__u16 bypass;
+	__u16 fail_open;
+};
+
 #endif /* _XT_NFQ_TARGET_H */
net/netfilter/xt_NFQUEUE.c +19 -0
diff -ruNp org/net/netfilter/xt_NFQUEUE.c new/net/netfilter/xt_NFQUEUE.c
@@ -94,6 +94,17 @@ nfqueue_tg_v2(struct sk_buff *skb, const
 	return ret;
 }
 
+static unsigned int
+nfqueue_tg_v3(struct sk_buff *skb, const struct xt_action_param *par)
+{
+	const struct xt_NFQ_info_v3 *info = par->targinfo;
+	unsigned int ret = nfqueue_tg_v1(skb, par);
+
+	if (info->fail_open)
+		ret |= NF_VERDICT_FLAG_FAIL_OPEN;
+	return ret;
+}
+
 static int nfqueue_tg_check(const struct xt_tgchk_param *par)
 {
 	const struct xt_NFQ_info_v2 *info = par->targinfo;
@@ -144,6 +155,14 @@ static struct xt_target nfqueue_tg_reg[]
 		.targetsize	= sizeof(struct xt_NFQ_info_v2),
 		.me		= THIS_MODULE,
 	},
+	{
+		.name		= "NFQUEUE",
+		.revision	= 3,
+		.family		= NFPROTO_UNSPEC,
+		.target		= nfqueue_tg_v3,
+		.targetsize	= sizeof(struct xt_NFQ_info_v3),
+		.me		= THIS_MODULE,
+	},
 };
 
 static int __init nfqueue_tg_init(void)
maintainer
#7 lore raw
In reply to Krishna Kumar
Quoted 15 lines · Krishna Kumar

Many users of an IBM security product, which uses netfilter's NFQUEUE target to process packets in userspace, face a problem of dropped connections during heavy load. Incoming packets are queued and processed by the security module, which does deep packet analysis to decide whether to accept or reject them. However during heavy load, NFQUEUE queue (default 1024 entries) fills up and connections fail after large number of packets drop during enqueue. Increasing the queue size delays the problem and also worsens latency.

This patch set implements a "failopen" support to help keep connections open during such failures. This is achieved by allowing acceptance of packets temporarily when the queue is full, which enables existing connections to be kept alive. Customers prefer this option as similar feature is available on other systems.

Quoted 5 lines

This patch set implements failopen for NFQUEUE (though a similar patch for IPQUEUE is also implemented but not submitted at this time). I will submit the iptables changes which controls turning failopen mode on/off later. The original requirement for sysctl option is not implemented - please let me know whether that is acceptable/preferable.

I think that exposing this feature as userspace-changeable via netlink (eg. by adding "NFQA_CFG_FAILOPEN" attribute) rather than via ruleset would make most sense, as only the application can know wheter it can cope with missing packets.

maintainer
#5 lore raw

Re: [RFC] [PATCH 4/4] netfilter: Enable fail-open

In reply to Krishna Kumar
Quoted 8 lines · Krishna Kumar

Define xt_NFQ_info_v3 to get fail-open argument from iptables. Also enable FAIL_OPEN.

diff -ruNp org/include/linux/netfilter/xt_NFQUEUE.h new/include/linux/netfilter/xt_NFQUEUE.h

include/linux/netfilter/xt_NFQUEUE.h struct xt_NFQ_info_v2 { · line 34

Quoted 9 lines
include/linux/netfilter/xt_NFQUEUE.h +6 -0
@@ -26,4 +26,11 @@ struct xt_NFQ_info_v2 {
 	__u16 bypass;
 };
 
+struct xt_NFQ_info_v3 {
+	__u16 queuenum;
+	__u16 queues_total;
+	__u16 bypass;
+	__u16 fail_open;
+};

Minor nit:

This shouldn't be necessary; bypass is always 0 or 1.
You could just rename it to "options" or something
like that. Would also mean that you could have the v2 target revision use the same target callback as v3 (since struct layout would be the same).

#6 lore raw

Re: [RFC] [PATCH 4/4] netfilter: Enable fail-open

Quoted 21 lines · Florian Westphal
Quoted 8 lines · Krishna Kumar

Define xt_NFQ_info_v3 to get fail-open argument from iptables. Also enable FAIL_OPEN.

diff -ruNp org/include/linux/netfilter/xt_NFQUEUE.h new/include/linux/netfilter/xt_NFQUEUE.h

include/linux/netfilter/xt_NFQUEUE.h struct xt_NFQ_info_v2 { · line 34

Quoted 9 lines
include/linux/netfilter/xt_NFQUEUE.h +6 -0
@@ -26,4 +26,11 @@ struct xt_NFQ_info_v2 {
 	__u16 bypass;
 };
 
+struct xt_NFQ_info_v3 {
+	__u16 queuenum;
+	__u16 queues_total;
+	__u16 bypass;
+	__u16 fail_open;
+};

Minor nit:

Quoted 5 lines

This shouldn't be necessary; bypass is always 0 or 1.
You could just rename it to "options" or something
like that. Would also mean that you could have the v2 target revision use the same target callback as v3 (since struct layout would be the same).

Yes, something like "flags" can make it.

Where flag (1 << 0) is bypass to ensure backward compatibility.

#8 lore raw
In reply to Florian Westphal
Quoted 22 lines · Florian Westphal
Quoted 15 lines · Krishna Kumar

Many users of an IBM security product, which uses netfilter's NFQUEUE target to process packets in userspace, face a problem of dropped connections during heavy load. Incoming packets are queued and processed by the security module, which does deep packet analysis to decide whether to accept or reject them. However during heavy load, NFQUEUE queue (default 1024 entries) fills up and connections fail after large number of packets drop during enqueue. Increasing the queue size delays the problem and also worsens latency.

This patch set implements a "failopen" support to help keep connections open during such failures. This is achieved by allowing acceptance of packets temporarily when the queue is full, which enables existing connections to be kept alive. Customers prefer this option as similar feature is available on other systems.

Quoted 5 lines

This patch set implements failopen for NFQUEUE (though a similar patch for IPQUEUE is also implemented but not submitted at this time). I will submit the iptables changes which controls turning failopen mode on/off later. The original requirement for sysctl option is not implemented - please let me know whether that is acceptable/preferable.

Quoted 4 lines

I think that exposing this feature as userspace-changeable via netlink (eg. by adding "NFQA_CFG_FAILOPEN" attribute) rather than via ruleset would make most sense, as only the application can know wheter it can cope with missing packets.

Agreed.

I have a patch here to add NFAQ_CFG_FLAGS attribute, we can add some new flag to specify this behaviour.

I'm using that with NFQNL_F_CONNTRACK flag to integrate nfnetlink_queue with conntrack so it sends the conntrack together with the packet via nfnetlink_queue.

#9 lore raw
In reply to Florian Westphal

Florian Westphal <[email protected]> wrote on 05/07/2012 01:40:29 PM:

Quoted 24 lines

Re: [RFC] [PATCH 0/4] netfilter: "fail-open" feature support for NFQUEUE

Quoted 15 lines · Krishna Kumar

Many users of an IBM security product, which uses netfilter's NFQUEUE target to process packets in userspace, face a problem of dropped connections during heavy load. Incoming packets are queued and processed by the security module, which does deep packet analysis to decide whether to accept or reject them. However during heavy load, NFQUEUE queue (default 1024 entries) fills up and connections fail after large number of packets drop during enqueue. Increasing the queue size delays the problem and also worsens latency.

This patch set implements a "failopen" support to help keep connections open during such failures. This is achieved by allowing acceptance of packets temporarily when the queue is full, which enables existing connections to be kept alive. Customers prefer this option as similar feature is available on other systems.

Quoted 5 lines

This patch set implements failopen for NFQUEUE (though a similar patch for IPQUEUE is also implemented but not submitted at this time). I will submit the iptables changes which controls turning failopen mode on/off later. The original requirement for sysctl option is not implemented - please let me know whether that is acceptable/preferable.

Quoted 4 lines

I think that exposing this feature as userspace-changeable via netlink (eg. by adding "NFQA_CFG_FAILOPEN" attribute) rather than via ruleset would make most sense, as only the application can know wheter it can cope with missing packets.

Thanks for your review. With this change, is there any reason to modify xt_NFQ_info_v2's bypass field, since app can specify this option directly? I tested without this for now and it works.

Thanks,
- KK

maintainer
#10 lore raw
Quoted 9 lines · Krishna Kumar2 <[email protected]> wrote:

Florian Westphal <[email protected]> wrote on 05/07/2012 01:40:29 PM:

Quoted 4 lines

I think that exposing this feature as userspace-changeable via netlink (eg. by adding "NFQA_CFG_FAILOPEN" attribute) rather than via ruleset would make most sense, as only the application can know wheter it can cope with missing packets.

Thanks for your review. With this change, is there any reason to modify xt_NFQ_info_v2's bypass field, since app can specify this option directly? I tested without this for now and it works.

I don't think so. If the netlink attribute works for you we should leave xt_NFQUEUE as-is.

Regards,
Florian