Skip to contentSkip to conversation

rework of userspace expectation support

PATCH In review Pablo Neira Ayuso 3 participants 14 messages Apr 12, 2011 – Jun 13, 2011

Subsystems (2)
lore
Share & tools
lore

Pablo Neira Ayuso

author maintainer
#0 lore raw

cover rework of userspace expectation support

Hi Patrick,

The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.

The idea to fix this is to extend the iptables CT target to explicit allocate the helper extension for conntracks that are suppose to behave as master for user-space expectations.

In the case of the userspace FTP helper, people would need to add the following rule:

iptables -A PREROUTING -t raw \
	-p tcp --dport 21 -j CT --userspace-helper

Thus, we can store the list of expectations that belong to one master, and delete them in case that the master vanishes.

---

Pablo Neira Ayuso (2):
netfilter: CT: allow to set userspace helper status flag
netfilter: nf_ct_expect: rework userspace expectation support

 include/linux/netfilter/nf_conntrack_common.h |    4 ++
 include/linux/netfilter/xt_CT.h               |    3 +
 include/net/netfilter/nf_conntrack_expect.h   |    1 
 net/netfilter/nf_conntrack_expect.c           |   63 ++++++++-----------------
 net/netfilter/nf_conntrack_helper.c           |   12 +++++
 net/netfilter/nf_conntrack_netlink.c          |    5 ++
 net/netfilter/xt_CT.c                         |    8 ++-
 7 files changed, 48 insertions(+), 48 deletions(-)
author maintainer
#1 lore raw

PATCH 1/2 netfilter: CT: allow to set userspace helper status flag

This patch is new feature and bug fix at the same time.

With this patch, the CT target allows to set the conntrack's userspace helper status flags. This flag is used to tell the conntrack system to explicitly allocate the helper extension.

This helper extension is useful to link the userspace expectations with the master conntrack that is being tracked from one userspace helper.

This feature fixes a problem in the current approach of the userspace helper support. Basically, if the master conntrack that has got a userspace expectation vanishes, the expectations point to one invalid memory address. Thus, triggering an oops in the expectation deletion event path.

I decided not to add a new revision of the CT target because I only needed to add a new flag for it. I'll document in this issue in the iptables manpages. I have also changed the return value from EINVAL to EOPNOTSUPP if one flag not supported is specified. Thus, in the future adding new features that only require a new flag can be added without a new revision.

---
 include/linux/netfilter/nf_conntrack_common.h |    4 ++++
 include/linux/netfilter/xt_CT.h               |    3 ++-
 net/netfilter/nf_conntrack_helper.c           |   12 ++++++++++++
 net/netfilter/xt_CT.c                         |    8 +++++---
 4 files changed, 23 insertions(+), 4 deletions(-)
include/linux/netfilter/nf_conntrack_common.h +4 -0
index 50cdc25..6097edd 100644
@@ -80,6 +80,10 @@ enum ip_conntrack_status {
 	/* Conntrack is a fake untracked entry */
 	IPS_UNTRACKED_BIT = 12,
 	IPS_UNTRACKED = (1 << IPS_UNTRACKED_BIT),
+
+	/* Conntrack has a userspace helper. */
+	IPS_USERSPACE_HELPER_BIT = 13,
+	IPS_USERSPACE_HELPER = (1 << IPS_USERSPACE_HELPER_BIT),
 };
 
 /* Connection tracking event types */
include/linux/netfilter/xt_CT.h +2 -1
index b56e768..6390f09 100644
@@ -3,7 +3,8 @@
 
 #include <linux/types.h>
 
-#define XT_CT_NOTRACK	0x1
+#define XT_CT_NOTRACK		0x1
+#define XT_CT_USERSPACE_HELPER	0x2
 
 struct xt_ct_target_info {
 	__u16 flags;
net/netfilter/nf_conntrack_helper.c +12 -0
index 1bdfea3..4d90102 100644
@@ -121,6 +121,18 @@ int __nf_ct_try_assign_helper(struct nf_conn *ct, struct nf_conn *tmpl,
 	int ret = 0;
 
 	if (tmpl != NULL) {
+		/* we've got a userspace helper. */
+		if (tmpl->status & IPS_USERSPACE_HELPER) {
+			help = nf_ct_helper_ext_add(ct, flags);
+			if (help == NULL) {
+				ret = -ENOMEM;
+				goto out;
+			}
+			rcu_assign_pointer(help->helper, NULL);
+			__set_bit(IPS_USERSPACE_HELPER_BIT, &ct->status);
+			ret = 0;
+			goto out;
+		}
 		help = nfct_help(tmpl);
 		if (help != NULL)
 			helper = help->helper;
net/netfilter/xt_CT.c +5 -3
index 782e519..5f6279b 100644
@@ -62,8 +62,8 @@ static int xt_ct_tg_check(const struct xt_tgchk_param *par)
 	int ret = 0;
 	u8 proto;
 
-	if (info->flags & ~XT_CT_NOTRACK)
-		return -EINVAL;
+	if (info->flags & ~(XT_CT_NOTRACK | XT_CT_USERSPACE_HELPER))
+		return -EOPNOTSUPP;
 
 	if (info->flags & XT_CT_NOTRACK) {
 		ct = nf_ct_untracked_get();
@@ -92,7 +92,9 @@ static int xt_ct_tg_check(const struct xt_tgchk_param *par)
 				  GFP_KERNEL))
 		goto err3;
 
-	if (info->helper[0]) {
+	if (info->flags & XT_CT_USERSPACE_HELPER) {
+		__set_bit(IPS_USERSPACE_HELPER_BIT, &ct->status);
+	} else if (info->helper[0]) {
 		ret = -ENOENT;
 		proto = xt_ct_find_proto(par);
 		if (!proto)
author maintainer
#2 lore raw

PATCH 2/2 netfilter: nf_ct_expect: rework userspace expectation support

In reply to Pablo Neira Ayuso

This partially reworks bc01befdcf3e40979eb518085a075cbf0aacede0 which added userspace expectation support.

This patch removes the nf_ct_userspace_expect_list since now we force to use the new iptables CT target feature to add the helper extension for conntracks that have attached expectations from userspace.

A new version of the proof-of-concept code to implement userspace helpers from userspace is available at:

http://people.netfilter.org/pablo/userspace-conntrack-helpers/nf-ftp-helper-POC.tar.bz2

---
 include/net/netfilter/nf_conntrack_expect.h |    1 
 net/netfilter/nf_conntrack_expect.c         |   63 +++++++++------------------
 net/netfilter/nf_conntrack_netlink.c        |    5 ++
 3 files changed, 25 insertions(+), 44 deletions(-)
include/net/netfilter/nf_conntrack_expect.h +0 -1
index 0f8a8c5..4619caa 100644
@@ -91,7 +91,6 @@ static inline void nf_ct_unlink_expect(struct nf_conntrack_expect *exp)
 
 void nf_ct_remove_expectations(struct nf_conn *ct);
 void nf_ct_unexpect_related(struct nf_conntrack_expect *exp);
-void nf_ct_remove_userspace_expectations(void);
 
 /* Allocate space for an expectation: this is mandatory before calling
    nf_ct_expect_related.  You will have to call put afterwards. */
net/netfilter/nf_conntrack_expect.c +21 -42

134 lines not on this page · raw message

net/netfilter/nf_conntrack_netlink.c +4 -1
index 30bf8a1..7ab2227 100644
@@ -2019,6 +2019,10 @@ ctnetlink_create_expect(struct net *net, u16 zone,
 	}
 	help = nfct_help(ct);
 	if (!help) {
+		err = -EOPNOTSUPP;
+		goto out;
+	}
+	if (test_bit(IPS_USERSPACE_HELPER_BIT, &ct->status)) {
 		if (!cda[CTA_EXPECT_TIMEOUT]) {
 			err = -EINVAL;
 			goto out;
@@ -2208,7 +2212,6 @@ static void __exit ctnetlink_exit(void)
 {
 	pr_info("ctnetlink: unregistering from nfnetlink.\n");
 
-	nf_ct_remove_userspace_expectations();
 #ifdef CONFIG_NF_CONNTRACK_EVENTS
 	nf_ct_expect_unregister_notifier(&ctnl_notifier_exp);
 	nf_conntrack_unregister_notifier(&ctnl_notifier);
#3 lore raw
In reply to Pablo Neira Ayuso

Am 12.04.2011 23:59, schrieb Pablo Neira Ayuso:

Quoted 6 lines

Hi Patrick,

The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.

Just wondering, how can this happen? We take a reference for userspace expectations just as we do for kernel expectations.

Ok, I see, we are releasing it again at the end of ctnetlink_create_expect(), that seems to be the actual problem if I'm not mistaken.

Quoted 13 lines

The idea to fix this is to extend the iptables CT target to explicit allocate the helper extension for conntracks that are suppose to behave as master for user-space expectations.

In the case of the userspace FTP helper, people would need to add the following rule:

iptables -A PREROUTING -t raw \
	-p tcp --dport 21 -j CT --userspace-helper

Thus, we can store the list of expectations that belong to one master, and delete them in case that the master vanishes.

author maintainer
#4 lore raw
Quoted 8 lines · Patrick McHardy

Am 12.04.2011 23:59, schrieb Pablo Neira Ayuso:

Quoted 6 lines

Hi Patrick,

The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.

Quoted 6 lines

Just wondering, how can this happen? We take a reference for userspace expectations just as we do for kernel expectations.

Ok, I see, we are releasing it again at the end of ctnetlink_create_expect(), that seems to be the actual problem if I'm not mistaken.

Indeed, we have keep that reference, that would fix the problem.

Still, with the curent approach the userspace expectation will be valid after the master conntrack has expired.

So we can do the following: Fix this refcount issue in -stable and current, and schedule these patches for nf-next to change the behaviour.

What do you think?

#5 lore raw

Am 13.04.2011 13:47, schrieb Pablo Neira Ayuso:

Quoted 9 lines · Patrick McHardy
Quoted 8 lines

Am 12.04.2011 23:59, schrieb Pablo Neira Ayuso:

Quoted 6 lines

Hi Patrick,

The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.

Quoted 8 lines
Quoted 6 lines

Just wondering, how can this happen? We take a reference for userspace expectations just as we do for kernel expectations.

Ok, I see, we are releasing it again at the end of ctnetlink_create_expect(), that seems to be the actual problem if I'm not mistaken.

Indeed, we have keep that reference, that would fix the problem.

We definitely need to hold it anyways since destroy_conntrack() releases it again.

Quoted 7 lines

Still, with the curent approach the userspace expectation will be valid after the master conntrack has expired.

So we can do the following: Fix this refcount issue in -stable and current, and schedule these patches for nf-next to change the behaviour.

What do you think?

I don't know, what's the difference to non-userspace expectations? The same applies to them, we don't require the master to still be active for expectations.

author maintainer
#6 lore raw
Quoted 25 lines · Patrick McHardy

Am 13.04.2011 13:47, schrieb Pablo Neira Ayuso:

Quoted 9 lines · Patrick McHardy
Quoted 8 lines

Am 12.04.2011 23:59, schrieb Pablo Neira Ayuso:

Quoted 6 lines

Hi Patrick,

The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.

Quoted 8 lines
Quoted 6 lines

Just wondering, how can this happen? We take a reference for userspace expectations just as we do for kernel expectations.

Ok, I see, we are releasing it again at the end of ctnetlink_create_expect(), that seems to be the actual problem if I'm not mistaken.

Indeed, we have keep that reference, that would fix the problem.

We definitely need to hold it anyways since destroy_conntrack() releases it again.

Quoted 3 lines

Still, with the curent approach the userspace expectation will be valid after the master conntrack has expired.

Quoted 8 lines
Quoted 4 lines

So we can do the following: Fix this refcount issue in -stable and current, and schedule these patches for nf-next to change the behaviour.

What do you think?

I don't know, what's the difference to non-userspace expectations? The same applies to them, we don't require the master to still be active for expectations.

kernelspace expectations are released if the master is destroyed. userspace expectations will not.

The helper extension is used to store a list of expectations for this master conntrack, so we can release the expectations that are associated. This is not true for userspace expectations, since the master has no list with expectations.

#7 lore raw

Am 13.04.2011 14:11, schrieb Pablo Neira Ayuso:

Quoted 29 lines · Patrick McHardy
Quoted 28 lines

Am 13.04.2011 13:47, schrieb Pablo Neira Ayuso:

Quoted 9 lines · Patrick McHardy
Quoted 8 lines

Am 12.04.2011 23:59, schrieb Pablo Neira Ayuso:

Quoted 6 lines

Hi Patrick,

The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.

Quoted 8 lines
Quoted 6 lines

Just wondering, how can this happen? We take a reference for userspace expectations just as we do for kernel expectations.

Ok, I see, we are releasing it again at the end of ctnetlink_create_expect(), that seems to be the actual problem if I'm not mistaken.

Indeed, we have keep that reference, that would fix the problem.

We definitely need to hold it anyways since destroy_conntrack() releases it again.

Quoted 6 lines

Still, with the curent approach the userspace expectation will be valid after the master conntrack has expired.

So we can do the following: Fix this refcount issue in -stable and current, and schedule these patches for nf-next to change the behaviour.

Quoted 8 lines
Quoted 5 lines
Quoted 1 line

What do you think?

I don't know, what's the difference to non-userspace expectations? The same applies to them, we don't require the master to still be active for expectations.

kernelspace expectations are released if the master is destroyed. userspace expectations will not.

I see, you're talking about unfulfilled expectations. Still, we're releasing expectations in destroy_conntrack(), if we fix the refcount issue, the master conntrack will not be destroyed while userspace expectations exist.

Quoted 4 lines

The helper extension is used to store a list of expectations for this master conntrack, so we can release the expectations that are associated. This is not true for userspace expectations, since the master has no list with expectations.

This raises the question - why are we treating userspace differently in this regard?

author maintainer
#8 lore raw
Quoted 36 lines · Patrick McHardy

Am 13.04.2011 14:11, schrieb Pablo Neira Ayuso:

Quoted 35 lines · Patrick McHardy
Quoted 33 lines

Am 13.04.2011 13:47, schrieb Pablo Neira Ayuso:

Quoted 9 lines · Patrick McHardy
Quoted 8 lines

Am 12.04.2011 23:59, schrieb Pablo Neira Ayuso:

Quoted 6 lines

Hi Patrick,

The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.

Quoted 8 lines
Quoted 6 lines

Just wondering, how can this happen? We take a reference for userspace expectations just as we do for kernel expectations.

Ok, I see, we are releasing it again at the end of ctnetlink_create_expect(), that seems to be the actual problem if I'm not mistaken.

Indeed, we have keep that reference, that would fix the problem.

We definitely need to hold it anyways since destroy_conntrack() releases it again.

Quoted 7 lines

Still, with the curent approach the userspace expectation will be valid after the master conntrack has expired.

So we can do the following: Fix this refcount issue in -stable and current, and schedule these patches for nf-next to change the behaviour.

What do you think?

I don't know, what's the difference to non-userspace expectations? The same applies to them, we don't require the master to still be active for expectations.

Quoted 7 lines
Quoted 2 lines

kernelspace expectations are released if the master is destroyed. userspace expectations will not.

I see, you're talking about unfulfilled expectations. Still, we're releasing expectations in destroy_conntrack(), if we fix the refcount issue, the master conntrack will not be destroyed while userspace expectations exist.

Exactly.

Quoted 7 lines
Quoted 4 lines

The helper extension is used to store a list of expectations for this master conntrack, so we can release the expectations that are associated. This is not true for userspace expectations, since the master has no list with expectations.

This raises the question - why are we treating userspace differently in this regard?

I think we should treat them link the kernelspace expectations.

I'll send a patch to fix this issue, and then tell me if you're OK with this approach so that we can apply it to nf-next.

author maintainer
#9 lore raw
In reply to Patrick McHardy
Quoted 4 lines · Patrick McHardy

I see, you're talking about unfulfilled expectations. Still, we're releasing expectations in destroy_conntrack(), if we fix the refcount issue, the master conntrack will not be destroyed while userspace expectations exist.

I missed one point. We don't seem to increase the refcnt for unfulfilled expectations.

The following example shows a unfulfilled FTP expectation:

# conntrack -L exp
262 proto=6 src=192.168.1.128 dst=130.89.149.226 sport=0 dport=20412
conntrack v1.0.0 (conntrack-tools): 1 expectations have been shown.
# conntrack -L
tcp      6 431979 ESTABLISHED src=192.168.1.128 dst=130.89.149.226
sport=37348 dport=21 packets=9 bytes=503 src=130.89.149.226
dst=192.168.1.128 sport=21 dport=37348 packets=8 bytes=564 [ASSURED]
mark=0 delta-time=29 use=1

The use field of the master conntrack shows 1.

So, I think that these patches are still the way to fix the problem with the expectations created through ctnetlink.

#10 lore raw

Am 20.04.2011 14:10, schrieb Pablo Neira Ayuso:

Quoted 8 lines · Patrick McHardy
Quoted 4 lines

I see, you're talking about unfulfilled expectations. Still, we're releasing expectations in destroy_conntrack(), if we fix the refcount issue, the master conntrack will not be destroyed while userspace expectations exist.

I missed one point. We don't seem to increase the refcnt for unfulfilled expectations.

Yes, that's not necessary since unfulfilled expectations are cleaned up once the master is destroyed. Its just the ct->master pointer of an expected connection that takes a reference.

I thought this is what your patches try to fix?

Quoted 12 lines

The following example shows a unfulfilled FTP expectation:

# conntrack -L exp
262 proto=6 src=192.168.1.128 dst=130.89.149.226 sport=0 dport=20412
conntrack v1.0.0 (conntrack-tools): 1 expectations have been shown.
# conntrack -L
tcp      6 431979 ESTABLISHED src=192.168.1.128 dst=130.89.149.226
sport=37348 dport=21 packets=9 bytes=503 src=130.89.149.226
dst=192.168.1.128 sport=21 dport=37348 packets=8 bytes=564 [ASSURED]
mark=0 delta-time=29 use=1
Quoted 4 lines

The use field of the master conntrack shows 1.

So, I think that these patches are still the way to fix the problem with the expectations created through ctnetlink.

Just for my understanding, why not simply take the reference for the master conntrack for "fulfilled" expectations (IOW conntracks that have a master assigned), which is necessary anyways since destroy_conntrack() attempts to release that reference again, and clean up userspace expectations once the master conntrack vanishes?

This would match what we do for kernel expectations and I don't really see why we would want to keep an unfulfilled userspace expectation without the corresponding master.

author maintainer
#11 lore raw
Quoted 9 lines · Patrick McHardy

Am 20.04.2011 14:10, schrieb Pablo Neira Ayuso:

Quoted 2 lines

I missed one point. We don't seem to increase the refcnt for unfulfilled expectations.

Yes, that's not necessary since unfulfilled expectations are cleaned up once the master is destroyed. Its just the ct->master pointer of an expected connection that takes a reference.

I thought this is what your patches try to fix?

Yes, my patches add the helper extension to the conntrack by means of the CT target. Thus, in the case of userspace helpers, the master conntrack can store the list of unfulfilled expectations that it owns.

Sorry, I misunderstood your previous email, I thought you were telling that we bump master's refcnt for unfulfilled expectation, you were telling the opposite :-).

Quoted 14 lines
Quoted 14 lines

The following example shows a unfulfilled FTP expectation:

# conntrack -L exp
262 proto=6 src=192.168.1.128 dst=130.89.149.226 sport=0 dport=20412
conntrack v1.0.0 (conntrack-tools): 1 expectations have been shown.
# conntrack -L
tcp      6 431979 ESTABLISHED src=192.168.1.128 dst=130.89.149.226
sport=37348 dport=21 packets=9 bytes=503 src=130.89.149.226
dst=192.168.1.128 sport=21 dport=37348 packets=8 bytes=564 [ASSURED]
mark=0 delta-time=29 use=1

The use field of the master conntrack shows 1.

Quoted 8 lines
Quoted 2 lines

So, I think that these patches are still the way to fix the problem with the expectations created through ctnetlink.

Just for my understanding, why not simply take the reference for the master conntrack for "fulfilled" expectations (IOW conntracks that have a master assigned), which is necessary anyways since destroy_conntrack() attempts to release that reference again, and clean up userspace expectations once the master conntrack vanishes?

That's fine to me.

Quoted 3 lines

This would match what we do for kernel expectations and I don't really see why we would want to keep an unfulfilled userspace expectation without the corresponding master.

Agreed.

Then, we need to allocate the conntrack helper extension to store the list of unfulfilled expectations, which is what my patches do.

#12 lore raw

Patrick, are you going to merge these?

I don't see it in nf-next-2.6.git.

Quoted 6 lines · Pablo Neira Ayuso
Quoted 1 line · Patrick McHardy

I thought this is what your patches try to fix?

Yes, my patches add the helper extension to the conntrack by means of the CT target. Thus, in the case of userspace helpers, the master conntrack can store the list of unfulfilled expectations that it owns.

Its not clear to me why the CT master isn't added on a master-less connection at the time that a userspace expectation is set (instead of having to use the CT target in the raw table to create the master for all connections that might potentially need a userspace expectation).

However, I'll take any fix that stops the oopsing.

Thanks,
Sam