rework of userspace expectation support
Subsystems (2)
Pablo Neira Ayuso
maintainercover rework of userspace expectation support
Hi Patrick,
The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.
The idea to fix this is to extend the iptables CT target to explicit allocate the helper extension for conntracks that are suppose to behave as master for user-space expectations.
In the case of the userspace FTP helper, people would need to add the following rule:
iptables -A PREROUTING -t raw \ -p tcp --dport 21 -j CT --userspace-helper
Thus, we can store the list of expectations that belong to one master, and delete them in case that the master vanishes.
---
Pablo Neira Ayuso (2):
netfilter: CT: allow to set userspace helper status flag
netfilter: nf_ct_expect: rework userspace expectation support
include/linux/netfilter/nf_conntrack_common.h | 4 ++ include/linux/netfilter/xt_CT.h | 3 + include/net/netfilter/nf_conntrack_expect.h | 1 net/netfilter/nf_conntrack_expect.c | 63 ++++++++----------------- net/netfilter/nf_conntrack_helper.c | 12 +++++ net/netfilter/nf_conntrack_netlink.c | 5 ++ net/netfilter/xt_CT.c | 8 ++- 7 files changed, 48 insertions(+), 48 deletions(-)
Pablo Neira Ayuso
maintainerPATCH 1/2 netfilter: CT: allow to set userspace helper status flag
This patch is new feature and bug fix at the same time.
With this patch, the CT target allows to set the conntrack's userspace helper status flags. This flag is used to tell the conntrack system to explicitly allocate the helper extension.
This helper extension is useful to link the userspace expectations with the master conntrack that is being tracked from one userspace helper.
This feature fixes a problem in the current approach of the userspace helper support. Basically, if the master conntrack that has got a userspace expectation vanishes, the expectations point to one invalid memory address. Thus, triggering an oops in the expectation deletion event path.
I decided not to add a new revision of the CT target because I only needed to add a new flag for it. I'll document in this issue in the iptables manpages. I have also changed the return value from EINVAL to EOPNOTSUPP if one flag not supported is specified. Thus, in the future adding new features that only require a new flag can be added without a new revision.
--- include/linux/netfilter/nf_conntrack_common.h | 4 ++++ include/linux/netfilter/xt_CT.h | 3 ++- net/netfilter/nf_conntrack_helper.c | 12 ++++++++++++ net/netfilter/xt_CT.c | 8 +++++--- 4 files changed, 23 insertions(+), 4 deletions(-)
include/linux/netfilter/nf_conntrack_common.h +4 -0
@@ -80,6 +80,10 @@ enum ip_conntrack_status {/* Conntrack is a fake untracked entry */ IPS_UNTRACKED_BIT =12 ,IPS_UNTRACKED = (1 <<IPS_UNTRACKED_BIT ), + +/* Conntrack has a userspace helper. */ +IPS_USERSPACE_HELPER_BIT =13 , + IPS_USERSPACE_HELPER = (1 << IPS_USERSPACE_HELPER_BIT), };/* Connection tracking event types */
include/linux/netfilter/xt_CT.h +2 -1
@@ -3,7 +3,8 @@#include <linux/types.h> -#define XT_CT_NOTRACK 0x1 +#define XT_CT_NOTRACK 0x1 +#define XT_CT_USERSPACE_HELPER 0x2struct xt_ct_target_info {__u16 flags;
net/netfilter/nf_conntrack_helper.c +12 -0
@@ -121,6 +121,18 @@ int __nf_ct_try_assign_helper(struct nf_conn *ct, struct nf_conn *tmpl,int ret =0 ;if (tmpl !=NULL ) { +/* we've got a userspace helper. */ +if (tmpl->status &IPS_USERSPACE_HELPER ) { + help =nf_ct_helper_ext_add (ct, flags); +if (help ==NULL ) { + ret = -ENOMEM ; + goto out; + } +rcu_assign_pointer (help->helper,NULL ); +__set_bit (IPS_USERSPACE_HELPER_BIT , &ct->status); + ret =0 ; + goto out; + } help =nfct_help (tmpl);if (help !=NULL ) helper = help->helper;
net/netfilter/xt_CT.c +5 -3
@@ -62,8 +62,8 @@ static int xt_ct_tg_check(const struct xt_tgchk_param *par)int ret =0 ;u8 proto; -if (info->flags & ~XT_CT_NOTRACK ) -return -EINVAL ; +if (info->flags & ~(XT_CT_NOTRACK |XT_CT_USERSPACE_HELPER )) +return -EOPNOTSUPP ;if (info->flags &XT_CT_NOTRACK ) { ct =nf_ct_untracked_get ();
@@ -92,7 +92,9 @@ static int xt_ct_tg_check(const struct xt_tgchk_param *par) GFP_KERNEL)) goto err3; -if (info->helper[0 ]) { +if (info->flags &XT_CT_USERSPACE_HELPER ) { +__set_bit (IPS_USERSPACE_HELPER_BIT , &ct->status); + }else if (info->helper[0 ]) { ret = -ENOENT ; proto =xt_ct_find_proto (par);if (!proto)
Pablo Neira Ayuso
maintainerPATCH 2/2 netfilter: nf_ct_expect: rework userspace expectation support
In reply to Pablo Neira AyusoThis partially reworks bc01befdcf3e40979eb518085a075cbf0aacede0 which added userspace expectation support.
This patch removes the nf_ct_userspace_expect_list since now we force to use the new iptables CT target feature to add the helper extension for conntracks that have attached expectations from userspace.
A new version of the proof-of-concept code to implement userspace helpers from userspace is available at:
http://people.netfilter.org/pablo/userspace-conntrack-helpers/nf-ftp-helper-POC.tar.bz2
--- include/net/netfilter/nf_conntrack_expect.h | 1 net/netfilter/nf_conntrack_expect.c | 63 +++++++++------------------ net/netfilter/nf_conntrack_netlink.c | 5 ++ 3 files changed, 25 insertions(+), 44 deletions(-)
include/net/netfilter/nf_conntrack_expect.h +0 -1
@@ -91,7 +91,6 @@ static inline void nf_ct_unlink_expect(struct nf_conntrack_expect *exp)void nf_ct_remove_expectations (struct nf_conn *ct);void nf_ct_unexpect_related (struct nf_conntrack_expect *exp); -void nf_ct_remove_userspace_expectations (void );/* Allocate space for an expectation: this is mandatory before calling nf_ct_expect_related. You will have to call put afterwards. */
net/netfilter/nf_conntrack_expect.c +21 -42
134 lines not on this page · raw message
net/netfilter/nf_conntrack_netlink.c +4 -1
@@ -2019,6 +2019,10 @@ ctnetlink_create_expect(struct net *net, u16 zone, } help =nfct_help (ct);if (!help) { + err = -EOPNOTSUPP ; + goto out; + } +if (test_bit (IPS_USERSPACE_HELPER_BIT , &ct->status)) {if (!cda[CTA_EXPECT_TIMEOUT ]) { err = -EINVAL ; goto out;
@@ -2208,7 +2212,6 @@ static void __exit ctnetlink_exit(void) {pr_info ("ctnetlink: unregistering from nfnetlink.\n" ); -nf_ct_remove_userspace_expectations ();#ifdef CONFIG_NF_CONNTRACK_EVENTS nf_ct_expect_unregister_notifier (&ctnl_notifier_exp);nf_conntrack_unregister_notifier (&ctnl_notifier);
Am 12.04.2011 23:59, schrieb Pablo Neira Ayuso:
Quoted 6 lines
Hi Patrick,
The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.
Just wondering, how can this happen? We take a reference for userspace expectations just as we do for kernel expectations.
Ok, I see, we are releasing it again at the end of ctnetlink_create_expect(), that seems to be the actual problem if I'm not mistaken.
Quoted 13 lines
The idea to fix this is to extend the iptables CT target to explicit allocate the helper extension for conntracks that are suppose to behave as master for user-space expectations.
In the case of the userspace FTP helper, people would need to add the following rule:
iptables -A PREROUTING -t raw \ -p tcp --dport 21 -j CT --userspace-helperThus, we can store the list of expectations that belong to one master, and delete them in case that the master vanishes.
Pablo Neira Ayuso
maintainerQuoted 8 lines · Patrick McHardy
Am 12.04.2011 23:59, schrieb Pablo Neira Ayuso:
Quoted 6 lines
Hi Patrick,
The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.
Quoted 6 lines
Just wondering, how can this happen? We take a reference for userspace expectations just as we do for kernel expectations.
Ok, I see, we are releasing it again at the end of
ctnetlink_create_expect(), that seems to be the actual problem if I'm not mistaken.
Indeed, we have keep that reference, that would fix the problem.
Still, with the curent approach the userspace expectation will be valid after the master conntrack has expired.
So we can do the following: Fix this refcount issue in -stable and current, and schedule these patches for nf-next to change the behaviour.
What do you think?
Am 13.04.2011 13:47, schrieb Pablo Neira Ayuso:
Quoted 9 lines · Patrick McHardy
Quoted 8 lines
Am 12.04.2011 23:59, schrieb Pablo Neira Ayuso:
Quoted 6 lines
Hi Patrick,
The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.
Quoted 8 lines
Quoted 6 lines
Just wondering, how can this happen? We take a reference for userspace expectations just as we do for kernel expectations.
Ok, I see, we are releasing it again at the end of
ctnetlink_create_expect(), that seems to be the actual problem if I'm not mistaken.Indeed, we have keep that reference, that would fix the problem.
We definitely need to hold it anyways since destroy_conntrack() releases it again.
Quoted 7 lines
Still, with the curent approach the userspace expectation will be valid after the master conntrack has expired.
So we can do the following: Fix this refcount issue in -stable and current, and schedule these patches for nf-next to change the behaviour.
What do you think?
I don't know, what's the difference to non-userspace expectations? The same applies to them, we don't require the master to still be active for expectations.
Pablo Neira Ayuso
maintainerQuoted 25 lines · Patrick McHardy
Am 13.04.2011 13:47, schrieb Pablo Neira Ayuso:
Quoted 9 lines · Patrick McHardy
Quoted 8 lines
Am 12.04.2011 23:59, schrieb Pablo Neira Ayuso:
Quoted 6 lines
Hi Patrick,
The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.
Quoted 8 lines
Quoted 6 lines
Just wondering, how can this happen? We take a reference for userspace expectations just as we do for kernel expectations.
Ok, I see, we are releasing it again at the end of
ctnetlink_create_expect(), that seems to be the actual problem if I'm not mistaken.Indeed, we have keep that reference, that would fix the problem.
We definitely need to hold it anyways since
destroy_conntrack()releases it again.Quoted 3 lines
Still, with the curent approach the userspace expectation will be valid after the master conntrack has expired.
Quoted 8 lines
Quoted 4 lines
So we can do the following: Fix this refcount issue in -stable and current, and schedule these patches for nf-next to change the behaviour.
What do you think?
I don't know, what's the difference to non-userspace expectations? The same applies to them, we don't require the master to still be active for expectations.
kernelspace expectations are released if the master is destroyed. userspace expectations will not.
The helper extension is used to store a list of expectations for this master conntrack, so we can release the expectations that are associated. This is not true for userspace expectations, since the master has no list with expectations.
Am 13.04.2011 14:11, schrieb Pablo Neira Ayuso:
Quoted 29 lines · Patrick McHardy
Quoted 28 lines
Am 13.04.2011 13:47, schrieb Pablo Neira Ayuso:
Quoted 9 lines · Patrick McHardy
Quoted 8 lines
Am 12.04.2011 23:59, schrieb Pablo Neira Ayuso:
Quoted 6 lines
Hi Patrick,
The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.
Quoted 8 lines
Quoted 6 lines
Just wondering, how can this happen? We take a reference for userspace expectations just as we do for kernel expectations.
Ok, I see, we are releasing it again at the end of
ctnetlink_create_expect(), that seems to be the actual problem if I'm not mistaken.Indeed, we have keep that reference, that would fix the problem.
We definitely need to hold it anyways since
destroy_conntrack()releases it again.Quoted 6 lines
Still, with the curent approach the userspace expectation will be valid after the master conntrack has expired.
So we can do the following: Fix this refcount issue in -stable and current, and schedule these patches for nf-next to change the behaviour.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
What do you think?
I don't know, what's the difference to non-userspace expectations? The same applies to them, we don't require the master to still be active for expectations.
kernelspace expectations are released if the master is destroyed. userspace expectations will not.
I see, you're talking about unfulfilled expectations. Still, we're releasing expectations in destroy_conntrack(), if we fix the refcount issue, the master conntrack will not be destroyed while userspace expectations exist.
Quoted 4 lines
The helper extension is used to store a list of expectations for this master conntrack, so we can release the expectations that are associated. This is not true for userspace expectations, since the master has no list with expectations.
This raises the question - why are we treating userspace differently in this regard?
Pablo Neira Ayuso
maintainerQuoted 36 lines · Patrick McHardy
Am 13.04.2011 14:11, schrieb Pablo Neira Ayuso:
Quoted 35 lines · Patrick McHardy
Quoted 33 lines
Am 13.04.2011 13:47, schrieb Pablo Neira Ayuso:
Quoted 9 lines · Patrick McHardy
Quoted 8 lines
Am 12.04.2011 23:59, schrieb Pablo Neira Ayuso:
Quoted 6 lines
Hi Patrick,
The following patches rework the userspace expectation support to fix one problematic scenario: if the master conntrack vanishes while there are still userspace expectations, we hit an oops in the destroy event path for expectations.
Quoted 8 lines
Quoted 6 lines
Just wondering, how can this happen? We take a reference for userspace expectations just as we do for kernel expectations.
Ok, I see, we are releasing it again at the end of
ctnetlink_create_expect(), that seems to be the actual problem if I'm not mistaken.Indeed, we have keep that reference, that would fix the problem.
We definitely need to hold it anyways since
destroy_conntrack()releases it again.Quoted 7 lines
Still, with the curent approach the userspace expectation will be valid after the master conntrack has expired.
So we can do the following: Fix this refcount issue in -stable and current, and schedule these patches for nf-next to change the behaviour.
What do you think?
I don't know, what's the difference to non-userspace expectations? The same applies to them, we don't require the master to still be active for expectations.
Quoted 7 lines
Quoted 2 lines
kernelspace expectations are released if the master is destroyed. userspace expectations will not.
I see, you're talking about unfulfilled expectations. Still, we're releasing expectations in
destroy_conntrack(), if we fix the refcount issue, the master conntrack will not be destroyed while userspace expectations exist.
Exactly.
Quoted 7 lines
Quoted 4 lines
The helper extension is used to store a list of expectations for this master conntrack, so we can release the expectations that are associated. This is not true for userspace expectations, since the master has no list with expectations.
This raises the question - why are we treating userspace differently in this regard?
I think we should treat them link the kernelspace expectations.
I'll send a patch to fix this issue, and then tell me if you're OK with this approach so that we can apply it to nf-next.
Pablo Neira Ayuso
maintainerQuoted 4 lines · Patrick McHardy
I see, you're talking about unfulfilled expectations. Still, we're releasing expectations in
destroy_conntrack(), if we fix the refcount issue, the master conntrack will not be destroyed while userspace expectations exist.
I missed one point. We don't seem to increase the refcnt for unfulfilled expectations.
The following example shows a unfulfilled FTP expectation:
# conntrack -L exp 262 proto=6 src=192.168.1.128 dst=130.89.149.226 sport=0 dport=20412 conntrack v1.0.0 (conntrack-tools): 1 expectations have been shown.
# conntrack -L tcp 6 431979 ESTABLISHED src=192.168.1.128 dst=130.89.149.226 sport=37348 dport=21 packets=9 bytes=503 src=130.89.149.226 dst=192.168.1.128 sport=21 dport=37348 packets=8 bytes=564 [ASSURED] mark=0 delta-time=29 use=1
The use field of the master conntrack shows 1.
So, I think that these patches are still the way to fix the problem with the expectations created through ctnetlink.
Am 20.04.2011 14:10, schrieb Pablo Neira Ayuso:
Quoted 8 lines · Patrick McHardy
Quoted 4 lines
I see, you're talking about unfulfilled expectations. Still, we're releasing expectations in
destroy_conntrack(), if we fix the refcount issue, the master conntrack will not be destroyed while userspace expectations exist.I missed one point. We don't seem to increase the refcnt for unfulfilled expectations.
Yes, that's not necessary since unfulfilled expectations are cleaned up once the master is destroyed. Its just the ct->master pointer of an expected connection that takes a reference.
I thought this is what your patches try to fix?
Quoted 12 lines
The following example shows a unfulfilled FTP expectation:
# conntrack -L exp 262 proto=6 src=192.168.1.128 dst=130.89.149.226 sport=0 dport=20412 conntrack v1.0.0 (conntrack-tools): 1 expectations have been shown.# conntrack -L tcp 6 431979 ESTABLISHED src=192.168.1.128 dst=130.89.149.226 sport=37348 dport=21 packets=9 bytes=503 src=130.89.149.226 dst=192.168.1.128 sport=21 dport=37348 packets=8 bytes=564 [ASSURED] mark=0 delta-time=29 use=1
Quoted 4 lines
The use field of the master conntrack shows 1.
So, I think that these patches are still the way to fix the problem with the expectations created through ctnetlink.
Just for my understanding, why not simply take the reference for the master conntrack for "fulfilled" expectations (IOW conntracks that have a master assigned), which is necessary anyways since destroy_conntrack() attempts to release that reference again, and clean up userspace expectations once the master conntrack vanishes?
This would match what we do for kernel expectations and I don't really see why we would want to keep an unfulfilled userspace expectation without the corresponding master.
Pablo Neira Ayuso
maintainerQuoted 9 lines · Patrick McHardy
Am 20.04.2011 14:10, schrieb Pablo Neira Ayuso:
Quoted 2 lines
I missed one point. We don't seem to increase the refcnt for unfulfilled expectations.
Yes, that's not necessary since unfulfilled expectations are cleaned up once the master is destroyed. Its just the
ct->masterpointer of an expected connection that takes a reference.I thought this is what your patches try to fix?
Yes, my patches add the helper extension to the conntrack by means of the CT target. Thus, in the case of userspace helpers, the master conntrack can store the list of unfulfilled expectations that it owns.
Sorry, I misunderstood your previous email, I thought you were telling that we bump master's refcnt for unfulfilled expectation, you were telling the opposite :-).
Quoted 14 lines
Quoted 14 lines
The following example shows a unfulfilled FTP expectation:
# conntrack -L exp 262 proto=6 src=192.168.1.128 dst=130.89.149.226 sport=0 dport=20412 conntrack v1.0.0 (conntrack-tools): 1 expectations have been shown.# conntrack -L tcp 6 431979 ESTABLISHED src=192.168.1.128 dst=130.89.149.226 sport=37348 dport=21 packets=9 bytes=503 src=130.89.149.226 dst=192.168.1.128 sport=21 dport=37348 packets=8 bytes=564 [ASSURED] mark=0 delta-time=29 use=1The use field of the master conntrack shows 1.
Quoted 8 lines
Quoted 2 lines
So, I think that these patches are still the way to fix the problem with the expectations created through ctnetlink.
Just for my understanding, why not simply take the reference for the master conntrack for "fulfilled" expectations (IOW conntracks that have a master assigned), which is necessary anyways since
destroy_conntrack()attempts to release that reference again, and clean up userspace expectations once the master conntrack vanishes?
That's fine to me.
Quoted 3 lines
This would match what we do for kernel expectations and I don't really see why we would want to keep an unfulfilled userspace expectation without the corresponding master.
Agreed.
Then, we need to allocate the conntrack helper extension to store the list of unfulfilled expectations, which is what my patches do.
Patrick, are you going to merge these?
I don't see it in nf-next-2.6.git.
Quoted 6 lines · Pablo Neira Ayuso
Quoted 1 line · Patrick McHardy
I thought this is what your patches try to fix?
Yes, my patches add the helper extension to the conntrack by means of the CT target. Thus, in the case of userspace helpers, the master conntrack can store the list of unfulfilled expectations that it owns.
Its not clear to me why the CT master isn't added on a master-less connection at the time that a userspace expectation is set (instead of having to use the CT target in the raw table to create the master for all connections that might potentially need a userspace expectation).
However, I'll take any fix that stops the oopsing.
Thanks,
Sam
Quoted 3 lines · Sam Roberts
Patrick, are you going to merge these?
I don't see it in nf-next-2.6.git.
Btw, here's a reproduction tool that will oops:
https://github.com/sam-github/nfct-expect-oops
Cheers,
Sam