net/sched: cls_u32: fix refcount leak
Subsystems (4)
cover net/sched: cls_u32: fix refcount leak
a refcount leak in the error path of u32_change() has been recently introduced. It can be observed with the following commands:
[root@f31 ~]# tc filter replace dev eth0 ingress protocol ip prio 97 \
Quoted 1 line
u32 match ip src 127.0.0.1/32 indev notexist20 flowid 1:1 action drop
RTNETLINK answers: Invalid argument
We have an error talking to the kernel
[root@f31 ~]# tc filter replace dev eth0 ingress protocol ip prio 98 \
Quoted 1 line
handle 42:42 u32 divisor 256
Error: cls_u32: Divisor can only be used on a hash table. We have an error talking to the kernel
[root@f31 ~]# tc filter replace dev eth0 ingress protocol ip prio 99 \
Quoted 1 line
u32 ht 47:47
Error: cls_u32: Specified hash table not found.
We have an error talking to the kernel
they all legitimately return -EINVAL; however, they leave semi-configured filters at eth0 tc ingress:
[root@f31 ~]# tc filter show dev eth0 ingress filter protocol ip pref 97 u32 chain 0
filter protocol ip pref 97 u32 chain 0 fh 800: ht divisor 1 filter protocol ip pref 98 u32 chain 0
filter protocol ip pref 98 u32 chain 0 fh 801: ht divisor 1 filter protocol ip pref 99 u32 chain 0
filter protocol ip pref 99 u32 chain 0 fh 802: ht divisor 1
With older kernels, filters were unconditionally considered empty (and thus de-refcounted) on the error path of ->change().
After commit 8b64678e0af8 ("net: sched: refactor tp insert/delete for concurrent execution"), filters were considered empty when the walk() function didn't set 'walker.stop' to 1.
Finally, with commit 6676d5e416ee ("net: sched: set dedicated tcf_walker flag when tp is empty"), tc filters are considered empty unless the walker function is called with a non-NULL handle. This last change doesn't fit cls_u32 design, because at least the "root hnode" is (almost) always non-NULL, as it's allocated in u32_init().
- patch 1/2 is a proposal to restore the original kernel behavior, where no filter was installed in the error path of u32_change().
- patch 2/2 adds tdc selftests that can be ued to verify the correct behavior of u32 in the error path of ->change().
Davide Caratti (2):
net/sched: cls_u32: fix refcount leak in the error path of
u32_change()
tc-testing: initial tdc selftests for cls_u32
net/sched/cls_u32.c | 25 +++ .../tc-testing/tc-tests/filters/tests.json | 22 -- .../tc-testing/tc-tests/filters/u32.json | 205 ++++++++++++++++++ 3 files changed, 230 insertions(+), 22 deletions(-) create mode 100644 tools/testing/selftests/tc-testing/tc-tests/filters/u32.json
Signature · 1 line
2.23.0
PATCH 1/2 net/sched: cls_u32: fix refcount leak in the error path of u32_change()
when users replace cls_u32 filters with new ones having wrong parameters, so that u32_change() fails to validate them, the kernel doesn't roll-back correctly, and leaves semi-configured rules.
Fix this in u32_walk(), avoiding a call to the walker function on filters that don't have a match rule connected. The side effect is, these "empty" filters are not even dumped when present; but that shouldn't be a problem as long as we are restoring the original behaviour, where semi-configured filters were not even added in the error path of u32_change().
--- net/sched/cls_u32.c | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+)
net/sched/cls_u32.c +25 -0
@@ -1108,10 +1108,33 @@ static int u32_change(struct net *net, struct sk_buff *in_skb,return err; } +static bool u32_hnode_empty (struct tc_u_hnode *ht,bool *non_root_ht) +{ +int i; + +if (!ht) +return true; +if (!ht->is_root) { + *non_root_ht = true; +return false; + } +if (*non_root_ht) +return false; +if (ht->refcnt <2 ) +return true; + +for (i =0 ; i <= ht->divisor; i++) { +if (rtnl_dereference (ht->ht[i])) +return false; + } +return true; +} +static void u32_walk (struct tcf_proto *tp,struct tcf_walker *arg,bool rtnl_held) {struct tc_u_common *tp_c = tp->data; +bool non_root_ht = false;struct tc_u_hnode *ht;struct tc_u_knode *n;unsigned int h;
@@ -1124,6 +1147,8 @@ static void u32_walk(struct tcf_proto *tp, struct tcf_walker *arg, ht = rtnl_dereference(ht->next)) {if (ht->prio != tp->prio)continue ; +if (u32_hnode_empty (ht, &non_root_ht)) +return ;if (arg->count >= arg->skip) {if (arg->fn (tp, ht, arg) <0 ) { arg->stop =1 ;
Signature · 1 line
2.23.0
Jamal Hadi Salim
maintainerRe: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Quoted 4 lines · Davide Caratti
when users replace
cls_u32filters with new ones having wrong parameters, so thatu32_change()fails to validate them, the kernel doesn't roll-back correctly, and leaves semi-configured rules.
Quoted 8 lines
Fix this in
u32_walk(), avoiding a call to the walker function on filters that don't have a match rule connected. The side effect is, these "empty" filters are not even dumped when present; but that shouldn't be a problem as long as we are restoring the original behaviour, where semi-configured filters were not even added in the error path ofu32_change().Fixes 6676d5e416ee ("net: sched: set dedicated tcf_walker flag when tp is empty")Signed-off-by Davide Caratti <[email protected]>
Hi Davide,
Great catch (and good test case addition),
but I am not sure about the fix.
Unless I am misunderstanding the flow is:
You enter bad rules, validation fails, partial state had already been created (in this case root hts) before validation failure, you then leave the partial state in the kernel but when someone dumps you hide these bad tables?
It sounds like the root cause is there is a missing destroy() invocation somewhere during the create/validation failure - which is what needs fixing... Those dangling tables should not have been inserted; maybe somewhere along the code path for tc_new_tfilter(). Note "replace" is essentially "create if it doesnt
exist" semantic therefore NLM_F_CREATE will be set.
Since this is a core cls issue - I would check all other classifiers with similar aggrevation - make some attribute fail in the middle or end.
Very likely only u32 is the victim.
cheers,
jamal
Jamal Hadi Salim
maintainerRe: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Hi Davide,
I ran your test on my laptop (4.19) and nothing dangled. Looking at that area of the code difference 4.19 vs current net-next there was a destroy() in there that migrated into the inner guts of tcf_chain_tp_delete_empty() Vlad? My gut feeling is restoring the old logic like this would work at least for u32:
------------
net/sched/cls_api.c +3 -1
@@ -2135,8 +2135,10 @@ static int tc_new_tfilter(struct sk_buff *skb, struct nlmsghdr *n, } errout: -if (err && tp_created) +if (err && tp_created) { +tcf_proto_destroy (tp, rtnl_held, true,NULL );tcf_chain_tp_delete_empty (chain, tp, rtnl_held,NULL ); + } errout_tp:if (chain) {if (tp && !IS_ERR (tp))
-----
Maybe even better tcf_proto_put(tp, rtnl_held, NULL) directly instead and no need for tcf_chain_tp_delete_empty().
Of course above not even compile tested and may have consequences for other classifiers (flower would be a good test) and concurency intentions. Thoughts?
cheers,
jamal
Quoted 41 lines · Jamal Hadi Salim
Quoted 4 lines · Davide Caratti
when users replace
cls_u32filters with new ones having wrong parameters, so thatu32_change()fails to validate them, the kernel doesn't roll-back correctly, and leaves semi-configured rules.Quoted 8 lines
Fix this in
u32_walk(), avoiding a call to the walker function on filters that don't have a match rule connected. The side effect is, these "empty" filters are not even dumped when present; but that shouldn't be a problem as long as we are restoring the original behaviour, where semi-configured filters were not even added in the error path ofu32_change().Fixes 6676d5e416ee ("net: sched: set dedicated tcf_walker flag when tp is empty")Signed-off-by Davide Caratti <[email protected]>Hi Davide,
Great catch (and good test case addition),
but I am not sure about the fix.Unless I am misunderstanding the flow is:
You enter bad rules, validation fails, partial state had already been created (in this case root hts) before validation failure, you then leave the partial state in the kernel but when someone dumps you hide these bad tables?It sounds like the root cause is there is a missing
destroy()invocation somewhere during the create/validation failure - which is what needs fixing... Those dangling tables should not have been inserted; maybe somewhere along the code path fortc_new_tfilter(). Note "replace" is essentially "create if it doesnt
exist" semantic thereforeNLM_F_CREATEwill be set.Since this is a core cls issue - I would check all other classifiers with similar aggrevation - make some attribute fail in the middle or end.
Very likely only u32 is the victim.cheers,
jamal
Re: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Quoted 31 lines · Jamal Hadi Salim
Hi Davide,
I ran your test on my laptop (4.19) and nothing dangled. Looking at that area of the code difference 4.19 vs current net-next there was a
destroy()in there that migrated into the inner guts oftcf_chain_tp_delete_empty()Vlad? My gut feeling is restoring the old logic like this would work at least for u32:------------
net/sched/cls_api.c +3
-1@@ -2135,8 +2135,10 @@ static int tc_new_tfilter(struct sk_buff *skb, struct nlmsghdr *n, } errout: -if (err && tp_created) +if (err && tp_created) { +tcf_proto_destroy (tp, rtnl_held, true,NULL );tcf_chain_tp_delete_empty (chain, tp, rtnl_held,NULL ); + } errout_tp:if (chain) {if (tp && !IS_ERR(tp))-----
Maybe even better
tcf_proto_put(tp, rtnl_held, NULL)directly instead and no need fortcf_chain_tp_delete_empty().
Quoted 50 lines
Of course above not even compile tested and may have consequences for other classifiers (flower would be a good test) and concurency intentions. Thoughts?
cheers,
jamalQuoted 42 lines · Jamal Hadi Salim
Quoted 10 lines · Davide Caratti
when users replace
cls_u32filters with new ones having wrong parameters, so thatu32_change()fails to validate them, the kernel doesn't roll-back correctly, and leaves semi-configured rules.Fix this in
u32_walk(), avoiding a call to the walker function on filters that don't have a match rule connected. The side effect is, these "empty" filters are not even dumped when present; but that shouldn't be a problem as long as we are restoring the original behaviour, where semi-configured filters were not even added in the error path ofu32_change().Quoted 3 lines
Fixes 6676d5e416ee ("net: sched: set dedicated tcf_walker flag when tp isempty")
Signed-off-by Davide Caratti <[email protected]>Hi Davide,
Great catch (and good test case addition),
but I am not sure about the fix.Unless I am misunderstanding the flow is:
You enter bad rules, validation fails, partial state had already been created (in this case root hts) before validation failure, you then leave the partial state in the kernel but when someone dumps you hide these bad tables?It sounds like the root cause is there is a missing
destroy()invocation somewhere during the create/validation failure - which is what needs fixing... Those dangling tables should not have been inserted; maybe somewhere along the code path fortc_new_tfilter(). Note "replace" is essentially "create if it doesnt
exist" semantic thereforeNLM_F_CREATEwill be set.Since this is a core cls issue - I would check all other classifiers with similar aggrevation - make some attribute fail in the middle or end.
Very likely only u32 is the victim.cheers,
jamal
Hi Jamal,
Just destroying tp unconditionally will break unlocked case (flower) because of possibility of concurrent insertion of new filters to the same tp instance.
The root cause here is precisely described by Davide in cover letter - to accommodate concurrent insertions cls API verifies that tp instance is empty before deleting it and since there is no cls ops to do it directly, it relies on checking that walk() stopped without accessing any filters instead. Unfortunately, somw classifier implementations assumed that there is always at least one filter on classifier (I fixed several of these) and now Davide also uncovered this leak in u32.
As a simpler solution to fix such issues once and for all I can propose not to perform the walk() check at all and assume that any classifier implementation that doesn't have TCF_PROTO_OPS_DOIT_UNLOCKED flag set is empty in tcf_chain_tp_delete_empty() (there is no possibility of concurrent insertion when synchronizing with rtnl).
WDYT?
Regards,
Vlad
Jamal Hadi Salim
maintainerRe: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Quoted 6 lines · Vlad Buslov
Hi Jamal,
Just destroying tp unconditionally will break unlocked case (flower) because of possibility of concurrent insertion of new filters to the same tp instance.
I was worried about that. So rtnlheld doesnt help?
Quoted 8 lines
The root cause here is precisely described by Davide in cover letter - to accommodate concurrent insertions cls API verifies that tp instance is empty before deleting it and since there is no cls ops to do it directly, it relies on checking that
walk()stopped without accessing any filters instead. Unfortunately, somw classifier implementations assumed that there is always at least one filter on classifier (I fixed several of these) and now Davide also uncovered this leak in u32.
Quoted 7 lines
As a simpler solution to fix such issues once and for all I can propose not to perform the
walk()check at all and assume that any classifier implementation that doesn't haveTCF_PROTO_OPS_DOIT_UNLOCKEDflag set is empty intcf_chain_tp_delete_empty()(there is no possibility of concurrent insertion when synchronizing with rtnl).WDYT?
IMO that would be a cleaner fix give walk() is used for other operations and this is a core cls issue.
Also: documenting what it takes for a classifier to support TCF_PROTO_OPS_DOIT_UNLOCKED is useful (you may have done this in some commit already).
cheers,
jamal
Re: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Quoted 11 lines · Jamal Hadi Salim
Quoted 6 lines · Vlad Buslov
Hi Jamal,
Just destroying tp unconditionally will break unlocked case (flower) because of possibility of concurrent insertion of new filters to the same tp instance.
I was worried about that. So rtnlheld doesnt help?
Rtnl_held flag can be set for multiple other reasons besides locked/unlocked classifier implementation (parent Qdisc doesn't support unlocked execution, retry in tc_new_tfilter(), etc.).
Quoted 15 lines
Quoted 15 lines
The root cause here is precisely described by Davide in cover letter - to accommodate concurrent insertions cls API verifies that tp instance is empty before deleting it and since there is no cls ops to do it directly, it relies on checking that
walk()stopped without accessing any filters instead. Unfortunately, somw classifier implementations assumed that there is always at least one filter on classifier (I fixed several of these) and now Davide also uncovered this leak in u32.As a simpler solution to fix such issues once and for all I can propose not to perform the
walk()check at all and assume that any classifier implementation that doesn't haveTCF_PROTO_OPS_DOIT_UNLOCKEDflag set is empty intcf_chain_tp_delete_empty()(there is no possibility of concurrent insertion when synchronizing with rtnl).
Quoted 7 lines
Quoted 1 line
WDYT?
IMO that would be a cleaner fix give
walk()is used for other operations and this is a core cls issue.
Also: documenting what it takes for a classifier to supportTCF_PROTO_OPS_DOIT_UNLOCKEDis useful (you may have done this in some commit already).
Well, idea was not to have classifier implement any specific API. If classifier has TCF_PROTO_OPS_DOIT_UNLOCKED, then cls API doesn't obtain rtnl and it is up to classifier to implement whatever locking necessary internally. However, my approach to checking for empty classifier instance uncovered multiple bugs and inconsistencies in classifier implementations of ops->walk().
So I guess the requirement now is for unlocked classifier to have sane implementation of ops->walk() that doesn't assume >1 filters and correctly handles the case when insertion of first filter after classifier instance is created fails.
Quoted 3 lines
cheers,
jamal
Re: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Quoted 47 lines · Vlad Buslov
Quoted 11 lines · Jamal Hadi Salim
Quoted 6 lines · Vlad Buslov
Hi Jamal,
Just destroying tp unconditionally will break unlocked case (flower) because of possibility of concurrent insertion of new filters to the same tp instance.
I was worried about that. So rtnlheld doesnt help?
Rtnl_heldflag can be set for multiple other reasons besides locked/unlocked classifier implementation (parent Qdisc doesn't support unlocked execution, retry intc_new_tfilter(), etc.).Quoted 15 lines
Quoted 15 lines
The root cause here is precisely described by Davide in cover letter - to accommodate concurrent insertions cls API verifies that tp instance is empty before deleting it and since there is no cls ops to do it directly, it relies on checking that
walk()stopped without accessing any filters instead. Unfortunately, somw classifier implementations assumed that there is always at least one filter on classifier (I fixed several of these) and now Davide also uncovered this leak in u32.As a simpler solution to fix such issues once and for all I can propose not to perform the
walk()check at all and assume that any classifier implementation that doesn't haveTCF_PROTO_OPS_DOIT_UNLOCKEDflag set is empty intcf_chain_tp_delete_empty()(there is no possibility of concurrent insertion when synchronizing with rtnl).Quoted 7 lines
Quoted 1 line
WDYT?
IMO that would be a cleaner fix give
walk()is used for other operations and this is a core cls issue.
Also: documenting what it takes for a classifier to supportTCF_PROTO_OPS_DOIT_UNLOCKEDis useful (you may have done this in some commit already).Well, idea was not to have classifier implement any specific API. If classifier has
TCF_PROTO_OPS_DOIT_UNLOCKED, then cls API doesn't obtain rtnl and it is up to classifier to implement whatever locking necessary internally. However, my approach to checking for empty classifier instance uncovered multiple bugs and inconsistencies in classifier implementations ofops->walk().
Quoted 4 lines
So I guess the requirement now is for unlocked classifier to have sane implementation of
ops->walk()that doesn't assume >1 filters and correctly handles the case when insertion of first filter after classifier instance is created fails.
BTW another approach would be to extend ops with new callback delete_empty(), require unlocked implementation to define it and move functionality of tcf_proto_check_delete() there. Such approach would remove the need for (ab)using ops->walk() for this since internally in classifier implementation there is always a way to correctly verify that classifier instance is empty. Don't know which approach is better in this case.
Quoted 4 lines
Quoted 4 lines
cheers,
jamal
Jamal Hadi Salim
maintainerRe: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Quoted 8 lines · Vlad Buslov
BTW another approach would be to extend ops with new callback
delete_empty(), require unlocked implementation to define it and move functionality oftcf_proto_check_delete()there. Such approach would remove the need for (ab)usingops->walk()for this since internally in classifier implementation there is always a way to correctly verify that classifier instance is empty. Don't know which approach is better in this case.
I see both as complementing each other. delete_empty() could serves like guidance almost for someone who wants to implement parallelization (and stops abuse of walk()) and TCF_PROTO_OPS_DOIT_UNLOCKED is more of a shortcut. IOW, you could at the top of tcf_proto_check_delete() return true if TCF_PROTO_OPS_DOIT_UNLOCKED is set while still invoking delete_empty() (instead of walk()) if you go past that. Would that work?
cheers,
jamal
Jamal Hadi Salim
maintainerRe: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Quoted 6 lines · Jamal Hadi Salim
I see both as complementing each other.
delete_empty()could serves like guidance almost for someone who wants to implement parallelization (and stops abuse ofwalk()) andTCF_PROTO_OPS_DOIT_UNLOCKEDis more of a shortcut. IOW, you could at the top oftcf_proto_check_delete()return true ifTCF_PROTO_OPS_DOIT_UNLOCKEDis set while still invoking
Something like attached...
cheers,
jamal
net/sched/cls_api.c +5 -0
@@ -331,6 +331,11 @@ static bool tcf_proto_is_empty(struct tcf_proto *tp, bool rtnl_held)static bool tcf_proto_check_delete (struct tcf_proto *tp,bool rtnl_held) { +if (!(tp->ops->flags &TCF_PROTO_OPS_DOIT_UNLOCKED )) { + tp->deleting = true; +return tp->deleting; + } +spin_lock (&tp->lock);if (tcf_proto_is_empty (tp, rtnl_held)) tp->deleting = true;
Jamal Hadi Salim
maintainerRe: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Quoted 11 lines · Jamal Hadi Salim
Quoted 6 lines · Jamal Hadi Salim
I see both as complementing each other.
delete_empty()could serves like guidance almost for someone who wants to implement parallelization (and stops abuse ofwalk()) andTCF_PROTO_OPS_DOIT_UNLOCKEDis more of a shortcut. IOW, you could at the top oftcf_proto_check_delete()return true ifTCF_PROTO_OPS_DOIT_UNLOCKEDis set while still invokingSomething like attached...
Vlad,
I tested this and it seems to fix the issue. But there may be other consequences...
cheers,
jamal
Re: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Quoted 11 lines · Jamal Hadi Salim
Quoted 10 lines · Jamal Hadi Salim
Quoted 6 lines · Jamal Hadi Salim
I see both as complementing each other.
delete_empty()could serves like guidance almost for someone who wants to implement parallelization (and stops abuse ofwalk()) andTCF_PROTO_OPS_DOIT_UNLOCKEDis more of a shortcut. IOW, you could at the top oftcf_proto_check_delete()return true ifTCF_PROTO_OPS_DOIT_UNLOCKEDis set while still invoking
Quoted 8 lines
Quoted 1 line
Something like attached...
Vlad,
I tested this and it seems to fix the issue. But there may be other consequences...cheers,
jamal
Hi Jamal,
Yes, I think the patch would work. However, we don't really need the flags check, if we are going to implement the new ops->delete_empty() callback because it can work like this:
if (!tp->ops->delete_empty) {
tp->deleting = true;
return tp->deleting;
} else {
return tp->ops->delete_empty(tp);
}
WDYT?
Regards,
Vlad
Jamal Hadi Salim
maintainerRe: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Quoted 8 lines · Vlad Buslov
Hi Jamal,
Yes, I think the patch would work. However, we don't really need the flags check, if we are going to implement the new
ops->delete_empty()callback because it can work like this:
Quoted 8 lines
if (!tp->ops->delete_empty) { tp->deleting = true; return tp->deleting; } else {return
tp->ops->delete_empty(tp);
}WDYT?
Looks reasonable - we kill two birds with one stone. We'd need to revert the patch David took in.
cheers,
jamal
Re: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Quoted 14 lines · Jamal Hadi Salim
Quoted 13 lines · Vlad Buslov
Hi Jamal,
Yes, I think the patch would work. However, we don't really need the flags check, if we are going to implement the new
ops->delete_empty()callback because it can work like this:if (!tp->ops->delete_empty) { tp->deleting = true; return tp->deleting; } else {return
tp->ops->delete_empty(tp);
}
Quoted 7 lines
Quoted 1 line
WDYT?
Looks reasonable - we kill two birds with one stone. We'd need to revert the patch David took in.
cheers,
jamal
... ok, I can try to do a series that reverts the fix on cls_u32 and implements the check_delete_empty for cls_flower. You might find it above the christmas tree (*)?
Signature · 3 lines
davide (*) above, not under, because the tree is reversed.
Jamal Hadi Salim
maintainerRe: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
On 2019-12-20 10:20 a.m., Davide Caratti wrote:
[..]
Quoted 5 lines
... ok, I can try to do a series that reverts the fix on
cls_u32and implements thecheck_delete_emptyforcls_flower. You might find it above the christmas tree (*)?
Thanks Davide. Please go for it..
cheers,
jamal
Re: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
In reply to Vlad Buslovhi Jamal and Vlad,
thanks a lot for sharing your thoughts.
Quoted 3 lines · Vlad Buslov
Quoted 3 lines
Quoted 3 lines
IMO that would be a cleaner fix give
walk()is used for other operations and this is a core cls issue.
[...]
Quoted 4 lines
Quoted 4 lines
So I guess the requirement now is for unlocked classifier to have sane implementation of
ops->walk()that doesn't assume >1 filters and correctly handles the case when insertion of first filter after classifier instance is created fails.
I tried forcing an error in matchall, and didn't observe this problem:
[root@f31 ~]# perf record -e probe:mall_change__return -agR -- tc filter add dev dam0 parent root matchall skip_sw action drop RTNETLINK answers: Operation not supported
We have an error talking to the kernel
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.225 MB perf.data (1 samples) ]
[root@f31 ~]# perf script
tc 115241 [002] 19665.372130: probe:mall_change__return: (ffffffffc115f930 <- ffffffffb98a7266) ret=0xffffffa1
ffffffffb9066790 kretprobe_trampoline+0x0 (vmlinux)
7fa64c16cb77 __libc_sendmsg+0x17 (/usr/lib64/libc-2.30.so)[root@f31 ~]# tc filter show dev dam0
[root@f31 ~]#
and similar test can be also carried for the other classifiers (on unpatched kernel), so it should be easy to understand if there are other filter that show the same problem.
Quoted 7 lines
BTW another approach would be to extend ops with new callback
delete_empty(), require unlocked implementation to define it and move functionality oftcf_proto_check_delete()there. Such approach would remove the need for (ab)usingops->walk()for this since internally in classifier implementation there is always a way to correctly verify that classifier instance is empty. Don't know which approach is better in this case.
I like the idea of using walk() only when filters are dumped, and handling the check for empty filters on deletion with a separate callback. That would allow de-refcounting the filter in the error path unconditionally
for all classifiers, like old kernel was doing, unless the classifier
implements its own "check empty" function.
how does it sound?
thanks!
Signature · 1 line
davide
Jamal Hadi Salim
maintainerRe: [PATCH net 1/2] net/sched: cls_u32: fix refcount leak in the error path of u32_change()
Quoted 20 lines · Davide Caratti
hi Jamal and Vlad,
thanks a lot for sharing your thoughts.
Quoted 2 lines · Vlad Buslov
Quoted 2 lines
Quoted 2 lines
IMO that would be a cleaner fix give
walk()is used for other operations and this is a core cls issue.I tried forcing an error in matchall, and didn't observe this problem:
[root@f31 ~]# perf record -e probe:mall_change__return -agR -- tc filter add dev dam0 parent root matchall
skip_swaction drop RTNETLINK answers: Operation not supported
We have an error talking to the kernel
[ perf record: Woken up 1 times to write data ]
[ perf record: Captured and wrote 1.225 MB perf.data (1 samples) ]
[root@f31 ~]# perf script
tc 115241 [002] 19665.372130: probe:mall_change__return: (ffffffffc115f930<-ffffffffb98a7266) ret=0xffffffa1ffffffffb9066790 kretprobe_trampoline+0x0 (vmlinux) 7fa64c16cb77 __libc_sendmsg+0x17 (/usr/lib64/libc-2.30.so)
Quoted 7 lines
[root@f31 ~]# tc filter show dev dam0
[root@f31 ~]#and similar test can be also carried for the other classifiers (on unpatched kernel), so it should be easy to understand if there are other filter that show the same problem.
I think this one works because of the simpler walk().
In the corner cases you caught u32 is more complex. It would create multiple tp and so the list would not appear empty.
Quoted 8 lines
Quoted 7 lines
BTW another approach would be to extend ops with new callback
delete_empty(), require unlocked implementation to define it and move functionality oftcf_proto_check_delete()there. Such approach would remove the need for (ab)usingops->walk()for this since internally in classifier implementation there is always a way to correctly verify that classifier instance is empty. Don't know which approach is better in this case.
Quoted 7 lines
I like the idea of using
walk()only when filters are dumped, and handling the check for empty filters on deletion with a separate callback. That would allow de-refcounting the filter in the error path unconditionally
for all classifiers, like old kernel was doing, unless the classifier
implements its own "check empty" function.how does it sound?
Agreed.
Note: my comment on other email was also to consider using TCF_PROTO_OPS_DOIT_UNLOCKED for obvious cases.
cheers,
jamal
PATCH 2/2 tc-testing: initial tdc selftests for cls_u32
In reply to Davide Caratti- move test "e9a3 - Add u32 with source match" to u32.json, and change the match pattern to catch all hnodes
- add testcases for relevant error paths of cls_u32 module
--- .../tc-testing/tc-tests/filters/tests.json | 22 -- .../tc-testing/tc-tests/filters/u32.json | 205 ++++++++++++++++++ 2 files changed, 205 insertions(+), 22 deletions(-) create mode 100644 tools/testing/selftests/tc-testing/tc-tests/filters/u32.json
tools/testing/selftests/tc-testing/tc-tests/filters/tests.json +0 -22
@@ -1,26 +1,4 @@ [ - { - "id": "e9a3", - "name": "Add u32 with source match", - "category": [ - "filter", - "u32" - ], - "plugins": { - "requires": "nsPlugin" - }, - "setup": [ - "$TC qdisc add dev $DEV1 ingress" - ], - "cmdUnderTest": "$TC filter add dev $DEV1 parent ffff: protocol ip prio 1 u32 match ip src 127.0.0.1/32 flowid 1:1 action ok", - "expExitCode": "0", - "verifyCmd": "$TC filter show dev $DEV1 parent ffff:", - "matchPattern": "match 7f000001/ffffffff at 12", - "matchCount": "1", - "teardown": [ - "$TC qdisc del dev $DEV1 ingress" - ] - }, { "id": "2638", "name": "Add matchall and try to get it",
tools/testing/selftests/tc-testing/tc-tests/filters/u32.json new file +205 -0
205 lines not on this page · raw message
Signature · 1 line
2.23.0
David Miller
maintainerFrom: Davide Caratti <[email protected]>
Date: Wed, 18 Dec 2019 00:00:03 +0100
Quoted 2 lines
a refcount leak in the error path of
u32_change()has been recently introduced. It can be observed with the following commands:
...
Quoted 2 lines
they all legitimately return
-EINVAL; however, they leave semi-configured filters at eth0 tc ingress:
...
Quoted 11 lines
With older kernels, filters were unconditionally considered empty (and thus de-refcounted) on the error path of ->change().
After commit8b64678e0af8("net: sched: refactor tp insert/delete for concurrent execution"), filters were considered empty when thewalk()function didn't set 'walker.stop' to 1.
Finally, with commit6676d5e416ee("net: sched: set dedicatedtcf_walkerflag when tp is empty"), tc filters are considered empty unless the walker function is called with a non-NULL handle. This last change doesn't fitcls_u32design, because at least the "root hnode" is (almost) always non-NULL, as it's allocated inu32_init().
Quoted 4 lines
- patch 1/2 is a proposal to restore the original kernel behavior, where no filter was installed in the error path of
u32_change().
- patch 2/2 adds tdc selftests that can be ued to verify the correct behavior of u32 in the error path of ->change().
Series applied, thanks.
Jamal Hadi Salim
maintainerQuoted 20 lines · David Miller
From: Davide Caratti <[email protected]>
Date: Wed, 18 Dec 2019 00:00:03 +0100Quoted 2 lines
a refcount leak in the error path of
u32_change()has been recently introduced. It can be observed with the following commands:...
Quoted 2 lines
they all legitimately return
-EINVAL; however, they leave semi-configured filters at eth0 tc ingress:...
Quoted 11 lines
With older kernels, filters were unconditionally considered empty (and thus de-refcounted) on the error path of ->change().
After commit8b64678e0af8("net: sched: refactor tp insert/delete for concurrent execution"), filters were considered empty when thewalk()function didn't set 'walker.stop' to 1.
Finally, with commit6676d5e416ee("net: sched: set dedicatedtcf_walkerflag when tp is empty"), tc filters are considered empty unless the walker function is called with a non-NULL handle. This last change doesn't fitcls_u32design, because at least the "root hnode" is (almost) always non-NULL, as it's allocated inu32_init().
Quoted 7 lines
Quoted 4 lines
- patch 1/2 is a proposal to restore the original kernel behavior, where no filter was installed in the error path of
u32_change().
- patch 2/2 adds tdc selftests that can be ued to verify the correct behavior of u32 in the error path of ->change().Series applied, thanks.
There is still an ongoing discussion on this patch set...
cheers,
jamal