evaluate: kill anon sets with one element
Florian Westphal
maintainerPATCH 1/1 evaluate: kill anon sets with one element
convert "ip saddr { 1.1.1.1 }" to "ip saddr 1.1.1.1". Both do the same, but second form is faster since no single-element anon set is created.
Fix up the remaining test cases to expect transformations of the form "meta l4proto { 33-55}" to "meta l4proto 33-55".
--- src/evaluate.c | 19 ++++++ tests/py/any/meta.t | 8 +-- tests/py/any/meta.t.json | 49 +++++++------ tests/py/any/meta.t.payload | 35 +++------- tests/py/ip/igmp.t | 6 +- tests/py/ip/igmp.t.json | 56 +++------------ tests/py/ip/igmp.t.payload | 76 +++------------------ tests/shell/testcases/nft-f/0016redefines_1 | 2 +- 8 files changed, 81 insertions(+), 170 deletions(-)
src/evaluate.c +19 -0
@@ -1253,6 +1253,7 @@ static int expr_evaluate_set_elem(struct eval_ctx *ctx, struct expr **expr)static int expr_evaluate_set (struct eval_ctx *ctx,struct expr **expr) {struct expr *set = *expr, *i, *next; +unsigned int count =0 ;list_for_each_entry_safe (i, next, &set->expressions, list) {if (list_member_evaluate (ctx, &i) <0 )
@@ -1277,6 +1278,8 @@ static int expr_evaluate_set(struct eval_ctx *ctx, struct expr **expr)return expr_error (ctx->msgs, i,"Set member is not constant" ); + count++; +if (i->etype ==EXPR_SET ) {/* Merge recursive set definitions */ list_splice_tail_init (&i->expressions, &i->list);
@@ -1288,6 +1291,22 @@ static int expr_evaluate_set(struct eval_ctx *ctx, struct expr **expr) set->set_flags |=NFT_SET_INTERVAL ; } +if (!ctx->set && count ==1 ) { + i = list_first_entry(&set->expressions, struct expr, list); +if (i->etype ==EXPR_SET_ELEM ) { +switch (i->key->etype) { +case EXPR_RANGE : +case EXPR_VALUE : + *expr = i->key; + i->key =NULL ; +expr_free (set); +return 0 ; +default : +break ; + } + } + } + set->set_flags |=NFT_SET_CONSTANT ; set->dtype = ctx->ectx.dtype;
tests/py/any/meta.t +4 -4
@@ -29,7 +29,7 @@ meta l4proto 33-45;ok meta l4proto != 33-45;ok meta l4proto { 33, 55, 67, 88};ok;meta l4proto { 33, 55, 67, 88} meta l4proto != { 33, 55, 67, 88};ok -meta l4proto != { 33-55};ok +meta l4proto != { 33-55};ok;meta l4proto != 33-55 meta priority root;ok meta priority none;ok
@@ -78,7 +78,7 @@ meta iiftype ppp;ok meta oif "lo" accept;ok;oif "lo" accept meta oif != "lo" accept;ok;oif != "lo" accept -meta oif {"lo"} accept;ok;oif {"lo"} accept +meta oif {"lo"} accept;ok;oif "lo" accept meta oifname "dummy0";ok;oifname "dummy0" meta oifname != "dummy0";ok;oifname != "dummy0"
@@ -166,9 +166,9 @@ meta iifgroup != 0;ok;iifgroup != "default" meta iifgroup "default";ok;iifgroup "default" meta iifgroup != "default";ok;iifgroup != "default" meta iifgroup { 11,33};ok;iifgroup { 11,33} -meta iifgroup {11-33};ok;iifgroup {11-33} +meta iifgroup {11-33};ok;iifgroup 11-33 meta iifgroup != { 11,33};ok;iifgroup != { 11,33} -meta iifgroup != {11-33};ok;iifgroup != {11-33} +meta iifgroup != {11-33};ok;iifgroup != 11-33 meta oifgroup 0;ok;oifgroup "default" meta oifgroup != 0;ok;oifgroup != "default" meta oifgroup "default";ok;oifgroup "default"
tests/py/any/meta.t.json +29 -20
100 lines not on this page · raw message
tests/py/any/meta.t.payload +9 -26
@@ -126,13 +126,10 @@ ip test-ip4 input [ lookup reg 1 set __set%d 0x1 ] # meta l4proto != { 33-55} -__set%d test-ip4 7 -__set%d test-ip4 0 - element 00000000 : 1 [end] element 00000021 : 0 [end] element 00000038 : 1 [end] ip test-ip4 input [ meta load l4proto => reg 1 ] [ byteorder reg 1 = hton(reg 1, 2, 1) ] - [ lookup reg 1 set __set%d 0x1 ] + [ range neq reg 1 0x00000021 0x00000037 ] # meta mark 0x4 ip test-ip4 input
@@ -285,12 +282,9 @@ ip test-ip4 input [ immediate reg 0 accept ] # meta oif {"lo"} accept -__set%d test-ip4 3 -__set%d test-ip4 0 - element 00000001 : 0 [end] ip test-ip4 input [ meta load oif => reg 1 ] - [ lookup reg 1 set __set%d ] + [ cmp eq reg 1 0x00000001 ] [ immediate reg 0 accept ] # meta oifname "dummy0"
@@ -652,12 +646,9 @@ ip test-ip4 input [ cmp neq reg 1 0x00000000 ] # meta iifgroup {"default"} -__set%d test-ip4 3 -__set%d test-ip4 0 - element 00000000 : 0 [end] ip test-ip4 input [ meta load iifgroup => reg 1 ] - [ lookup reg 1 set __set%d ] + [ cmp eq reg 1 0x00000000 ] # meta iifgroup { 11,33} __set%d test-ip4 3
@@ -668,13 +659,11 @@ ip test-ip4 input [ lookup reg 1 set __set%d ] # meta iifgroup {11-33} -__set%d test-ip4 7 -__set%d test-ip4 0 - element 00000000 : 1 [end] element 0b000000 : 0 [end] element 22000000 : 1 [end] ip test-ip4 input [ meta load iifgroup => reg 1 ] [ byteorder reg 1 = hton(reg 1, 4, 4) ] - [ lookup reg 1 set __set%d ] + [ cmp gte reg 1 0x0b000000 ] + [ cmp lte reg 1 0x21000000 ] # meta iifgroup != { 11,33} __set%d test-ip4 3
@@ -685,13 +674,10 @@ ip test-ip4 input [ lookup reg 1 set __set%d 0x1 ] # meta iifgroup != {11-33} -__set%d test-ip4 7 -__set%d test-ip4 0 - element 00000000 : 1 [end] element 0b000000 : 0 [end] element 22000000 : 1 [end] -ip test-ip4 input +ip test-ip4 input [ meta load iifgroup => reg 1 ] [ byteorder reg 1 = hton(reg 1, 4, 4) ] - [ lookup reg 1 set __set%d 0x1 ] + [ range neq reg 1 0x0b000000 0x21000000 ] # meta oifgroup 0 ip test-ip4 input
@@ -714,12 +700,9 @@ ip test-ip4 input [ cmp neq reg 1 0x00000000 ] # meta oifgroup != {"default"} -__set%d test-ip4 3 -__set%d test-ip4 0 - element 00000000 : 0 [end] -ip test-ip4 input +ip test-ip4 input [ meta load oifgroup => reg 1 ] - [ lookup reg 1 set __set%d 0x1 ] + [ cmp neq reg 1 0x00000000 ] # meta oifgroup { 11,33} __set%d test-ip4 3
tests/py/ip/igmp.t +2 -4
@@ -16,10 +16,8 @@ igmp checksum 12343;ok igmp checksum != 12343;ok igmp checksum 11-343;ok igmp checksum != 11-343;ok -igmp checksum { 11-343};ok -igmp checksum != { 11-343};ok -igmp checksum { 1111, 222, 343};ok -igmp checksum != { 1111, 222, 343};ok +igmp checksum { 1111, 222, 343, 11-22 };ok +igmp checksum != { 1111, 222, 343, 11-22 };ok igmp mrt 10;ok igmp mrt != 10;ok
tests/py/ip/igmp.t.json +9 -47
82 lines not on this page · raw message
tests/py/ip/igmp.t.payload +8 -68
101 lines not on this page · raw message
tests/shell/testcases/nft-f/0016redefines_1 +1 -1
@@ -17,7 +17,7 @@ table ip x { EXPECTED="table ip x { chain y { ip saddr { 1.1.1.1, 2.2.2.2 } - ip saddr { 3.3.3.3 } + ip saddr 3.3.3.3 } }"
Signature · 1 line
2.21.0
Pablo Neira Ayuso
maintainerQuoted 6 lines · Florian Westphal
convert "ip saddr { 1.1.1.1 }" to "ip saddr 1.1.1.1". Both do the same, but second form is faster since no single-element anon set is created.
Fix up the remaining test cases to expect transformations of the form "meta l4proto { 33-55}" to "meta l4proto 33-55".
Last time we discussed this I think we agreed to spew a warning for this to educate people on this.
My concern is: This is an optimization, are we going to do transparent optimizations of the ruleset? I would like to explore at some point automatic transformations for rulesets, also spot shadowed rules, overlaps, and other sort of inconsistencies.
Are we going to do all that transparently?
Asking this because this is an optimization after all, and I'm not sure I want to step in into making optimizations transparently. Even if this one is fairly trivial.
I also don't like this path because we introduce one more assymmetry between what the user adds a what the user fetches from the kernel.
Thanks.
Florian Westphal
maintainerQuoted 10 lines · Pablo Neira Ayuso
Quoted 6 lines · Florian Westphal
convert "ip saddr { 1.1.1.1 }" to "ip saddr 1.1.1.1". Both do the same, but second form is faster since no single-element anon set is created.
Fix up the remaining test cases to expect transformations of the form "meta l4proto { 33-55}" to "meta l4proto 33-55".
Last time we discussed this I think we agreed to spew a warning for this to educate people on this.
I decided against it.
Why adding a warning? We do not change what the rule does, and we do not collapse different rules into one.
Quoted 6 lines
My concern is: This is an optimization, are we going to do transparent optimizations of the ruleset? I would like to explore at some point automatic transformations for rulesets, also spot shadowed rules, overlaps, and other sort of inconsistencies.
Are we going to do all that transparently?
I think it could be done on a case-by-case basis.
Quoted 6 lines
Asking this because this is an optimization after all, and I'm not sure I want to step in into making optimizations transparently. Even if this one is fairly trivial.
I also don't like this path because we introduce one more assymmetry between what the user adds a what the user fetches from the kernel.
nft add rule ip filter input ip protocol tcp tcp dport 22 -> tcp dport 22
nft add rule inet filter input tcp dport 22 tcp sport 55 -> tcp sport 55 tcp dport 22
I don't think there is any point in warning about this, so why warn about tcp dport { 22 } ?
A warning should, IMO, be reserved for something where we detect a actual problem, e.g.
tcp dport 22 tcp dport 55
and then we could print a 'will never match' warning for nft -f, and maybe even refuse it for "nft add rule .."
Full ruleset optimization should, IMO, be done transparently as well if we're confident such transformations are done correctly.
I would however only do it for nft -f, not for "nft add rule ..."
I see rulesets appearing that are very iptables like, e.g.
tcp dport 22 accept
tcp dport 80 accept
and so on, and I think it makes sense to allow nft to compact this. We could add optimization level support to nft, i.e. only do basic per-rule and do cross-rule optimizations only with a command line parameter.
I think a trivial one such as s/{ 22 }/22/ should just be done automatically.
Pablo Neira Ayuso
maintainerQuoted 9 lines · Florian Westphal
Quoted 8 lines · Pablo Neira Ayuso
Quoted 6 lines · Florian Westphal
convert "ip saddr { 1.1.1.1 }" to "ip saddr 1.1.1.1". Both do the same, but second form is faster since no single-element anon set is created.
Fix up the remaining test cases to expect transformations of the form "meta l4proto { 33-55}" to "meta l4proto 33-55".
Quoted 6 lines
Quoted 2 lines
Last time we discussed this I think we agreed to spew a warning for this to educate people on this.
I decided against it.
Why adding a warning? We do not change what the rule does, and we do not collapse different rules into one.
Yes, there is no semantic change.
Should we also do transparent transformation for
ct state { established,new }I have seen rulesets like this too.
Quoted 8 lines
Quoted 6 lines
My concern is: This is an optimization, are we going to do transparent optimizations of the ruleset? I would like to explore at some point automatic transformations for rulesets, also spot shadowed rules, overlaps, and other sort of inconsistencies.
Are we going to do all that transparently?
I think it could be done on a case-by-case basis.
OK.
Quoted 21 lines
Quoted 6 lines
Asking this because this is an optimization after all, and I'm not sure I want to step in into making optimizations transparently. Even if this one is fairly trivial.
I also don't like this path because we introduce one more assymmetry between what the user adds a what the user fetches from the kernel.
nft add rule ip filter input ip protocol tcp tcp dport 22 -> tcp dport 22
nft add rule inet filter input tcp dport 22 tcp sport 55 -> tcp sport 55 tcp dport 22
I don't think there is any point in warning about this, so why warn about tcp dport { 22 } ?
A warning should, IMO, be reserved for something where we detect a actual problem, e.g.
tcp dport 22 tcp dport 55
Quoted 7 lines
and then we could print a 'will never match' warning for nft -f, and maybe even refuse it for "nft add rule .."
Full ruleset optimization should, IMO, be done transparently as well if we're confident such transformations are done correctly.
I would however only do it for nft -f, not for "nft add rule ..."
Why only for nft -f and not for "nft add rule" ?
Quoted 6 lines
I see rulesets appearing that are very iptables like, e.g.
tcp dport 22 accept
tcp dport 80 acceptand so on, and I think it makes sense to allow nft to compact this.
Yes, we can do transformations for that case too.
Quoted 2 lines
We could add optimization level support to nft, i.e. only do basic per-rule and do cross-rule optimizations only with a command line parameter.
We can add a new parameter to optimize rulesets, we can start with something simple, ie.
* collapse consecutive several rules that come with the same selectors, only values change.
* turn { 22 } into 22.
* turn ct state {new, established } into ct new,established.
Quoted 1 line
I think a trivial one such as s/{ 22 }/22/ should just be done automatically.
We could, yes. But I would prefer if we place all optimizations under some new option and we document them.
I was expecting to have a discussion on this during the NFWS :-).
Pablo Neira Ayuso
maintainerOn Fri, May 24, 2019 at 11:25:06PM +0200, Pablo Neira Ayuso wrote:
[...]
Quoted 9 lines
We can add a new parameter to optimize rulesets, we can start with something simple, ie.
* collapse consecutive several rules that come with the same selectors, only values change.
* turn { 22 } into 22.
* turn ct state {new, established } into ct new,established.
This new optimization option would work both for "nft add rule" and "nft -f", and we can also include a mode that just prints the optimization, similar to iptables-translate. So users can diff their rulesets before and after.
Florian Westphal
maintainerQuoted 17 lines · Pablo Neira Ayuso
Quoted 9 lines · Florian Westphal
Quoted 8 lines · Pablo Neira Ayuso
Quoted 6 lines · Florian Westphal
convert "ip saddr { 1.1.1.1 }" to "ip saddr 1.1.1.1". Both do the same, but second form is faster since no single-element anon set is created.
Fix up the remaining test cases to expect transformations of the form "meta l4proto { 33-55}" to "meta l4proto 33-55".
Quoted 6 lines
Quoted 2 lines
Last time we discussed this I think we agreed to spew a warning for this to educate people on this.
I decided against it.
Why adding a warning? We do not change what the rule does, and we do not collapse different rules into one.
Quoted 7 lines
Yes, there is no semantic change.
Should we also do transparent transformation for
ct state { established,new }I have seen rulesets like this too.
Hmm, to what should this be transformed?
ct state established,new ?
We can do it for this one, established and new can never be set at the same time (or any other state for that matter).
[..]
Quoted 6 lines
Quoted 4 lines
Full ruleset optimization should, IMO, be done transparently as well if we're confident such transformations are done correctly.
I would however only do it for nft -f, not for "nft add rule ..."
Why only for nft -f and not for "nft add rule" ?
I think if you do a 'nft add rule' then you expect to add a rule and not to modify an existing one.
There is also the performance aspect, for 'nft add' you can just build the rule object and pass that to kernel, but for optimizing you need to fetch what is there and perform analysis on it.
I should have be more specific, with "nft -f" i meant a full rule file, not a partial one (esp. one that has a 'flush ruleset'), or something that at least contains a complete subset, e.g. a new chain with n rules.
We could add a command line switch for this and allow optimizing in any case, or add a new 'nft compact' or 'nft optimize' command that would fetch the ruleset and print the 'optimized' version what nft thinks can be coalesced/omitted etc.
What do you think?
Quoted 8 lines
We can add a new parameter to optimize rulesets, we can start with something simple, ie.
* collapse consecutive several rules that come with the same selectors, only values change.
* turn { 22 } into 22.
Quoted 8 lines
* turn ct state {new, established } into ct new,established.
Quoted 1 line
I think a trivial one such as s/{ 22 }/22/ should just be done automatically.
We could, yes. But I would prefer if we place all optimizations under some new option and we document them.
I was expecting to have a discussion on this during the NFWS :-).
Sure, lets do that.