Skip to contentSkip to conversation

evaluate: kill anon sets with one element

PATCH In review Florian Westphal 2 participants 6 messages May 19, 2019 – May 24, 2019 netfilter

lore
Share & tools
lore

Florian Westphal

author maintainer
#0 lore raw

PATCH 1/1 evaluate: kill anon sets with one element

convert "ip saddr { 1.1.1.1 }" to "ip saddr 1.1.1.1". Both do the same, but second form is faster since no single-element anon set is created.

Fix up the remaining test cases to expect transformations of the form "meta l4proto { 33-55}" to "meta l4proto 33-55".

---
 src/evaluate.c                              | 19 ++++++
 tests/py/any/meta.t                         |  8 +--
 tests/py/any/meta.t.json                    | 49 +++++++------
 tests/py/any/meta.t.payload                 | 35 +++-------
 tests/py/ip/igmp.t                          |  6 +-
 tests/py/ip/igmp.t.json                     | 56 +++------------
 tests/py/ip/igmp.t.payload                  | 76 +++------------------
 tests/shell/testcases/nft-f/0016redefines_1 |  2 +-
 8 files changed, 81 insertions(+), 170 deletions(-)
src/evaluate.c +19 -0
index 21d9e146e587..cfdf48c6f5b7 100644
@@ -1253,6 +1253,7 @@ static int expr_evaluate_set_elem(struct eval_ctx *ctx, struct expr **expr)
 static int expr_evaluate_set(struct eval_ctx *ctx, struct expr **expr)
 {
 	struct expr *set = *expr, *i, *next;
+	unsigned int count = 0;
 
 	list_for_each_entry_safe(i, next, &set->expressions, list) {
 		if (list_member_evaluate(ctx, &i) < 0)
@@ -1277,6 +1278,8 @@ static int expr_evaluate_set(struct eval_ctx *ctx, struct expr **expr)
 			return expr_error(ctx->msgs, i,
 					  "Set member is not constant");
 
+		count++;
+
 		if (i->etype == EXPR_SET) {
 			/* Merge recursive set definitions */
 			list_splice_tail_init(&i->expressions, &i->list);
@@ -1288,6 +1291,22 @@ static int expr_evaluate_set(struct eval_ctx *ctx, struct expr **expr)
 			set->set_flags |= NFT_SET_INTERVAL;
 	}
 
+	if (!ctx->set && count == 1) {
+		i = list_first_entry(&set->expressions, struct expr, list);
+		if (i->etype == EXPR_SET_ELEM) {
+			switch (i->key->etype) {
+			case EXPR_RANGE:
+			case EXPR_VALUE:
+				*expr = i->key;
+				i->key = NULL;
+				expr_free(set);
+				return 0;
+			default:
+				break;
+			}
+		}
+	}
+
 	set->set_flags |= NFT_SET_CONSTANT;
 
 	set->dtype = ctx->ectx.dtype;
tests/py/any/meta.t +4 -4
index 3d3ddad94aa3..8aea664794ad 100644
@@ -29,7 +29,7 @@ meta l4proto 33-45;ok
 meta l4proto != 33-45;ok
 meta l4proto { 33, 55, 67, 88};ok;meta l4proto { 33, 55, 67, 88}
 meta l4proto != { 33, 55, 67, 88};ok
-meta l4proto != { 33-55};ok
+meta l4proto != { 33-55};ok;meta l4proto != 33-55
 
 meta priority root;ok
 meta priority none;ok
@@ -78,7 +78,7 @@ meta iiftype ppp;ok
 
 meta oif "lo" accept;ok;oif "lo" accept
 meta oif != "lo" accept;ok;oif != "lo" accept
-meta oif {"lo"} accept;ok;oif {"lo"} accept
+meta oif {"lo"} accept;ok;oif "lo" accept
 
 meta oifname "dummy0";ok;oifname "dummy0"
 meta oifname != "dummy0";ok;oifname != "dummy0"
@@ -166,9 +166,9 @@ meta iifgroup != 0;ok;iifgroup != "default"
 meta iifgroup "default";ok;iifgroup "default"
 meta iifgroup != "default";ok;iifgroup != "default"
 meta iifgroup { 11,33};ok;iifgroup { 11,33}
-meta iifgroup {11-33};ok;iifgroup {11-33}
+meta iifgroup {11-33};ok;iifgroup 11-33
 meta iifgroup != { 11,33};ok;iifgroup != { 11,33}
-meta iifgroup != {11-33};ok;iifgroup != {11-33}
+meta iifgroup != {11-33};ok;iifgroup != 11-33
 meta oifgroup 0;ok;oifgroup "default"
 meta oifgroup != 0;ok;oifgroup != "default"
 meta oifgroup "default";ok;oifgroup "default"
tests/py/any/meta.t.json +29 -20

100 lines not on this page · raw message

tests/py/any/meta.t.payload +9 -26
index 2fb352430663..9733167cf07b 100644
@@ -126,13 +126,10 @@ ip test-ip4 input
   [ lookup reg 1 set __set%d 0x1 ]
 
 # meta l4proto != { 33-55}
-__set%d test-ip4 7
-__set%d test-ip4 0
-	element 00000000  : 1 [end]	element 00000021  : 0 [end]	element 00000038  : 1 [end]
 ip test-ip4 input
   [ meta load l4proto => reg 1 ]
   [ byteorder reg 1 = hton(reg 1, 2, 1) ]
-  [ lookup reg 1 set __set%d 0x1 ]
+  [ range neq reg 1 0x00000021 0x00000037 ]
 
 # meta mark 0x4
 ip test-ip4 input
@@ -285,12 +282,9 @@ ip test-ip4 input
   [ immediate reg 0 accept ]
 
 # meta oif {"lo"} accept
-__set%d test-ip4 3
-__set%d test-ip4 0
-	element 00000001  : 0 [end]
 ip test-ip4 input
   [ meta load oif => reg 1 ]
-  [ lookup reg 1 set __set%d ]
+  [ cmp eq reg 1 0x00000001 ]
   [ immediate reg 0 accept ]
 
 # meta oifname "dummy0"
@@ -652,12 +646,9 @@ ip test-ip4 input
   [ cmp neq reg 1 0x00000000 ]
 
 # meta iifgroup {"default"}
-__set%d test-ip4 3
-__set%d test-ip4 0
-	element 00000000  : 0 [end]
 ip test-ip4 input
   [ meta load iifgroup => reg 1 ]
-  [ lookup reg 1 set __set%d ]
+  [ cmp eq reg 1 0x00000000 ]
 
 # meta iifgroup { 11,33}
 __set%d test-ip4 3
@@ -668,13 +659,11 @@ ip test-ip4 input
   [ lookup reg 1 set __set%d ]
 
 # meta iifgroup {11-33}
-__set%d test-ip4 7
-__set%d test-ip4 0
-	element 00000000  : 1 [end]	element 0b000000  : 0 [end]	element 22000000  : 1 [end]
 ip test-ip4 input
   [ meta load iifgroup => reg 1 ]
   [ byteorder reg 1 = hton(reg 1, 4, 4) ]
-  [ lookup reg 1 set __set%d ]
+  [ cmp gte reg 1 0x0b000000 ]
+  [ cmp lte reg 1 0x21000000 ]
 
 # meta iifgroup != { 11,33}
 __set%d test-ip4 3
@@ -685,13 +674,10 @@ ip test-ip4 input
   [ lookup reg 1 set __set%d 0x1 ]
 
 # meta iifgroup != {11-33}
-__set%d test-ip4 7
-__set%d test-ip4 0
-	element 00000000  : 1 [end]	element 0b000000  : 0 [end]	element 22000000  : 1 [end]
-ip test-ip4 input
+ip test-ip4 input 
   [ meta load iifgroup => reg 1 ]
   [ byteorder reg 1 = hton(reg 1, 4, 4) ]
-  [ lookup reg 1 set __set%d 0x1 ]
+  [ range neq reg 1 0x0b000000 0x21000000 ]
 
 # meta oifgroup 0
 ip test-ip4 input
@@ -714,12 +700,9 @@ ip test-ip4 input
   [ cmp neq reg 1 0x00000000 ]
 
 # meta oifgroup != {"default"}
-__set%d test-ip4 3
-__set%d test-ip4 0
-	element 00000000  : 0 [end]
-ip test-ip4 input
+ip test-ip4 input 
   [ meta load oifgroup => reg 1 ]
-  [ lookup reg 1 set __set%d 0x1 ]
+  [ cmp neq reg 1 0x00000000 ]
 
 # meta oifgroup { 11,33}
 __set%d test-ip4 3
tests/py/ip/igmp.t +2 -4
index 939dcc32b248..7a39348a7f84 100644
@@ -16,10 +16,8 @@ igmp checksum 12343;ok
 igmp checksum != 12343;ok
 igmp checksum 11-343;ok
 igmp checksum != 11-343;ok
-igmp checksum { 11-343};ok
-igmp checksum != { 11-343};ok
-igmp checksum { 1111, 222, 343};ok
-igmp checksum != { 1111, 222, 343};ok
+igmp checksum { 1111, 222, 343, 11-22 };ok
+igmp checksum != { 1111, 222, 343, 11-22 };ok
 
 igmp mrt 10;ok
 igmp mrt != 10;ok
tests/py/ip/igmp.t.json +9 -47

82 lines not on this page · raw message

tests/py/ip/igmp.t.payload +8 -68

101 lines not on this page · raw message

tests/shell/testcases/nft-f/0016redefines_1 +1 -1
index d0148d65da54..a6e857bd13c0 100755
@@ -17,7 +17,7 @@ table ip x {
 EXPECTED="table ip x {
 	chain y {
 		ip saddr { 1.1.1.1, 2.2.2.2 }
-		ip saddr { 3.3.3.3 }
+		ip saddr 3.3.3.3
 	}
 }"
 
Signature · 1 line
2.21.0
#1 lore raw
Quoted 6 lines · Florian Westphal

convert "ip saddr { 1.1.1.1 }" to "ip saddr 1.1.1.1". Both do the same, but second form is faster since no single-element anon set is created.

Fix up the remaining test cases to expect transformations of the form "meta l4proto { 33-55}" to "meta l4proto 33-55".

Last time we discussed this I think we agreed to spew a warning for this to educate people on this.

My concern is: This is an optimization, are we going to do transparent optimizations of the ruleset? I would like to explore at some point automatic transformations for rulesets, also spot shadowed rules, overlaps, and other sort of inconsistencies.

Are we going to do all that transparently?

Asking this because this is an optimization after all, and I'm not sure I want to step in into making optimizations transparently. Even if this one is fairly trivial.

I also don't like this path because we introduce one more assymmetry between what the user adds a what the user fetches from the kernel.

Thanks.

author maintainer
#2 lore raw
Quoted 10 lines · Pablo Neira Ayuso
Quoted 6 lines · Florian Westphal

convert "ip saddr { 1.1.1.1 }" to "ip saddr 1.1.1.1". Both do the same, but second form is faster since no single-element anon set is created.

Fix up the remaining test cases to expect transformations of the form "meta l4proto { 33-55}" to "meta l4proto 33-55".

Last time we discussed this I think we agreed to spew a warning for this to educate people on this.

I decided against it.
Why adding a warning? We do not change what the rule does, and we do not collapse different rules into one.

Quoted 6 lines

My concern is: This is an optimization, are we going to do transparent optimizations of the ruleset? I would like to explore at some point automatic transformations for rulesets, also spot shadowed rules, overlaps, and other sort of inconsistencies.

Are we going to do all that transparently?

I think it could be done on a case-by-case basis.

Quoted 6 lines

Asking this because this is an optimization after all, and I'm not sure I want to step in into making optimizations transparently. Even if this one is fairly trivial.

I also don't like this path because we introduce one more assymmetry between what the user adds a what the user fetches from the kernel.

nft add rule ip filter input ip protocol tcp tcp dport 22 -> tcp dport 22

nft add rule inet filter input tcp dport 22 tcp sport 55 -> tcp sport 55 tcp dport 22

I don't think there is any point in warning about this, so why warn about tcp dport { 22 } ?

A warning should, IMO, be reserved for something where we detect a actual problem, e.g.

tcp dport 22 tcp dport 55

and then we could print a 'will never match' warning for nft -f, and maybe even refuse it for "nft add rule .."

Full ruleset optimization should, IMO, be done transparently as well if we're confident such transformations are done correctly.

I would however only do it for nft -f, not for "nft add rule ..."

I see rulesets appearing that are very iptables like, e.g.

tcp dport 22 accept
tcp dport 80 accept

and so on, and I think it makes sense to allow nft to compact this. We could add optimization level support to nft, i.e. only do basic per-rule and do cross-rule optimizations only with a command line parameter.

I think a trivial one such as s/{ 22 }/22/ should just be done automatically.

#3 lore raw
Quoted 9 lines · Florian Westphal
Quoted 8 lines · Pablo Neira Ayuso
Quoted 6 lines · Florian Westphal

convert "ip saddr { 1.1.1.1 }" to "ip saddr 1.1.1.1". Both do the same, but second form is faster since no single-element anon set is created.

Fix up the remaining test cases to expect transformations of the form "meta l4proto { 33-55}" to "meta l4proto 33-55".

Quoted 6 lines
Quoted 2 lines

Last time we discussed this I think we agreed to spew a warning for this to educate people on this.

I decided against it.
Why adding a warning? We do not change what the rule does, and we do not collapse different rules into one.

Yes, there is no semantic change.

Should we also do transparent transformation for

        ct state { established,new }

I have seen rulesets like this too.

Quoted 8 lines
Quoted 6 lines

My concern is: This is an optimization, are we going to do transparent optimizations of the ruleset? I would like to explore at some point automatic transformations for rulesets, also spot shadowed rules, overlaps, and other sort of inconsistencies.

Are we going to do all that transparently?

I think it could be done on a case-by-case basis.

OK.

Quoted 21 lines
Quoted 6 lines

Asking this because this is an optimization after all, and I'm not sure I want to step in into making optimizations transparently. Even if this one is fairly trivial.

I also don't like this path because we introduce one more assymmetry between what the user adds a what the user fetches from the kernel.

nft add rule ip filter input ip protocol tcp tcp dport 22 -> tcp dport 22

nft add rule inet filter input tcp dport 22 tcp sport 55 -> tcp sport 55 tcp dport 22

I don't think there is any point in warning about this, so why warn about tcp dport { 22 } ?

A warning should, IMO, be reserved for something where we detect a actual problem, e.g.

tcp dport 22 tcp dport 55

Quoted 7 lines

and then we could print a 'will never match' warning for nft -f, and maybe even refuse it for "nft add rule .."

Full ruleset optimization should, IMO, be done transparently as well if we're confident such transformations are done correctly.

I would however only do it for nft -f, not for "nft add rule ..."

Why only for nft -f and not for "nft add rule" ?

Quoted 6 lines

I see rulesets appearing that are very iptables like, e.g.

tcp dport 22 accept
tcp dport 80 accept

and so on, and I think it makes sense to allow nft to compact this.

Yes, we can do transformations for that case too.

Quoted 2 lines

We could add optimization level support to nft, i.e. only do basic per-rule and do cross-rule optimizations only with a command line parameter.

We can add a new parameter to optimize rulesets, we can start with something simple, ie.

* collapse consecutive several rules that come with the same selectors, only values change.

* turn { 22 } into 22.

* turn ct state {new, established } into ct new,established.

Quoted 1 line

I think a trivial one such as s/{ 22 }/22/ should just be done automatically.

We could, yes. But I would prefer if we place all optimizations under some new option and we document them.

I was expecting to have a discussion on this during the NFWS :-).

#4 lore raw

On Fri, May 24, 2019 at 11:25:06PM +0200, Pablo Neira Ayuso wrote:
[...]

Quoted 9 lines

We can add a new parameter to optimize rulesets, we can start with something simple, ie.

* collapse consecutive several rules that come with the same selectors, only values change.

* turn { 22 } into 22.

* turn ct state {new, established } into ct new,established.

This new optimization option would work both for "nft add rule" and "nft -f", and we can also include a mode that just prints the optimization, similar to iptables-translate. So users can diff their rulesets before and after.

author maintainer
#5 lore raw
In reply to Pablo Neira Ayuso
Quoted 17 lines · Pablo Neira Ayuso
Quoted 9 lines · Florian Westphal
Quoted 8 lines · Pablo Neira Ayuso
Quoted 6 lines · Florian Westphal

convert "ip saddr { 1.1.1.1 }" to "ip saddr 1.1.1.1". Both do the same, but second form is faster since no single-element anon set is created.

Fix up the remaining test cases to expect transformations of the form "meta l4proto { 33-55}" to "meta l4proto 33-55".

Quoted 6 lines
Quoted 2 lines

Last time we discussed this I think we agreed to spew a warning for this to educate people on this.

I decided against it.
Why adding a warning? We do not change what the rule does, and we do not collapse different rules into one.

Quoted 7 lines

Yes, there is no semantic change.

Should we also do transparent transformation for

        ct state { established,new }

I have seen rulesets like this too.

Hmm, to what should this be transformed?
ct state established,new ?

We can do it for this one, established and new can never be set at the same time (or any other state for that matter).

[..]

Quoted 6 lines
Quoted 4 lines

Full ruleset optimization should, IMO, be done transparently as well if we're confident such transformations are done correctly.

I would however only do it for nft -f, not for "nft add rule ..."

Why only for nft -f and not for "nft add rule" ?

I think if you do a 'nft add rule' then you expect to add a rule and not to modify an existing one.

There is also the performance aspect, for 'nft add' you can just build the rule object and pass that to kernel, but for optimizing you need to fetch what is there and perform analysis on it.

I should have be more specific, with "nft -f" i meant a full rule file, not a partial one (esp. one that has a 'flush ruleset'), or something that at least contains a complete subset, e.g. a new chain with n rules.

We could add a command line switch for this and allow optimizing in any case, or add a new 'nft compact' or 'nft optimize' command that would fetch the ruleset and print the 'optimized' version what nft thinks can be coalesced/omitted etc.

What do you think?

Quoted 8 lines

We can add a new parameter to optimize rulesets, we can start with something simple, ie.

* collapse consecutive several rules that come with the same selectors, only values change.

* turn { 22 } into 22.

Quoted 8 lines

* turn ct state {new, established } into ct new,established.

Quoted 1 line

I think a trivial one such as s/{ 22 }/22/ should just be done automatically.

We could, yes. But I would prefer if we place all optimizations under some new option and we document them.

I was expecting to have a discussion on this during the NFWS :-).

Sure, lets do that.