Skip to contentSkip to conversation

tests: Automated regression testing

PATCH Superseded Ana Rey 3 participants 9 messages Sep 15, 2014 – Sep 18, 2014 netfilter

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore

Ana Rey

author
#0 lore raw

v2 cover tests: Automated regression testing

This is a new version of the automated regression testing of nftables.

There is no infrastructure to allow us to check all options/features in nft. So, if anyone sends a patch, we cannot check if it breaks something.

I send in this patchset the nftables automated regression tests. It contains a python script (nft-add.py) and a set of test files.

This allow us to check the input of rules of nft-tool from the command-line and the output from nft-tool of this rule. Then, it compares if the rule input matches the rule output automatically.

We also have plans to add automated regression testing in the packet path in the future, which should come in a follow up step.

Comments welcome, thanks

[Changes in v2]

* More readable variables and functions names.
* Consistent errors and warning messages.
* I move the script and tests files into tests/regression folder.
* I add a variable that specifies the folders that contains the test files. The script only looks for test files in these folders.
* I split several functions in smaller chunks for maintenability reasons.
* I use "-nnn" option to nft list table.
* I add signal handlers to catch when the user stops the test.

Ana Rey (6):
tests: Add Automated regression testing
tests: Add ip6 folder with test files.
tests: Add inet folder with test files.
tests: Add arp folder with test files.
tests: Add bridge folder with test files.
tests: Add any folder with test files.

 tests/regression/any/ct.t        |  105 +++++
 tests/regression/any/frag.t      |   64 +++
 tests/regression/any/limit.t     |   12 +
 tests/regression/any/log.t       |   27 ++
 tests/regression/any/meta.t      |  160 ++++++++
 tests/regression/any/queue.t     |   15 +
 tests/regression/arp/arp.t       |   53 +++
 tests/regression/arp/chains.t    |    5 +
 tests/regression/bridge/chains.t |    7 +
 tests/regression/inet/ah.t       |   58 +++
 tests/regression/inet/comp.t     |   31 ++
 tests/regression/inet/dccp.t     |   31 ++
 tests/regression/inet/esp.t      |   23 ++
 tests/regression/inet/sctp.t     |   42 ++
 tests/regression/inet/tcp.t      |  103 +++++
 tests/regression/inet/udp.t      |   49 +++
 tests/regression/inet/udplite.t  |   42 ++
 tests/regression/ip/chains.t     |   22 ++
 tests/regression/ip/icmp.t       |   93 +++++
 tests/regression/ip/ip.t         |  107 +++++
 tests/regression/ip/nat.t        |   18 +
 tests/regression/ip/reject.t     |    5 +
 tests/regression/ip/sets.t       |   31 ++
 tests/regression/ip6/chains.t    |   18 +
 tests/regression/ip6/dst.t       |   25 ++
 tests/regression/ip6/hbh.t       |   25 ++
 tests/regression/ip6/icmpv6.t    |   96 +++++
 tests/regression/ip6/ip6.t       |  142 +++++++
 tests/regression/ip6/mh.t        |   49 +++
 tests/regression/ip6/nat.t       |    6 +
 tests/regression/ip6/reject.t    |    5 +
 tests/regression/ip6/rt.t        |   45 +++
 tests/regression/ip6/sets.t      |   22 ++
 tests/regression/ip6/vmap.t      |   54 +++
 tests/regression/nft-test.py     |  805 ++++++++++++++++++++++++++++++++++++++
 35 files changed, 2395 insertions(+)
 create mode 100644 tests/regression/any/ct.t
 create mode 100644 tests/regression/any/frag.t
 create mode 100644 tests/regression/any/limit.t
 create mode 100644 tests/regression/any/log.t
 create mode 100644 tests/regression/any/meta.t
 create mode 100644 tests/regression/any/queue.t
 create mode 100644 tests/regression/arp/arp.t
 create mode 100644 tests/regression/arp/chains.t
 create mode 100644 tests/regression/bridge/chains.t
 create mode 100644 tests/regression/inet/ah.t
 create mode 100644 tests/regression/inet/comp.t
 create mode 100644 tests/regression/inet/dccp.t
 create mode 100644 tests/regression/inet/esp.t
 create mode 100644 tests/regression/inet/sctp.t
 create mode 100644 tests/regression/inet/tcp.t
 create mode 100644 tests/regression/inet/udp.t
 create mode 100644 tests/regression/inet/udplite.t
 create mode 100644 tests/regression/ip/chains.t
 create mode 100644 tests/regression/ip/icmp.t
 create mode 100644 tests/regression/ip/ip.t
 create mode 100644 tests/regression/ip/nat.t
 create mode 100644 tests/regression/ip/reject.t
 create mode 100644 tests/regression/ip/sets.t
 create mode 100644 tests/regression/ip6/chains.t
 create mode 100644 tests/regression/ip6/dst.t
 create mode 100644 tests/regression/ip6/hbh.t
 create mode 100644 tests/regression/ip6/icmpv6.t
 create mode 100644 tests/regression/ip6/ip6.t
 create mode 100644 tests/regression/ip6/mh.t
 create mode 100644 tests/regression/ip6/nat.t
 create mode 100644 tests/regression/ip6/reject.t
 create mode 100644 tests/regression/ip6/rt.t
 create mode 100644 tests/regression/ip6/sets.t
 create mode 100644 tests/regression/ip6/vmap.t
 create mode 100755 tests/regression/nft-test.py
Signature · 1 line
1.7.10.4
author
#1 lore raw

PATCH v2 1/6 tests: Add Automated regression testing

Here, the automated regression testing to nftables and some test files.

This is a python script to check the command-line in nft.

This script checks the input of rules of nft-tool from the command-line and the output of them to the command-line. A bit more details here below.

A) What is it checking?

This script tests two different paths:

1) The input of rules of nft-tool from the command-line. It checks the different steps from the command-line to the kernel: parse step, evaluate step, compile step, the generate of netlink message and after this is sent into the kernel.

2) The output that is obtained from the kernel. It checks the different steps from the kernel to the command-line: getting the netlink message, the parse step, the postproces step, the textify step and the listing the rule in the command-line.

As a last step, It compares the rule is added and rule is listed by nft.

B) What options are available?

The script offers the following options:

1) Execute all set of test files (or one test file):

./nft-test.sh => Run all test files
./nft-test.sh path/file.t => Run this test file

So, It tests the input of rules of nft-tool from the command-line and then, It checks if the rule is added correctly.

If there is a problem, It lists the differences between the rule is added and the rule is listed by nft.

(If there are more than one family of table indicated in the test file and there is an error or a warning in this execution of the rule, the execution of this rule stop and it does not run in the others families of the tables).

2) List all rules are added in nft-tool while this script is run. (It is similar a debug mode of this test.)

./nft-test.sh -d
./nft-test.sh -d path/file.t

3) Run rules that they need to fix: This mode runs the lines of a file that starts with a "-" symbol (these rules only). ./nft-test.sh -e
./nft-test.sh -e path/file.t

4) Keep testing all families on error. Run all rules in all families of the tables defined in the test file. (although there were an error or a warning in a previous families.)
./nft-test.sh -f
./nft-test.sh -f path/file.t

C) What is the structure of the test file?

A test file contains a set of rules that are added in the system.

Here, an example of a test file:

*ip;test-ipv4                               # line 1
*ip6;test-ipv6                              # line 2
*inet;test-inet                             # line 3

:input;type filter hook input priority 0 # line 4

ah hdrlength != 11-23;ok;ah hdrlength < 11 ah hdrlength > 23 # line 5
- tcp dport != {22-25} # line 6

!set1 ipv4_addr;ok                          # line 7
?set1 192.168.3.8 192.168.3.9;ok            # line 8
 # This is a commented-line.                # line 9

1) Tables:
# Line 1: it defines a table where chains and rules are added.
It defines a table. the name of the table is test-ip and the family is ip.
In line 2 and 3, It define more tables of different families (ip6 and inet). It's possible to add different type of tables.

2) Chains:
# Line 4: It defines the chain/s (and the type, hook and priority of
this chain) where rules are added. The name of this chain is "input". The type is "filter", the hook is "input" and the priority is 0.

3) Rules:
line: 4: This line is divided by a ";" character.
Part 1: "ah hdrlength != 11-23" is the rule to check. Part 2: "ok" is the result expected with the execute of this rule.
(This rule is added without errors.)
Part 3: "ah hdrlength < 11 ah hdrlength > 23". This is the look of
the rule if it is run in the command-line. If the look of the output rule is the same that the rule in the input, this part is omit.

4) Marked-line:
Line 6: This is a marked-line. It means this rule is not run in a general execution of this script.
If if want to execute this line, It's necessary run this script with "-r" option.

It's useful to mark a known bugs or lines that don't want to execute.

5) Named set:
Line 7: It adds a new set. The name of this set is "set1" and the type of this set is "ipv4_add"
Line 8: It adds two element into the set1 set: "192.168.3.8" and "192.168.3.9" A whitespace divide the diferent elements of the set.

The Anonymous sets is added as a normal rule. It doesn't an especial handling.

6) Comments:
Line 9: "#" symbol means that line is a comment about the test.

D) The test folders
The test files are divide in directory: ip, ip6, inet, arp, bridge and any folders:

* "ip" folder: Here are the test files are executed in ip and inet
table.
* "ip" folder: Here are the test files are executed in ip6 and inet
table.
* "inet" folder: Here are the test files are executed in ip, ip6 and
inet table.
* "arp" folder: Here are the test files are executed in arp tables.
"bridge" folder: Here are the test files are executed in bridge table.
* "any" folder: Here are the test files are executed in ip, ip6, inet,
arp and bridge tables.

Moreover, It adds the "ip4" folder with expecific test files for ip and inet tables.

---
 tests/regression/ip/chains.t |   22 ++
 tests/regression/ip/icmp.t   |   93 +++++
 tests/regression/ip/ip.t     |  107 ++++++
 tests/regression/ip/nat.t    |   18 +
 tests/regression/ip/reject.t |    5 +
 tests/regression/ip/sets.t   |   31 ++
 tests/regression/nft-test.py |  805 ++++++++++++++++++++++++++++++++++++++++++
 7 files changed, 1081 insertions(+)
 create mode 100644 tests/regression/ip/chains.t
 create mode 100644 tests/regression/ip/icmp.t
 create mode 100644 tests/regression/ip/ip.t
 create mode 100644 tests/regression/ip/nat.t
 create mode 100644 tests/regression/ip/reject.t
 create mode 100644 tests/regression/ip/sets.t
 create mode 100755 tests/regression/nft-test.py
tests/regression/ip/chains.t new file +22 -0
new file mode 100644
index 0000000..92a1147
@@ -0,0 +1,22 @@
+*ip;test-ip4
+-*inet;test-inet
+
+# filter chains available are: input, output, forward, prerouting, postrouting
+:filter-input;type filter hook input priority 0
+:filter-pre;type filter hook prerouting priority 0
+:filter-forw;type filter hook forward priority 0
+:filter-out;type filter hook output priority 0
+:filter-post;type filter hook postrouting priority 0
+# nat chains available are: input, output, prerouting, postrouting
+:nat-input-t;type nat hook input priority 0
+:nat-pre-t;type nat hook prerouting priority 0
+:nat-out-t;type nat hook output priority 0
+:nat-post-t;type nat hook postrouting priority 0
+# route chain available are: output
+:route-out-t;type route hook output priority 0
+
+#ip daddr 192.168.0.1-192.168.0.250;ok
+#ip daddr 192.168.0.1;ok
+#ip daddr 192.168.0.1 drop;ok
+#ip daddr 192.168.0.2 log;ok
+#ip daddr 192.168.0.2 log;ok
tests/regression/ip/icmp.t new file +93 -0

93 lines not on this page · raw message

tests/regression/ip/ip.t new file +107 -0

107 lines not on this page · raw message

tests/regression/ip/nat.t new file +18 -0
new file mode 100644
index 0000000..23e0bce
@@ -0,0 +1,18 @@
+*ip;test-ip4
+# bug: Nat tables is not supported yet in inet table.
+-*inet;test-inet
+
+:output;type nat hook output priority 0
+
+iifname eth0 tcp dport 80-90 dnat 192.168.3.2;ok
+iifname eth0 tcp dport != 80-90 dnat 192.168.3.2;ok
+iifname eth0 tcp dport {80, 90, 23} dnat 192.168.3.2;ok
+- iifname eth0 tcp dport != {80, 90, 23} dnat 192.168.3.2;ok
+
+iifname eth0 tcp sport 23-34 snat 192.168.3.2;ok
+
+- iifname eth0 tcp dport != {80, 90, 23} dnat 192.168.3.2;ok
+# BUG: invalid expression type set
+# nft: src/evaluate.c:975: expr_evaluate_relational: Assertion '0' failed.
+
+iifname eth0 tcp dport != 23-34 dnat 192.168.3.2;ok
tests/regression/ip/reject.t new file +5 -0
new file mode 100644
index 0000000..e7fb15b
@@ -0,0 +1,5 @@
+*ip;test-ip4
+*ip;test-inet
+:output;type filter hook output priority 0
+
+reject;ok
tests/regression/ip/sets.t new file +31 -0
new file mode 100644
index 0000000..a74d308
@@ -0,0 +1,31 @@
+*ip;test-ip4
+*inet;test-inet
+:input;type filter hook input priority 0
+
+!set_ipv4_add ipv4_addr;ok
+!set_inet inet_proto;ok
+!set_inet_serv inet_service;ok
+!set_time time;ok
+
+!set1 ipv4_addr;ok
+?set1 192.168.3.4;ok
+
+?set1 192.168.3.4;fail
+?set1 192.168.3.5 192.168.3.6;ok
+?set1 192.168.3.5 192.168.3.6;fail
+?set1 192.168.3.8 192.168.3.9;ok
+?set1 192.168.3.10 192.168.3.11;ok
+?set1 1234:1234:1234:1234:1234:1234:1234:1234;fail
+?set2 192.168.3.4;fail
+
+!set2 ipv4_addr;ok
+?set2 192.168.3.4;ok
+?set2 192.168.3.5 192.168.3.6;ok
+?set2 192.168.3.5 192.168.3.6;fail
+?set2 192.168.3.8 192.168.3.9;ok
+?set2 192.168.3.10 192.168.3.11;ok
+
+-ip saddr @set1 drop;ok
+-ip saddr @set2 drop;ok
+-ip saddr @set33 drop;fail
+-ip saddr @set21 drop;fail
tests/regression/nft-test.py new file +805 -0

805 lines not on this page · raw message

Signature · 1 line
1.7.10.4
author
#2 lore raw

PATCH v2 2/6 tests: Add ip6 folder with test files.

In reply to Ana Rey

"ip6" folder contains the test files that are executed in ip6 and inet family of tables.

These test files are executed with nft-tests.py

---
 tests/regression/ip6/chains.t |   18 ++++++
 tests/regression/ip6/dst.t    |   25 ++++++++
 tests/regression/ip6/hbh.t    |   25 ++++++++
 tests/regression/ip6/icmpv6.t |   96 ++++++++++++++++++++++++++++
 tests/regression/ip6/ip6.t    |  142 +++++++++++++++++++++++++++++++++++++++++
 tests/regression/ip6/mh.t     |   49 ++++++++++++++
 tests/regression/ip6/nat.t    |    6 ++
 tests/regression/ip6/reject.t |    5 ++
 tests/regression/ip6/rt.t     |   45 +++++++++++++
 tests/regression/ip6/sets.t   |   22 +++++++
 tests/regression/ip6/vmap.t   |   54 ++++++++++++++++
 11 files changed, 487 insertions(+)
 create mode 100644 tests/regression/ip6/chains.t
 create mode 100644 tests/regression/ip6/dst.t
 create mode 100644 tests/regression/ip6/hbh.t
 create mode 100644 tests/regression/ip6/icmpv6.t
 create mode 100644 tests/regression/ip6/ip6.t
 create mode 100644 tests/regression/ip6/mh.t
 create mode 100644 tests/regression/ip6/nat.t
 create mode 100644 tests/regression/ip6/reject.t
 create mode 100644 tests/regression/ip6/rt.t
 create mode 100644 tests/regression/ip6/sets.t
 create mode 100644 tests/regression/ip6/vmap.t
tests/regression/ip6/chains.t new file +18 -0
new file mode 100644
index 0000000..ef975b2
@@ -0,0 +1,18 @@
+*ip6;test-ip6
+-*inet;test-inet
+
+# filter chains available are: input, output, forward, forward, prerouting and postrouting.
+:filter-input;type filter hook input priority 0
+:filter-prer;type filter hook prerouting priority 0
+:filter-forw-t;type filter hook forward priority 0
+:filter-out-t;type filter hook output priority 0
+:filter-post-t;type filter hook postrouting priority 0
+
+# nat chains available are: input, output, forward, prerouting and postrouting.
+:nat-input;type nat hook input priority 0
+:nat-prerouting;type nat hook prerouting priority 0
+:nat-output;type nat hook output priority 0
+:nat-postrou;type nat hook postrouting priority 0
+
+# route chain available is output.
+:route-out;type route hook output priority 0
tests/regression/ip6/dst.t new file +25 -0
new file mode 100644
index 0000000..1b1bc52
@@ -0,0 +1,25 @@
+*ip6;test-ip6
+*inet;test-inet
+:input;type filter hook input priority 0
+
+dst nexthdr 22;ok
+dst nexthdr != 233;ok
+dst nexthdr 33-45;ok;dst nexthdr >= 33 dst nexthdr <= 45
+dst nexthdr != 33-45;ok;dst nexthdr < 33 dst nexthdr > 45
+dst nexthdr { 33, 55, 67, 88};ok
+- dst nexthdr != { 33, 55, 67, 88};ok
+dst nexthdr { 33-55};ok
+- dst nexthdr != { 33-55};ok
+dst nexthdr { udplite, ipcomp, udp, ah, sctp, esp, dccp, tcp, ipv6-icmp};ok;dst nexthdr { 51, 50, 17, 136, 58, 6, 33, 132, 108}
+- dst nexthdr != { udplite, ipcomp, udp, ah, sctp, esp, dccp, tcp, ipv6-icmp};ok
+dst nexthdr icmp;ok;dst nexthdr 1
+dst nexthdr != icmp;ok;dst nexthdr != 1
+
+dst hdrlength 22;ok
+dst hdrlength != 233;ok
+dst hdrlength 33-45;ok;dst hdrlength >= 33 dst hdrlength <= 45
+dst hdrlength != 33-45;ok;dst hdrlength < 33 dst hdrlength > 45
+dst hdrlength { 33, 55, 67, 88};ok
+- dst hdrlength != { 33, 55, 67, 88};ok
+dst hdrlength { 33-55};ok
+- dst hdrlength != { 33-55};ok
tests/regression/ip6/hbh.t new file +25 -0
new file mode 100644
index 0000000..b274b8b
@@ -0,0 +1,25 @@
+*ip6;test-ip6
+*inet;test-inet
+:filter-input;type filter hook input priority 0
+
+hbh hdrlength 22;ok
+hbh hdrlength != 233;ok
+hbh hdrlength 33-45;ok;hbh hdrlength >= 33 hbh hdrlength <= 45
+hbh hdrlength != 33-45;ok;hbh hdrlength < 33 hbh hdrlength > 45
+hbh hdrlength {33, 55, 67, 88};ok
+- hbh hdrlength != {33, 55, 67, 88};ok
+hbh hdrlength { 33-55};ok
+- hbh hdrlength != {33-55};ok
+
+hbh nexthdr {esp, ah, comp, udp, udplite, tcp, dccp, sctp, icmpv6};ok;hbh nexthdr { 58, 136, 51, 50, 6, 17, 132, 33, 108}
+- hbh nexthdr != {esp, ah, comp, udp, udplite, tcp, dccp, sctp, icmpv6};ok
+hbh nexthdr 22;ok
+hbh nexthdr != 233;ok
+hbh nexthdr 33-45;ok;hbh nexthdr >= 33 hbh nexthdr <= 45
+hbh nexthdr != 33-45;ok;hbh nexthdr < 33 hbh nexthdr > 45
+hbh nexthdr {33, 55, 67, 88};ok
+- hbh nexthdr != {33, 55, 67, 88};ok
+hbh nexthdr { 33-55};ok
+- hbh nexthdr != {33-55};ok
+hbh nexthdr ip;ok;hbh nexthdr 0
+hbh nexthdr != ip;ok;hbh nexthdr != 0
tests/regression/ip6/icmpv6.t new file +96 -0

96 lines not on this page · raw message

tests/regression/ip6/ip6.t new file +142 -0

142 lines not on this page · raw message

tests/regression/ip6/mh.t new file +49 -0

49 lines not on this page · raw message

tests/regression/ip6/nat.t new file +6 -0
new file mode 100644
index 0000000..2fb4ac8
@@ -0,0 +1,6 @@
+*ip6;test-ip6
+- *inet;test-inet
+:input;type nat hook input priority 0
+
+tcp dport 80-90 dnat 2001:838:35f:1::-2001:838:35f:2:: :80-100;ok
+tcp dport 80-90 dnat 2001:838:35f:1::-2001:838:35f:2:: :100;ok
tests/regression/ip6/reject.t new file +5 -0
new file mode 100644
index 0000000..b49c50b
@@ -0,0 +1,5 @@
+*ip6;test-ip6
+*inet;test-inet
+:output;type filter hook output priority 0
+
+reject;ok
tests/regression/ip6/rt.t new file +45 -0

45 lines not on this page · raw message

tests/regression/ip6/sets.t new file +22 -0
new file mode 100644
index 0000000..4938929
@@ -0,0 +1,22 @@
+*ip6;test-ip6
+*inet;test-inet
+:input;type filter hook input priority 0
+
+!set_ipv6_add1 ipv6_addr;ok
+!set_inet1 inet_proto;ok
+!set_inet inet_service;ok
+!set_time time;ok
+
+?set2 192.168.3.4;fail
+!set2 ipv6_addr;ok
+?set2 1234:1234::1234:1234:1234:1234:1234;ok
+?set2 1234:1234::1234:1234:1234:1234:1234;fail
+?set2 1234::1234:1234:1234;ok
+?set2 1234:1234:1234:1234:1234::1234:1234 1234:1234::123;ok
+?set2 192.168.3.8 192.168.3.9;fail
+?set2 1234:1234::1234:1234:1234:1234;ok
+?set2 1234:1234::1234:1234:1234:1234;fail
+?set2 1234:1234:1234::1234;ok
+
+ip6 saddr @set2 drop;ok
+ip6 saddr @set33 drop;fail
tests/regression/ip6/vmap.t new file +54 -0

54 lines not on this page · raw message

Signature · 1 line
1.7.10.4
author
#3 lore raw

PATCH v2 3/6 tests: Add inet folder with test files.

In reply to Ana Rey

"inet" folder contains the test files that are executed in ipv4, ipv6 and inet family of tables.

These test files are executed with nft-tests.py

---
 tests/regression/inet/ah.t      |   58 ++++++++++++++++++++++
 tests/regression/inet/comp.t    |   31 ++++++++++++
 tests/regression/inet/dccp.t    |   31 ++++++++++++
 tests/regression/inet/esp.t     |   23 +++++++++
 tests/regression/inet/sctp.t    |   42 ++++++++++++++++
 tests/regression/inet/tcp.t     |  103 +++++++++++++++++++++++++++++++++++++++
 tests/regression/inet/udp.t     |   49 +++++++++++++++++++
 tests/regression/inet/udplite.t |   42 ++++++++++++++++
 8 files changed, 379 insertions(+)
 create mode 100644 tests/regression/inet/ah.t
 create mode 100644 tests/regression/inet/comp.t
 create mode 100644 tests/regression/inet/dccp.t
 create mode 100644 tests/regression/inet/esp.t
 create mode 100644 tests/regression/inet/sctp.t
 create mode 100644 tests/regression/inet/tcp.t
 create mode 100644 tests/regression/inet/udp.t
 create mode 100644 tests/regression/inet/udplite.t
tests/regression/inet/ah.t new file +58 -0
new file mode 100644
index 0000000..ebc04f7
@@ -0,0 +1,58 @@
+*ip;test-ip4
+*ip6;test-ip6
+*inet;test-inet
+:input;type filter hook input priority 0
+
+# nexthdr Bug to list table.
+
+- ah nexthdr esp;ok
+- ah nexthdr ah;ok
+- ah nexthdr comp;ok
+- ah nexthdr udp;ok
+- ah nexthdr udplite;ok
+- ah nexthdr tcp;ok
+- ah nexthdr dccp;ok
+- ah nexthdr sctp;ok
+
+- ah nexthdr { esp, ah, comp, udp, udplite, tcp, dccp, sctp};ok
+- ah nexthdr != { esp, ah, comp, udp, udplite, tcp, dccp, sctp};ok
+
+ah hdrlength 11-23;ok;ah hdrlength >= 11 ah hdrlength <= 23
+ah hdrlength != 11-23;ok;ah hdrlength < 11 ah hdrlength > 23
+ah hdrlength { 11-23};ok
+- ah hdrlength != { 11-23};ok
+ah hdrlength {11, 23, 44 };ok
+- ah hdrlength != {11-23 };ok
+
+ah reserved 22;ok
+ah reserved != 233;ok
+ah reserved 33-45;ok;ah reserved >= 33 ah reserved <= 45
+ah reserved != 33-45;ok;ah reserved < 33 ah reserved > 45
+ah reserved {23, 100};ok
+- ah reserved != {33, 55, 67, 88};ok
+ah reserved { 33-55};ok
+- ah reserved != { 33-55};ok
+
+ah spi 111;ok
+ah spi != 111;ok
+ah spi 111-222;ok;ah spi >= 111 ah spi <= 222
+ah spi != 111-222;ok;ah spi < 111 ah spi > 222
+ah spi {111, 122};ok
+- ah spi != {111, 122};ok
+# BUG: invalid expression type set
+# nft: src/evaluate.c:975: expr_evaluate_relational: Assertion '0' failed.
+
+ah spi { 111-122};ok
+- ah spi != { 111-122};ok
+# BUG: invalid expression type set
+# nft: src/evaluate.c:975: expr_evaluate_relational: Assertion '0' failed.
+
+# sequence
+ah sequence 123;ok
+ah sequence != 123;ok
+ah sequence {23, 25, 33};ok
+- ah sequence != {23, 25, 33};ok
+ah sequence { 23-33};ok
+- ah sequence != { 33-44};ok
+ah sequence 23-33;ok;ah sequence >= 23 ah sequence <= 33
+ah sequence != 23-33;ok;ah sequence < 23 ah sequence > 33
tests/regression/inet/comp.t new file +31 -0
new file mode 100644
index 0000000..32db32b
@@ -0,0 +1,31 @@
+*ip;test-ip4
+*ip6;test-ip6
+*inet;test-inet
+
+:input;type filter hook input priority 0
+
+# BUG: Do no list table.
+- comp nexthdr esp;ok;comp nexthdr 50
+comp nexthdr != esp;ok
+
+- comp nexthdr {esp, ah, comp, udp, udplite, tcp, tcp, dccp, sctp};ok
+# comp flags ## 8-bit field.  Reserved for future use.  MUST be set to zero.
+
+# Bug comp flags: to list. List the decimal value.
+comp flags 0x00;ok
+comp flags != 0x23;ok
+comp flags 0x33-0x45;ok
+comp flags != 0x33-0x45;ok
+comp flags {0x33, 0x55, 0x67, 0x88};ok
+- comp flags != {0x33, 0x55, 0x67, 0x88};ok
+comp flags { 0x33-0x55};ok
+- comp flags != { 0x33-0x55};ok
+
+comp cpi 22;ok
+comp cpi != 233;ok
+comp cpi 33-45;ok;comp cpi >= 33 comp cpi <= 45
+comp cpi != 33-45;ok;comp cpi < 33 comp cpi > 45
+comp cpi {33, 55, 67, 88};ok
+- comp cpi != {33, 55, 67, 88};ok
+comp cpi { 33-55};ok
+- comp cpi != { 33-55};ok
tests/regression/inet/dccp.t new file +31 -0
new file mode 100644
index 0000000..fd25d82
@@ -0,0 +1,31 @@
+*ip;test-ip4
+*ip6;test-ip6
+*inet;test-inet
+:input;type filter hook input priority 0
+
+dccp sport 21-35;ok;dccp sport >= 21 dccp sport <= 35
+dccp sport != 21-35;ok;dccp sport < 21 dccp sport > 35
+dccp sport {23, 24, 25};ok;dccp sport { 23, 24, 25}
+- dccp sport != { 27, 34};ok
+# BUG: invalid expression type set
+# nft: src/evaluate.c:975: expr_evaluate_relational: Assertion '0' failed.
+
+dccp sport { ftp-data - re-mail-ck};ok;dccp sport { 20-50}
+dccp sport ftp-data - re-mail-ck;ok;dccp sport >= 20 dccp sport <= 50
+dccp sport { 20-50};ok
+# dccp sport != {27-34};ok
+- BUG: invalid expression type set
+- nft: src/evaluate.c:975: expr_evaluate_relational: Assertion '0' failed.
+
+# dccp dport 21-35;ok
+# dccp dport != 21-35;ok
+dccp dport {23, 24, 25};ok
+# dccp dport != {27, 34};ok
+dccp dport { 20-50};ok
+# dccp dport != {27-34};ok
+
+# BUG dccp type
+# dccp type {request, response, data, ack, dataack, closereq, close, reset, sync, syncack};ok
+# dccp type != {request, response, data, ack, dataack, closereq, close, reset, sync, syncack};ok
+# dccp type request;ok
+# dccp type != request;ok
tests/regression/inet/esp.t new file +23 -0
new file mode 100644
index 0000000..1f23aa4
@@ -0,0 +1,23 @@
+*ip;test-ip4
+*ip6;test-ip6
+*inet;test-inet
+:input;type filter hook input priority 0
+
+esp spi 100;ok
+esp spi != 100;ok
+esp spi 111-222;ok;esp spi >= 111 esp spi <= 222
+esp spi != 111-222;ok;esp spi < 111 esp spi > 222
+esp spi { 100, 102};ok
+- esp spi != { 100, 102};ok
+esp spi { 100-102};ok
+- esp spi {100-102};ok
+
+esp sequence 22;ok
+esp sequence 22-24;ok;esp sequence >= 22 esp sequence <= 24
+esp sequence != 22-24;ok;esp sequence < 22 esp sequence > 24
+esp sequence { 22, 24};ok
+- esp sequence != { 22, 24};ok
+# BUG: invalid expression type set
+# nft: src/evaluate.c:975: expr_evaluate_relational: Assertion '0' failed.
+esp sequence { 22-25};ok
+- esp sequence != { 22-25};ok
tests/regression/inet/sctp.t new file +42 -0

42 lines not on this page · raw message

tests/regression/inet/tcp.t new file +103 -0

103 lines not on this page · raw message

tests/regression/inet/udp.t new file +49 -0

49 lines not on this page · raw message

tests/regression/inet/udplite.t new file +42 -0

42 lines not on this page · raw message

Signature · 1 line
1.7.10.4
author
#4 lore raw

PATCH v2 4/6 tests: Add arp folder with test files.

In reply to Ana Rey

"arp" folder contains the test files that are executed in arp tables.

These test files are executed with nft-tests.py

---
 tests/regression/arp/arp.t    |   53 +++++++++++++++++++++++++++++++++++++++++
 tests/regression/arp/chains.t |    5 ++++
 2 files changed, 58 insertions(+)
 create mode 100644 tests/regression/arp/arp.t
 create mode 100644 tests/regression/arp/chains.t
tests/regression/arp/arp.t new file +53 -0
new file mode 100644
index 0000000..797e394
@@ -0,0 +1,53 @@
+*arp;test-arp
+# filter chains available are: input, output, forward
+:input;type filter hook input priority 0
+
+arp htype 1;ok
+arp htype != 1;ok
+arp htype 22;ok
+arp htype != 233;ok
+arp htype 33-45;ok;arp htype >= 33 arp htype <= 45
+arp htype != 33-45;ok;arp htype < 33 arp htype > 45
+arp htype { 33, 55, 67, 88};ok
+- arp htype != { 33, 55, 67, 88};ok
+arp htype { 33-55};ok
+- arp htype != { 33-55};ok
+
+arp ptype 0x0800;ok
+
+arp hlen 22;ok
+arp hlen != 233;ok
+arp hlen 33-45;ok;arp hlen >= 33 arp hlen <= 45
+arp hlen != 33-45;ok;arp hlen < 33 arp hlen > 45
+arp hlen { 33, 55, 67, 88};ok
+- arp hlen != { 33, 55, 67, 88};ok
+arp hlen { 33-55};ok
+- arp hlen != { 33-55};ok
+
+arp plen 22;ok
+arp plen != 233;ok
+arp plen 33-45;ok;arp plen >= 33 arp plen <= 45
+arp plen != 33-45;ok;arp plen < 33 arp plen > 45
+arp plen { 33, 55, 67, 88};ok
+- arp plen != { 33, 55, 67, 88};ok
+arp plen { 33-55};ok
+- arp plen != {33-55};ok
+
+arp operation {nak, inreply, inrequest, rreply, rrequest, reply, request};ok
+- arp operation != {nak, inreply, inrequest, rreply, rrequest, reply, request};ok
+arp operation request;ok
+arp operation reply;ok
+arp operation rrequest;ok
+arp operation rreply;ok
+arp operation inrequest;ok
+arp operation inreply;ok
+arp operation nak;ok
+arp operation reply;ok
+arp operation != request;ok
+arp operation != reply;ok
+arp operation != rrequest;ok
+arp operation != rreply;ok
+arp operation != inrequest;ok
+arp operation != inreply;ok
+arp operation != nak;ok
+arp operation != reply;ok
tests/regression/arp/chains.t new file +5 -0
new file mode 100644
index 0000000..6b0c042
@@ -0,0 +1,5 @@
+*arp;test-arp
+- filter chains available are: input, output, forward
+:input;type filter hook input priority 0
+:forward;type filter hook forward priority 0
+:output;type filter hook output priority 0
Signature · 1 line
1.7.10.4
author
#5 lore raw

PATCH v2 5/6 tests: Add bridge folder with test files.

In reply to Ana Rey

"bridge" folder contains the test files that are executed in bridge tables.

These test files are executed with nft-tests.py

---
 tests/regression/bridge/chains.t |    7 +++++++
 1 file changed, 7 insertions(+)
 create mode 100644 tests/regression/bridge/chains.t
tests/regression/bridge/chains.t new file +7 -0
new file mode 100644
index 0000000..3fb1c89
@@ -0,0 +1,7 @@
+*bridge;test-bridge
+# filter chains available are: input, output, forward
+:filter-input;type filter hook input priority 0
+:filter-forward;type filter hook forward priority 0
+:filter-output;type filter hook output priority 0
+
+- ether type 1;ok
Signature · 1 line
1.7.10.4
author
#6 lore raw

PATCH v2 6/6 tests: Add any folder with test files.

In reply to Ana Rey

"any" folder contains the test files that are executed in ipv4, ipv6, inet, arp, bridge family of tables.

These test files are executed with nft-tests.py

---
 tests/regression/any/ct.t    |  105 +++++++++++++++++++++++++++
 tests/regression/any/frag.t  |   64 +++++++++++++++++
 tests/regression/any/limit.t |   12 ++++
 tests/regression/any/log.t   |   27 +++++++
 tests/regression/any/meta.t  |  160 ++++++++++++++++++++++++++++++++++++++++++
 tests/regression/any/queue.t |   15 ++++
 6 files changed, 383 insertions(+)
 create mode 100644 tests/regression/any/ct.t
 create mode 100644 tests/regression/any/frag.t
 create mode 100644 tests/regression/any/limit.t
 create mode 100644 tests/regression/any/log.t
 create mode 100644 tests/regression/any/meta.t
 create mode 100644 tests/regression/any/queue.t
tests/regression/any/ct.t new file +105 -0
new file mode 100644
index 0000000..512ca78
@@ -0,0 +1,105 @@
+*ip;test-ip4
+*ip6;test-ip6
+*inet;test-inet
+# ct expresion is not supported in arp and bridge family yet.
+- *arp;test-arp
+- *bridge;test-bridge
+
+:output;type filter hook output priority 0
+
+ct state new,established, related, untracked;ok;ct state established,related,new,untracked
+ct state != related;ok
+ct state {new,established, related, untracked};ok
+- ct state != {new,established, related, untracked};ok
+ct state invalid drop;ok
+ct state established accept;ok
+
+ct direction original;ok
+ct direction != original;ok
+ct direction reply;ok
+ct direction != reply;ok
+ct direction {reply, original};ok
+- ct direction != {reply, original};ok
+
+ct status expected;ok
+ct status != expected;ok
+ct status seen-reply;ok
+ct status != seen-reply;ok
+ct status {expected, seen-reply, assured, confirmed, dying};ok
+
+# SYMBOL("snat", IPS_SRC_NAT)
+# SYMBOL("dnat", IPS_DST_NAT)
+- ct status snat;ok
+- ct status dnat;ok
+
+ct mark 0;ok
+ct mark or 0x23 == 0x11;ok
+ct mark or 0x3 != 0x1;ok
+ct mark and 0x23 == 0x11;ok
+ct mark and 0x3 != 0x1;ok
+ct mark xor 0x23 == 0x11;ok
+ct mark xor 0x3 != 0x1;ok
+
+ct mark 0x32;ok
+ct mark != 0x32;ok
+ct mark 0x32-0x45;ok
+ct mark != 0x32-0x43;ok
+ct mark {0x32, 0x2222, 0x42de3};ok
+- ct mark != {0x32, 0x2222, 0x42de3};ok
+
+# ct mark != {0x32, 0x2222, 0x42de3};ok
+# BUG: invalid expression type set
+# nft: src/evaluate.c:975: expr_evaluate_relational: Assertion '0' failed.
+
+ct mark set 0x11 xor 0x1331;ok
+ct mark set 0x11333 and 0x11;ok
+ct mark set 0x12 or 0x11;ok
+ct mark set 0x11;ok
+
+ct expiration 30;ok
+ct expiration 22;ok
+ct expiration != 233;ok
+ct expiration 33-45;ok
+# BUG: ct expiration 33-45 and  ct expiration != 33-45
+# Broken output: ct expiration >= "33s" ct expiration <= "9709d53m20s"
+ct expiration != 33-45;ok
+ct expiration {33, 55, 67, 88};ok
+- ct expiration != {33, 55, 67, 88};ok
+ct expiration {33-55};ok
+# BUG: ct expiration {33-55}
+# Broken output: ct expiration { "4271d23h25m52s"-"8738d3h11m59s" }
+- ct expiration != {33-55};ok
+
+ct helper "ftp";ok
+ct helper "12345678901234567";fail
+
+# BUG: ct l3proto "Layer  3 protocol of the connection"
+# nft add rule ip test input ct l3proto arp
+# <cmdline>:1:35-37: Error: Can t parse symbolic invalid expressions
+
+
+# If table is ip6 or inet or bridge family,, It is failed. I can not test it
+# ct saddr 1.2.3.4;ok
+
+# BUG: ct saddr 192.168.3.4
+# <cmdline>:1:1-43: Error: Could not process rule: Invalid argument
+# add rule ip test input ct saddr 192.168.3.4
+# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+- ct saddr 192.168.3.4;ok
+- ct daddr 192.168.3.4;ok
+
+# BUG: ct protocol tcp
+# <cmdline>:1:1-37: Error: Could not process rule: Invalid argument
+# input ct protocol bgp <cmdline>:1:36-38: Error: Could not resolve protocol name
+# ct protocol tcp;ok
+- ct protocol tcp;ok
+
+- ct proto-src udp;ok
+- ct proto-dst udp;ok
+# BUG: ct proto-src udp and ct proto-dst udp
+# <cmdline>:1:37-39: Error: datatype mismatch, expected invalid, expression has type Internet protocol
+# add rule ip test input ct proto-src udp
+#                       ~~~~~~~~~~~~ ^^^
+# <cmdline>:1:37-39: Error: datatype mismatch, expected invalid, expression has type Internet protocol
+# add rule ip test input ct proto-dst udp
+#                        ~~~~~~~~~~~~ ^^^
tests/regression/any/frag.t new file +64 -0

64 lines not on this page · raw message

tests/regression/any/limit.t new file +12 -0
new file mode 100644
index 0000000..9af1ea8
@@ -0,0 +1,12 @@
+*ip;test-ip4
+*ip6;test-ip6
+*inet;test-inet
+*arp;test-arp
+*bridge;test-bridge
+:output;type filter hook output priority 0
+
+limit rate 400/minute;ok
+limit rate 20/second;ok
+limit rate 400/hour;ok
+limit rate 400/week;ok
+limit rate 40/day;ok
tests/regression/any/log.t new file +27 -0
new file mode 100644
index 0000000..2bc2543
@@ -0,0 +1,27 @@
+*ip;test-ip4
+*ip6;test-ip6
+*inet;test-inet
+- *arp;test-arp
+- *bridge;test-bridge
+:output;type filter hook output priority 0
+
+ct direction original log;ok
+log;ok
+log level emerg;ok
+log level alert;ok
+log level crit;ok
+log level err;ok
+log level warn;ok;log
+log level notice;ok
+log level info;ok
+log level debug;ok
+
+log level emerg group 2;fail
+log level alert group 2 prefix "log test2";fail
+
+udp dport 200 log prefix aaaaa-aaaaaa group 2 snaplen 33;ok;udp dport 200 log prefix "aaaaa-aaaaaa" group 2 snaplen 33
+# TODO: Add an exception: 'queue-threshold' attribute needs 'group' attribute
+# The correct rule is log group 2 queue-threshold 2
+log group 2 queue-threshold 2;ok
+log group 2 snaplen 33;ok
+tcp dport 300 log group 2 prefix \"IPTABLES-NFT SSH\";ok;tcp dport 300 log prefix "IPTABLES-NFT SSH" group 2
tests/regression/any/meta.t new file +160 -0

160 lines not on this page · raw message

tests/regression/any/queue.t new file +15 -0
new file mode 100644
index 0000000..8307411
@@ -0,0 +1,15 @@
+*ip;test-ip4
+*ip6;test-ip6
+*inet;test-inet
+*arp;test-arp
+*bridge;test-bridge
+
+:output;type filter hook output priority 0
+
+queue;ok;queue num 0
+queue num 2;ok
+queue num 2-3;ok
+- queue num {3, 4, 6};ok
+queue num 4-5 fanout bypass;ok;queue num 4-5 bypass fanout
+queue num 4-5 fanout;ok
+queue num 4-5 bypass;ok
Signature · 1 line
1.7.10.4
#7 lore raw
In reply to Ana Rey
Quoted 8 lines · Ana Rey

This is a new version of the automated regression testing of nftables.

There is no infrastructure to allow us to check all options/features in nft. So, if anyone sends a patch, we cannot check if it breaks something.

I send in this patchset the nftables automated regression tests. It contains a python script (nft-add.py) and a set of test files.

Quoted 8 lines

This allow us to check the input of rules of nft-tool from the command-line and the output from nft-tool of this rule. Then, it compares if the rule input matches the rule output automatically.

We also have plans to add automated regression testing in the packet path in the future, which should come in a follow up step.

Comments welcome, thanks

I think it's great to have this and would suggest pulling it into the tree now and continue the work there.

So if you don't have any objections, I'll apply the series.

#8 lore raw
Quoted 16 lines · Patrick McHardy
Quoted 15 lines · Ana Rey

This is a new version of the automated regression testing of nftables.

There is no infrastructure to allow us to check all options/features in nft. So, if anyone sends a patch, we cannot check if it breaks something.

I send in this patchset the nftables automated regression tests. It contains a python script (nft-add.py) and a set of test files.

This allow us to check the input of rules of nft-tool from the command-line and the output from nft-tool of this rule. Then, it compares if the rule input matches the rule output automatically.

We also have plans to add automated regression testing in the packet path in the future, which should come in a follow up step.

Quoted 6 lines
Quoted 1 line

Comments welcome, thanks

I think it's great to have this and would suggest pulling it into the tree now and continue the work there.

So if you don't have any objections, I'll apply the series.

Please wait, she's going to send a final v3.

Thanks.