nftables rule optimization - dropping invalid in ingress?

As per advice by Kerin Millar, this is a continuation of another discussion [1] which resulted in a different topic.

Actually, I have read about all this in older posts here. I should have probably clarified better the forest, not just the trees.

The rules I mention (along with a few others) were inspired by a few sources - some using iptables (where INVALID may be different in its code definition from nftables and thus need such rules). That said, I have actually tested and am aware that e.g. Xmas is an invalid TCP packet that will be dropped by conntrack anyway. Similarly, the others too.

However, in the setup I am trying to implement, I am attempting to be "clever" and optimize things by dropping bad traffic earlier, so I am doing it in the ingress hook where, AFAICS, conntrack is not available. Why ingress? - Because I am following the general principle that attacks should be stopped as soon and as far as possible, rather than allowing them go further inside (in this case - next hooks). And even though the next hook (prerouting) can drop e.g. Xmas of FINSYN as invalid, I assume it would be a waste of CPU cycles to allow further processing of such traffic. So, I thought: why not prevent the unnecessary load on stateful conntrack? - Hence the whole idea to drop early.

OTOH, adding more rules to ingress adds CPU cycles itself.

Which is more optimal - dropping early or not piling up extra rules in ingress? Looking for an answer to that, I have done this:

As per earlier advise from you in a different context, I checked this:

# zgrep BPFILTER /proc/config.gz 
# CONFIG_BPFILTER is not set

If I am reading this correctly, it means there is no BPF JIT optimization. Is this normal? Is BPF still experimental and for that reason not available? I don't know, which is why I asked and still hope for an answer:

https://marc.info/?l=netfilter&m=171345423924347&w=2

Why am I referring to BPF? - Because I suppose having it available would make the difference between the "drop early" (in ingress) and "drop as invalid" (in prerouting) cases negligible.

Now, the question comes down to: How big is the actual difference? Is it negligible right now (without BPF)? - I really don't know. Hence this other thread:

https://marc.info/?l=netfilter&m=171354240711565&w=2

Any info and advice is very welcome, as the whole thing discussed here is very unclear to me.

--

[1] https://marc.info/?l=netfilter&m=171358042732609&w=2

INVALID packets are those that netfilter has no idea what to do with and will eventually drop them after they fit no existing ACCEPT rules. A couple examples of INVALID packets would be (1) a TCP RESET for a non-existent conn (possibly one that was just reset, possibly a DoS attack), and (2) a TCP data packet for a non-existent conn. Neither of these is routable because there is no place to send them. Thus they are dropped in due time. The choice is whether to drop them after they pass through all the rules they will pass through or to drop them as soon as possible after conntrack tags them as INVALID.

With iptables, I found the earliest I could drop bad traffic (large blocksets of addrs I never want access to or from whether or not they are INVALID, and all other INVALID packets) was at the top of PREROUTING in table mangle. I would think nftables is similar.

The most optimal involves the least amount of processing. I would think conntrack is more efficient at tagging INVALID packets than a bunch of rules somewhere. Then it takes only one rule to drop INVALID packets early in PREROUTING. Or two if you jump to a chain that has a DROP rule; said chain would allow you to easily add or remove a log rule.

Neal

After spending some time looking for more info and based on this:

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/net/netfilter/nf_conntrack_proto_tcp.c?h=v5.8#n700

I think I figured it out:

tcp flags & (fin|syn|rst|ack|urg) != {
		syn,
		syn|urg,
		syn|ack,
		rst,
		rst|ack,
		fin|ack,
		fin|ack|urg,
		ack,
		ack|urg
	} drop comment "TCP invalid"

This checks the listed values against the mask "fin|syn|rst|ack|urg". The same values and mask are used in the conntrack code, i.e. it drops invalid TCP packets.

According to my own tests, this works in the ingress hook, i.e. early drop.

The only question that remains is performance measurement and comparison, as mentioned.

Please let me know what you think.

The rule looks good. Borrowing from the conntrack code was a bright idea.

If using the ingress hook in this way is to make any measurable difference to your load average at all, my expectation would be for it be observable in the event that you are subjected to a concentrated flood of invalid TCP packets. You could use hping3 to conduct a series of stress tests.

Kerin Millar

I was thinking the same.

The question is how to measure the difference correctly. I am not a network expert. I see some articles about firewall performance testing using iperf3 but in a VM-to-VM test (which I use) it is quite inconsistent. Additionally, iperf3 is not able to do what hping3 does and I have no idea how to measure anything using hping3, which is probably the correct test, as it would trigger the rules.

My clumsy attempt to have at least some comparison:

Testing procedure
-----------------

1. Design the ruleset
2. Put the rule to be tested in a separate file and include that file
3. In the file, have the line with the rule, followed by a single 'continue' on a new line. Repeat this combination 1000 times (to make it have an observable effect)
4. In VM1 run (where rules reside):

iperf3 -s -p <port>
5. In VM2 run:

for (( i=0; i<10; i++)); do iperf3 -c 10.137.0.82 -p <port> -V; done

6. Note the speed after each iteration.

I am doing it 10 times because my observation shows that the first iteration is always faster and the next ones decline to a point of "saturation" (my interpretation - some resource exhaustion, e.g. memory buffer). That is the inconsistency, mentioned above and my attempt to get rid of it. Additionally, I also tested 'time ./my-firewall' to compare this too.

Results
-------

# Early drop:

time to load = 23 sec
Bitrates (in Gbits/sec):

3.53
2.88
2.92
2.91
2.66
2.80
2.83
2.79
2.83
2.82

# Drop in prerouting (using conntrack and 'invalid'):

time to load = 1.326 s
Bitrates:

4.19
3.48
3.46
3.50
3.53
3.52
3.51
3.48
3.42
3.40

For comparison, the speed without any ruleset (after 'nft flush ruleset'):

Bitrates:

5.54
4.96
4.66
4.33
4.32
4.32
4.33
4.34
4.31
4.32

Summarized
----------

Given the rules are repeated 1000 times, early drop is ~82% of the speed of dropping using conntrack in prerouting and takes x20 more time to load.

My interpretation (may be completely wrong, so corrections are welcome)
-----------------

conntrack is faster because it is compiled (binary), i.e. the same rule written in nftables syntax needs additional processing (hence the much higher load time), which makes it less efficient.

Additional info
---------------

If the rule is set only once (not repeated 1000 times, as in the test) there is no difference between the 3 cases above - same speed. So, ceteris paribus, the question comes down to: is there a security benefit in dropping invalid packets earlier (e.g. right after dropping fragments and bogons)?

I would highly appreciate the thoughts of the experts here.

Another test, using only invalid packets, shows the opposite result:

timeout 65s hping3 <host> --flood --xmas

When dropping in ingress:

14455889 packets transmitted
(on <host>) load average: 0.26, 0.30, 0.18

When dropping in prerouting (conntrack 'invalid'):

13794361 packets transmitted
(on <host>) load average: 0.55, 0.29, 0.19

So, after all, dropping in ingress seems faster (5% more packets for the same time) and x2 less resource intensive.

Sent with Proton Mail secure email.

I'd be very interested in seeing some statistics on how many actual invalid packets you see on a live link. Stick some counters in there and collect dropped versus passed packets...

My naive guess would be there are only tiny percentage of rejected packets.

Eric

Dňa 21. apríla 2024 3:45:31 UTC používateľ Eric <[email protected]> napísal:

Your guess is not as naive, at least on my low traffic server i have ~70 drops (IPv4) in last 7 days. On my router it is ~370 drops in both, the INPUT & FORWARD chains (iptables), in last ~100 days. That are really tiny fractions of other drops, but many drops (by IP) happens before INVALID rule.

regards

Slavko
https://www.slavino.sk/

This particular system is a desktop one (rebooted often), so that kind of stats won't make any sense.

Without a particular attack - quite possible. However, it is always good to learn what is better/worse/futile.

[Again, this is iptables; your mileage with nftables may vary.]

From my firewall that's been up 30 days; I think these are reasonable numbers. It shows the total packets that passed PREROUTING, the packets from internet dropped due to my blocklists (which probably includes at least some INVALID packets), and the remaining INVALID packets from internet and internal sources. These two are the only DROPs in PREROUTING. Ballpark, about 0.5% of the packets are INVALID. Small, but not necessarily 'tiny'.
-----
*mangle
:PREROUTING ACCEPT [728638:3046835361]

[43686:2232175] -A PREROUTING -i eth3 \
    -m set --match-set blockSetHost src -j blDrop
[37712:1840302] -A PREROUTING \
    -m state --state INVALID -j invdrop

-----

Note that dropping them at the top of PREROUTING prevents them from passing through the rest of the rules in PREROUTING (and mangle), and rules in nat, and any rules in filter they might hit before finally being DROPped.

N