nftables DNAT routes to wrong iface

Hello,

I am trying to do this:

Both local IP addresses, when access from outside:

enp2s0.310: 193.72.186.130:8080
br0: 46.140.72.218:8080

are DNATed to

enp2s0.202: 192.168.202.10:80 (a remote machine)

I know there are two steps:

- the incoming DNAT in prerouting (and the forward accept)

     -> this part I am stuck, see below

- then when the reply comes back, route to the proper interface where it came from (using conntrack + marks + specific routing tables) -> this part I have not done yet -- it would be required for 193.72.186.130:8080 obviously because the default route does not go there.

What I observe:

telnet 46.140.72.218 8080 from outside works (*), it connects to 192.168.202.10:80 and there is nothing bizarre in tcpdump either on br0 nor on enp2s0.202 (no delays, lost packets, e.g.) (**)

telnet 193.72.186.130 8080 gives this on enp2s0.300

      IP 193.72.186.190.52636 > 193.72.186.130.8080 (normal)
      IP 193.72.186.190.52636 > 192.168.202.10.80   (good, it was DNATted, BUT should be on enp2s0.202!)

aka the DNAT is executed, but then 192.168.202.10 is not routed
correctly. From the diagram https://wiki.nftables.org/wiki-nftables/index.php/Netfilter_hooks
I thought that routing would be done AFTER prerouting/DNAT.

The routing table has:

default via 46.140.72.217 dev br0 onlink
46.140.72.216/29 dev br0 proto kernel scope link src 46.140.72.218 192.168.202.0/24 dev enp2s0.202 proto kernel scope link src 192.168.202.2 193.72.186.128/26 dev enp2s0.300 proto kernel scope link src 193.72.186.130

(*) not from 193.72.186.0/24, however, gets the same bug.

The nftables config:

table ip filter {
        # obviously a later goal is also to encode the L4 protocol here
        # and not hardcode it in the prerouting
        map multihoming_ext {
                type ipv4_addr . inet_service : ipv4_addr . inet_service
                elements = { 193.72.186.130 . 8080 : 192.168.202.10 . 80,
                             46.140.72.218 . 8080 : 192.168.202.10 . 80 }
        }

        set w_all {
                type ipv4_addr
                flags interval
                elements = { 46.140.72.216/29,
                             192.168.202.1, 193.72.186.0/24
                           }
        }

        chain input {
                type filter hook input priority filter; policy drop;
                ct state invalid counter packets 3 bytes 120 drop
                ct state { established, related } counter packets 1092 bytes 76846 accept
                iif "lo" counter packets 0 bytes 0 accept
                counter packets 324 bytes 13539 jump whitelist
                counter packets 323 bytes 13455 jump blacknets
                counter packets 323 bytes 13455 jump blacklist
                counter packets 323 bytes 13455 jump incoming
        }

        chain forward {
                type filter hook forward priority filter; policy drop;
                ct state invalid counter packets 0 bytes 0 drop
                ct state { established, related } counter packets 12 bytes 548 accept

                # already after DNAT, obviously
                iifname "br0" ip daddr 192.168.202.10 tcp dport 80 accept
                iifname "enp2s0.300" ip daddr 192.168.202.10 tcp dport 80 accept
        }

        chain output {
                type filter hook output priority filter; policy accept;
                ct state invalid counter packets 0 bytes 0 drop
                ct state { established, related } counter packets 890 bytes 182096 accept
                oif "lo" counter packets 0 bytes 0 accept
                counter packets 1 bytes 76 jump outgoing
        }

        chain rejectcounter {
                meta l4proto tcp counter packets 0 bytes 0 reject with tcp reset
                meta l4proto udp counter packets 0 bytes 0 reject
                counter packets 0 bytes 0 drop
        }

        chain dropcounter {
                counter packets 320 bytes 13349 drop
        }

        chain whitelist {
                ip saddr @w_all counter packets 1 bytes 84 accept
        }

        chain blacknets {
        }

        chain blacklist {
        }

        chain incoming {
                icmp type echo-request counter packets 3 bytes 106 accept
                icmp type echo-reply counter packets 0 bytes 0 accept
                tcp dport 22 ip saddr { 46.140.72.216/29, 192.168.202.1, 193.72.186.0/24 } counter packets 0 bytes 0 jump dropcounter
                udp dport 22 ip saddr { 46.140.72.216/29, 192.168.202.1, 193.72.186.0/24 } counter packets 0 bytes 0 jump dropcounter
                counter packets 320 bytes 13349 jump dropcounter
        }

        chain outgoing {
        }

        chain multihoming_prerouting {
                type nat hook prerouting priority dstnat; policy accept;

                # this is the DNAT
                dnat ip to ip daddr . tcp dport map @multihoming_ext
        }
}
table ip myhelpers {
        chain prerouting {
                type filter hook prerouting priority filter; policy accept;
        }
}

[ I removed ip6 entries ]

Any idea what could be wrong?

Thank you.

(**) very nice:
br0:
IP 46.140.72.222.60394 > 46.140.72.218.8080
IP 46.140.72.218.8080 > 46.140.72.222.60394
IP 46.140.72.222.60394 > 46.140.72.218.8080

enp2s0.202 (another connection):
IP 46.140.72.222.57790 > 192.168.202.10.80
IP 192.168.202.10.80 > 46.140.72.222.57790
IP 46.140.72.222.57790 > 192.168.202.10.80

Hi,

Oops! Old reply (please disregard): '193.72.186.128/26' is poorly subnetted; the correct base address is 193.72.186.192/26. Did you mean '193.72.186.128/25'? As a result, '193.72.186.130' is not part of the subnet that you are routing. Consider if 25 bits is the correct mask.

New reply (and now public): '193.72.186.128/26' does not cover all of '193.72.186.0/24', and some packets will not get routed. You could try changing your route to '193.72.186.0/24', or edit the firewall policy to reflect the '193.72.186.128/26' route.

sunny

Sent with Proton Mail secure email.

Hello,

Why should it? The firewall 193.72.186.130 on which the DNAT is done is connected to 193.72.186.128/26 because this is how the router 193.72.186.129 on the other side of the VLAN 300 defined that VLAN:

193.72.186.128/26 dev enp5s0.300 proto kernel scope link src 193.72.186.129

Indeed, the addresses 193.72.186.128 (reserved, subnet) to 193.72.186.191 (reserved, subnet broadcast) correspond to a 193.72.186.128/26 aka 32 - 26 = 6 free bit, so 64 addresses.

On the same VLANs there are other machines, such as 193.72.186.190 that I used for my test.

BTW, those (193.72.186.129, 193.72.186.190) are ping-reachable from the Internet (193.72.186.130 is currently down, as it is a test machine). The setup works, it's just the DNAT that routes wrong for some reason (I am an nftables beginner). Do not try random connections to those IP addresses (ping is ok), you might trigger the IDS.

The DNAT problem is that when a machine from 193.72.186.128/26 (e.g. 193.72.186.190 in the example) sends a TCP datagram on port 8080 of 193.72.186.130, this gets correctly DNATted to 192.168.202.10 port 80, however, then the routing entry:

is not respected, and the datagram is sent on enp2s0.300.

In another case, which is 46.140.72.218:8080 (on enp2s0.400), the DNAT correctly works and sends to 192.168.202.10 on enp2s0.202.

As said in the mail, the reply packet won't work on 193.72.186.128/26, that's expected (not default route, and no conntrack/mark set yet), but it's the initial packet which is routed wrong.

Thank you & have a nice day.

Hello,

Ok, my fault!

There still was an old configuration lying around which did:

root@test:~# ip rule
0:      from all lookup local
32764:  from 193.72.186.128/26 lookup 193
32766:  from all lookup main
32767:  from all lookup default

obviously, that's the bug.

If I remove those, then it works like it should.

I will now work on the conntrack part.