Source IP incorrect on multi homed systems
Dear group,
I was wondering how wireguard [Linux kernel] or wireguard-go [FreeBSD] are supposed to decide which IP address to use for replying?
I have seen both on FreeBSD and Linux that wireguard seems to use the IP address of the outgoing interface, i.e. the one with the route returning to the sender. However in multi homed situations, this can be wrong, let's take this example:
19:57:24.607526 net1 In IP 194.5.220.43.60770 > 147.78.195.254.51820: UDP, length 148
19:57:24.608358 net2 Out IP 195.141.200.73.51820 > 194.5.220.43.60770: UDP, length 92The initiator sends from 194.5.220.43 to the receiver 147.78.195.254. Wireguard then replies with the source IP of 195.141.200.73 instead of 147.78.195.254.
As the node is multi homed, the packet might leave through any of its uplinks and thus return with a random (unexpected) IP address and will not pass NAT rules on firewalls and finally be dropped. F.i. in above example the firewall drops the packet from 195.141.200.73, because there is no session entry for that.
I have observed this behaviour both on Linux 6.1.11 as well as wireguard-go 0.0.20220316_8,1 on FreeBSD and in both cases the connection will break depending on which active interface is taken as exit.
I would argue that wireguard should by default invert the IP addresses, i.e. switch dst=src, src=dst and then reply with that, instead of adapting an interface specific address, or is there a good reason for the current behaviour?
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
Hello Omkhar,
I tend to disagree. The problem is not the routing, but the selected source address, which is independent of routing. To be more specific: as there is BGP routing on all all interfaces, 147.78.195.254 is an accepted IP address on any interface.
Best regards,
Nico
Quoted 39 lines · Omkhar Arasaratnam
This looks like an asymmetric routing issue from what you’re describing, not a wireguard issue.
You may want to look into policy based routing to address it.
On Sat, Feb 18, 2023 at 15:54 Nico Schottelius <[email protected]> wrote:
Dear group,
I was wondering how wireguard [Linux kernel] or wireguard-go [FreeBSD] are supposed to decide which IP address to use for replying?
I have seen both on FreeBSD and Linux that wireguard seems to use the IP address of the outgoing interface, i.e. the one with the route returning to the sender. However in multi homed situations, this can be wrong, let's take this example:
19:57:24.607526 net1 In IP 194.5.220.43.60770 > 147.78.195.254.51820: UDP, length 148 19:57:24.608358 net2 Out IP 195.141.200.73.51820 > 194.5.220.43.60770: UDP, length 92The initiator sends from 194.5.220.43 to the receiver 147.78.195.254. Wireguard then replies with the source IP of 195.141.200.73 instead of 147.78.195.254.
As the node is multi homed, the packet might leave through any of its uplinks and thus return with a random (unexpected) IP address and will not pass NAT rules on firewalls and finally be dropped. F.i. in above example the firewall drops the packet from 195.141.200.73, because there is no session entry for that.
I have observed this behaviour both on Linux 6.1.11 as well as wireguard-go 0.0.20220316_8,1 on FreeBSD and in both cases the connection will break depending on which active interface is taken as exit.
I would argue that wireguard should by default invert the IP addresses, i.e. switch dst=src, src=dst and then reply with that, instead of adapting an interface specific address, or is there a good reason for the current behaviour?
Quoted 6 lines
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
--
Sustainable and modern Infrastructures by ungleich.ch
Generally all OSs will if sending from a local process will use the address of the outgoing interface for the packet.
If the packet is forwarded and no NAT is used the address will be routed via the interface suggested by the routing table.
So local routing can be a real pain, policy based routing is an option. The other option could be to setup an 'output' NAT to an address which is multi-homed.
I have a system running which is multi-homed with out issue other than the actual routing machine. This machine is BGP connected to three locations.
There is no NAT setup and because I also add the wireguard link addresses to the BGP sessions.
Cheers
Quoted 39 lines · Nico Schottelius
Dear group,
I was wondering how wireguard [Linux kernel] or wireguard-go [FreeBSD] are supposed to decide which IP address to use for replying?
I have seen both on FreeBSD and Linux that wireguard seems to use the IP address of the outgoing interface, i.e. the one with the route returning to the sender. However in multi homed situations, this can be wrong, let's take this example:
19:57:24.607526 net1 In IP 194.5.220.43.60770 > 147.78.195.254.51820: UDP, length 148 19:57:24.608358 net2 Out IP 195.141.200.73.51820 > 194.5.220.43.60770: UDP, length 92The initiator sends from 194.5.220.43 to the receiver 147.78.195.254. Wireguard then replies with the source IP of 195.141.200.73 instead of 147.78.195.254.
As the node is multi homed, the packet might leave through any of its uplinks and thus return with a random (unexpected) IP address and will not pass NAT rules on firewalls and finally be dropped. F.i. in above example the firewall drops the packet from 195.141.200.73, because there is no session entry for that.
I have observed this behaviour both on Linux 6.1.11 as well as wireguard-go 0.0.20220316_8,1 on FreeBSD and in both cases the connection will break depending on which active interface is taken as exit.
I would argue that wireguard should by default invert the IP addresses, i.e. switch dst=src, src=dst and then reply with that, instead of adapting an interface specific address, or is there a good reason for the current behaviour?
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
Let me rephrase the problem statement:
- ping and http calls to the multi homed machine work correctly:
I can ping 147.78.195.254 and the reply contains the same address. I can ping 195.141.200.73 and the reply contains the same address. I can curl 147.78.195.254 and the reply contains the same address. I can curl 195.141.200.73 and the reply contains the same address.
- wireguard does NOT work because it changes the reply address:
A packet sent to 147.78.195.254 is being replied with 195.141.200.73
In general, processes reply with the IP address that was used to contact them and not with the outgoing interface address, which would also break adding IP addresses to the loopback interface.
For full detail, see ip addresses [0] and routing below [1] and tests executed [2].
I believe that this is a bug in wireguard.
--------------------------------------------------------------------------------
[2]
Let's see how it looks like in detail:
1) ping to 147.78.195.254: works
[9:14] nb3:~% ping -c2 147.78.195.254
PING 147.78.195.254 (147.78.195.254) 56(84) bytes of data. 64 bytes from 147.78.195.254: icmp_seq=1 ttl=53 time=7.27 ms 64 bytes from 147.78.195.254: icmp_seq=2 ttl=53 time=6.30 ms
--- 147.78.195.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 6.296/6.781/7.267/0.485 ms
/ # tcpdump -ni any host 194.5.220.43
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
08:14:48.379618 net1 In IP 194.5.220.43 > 147.78.195.254: ICMP echo request, id 89, seq 1, length 64 08:14:48.379651 net2 Out IP 147.78.195.254 > 194.5.220.43: ICMP echo reply, id 89, seq 1, length 64 08:14:49.380340 net1 In IP 194.5.220.43 > 147.78.195.254: ICMP echo request, id 89, seq 2, length 64 08:14:49.380392 net2 Out IP 147.78.195.254 > 194.5.220.43: ICMP echo reply, id 89, seq 2, length 64
2) ping to 195.141.200.73
[9:14] nb3:~% ping -c2 195.141.200.73
PING 195.141.200.73 (195.141.200.73) 56(84) bytes of data. 64 bytes from 195.141.200.73: icmp_seq=1 ttl=53 time=11.3 ms 64 bytes from 195.141.200.73: icmp_seq=2 ttl=53 time=6.81 ms
--- 195.141.200.73 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 6.813/9.057/11.301/2.244 ms
[9:15] nb3:~%
/ # tcpdump -ni any host 194.5.220.43
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
08:16:19.257697 net2 In IP 194.5.220.43 > 195.141.200.73: ICMP echo request, id 91, seq 1, length 64 08:16:19.257730 net2 Out IP 195.141.200.73 > 194.5.220.43: ICMP echo reply, id 91, seq 1, length 64 08:16:20.250948 net2 In IP 194.5.220.43 > 195.141.200.73: ICMP echo request, id 91, seq 2, length 64 08:16:20.250980 net2 Out IP 195.141.200.73 > 194.5.220.43: ICMP echo reply, id 91, seq 2, length 64
3) http to 147.78.195.254
[9:16] nb3:~% curl -s 147.78.195.254 > /dev/null ; echo $?
0
/ # tcpdump -ni any host 194.5.220.43
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
08:17:04.082945 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [S], seq 1405408358, win 64240, options [mss 1460,sackOK,TS val 1380610701 ecr 0,nop,wscale 7], length 0 08:17:04.082983 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [S.], seq 3790092363, ack 1405408359, win 65160, options [mss 1460,sackOK,TS val 520503591 ecr 1380610701,nop,wscale 7], length 0 08:17:04.089996 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 1380610709 ecr 520503591], length 0 08:17:04.090121 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [P.], seq 1:79, ack 1, win 502, options [nop,nop,TS val 1380610709 ecr 520503591], length 78: HTTP: GET / HTTP/1.1 08:17:04.090136 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [.], ack 79, win 509, options [nop,nop,TS val 520503598 ecr 1380610709], length 0 08:17:04.090301 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [P.], seq 1:239, ack 79, win 509, options [nop,nop,TS val 520503598 ecr 1380610709], length 238: HTTP: HTTP/1.1 200 OK 08:17:04.090381 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [P.], seq 239:854, ack 79, win 509, options [nop,nop,TS val 520503598 ecr 1380610709], length 615: HTTP 08:17:04.096058 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [.], ack 239, win 501, options [nop,nop,TS val 1380610715 ecr 520503598], length 0 08:17:04.096059 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [.], ack 854, win 497, options [nop,nop,TS val 1380610715 ecr 520503598], length 0 08:17:04.096339 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [F.], seq 79, ack 854, win 501, options [nop,nop,TS val 1380610715 ecr 520503598], length 0 08:17:04.096450 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [F.], seq 854, ack 80, win 509, options [nop,nop,TS val 520503604 ecr 1380610715], length 0 08:17:04.102609 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [.], ack 855, win 501, options [nop,nop,TS val 1380610721 ecr 520503604], length 0
4) http to 195.141.200.73
[9:17] nb3:~% curl -s 195.141.200.73 > /dev/null ; echo $? 0
/ # tcpdump -ni any host 194.5.220.43
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
08:18:05.951066 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [S], seq 1556080700, win 64240, options [mss 1460,sackOK,TS val 765965336 ecr 0,nop,wscale 7], length 0 08:18:05.951106 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [S.], seq 3465881361, ack 1556080701, win 65160, options [mss 1460,sackOK,TS val 3168643538 ecr 765965336,nop,wscale 7], length 0 08:18:05.958699 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 765965342 ecr 3168643538], length 0 08:18:05.958749 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [P.], seq 1:79, ack 1, win 502, options [nop,nop,TS val 765965342 ecr 3168643538], length 78: HTTP: GET / HTTP/1.1 08:18:05.958763 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [.], ack 79, win 509, options [nop,nop,TS val 3168643545 ecr 765965342], length 0 08:18:05.959216 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [P.], seq 1:239, ack 79, win 509, options [nop,nop,TS val 3168643546 ecr 765965342], length 238: HTTP: HTTP/1.1 200 OK 08:18:05.959327 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [P.], seq 239:854, ack 79, win 509, options [nop,nop,TS val 3168643546 ecr 765965342], length 615: HTTP 08:18:05.965244 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [.], ack 239, win 501, options [nop,nop,TS val 765965350 ecr 3168643546], length 0 08:18:05.965348 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [.], ack 854, win 497, options [nop,nop,TS val 765965350 ecr 3168643546], length 0 08:18:05.965487 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [F.], seq 79, ack 854, win 501, options [nop,nop,TS val 765965350 ecr 3168643546], length 0 08:18:05.965573 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [F.], seq 854, ack 80, win 509, options [nop,nop,TS val 3168643552 ecr 765965350], length 0 08:18:05.971916 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [.], ack 855, win 501, options [nop,nop,TS val 765965356 ecr 3168643552], length 0
[0]
wireguard "server" that changes the source ip:
/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
3: eth0@if29: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 66:4a:9c:12:5b:6c brd ff:ff:ff:ff:ff:ff
inet6 2a0a:e5c0:10:1e:7f21:83ca:a7d:46d2/128 scope global
valid_lft forever preferred_lft forever
inet6 fe80::644a:9cff:fe12:5b6c/64 scope link
valid_lft forever preferred_lft forever
4: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 3c:ec:ef:cb:d8:1b brd ff:ff:ff:ff:ff:ff
inet 147.78.195.254/27 brd 147.78.195.255 scope global net1
valid_lft forever preferred_lft forever
inet6 2a0a:e5c0:1:8::53/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::3eec:efff:fecb:d81b/64 scope link
valid_lft forever preferred_lft forever
5: v1477819464: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN qlen 1000
link/[65534]
inet 147.78.194.65/26 scope global v1477819464
valid_lft forever preferred_lft forever
inet6 2a0a:e5c0:2e::1/64 scope global
valid_lft forever preferred_lft forever
26: net2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000
link/ether 3c:ec:ef:cb:d8:1c brd ff:ff:ff:ff:ff:ff
inet 195.141.200.73/31 scope global net2
valid_lft forever preferred_lft forever
inet6 2001:1700:3500:2::12/124 scope global
valid_lft forever preferred_lft forever
inet6 fe80::3eec:efff:fecb:d81c/64 scope link
valid_lft forever preferred_lft forever/ #
wireguard client behind nat:
nb3:/etc/wireguard# curl -4 ifconfig.io
194.5.220.43
nb3:/etc/wireguard# ip a sh dev wlan0
2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 84:5c:f3:ed:52:9c brd ff:ff:ff:ff:ff:ff
inet 192.168.4.85/24 brd 192.168.4.255 scope global dynamic noprefixroute wlan0
valid_lft 317sec preferred_lft 242sec
inet6 2a0a:e5c0:13:0:865c:f3ff:feed:529c/64 scope global dynamic mngtmpaddr noprefixroute
valid_lft 86394sec preferred_lft 14394sec
inet6 fe80::865c:f3ff:feed:529c/64 scope link
valid_lft forever preferred_lft forever
nb3:/etc/wireguard#[1]
/ # ip route get 194.5.220.43
194.5.220.43 via 195.141.200.72 dev net2 src 195.141.200.73 / #
Quoted 14 lines · Mike O'Connor
Generally all OSs will if sending from a local process will use the address of the outgoing interface for the packet.
If the packet is forwarded and no NAT is used the address will be routed via the interface suggested by the routing table.
So local routing can be a real pain, policy based routing is an option. The other option could be to setup an 'output' NAT to an address which is multi-homed.
I have a system running which is multi-homed with out issue other than the actual routing machine. This machine is BGP connected to three locations.
Quoted 47 lines
There is no NAT setup and because I also add the wireguard link addresses to the BGP sessions.
Cheers
Quoted 39 lines · Nico Schottelius
Dear group,
I was wondering how wireguard [Linux kernel] or wireguard-go [FreeBSD] are supposed to decide which IP address to use for replying?
I have seen both on FreeBSD and Linux that wireguard seems to use the IP address of the outgoing interface, i.e. the one with the route returning to the sender. However in multi homed situations, this can be wrong, let's take this example:
19:57:24.607526 net1 In IP 194.5.220.43.60770 > 147.78.195.254.51820: UDP, length 148 19:57:24.608358 net2 Out IP 195.141.200.73.51820 > 194.5.220.43.60770: UDP, length 92The initiator sends from 194.5.220.43 to the receiver 147.78.195.254. Wireguard then replies with the source IP of 195.141.200.73 instead of 147.78.195.254.
As the node is multi homed, the packet might leave through any of its uplinks and thus return with a random (unexpected) IP address and will not pass NAT rules on firewalls and finally be dropped. F.i. in above example the firewall drops the packet from 195.141.200.73, because there is no session entry for that.
I have observed this behaviour both on Linux 6.1.11 as well as wireguard-go 0.0.20220316_8,1 on FreeBSD and in both cases the connection will break depending on which active interface is taken as exit.
I would argue that wireguard should by default invert the IP addresses, i.e. switch dst=src, src=dst and then reply with that, instead of adapting an interface specific address, or is there a good reason for the current behaviour?
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
--
Sustainable and modern Infrastructures by ungleich.ch
Have you tried setting the preferred src address of the route(s) to the addresses you desire?
From "man ip":
Quoted 1 line
src ADDRESS the source address to prefer when sending to the destinations covered by the route prefix.
Quoted 20 lines · Nico Schottelius
Let me rephrase the problem statement:
- ping and http calls to the multi homed machine work correctly:
I can ping 147.78.195.254 and the reply contains the same address. I can ping 195.141.200.73 and the reply contains the same address. I can curl 147.78.195.254 and the reply contains the same address. I can curl 195.141.200.73 and the reply contains the same address.- wireguard does NOT work because it changes the reply address:
A packet sent to 147.78.195.254 is being replied with 195.141.200.73In general, processes reply with the IP address that was used to contact them and not with the outgoing interface address, which would also break adding IP addresses to the loopback interface.
For full detail, see ip addresses [0] and routing below [1] and tests executed [2].
I believe that this is a bug in wireguard.
Hello Mikma,
Quoted 5 lines · Mikma
Have you tried setting the preferred src address of the route(s) to the addresses you desire?
From "man ip":
Quoted 1 line
src ADDRESS the source address to prefer when sending to the destinations covered by the route prefix.
unfortunately this does not solve the problem. The expected behaviour of wireguard is to reply with the same IP address, like nginx and the kernel ICMP handler do, not with a route based outgoing interface IP address.
In a BGP based environment the route can vary dynamically and I showed a stripped down version to make it easier to understand. In practices, many of our systems have 4-7 different upstreams and the packet can come in on any interface and should leave the machine on the current correct interface depending on the route import.
In no case however, wireguard should change the response address, because this breaks stateful firewalls.
As demonstrated in my last email, both the in-kernel ICMP handler as well as user space applications like nginx behave correctly on the same machine.
I briefly checked the wireguard source code and I did not right away spot the network handling part that sets the source IP, so I am wondering if this bug is due to wireguard not handling it at all?
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
Aside from nginx + icmp being handled correctly as a reference, I want to further elaborate on this case to show that something is really wrong with the current behaviour:
A typical scenario for routers is to have a lot of global reachable IP addresses (IPv6, IPv4) assigned to the loopback interface, such as this system:
[13:11] router2.place6:~# ip a sh dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 2a0a:e5c0:1e:a::b/128 scope global
valid_lft forever preferred_lft forever
inet6 2a0a:e5c0:1e:a::a/128 scope global
valid_lft forever preferred_lft forever
inet6 2a0a:e5c0:2:a::b/128 scope global
valid_lft forever preferred_lft forever
inet6 2a0a:e5c0:2:a::a/128 scope global
valid_lft forever preferred_lft forever
inet6 2a0a:e5c0:2:1::7/128 scope global
valid_lft forever preferred_lft forever
inet6 2a0a:e5c0:2:1::6/128 scope global
valid_lft forever preferred_lft forever
inet6 2a0a:e5c0:2:1::5/128 scope global
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft foreverThe motivation behind that is that independent of the actual routing interface, these IP addresses are always reachable.
Now in the case of wireguard selecting the source IP based on the outgoing interface, this is never going to work, as lo cannot send packets to the outside world.
Quoted 168 lines · Nico Schottelius
Let me rephrase the problem statement:
- ping and http calls to the multi homed machine work correctly:
I can ping 147.78.195.254 and the reply contains the same address. I can ping 195.141.200.73 and the reply contains the same address. I can curl 147.78.195.254 and the reply contains the same address. I can curl 195.141.200.73 and the reply contains the same address.- wireguard does NOT work because it changes the reply address:
A packet sent to 147.78.195.254 is being replied with 195.141.200.73In general, processes reply with the IP address that was used to contact them and not with the outgoing interface address, which would also break adding IP addresses to the loopback interface.
For full detail, see ip addresses [0] and routing below [1] and tests executed [2].
I believe that this is a bug in wireguard.
--------------------------------------------------------------------------------
[2]
Let's see how it looks like in detail:
1) ping to 147.78.195.254: works
[9:14] nb3:~% ping -c2 147.78.195.254
PING 147.78.195.254 (147.78.195.254) 56(84) bytes of data. 64 bytes from 147.78.195.254: icmp_seq=1 ttl=53 time=7.27 ms 64 bytes from 147.78.195.254: icmp_seq=2 ttl=53 time=6.30 ms--- 147.78.195.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 6.296/6.781/7.267/0.485 ms/ # tcpdump -ni any host 194.5.220.43
tcpdump: data link typeLINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-typeLINUX_SLL2(Linux cooked v2), snapshot length 262144 bytes08:14:48.379618 net1 In IP 194.5.220.43 > 147.78.195.254: ICMP echo request, id 89, seq 1, length 64 08:14:48.379651 net2 Out IP 147.78.195.254 > 194.5.220.43: ICMP echo reply, id 89, seq 1, length 64 08:14:49.380340 net1 In IP 194.5.220.43 > 147.78.195.254: ICMP echo request, id 89, seq 2, length 64 08:14:49.380392 net2 Out IP 147.78.195.254 > 194.5.220.43: ICMP echo reply, id 89, seq 2, length 642) ping to 195.141.200.73
[9:14] nb3:~% ping -c2 195.141.200.73
PING 195.141.200.73 (195.141.200.73) 56(84) bytes of data. 64 bytes from 195.141.200.73: icmp_seq=1 ttl=53 time=11.3 ms 64 bytes from 195.141.200.73: icmp_seq=2 ttl=53 time=6.81 ms--- 195.141.200.73 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 6.813/9.057/11.301/2.244 ms
[9:15] nb3:~%
/ # tcpdump -ni any host 194.5.220.43
tcpdump: data link typeLINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-typeLINUX_SLL2(Linux cooked v2), snapshot length 262144 bytes08:16:19.257697 net2 In IP 194.5.220.43 > 195.141.200.73: ICMP echo request, id 91, seq 1, length 64 08:16:19.257730 net2 Out IP 195.141.200.73 > 194.5.220.43: ICMP echo reply, id 91, seq 1, length 64 08:16:20.250948 net2 In IP 194.5.220.43 > 195.141.200.73: ICMP echo request, id 91, seq 2, length 64 08:16:20.250980 net2 Out IP 195.141.200.73 > 194.5.220.43: ICMP echo reply, id 91, seq 2, length 643) http to 147.78.195.254
[9:16] nb3:~% curl -s 147.78.195.254 >
/dev/null; echo $?
0
/ # tcpdump -ni any host 194.5.220.43
tcpdump: data link typeLINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-typeLINUX_SLL2(Linux cooked v2), snapshot length 262144 bytes08:17:04.082945 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [S], seq 1405408358, win 64240, options [mss 1460,sackOK,TS val 1380610701 ecr 0,nop,wscale 7], length 0 08:17:04.082983 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [S.], seq 3790092363, ack 1405408359, win 65160, options [mss 1460,sackOK,TS val 520503591 ecr 1380610701,nop,wscale 7], length 0 08:17:04.089996 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 1380610709 ecr 520503591], length 0 08:17:04.090121 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [P.], seq 1:79, ack 1, win 502, options [nop,nop,TS val 1380610709 ecr 520503591], length 78: HTTP: GET / HTTP/1.1 08:17:04.090136 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [.], ack 79, win 509, options [nop,nop,TS val 520503598 ecr 1380610709], length 0 08:17:04.090301 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [P.], seq 1:239, ack 79, win 509, options [nop,nop,TS val 520503598 ecr 1380610709], length 238: HTTP: HTTP/1.1 200 OK 08:17:04.090381 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [P.], seq 239:854, ack 79, win 509, options [nop,nop,TS val 520503598 ecr 1380610709], length 615: HTTP 08:17:04.096058 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [.], ack 239, win 501, options [nop,nop,TS val 1380610715 ecr 520503598], length 0 08:17:04.096059 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [.], ack 854, win 497, options [nop,nop,TS val 1380610715 ecr 520503598], length 0 08:17:04.096339 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [F.], seq 79, ack 854, win 501, options [nop,nop,TS val 1380610715 ecr 520503598], length 0 08:17:04.096450 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [F.], seq 854, ack 80, win 509, options [nop,nop,TS val 520503604 ecr 1380610715], length 0 08:17:04.102609 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [.], ack 855, win 501, options [nop,nop,TS val 1380610721 ecr 520503604], length 04) http to 195.141.200.73
[9:17] nb3:~% curl -s 195.141.200.73 >
/dev/null; echo $? 0/ # tcpdump -ni any host 194.5.220.43
tcpdump: data link typeLINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-typeLINUX_SLL2(Linux cooked v2), snapshot length 262144 bytes08:18:05.951066 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [S], seq 1556080700, win 64240, options [mss 1460,sackOK,TS val 765965336 ecr 0,nop,wscale 7], length 0 08:18:05.951106 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [S.], seq 3465881361, ack 1556080701, win 65160, options [mss 1460,sackOK,TS val 3168643538 ecr 765965336,nop,wscale 7], length 0 08:18:05.958699 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 765965342 ecr 3168643538], length 0 08:18:05.958749 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [P.], seq 1:79, ack 1, win 502, options [nop,nop,TS val 765965342 ecr 3168643538], length 78: HTTP: GET / HTTP/1.1 08:18:05.958763 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [.], ack 79, win 509, options [nop,nop,TS val 3168643545 ecr 765965342], length 0 08:18:05.959216 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [P.], seq 1:239, ack 79, win 509, options [nop,nop,TS val 3168643546 ecr 765965342], length 238: HTTP: HTTP/1.1 200 OK 08:18:05.959327 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [P.], seq 239:854, ack 79, win 509, options [nop,nop,TS val 3168643546 ecr 765965342], length 615: HTTP 08:18:05.965244 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [.], ack 239, win 501, options [nop,nop,TS val 765965350 ecr 3168643546], length 0 08:18:05.965348 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [.], ack 854, win 497, options [nop,nop,TS val 765965350 ecr 3168643546], length 0 08:18:05.965487 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [F.], seq 79, ack 854, win 501, options [nop,nop,TS val 765965350 ecr 3168643546], length 0 08:18:05.965573 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [F.], seq 854, ack 80, win 509, options [nop,nop,TS val 3168643552 ecr 765965350], length 0 08:18:05.971916 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [.], ack 855, win 501, options [nop,nop,TS val 765965356 ecr 3168643552], length 0[0]
wireguard "server" that changes the source ip:/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if29: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP link/ether 66:4a:9c:12:5b:6c brd ff:ff:ff:ff:ff:ff inet6 2a0a:e5c0:10:1e:7f21:83ca:a7d:46d2/128 scope global valid_lft forever preferred_lft forever inet6 fe80::644a:9cff:fe12:5b6c/64 scope link valid_lft forever preferred_lft forever 4: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 3c:ec:ef:cb:d8:1b brd ff:ff:ff:ff:ff:ff inet 147.78.195.254/27 brd 147.78.195.255 scope global net1 valid_lft forever preferred_lft forever inet6 2a0a:e5c0:1:8::53/64 scope global valid_lft forever preferred_lft forever inet6 fe80::3eec:efff:fecb:d81b/64 scope link valid_lft forever preferred_lft forever 5: v1477819464: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN qlen 1000 link/[65534] inet 147.78.194.65/26 scope global v1477819464 valid_lft forever preferred_lft forever inet6 2a0a:e5c0:2e::1/64 scope global valid_lft forever preferred_lft forever 26: net2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 3c:ec:ef:cb:d8:1c brd ff:ff:ff:ff:ff:ff inet 195.141.200.73/31 scope global net2 valid_lft forever preferred_lft forever inet6 2001:1700:3500:2::12/124 scope global valid_lft forever preferred_lft forever inet6 fe80::3eec:efff:fecb:d81c/64 scope link valid_lft forever preferred_lft forever/ #
wireguard client behind nat:
nb3:/etc/wireguard# curl -4 ifconfig.io 194.5.220.43 nb3:/etc/wireguard# ip a sh dev wlan0 2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 84:5c:f3:ed:52:9c brd ff:ff:ff:ff:ff:ff inet 192.168.4.85/24 brd 192.168.4.255 scope global dynamic noprefixroute wlan0 valid_lft 317sec preferred_lft 242sec inet6 2a0a:e5c0:13:0:865c:f3ff:feed:529c/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 86394sec preferred_lft 14394sec inet6 fe80::865c:f3ff:feed:529c/64 scope link valid_lft forever preferred_lft forever nb3:/etc/wireguard#
Quoted 69 lines
[1]
/ # ip route get 194.5.220.43
194.5.220.43 via 195.141.200.72 dev net2 src 195.141.200.73 / #Quoted 61 lines · Mike O'Connor
Generally all OSs will if sending from a local process will use the address of the outgoing interface for the packet.
If the packet is forwarded and no NAT is used the address will be routed via the interface suggested by the routing table.
So local routing can be a real pain, policy based routing is an option. The other option could be to setup an 'output' NAT to an address which is multi-homed.
I have a system running which is multi-homed with out issue other than the actual routing machine. This machine is BGP connected to three locations.
There is no NAT setup and because I also add the wireguard link addresses to the BGP sessions.
Cheers
Quoted 39 lines · Nico Schottelius
Dear group,
I was wondering how wireguard [Linux kernel] or wireguard-go [FreeBSD] are supposed to decide which IP address to use for replying?
I have seen both on FreeBSD and Linux that wireguard seems to use the IP address of the outgoing interface, i.e. the one with the route returning to the sender. However in multi homed situations, this can be wrong, let's take this example:
19:57:24.607526 net1 In IP 194.5.220.43.60770 > 147.78.195.254.51820: UDP, length 148 19:57:24.608358 net2 Out IP 195.141.200.73.51820 > 194.5.220.43.60770: UDP, length 92The initiator sends from 194.5.220.43 to the receiver 147.78.195.254. Wireguard then replies with the source IP of 195.141.200.73 instead of 147.78.195.254.
As the node is multi homed, the packet might leave through any of its uplinks and thus return with a random (unexpected) IP address and will not pass NAT rules on firewalls and finally be dropped. F.i. in above example the firewall drops the packet from 195.141.200.73, because there is no session entry for that.
I have observed this behaviour both on Linux 6.1.11 as well as wireguard-go 0.0.20220316_8,1 on FreeBSD and in both cases the connection will break depending on which active interface is taken as exit.
I would argue that wireguard should by default invert the IP addresses, i.e. switch dst=src, src=dst and then reply with that, instead of adapting an interface specific address, or is there a good reason for the current behaviour?
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
--
Sustainable and modern Infrastructures by ungleich.ch
Hey Sebastian,
Quoted 6 lines · Sebastian Hyrwall
It is kinda. It's been mentioned multiple times over the years but no one seems to want to fix it. Atleast you should be able to specify bind/src ip in the config. I gave up WG because of it. Wasn't accepted by my projects security policy since src ip could not be configured.
There is an unofficial patch however,
https://github.com/torvalds/linux/commit/5fa98082093344c86345f9f63305cae9d5f9f281
the binding is somewhat related to this issue and I was looking for that feature some time ago, too. While it is correlated and I would really appreciate binding support, I am not sure whether the linked patch does actually fix the problem I am seeing in multi homed devices.
As long as wireguard does not reply with the same IP address it was contacted with, packets will get dropped on stateful firewalls, because the returning packet does not match the state session database.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
Hi,
I don't think no one wants to fix it, there are several users having this issue. I rather guess no one could find a suitable solution to fix it.
@Nico: did you try to delete the affected route and add it again with the correct source IP ?
as I mentioned it in https://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html
ip route del <NET>
ip route add <NET> dev <ALIAS_DEV> src <SRC_IP>
This way I was able to (at least temporary) fix this issue on multi homed systems.
Kind regards,
Christoph
Am 19.02.2023 um 13:13 schrieb Nico Schottelius:
Quoted 25 lines
Hey Sebastian,
Quoted 6 lines · Sebastian Hyrwall
It is kinda. It's been mentioned multiple times over the years but no one seems to want to fix it. Atleast you should be able to specify bind/src ip in the config. I gave up WG because of it. Wasn't accepted by my projects security policy since src ip could not be configured.
There is an unofficial patch however,
https://github.com/torvalds/linux/commit/5fa98082093344c86345f9f63305cae9d5f9f281
the binding is somewhat related to this issue and I was looking for that feature some time ago, too. While it is correlated and I would really appreciate binding support, I am not sure whether the linked patch does actually fix the problem I am seeing in multi homed devices.
As long as wireguard does not reply with the same IP address it was contacted with, packets will get dropped on stateful firewalls, because the returning packet does not match the state session database.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
Without getting into the debate of whether wireguard is acting correctly or not, I think there is a possible workaround.
1. In the iptables mangle table PREROUTING, match the incoming interface and destination address and --set-xmark a firewall MARK unique to this interface/destination
2. Create a new ip route table that sets the default route to go out on the interface with the source address you want (same as destination address in iptables)
3. Create a new ip rule that sends all packets with firewall mark set in iptables to the routing table you just created
Repeat above for each interface/address you need to mangle, with a unique firewall mark and routing table for each.
It may be necessary to use CONNMARK in PREROUTING and OUTPUT to --restore_mark. I can't remember if this is needed or not, its been a while since I configured iptables with this.
This should ensure that any packet that comes into an interface/address is replied to from the same interface/address.
David
Quoted 43 lines · Christoph Loesch
Hi,
I don't think no one wants to fix it, there are several users having this issue. I rather guess no one could find a suitable solution to fix it.
@Nico: did you try to delete the affected route and add it again with the correct source IP ?
as I mentioned it in https://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html
ip route del <NET>
ip route add <NET> dev <ALIAS_DEV> src <SRC_IP>This way I was able to (at least temporary) fix this issue on multi homed systems.
Kind regards,
ChristophAm 19.02.2023 um 13:13 schrieb Nico Schottelius:
Quoted 25 lines
Hey Sebastian,
Quoted 6 lines · Sebastian Hyrwall
It is kinda. It's been mentioned multiple times over the years but no one seems to want to fix it. Atleast you should be able to specify bind/src ip in the config. I gave up WG because of it. Wasn't accepted by my projects security policy since src ip could not be configured.
There is an unofficial patch however,
https://github.com/torvalds/linux/commit/5fa98082093344c86345f9f63305cae9d5f9f281
the binding is somewhat related to this issue and I was looking for that feature some time ago, too. While it is correlated and I would really appreciate binding support, I am not sure whether the linked patch does actually fix the problem I am seeing in multi homed devices.
As long as wireguard does not reply with the same IP address it was contacted with, packets will get dropped on stateful firewalls, because the returning packet does not match the state session database.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
You should get into that debate. Proposing firewall workarounds is not a correct solution so please don't do it. It needs to be fixed. It's an immature VPN solution that always just proposed a workaround instead of fixing the problem. It seems to be designed by people that are good at software and cryptography but has no clue about networking stacks.
Quoted 68 lines · David Kerr
Without getting into the debate of whether wireguard is acting correctly or not, I think there is a possible workaround.
1. In the iptables mangle table PREROUTING, match the incoming interface and destination address and --set-xmark a firewall MARK unique to this interface/destination
2. Create a new ip route table that sets the default route to go out on the interface with the source address you want (same as destination address in iptables)
3. Create a new ip rule that sends all packets with firewall mark set in iptables to the routing table you just createdRepeat above for each interface/address you need to mangle, with a unique firewall mark and routing table for each.
It may be necessary to use CONNMARK in PREROUTING and OUTPUT to --restore_mark. I can't remember if this is needed or not, its been a while since I configured iptables with this.
This should ensure that any packet that comes into an interface/address is replied to from the same interface/address.
David
Quoted 42 lines · Christoph Loesch
Hi,
I don't think no one wants to fix it, there are several users having this issue. I rather guess no one could find a suitable solution to fix it.
@Nico: did you try to delete the affected route and add it again with the correct source IP ?
as I mentioned it in https://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html
ip route del <NET>
ip route add <NET> dev <ALIAS_DEV> src <SRC_IP>This way I was able to (at least temporary) fix this issue on multi homed systems.
Kind regards,
ChristophAm 19.02.2023 um 13:13 schrieb Nico Schottelius:
Quoted 25 lines
Hey Sebastian,
Quoted 6 lines · Sebastian Hyrwall
It is kinda. It's been mentioned multiple times over the years but no one seems to want to fix it. Atleast you should be able to specify bind/src ip in the config. I gave up WG because of it. Wasn't accepted by my projects security policy since src ip could not be configured.
There is an unofficial patch however,
https://github.com/torvalds/linux/commit/5fa98082093344c86345f9f63305cae9d5f9f281
the binding is somewhat related to this issue and I was looking for that feature some time ago, too. While it is correlated and I would really appreciate binding support, I am not sure whether the linked patch does actually fix the problem I am seeing in multi homed devices.
As long as wireguard does not reply with the same IP address it was contacted with, packets will get dropped on stateful firewalls, because the returning packet does not match the state session database.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
FWIW, while clever, I don't think that iptables mark solves all cases. E.g., consider an interface with multiple addresses, where a packet comes in on a secondary address. The proposed rule would send it out the right interface, but still with the wrong (primary) address picked from the interface...
With IPv6 it's common to assign an address to a service rather than a host so services can move easily. So multiple addresses per interface are the rule, not the exception.
I do the same with IPv4 inside addresses, though these days public IPv4 addresses are scarce enough that it's not common for public IPs. It amounts to the same issue - the NAT tracking is stateful.
Trying to work around this with routing seems like a maze of twisty passages - so I agree that the right solution is for WG to respond from the address that receives a packet.
Quoted 68 lines · David Kerr
Without getting into the debate of whether wireguard is acting correctly or not, I think there is a possible workaround.
1. In the iptables mangle table PREROUTING, match the incoming interface and destination address and --set-xmark a firewall MARK unique to this interface/destination
2. Create a new ip route table that sets the default route to go out on the interface with the source address you want (same as destination address in iptables)
3. Create a new ip rule that sends all packets with firewall mark set in iptables to the routing table you just createdRepeat above for each interface/address you need to mangle, with a unique firewall mark and routing table for each.
It may be necessary to use CONNMARK in PREROUTING and OUTPUT to --restore_mark. I can't remember if this is needed or not, its been a while since I configured iptables with this.
This should ensure that any packet that comes into an interface/address is replied to from the same interface/address.
David
Quoted 42 lines · Christoph Loesch
Hi,
I don't think no one wants to fix it, there are several users having this issue. I rather guess no one could find a suitable solution to fix it.
@Nico: did you try to delete the affected route and add it again with the correct source IP ?
as I mentioned it inhttps://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html
ip route del <NET>
ip route add <NET> dev <ALIAS_DEV> src <SRC_IP>This way I was able to (at least temporary) fix this issue on multi homed systems.
Kind regards,
ChristophAm 19.02.2023 um 13:13 schrieb Nico Schottelius:
Quoted 25 lines
Hey Sebastian,
Quoted 6 lines · Sebastian Hyrwall
It is kinda. It's been mentioned multiple times over the years but no one seems to want to fix it. Atleast you should be able to specify bind/src ip in the config. I gave up WG because of it. Wasn't accepted by my projects security policy since src ip could not be configured.
There is an unofficial patch however,
https://github.com/torvalds/linux/commit/5fa98082093344c86345f9f63305cae9d5f9f281
the binding is somewhat related to this issue and I was looking for that feature some time ago, too. While it is correlated and I would really appreciate binding support, I am not sure whether the linked patch does actually fix the problem I am seeing in multi homed devices.
As long as wireguard does not reply with the same IP address it was contacted with, packets will get dropped on stateful firewalls, because the returning packet does not match the state session database.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
Den sön 19 feb. 2023 kl 18:06 skrev Sebastian Hyrvall <[email protected]>:
Quoted 5 lines
You should get into that debate. Proposing firewall workarounds is not a correct solution so please don't do it. It needs to be fixed. It's an immature VPN solution that always just proposed a workaround instead of fixing the problem.
I would make sure that you are not mis-ascribing the problem* to "an immature VPN" and not what the default UDP behaviour of the kernel is, to pick a working interface to send packets from based on the routing table, in which any/all udp based tunnel would suffer the same problem. If you google it, you may find that other udp transports face the same "problem".
*) https://en.wiktionary.org/wiki/Chesterton%27s_fence
Signature · 1 line
May the most significant bit of your life be positive.
It is the default behavior of the kernel. But all networking software dealing in security knows how to correctly behave. You are welcome to inform me of something else suffering the same problem.
Quoted 14 lines · Janne Johansson
Den sön 19 feb. 2023 kl 18:06 skrev Sebastian Hyrvall <[email protected]>:
Quoted 4 lines
You should get into that debate. Proposing firewall workarounds is not a correct solution so please don't do it. It needs to be fixed. It's an immature VPN solution that always just proposed a workaround instead of fixing the problem.
I would make sure that you are not mis-ascribing the problem* to "an immature VPN" and not what the default UDP behaviour of the kernel is, to pick a working interface to send packets from based on the routing table, in which any/all udp based tunnel would suffer the same problem. If you google it, you may find that other udp transports face the same "problem".
My proposed workaround specifically stated to match on both the interface and destination address, and to set a route with both interface and [source] address. This allows for multiple IP addresses on the same interface -- which you can do with both IPv4 and IPv6.
But yes, it is a nasty hack. You really need to understand what is going on between the firewall and routing tables/rules and it is easy to get confused.
Quoted 89 lines · tlhackque
FWIW, while clever, I don't think that iptables mark solves all cases. E.g., consider an interface with multiple addresses, where a packet comes in on a secondary address. The proposed rule would send it out the right interface, but still with the wrong (primary) address picked from the interface...
With IPv6 it's common to assign an address to a service rather than a host so services can move easily. So multiple addresses per interface are the rule, not the exception.
I do the same with IPv4 inside addresses, though these days public IPv4 addresses are scarce enough that it's not common for public IPs. It amounts to the same issue - the NAT tracking is stateful.
Trying to work around this with routing seems like a maze of twisty passages - so I agree that the right solution is for WG to respond from the address that receives a packet.
Quoted 68 lines · David Kerr
Without getting into the debate of whether wireguard is acting correctly or not, I think there is a possible workaround.
1. In the iptables mangle table PREROUTING, match the incoming interface and destination address and --set-xmark a firewall MARK unique to this interface/destination
2. Create a new ip route table that sets the default route to go out on the interface with the source address you want (same as destination address in iptables)
3. Create a new ip rule that sends all packets with firewall mark set in iptables to the routing table you just createdRepeat above for each interface/address you need to mangle, with a unique firewall mark and routing table for each.
It may be necessary to use CONNMARK in PREROUTING and OUTPUT to --restore_mark. I can't remember if this is needed or not, its been a while since I configured iptables with this.
This should ensure that any packet that comes into an interface/address is replied to from the same interface/address.
David
Quoted 42 lines · Christoph Loesch
Hi,
I don't think no one wants to fix it, there are several users having this issue. I rather guess no one could find a suitable solution to fix it.
@Nico: did you try to delete the affected route and add it again with the correct source IP ?
as I mentioned it inhttps://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html
ip route del <NET>
ip route add <NET> dev <ALIAS_DEV> src <SRC_IP>This way I was able to (at least temporary) fix this issue on multi homed systems.
Kind regards,
ChristophAm 19.02.2023 um 13:13 schrieb Nico Schottelius:
Quoted 25 lines
Hey Sebastian,
Quoted 6 lines · Sebastian Hyrwall
It is kinda. It's been mentioned multiple times over the years but no one seems to want to fix it. Atleast you should be able to specify bind/src ip in the config. I gave up WG because of it. Wasn't accepted by my projects security policy since src ip could not be configured.
There is an unofficial patch however,
https://github.com/torvalds/linux/commit/5fa98082093344c86345f9f63305cae9d5f9f281
the binding is somewhat related to this issue and I was looking for that feature some time ago, too. While it is correlated and I would really appreciate binding support, I am not sure whether the linked patch does actually fix the problem I am seeing in multi homed devices.
As long as wireguard does not reply with the same IP address it was contacted with, packets will get dropped on stateful firewalls, because the returning packet does not match the state session database.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
BTW, DNS is a common UDP (well, mostly) protocol that encountered the same issue.
See RFC 2181 <https://www.rfc-editor.org/rfc/rfc2181.html> (1997), where you'll find (emphasis added):
Quoted 32 lines
4 <https://www.rfc-editor.org/rfc/rfc2181.html#section-4>. Server Reply Source Address Selection
Most, if not all, DNS clients, expect the address from which a reply is received to be the same address as that to which the query eliciting the reply was sent. This is true for servers acting as clients for the purposes of recursive query resolution, as well as simple resolver clients. The address, along with the identifier (ID) in the reply is used for disambiguating replies, and filtering spurious responses. This may, or may not, have been intended when the DNS was designed, but is now a fact of life.
Some multi-homed hosts running DNS servers generate a reply using a source address that is not the same as the destination address from the client's request packet.
_**Such replies will be discarded by the client because the source address of the reply does not match that of a host to which the client sent the original request.** _ That is, it
appears to be an unsolicited response.4.1 <https://www.rfc-editor.org/rfc/rfc2181.html#section-4.1>. UDP Source Address Selection
***To avoid these problems, servers when responding to queries using
UDP_must_causethe reply to be sent with the source address field in the IP header set to the address that was in the destination address field of the IP header of the packet containing the query causing the response.** *If this would cause the response to be sent from an IP
address that is not permitted for this purpose, then the response may be sent from any legal IP address allocated to the server. That address should be chosen to maximise the possibility that the client will be able to use it for further queries. Servers configured in such a way that not all their addresses are equally reachable from all potential clients need take particular care when responding to queries sent to anycast, multicast, or similar, addresses.
Quoted 80 lines · tlhackque
FWIW, while clever, I don't think that iptables mark solves all cases. E.g., consider an interface with multiple addresses, where a packet comes in on a secondary address. The proposed rule would send it out the right interface, but still with the wrong (primary) address picked from the interface...
With IPv6 it's common to assign an address to a service rather than a host so services can move easily. So multiple addresses per interface are the rule, not the exception.
I do the same with IPv4 inside addresses, though these days public IPv4 addresses are scarce enough that it's not common for public IPs. It amounts to the same issue - the NAT tracking is stateful.
Trying to work around this with routing seems like a maze of twisty passages - so I agree that the right solution is for WG to respond from the address that receives a packet.
Quoted 70 lines · David Kerr
Without getting into the debate of whether wireguard is acting correctly or not, I think there is a possible workaround.
1. In the iptables mangle table PREROUTING, match the incoming interface and destination address and --set-xmark a firewall MARK unique to this interface/destination
2. Create a new ip route table that sets the default route to go out on the interface with the source address you want (same as destination address in iptables)
3. Create a new ip rule that sends all packets with firewall mark set in iptables to the routing table you just createdRepeat above for each interface/address you need to mangle, with a unique firewall mark and routing table for each.
It may be necessary to use CONNMARK in PREROUTING and OUTPUT to --restore_mark. I can't remember if this is needed or not, its been a while since I configured iptables with this.
This should ensure that any packet that comes into an interface/address is replied to from the same interface/address.
David
Quoted 44 lines · Christoph Loesch
Hi,
I don't think no one wants to fix it, there are several users having this issue. I rather guess no one could find a suitable solution to fix it.
@Nico: did you try to delete the affected route and add it again with the correct source IP ?
as I mentioned it inhttps://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html
ip route del <NET>
ip route add <NET> dev <ALIAS_DEV> src <SRC_IP>This way I was able to (at least temporary) fix this issue on multi homed systems.
Kind regards,
ChristophAm 19.02.2023 um 13:13 schrieb Nico Schottelius:
Quoted 26 lines
Hey Sebastian,
Quoted 7 lines · Sebastian Hyrwall
It is kinda. It's been mentioned multiple times over the years but no one seems to want to fix it. Atleast you should be able to specify bind/src ip in the config. I gave up WG because of it. Wasn't accepted by my projects security policy since src ip could not be configured.
There is an unofficial patch however,
https://github.com/torvalds/linux/commit/5fa98082093344c86345f9f63305cae9d5f9f281
the binding is somewhat related to this issue and I was looking for that feature some time ago, too. While it is correlated and I would really appreciate binding support, I am not sure whether the linked patch does actually fix the problem I am seeing in multi homed devices.
As long as wireguard does not reply with the same IP address it was contacted with, packets will get dropped on stateful firewalls, because the returning packet does not match the state session database.
Best regards,
Nico
Signature · 1 line
Sustainable and modern Infrastructures by ungleich.ch
Signature · 2 lines
This communication may not represent my employer's views, if any, on the matters discussed.
Quoted 4 lines · David Kerr
My proposed workaround specifically stated to match on both the interface and destination address, and to set a route with both interface and [source] address. This allows for multiple IP addresses on the same interface -- which you can do with both IPv4 and IPv6.
Fair enough. Of course, that means having a unique rule and mark for each if/destination address, which you now have to manage - and avoid conflicts with all other uses of mark. One of which is wg-quick...
"manage" includes remembering to add/remove the rule and allocate/deallocate the mark synchronously with wg-enabled IP addresses - and if wg is listening on all addresses, that means every ip address.
You can get there, but as I said, it's a maze of twisty passages and the complications of managing it pile up.
Quoted 93 lines
But yes, it is a nasty hack. You really need to understand what is going on between the firewall and routing tables/rules and it is easy to get confused.
Quoted 87 lines · tlhackque
FWIW, while clever, I don't think that iptables mark solves all cases. E.g., consider an interface with multiple addresses, where a packet comes in on a secondary address. The proposed rule would send it out the right interface, but still with the wrong (primary) address picked from the interface...
With IPv6 it's common to assign an address to a service rather than a host so services can move easily. So multiple addresses per interface are the rule, not the exception.
I do the same with IPv4 inside addresses, though these days public IPv4 addresses are scarce enough that it's not common for public IPs. It amounts to the same issue - the NAT tracking is stateful.
Trying to work around this with routing seems like a maze of twisty passages - so I agree that the right solution is for WG to respond from the address that receives a packet.
Quoted 68 lines · David Kerr
Without getting into the debate of whether wireguard is acting correctly or not, I think there is a possible workaround.
1. In the iptables mangle table PREROUTING, match the incoming interface and destination address and --set-xmark a firewall MARK unique to this interface/destination
2. Create a new ip route table that sets the default route to go out on the interface with the source address you want (same as destination address in iptables)
3. Create a new ip rule that sends all packets with firewall mark set in iptables to the routing table you just createdRepeat above for each interface/address you need to mangle, with a unique firewall mark and routing table for each.
It may be necessary to use CONNMARK in PREROUTING and OUTPUT to --restore_mark. I can't remember if this is needed or not, its been a while since I configured iptables with this.
This should ensure that any packet that comes into an interface/address is replied to from the same interface/address.
David
Quoted 42 lines · Christoph Loesch
Hi,
I don't think no one wants to fix it, there are several users having this issue. I rather guess no one could find a suitable solution to fix it.
@Nico: did you try to delete the affected route and add it again with the correct source IP ?
as I mentioned it inhttps://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html
ip route del <NET>
ip route add <NET> dev <ALIAS_DEV> src <SRC_IP>This way I was able to (at least temporary) fix this issue on multi homed systems.
Kind regards,
ChristophAm 19.02.2023 um 13:13 schrieb Nico Schottelius:
Quoted 25 lines
Hey Sebastian,
Quoted 6 lines · Sebastian Hyrwall
It is kinda. It's been mentioned multiple times over the years but no one seems to want to fix it. Atleast you should be able to specify bind/src ip in the config. I gave up WG because of it. Wasn't accepted by my projects security policy since src ip could not be configured.
There is an unofficial patch however,
https://github.com/torvalds/linux/commit/5fa98082093344c86345f9f63305cae9d5f9f281
the binding is somewhat related to this issue and I was looking for that feature some time ago, too. While it is correlated and I would really appreciate binding support, I am not sure whether the linked patch does actually fix the problem I am seeing in multi homed devices.
As long as wireguard does not reply with the same IP address it was contacted with, packets will get dropped on stateful firewalls, because the returning packet does not match the state session database.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
Signature · 2 lines
This communication may not represent my employer's views, if any, on the matters discussed.
Indeed this is how you typically set up a multihomed service (addresses on lo and then announce that using BGP or something).
If you use one of the network links directly for the service and that link network goes down (it may not even be in your AS so you may not know?) then the service is offline.
use a route-map in your bgp config to set the src address of routes to the address on lo, that works for wg :)
/Peter
Quoted 280 lines · Nico Schottelius
Aside from nginx + icmp being handled correctly as a reference, I want to further elaborate on this case to show that something is really wrong with the current behaviour:
A typical scenario for routers is to have a lot of global reachable IP addresses (IPv6, IPv4) assigned to the loopback interface, such as this system:
[13:11] router2.place6:~# ip a sh dev lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 2a0a:e5c0:1e:a::b/128 scope global valid_lft forever preferred_lft forever inet6 2a0a:e5c0:1e:a::a/128 scope global valid_lft forever preferred_lft forever inet6 2a0a:e5c0:2:a::b/128 scope global valid_lft forever preferred_lft forever inet6 2a0a:e5c0:2:a::a/128 scope global valid_lft forever preferred_lft forever inet6 2a0a:e5c0:2:1::7/128 scope global valid_lft forever preferred_lft forever inet6 2a0a:e5c0:2:1::6/128 scope global valid_lft forever preferred_lft forever inet6 2a0a:e5c0:2:1::5/128 scope global valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft foreverThe motivation behind that is that independent of the actual routing interface, these IP addresses are always reachable.
Now in the case of wireguard selecting the source IP based on the outgoing interface, this is never going to work, as lo cannot send packets to the outside world.
Quoted 168 lines · Nico Schottelius
Let me rephrase the problem statement:
- ping and http calls to the multi homed machine work correctly:
I can ping 147.78.195.254 and the reply contains the same address. I can ping 195.141.200.73 and the reply contains the same address. I can curl 147.78.195.254 and the reply contains the same address. I can curl 195.141.200.73 and the reply contains the same address.- wireguard does NOT work because it changes the reply address:
A packet sent to 147.78.195.254 is being replied with 195.141.200.73In general, processes reply with the IP address that was used to contact them and not with the outgoing interface address, which would also break adding IP addresses to the loopback interface.
For full detail, see ip addresses [0] and routing below [1] and tests executed [2].
I believe that this is a bug in wireguard.
--------------------------------------------------------------------------------
[2]
Let's see how it looks like in detail:
1) ping to 147.78.195.254: works
[9:14] nb3:~% ping -c2 147.78.195.254
PING 147.78.195.254 (147.78.195.254) 56(84) bytes of data. 64 bytes from 147.78.195.254: icmp_seq=1 ttl=53 time=7.27 ms 64 bytes from 147.78.195.254: icmp_seq=2 ttl=53 time=6.30 ms--- 147.78.195.254 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms rtt min/avg/max/mdev = 6.296/6.781/7.267/0.485 ms/ # tcpdump -ni any host 194.5.220.43
tcpdump: data link typeLINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-typeLINUX_SLL2(Linux cooked v2), snapshot length 262144 bytes08:14:48.379618 net1 In IP 194.5.220.43 > 147.78.195.254: ICMP echo request, id 89, seq 1, length 64 08:14:48.379651 net2 Out IP 147.78.195.254 > 194.5.220.43: ICMP echo reply, id 89, seq 1, length 64 08:14:49.380340 net1 In IP 194.5.220.43 > 147.78.195.254: ICMP echo request, id 89, seq 2, length 64 08:14:49.380392 net2 Out IP 147.78.195.254 > 194.5.220.43: ICMP echo reply, id 89, seq 2, length 642) ping to 195.141.200.73
[9:14] nb3:~% ping -c2 195.141.200.73
PING 195.141.200.73 (195.141.200.73) 56(84) bytes of data. 64 bytes from 195.141.200.73: icmp_seq=1 ttl=53 time=11.3 ms 64 bytes from 195.141.200.73: icmp_seq=2 ttl=53 time=6.81 ms--- 195.141.200.73 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 6.813/9.057/11.301/2.244 ms
[9:15] nb3:~%
/ # tcpdump -ni any host 194.5.220.43
tcpdump: data link typeLINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-typeLINUX_SLL2(Linux cooked v2), snapshot length 262144 bytes08:16:19.257697 net2 In IP 194.5.220.43 > 195.141.200.73: ICMP echo request, id 91, seq 1, length 64 08:16:19.257730 net2 Out IP 195.141.200.73 > 194.5.220.43: ICMP echo reply, id 91, seq 1, length 64 08:16:20.250948 net2 In IP 194.5.220.43 > 195.141.200.73: ICMP echo request, id 91, seq 2, length 64 08:16:20.250980 net2 Out IP 195.141.200.73 > 194.5.220.43: ICMP echo reply, id 91, seq 2, length 643) http to 147.78.195.254
[9:16] nb3:~% curl -s 147.78.195.254 >
/dev/null; echo $?
0
/ # tcpdump -ni any host 194.5.220.43
tcpdump: data link typeLINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-typeLINUX_SLL2(Linux cooked v2), snapshot length 262144 bytes08:17:04.082945 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [S], seq 1405408358, win 64240, options [mss 1460,sackOK,TS val 1380610701 ecr 0,nop,wscale 7], length 0 08:17:04.082983 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [S.], seq 3790092363, ack 1405408359, win 65160, options [mss 1460,sackOK,TS val 520503591 ecr 1380610701,nop,wscale 7], length 0 08:17:04.089996 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 1380610709 ecr 520503591], length 0 08:17:04.090121 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [P.], seq 1:79, ack 1, win 502, options [nop,nop,TS val 1380610709 ecr 520503591], length 78: HTTP: GET / HTTP/1.1 08:17:04.090136 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [.], ack 79, win 509, options [nop,nop,TS val 520503598 ecr 1380610709], length 0 08:17:04.090301 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [P.], seq 1:239, ack 79, win 509, options [nop,nop,TS val 520503598 ecr 1380610709], length 238: HTTP: HTTP/1.1 200 OK 08:17:04.090381 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [P.], seq 239:854, ack 79, win 509, options [nop,nop,TS val 520503598 ecr 1380610709], length 615: HTTP 08:17:04.096058 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [.], ack 239, win 501, options [nop,nop,TS val 1380610715 ecr 520503598], length 0 08:17:04.096059 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [.], ack 854, win 497, options [nop,nop,TS val 1380610715 ecr 520503598], length 0 08:17:04.096339 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [F.], seq 79, ack 854, win 501, options [nop,nop,TS val 1380610715 ecr 520503598], length 0 08:17:04.096450 net2 Out IP 147.78.195.254.80 > 194.5.220.43.39274: Flags [F.], seq 854, ack 80, win 509, options [nop,nop,TS val 520503604 ecr 1380610715], length 0 08:17:04.102609 net1 In IP 194.5.220.43.39274 > 147.78.195.254.80: Flags [.], ack 855, win 501, options [nop,nop,TS val 1380610721 ecr 520503604], length 04) http to 195.141.200.73
[9:17] nb3:~% curl -s 195.141.200.73 >
/dev/null; echo $? 0/ # tcpdump -ni any host 194.5.220.43
tcpdump: data link typeLINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode listening on any, link-typeLINUX_SLL2(Linux cooked v2), snapshot length 262144 bytes08:18:05.951066 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [S], seq 1556080700, win 64240, options [mss 1460,sackOK,TS val 765965336 ecr 0,nop,wscale 7], length 0 08:18:05.951106 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [S.], seq 3465881361, ack 1556080701, win 65160, options [mss 1460,sackOK,TS val 3168643538 ecr 765965336,nop,wscale 7], length 0 08:18:05.958699 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [.], ack 1, win 502, options [nop,nop,TS val 765965342 ecr 3168643538], length 0 08:18:05.958749 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [P.], seq 1:79, ack 1, win 502, options [nop,nop,TS val 765965342 ecr 3168643538], length 78: HTTP: GET / HTTP/1.1 08:18:05.958763 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [.], ack 79, win 509, options [nop,nop,TS val 3168643545 ecr 765965342], length 0 08:18:05.959216 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [P.], seq 1:239, ack 79, win 509, options [nop,nop,TS val 3168643546 ecr 765965342], length 238: HTTP: HTTP/1.1 200 OK 08:18:05.959327 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [P.], seq 239:854, ack 79, win 509, options [nop,nop,TS val 3168643546 ecr 765965342], length 615: HTTP 08:18:05.965244 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [.], ack 239, win 501, options [nop,nop,TS val 765965350 ecr 3168643546], length 0 08:18:05.965348 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [.], ack 854, win 497, options [nop,nop,TS val 765965350 ecr 3168643546], length 0 08:18:05.965487 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [F.], seq 79, ack 854, win 501, options [nop,nop,TS val 765965350 ecr 3168643546], length 0 08:18:05.965573 net2 Out IP 195.141.200.73.80 > 194.5.220.43.41484: Flags [F.], seq 854, ack 80, win 509, options [nop,nop,TS val 3168643552 ecr 765965350], length 0 08:18:05.971916 net2 In IP 194.5.220.43.41484 > 195.141.200.73.80: Flags [.], ack 855, win 501, options [nop,nop,TS val 765965356 ecr 3168643552], length 0[0]
wireguard "server" that changes the source ip:/ # ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 3: eth0@if29: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP link/ether 66:4a:9c:12:5b:6c brd ff:ff:ff:ff:ff:ff inet6 2a0a:e5c0:10:1e:7f21:83ca:a7d:46d2/128 scope global valid_lft forever preferred_lft forever inet6 fe80::644a:9cff:fe12:5b6c/64 scope link valid_lft forever preferred_lft forever 4: net1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 3c:ec:ef:cb:d8:1b brd ff:ff:ff:ff:ff:ff inet 147.78.195.254/27 brd 147.78.195.255 scope global net1 valid_lft forever preferred_lft forever inet6 2a0a:e5c0:1:8::53/64 scope global valid_lft forever preferred_lft forever inet6 fe80::3eec:efff:fecb:d81b/64 scope link valid_lft forever preferred_lft forever 5: v1477819464: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN qlen 1000 link/[65534] inet 147.78.194.65/26 scope global v1477819464 valid_lft forever preferred_lft forever inet6 2a0a:e5c0:2e::1/64 scope global valid_lft forever preferred_lft forever 26: net2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP qlen 1000 link/ether 3c:ec:ef:cb:d8:1c brd ff:ff:ff:ff:ff:ff inet 195.141.200.73/31 scope global net2 valid_lft forever preferred_lft forever inet6 2001:1700:3500:2::12/124 scope global valid_lft forever preferred_lft forever inet6 fe80::3eec:efff:fecb:d81c/64 scope link valid_lft forever preferred_lft forever/ #
wireguard client behind nat:
nb3:/etc/wireguard# curl -4 ifconfig.io 194.5.220.43 nb3:/etc/wireguard# ip a sh dev wlan0 2: wlan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 84:5c:f3:ed:52:9c brd ff:ff:ff:ff:ff:ff inet 192.168.4.85/24 brd 192.168.4.255 scope global dynamic noprefixroute wlan0 valid_lft 317sec preferred_lft 242sec inet6 2a0a:e5c0:13:0:865c:f3ff:feed:529c/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 86394sec preferred_lft 14394sec inet6 fe80::865c:f3ff:feed:529c/64 scope link valid_lft forever preferred_lft forever nb3:/etc/wireguard#Quoted 69 lines
[1]
/ # ip route get 194.5.220.43
194.5.220.43 via 195.141.200.72 dev net2 src 195.141.200.73 / #Quoted 61 lines · Mike O'Connor
Generally all OSs will if sending from a local process will use the address of the outgoing interface for the packet.
If the packet is forwarded and no NAT is used the address will be routed via the interface suggested by the routing table.
So local routing can be a real pain, policy based routing is an option. The other option could be to setup an 'output' NAT to an address which is multi-homed.
I have a system running which is multi-homed with out issue other than the actual routing machine. This machine is BGP connected to three locations.
There is no NAT setup and because I also add the wireguard link addresses to the BGP sessions.
Cheers
Quoted 39 lines · Nico Schottelius
Dear group,
I was wondering how wireguard [Linux kernel] or wireguard-go [FreeBSD] are supposed to decide which IP address to use for replying?
I have seen both on FreeBSD and Linux that wireguard seems to use the IP address of the outgoing interface, i.e. the one with the route returning to the sender. However in multi homed situations, this can be wrong, let's take this example:
19:57:24.607526 net1 In IP 194.5.220.43.60770 > 147.78.195.254.51820: UDP, length 148 19:57:24.608358 net2 Out IP 195.141.200.73.51820 > 194.5.220.43.60770: UDP, length 92The initiator sends from 194.5.220.43 to the receiver 147.78.195.254. Wireguard then replies with the source IP of 195.141.200.73 instead of 147.78.195.254.
As the node is multi homed, the packet might leave through any of its uplinks and thus return with a random (unexpected) IP address and will not pass NAT rules on firewalls and finally be dropped. F.i. in above example the firewall drops the packet from 195.141.200.73, because there is no session entry for that.
I have observed this behaviour both on Linux 6.1.11 as well as wireguard-go 0.0.20220316_8,1 on FreeBSD and in both cases the connection will break depending on which active interface is taken as exit.
I would argue that wireguard should by default invert the IP addresses, i.e. switch dst=src, src=dst and then reply with that, instead of adapting an interface specific address, or is there a good reason for the current behaviour?
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch--
Sustainable and modern Infrastructures by ungleich.ch
Hello Christoph,
Quoted 1 line · Christoph Loesch
@Nico: did you try to delete the affected route and add it again with the correct source IP ?
No, I did not because the routes are really dynamic on the affected systems and I would need to overwrite the BGP routes with a better metric, which in turn will likely break the return path.
Quoted 6 lines
as I mentioned it in https://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html
ip route del <NET>
ip route add <NET> dev <ALIAS_DEV> src <SRC_IP>This way I was able to (at least temporary) fix this issue on multi homed systems.
Much appreciate the hint. However changing routes manually on as many routers/vpn endpoints as we have is not a practical solution. To fix the current project's issue we have shifted the VPN endpoint to a single homed device for the moment.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
Hey Janne,
Quoted 1 line · Janne Johansson
I am happy to have learned a new principle today, thanks for that.
And to be sure that everyone is on the same page:
Wireguard should reply by default with the source address that used to be the destination address, but at the moment wireguard is not doing that at the moment.
If anyone disagrees with above statement, please let me know.
--
Sustainable and modern Infrastructures by ungleich.ch
Quoted 9 lines · tlhackque
Quoted 9 lines
[...]
4.1 <https://www.rfc-editor.org/rfc/rfc2181.html#section-4.1>. UDP Source Address Selection***To avoid these problems, servers when responding to queries
using UDP_must_causethe reply to be sent with the source address field in the IP header set to the address that was in the destination address field of the IP header of the packet containing the query causing the response.** *
OMG, we really have seen everything already, haven't we?
Jason, what do you think about adopting the RFC2181 Source Address Selection algorithm for wireguard?
If I am not mistaken that would mean in practice:
if orignal_pkg.ip_dst == one_of_my_ips then
return_pkg.ip.src = orignal_pkg.ip_dst
return_pkg.ip.dst = orignal_pkg.ip_src
fi
For me that sounds like a sane approach (aside from my very simplified algorithm).
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
Quoted 9 lines · On Sun, 19 Feb 2023 21:18:34 +0100 Nico Schottelius…
If I am not mistaken that would mean in practice:
if orignal_pkg.ip_dst ==
one_of_my_ipsthen
return_pkg.ip.src = orignal_pkg.ip_dst
return_pkg.ip.dst = orignal_pkg.ip_src
fiFor me that sounds like a sane approach (aside from my very simplified algorithm).
Except there is no request and response in WG, and as such no original or return packet. Another peer contacts you, then some time later you contact the other peer. Or the other way round.
WG-wise what will need to be done is to store in the each peer's information structure the local IP that we are supposed to use for communication with that peer; and updating it when receiving packets from the peer, using the destination of those. So you would see a "Local IP" in each "peer" section when doing a "wg show".
Also, until there is such IP initially stored, it will have to be some default outgoing IP of the system towards that peer. BTW, how would this work in your setup, what if not the peer contacts you first, but your machine needs to contact the peer?
Signature · 2 lines
With respect, Roman
Quoted 22 lines · Roman Mamedov
Quoted 9 lines · On Sun, 19 Feb 2023 21:18:34 +0100 Nico Schottelius<[email protected]…
If I am not mistaken that would mean in practice:
if orignal_pkg.ip_dst ==
one_of_my_ipsthen
return_pkg.ip.src = orignal_pkg.ip_dst
return_pkg.ip.dst = orignal_pkg.ip_src
fiFor me that sounds like a sane approach (aside from my very simplified algorithm).
Except there is no request and response in WG, and as such no original or return packet. Another peer contacts you, then some time later you contact the other peer. Or the other way round.
WG-wise what will need to be done is to store in the each peer's information structure the local IP that we are supposed to use for communication with that peer; and updating it when receiving packets from the peer, using the destination of those. So you would see a "Local IP" in each "peer" section when doing a "wg show".
Quoted 5 lines
Also, until there is such IP initially stored, it will have to be some default outgoing IP of the system towards that peer. BTW, how would this work in your setup, what if not the peer contacts you first, but your machine needs to contact the peer?
The situation can be (and often is) the same for both peers.
If you're the initiator, you send to the peer address using its configured or DNS IP address, and normal routing. You note the address used to send, and use it for future communications to that peer. The first packet sets state in the posited firewall/nat. Subsequent packets using the same source address ensures that the firewall sees them as the same flow.
When the peer gets around to saying something - which it will at latest when the keepalive timer goes off, but probably sooner, it will have noted your source address and it's local IP address (the one you used). So it will send using the source address that you know about.
This is the same algorithm used by the peer, so they should agree.
When either end detects and address change, the process restarts.
There is a possibility that the initial packets pass in flight, but I think that would at most result in a dropped packet, which will be resent.
I don't think there's a deadlock, but in the event of thrashing, a tie-breaker of using the lowest candidate IP address generally works..
When there are multiple choices, it doesn't really matter which pair of IP addresses are picked, as long as they're stable while the systems reside on the same networks. (E.G. it could be two notebook PCs in different hotel rooms, not just two fixed servers or one fixed server and one mobile.) The goal is to establish a flow that stateful packet inspection, NAT, routing can recognize and use to keep a pinhole open...
I don't have time at the moment to work out the corner cases, but that's the overall approach.
Hey Roman,
Quoted 17 lines · Roman Mamedov
Quoted 9 lines · On Sun, 19 Feb 2023 21:18:34 +0100 Nico Schottelius…
If I am not mistaken that would mean in practice:
if orignal_pkg.ip_dst ==
one_of_my_ipsthen
return_pkg.ip.src = orignal_pkg.ip_dst
return_pkg.ip.dst = orignal_pkg.ip_src
fiFor me that sounds like a sane approach (aside from my very simplified algorithm).
Except there is no request and response in WG, and as such no original or return packet. Another peer contacts you, then some time later you contact the other peer. Or the other way round.
Quoted 5 lines
WG-wise what will need to be done is to store in the each peer's information structure the local IP that we are supposed to use for communication with that peer; and updating it when receiving packets from the peer, using the destination of those. So you would see a "Local IP" in each "peer" section when doing a "wg show".
That is very interesting, thanks for the insight. Reading above paragraph, I was having a very similar thought that we need to record the local IP.
Quoted 4 lines
Also, until there is such IP initially stored, it will have to be some default outgoing IP of the system towards that peer. BTW, how would this work in your setup, what if not the peer contacts you first, but your machine needs to contact the peer?
So far this situation doesn't exist for us, because only servers are multi homed.
However, having an option to specify something a local address in each peer section would probably be a good solution to disambiguate it and if not specified, use the default, as in whatever other processes are using that don't define it explicitly - i.e. follow the process of least surprise.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
Quoted 2 lines · Nico Schottelius
So far this situation doesn't exist for us, because only servers are multi homed.
It's not that uncommon; consider a docked notebook that has a WiFi address and an Ethernet address on the same subnet.
While typically the routing priorities favor the Ethernet, the mobile will have both addresses.
In a car, you can have WiFi thru the car and mobile data. (Not saying I like this, but ..)
There are probably other cases, but I wouldn't assume it's only a server issue.
As I also noted in another note: two servers can have the same issue, if both are multi-homed.
The solution really needs to be symmetric.
Actually in my case (I'm not the originator of this thread), I don't run BGP. But I do have both site-site and mobile-site clients. Much simpler environment, but same issue.
I do understand UDP.
As I've noted, DNS UDP has the same issue, and an RFC was issued to clarify that responses MUST come from the address on which a query is received.
WG isn't quite the same, as it isn't a request/response protocol. But it is a flow between two endpoints, and NAT/firewalls will open a pinhole for incoming packets when they see an outbound packet.
One of the nice things about WG is that except for this issue, it has no dependencies on custom routing (or anything but UDP) and "just works". It should "just work" on multihomed hosts, without handstands, BGP routing, different ports, and the like. It also needs to work where it's not feasible to layer on work-arounds, such as VPSs where you don't get to pick your kernel...or your firewall.
Picking stable endpoint addresses would make the traffic look like the kind of flow that these middleboxes recognize, and things would "just work".
Quoted 125 lines · John Lauro
I think the ip route with src would work, but only as a short lived work around. The problem with that is if dealing with dynamic routes is it could go a way when a link is down and then come back and the src setting would be lost. You would need the bgp software to add the src.
UDP is connectionless. Sending back out the same as it's coming in isn't strictly the same. The streams are not attached the same as they would be with TCP on nginx or a reply with icmp. You should be able to whitelist the udp port on the NAT devices, as it shouldn't use state info.
I am not sure if you are attempting to do site to site or client to server/site and which end has the NAT (or both). What I do for site to site is use a different port for each connection and have a separate BGP connection for each possible connection (ie: different one for different network providers). Have a full mesh with 8 sites and upto 3 providers per site.
That said, you probably have floating IPs on the client side, and don't want to lock in a single IP on the multi-homed server side? You could nat the incoming IPs on the border from an internal IP and then then lock to a single private IP on the wireguard server for in/out and that border nat would force the reply back to the same gateway it came in from.
I know, you don't want work arounds, just want to mention it's not the same as comparing a single stream to something that handles routing though it. As you are doing bgp and redundant routes I assume you also reset
rp_filteron all nat/wireguard/routers so the routers will allow packets to come from different sources.On Sun, Feb 19, 2023 at 12:07 PM tlhackque <[email protected]> wrote:
FWIW, while clever, I don't think that iptables mark solves all cases.
E.g., consider an interface with multiple addresses, where a packet comes in on a secondary address. The proposed rule would send it out the right interface, but still with the wrong (primary) address picked
from the interface...With IPv6 it's common to assign an address to a service rather than a host so services can move easily. So multiple addresses per interface
are the rule, not the exception.I do the same with IPv4 inside addresses, though these days public IPv4
addresses are scarce enough that it's not common for public IPs. It amounts to the same issue - the NAT tracking is stateful.Trying to work around this with routing seems like a maze of twisty passages - so I agree that the right solution is for WG to respond from
the address that receives a packet.On 19-Feb-23 11:32, David Kerr wrote:
> Without getting into the debate of whether wireguard is acting > correctly or not, I think there is a possible workaround. >
> 1. In the iptables mangle table PREROUTING, match the incoming > interface and destination address and --set-xmark a firewall MARK > unique to this interface/destination
> 2. Create a new ip route table that sets the default route to go out > on the interface with the source address you want (same as destination
> address in iptables)
> 3. Create a new ip rule that sends all packets with firewall mark set
> in iptables to the routing table you just created
>
> Repeat above for each interface/address you need to mangle, with a > unique firewall mark and routing table for each.
>
> It may be necessary to use CONNMARK in PREROUTING and OUTPUT to > --restore_mark. I can't remember if this is needed or not, its been a
> while since I configured iptables with this.
>
> This should ensure that any packet that comes into an
> interface/address is replied to from the same interface/address. >
> David
>
>
> On Sun, Feb 19, 2023 at 9:44 AM Christoph Loesch<[email protected]> wrote:
>> Hi,
>>
>> I don't think no one wants to fix it, there are several users having this issue. I rather guess no one could find a suitable solution to fix it.
>>
>> @Nico: did you try to delete the affected route and add it again with the correct source IP ?
>>
>> as I mentioned it inhttps://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html
<http://lists.zx2c4.com/pipermail/wireguard/2021-November/007324.html>
>>
>> ip route del <NET>
>> ip route add <NET> dev <ALIAS_DEV> src <SRC_IP>
>>
>> This way I was able to (at least temporary) fix this issue on multi homed systems.
>>
>> Kind regards,
>> Christoph
>>
>> Am 19.02.2023 um 13:13 schrieb Nico Schottelius:
>>> Hey Sebastian,
>>>
>>> Sebastian Hyrwall<[email protected]> writes:
>>>
>>>> It is kinda. It's been mentioned multiple times over the years but no one seems to want to fix it. Atleast you should be able to specify bind/src ip in the
>>>> config. I gave up WG because of it. Wasn't accepted by my projects security policy since src ip could not be configured. >>>>
>>>> There is an unofficial patch however,
>>>>
>>>>
https://github.com/torvalds/linux/commit/5fa98082093344c86345f9f63305cae9d5f9f281
>>> the binding is somewhat related to this issue and I was looking for that
>>> feature some time ago, too. While it is correlated and I would really
>>> appreciate binding support, I am not sure whether the linked patch does
>>> actually fix the problem I am seeing in multi homed devices. >>>
>>> As long as wireguard does not reply with the same IP address it was
>>> contacted with, packets will get dropped on stateful firewalls, because
>>> the returning packet does not match the state session database. >>>
>>> Best regards,
>>>
>>> Nico
>>>
>>> --
>>> Sustainable and modern Infrastructures by ungleich.ch
<http://ungleich.ch>
Src addr code review (Was: Source IP incorrect on multi homed systems)
In reply to Nico SchotteliusHi,
I though it might be useful to do some quick and dirty code review instead of speculating wildly to figure out where these source IP selection problems could be coming from ;)
From previous code deep dives I know the udp_tunnel_xmit_skb function is where tunnel packets get handed off to the kernel. So in net/wireguard/socket.c:send4 we have:
udp_tunnel_xmit_skb(rt, sock, skb, fl.saddr, fl.daddr, ds, ip4_dst_hoplimit(&rt->dst), 0, fl.fl4_sport, fl.fl4_dport, false, false);
Where fl.saddr is the source address that's supposedly wrong (sometimes? I guess?) Where does that come from?
Let's look at the code (heavily culled):
struct flowi4 fl = {
.saddr = endpoint->src4.s_addr,
};
if (cache)
rt = dst_cache_get_ip4(cache, &fl.saddr);
if (!rt) {
if (unlikely(!inet_confirm_addr(sock_net(sock), NULL, 0,
fl.saddr, RT_SCOPE_HOST)))
fl.saddr = 0;
if (unlikely(endpoint->src_if4 && ((IS_ERR(rt) &&
PTR_ERR(rt) == -EINVAL) || (!IS_ERR(rt) &&
rt->dst.dev->ifindex != endpoint->src_if4))))
fl.saddr = 0;Well it's initialized from endpoint->src4.s_addr, overwritten with zero in some cases, which I believe lets the kernel do it's regular source addr selection, and populated from something called dst_cache at some callsites.
@Nico could it perhaps simply be that you're hitting one of these zero'ing cases and that's why it's using regular kernel src addr selection instead of the cached endpoint src4 address?
The first case !inet_confirm_addr(..., RT_SCOPE_HOST) ought to confirm that the saddr is actually still a local address. Makes sens if the address we remembered was removed from the interface we can't use it anymore.
The second case looks like it's checking if the (sometimes cached) src_if4 interface index is still what the route we're about to use points to.
If neither of those seem likely we can keep reading :)
--Daniel
Re: Src addr code review (Was: Source IP incorrect on multi homed systems)
Hi all,
I've hacked that source code myself months ago, and it works well on my use case (I have 4 dual stack pppoe wan set on my openwrt router, and seted a wireguard sever on it), my hack will pickup the dst_addr from incoming handshake packet in kernel sk_buff, and then use that addr as src_addr to reply.
I'm not good at source code, and I know that my hack may be ugly, but it works, hope this patch can help:
https://github.com/openwrt/packages/issues/9538#issuecomment-1150592803
Quoted 55 lines · Daniel Gröber <[email protected]> 于2023年2月20日周一 06:42写道:
Hi,
I though it might be useful to do some quick and dirty code review instead of speculating wildly to figure out where these source IP selection problems could be coming from ;)
From previous code deep dives I know the
udp_tunnel_xmit_skbfunction is where tunnel packets get handed off to the kernel. So in net/wireguard/socket.c:send4 we have:udp_tunnel_xmit_skb(rt, sock, skb, fl.saddr, fl.daddr, ds, ip4_dst_hoplimit(&rt->dst), 0, fl.fl4_sport, fl.fl4_dport, false, false);Where fl.saddr is the source address that's supposedly wrong (sometimes? I guess?) Where does that come from?
Let's look at the code (heavily culled):
struct flowi4 fl = { .saddr = endpoint->src4.s_addr, }; if (cache) rt = dst_cache_get_ip4(cache, &fl.saddr); if (!rt) { if (unlikely(!inet_confirm_addr(sock_net(sock), NULL, 0, fl.saddr, RT_SCOPE_HOST))) fl.saddr = 0; if (unlikely(endpoint->src_if4 && ((IS_ERR(rt) && PTR_ERR(rt) == -EINVAL) || (!IS_ERR(rt) && rt->dst.dev->ifindex != endpoint->src_if4)))) fl.saddr = 0;Well it's initialized from
endpoint->src4.s_addr, overwritten with zero in some cases, which I believe lets the kernel do it's regular source addr selection, and populated from something calleddst_cacheat some callsites.@Nico could it perhaps simply be that you're hitting one of these zero'ing cases and that's why it's using regular kernel src addr selection instead of the cached endpoint src4 address?
The first case !inet_confirm_addr(...,
RT_SCOPE_HOST) ought to confirm that the saddr is actually still a local address. Makes sens if the address we remembered was removed from the interface we can't use it anymore.The second case looks like it's checking if the (sometimes cached)
src_if4interface index is still what the route we're about to use points to.If neither of those seem likely we can keep reading :)
--Daniel
Yes, wg is not a request/response protocol. But it does have some state. Can't wireguard remember the last local address that each peer sent traffic? It is just like the tracking already in use for peer ip address. If there is an "last address" it would be nice if we could hint the kernel to use that as the source address, with a fallback to the current behavior if the address is not available. It might solve a couple of problems. I just don't know if it is possible to hint the source address without enforcing it. It not, wg would have to deal with cases when the address is gone.
Regards,
Luiz
Re: Src addr code review (Was: Source IP incorrect on multi homed systems)
In reply to Daniel GröberHey Daniel,
thanks a lot for diving in ...
Quoted 7 lines · Daniel Gröber
Let's look at the code (heavily culled):
struct flowi4 fl = { .saddr = endpoint->src4.s_addr, }; if (cache) rt = dst_cache_get_ip4(cache, &fl.saddr);
What I am wondering is, how did it get into the cache in the first place?
Quoted 5 lines
[...]
@Nico could it perhaps simply be that you're hitting one of these zero'ing cases and that's why it's using regular kernel src addr selection instead of the cached endpoint src4 address?
That could absolutely be the case. What is funky is that I see the problem on two very different systems, but maybe it's a good time to elaborate on this:
- System A:
- Wireguard module loaded on the host
- Wireguard wg-quick used within a kubernetes pods that has permissions for managing wireguard
- The same pod also runs bird for BGP peering
- System B:
- Wireguard running as wireguard-go on OpnSense / FreeBSD
- BGP running with frr
Both systems exhibit the behaviour, but maybe it's better to focus on System A first, as this seems to be more the "upstream" source.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
Re: Src addr code review (Was: Source IP incorrect on multi homed systems)
In reply to 曹煜Hello 曹煜,
on github it seems your patch was applied / the issue was closed - is that the correct current status?
Best regards,
Nico
Quoted 10 lines · 曹煜
Hi all,
I've hacked that source code myself months ago, and it works well on my use case (I have 4 dual stack pppoe wan set on my openwrt router, and seted a wireguard sever on it), my hack will pickup thedst_addrfrom incoming handshake packet in kernelsk_buff, and then use that addr assrc_addrto reply.
I'm not good at source code, and I know that my hack may be ugly, but it works, hope this patch can help:
https://github.com/openwrt/packages/issues/9538#issuecomment-1150592803
Quoted 56 lines
Quoted 55 lines · Daniel Gröber <[email protected]> 于2023年2月20日周一 06:42写道:
Hi,
I though it might be useful to do some quick and dirty code review instead of speculating wildly to figure out where these source IP selection problems could be coming from ;)
From previous code deep dives I know the
udp_tunnel_xmit_skbfunction is where tunnel packets get handed off to the kernel. So in net/wireguard/socket.c:send4 we have:udp_tunnel_xmit_skb(rt, sock, skb, fl.saddr, fl.daddr, ds, ip4_dst_hoplimit(&rt->dst), 0, fl.fl4_sport, fl.fl4_dport, false, false);Where fl.saddr is the source address that's supposedly wrong (sometimes? I guess?) Where does that come from?
Let's look at the code (heavily culled):
struct flowi4 fl = { .saddr = endpoint->src4.s_addr, }; if (cache) rt = dst_cache_get_ip4(cache, &fl.saddr); if (!rt) { if (unlikely(!inet_confirm_addr(sock_net(sock), NULL, 0, fl.saddr, RT_SCOPE_HOST))) fl.saddr = 0; if (unlikely(endpoint->src_if4 && ((IS_ERR(rt) && PTR_ERR(rt) == -EINVAL) || (!IS_ERR(rt) && rt->dst.dev->ifindex != endpoint->src_if4)))) fl.saddr = 0;Well it's initialized from
endpoint->src4.s_addr, overwritten with zero in some cases, which I believe lets the kernel do it's regular source addr selection, and populated from something calleddst_cacheat some callsites.@Nico could it perhaps simply be that you're hitting one of these zero'ing cases and that's why it's using regular kernel src addr selection instead of the cached endpoint src4 address?
The first case !inet_confirm_addr(...,
RT_SCOPE_HOST) ought to confirm that the saddr is actually still a local address. Makes sens if the address we remembered was removed from the interface we can't use it anymore.The second case looks like it's checking if the (sometimes cached)
src_if4interface index is still what the route we're about to use points to.If neither of those seem likely we can keep reading :)
--Daniel
--
Sustainable and modern Infrastructures by ungleich.ch
Re: Src addr code review (Was: Source IP incorrect on multi homed systems)
Hi Nico,
That issue was closed by myself, but the patch didn't get applied cause the issue was came from wireguard itself, and the maintener told me that I should send my patch to wireguard upstream (but I just gave up for sending it to wireguard team).
Quoted 83 lines · Nico Schottelius <[email protected]> 于2023年2月20日周一 18:41写道:
Hello 曹煜,
on github it seems your patch was applied / the issue was closed - is that the correct current status?
Best regards,
Nico
Quoted 10 lines · 曹煜
Hi all,
I've hacked that source code myself months ago, and it works well on my use case (I have 4 dual stack pppoe wan set on my openwrt router, and seted a wireguard sever on it), my hack will pickup thedst_addrfrom incoming handshake packet in kernelsk_buff, and then use that addr assrc_addrto reply.
I'm not good at source code, and I know that my hack may be ugly, but it works, hope this patch can help:
https://github.com/openwrt/packages/issues/9538#issuecomment-1150592803Quoted 56 lines
Quoted 55 lines · Daniel Gröber <[email protected]> 于2023年2月20日周一 06:42写道:
Hi,
I though it might be useful to do some quick and dirty code review instead of speculating wildly to figure out where these source IP selection problems could be coming from ;)
From previous code deep dives I know the
udp_tunnel_xmit_skbfunction is where tunnel packets get handed off to the kernel. So in net/wireguard/socket.c:send4 we have:udp_tunnel_xmit_skb(rt, sock, skb, fl.saddr, fl.daddr, ds, ip4_dst_hoplimit(&rt->dst), 0, fl.fl4_sport, fl.fl4_dport, false, false);Where fl.saddr is the source address that's supposedly wrong (sometimes? I guess?) Where does that come from?
Let's look at the code (heavily culled):
struct flowi4 fl = { .saddr = endpoint->src4.s_addr, }; if (cache) rt = dst_cache_get_ip4(cache, &fl.saddr); if (!rt) { if (unlikely(!inet_confirm_addr(sock_net(sock), NULL, 0, fl.saddr, RT_SCOPE_HOST))) fl.saddr = 0; if (unlikely(endpoint->src_if4 && ((IS_ERR(rt) && PTR_ERR(rt) == -EINVAL) || (!IS_ERR(rt) && rt->dst.dev->ifindex != endpoint->src_if4)))) fl.saddr = 0;Well it's initialized from
endpoint->src4.s_addr, overwritten with zero in some cases, which I believe lets the kernel do it's regular source addr selection, and populated from something calleddst_cacheat some callsites.@Nico could it perhaps simply be that you're hitting one of these zero'ing cases and that's why it's using regular kernel src addr selection instead of the cached endpoint src4 address?
The first case !inet_confirm_addr(...,
RT_SCOPE_HOST) ought to confirm that the saddr is actually still a local address. Makes sens if the address we remembered was removed from the interface we can't use it anymore.The second case looks like it's checking if the (sometimes cached)
src_if4interface index is still what the route we're about to use points to.If neither of those seem likely we can keep reading :)
--Daniel
--
Sustainable and modern Infrastructures by ungleich.ch
I think the ip route with src would work, but only as a short lived work around. The problem with it is if dealing with dynamic routes is it could go a way when a link is down and then come back and the src setting would be lost. You would need the bgp software to add the src.
UDP is connectionless. Sending back out the same as it's coming in isn't strictly the same. The streams are not attached the same as they would be with TCP on nginx or a reply with icmp. You should be able to whitelist the udp port on the NAT devices, as it shouldn't use state info.
I am not sure if you are attempting to do site to site or client to server/site and which end has the NAT (or both). What I do for site to site is use a different port for each connection and have a separate BGP connection for each possible connection (ie: different one for different network providers). Have a full mesh with 8 sites and upto 3 providers per site.
That said, you probably have floating IPs on the client side, and don't want to lock in a single IP on the multi-homed server side? You could nat the incoming IPs on the border from an internal IP and then then lock to a single private IP on the wireguard server for in/out and that border nat would force the reply back to the same gateway it came in from.
I know, you don't want work arounds, just want to mention it's not the same as comparing a single stream to something that handles routing though it. As you are doing bgp and redundant routes I assume you also reset rp_filter on all nat/wireguard/routers so the routers will allow packets to come from different sources.
Quoted 18 lines · tlhackque
FWIW, while clever, I don't think that iptables mark solves all cases. E.g., consider an interface with multiple addresses, where a packet comes in on a secondary address. The proposed rule would send it out the right interface, but still with the wrong (primary) address picked from the interface...
With IPv6 it's common to assign an address to a service rather than a host so services can move easily. So multiple addresses per interface are the rule, not the exception.
I do the same with IPv4 inside addresses, though these days public IPv4 addresses are scarce enough that it's not common for public IPs. It amounts to the same issue - the NAT tracking is stateful.
Trying to work around this with routing seems like a maze of twisty passages - so I agree that the right solution is for WG to respond from the address that receives a packet.
Re: Src addr code review (Was: Source IP incorrect on multi homed systems)
In reply to Nico SchotteliusHi Nico,
Quoted 10 lines · Nico Schottelius
Quoted 7 lines · Daniel Gröber
Let's look at the code (heavily culled):
struct flowi4 fl = { .saddr = endpoint->src4.s_addr, }; if (cache) rt = dst_cache_get_ip4(cache, &fl.saddr);What I am wondering is, how did it get into the cache in the first place?
Right so, endpoint->src4 is set in wg_socket_set_peer_endpoint, which is called either trough through wg_socket_endpoint_from_skb in the handshake receive code or wg_socket_set_peer_endpoint in the data path.
The _from_skb variant also calls wg_socket_endpoint_from_skb. Here we're remembering the src addr of the (received) packet in addr4 and the dst addr we're going to use for sending as src4 as you'd expect:
endpoint->addr4.sin_family = AF_INET; endpoint->addr4.sin_port = udp_hdr(skb)->source; endpoint->addr4.sin_addr.s_addr = ip_hdr(skb)->saddr; endpoint->src4.s_addr = ip_hdr(skb)->daddr; endpoint->src_if4 = skb->skb_iif;
The dst_cache is set just after those zero'ing conditionals we were looking at before. It's cleared whenever the endpoint/port changes or one of those cases is hit. Note the dst_cache is only used for data packets, so handshakes would be unaffected if it was the cause of your woes.
Quoted 9 lines
Quoted 3 lines
@Nico could it perhaps simply be that you're hitting one of these zero'ing cases and that's why it's using regular kernel src addr selection instead of the cached endpoint src4 address?
That could absolutely be the case. What is funky is that I see the problem on two very different systems
Both systems exhibit the behaviour, but maybe it's better to focus on System A first, as this seems to be more the "upstream" source.
It is weird indeed, but yeah. One thing at a time.
BTW, what kernel version/distro are we dealing with?
--Daniel