Wireguard uses incorrect interface - routing issue
Hello again,
I'm sorry to flood the mailing list with wireguard bugs, but it seems there is yet another routing bug in wireguard - happy to be wrong, but here are my findings:
a) system has source based routing on via ip rule:
[11:07] server141.place10:~# ip rule ls
0: from all lookup local 32765: from 192.168.1.0/24 lookup 42 32766: from all lookup main 32767: from all lookup default
[11:07] server141.place10:~# ip route sh table 42 194.5.220.0/24 via 192.168.1.254 dev eth1 proto bird metric 32 194.187.90.23 via 192.168.1.254 dev eth1 proto bird metric 32 212.103.65.231 via 192.168.1.254 dev eth1 proto bird metric 32 [11:08] server141.place10:~#
This should ensure that packets towards 194.187.90.23 travel via eth1.
b) tcpdump for verification
Using "tcpdump -ni any port 4000" I observe:
11:10:22.445638 eth0 Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148 11:10:27.447026 eth0 Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148 11:10:32.448329 eth0 Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148 11:10:37.449719 eth0 Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148
c) Route in main table
There is indeed a route in the main routing table that matches, too:
[11:08] server141.place10:~# ip r get 194.187.90.23 194.187.90.23 via 10.5.2.123 dev eth0 src 192.168.1.149 uid 0
cache
d) ip rule not working (?)
So from what I can observe it is that ip rule does not work together with wireguard / wireguard routing takes the route from main fib instead of from the separate table.
I am not sure if this is related at all to the IP address binding bug, but it appears in a similar context from our tests.
BR,
Nico
Signature · 1 line
Sustainable and modern Infrastructures by ungleich.ch
p.s.: the route lookup looks correct on the machine, when selecting the source IP:
[11:15] server141.place10:~# ip r get 194.187.90.23
194.187.90.23 via inet6 fe80::3eec:efff:fecb:d81a dev eth0 src 192.168.1.149 uid 0
cache
[11:16] server141.place10:~# ip r get 194.187.90.23 from 192.168.1.149 194.187.90.23 from 192.168.1.149 via 192.168.1.254 dev eth1 table 42 uid 0
cache
wireguard still uses the wrong interface:
11:20:13.115154 eth0 Out IP 192.168.1.149.60031 > 194.187.90.23.4000: UDP, length 148
Signature · 1 line
Sustainable and modern Infrastructures by ungleich.ch
Quoted 11 lines · Nico Schottelius
p.s.: the route lookup looks correct on the machine, when selecting the source IP:
[11:15] server141.place10:~# ip r get 194.187.90.23
194.187.90.23 via inet6 fe80::3eec:efff:fecb:d81a dev eth0 src 192.168.1.149 uid 0
cache
[11:16] server141.place10:~# ip r get 194.187.90.23 from 192.168.1.149 194.187.90.23 from 192.168.1.149 via 192.168.1.254 dev eth1 table 42 uid 0
cache
Quoted 3 lines
wireguard still uses the wrong interface:
11:20:13.115154 eth0 Out IP 192.168.1.149.60031 > 194.187.90.23.4000: UDP, length 148
I haven't looked at the details yet but this smells like the same route caching issue I found a while ago:
https://lists.zx2c4.com/pipermail/wireguard/2023-July/008111.html
Does up/down'ing the interface make the problem go away? IIRC that will re-initialize the udp socket and thus clear the route chache.
FYI Nico: It may be time to escalate these bugs to the network subsystem maintainers on [email protected] since Jason is not reading this list anymore AFAICT.
get_maintainer.pl spits out this list of emails to send To:
Jason A. Donenfeld" <[email protected]>,
"David S. Miller" <[email protected]>,
Eric Dumazet <[email protected]>,
Jakub Kicinski <[email protected]>,
Paolo Abeni <[email protected]>,
[email protected],
[email protected],
[email protected]
Do add me to CC as well. Before sending I'd recommend working out an ip-netns based reproducer script -- makes it harder to ignore the report as "ugh, too much work" ;)
Let me know if you need help with that,
--Daniel
The better approach would be to exclude the IPs from your WireGuard AllowedIPs. I always exclude IPs if I can before doing policy based routing.
https://www.procustodibus.com/blog/2021/03/wireguard-allowedips-calculator/
Jun 21, 2024, 5:15 AM by [email protected]:
Quoted 55 lines
Hello again,
I'm sorry to flood the mailing list with wireguard bugs, but it seems there is yet another routing bug in wireguard - happy to be wrong, but here are my findings:
a) system has source based routing on via ip rule:
[11:07] server141.place10:~# ip rule ls
0: from all lookup local 32765: from 192.168.1.0/24 lookup 42 32766: from all lookup main 32767: from all lookup default[11:07] server141.place10:~# ip route sh table 42 194.5.220.0/24 via 192.168.1.254 dev eth1 proto bird metric 32 194.187.90.23 via 192.168.1.254 dev eth1 proto bird metric 32 212.103.65.231 via 192.168.1.254 dev eth1 proto bird metric 32 [11:08] server141.place10:~#
This should ensure that packets towards 194.187.90.23 travel via eth1.
b) tcpdump for verification
Using "tcpdump -ni any port 4000" I observe:
11:10:22.445638 eth0 Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148 11:10:27.447026 eth0 Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148 11:10:32.448329 eth0 Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148 11:10:37.449719 eth0 Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148c) Route in main table
There is indeed a route in the main routing table that matches, too:
[11:08] server141.place10:~# ip r get 194.187.90.23 194.187.90.23 via 10.5.2.123 dev eth0 src 192.168.1.149 uid 0
cached) ip rule not working (?)
So from what I can observe it is that ip rule does not work together with wireguard / wireguard routing takes the route from main fib instead of from the separate table.
I am not sure if this is related at all to the IP address binding bug, but it appears in a similar context from our tests.
BR,
Nico
Signature · 1 line
Sustainable and modern Infrastructures by ungleich.ch
Hi,
Quoted 3 lines · Stephan von Krawczynski
... and in case you do find someone interested at all there is still the problem of no signaling to anyone when a client connects.
I hardly can remember the decade when all this was implemented in cipe.
Yeah. Can be hard to get attention on netdev, but I've been advised that when the maintainance of a (sub)subsystem is in question that is an issue they'll take notice of. So be sure to lament the fact that Jason hasn't been responding in at least a year on this ML ;)
IIRC we have a patch for netlink notifications on handshakes flying around somewhere tho. Just needs some more work.
Quoted 5 lines · Diyaa Alkanakre
The better approach would be to exclude the IPs from your WireGuard AllowedIPs. I always exclude IPs if I can before doing policy based routing.
https://www.procustodibus.com/blog/2021/03/wireguard-allowedips-calculator/
Interesting approach, thanks for the pointer :)
--Daniel
Diyaa,
this is about the *outside* tunnel IP address that wireguard uses to establish connection, not about inside routing.
BR,
Nico
Quoted 62 lines · Diyaa Alkanakre
The better approach would be to exclude the IPs from your WireGuard AllowedIPs. I always exclude IPs if I can before doing policy based routing.
https://www.procustodibus.com/blog/2021/03/wireguard-allowedips-calculator/
Jun 21, 2024, 5:15 AM by [email protected]:
Quoted 55 lines
Hello again,
I'm sorry to flood the mailing list with wireguard bugs, but it seems there is yet another routing bug in wireguard - happy to be wrong, but here are my findings:
a) system has source based routing on via ip rule:
[11:07] server141.place10:~# ip rule ls
0: from all lookup local 32765: from 192.168.1.0/24 lookup 42 32766: from all lookup main 32767: from all lookup default[11:07] server141.place10:~# ip route sh table 42 194.5.220.0/24 via 192.168.1.254 dev eth1 proto bird metric 32 194.187.90.23 via 192.168.1.254 dev eth1 proto bird metric 32 212.103.65.231 via 192.168.1.254 dev eth1 proto bird metric 32 [11:08] server141.place10:~#
This should ensure that packets towards 194.187.90.23 travel via eth1.
b) tcpdump for verification
Using "tcpdump -ni any port 4000" I observe:
11:10:22.445638 eth0 Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148 11:10:27.447026 eth0 Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148 11:10:32.448329 eth0 Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148 11:10:37.449719 eth0 Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148c) Route in main table
There is indeed a route in the main routing table that matches, too:
[11:08] server141.place10:~# ip r get 194.187.90.23 194.187.90.23 via 10.5.2.123 dev eth0 src 192.168.1.149 uid 0
cached) ip rule not working (?)
So from what I can observe it is that ip rule does not work together with wireguard / wireguard routing takes the route from main fib instead of from the separate table.
I am not sure if this is related at all to the IP address binding bug, but it appears in a similar context from our tests.
BR,
Nico
Signature · 1 line
Sustainable and modern Infrastructures by ungleich.ch
Signature · 2 lines
-- Sustainable and modern Infrastructures by ungleich.ch
Good morning Daniel,
Quoted 10 lines · Daniel Gröber
Quoted 3 lines
wireguard still uses the wrong interface:
11:20:13.115154 eth0 Out IP 192.168.1.149.60031 > 194.187.90.23.4000: UDP, length 148
I haven't looked at the details yet but this smells like the same route caching issue I found a while ago:
https://lists.zx2c4.com/pipermail/wireguard/2023-July/008111.htmlDoes up/down'ing the interface make the problem go away? IIRC that will re-initialize the udp socket and thus clear the route chache.
Up & down does *not* fix it, however a *reboot* did. I've the feeling that this is a race condition together with bird running on the machine. I suspect the following is happening:
- machine starts
- ip rule is used to move traffic into table 42 (part of the container startup)
- table 42 is populated by bird with static routes (part of bird startup)
-- at this stage wireguard works
- bird establishes iBGP sessions and receives alternate routes for the target in the main routing table
- wireguard restart is triggered and from that moment on wireguard uses the route from the main table
-- at this stage wireguard is broken/takes the route from the main table
This is so far a theory, I'll need to verify that, maybe a simple test script as you suggested makes sense.
Quoted 3 lines
FYI Nico: It may be time to escalate these bugs to the network subsystem maintainers on [email protected] since Jason is not reading this list anymore AFAICT.
That is a very good point and I shall do so next week!
Quoted 10 lines
get_maintainer.pl spits out this list of emails to send To:
Jason A. Donenfeld" <[email protected]>,
"David S. Miller" <[email protected]>,
Eric Dumazet <[email protected]>,
Jakub Kicinski <[email protected]>,
Paolo Abeni <[email protected]>,
[email protected],
[email protected],
[email protected]
Thanks for looking up!
Quoted 3 lines
Do add me to CC as well. Before sending I'd recommend working out an ip-netns based reproducer script -- makes it harder to ignore the report as "ugh, too much work" ;)
Understood and ...
Quoted 1 line
Let me know if you need help with that,
... would certainly appreciate that.
You are on matrix, too, aren't you?
I'm @nico:ungleich.ch, might be easier for coordination.
Best regards from sunny Glarus,
Nico
Signature · 2 lines
-- Sustainable and modern Infrastructures by ungleich.ch
Quoted 39 lines · On Fri, 21 Jun 2024 14:29:26 +0200 Daniel Gröber <[email protected]> wrote:
Quoted 11 lines · Nico Schottelius
p.s.: the route lookup looks correct on the machine, when selecting the source IP:
[11:15] server141.place10:~# ip r get 194.187.90.23 194.187.90.23 via inet6 fe80::3eec:efff:fecb:d81a dev eth0 src 192.168.1.149 uid 0 cache
[11:16] server141.place10:~# ip r get 194.187.90.23 from 192.168.1.149 194.187.90.23 from 192.168.1.149 via 192.168.1.254 dev eth1 table 42 uid 0
cacheQuoted 4 lines
wireguard still uses the wrong interface:
11:20:13.115154 eth0 Out IP 192.168.1.149.60031 > 194.187.90.23.4000:
UDP, length 148I haven't looked at the details yet but this smells like the same route caching issue I found a while ago:
https://lists.zx2c4.com/pipermail/wireguard/2023-July/008111.htmlDoes up/down'ing the interface make the problem go away? IIRC that will re-initialize the udp socket and thus clear the route chache.
FYI Nico: It may be time to escalate these bugs to the network subsystem maintainers on [email protected] since Jason is not reading this list anymore AFAICT.
get_maintainer.pl spits out this list of emails to send To:
Jason A. Donenfeld" <[email protected]>,
"David S. Miller" <[email protected]>,
Eric Dumazet <[email protected]>,
Jakub Kicinski <[email protected]>,
Paolo Abeni <[email protected]>,
[email protected],
[email protected],
[email protected]
Quoted 6 lines
Do add me to CC as well. Before sending I'd recommend working out an ip-netns based reproducer script -- makes it harder to ignore the report as "ugh, too much work" ;)
Let me know if you need help with that,
--Daniel
... and in case you do find someone interested at all there is still the problem of no signaling to anyone when a client connects.
I hardly can remember the decade when all this was implemented in cipe.
Signature · 2 lines
Regards, Stephan