Skip to contentSkip to conversation

Wireguard uses incorrect interface - routing issue

DISCUSSION Nico Schottelius 4 participants 8 messages Jun 21, 2024 – Nov 18, 2024

lore
Share & tools
lore
#0 lore raw

Hello again,

I'm sorry to flood the mailing list with wireguard bugs, but it seems there is yet another routing bug in wireguard - happy to be wrong, but here are my findings:

a) system has source based routing on via ip rule:

[11:07] server141.place10:~# ip rule ls

0:      from all lookup local
32765:  from 192.168.1.0/24 lookup 42
32766:  from all lookup main
32767:  from all lookup default

[11:07] server141.place10:~# ip route sh table 42 194.5.220.0/24 via 192.168.1.254 dev eth1 proto bird metric 32 194.187.90.23 via 192.168.1.254 dev eth1 proto bird metric 32 212.103.65.231 via 192.168.1.254 dev eth1 proto bird metric 32 [11:08] server141.place10:~#

This should ensure that packets towards 194.187.90.23 travel via eth1.

b) tcpdump for verification

Using "tcpdump -ni any port 4000" I observe:

11:10:22.445638 eth0  Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148
11:10:27.447026 eth0  Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148
11:10:32.448329 eth0  Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148
11:10:37.449719 eth0  Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148

c) Route in main table

There is indeed a route in the main routing table that matches, too:

[11:08] server141.place10:~# ip r get 194.187.90.23 194.187.90.23 via 10.5.2.123 dev eth0 src 192.168.1.149 uid 0
cache

d) ip rule not working (?)

So from what I can observe it is that ip rule does not work together with wireguard / wireguard routing takes the route from main fib instead of from the separate table.

I am not sure if this is related at all to the IP address binding bug, but it appears in a similar context from our tests.

BR,

Nico

Signature · 1 line
Sustainable and modern Infrastructures by ungleich.ch
#1 lore raw

p.s.: the route lookup looks correct on the machine, when selecting the source IP:

[11:15] server141.place10:~# ip r get 194.187.90.23
194.187.90.23 via inet6 fe80::3eec:efff:fecb:d81a dev eth0 src 192.168.1.149 uid 0
cache
[11:16] server141.place10:~# ip r get 194.187.90.23 from 192.168.1.149 194.187.90.23 from 192.168.1.149 via 192.168.1.254 dev eth1 table 42 uid 0
cache

wireguard still uses the wrong interface:

11:20:13.115154 eth0 Out IP 192.168.1.149.60031 > 194.187.90.23.4000: UDP, length 148

Signature · 1 line
Sustainable and modern Infrastructures by ungleich.ch
#2 lore raw
Quoted 11 lines · Nico Schottelius

p.s.: the route lookup looks correct on the machine, when selecting the source IP:

[11:15] server141.place10:~# ip r get 194.187.90.23
194.187.90.23 via inet6 fe80::3eec:efff:fecb:d81a dev eth0 src 192.168.1.149 uid 0
cache
[11:16] server141.place10:~# ip r get 194.187.90.23 from 192.168.1.149 194.187.90.23 from 192.168.1.149 via 192.168.1.254 dev eth1 table 42 uid 0
cache

Quoted 3 lines

wireguard still uses the wrong interface:

11:20:13.115154 eth0 Out IP 192.168.1.149.60031 > 194.187.90.23.4000: UDP, length 148

I haven't looked at the details yet but this smells like the same route caching issue I found a while ago:
https://lists.zx2c4.com/pipermail/wireguard/2023-July/008111.html

Does up/down'ing the interface make the problem go away? IIRC that will re-initialize the udp socket and thus clear the route chache.

FYI Nico: It may be time to escalate these bugs to the network subsystem maintainers on [email protected] since Jason is not reading this list anymore AFAICT.

get_maintainer.pl spits out this list of emails to send To:

Jason A. Donenfeld" <[email protected]>,
"David S. Miller" <[email protected]>,
Eric Dumazet <[email protected]>,
Jakub Kicinski <[email protected]>,
Paolo Abeni <[email protected]>,
[email protected],
[email protected],
[email protected]

Do add me to CC as well. Before sending I'd recommend working out an ip-netns based reproducer script -- makes it harder to ignore the report as "ugh, too much work" ;)

Let me know if you need help with that,
--Daniel

#3 lore raw
Quoted 39 lines · On Fri, 21 Jun 2024 14:29:26 +0200 Daniel Gröber <[email protected]> wrote:
Quoted 11 lines · Nico Schottelius

p.s.: the route lookup looks correct on the machine, when selecting the source IP:

[11:15] server141.place10:~# ip r get 194.187.90.23 194.187.90.23 via inet6 fe80::3eec:efff:fecb:d81a dev eth0 src 192.168.1.149 uid 0 cache
[11:16] server141.place10:~# ip r get 194.187.90.23 from 192.168.1.149 194.187.90.23 from 192.168.1.149 via 192.168.1.254 dev eth1 table 42 uid 0
cache

Quoted 4 lines

wireguard still uses the wrong interface:

11:20:13.115154 eth0 Out IP 192.168.1.149.60031 > 194.187.90.23.4000:
UDP, length 148

I haven't looked at the details yet but this smells like the same route caching issue I found a while ago:
https://lists.zx2c4.com/pipermail/wireguard/2023-July/008111.html

Does up/down'ing the interface make the problem go away? IIRC that will re-initialize the udp socket and thus clear the route chache.

FYI Nico: It may be time to escalate these bugs to the network subsystem maintainers on [email protected] since Jason is not reading this list anymore AFAICT.

get_maintainer.pl spits out this list of emails to send To:

Jason A. Donenfeld" <[email protected]>,
"David S. Miller" <[email protected]>,
Eric Dumazet <[email protected]>,
Jakub Kicinski <[email protected]>,
Paolo Abeni <[email protected]>,
[email protected],
[email protected],
[email protected]

Quoted 6 lines

Do add me to CC as well. Before sending I'd recommend working out an ip-netns based reproducer script -- makes it harder to ignore the report as "ugh, too much work" ;)

Let me know if you need help with that,
--Daniel

... and in case you do find someone interested at all there is still the problem of no signaling to anyone when a client connects.
I hardly can remember the decade when all this was implemented in cipe.

Signature · 2 lines
Regards,
Stephan
#4 lore raw

Hi,

Quoted 3 lines · Stephan von Krawczynski

... and in case you do find someone interested at all there is still the problem of no signaling to anyone when a client connects.
I hardly can remember the decade when all this was implemented in cipe.

Yeah. Can be hard to get attention on netdev, but I've been advised that when the maintainance of a (sub)subsystem is in question that is an issue they'll take notice of. So be sure to lament the fact that Jason hasn't been responding in at least a year on this ML ;)

IIRC we have a patch for netlink notifications on handshakes flying around somewhere tho. Just needs some more work.

Quoted 5 lines · Diyaa Alkanakre

The better approach would be to exclude the IPs from your WireGuard AllowedIPs. I always exclude IPs if I can before doing policy based routing.

https://www.procustodibus.com/blog/2021/03/wireguard-allowedips-calculator/

Interesting approach, thanks for the pointer :)

--Daniel

#5 lore raw
In reply to Daniel Gröber

Good morning Daniel,

Quoted 10 lines · Daniel Gröber
Quoted 3 lines

wireguard still uses the wrong interface:

11:20:13.115154 eth0 Out IP 192.168.1.149.60031 > 194.187.90.23.4000: UDP, length 148

I haven't looked at the details yet but this smells like the same route caching issue I found a while ago:
https://lists.zx2c4.com/pipermail/wireguard/2023-July/008111.html

Does up/down'ing the interface make the problem go away? IIRC that will re-initialize the udp socket and thus clear the route chache.

Up & down does *not* fix it, however a *reboot* did. I've the feeling that this is a race condition together with bird running on the machine. I suspect the following is happening:

- machine starts
- ip rule is used to move traffic into table 42 (part of the container startup)
- table 42 is populated by bird with static routes (part of bird startup)

-- at this stage wireguard works

- bird establishes iBGP sessions and receives alternate routes for the target in the main routing table
- wireguard restart is triggered and from that moment on wireguard uses the route from the main table

-- at this stage wireguard is broken/takes the route from the main table

This is so far a theory, I'll need to verify that, maybe a simple test script as you suggested makes sense.

Quoted 3 lines

FYI Nico: It may be time to escalate these bugs to the network subsystem maintainers on [email protected] since Jason is not reading this list anymore AFAICT.

That is a very good point and I shall do so next week!

Quoted 10 lines

get_maintainer.pl spits out this list of emails to send To:

Jason A. Donenfeld" <[email protected]>,
"David S. Miller" <[email protected]>,
Eric Dumazet <[email protected]>,
Jakub Kicinski <[email protected]>,
Paolo Abeni <[email protected]>,
[email protected],
[email protected],
[email protected]

Thanks for looking up!

Quoted 3 lines

Do add me to CC as well. Before sending I'd recommend working out an ip-netns based reproducer script -- makes it harder to ignore the report as "ugh, too much work" ;)

Understood and ...

Quoted 1 line

Let me know if you need help with that,

... would certainly appreciate that.

You are on matrix, too, aren't you?
I'm @nico:ungleich.ch, might be easier for coordination.

Best regards from sunny Glarus,

Nico

Signature · 2 lines
-- 
Sustainable and modern Infrastructures by ungleich.ch
#6 lore raw
In reply to Nico Schottelius

The better approach would be to exclude the IPs from your WireGuard AllowedIPs. I always exclude IPs if I can before doing policy based routing.

https://www.procustodibus.com/blog/2021/03/wireguard-allowedips-calculator/

Jun 21, 2024, 5:15 AM by [email protected]:

Quoted 55 lines

Hello again,

I'm sorry to flood the mailing list with wireguard bugs, but it seems there is yet another routing bug in wireguard - happy to be wrong, but here are my findings:

a) system has source based routing on via ip rule:

[11:07] server141.place10:~# ip rule ls

0:      from all lookup local
32765:  from 192.168.1.0/24 lookup 42
32766:  from all lookup main
32767:  from all lookup default

[11:07] server141.place10:~# ip route sh table 42 194.5.220.0/24 via 192.168.1.254 dev eth1 proto bird metric 32 194.187.90.23 via 192.168.1.254 dev eth1 proto bird metric 32 212.103.65.231 via 192.168.1.254 dev eth1 proto bird metric 32 [11:08] server141.place10:~#

This should ensure that packets towards 194.187.90.23 travel via eth1.

b) tcpdump for verification

Using "tcpdump -ni any port 4000" I observe:

11:10:22.445638 eth0  Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148
11:10:27.447026 eth0  Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148
11:10:32.448329 eth0  Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148
11:10:37.449719 eth0  Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148

c) Route in main table

There is indeed a route in the main routing table that matches, too:

[11:08] server141.place10:~# ip r get 194.187.90.23 194.187.90.23 via 10.5.2.123 dev eth0 src 192.168.1.149 uid 0
cache

d) ip rule not working (?)

So from what I can observe it is that ip rule does not work together with wireguard / wireguard routing takes the route from main fib instead of from the separate table.

I am not sure if this is related at all to the IP address binding bug, but it appears in a similar context from our tests.

BR,

Nico

Signature · 1 line
Sustainable and modern Infrastructures by ungleich.ch
#7 lore raw

Diyaa,

this is about the *outside* tunnel IP address that wireguard uses to establish connection, not about inside routing.

BR,

Nico

Quoted 62 lines · Diyaa Alkanakre

The better approach would be to exclude the IPs from your WireGuard AllowedIPs. I always exclude IPs if I can before doing policy based routing.

https://www.procustodibus.com/blog/2021/03/wireguard-allowedips-calculator/

Jun 21, 2024, 5:15 AM by [email protected]:

Quoted 55 lines

Hello again,

I'm sorry to flood the mailing list with wireguard bugs, but it seems there is yet another routing bug in wireguard - happy to be wrong, but here are my findings:

a) system has source based routing on via ip rule:

[11:07] server141.place10:~# ip rule ls

0:      from all lookup local
32765:  from 192.168.1.0/24 lookup 42
32766:  from all lookup main
32767:  from all lookup default

[11:07] server141.place10:~# ip route sh table 42 194.5.220.0/24 via 192.168.1.254 dev eth1 proto bird metric 32 194.187.90.23 via 192.168.1.254 dev eth1 proto bird metric 32 212.103.65.231 via 192.168.1.254 dev eth1 proto bird metric 32 [11:08] server141.place10:~#

This should ensure that packets towards 194.187.90.23 travel via eth1.

b) tcpdump for verification

Using "tcpdump -ni any port 4000" I observe:

11:10:22.445638 eth0  Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148
11:10:27.447026 eth0  Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148
11:10:32.448329 eth0  Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148
11:10:37.449719 eth0  Out IP 192.168.1.149.58591 > 194.187.90.23.4000: UDP, length 148

c) Route in main table

There is indeed a route in the main routing table that matches, too:

[11:08] server141.place10:~# ip r get 194.187.90.23 194.187.90.23 via 10.5.2.123 dev eth0 src 192.168.1.149 uid 0
cache

d) ip rule not working (?)

So from what I can observe it is that ip rule does not work together with wireguard / wireguard routing takes the route from main fib instead of from the separate table.

I am not sure if this is related at all to the IP address binding bug, but it appears in a similar context from our tests.

BR,

Nico

Signature · 1 line
Sustainable and modern Infrastructures by ungleich.ch
Signature · 2 lines
-- 
Sustainable and modern Infrastructures by ungleich.ch