Should I expect faster recovery after one side goes down

I'm not sure what is really going on but I have seen some very long delays after one side of the link goes down, while the other keeps sending packets. The work around is to restart the local side once the remote side is back up.

When I do some testing and say reboot the router the a wg tunnel terminates at, while continuing to use the laptop at the other end, after the router is back up very little traffic seems to get through or there is a very large latency. Restarting the iptables service with systemd will also hang. I don't know if that is forever or just a very long time. If I restart wireguard on the laptop (which deletes and recreates the device) things will start working normally again.

Is there some information I can collect that will illuminate what is going on here?

Hi Bruno,

The first question is - how long?

Jason

For "systemctl iptables stop" I have waited around a minute before using control C. After running "systemctl stop wireguard" or "systemctl restart wireguard" (which will delete wg0) "systemctl stop iptables" will run with no noticeable delay.

For network traffic, I waited around 10 minutes and things were still not working. Web page loads would still time out after a minute or two. But I did have a few DNS lookups succeed. I'm not sure if I did something that allowed a value to get cached (there is a local caching resolver on the affected machines) or if a response eventually made it through. After "systemctl restart wireguard" things start working normal right away. So I don't know the delay for specific traffic, but it looks to be at least a minute for most traffic. The problem does not seem to resolve for at least 10 minutes, though I don't think I have ever seen it resolve on its own.

This might be related to the amount or type of traffic backed up. The two machines where this was very noticeable in testing had all of their traffic routed through the tunnel other than the encapsulating packets. (DNS traffic gets tunnelled.) Playing with this on my work machine where only traffic destined for a few specific hosts was tunnelled, I am finding it hard to duplicate the problem.

Hello Bruno,

That's some pretty weird behavior, and it sounds like whatever the cause is is being obscured under layers of systemd. Perhaps come on into #wireguard on Freenode and we can debug this in real time? I've got a few ideas.

Jason

I don't have my laptop with me at work and breaking the wireguard tunnel to it will break my access to it from here. I could check configuration from work. I can bring it with me tomorrow, otherwise I'll probably get home too late for you tonight and I probably won't be up late enough to catch you early tomorrow.

Probably I'll be able to reproduce the issue from work.

I'm pretty sure I'm being bit by firewall rules on the router. It seems to be rejecting all of the tunnel packets and it has no reason to try to connect to the laptop the handshake never occurs again. I suspect that normally a connection established related rule lets things through. I just need to figure out how the start up packet is different so that it gets through. The systemd iptables service eventually seems to stop. Probably there is a DNS request that needs to timeout. I do some source address rewriting and it may be that the initial addresses used for the encapsulating packets are different than the ones later.
So most likely this is all on my end and not wireguard related.
Thanks for the tcpdump suggestion. I should have tried that sooner.

When I'm on the local network, 192.168.6.1 gets used for the initial source adddress and gets rewritten to 98.103.208.26 in order to make the source consistent for the laptop whether or not it is on the local network. (That way I don't need to allow connections from 192.168.6.1 somewhere else where it wouldn't be my router.) When this happens the source port seems to normally get changed. Wireguard on the laptop remembers the new source port and tries to keep using it after the router is rebooted. But during the reboot the router forgets about the port mapping so it ends up dropping the packets. It has no reason to send packets on its own to the laptop (and wouldn't know where to send them) so the port doesn't get corrected.

I think the correct fix is to know if I reboot the router for testing something, I need to also restart wireguard to make sure it is sending data to the expected port. This isn't going to be an issue in normal operation.

I found a way to make it work more automatically. The reason the port was getting reassigned was because the original connection packet was being tracked and was conflicting with the source nat mapping even though in reallity the connection was the same. By putting in CT --notrack rules I was able to block that traking and without the conflict the port doesn't get remapped. I don't need tracking or the original connection for my firewall rules so this should be OK. On testing it seems to work as expected. Now when I reboot my router, my laptop reconnects and the wireguard tunnel works without having to restart it.

Hi,

It sounds like one of these situations where persistent keepalives would be useful, doesn't it?

This way the laptop would create a new binding in your firewall.

It is definitely useful as the laptop is expected to be behind NAT, but it doesn't help with the rebooting the router breaking historical source NAT (on my local network) while the other end remembers where it last got a packet from. The solution below dealt with that.