Outgoing ping required in container environment (even with PersistentKeepalive)
Good morning,
another day news from the container land. When running wireguard in kubernetes, deleting the existing pod and replacing it with a new one, I see the following behaviour:
- The assigned IPv4 address stops being reachable (good so far)
- The assigned IPv4 address is then shortly reachable for about 5 seconds
- The assigned IPv4 address stops being reachable (not good)
- The assigned IPv4 address is again reachable, if I trigger a ping through the tunnel inside the container (ok, but why?)
I am using the following configuration:
--------------------------------------------------------------------------------
[Interface]
PrivateKey = ...
ListenPort = 51828
Address = 185.155.29.81/32
PostUp = iptables -t nat -I POSTROUTING -o ipv4 -j MASQUERADE
# upstream [Peer] Endpoint = vpn-...:51820 PublicKey = 6BRnQ+dmeFzVCH9RbM1pbJ7u3y3qrl+zUzzYCmC88kE= AllowedIPs = 0.0.0.0/1, 128.0.0.0/1 PersistentKeepalive = 25 --------------------------------------------------------------------------------
And the following container specification:
--------------------------------------------------------------------------------
spec:
containers:
- name: wireguard
image: ungleich/ungleich-wireguard:{{ $.Chart.AppVersion }} # We only support 1 listener at the moment
# Outgoing connections are not affected
ports:
- containerPort: 51820
securityContext:
capabilities:
# NET_ADMIN for wg
# NET_RAW for iptables
add: ["NET_ADMIN", "NET_RAW" ]
volumeMounts:
- name: wireguard-config
mountPath: "/etc/wireguard"
resources:
requests:
memory: {{ $v.memory | default "1Gi" }}
cpu: {{ $v.cpu | default "1000m" }}
limits:
memory: {{ $v.memory | default "1Gi" }}
cpu: {{ $v.cpu | default "1000m" }}
--------------------------------------------------------------------------------
The strange thing is that after issuing the ping once inside the container:
--------------------------------------------------------------------------------
[8:41] nb2:~% kubectl -n wireguard exec -ti wireguard-vpn-server-7db664db6f-zl4fz -- ping -c2 -4 google.com PING google.com (172.217.168.78): 56 data bytes
64 bytes from 172.217.168.78: seq=0 ttl=116 time=9.110 ms
64 bytes from 172.217.168.78: seq=1 ttl=116 time=6.664 ms
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 6.664/7.887/9.110 ms
--------------------------------------------------------------------------------
The connection stays correctly established.
If anyone has a pointer on what might be going on, any help is appreciated.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
A follow up: we "upgraded" the wireguard container to use the following entrypoint.sh instead:
--------------------------------------------------------------------------------
set -x
# Ensure everything is clean / show prior state
wg show
# Start all definitions
for conf in /etc/wireguard/*.conf; do
# Remove leftovers??
wg-quick down $conf
# Try to up and if any tunnel fails -> exit
wg-quick up "$conf" || (sleep 300; exit 1)
done
# Debug output
while true; do
# Establish tunnels, keepalive alone is not enough for some reason
ping -c2 185.203.112.1
ping -c2 2a0a:e5c0::
wg show
sleep 300done
--------------------------------------------------------------------------------
This establishes the connection reliably. I guess that should not be the case, but effectively, it is.
I was for a moment suspecting that the old container is overlapping running while the new one is created and whether there is a race condition, however two points speak against that being the source of the problem:
- Using the "Recreate strategy" in k8s, the container is first shut down. Using this, the behaviour does not change
- Even if my assumption was right, I'd expect a new handshake at some point to happen, but even minutes after restarting the container, the IPv4 address is not reachable.
Best regards,
Nico
Quoted 71 lines · Nico Schottelius
Good morning,
another day news from the container land. When running wireguard in kubernetes, deleting the existing pod and replacing it with a new one, I see the following behaviour:
- The assigned IPv4 address stops being reachable (good so far)
- The assigned IPv4 address is then shortly reachable for about 5 seconds
- The assigned IPv4 address stops being reachable (not good)
- The assigned IPv4 address is again reachable, if I trigger a ping through the tunnel inside the container (ok, but why?)I am using the following configuration:
--------------------------------------------------------------------------------
[Interface]
PrivateKey = ...
ListenPort = 51828
Address = 185.155.29.81/32
PostUp = iptables -t nat -I POSTROUTING -o ipv4 -j MASQUERADE# upstream [Peer] Endpoint = vpn-...:51820 PublicKey = 6BRnQ+dmeFzVCH9RbM1pbJ7u3y3qrl+zUzzYCmC88kE= AllowedIPs = 0.0.0.0/1, 128.0.0.0/1 PersistentKeepalive = 25 --------------------------------------------------------------------------------And the following container specification:
--------------------------------------------------------------------------------
spec:
containers:
- name: wireguard
image: ungleich/ungleich-wireguard:{{ $.Chart.AppVersion }} # We only support 1 listener at the moment
# Outgoing connections are not affected
ports:
- containerPort: 51820
securityContext:
capabilities:
#NET_ADMINfor wg
#NET_RAWfor iptables
add: ["NET_ADMIN", "NET_RAW" ]
volumeMounts:
- name: wireguard-config
mountPath: "/etc/wireguard"
resources:
requests:
memory: {{ $v.memory | default "1Gi" }}
cpu: {{ $v.cpu | default "1000m" }}
limits:
memory: {{ $v.memory | default "1Gi" }}
cpu: {{ $v.cpu | default "1000m" }}
--------------------------------------------------------------------------------The strange thing is that after issuing the ping once inside the container:
--------------------------------------------------------------------------------
[8:41] nb2:~% kubectl -n wireguard exec -ti wireguard-vpn-server-7db664db6f-zl4fz -- ping -c2 -4 google.com PING google.com (172.217.168.78): 56 data bytes
64 bytes from 172.217.168.78: seq=0 ttl=116 time=9.110 ms
64 bytes from 172.217.168.78: seq=1 ttl=116 time=6.664 ms--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 6.664/7.887/9.110 ms
--------------------------------------------------------------------------------
Quoted 8 lines
The connection stays correctly established.
If anyone has a pointer on what might be going on, any help is appreciated.
Best regards,
Nico
--
Sustainable and modern Infrastructures by ungleich.ch
Quoted 4 lines · On Sun, 08 May 2022 08:34:46 +0200 Nico Schottelius…
The connection stays correctly established.
If anyone has a pointer on what might be going on, any help is appreciated.
Maybe you don't have a corresponding firewall rule, and happen to rely on the ESTABLISHED,RELATED matching instead? This could explain the "works after a ping" behaviour. But then the PersistentKeepalive could be expected to help as well.
Signature · 2 lines
With respect, Roman