Skip to contentSkip to conversation

Outgoing ping required in container environment (even with PersistentKeepalive)

DISCUSSION Nico Schottelius 2 participants 3 messages May 8, 2022

lore
Share & tools
lore
#0 lore raw

Good morning,

another day news from the container land. When running wireguard in kubernetes, deleting the existing pod and replacing it with a new one, I see the following behaviour:

- The assigned IPv4 address stops being reachable (good so far)
- The assigned IPv4 address is then shortly reachable for about 5 seconds
- The assigned IPv4 address stops being reachable (not good)
- The assigned IPv4 address is again reachable, if I trigger a ping through the tunnel inside the container (ok, but why?)

I am using the following configuration:

--------------------------------------------------------------------------------
[Interface]
PrivateKey = ...
ListenPort = 51828
Address = 185.155.29.81/32
PostUp = iptables -t nat -I POSTROUTING -o ipv4 -j MASQUERADE

# upstream
[Peer]
Endpoint = vpn-...:51820
PublicKey = 6BRnQ+dmeFzVCH9RbM1pbJ7u3y3qrl+zUzzYCmC88kE=
AllowedIPs = 0.0.0.0/1, 128.0.0.0/1
PersistentKeepalive = 25
--------------------------------------------------------------------------------

And the following container specification:

--------------------------------------------------------------------------------
spec:
containers:
- name: wireguard
image: ungleich/ungleich-wireguard:{{ $.Chart.AppVersion }} # We only support 1 listener at the moment
# Outgoing connections are not affected
ports:
- containerPort: 51820
securityContext:
capabilities:
# NET_ADMIN for wg
# NET_RAW for iptables
add: ["NET_ADMIN", "NET_RAW" ]
volumeMounts:
- name: wireguard-config
mountPath: "/etc/wireguard"
resources:
requests:
memory: {{ $v.memory | default "1Gi" }}
cpu: {{ $v.cpu | default "1000m" }}
limits:
memory: {{ $v.memory | default "1Gi" }}
cpu: {{ $v.cpu | default "1000m" }}
--------------------------------------------------------------------------------

The strange thing is that after issuing the ping once inside the container:

--------------------------------------------------------------------------------
[8:41] nb2:~% kubectl -n wireguard exec -ti wireguard-vpn-server-7db664db6f-zl4fz -- ping -c2 -4 google.com PING google.com (172.217.168.78): 56 data bytes
64 bytes from 172.217.168.78: seq=0 ttl=116 time=9.110 ms
64 bytes from 172.217.168.78: seq=1 ttl=116 time=6.664 ms

--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 6.664/7.887/9.110 ms
--------------------------------------------------------------------------------

The connection stays correctly established.

If anyone has a pointer on what might be going on, any help is appreciated.

Best regards,

Nico

--
Sustainable and modern Infrastructures by ungleich.ch

#1 lore raw

A follow up: we "upgraded" the wireguard container to use the following entrypoint.sh instead:

--------------------------------------------------------------------------------
set -x

# Ensure everything is clean / show prior state
wg show

# Start all definitions
for conf in /etc/wireguard/*.conf; do
# Remove leftovers??
wg-quick down $conf

# Try to up and if any tunnel fails -> exit
wg-quick up "$conf" || (sleep 300; exit 1)
done

# Debug output
while true; do

    # Establish tunnels, keepalive alone is not enough for some reason
    ping -c2 185.203.112.1
    ping -c2 2a0a:e5c0::
    wg show
    sleep 300

done
--------------------------------------------------------------------------------

This establishes the connection reliably. I guess that should not be the case, but effectively, it is.

I was for a moment suspecting that the old container is overlapping running while the new one is created and whether there is a race condition, however two points speak against that being the source of the problem:

- Using the "Recreate strategy" in k8s, the container is first shut down. Using this, the behaviour does not change
- Even if my assumption was right, I'd expect a new handshake at some point to happen, but even minutes after restarting the container, the IPv4 address is not reachable.

Best regards,

Nico

Quoted 71 lines · Nico Schottelius

Good morning,

another day news from the container land. When running wireguard in kubernetes, deleting the existing pod and replacing it with a new one, I see the following behaviour:

- The assigned IPv4 address stops being reachable (good so far)
- The assigned IPv4 address is then shortly reachable for about 5 seconds
- The assigned IPv4 address stops being reachable (not good)
- The assigned IPv4 address is again reachable, if I trigger a ping through the tunnel inside the container (ok, but why?)

I am using the following configuration:

--------------------------------------------------------------------------------
[Interface]
PrivateKey = ...
ListenPort = 51828
Address = 185.155.29.81/32
PostUp = iptables -t nat -I POSTROUTING -o ipv4 -j MASQUERADE

# upstream
[Peer]
Endpoint = vpn-...:51820
PublicKey = 6BRnQ+dmeFzVCH9RbM1pbJ7u3y3qrl+zUzzYCmC88kE=
AllowedIPs = 0.0.0.0/1, 128.0.0.0/1
PersistentKeepalive = 25
--------------------------------------------------------------------------------

And the following container specification:

--------------------------------------------------------------------------------
spec:
containers:
- name: wireguard
image: ungleich/ungleich-wireguard:{{ $.Chart.AppVersion }} # We only support 1 listener at the moment
# Outgoing connections are not affected
ports:
- containerPort: 51820
securityContext:
capabilities:
# NET_ADMIN for wg
# NET_RAW for iptables
add: ["NET_ADMIN", "NET_RAW" ]
volumeMounts:
- name: wireguard-config
mountPath: "/etc/wireguard"
resources:
requests:
memory: {{ $v.memory | default "1Gi" }}
cpu: {{ $v.cpu | default "1000m" }}
limits:
memory: {{ $v.memory | default "1Gi" }}
cpu: {{ $v.cpu | default "1000m" }}
--------------------------------------------------------------------------------

The strange thing is that after issuing the ping once inside the container:

--------------------------------------------------------------------------------
[8:41] nb2:~% kubectl -n wireguard exec -ti wireguard-vpn-server-7db664db6f-zl4fz -- ping -c2 -4 google.com PING google.com (172.217.168.78): 56 data bytes
64 bytes from 172.217.168.78: seq=0 ttl=116 time=9.110 ms
64 bytes from 172.217.168.78: seq=1 ttl=116 time=6.664 ms

--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss round-trip min/avg/max = 6.664/7.887/9.110 ms
--------------------------------------------------------------------------------

Quoted 8 lines

The connection stays correctly established.

If anyone has a pointer on what might be going on, any help is appreciated.

Best regards,

Nico

--
Sustainable and modern Infrastructures by ungleich.ch

#2 lore raw
In reply to Nico Schottelius
Quoted 4 lines · On Sun, 08 May 2022 08:34:46 +0200 Nico Schottelius…

The connection stays correctly established.

If anyone has a pointer on what might be going on, any help is appreciated.

Maybe you don't have a corresponding firewall rule, and happen to rely on the ESTABLISHED,RELATED matching instead? This could explain the "works after a ping" behaviour. But then the PersistentKeepalive could be expected to help as well.

Signature · 2 lines
With respect,
Roman