Skip to contentSkip to conversation

Figuring out how vti works

DISCUSSION Joe M 2 participants 7 messages Sep 15, 2014 – Sep 18, 2014

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

Joe M

author
#0 lore raw

Hello Steffen Klassert,

Very sorry for this bother.

I could not figure out how vti works with ipsec and your patch was the latest to ip_vti.c. If you cannot help, please excuse me.

I cannot get the vpn traffic to get on the vti tunnel. tcpdump on vti does not show anything. I think the tunnel lookup code, for some reason, is not able to use the "vtil" tunnel.

The pings worked fine if I remove the ip_vti and ip_tunnel modules, the "mark=1" from /etc/ipsec.conf and the iptables mangle rules to set-mark.

This is with strongswan 5.2.0. Can you please help?

This is my setup on moon (master hostname)

cat /etc/ipsec.conf

# /etc/ipsec.conf - strongSwan IPsec configuration file

config setup

conn %default

        ikelifetime=60m
        keylife=20m
        rekeymargin=3m
        keyingtries=1
        keyexchange=ikev2
        mobike=no

conn master-bnglr

        leftid="C=CH, O=strongSwan, CN=master"
        leftcert=masterCert.der
        left=192.168.0.11
        leftsubnet=192.168.0.0/24
        rightid="C=CH, O=strongSwan, CN=bnglr"
        right=%any
        rightsubnet=192.168.1.0/24
        auto=add
        mark=1

sudo cat /etc/ipsec.secrets

# /etc/ipsec.secrets - strongSwan IPsec secrets file

: RSA masterKey.der

sudo ip tunnel list
vtil: ip/ip remote 192.168.1.232 local 192.168.0.11 ttl inherit ikey 0 okey 1
ip_vti0: ip/ip remote any local any ttl inherit nopmtudisc key 0

sudo ip route list
default via 192.168.0.1 dev enp4s0 metric 202
127.0.0.0/8 dev lo scope host
192.168.0.0/24 dev enp4s0 proto kernel scope link src 192.168.0.11 metric 202
192.168.1.0/24 dev vtil scope link

sudo ip xfrm policy
src 192.168.1.0/24 dst 192.168.0.0/24
dir fwd priority 2883
mark 1/0xffffffff
tmpl src <bnglr public ip> dst 192.168.0.11
proto esp reqid 2 mode tunnel
src 192.168.1.0/24 dst 192.168.0.0/24
dir in priority 2883
mark 1/0xffffffff
tmpl src <bnglr public ip> dst 192.168.0.11
proto esp reqid 2 mode tunnel
src 192.168.0.0/24 dst 192.168.1.0/24
dir out priority 2883
mark 1/0xffffffff
tmpl src 192.168.0.11 dst <bnglr public ip>
proto esp reqid 2 mode tunnel
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0

sudo ip xfrm state
src 192.168.0.11 dst <bnglr public ip>

        proto esp spi 0xc3b23fb1 reqid 2 mode tunnel
        replay-window 32 flag af-unspec
        mark 1/0xffffffff
        auth-trunc hmac(sha1) 0x33f17d71abbc9ccdbef83ecba9e1c0711c3767a0 96
        enc cbc(aes) 0xe790b24d9e9f71aec28f8ed00013f411
        encap type espinudp sport 4500 dport 8993 addr 0.0.0.0
src <bnglr public ip> dst 192.168.0.11
        proto esp spi 0xc8bcf9b0 reqid 2 mode tunnel
        replay-window 32 flag af-unspec
        mark 1/0xffffffff
        auth-trunc hmac(sha1) 0xb780288b0cf20aa7f010552837cc03a04e29198a 96
        enc cbc(aes) 0xd0db2ec7e9bb83cbc6a9d20feb6eab49
        encap type espinudp sport 8993 dport 4500 addr 0.0.0.0

I tried setting the mangle rules to set-mark but that did not help. I could not find any more documentation.

Thanks again and Sorry for the bother,
Joe

#1 lore raw
Quoted 10 lines · Joe M

Hello Steffen Klassert,

Very sorry for this bother.

I could not figure out how vti works with ipsec and your patch was the latest to ip_vti.c. If you cannot help, please excuse me.

I cannot get the vpn traffic to get on the vti tunnel. tcpdump on vti does not show anything. I think the tunnel lookup code, for some reason, is not able to use the "vtil" tunnel.

Do you know where the packets are getting dropped? netstat -i or /proc/net/xfrm_stat could give a hint.

Quoted 4 lines

The pings worked fine if I remove the ip_vti and ip_tunnel modules, the "mark=1" from /etc/ipsec.conf and the iptables mangle rules to set-mark.

You don't need to set the mark with iptables. You just have to ensure that the policy and state marks match the tunnel keys. I.e. direction in and forward must match the ikey, direction out must match the okey.

Quoted 3 lines

sudo ip tunnel list
vtil: ip/ip remote 192.168.1.232 local 192.168.0.11 ttl inherit ikey 0 okey 1

Your ikey does not match the policy and the state mark.

Quoted 10 lines

sudo ip xfrm policy
src 192.168.1.0/24 dst 192.168.0.0/24
dir fwd priority 2883
mark 1/0xffffffff
tmpl src <bnglr public ip> dst 192.168.0.11
proto esp reqid 2 mode tunnel
src 192.168.1.0/24 dst 192.168.0.0/24
dir in priority 2883
mark 1/0xffffffff

If you set mark 1 here, the tunnel should set ikey 1.

Quoted 4 lines

I tried setting the mangle rules to set-mark but that did not help. I could not find any more documentation.

Please try without setting a mark with netfilter.

author
#2 lore raw

Hello Steffen,

Thanks for responding. Sorry that it took me some time to gather all the information.

Quoted 1 line

Do you know where the packets are getting dropped?

All I can see from the below statistics is that the ip_vti0 tunnel is getting picked up instead of the vtil tunnel.

Quoted 1 line

netstat -i or /proc/net/xfrm_stat could give a hint.

master# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg

enp4s0    1500     2310      0     57 0          1974      0      0      0 BMRU
ip_vti0   1428        0      2      0 0             0     10      0      0 ORU
lo       65536      600      0      0 0           600      0      0      0 LRU
vtil      1428        0      0      0 0             0      0      0      0 OPRU
master# ip -statistics xfrm state
src 192.168.0.11 dst <client or alice or bnglr public ip>
        proto esp spi 0xc0b44648(3233039944) reqid 1(0x00000001) mode tunnel
        replay-window 32 seq 0x00000000 flag af-unspec (0x00100000)
        mark 1/0xffffffff
        auth-trunc hmac(sha1)
0x6fb52dc437eb26b65bd0dced995aa27e78a7e869 (160 bits) 96
        enc cbc(aes) 0x7e431bdc0ec138d0d8476c4afb0ccd63 (128 bits)
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 902(sec), hard 1200(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2014-09-17 17:38:06 use -
        stats:
          replay-window 0 replay 0 failed 0
src <client or alice or bnglr public ip> dst 192.168.0.11
        proto esp spi 0xc514e2d3(3306480339) reqid 1(0x00000001) mode tunnel
        replay-window 32 seq 0x00000000 flag af-unspec (0x00100000)
        mark 1/0xffffffff
        auth-trunc hmac(sha1)
0x8b4fd0749314d3656c962124e69c554ca03c9e11 (160 bits) 96
        enc cbc(aes) 0x27e2a7cae3a24c20584e841a16dcf89d (128 bits)
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 848(sec), hard 1200(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2014-09-17 17:38:05 use -
        stats:
          replay-window 0 replay 0 failed 0

master# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg

enp4s0    1500     3214      0     69 0          2886      0      0      0 BMRU
ip_vti0   1428        0      2      0 0             0     21      0      0 ORU
lo       65536      629      0      0 0           629      0      0      0 LRU
vtil      1428        0      0      0 0             0      1      0      0 OPRU

master# ping -c 1 -I 192.168.0.11 192.168.1.232
PING 192.168.1.232 (192.168.1.232) from 192.168.0.11 : 56(84) bytes of data.

--- 192.168.1.232 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

master# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg

enp4s0    1500     3217      0     69 0          2890      0      0      0 BMRU
ip_vti0   1428        0      2      0 0             0     21      0      0 ORU
lo       65536      629      0      0 0           629      0      0      0 LRU
vtil      1428        0      0      0 0             0      1      0      0 OPRU

master# ping -c 1 -I 192.168.0.11 192.168.1.232
PING 192.168.1.232 (192.168.1.232) from 192.168.0.11 : 56(84) bytes of data.

--- 192.168.1.232 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

master# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg

enp4s0    1500     3230      0     69 0          2897      0      0      0 BMRU
ip_vti0   1428        0      2      0 0             0     22      0      0 ORU
lo       65536      629      0      0 0           629      0      0      0 LRU
vtil      1428        0      0      0 0             0      1      0      0 OPRU

master# ping -c 1 -I 192.168.0.11 192.168.1.232
PING 192.168.1.232 (192.168.1.232) from 192.168.0.11 : 56(84) bytes of data.

--- 192.168.1.232 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

master# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg

enp4s0    1500     3250      0     69 0          2912      0      0      0 BMRU
ip_vti0   1428        0      2      0 0             0     22      0      0 ORU
lo       65536      629      0      0 0           629      0      0      0 LRU
vtil      1428        0      0      0 0             0      1      0      0 OPRU

master#

Quoted 1 line

From what I gather, nothing seems to be going through the vtil tunnel.

The ip_vti0 seems to get used instead.

Quoted 26 lines
Quoted 3 lines

The pings worked fine if I remove the ip_vti and ip_tunnel modules, the "mark=1" from /etc/ipsec.conf and the iptables mangle rules to set-mark.

You don't need to set the mark with iptables. You just have to ensure that the policy and state marks match the tunnel keys. I.e. direction in and forward must match the ikey, direction out must match the okey.

Quoted 3 lines

sudo ip tunnel list
vtil: ip/ip remote 192.168.1.232 local 192.168.0.11 ttl inherit ikey 0 okey 1

Your ikey does not match the policy and the state mark.

Quoted 10 lines

sudo ip xfrm policy
src 192.168.1.0/24 dst 192.168.0.0/24
dir fwd priority 2883
mark 1/0xffffffff
tmpl src <bnglr public ip> dst 192.168.0.11
proto esp reqid 2 mode tunnel
src 192.168.1.0/24 dst 192.168.0.0/24
dir in priority 2883
mark 1/0xffffffff

Quoted 9 lines

If you set mark 1 here, the tunnel should set ikey 1.

Quoted 4 lines

I tried setting the mangle rules to set-mark but that did not help. I could not find any more documentation.

Please try without setting a mark with netfilter.

I removed the iptables rules and set all policy to ACCEPT in iptables raw, nat, mangle and raw tables.

master# echo "1" | sudo tee /proc/sys/net/ipv4/ip_forward 1
master# modprobe ip_vti
master# ipsec start
Starting strongSwan 5.2.0 IPsec [starter]... master# ip tunnel add vtil mode vti local 192.168.0.11 remote 192.168.1.232 ikey 1 okey 1
master# ip link set vtil up
master# sleep 60
master# ip route add 192.168.1.0/24 dev vtil master# ip xfrm state
src 192.168.0.11 dst <client or alice or bnglr public ip>

        proto esp spi 0xc0b44648 reqid 1 mode tunnel
        replay-window 32 flag af-unspec
        mark 1/0xffffffff
        auth-trunc hmac(sha1) 0x6fb52dc437eb26b65bd0dced995aa27e78a7e869 96
        enc cbc(aes) 0x7e431bdc0ec138d0d8476c4afb0ccd63
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
src <client or alice or bnglr public ip> dst 192.168.0.11
        proto esp spi 0xc514e2d3 reqid 1 mode tunnel
        replay-window 32 flag af-unspec
        mark 1/0xffffffff
        auth-trunc hmac(sha1) 0x8b4fd0749314d3656c962124e69c554ca03c9e11 96
        enc cbc(aes) 0x27e2a7cae3a24c20584e841a16dcf89d
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0

master# ip xfrm policy
src 192.168.1.0/24 dst 192.168.0.0/24
dir fwd priority 2883
mark 1/0xffffffff
tmpl src <client or alice or bnglr public ip> dst 192.168.0.11
proto esp reqid 1 mode tunnel
src 192.168.1.0/24 dst 192.168.0.0/24
dir in priority 2883
mark 1/0xffffffff
tmpl src <client or alice or bnglr public ip> dst 192.168.0.11
proto esp reqid 1 mode tunnel
src 192.168.0.0/24 dst 192.168.1.0/24
dir out priority 2883
mark 1/0xffffffff
tmpl src 192.168.0.11 dst <client or alice or bnglr public ip>
proto esp reqid 1 mode tunnel
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0
master# ip tunnel list
ip_vti0: ip/ip remote any local any ttl inherit nopmtudisc key 0
vtil: ip/ip remote 192.168.1.232 local 192.168.0.11 ttl inherit key 1 master# ip route list
default via 192.168.0.1 dev enp4s0 metric 202
127.0.0.0/8 dev lo scope host
192.168.0.0/24 dev enp4s0 proto kernel scope link src 192.168.0.11 metric 202
192.168.1.0/24 dev vtil scope link
master# uname -a
Linux master 3.16.3 #90 SMP PREEMPT Wed Sep 17 13:39:17 CDT 2014 x86_64 Intel(R) Pentium(R) CPU G620 @ 2.60GHz GenuineIntel GNU/Linux master# ip -V
ip utility, iproute2-ss140804

sudo tcpdump -nS 'src port 500 or dst port 500 or src port 4500 or dst port 4500' -i enp4s0
Password:
error : ret -1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp4s0, link-type EN10MB (Ethernet), capture size 65535 bytes 17:56:48.651871 IP 192.168.0.11.4500 > <client or alice or bnglr public ip>.4500: isakmp-nat-keep-alive
17:56:56.934729 IP <client or alice or bnglr public ip>.4500 > 192.168.0.11.4500: isakmp-nat-keep-alive
17:57:08.652113 IP 192.168.0.11.4500 > <client or alice or bnglr public ip>.4500: isakmp-nat-keep-alive
17:57:16.934548 IP <client or alice or bnglr public ip>.4500 > 192.168.0.11.4500: isakmp-nat-keep-alive
17:57:28.652359 IP 192.168.0.11.4500 > <client or alice or bnglr public ip>.4500: isakmp-nat-keep-alive
17:57:36.938056 IP <client or alice or bnglr public ip>.4500 > 192.168.0.11.4500: isakmp-nat-keep-alive
17:57:48.652624 IP 192.168.0.11.4500 > <client or alice or bnglr public ip>.4500: isakmp-nat-keep-alive
17:57:56.935926 IP <client or alice or bnglr public ip>.4500 > 192.168.0.11.4500: isakmp-nat-keep-alive
17:58:08.652872 IP 192.168.0.11.4500 > <client or alice or bnglr public ip>.4500: isakmp-nat-keep-alive
17:58:17.005488 IP <client or alice or bnglr public ip>.4500 > 192.168.0.11.4500: isakmp-nat-keep-alive

and there is no tcpdump output on vtil interface.

master# cat /etc/ipsec.conf

# /etc/ipsec.conf - strongSwan IPsec configuration file

config setup

conn %default

        ikelifetime=60m
        keylife=20m
        rekeymargin=3m
        keyingtries=1
        keyexchange=ikev2
        mobike=no

conn master-bnglr

        leftid="C=CH, O=strongSwan, CN=master"
        leftcert=masterCert.der
        left=192.168.0.11
        leftsubnet=192.168.0.0/24
        rightid="C=CH, O=strongSwan, CN=bnglr"
        right=%any
        rightsubnet=192.168.1.0/24
        auto=add
        mark=1

Any other thoughts, please?

Thanks
Joe

author
#3 lore raw

Hello Steffen,

Checking further with printk's, I can see

vti_tunnel_init being called for both ip_vti0 and vtil tunnels. But, when vti_tunnel_xmit is called, it is called with ip_vti0 tunnel and not the vtil tunnel. I am not sure if I am setting the route wrong. I saw a comment that vti_tunnel_xmit is called by dev_queue_xmit function. I could not figure out what dev_queue_xmit does though.

master# echo "1" | sudo tee /proc/sys/net/ipv4/ip_forward
1
master# modprobe ip_vti
master# ipsec start
Starting strongSwan 5.2.0 IPsec [starter]...
master# ip tunnel add vtil mode vti local 192.168.0.11 remote 192.168.1.232 ikey 1 okey 1 master# ip link set vtil up
master# sleep 10
master# ip route add 192.168.1.0/24 dev vtil
master# ip route list
default via 192.168.0.1 dev enp4s0 metric 202
127.0.0.0/8 dev lo scope host
192.168.0.0/24 dev enp4s0 proto kernel scope link src 192.168.0.11 metric 202 192.168.1.0/24 dev vtil scope link

ip link list
.
.
.
13: ip_vti0@NONE: <NOARP,UP,LOWER_UP> mtu 1428 qdisc noqueue state UNKNOWN mode DEFAULT group default

    link/ipip 0.0.0.0 brd 0.0.0.0
14: vtil@NONE: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1428 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/ipip 192.168.0.11 peer 192.168.1.232

master# ip tunnel list
ip_vti0: ip/ip remote any local any ttl inherit nopmtudisc key 0
vtil: ip/ip remote 192.168.1.232 local 192.168.0.11 ttl inherit key 1

Just wanted to check if you have any suggestions to go about debugging this.

Thanks
Joe

#4 lore raw
Quoted 7 lines · Joe M

Hello Steffen,

Checking further with printk's, I can see

vti_tunnel_init being called for both ip_vti0 and vtil tunnels. But, when vti_tunnel_xmit is called, it is called with ip_vti0 tunnel and not the vtil tunnel. I am not sure if I am setting the route wrong.

If you can't get traffic into the tunnel, then the routing is likely to be wrong.

Quoted 16 lines

master# echo "1" | sudo tee /proc/sys/net/ipv4/ip_forward
1
master# modprobe ip_vti
master# ipsec start
Starting strongSwan 5.2.0 IPsec [starter]...
master# ip tunnel add vtil mode vti local 192.168.0.11 remote 192.168.1.232 ikey 1 okey 1 master# ip link set vtil up
master# sleep 10
master# ip route add 192.168.1.0/24 dev vtil
master# ip route list
default via 192.168.0.1 dev enp4s0 metric 202
127.0.0.0/8 dev lo scope host
192.168.0.0/24 dev enp4s0 proto kernel scope link src 192.168.0.11 metric 202 192.168.1.0/24 dev vtil scope link

Quoted 6 lines

ip link list
.
.
.
13: ip_vti0@NONE: <NOARP,UP,LOWER_UP> mtu 1428 qdisc noqueue state UNKNOWN mode DEFAULT group default

    link/ipip 0.0.0.0 brd 0.0.0.0

Why is the ip_vti0 interface up? This should be down after inserting the ip_vti module and I don't see that you've set it up with the commands above.

Set the ip_vti0 interface down and try to find out where your packets are getting dropped. As I said already, netstat -i and /proc/net/xfrm_stat can help.

#5 lore raw
In reply to Joe M
Quoted 13 lines · Joe M

master# echo "1" | sudo tee /proc/sys/net/ipv4/ip_forward 1
master# modprobe ip_vti
master# ipsec start
Starting strongSwan 5.2.0 IPsec [starter]... master# ip tunnel add vtil mode vti local 192.168.0.11 remote 192.168.1.232 ikey 1 okey 1
master# ip link set vtil up
master# sleep 60
master# ip route add 192.168.1.0/24 dev vtil master# ip xfrm state
src 192.168.0.11 dst <client or alice or bnglr public ip>

Not sure what <client or alice or bnglr public ip> is, but dst must match the remote tunnel endpoint i.e. 192.168.1.232 in your case.

author
#6 lore raw
In reply to Steffen Klassert

Hello Steffen,

Thanks for responding. Sorry that it took me some time to gather all the information.

Quoted 1 line

Do you know where the packets are getting dropped?

All I can see from the below statistics is that the ip_vti0 tunnel is getting picked up instead of the vtil tunnel.

Quoted 1 line

netstat -i or /proc/net/xfrm_stat could give a hint.

master# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg

enp4s0    1500     2310      0     57 0          1974      0      0      0 BMRU
ip_vti0   1428        0      2      0 0             0     10      0      0 ORU
lo       65536      600      0      0 0           600      0      0      0 LRU
vtil      1428        0      0      0 0             0      0      0      0 OPRU
master# ip -statistics xfrm state
src 192.168.0.11 dst <client or alice or bnglr public ip>
        proto esp spi 0xc0b44648(3233039944) reqid 1(0x00000001) mode tunnel
        replay-window 32 seq 0x00000000 flag af-unspec (0x00100000)
        mark 1/0xffffffff
        auth-trunc hmac(sha1)
0x6fb52dc437eb26b65bd0dced995aa27e78a7e869 (160 bits) 96
        enc cbc(aes) 0x7e431bdc0ec138d0d8476c4afb0ccd63 (128 bits)
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 902(sec), hard 1200(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2014-09-17 17:38:06 use -
        stats:
          replay-window 0 replay 0 failed 0
src <client or alice or bnglr public ip> dst 192.168.0.11
        proto esp spi 0xc514e2d3(3306480339) reqid 1(0x00000001) mode tunnel
        replay-window 32 seq 0x00000000 flag af-unspec (0x00100000)
        mark 1/0xffffffff
        auth-trunc hmac(sha1)
0x8b4fd0749314d3656c962124e69c554ca03c9e11 (160 bits) 96
        enc cbc(aes) 0x27e2a7cae3a24c20584e841a16dcf89d (128 bits)
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
        lifetime config:
          limit: soft (INF)(bytes), hard (INF)(bytes)
          limit: soft (INF)(packets), hard (INF)(packets)
          expire add: soft 848(sec), hard 1200(sec)
          expire use: soft 0(sec), hard 0(sec)
        lifetime current:
          0(bytes), 0(packets)
          add 2014-09-17 17:38:05 use -
        stats:
          replay-window 0 replay 0 failed 0

master# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg

enp4s0    1500     3214      0     69 0          2886      0      0      0 BMRU
ip_vti0   1428        0      2      0 0             0     21      0      0 ORU
lo       65536      629      0      0 0           629      0      0      0 LRU
vtil      1428        0      0      0 0             0      1      0      0 OPRU

master# ping -c 1 -I 192.168.0.11 192.168.1.232
PING 192.168.1.232 (192.168.1.232) from 192.168.0.11 : 56(84) bytes of data.

--- 192.168.1.232 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

master# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg

enp4s0    1500     3217      0     69 0          2890      0      0      0 BMRU
ip_vti0   1428        0      2      0 0             0     21      0      0 ORU
lo       65536      629      0      0 0           629      0      0      0 LRU
vtil      1428        0      0      0 0             0      1      0      0 OPRU

master# ping -c 1 -I 192.168.0.11 192.168.1.232
PING 192.168.1.232 (192.168.1.232) from 192.168.0.11 : 56(84) bytes of data.

--- 192.168.1.232 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

master# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg

enp4s0    1500     3230      0     69 0          2897      0      0      0 BMRU
ip_vti0   1428        0      2      0 0             0     22      0      0 ORU
lo       65536      629      0      0 0           629      0      0      0 LRU
vtil      1428        0      0      0 0             0      1      0      0 OPRU

master# ping -c 1 -I 192.168.0.11 192.168.1.232
PING 192.168.1.232 (192.168.1.232) from 192.168.0.11 : 56(84) bytes of data.

--- 192.168.1.232 ping statistics ---
1 packets transmitted, 0 received, 100% packet loss, time 0ms

master# netstat -i
Kernel Interface table
Iface MTU RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg

enp4s0    1500     3250      0     69 0          2912      0      0      0 BMRU
ip_vti0   1428        0      2      0 0             0     22      0      0 ORU
lo       65536      629      0      0 0           629      0      0      0 LRU
vtil      1428        0      0      0 0             0      1      0      0 OPRU

master#

Quoted 1 line

From what I gather, nothing seems to be going through the vtil tunnel.

The ip_vti0 seems to get used instead.

Quoted 26 lines
Quoted 3 lines

The pings worked fine if I remove the ip_vti and ip_tunnel modules, the "mark=1" from /etc/ipsec.conf and the iptables mangle rules to set-mark.

You don't need to set the mark with iptables. You just have to ensure that the policy and state marks match the tunnel keys. I.e. direction in and forward must match the ikey, direction out must match the okey.

Quoted 3 lines

sudo ip tunnel list
vtil: ip/ip remote 192.168.1.232 local 192.168.0.11 ttl inherit ikey 0 okey 1

Your ikey does not match the policy and the state mark.

Quoted 10 lines

sudo ip xfrm policy
src 192.168.1.0/24 dst 192.168.0.0/24
dir fwd priority 2883
mark 1/0xffffffff
tmpl src <bnglr public ip> dst 192.168.0.11
proto esp reqid 2 mode tunnel
src 192.168.1.0/24 dst 192.168.0.0/24
dir in priority 2883
mark 1/0xffffffff

Quoted 9 lines

If you set mark 1 here, the tunnel should set ikey 1.

Quoted 4 lines

I tried setting the mangle rules to set-mark but that did not help. I could not find any more documentation.

Please try without setting a mark with netfilter.

I removed the iptables rules and set all policy to ACCEPT in iptables raw, nat, mangle and raw tables.

master# echo "1" | sudo tee /proc/sys/net/ipv4/ip_forward 1
master# modprobe ip_vti
master# ipsec start
Starting strongSwan 5.2.0 IPsec [starter]... master# ip tunnel add vtil mode vti local 192.168.0.11 remote 192.168.1.232 ikey 1 okey 1
master# ip link set vtil up
master# sleep 60
master# ip route add 192.168.1.0/24 dev vtil master# ip xfrm state
src 192.168.0.11 dst <client or alice or bnglr public ip>

        proto esp spi 0xc0b44648 reqid 1 mode tunnel
        replay-window 32 flag af-unspec
        mark 1/0xffffffff
        auth-trunc hmac(sha1) 0x6fb52dc437eb26b65bd0dced995aa27e78a7e869 96
        enc cbc(aes) 0x7e431bdc0ec138d0d8476c4afb0ccd63
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
src <client or alice or bnglr public ip> dst 192.168.0.11
        proto esp spi 0xc514e2d3 reqid 1 mode tunnel
        replay-window 32 flag af-unspec
        mark 1/0xffffffff
        auth-trunc hmac(sha1) 0x8b4fd0749314d3656c962124e69c554ca03c9e11 96
        enc cbc(aes) 0x27e2a7cae3a24c20584e841a16dcf89d
        encap type espinudp sport 4500 dport 4500 addr 0.0.0.0

master# ip xfrm policy
src 192.168.1.0/24 dst 192.168.0.0/24
dir fwd priority 2883
mark 1/0xffffffff
tmpl src <client or alice or bnglr public ip> dst 192.168.0.11
proto esp reqid 1 mode tunnel
src 192.168.1.0/24 dst 192.168.0.0/24
dir in priority 2883
mark 1/0xffffffff
tmpl src <client or alice or bnglr public ip> dst 192.168.0.11
proto esp reqid 1 mode tunnel
src 192.168.0.0/24 dst 192.168.1.0/24
dir out priority 2883
mark 1/0xffffffff
tmpl src 192.168.0.11 dst <client or alice or bnglr public ip>
proto esp reqid 1 mode tunnel
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0
master# ip tunnel list
ip_vti0: ip/ip remote any local any ttl inherit nopmtudisc key 0
vtil: ip/ip remote 192.168.1.232 local 192.168.0.11 ttl inherit key 1 master# ip route list
default via 192.168.0.1 dev enp4s0 metric 202
127.0.0.0/8 dev lo scope host
192.168.0.0/24 dev enp4s0 proto kernel scope link src 192.168.0.11 metric 202
192.168.1.0/24 dev vtil scope link
master# uname -a
Linux master 3.16.3 #90 SMP PREEMPT Wed Sep 17 13:39:17 CDT 2014 x86_64 Intel(R) Pentium(R) CPU G620 @ 2.60GHz GenuineIntel GNU/Linux master# ip -V
ip utility, iproute2-ss140804

sudo tcpdump -nS 'src port 500 or dst port 500 or src port 4500 or dst port 4500' -i enp4s0
Password:
error : ret -1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on enp4s0, link-type EN10MB (Ethernet), capture size 65535 bytes 17:56:48.651871 IP 192.168.0.11.4500 > <client or alice or bnglr public ip>.4500: isakmp-nat-keep-alive
17:56:56.934729 IP <client or alice or bnglr public ip>.4500 > 192.168.0.11.4500: isakmp-nat-keep-alive
17:57:08.652113 IP 192.168.0.11.4500 > <client or alice or bnglr public ip>.4500: isakmp-nat-keep-alive
17:57:16.934548 IP <client or alice or bnglr public ip>.4500 > 192.168.0.11.4500: isakmp-nat-keep-alive
17:57:28.652359 IP 192.168.0.11.4500 > <client or alice or bnglr public ip>.4500: isakmp-nat-keep-alive
17:57:36.938056 IP <client or alice or bnglr public ip>.4500 > 192.168.0.11.4500: isakmp-nat-keep-alive
17:57:48.652624 IP 192.168.0.11.4500 > <client or alice or bnglr public ip>.4500: isakmp-nat-keep-alive
17:57:56.935926 IP <client or alice or bnglr public ip>.4500 > 192.168.0.11.4500: isakmp-nat-keep-alive
17:58:08.652872 IP 192.168.0.11.4500 > <client or alice or bnglr public ip>.4500: isakmp-nat-keep-alive
17:58:17.005488 IP <client or alice or bnglr public ip>.4500 > 192.168.0.11.4500: isakmp-nat-keep-alive

and there is no tcpdump output on vtil interface.

master# cat /etc/ipsec.conf

# /etc/ipsec.conf - strongSwan IPsec configuration file

config setup

conn %default

        ikelifetime=60m
        keylife=20m
        rekeymargin=3m
        keyingtries=1
        keyexchange=ikev2
        mobike=no

conn master-bnglr

        leftid="C=CH, O=strongSwan, CN=master"
        leftcert=masterCert.der
        left=192.168.0.11
        leftsubnet=192.168.0.0/24
        rightid="C=CH, O=strongSwan, CN=bnglr"
        right=%any
        rightsubnet=192.168.1.0/24
        auto=add
        mark=1

Any other thoughts, please?

Thanks
Joe