Skip to contentSkip to conversation

Packets marked by iptables only sent to the correct routing table sometimes

DISCUSSION Jeff Cook 3 participants 6 messages Oct 30, 2012 – Oct 31, 2012

lore
Share & tools
lore

Jeff Cook

author
#0 lore raw

Hello.

I am trying to route packets generated by a specific user out over a VPN. I have this configuration:

    $ sudo iptables -S -t nat
    -P PREROUTING ACCEPT
    -P OUTPUT ACCEPT
    -P POSTROUTING ACCEPT
    -A POSTROUTING -o tun0 -j MASQUERADE
    $ sudo iptables -S -t mangle
    -P PREROUTING ACCEPT
    -P INPUT ACCEPT
    -P FORWARD ACCEPT
    -P OUTPUT ACCEPT
    -P POSTROUTING ACCEPT
    -A OUTPUT -m owner --uid-owner guy -j MARK --set-xmark 0xb/0xffffffff
    $ sudo ip rule show
    0:      from all lookup local
    32765:  from all fwmark 0xb lookup 11
    32766:  from all lookup main
    32767:  from all lookup default
    $ sudo ip route show table 11
    10.8.0.5 dev tun0  proto kernel  scope link  src 10.8.0.6
    10.8.0.6 dev tun0  scope link
    10.8.0.1 via 10.8.0.5 dev tun0
    0.0.0.0/1 via 10.8.0.5 dev tun0
    $ sudo iptables -S -t raw
    -P PREROUTING ACCEPT
    -P OUTPUT ACCEPT
    -A OUTPUT -m owner --uid-owner guy -j TRACE
    -A OUTPUT -p tcp -m tcp --dport 80 -j TRACE

It seems that some sites work fine and use the VPN, but others don't and fall back to the normal interface. This is bad. This is a packet trace that used VPN:

    Oct 27 00:24:28 agent kernel: [612979.976052] TRACE:

raw:OUTPUT:rule:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999

    Oct 27 00:24:28 agent kernel: [612979.976105] TRACE:

raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999

    Oct 27 00:24:28 agent kernel: [612979.976164] TRACE:

mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999

    Oct 27 00:24:28 agent kernel: [612979.976210] TRACE:

mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:24:28 agent kernel: [612979.976269] TRACE:

nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:24:28 agent kernel: [612979.976320] TRACE:

filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:24:28 agent kernel: [612979.976367] TRACE:

mangle:POSTROUTING:policy:1 IN= OUT=tun0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:24:28 agent kernel: [612979.976414] TRACE:

nat:POSTROUTING:rule:1 IN= OUT=tun0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

and this is one that didn't:

    Oct 27 00:22:41 agent kernel: [612873.662559] TRACE:

raw:OUTPUT:rule:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999

    Oct 27 00:22:41 agent kernel: [612873.662609] TRACE:

raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999

    Oct 27 00:22:41 agent kernel: [612873.662664] TRACE:

mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999

    Oct 27 00:22:41 agent kernel: [612873.662709] TRACE:

mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:22:41 agent kernel: [612873.662761] TRACE:

nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:22:41 agent kernel: [612873.662808] TRACE:

filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:22:41 agent kernel: [612873.662855] TRACE:

mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb

I have already tried "ip route flush cache", to no avail. I do not know why the first packet goes through the correct routing table, and the second doesn't. Both are marked.

Once again, I do not want ALL packets system-wide to go through the VPN, I only want packets from a specific user (UID=999) to go through the VPN. I am testing ipchicken.com and walmart.com via links, from the same user, same shell. walmart.com appears to use the VPN; ipchicken.com does not.

I have tried iptables -t raw -A OUTPUT -j NOTRACK to circumvent conntrack interference, but this hasn't worked either.

Any help appreciated; need this resolved ASAP. If this is something that can't be resolved by volunteers on a mailing list and someone is available as a consultant and can look into this further, would appreciate it; email me privately with rate information and credentials.

Thanks
Jeff

#1 lore raw
Quoted 25 lines · Jeff Cook

Hello.

I am trying to route packets generated by a specific user out over a VPN. I have this configuration:

    $ sudo iptables -S -t nat
    -P PREROUTING ACCEPT
    -P OUTPUT ACCEPT
    -P POSTROUTING ACCEPT
    -A POSTROUTING -o tun0 -j MASQUERADE
    $ sudo iptables -S -t mangle
    -P PREROUTING ACCEPT
    -P INPUT ACCEPT
    -P FORWARD ACCEPT
    -P OUTPUT ACCEPT
    -P POSTROUTING ACCEPT
    -A OUTPUT -m owner --uid-owner guy -j MARK --set-xmark 0xb/0xffffffff
    $ sudo ip rule show
    0:      from all lookup local
    32765:  from all fwmark 0xb lookup 11
    32766:  from all lookup main
    32767:  from all lookup default
Quoted 5 lines
    $ sudo ip route show table 11
    10.8.0.5 dev tun0  proto kernel  scope link  src 10.8.0.6
    10.8.0.6 dev tun0  scope link
    10.8.0.1 via 10.8.0.5 dev tun0
    0.0.0.0/1 via 10.8.0.5 dev tun0
      ^^^^^^^^^

23.1.17.194, this doesn't go through tun0
209.68.27.16, this doesn't go through tun0

Address & CIDR => 209.68.27.16 & 128.0.0.0 => 128.0.0.0

Then: 128.0.0.0 != 0.0.0.0, then go to default route, likely to be eth0.

Quoted 109 lines
    $ sudo iptables -S -t raw
    -P PREROUTING ACCEPT
    -P OUTPUT ACCEPT
    -A OUTPUT -m owner --uid-owner guy -j TRACE
    -A OUTPUT -p tcp -m tcp --dport 80 -j TRACE

It seems that some sites work fine and use the VPN, but others don't and fall back to the normal interface. This is bad. This is a packet trace that used VPN:

    Oct 27 00:24:28 agent kernel: [612979.976052] TRACE:

raw:OUTPUT:rule:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999

    Oct 27 00:24:28 agent kernel: [612979.976105] TRACE:

raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999

    Oct 27 00:24:28 agent kernel: [612979.976164] TRACE:

mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999

    Oct 27 00:24:28 agent kernel: [612979.976210] TRACE:

mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:24:28 agent kernel: [612979.976269] TRACE:

nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:24:28 agent kernel: [612979.976320] TRACE:

filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:24:28 agent kernel: [612979.976367] TRACE:

mangle:POSTROUTING:policy:1 IN= OUT=tun0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:24:28 agent kernel: [612979.976414] TRACE:

nat:POSTROUTING:rule:1 IN= OUT=tun0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

and this is one that didn't:

    Oct 27 00:22:41 agent kernel: [612873.662559] TRACE:

raw:OUTPUT:rule:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999

    Oct 27 00:22:41 agent kernel: [612873.662609] TRACE:

raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999

    Oct 27 00:22:41 agent kernel: [612873.662664] TRACE:

mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999

    Oct 27 00:22:41 agent kernel: [612873.662709] TRACE:

mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:22:41 agent kernel: [612873.662761] TRACE:

nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:22:41 agent kernel: [612873.662808] TRACE:

filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:22:41 agent kernel: [612873.662855] TRACE:

mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb

I have already tried "ip route flush cache", to no avail. I do not know why the first packet goes through the correct routing table, and the second doesn't. Both are marked.

Once again, I do not want ALL packets system-wide to go through the VPN, I only want packets from a specific user (UID=999) to go through the VPN. I am testing ipchicken.com and walmart.com via links, from the same user, same shell. walmart.com appears to use the VPN; ipchicken.com does not.

I have tried iptables -t raw -A OUTPUT -j NOTRACK to circumvent conntrack interference, but this hasn't worked either.

Any help appreciated; need this resolved ASAP. If this is something that can't be resolved by volunteers on a mailing list and someone is available as a consultant and can look into this further, would appreciate it; email me privately with rate information and credentials.

Thanks
Jeff

#2 lore raw
Quoted 26 lines · Pablo Neira Ayuso
Quoted 25 lines · Jeff Cook

Hello.

I am trying to route packets generated by a specific user out over a VPN. I have this configuration:

    $ sudo iptables -S -t nat
    -P PREROUTING ACCEPT
    -P OUTPUT ACCEPT
    -P POSTROUTING ACCEPT
    -A POSTROUTING -o tun0 -j MASQUERADE
    $ sudo iptables -S -t mangle
    -P PREROUTING ACCEPT
    -P INPUT ACCEPT
    -P FORWARD ACCEPT
    -P OUTPUT ACCEPT
    -P POSTROUTING ACCEPT
    -A OUTPUT -m owner --uid-owner guy -j MARK --set-xmark 0xb/0xffffffff
    $ sudo ip rule show
    0:      from all lookup local
    32765:  from all fwmark 0xb lookup 11
    32766:  from all lookup main
    32767:  from all lookup default
Quoted 8 lines
Quoted 5 lines
    $ sudo ip route show table 11
    10.8.0.5 dev tun0  proto kernel  scope link  src 10.8.0.6
    10.8.0.6 dev tun0  scope link
    10.8.0.1 via 10.8.0.5 dev tun0
    0.0.0.0/1 via 10.8.0.5 dev tun0
      ^^^^^^^^^

23.1.17.194, this doesn't go through tun0

Sorry, I meant: 23.1.17.194, this goes through tun0.

Quoted 116 lines

209.68.27.16, this doesn't go through tun0

Address & CIDR => 209.68.27.16 & 128.0.0.0 => 128.0.0.0

Then: 128.0.0.0 != 0.0.0.0, then go to default route, likely to be eth0.

Quoted 109 lines
    $ sudo iptables -S -t raw
    -P PREROUTING ACCEPT
    -P OUTPUT ACCEPT
    -A OUTPUT -m owner --uid-owner guy -j TRACE
    -A OUTPUT -p tcp -m tcp --dport 80 -j TRACE

It seems that some sites work fine and use the VPN, but others don't and fall back to the normal interface. This is bad. This is a packet trace that used VPN:

    Oct 27 00:24:28 agent kernel: [612979.976052] TRACE:

raw:OUTPUT:rule:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999

    Oct 27 00:24:28 agent kernel: [612979.976105] TRACE:

raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999

    Oct 27 00:24:28 agent kernel: [612979.976164] TRACE:

mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999

    Oct 27 00:24:28 agent kernel: [612979.976210] TRACE:

mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:24:28 agent kernel: [612979.976269] TRACE:

nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:24:28 agent kernel: [612979.976320] TRACE:

filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:24:28 agent kernel: [612979.976367] TRACE:

mangle:POSTROUTING:policy:1 IN= OUT=tun0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:24:28 agent kernel: [612979.976414] TRACE:

nat:POSTROUTING:rule:1 IN= OUT=tun0 SRC=XXX.YYY.ZZZ.AAA DST=23.1.17.194 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=14494 DF PROTO=TCP SPT=57502 DPT=80 SEQ=2294732931 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6E01D0000000001030307) UID=999 GID=999 MARK=0xb

and this is one that didn't:

    Oct 27 00:22:41 agent kernel: [612873.662559] TRACE:

raw:OUTPUT:rule:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999

    Oct 27 00:22:41 agent kernel: [612873.662609] TRACE:

raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999

    Oct 27 00:22:41 agent kernel: [612873.662664] TRACE:

mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999

    Oct 27 00:22:41 agent kernel: [612873.662709] TRACE:

mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:22:41 agent kernel: [612873.662761] TRACE:

nat:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:22:41 agent kernel: [612873.662808] TRACE:

filter:OUTPUT:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb

    Oct 27 00:22:41 agent kernel: [612873.662855] TRACE:

mangle:POSTROUTING:policy:1 IN= OUT=eth0 SRC=XXX.YYY.ZZZ.AAA DST=209.68.27.16 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=40425 DF PROTO=TCP SPT=45305 DPT=80 SEQ=604973951 ACK=0 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A03A6B6960000000001030307) UID=999 GID=999 MARK=0xb

I have already tried "ip route flush cache", to no avail. I do not know why the first packet goes through the correct routing table, and the second doesn't. Both are marked.

Once again, I do not want ALL packets system-wide to go through the VPN, I only want packets from a specific user (UID=999) to go through the VPN. I am testing ipchicken.com and walmart.com via links, from the same user, same shell. walmart.com appears to use the VPN; ipchicken.com does not.

I have tried iptables -t raw -A OUTPUT -j NOTRACK to circumvent conntrack interference, but this hasn't worked either.

Any help appreciated; need this resolved ASAP. If this is something that can't be resolved by volunteers on a mailing list and someone is available as a consultant and can look into this further, would appreciate it; email me privately with rate information and credentials.

Thanks
Jeff

author
#3 lore raw
Quoted 32 lines · Pablo Neira Ayuso
Quoted 31 lines · Pablo Neira Ayuso
Quoted 30 lines · Jeff Cook

Hello.

I am trying to route packets generated by a specific user out over a VPN. I have this configuration:

    $ sudo iptables -S -t nat
    -P PREROUTING ACCEPT
    -P OUTPUT ACCEPT
    -P POSTROUTING ACCEPT
    -A POSTROUTING -o tun0 -j MASQUERADE
    $ sudo iptables -S -t mangle
    -P PREROUTING ACCEPT
    -P INPUT ACCEPT
    -P FORWARD ACCEPT
    -P OUTPUT ACCEPT
    -P POSTROUTING ACCEPT
    -A OUTPUT -m owner --uid-owner guy -j MARK --set-xmark 0xb/0xffffffff
    $ sudo ip rule show
    0:      from all lookup local
    32765:  from all fwmark 0xb lookup 11
    32766:  from all lookup main
    32767:  from all lookup default
    $ sudo ip route show table 11
    10.8.0.5 dev tun0  proto kernel  scope link  src 10.8.0.6
    10.8.0.6 dev tun0  scope link
    10.8.0.1 via 10.8.0.5 dev tun0
    0.0.0.0/1 via 10.8.0.5 dev tun0
Quoted 12 lines
Quoted 3 lines
      ^^^^^^^^^

23.1.17.194, this doesn't go through tun0

Sorry, I meant: 23.1.17.194, this goes through tun0.

Quoted 6 lines

209.68.27.16, this doesn't go through tun0

Address & CIDR => 209.68.27.16 & 128.0.0.0 => 128.0.0.0

Then: 128.0.0.0 != 0.0.0.0, then go to default route, likely to be eth0.

Thanks very much, I can verify that adding a route for 128.0.0.0/1 to table 11 fixes things.

Apologies for asking a naive question, but could you please inform me where 128.0.0.0/1 comes from and why it's ANDed against external IP addresses? I've tried to find info on Google about 128.0.0.0 and CIDR and unfortunately have not been able to find anything thus far that enlightens me as to why that route is necessary. I'd really like to understand, so if you spend some time explaining it to me I'd appreciate it.

Thanks again for your help. Will send 0.5 btc if you're interested; please include a Bitcoin deposit address and I will forward over the funds promptly. :)

Thanks
Jeff

#4 lore raw
Quoted 6 lines · Jeff Cook
Quoted 1 line
Quoted 1 line
Quoted 1 line
     0.0.0.0/1 via 10.8.0.5 dev tun0

Apologies for asking a naive question, but could you please inform me where 128.0.0.0/1 comes from and why it's ANDed against external IP addresses?

Haven't you got it backwards? Your route is 0.0.0.0/1, ie anything 0-127.x.x.x is routed, but 128.x.x.x isn't in the 0.0.0.0/1 net, hence it isn't?

Ed W

#5 lore raw
In reply to Jeff Cook
Quoted 49 lines · Jeff Cook
Quoted 32 lines · Pablo Neira Ayuso
Quoted 31 lines · Pablo Neira Ayuso
Quoted 30 lines · Jeff Cook

Hello.

I am trying to route packets generated by a specific user out over a VPN. I have this configuration:

    $ sudo iptables -S -t nat
    -P PREROUTING ACCEPT
    -P OUTPUT ACCEPT
    -P POSTROUTING ACCEPT
    -A POSTROUTING -o tun0 -j MASQUERADE
    $ sudo iptables -S -t mangle
    -P PREROUTING ACCEPT
    -P INPUT ACCEPT
    -P FORWARD ACCEPT
    -P OUTPUT ACCEPT
    -P POSTROUTING ACCEPT
    -A OUTPUT -m owner --uid-owner guy -j MARK --set-xmark 0xb/0xffffffff
    $ sudo ip rule show
    0:      from all lookup local
    32765:  from all fwmark 0xb lookup 11
    32766:  from all lookup main
    32767:  from all lookup default
    $ sudo ip route show table 11
    10.8.0.5 dev tun0  proto kernel  scope link  src 10.8.0.6
    10.8.0.6 dev tun0  scope link
    10.8.0.1 via 10.8.0.5 dev tun0
    0.0.0.0/1 via 10.8.0.5 dev tun0
Quoted 12 lines
Quoted 3 lines
      ^^^^^^^^^

23.1.17.194, this doesn't go through tun0

Sorry, I meant: 23.1.17.194, this goes through tun0.

Quoted 6 lines

209.68.27.16, this doesn't go through tun0

Address & CIDR => 209.68.27.16 & 128.0.0.0 => 128.0.0.0

Then: 128.0.0.0 != 0.0.0.0, then go to default route, likely to be eth0.

Thanks very much, I can verify that adding a route for 128.0.0.0/1 to table 11 fixes things.

Quoted 6 lines

Apologies for asking a naive question, but could you please inform me where 128.0.0.0/1 comes from and why it's ANDed against external IP addresses? I've tried to find info on Google about 128.0.0.0 and CIDR and unfortunately have not been able to find anything thus far that enlightens me as to why that route is necessary. I'd really like to understand, so if you spend some time explaining it to me I'd appreciate it.

Your mask is wrong. Using CIDR notation 0.0.0.0/1 matches networks from 0.0.0.0 to 127.255.255.255.

I'd suggest to add some default route to that table to get everything through tun0 instead of adding 128.0.0.0/1

Regards.