Returning nat packets vanishing after mangle:PREROUTING and conntrack processing

I have a multi-homed server, and have been routing packets selectively

between the public interfaces using iptables marking and iproute2 tables.

The setup worked well until I upgraded to 2.6.31 (Fedora 11->12 upgrade),

and then connection tracking apparently broke. I've struggled with it for

some time, but think I may be encountering new/changed kernel behavior

since 2.6.30 (which worked).

With the setup below, bringing down either public interface results in

normal conntrack behavior (packets are correctly nat'd back to their

source).

Sadly, I've been hacking with the settings so much to track the source of

the breakage that I no longer have the iptables/iproute2 setup that worked

in 2.6.30, but I can't see why the setup below shouldn't work as intended.

My (simplified) setup is as follows:

Definitions:

PRIV - internal IP on eth0 (PRIV-NET is subnet)

DSL - static public IP on eth1 (DSL-NET is subnet, DSL-GW is gateway)

CABLE - dynamic public IP on eth2 (CABLE-NET is subnet, CABLE-GW is

gateway)

INT - host IP on internal eth0 network

EXT - external host IP (reachable via eth1 or eth2)

router# tail -2 /etc/iproute2/rt_tables

200 cable

201 dsl

router# ip rule list

0: from all lookup local

32756: from all fwmark 0x2 lookup cable

32757: from all to <cable-dhcp-server> lookup cable

32758: from <CABLE> lookup cable

32759: from all fwmark 0x1 lookup dsl

32760: from <DSL> lookup dsl

32766: from all lookup main

32767: from all lookup default

router# ip route list

<DSL-NET> dev eth1 proto kernel scope link src <DSL>

<PRIV-NET> dev eth0 proto kernel scope link src <PRIV>

<CABLE-NET> dev eth2 proto kernel scope link src <CABLE>

169.254.0.0/16 dev eth0 scope link metric 1002

default via <DSL-GW> dev eth1

router# ip route list table default

default via <CABLE-GW> dev eth2

router# ip route list table dsl

<PRIV-NET> dev eth0 scope link src <PRIV>

<CABLE-NET> dev eth2 scope link src <CABLE>

default via <DSL-GW> dev eth1

router# ip route list table cable

<DSL-NET> dev eth1 scope link src <DSL>

<PRIV-NET> dev eth0 scope link src <PRIV>

default via <CABLE-GW> dev eth2

(simplified iptables config)

*raw

-A PREROUTING -p tcp -d <EXT> -j TRACE

-A PREROUTING -p tcp -s <EXT> -j TRACE

*nat

-A POSTROUTING -o eth1 -j SNAT --to-source <DSL>

-A POSTROUTING -o eth2 -j MASQUERADE

*mangle

-A PREROUTING -m state --state ESTABLISHED,RELATED -j CONNMARK

--restore-mark

-A PREROUTING -i eth0 -m state --state NEW -j CONNMARK --set-mark 2

-A PREROUTING -i eth1 -m state --state NEW -j CONNMARK --set-mark 1

-A PREROUTING -i eth2 -m state --state NEW -j CONNMARK --set-mark 2

-A PREROUTING -m connmark --mark 1 -j MARK --set-mark 1

-A PREROUTING -m connmark --mark 2 -j MARK --set-mark 2

-A PREROUTING -m state --state NEW -m connmark ! --mark 0 -j CONNMARK

--save-mark

-A OUTPUT -m state --state ESTABLISHED,RELATED -j CONNMARK --restore-mark

Here's the trace output and conntrack details from the following command

issued on the internal host <INT> (on the <PRIV> network):

<INT># curl -I http://<EXT>

router# tcpdump -i eth2 -nn 'host <EXT>'

09:08:04.524942 IP <CABLE>.52800 > <EXT>.80: Flags [S], seq 304914632, win

65535, options [mss 1460,nop,wscale 3,nop,nop,TS val 521390037 ecr

0,sackOK,eol], length 0

09:08:04.540042 IP <EXT>.80 > <CABLE>.52800: Flags [S.], seq 2326757745,

ack 304914633, win 5792, options [mss 1460,sackOK,TS val 3283546987 ecr

521390037,nop,wscale 7], length 0

router# conntrack -E -p tcp -d <EXT>

    [NEW] tcp      6 120 SYN_SENT src=<INT> dst=<EXT> sport=52800 dport=80

[UNREPLIED] src=<EXT> dst=<CABLE> sport=80 dport=52800 mark=2

[UPDATE] tcp 6 59 SYN_RECV src=<INT> dst=<EXT> sport=52800 dport=80

src=<EXT> dst=<CABLE> sport=80 dport=52800 mark=2

router# conntrack -L -p tcp -d <EXT> --sport 52800

tcp 6 55 SYN_RECV src=<INT> dst=<EXT> sport=52800 dport=80 packets=1

bytes=64 src=<EXT> dst=<CABLE> sport=80 dport=52800 packets=6 bytes=360

mark=2 secmark=0 use=2

router# cat syslog

Dec 18 09:08:04 home kernel: TRACE: raw:PREROUTING:policy:3 IN=eth0 OUT=

SRC=<INT> DST=<EXT> LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=45328 DF PROTO=TCP

SPT=52800 DPT=80 SEQ=304914632 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT

Dec 18 09:08:04 home kernel: TRACE: mangle:PREROUTING:rule:2 IN=eth0 OUT=

SRC=<INT> DST=<EXT> LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=45328 DF PROTO=TCP

SPT=52800 DPT=80 SEQ=304914632 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT

Dec 18 09:08:04 home kernel: TRACE: mangle:PREROUTING:rule:6 IN=eth0 OUT=

SRC=<INT> DST=<EXT> LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=45328 DF PROTO=TCP

SPT=52800 DPT=80 SEQ=304914632 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT

Dec 18 09:08:04 home kernel: TRACE: mangle:PREROUTING:rule:7 IN=eth0 OUT=

SRC=<INT> DST=<EXT> LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=45328 DF PROTO=TCP

SPT=52800 DPT=80 SEQ=304914632 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT

MARK=0x2

Dec 18 09:08:04 home kernel: TRACE: mangle:PREROUTING:policy:8 IN=eth0

OUT= SRC=<INT> DST=<EXT> LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=45328 DF

PROTO=TCP SPT=52800 DPT=80 SEQ=304914632 ACK=0 WINDOW=65535 RES=0x00 SYN

URGP=0 OPT MARK=0x2

Dec 18 09:08:04 home kernel: TRACE: nat:PREROUTING:policy:1 IN=eth0 OUT=

SRC=<INT> DST=<EXT> LEN=64 TOS=0x00 PREC=0x00 TTL=64 ID=45328 DF PROTO=TCP

SPT=52800 DPT=80 SEQ=304914632 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT

MARK=0x2

Dec 18 09:08:04 home kernel: TRACE: mangle:FORWARD:policy:1 IN=eth0

OUT=eth2 SRC=<INT> DST=<EXT> LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=45328 DF

PROTO=TCP SPT=52800 DPT=80 SEQ=304914632 ACK=0 WINDOW=65535 RES=0x00 SYN

URGP=0 OPT MARK=0x2

Dec 18 09:08:04 home kernel: TRACE: filter:FORWARD:policy:1 IN=eth0

OUT=eth2 SRC=<INT> DST=<EXT> LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=45328 DF

PROTO=TCP SPT=52800 DPT=80 SEQ=304914632 ACK=0 WINDOW=65535 RES=0x00 SYN

URGP=0 OPT MARK=0x2

Dec 18 09:08:04 home kernel: TRACE: mangle:POSTROUTING:policy:1 IN=

OUT=eth2 SRC=<INT> DST=<EXT> LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=45328 DF

PROTO=TCP SPT=52800 DPT=80 SEQ=304914632 ACK=0 WINDOW=65535 RES=0x00 SYN

URGP=0 OPT MARK=0x2

Dec 18 09:08:04 home kernel: TRACE: nat:POSTROUTING:rule:2 IN= OUT=eth2

SRC=<INT> DST=<EXT> LEN=64 TOS=0x00 PREC=0x00 TTL=63 ID=45328 DF PROTO=TCP

SPT=52800 DPT=80 SEQ=304914632 ACK=0 WINDOW=65535 RES=0x00 SYN URGP=0 OPT

MARK=0x2

Dec 18 09:08:04 home kernel: TRACE: raw:PREROUTING:policy:3 IN=eth2 OUT=

SRC=<EXT> DST=<CABLE> LEN=60 TOS=0x00 PREC=0x20 TTL=55 ID=0 DF PROTO=TCP

SPT=80 DPT=52800 SEQ=2326757745 ACK=304914633 WINDOW=5792 RES=0x00 ACK SYN

URGP=0 OPT

Dec 18 09:08:04 home kernel: TRACE: mangle:PREROUTING:rule:1 IN=eth2 OUT=

SRC=<EXT> DST=<CABLE> LEN=60 TOS=0x00 PREC=0x20 TTL=55 ID=0 DF PROTO=TCP

SPT=80 DPT=52800 SEQ=2326757745 ACK=304914633 WINDOW=5792 RES=0x00 ACK SYN

URGP=0 OPT

Dec 18 09:08:04 home kernel: TRACE: mangle:PREROUTING:rule:6 IN=eth2 OUT=

SRC=<EXT> DST=<CABLE> LEN=60 TOS=0x00 PREC=0x20 TTL=55 ID=0 DF PROTO=TCP

SPT=80 DPT=52800 SEQ=2326757745 ACK=304914633 WINDOW=5792 RES=0x00 ACK SYN

URGP=0 OPT MARK=0x2

Dec 18 09:08:04 home kernel: TRACE: mangle:PREROUTING:policy:8 IN=eth2

OUT= SRC=<EXT> DST=<CABLE> LEN=60 TOS=0x00 PREC=0x20 TTL=55 ID=0 DF

PROTO=TCP SPT=80 DPT=52800 SEQ=2326757745 ACK=304914633 WINDOW=5792

RES=0x00 ACK SYN URGP=0 OPT MARK=0x2

And that's where the packet dissappears... it apparently has been seen by

conntrack, but fails to appear in the nat:PREROUTING chain.

Any suggestion on how I can discover what is happening to the packet at

this point?

Please excuse any mistakes I made sanitizing the output, and let me know

if I can provide any further details that might help.

Thanks,

Scott

Hello,

[...]

It may be a source validation issue, which is common in multihomed setups. If sysctl net.ipv4.conf.<input_interface>.rp_filter is set to 1, does setting it to 0 fix the problem ?

[...]

AFAIK, only the first NEW packet of a connection appears in the nat chains. Subsquent packets don't. If the packet does not appear in the FORWARD nor INPUT chain, chances are it was dropped at the routing decision stage, where source validation is performed.

On Sat, 19 Dec 2009 14:12:29 +0100, Pascal Hambourg

Fantastic, works great. Changed to 'net.ipv4.conf.default.rp_filter = 0'

in sysctl.conf (was set to 1).

Oddly, I had rp_filter enabled on the system in kernel 2.6.30 and it

worked. Has rp_filter changed somehow in the newer kernel (or is it now

working 'correctly'?).

Thanks,

Scott

(Searching in kernel changelogs...)
Yes, rp_filter slighly changed in kernel 2.6.31 (commit 27fed4175acf81ddd91d9a4ee2fd298981f60295). IIUC it is the way that net.ipv4.conf.<interface>.rp_filter and net.ipv4.conf.all.rp_filter are combined together that changed from a logical AND to an arithmetic MAX. This was a fix for a previous patch in kernel 2.6.30 (commit c1cf8422f0512c2b14f0d66bce34abb0645c888a) which added support for reverse path filtering "loose mode" (actually a route presence check), changing rp_filter type from boolean to integer and assigning the value 2 to the new loose mode (see Documentation/networking/ip-sysctl.txt for details).

Before kernel 2.6.31 :
Actual rp_filter for <interface> = net.ipv4.conf.<interface>.rp_filter AND net.ipv4.conf.all.rp_filter
I.e. reverse path filtering is enabled in strict mode if rp_filter=1 for both "all" and the interface.

Since kernel 2.6.31 :
Actual rp_filter for <interface> = MAX(net.ipv4.conf.<interface>.rp_filter, net.ipv4.conf.all.rp_filter) I.e. reverse path filtering is enabled in strict mode if rp_filter=1 for either "all" or the interface.

If by "I had rp_filter enabled" you mean that only net.ipv4.conf.default.rp_filter was set to 1 and net.ipv4.conf.all.rp_filter was left to 0 (default), then with the kernel 2.6.30 the resulting AND was 0, so the reverse path filtering was disabled. But with the kernel 2.6.31 the resulting MAX is 1, so strict reverse path filtering is enabled.

Notes :
1) "Loose" reverse path filtering may be a bit better than no reverse path filtering and should work with your setup.
2) Reverse path filtering in kernel 2.6.32 uses the mark as in policy routing, so strict reverse path filtering may work better in multihomed setups like yours.

On Sat, 19 Dec 2009 19:39:51 +0100, Pascal Hambourg

Guess I need to read those more closely to catch subtle but important

changes in kernel behavior :)

[...]

Tried "loose" and it worked great on my multi-homed setup, thanks for the

tip.

Looking forward to it, I'll give it a try once I upgrade to .32

Cheers,

Scott