2.4.23 masquerading broken?

For security reasons, I upgraded to 2.4.23 last night. Now, suddenly, IP masquerading seems to be broken. When I use SNAT instead of masquerading, everything works.

Unfortunately, I think it's hard to reproduce the problem. Right after booting .23 for the first time, everything seemed to be okay. The problems started just an hour ago, after having the server running for fifteen hours without any problems.

Unfortunately there's not much more information I can provide. I can attach my iptables/rule/route file and keep my machine running in case anyone needs/wants more information. For now I'll just stick with SNAT. It works good enough for me.

Just FYI, and maybe someone else will have a similar problem.

Wilmer v/d Gaast. (not on the list)

+-------- .''`.     - -- ---+  +        - -- --- ---- ----- ------+
| lintux : :'  :  lintux.cx |  | OSS Programmer   www.bitlbee.org |
|   at   `. `~'  debian.org |  | www.algoritme.nl   www.lintux.cx |
+--- -- -  ` ---------------+  +------ ----- ---- --- -- -        +

#!/bin/sh

if [ "$1" != "start" ]; then

	exit;

fi

if ! ifconfig hensema > /dev/null 2> /dev/null; then sleep 1; fi

catnet="130.89.203.37"
utwente="utwente"
hensema="hensema"

echo -n 'Setting up routes and iptables... '

## Without this Linux will reject packets from outside the network echo 0 > /proc/sys/net/ipv4/conf/eth1/rp_filter

iptables -F
iptables -P INPUT ACCEPT
iptables -P OUTPUT ACCEPT
iptables -P FORWARD ACCEPT
iptables -t nat -F

iptables -A INPUT -i wlan0 -p tcp --dport ssh -j ACCEPT iptables -A INPUT -i wlan0 -p tcp --dport tinc -j ACCEPT iptables -A INPUT -i wlan0 -p udp --dport tinc -j ACCEPT iptables -A INPUT -i wlan0 -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -A INPUT -i wlan0 -p icmp -j ACCEPT
iptables -A INPUT -i wlan0 -j REJECT

iptables -A INPUT -i eth1 -p tcp --dport ssh -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport smtp -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport domain -j ACCEPT iptables -A INPUT -i eth1 -p udp --dport domain -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport www -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport imap2 -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport imap3 -j ACCEPT iptables -A INPUT -i eth1 -p tcp --dport imaps -j ACCEPT iptables -A INPUT -i eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -A INPUT -i eth1 -p icmp -j ACCEPT
iptables -A INPUT -i eth1 -j REJECT

iptables -A FORWARD -i wlan0 -j REJECT --reject-with icmp-host-unreachable #iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source $catnet

ip route flush table utwente
ip route flush table hensema

## This is a quick hack to flush the rule table. It's not yet perfect. ip rule list | \
cut -d ' ' -f2 | \
sed s/all/0\\/0/g | \
while read LINE; do

	ip rule del $LINE;

done

ip rule add prio 10 to 192.168.0.0/16 table main
ip rule add prio 12 iif hensema table $utwente ## For the ssh->hensema portfw ip rule add prio 15 from $catnet table $utwente
ip rule add prio 15 from 192.168.7.2 table $hensema
ip rule add prio 20 to 130.89.0.0/16 table $utwente
ip rule add prio 20 to 212.120.64.0/18 table $hensema
ip rule add prio 40 from 192.168.9.11 table $hensema
ip rule add prio 40 from 192.168.9.13 table $hensema
#ip rule add prio 40 from 192.168.9.14 table $hensema
#ip rule add prio 40 from 192.168.9.17 table $hensema
ip rule add prio 55 from 192.168.9.0/24 table $utwente
ip rule add prio 60 table main

ip route flush cache

ip route add table hensema 192.168.7.0/24 dev hensema ip route add table hensema default via 192.168.7.1 dev hensema ip route add table utwente 130.89.192.0/20 dev eth1 ip route add table utwente default via 130.89.192.1 dev eth1

ip route add 192.168.1.0/24 via 192.168.7.1 dev hensema

## Add default routes as fallbacks. The second route seems to be ## necessary to be able to bind() to eth1 and have things work. while ip route del default; do :; done
ip route add table main default via 192.168.7.1 dev hensema metric 0 ip route add table main default via 130.89.192.1 dev eth1 metric 1

ip route flush cache

## Port forward stuff
iptables -t nat -A PREROUTING -p tcp -d $catnet --dport 2222 -j DNAT --to 192.168.1.1:22 iptables -t nat -A PREROUTING -p tcp -d $catnet --dport 2121 -j DNAT --to 192.168.9.15:21 #iptables -t nat -A PREROUTING -p tcp -d $catnet --dport 222 -j DNAT --to 192.168.9.14:22 #iptables -t nat -s \! 192.168.9.0/24 -A POSTROUTING -o hensema -j MASQUERADE iptables -t nat -s \! 192.168.9.0/24 -A POSTROUTING -o hensema -j SNAT --to-source 192.168.7.2

echo 'OK'

Can you check the ringbuffer for error messages ? What happens to the packets when masquerading fails ?

Best regards,
Patrick

Hmm. Damn, forgot about the syslogs completely. :-(

Dec  2 16:42:30 tosca kernel: MASQUERADE: Route sent us somewhere else.
Dec  2 16:42:44 tosca last message repeated 11 times
Dec  2 16:42:47 tosca kernel: NET: 1 messages suppressed.
Dec  2 16:42:47 tosca kernel: MASQUERADE: Route sent us somewhere else.
Dec  2 16:42:51 tosca kernel: NET: 5 messages suppressed.
Dec  2 16:42:51 tosca kernel: MASQUERADE: Route sent us somewhere else.
Dec  2 16:42:57 tosca kernel: NET: 4 messages suppressed.
Dec  2 16:42:57 tosca kernel: MASQUERADE: Route sent us somewhere else.

And, well, it goes on like that. dmesg is full of messages like this.

The packages seem to get lost completely. At least I don't see them going out on eth1 (where they should go to).

Wilmer van der Gaast.

+-------- .''`.     - -- ---+  +        - -- --- ---- ----- ------+
| lintux : :'  :  lintux.cx |  | OSS Programmer   www.bitlbee.org |
|   at   `. `~'  debian.org |  | www.algoritme.nl   www.lintux.cx |
+--- -- -  ` ---------------+  +------ ----- ---- --- -- -        +

It work for me
(2.4.23 with linux abi, gcc 3.2 from RedHat 8.0 - netfilter like modules) regards
Haris Peco

It may be related to using advances routing features ..
Can you give information about the specific IPs ? Is it local traffic ?

Thanks,
Patrick

It happens when a packet to a host outside 130.89.*.* is routed through the eth1 (130.89.203.37) interface. That happens with any traffic not from .9.11 or .9.13. .9.11 and .9.13 don't have any problems, their traffic is routed to the other Internet interface without any problems.

Also, traffic to 130.89.*.* from any host is routed to eth1 correctly and "even" works.

So, in short:

192.168.9.10 -=> 130.89.1.1 works, through eth1
192.168.9.11 -=> 130.89.1.1 works, through eth1
192.168.9.10 -=> www.google.com doesn't work
192.168.9.11 -=> www.google.com works, through hensema (which is what I want)

Also, trying to ping www.google.com through eth1 (hensema is the default interface) from the bugging machine directly works.

I hope this clarifies something... If not, I can do some more testing tomorrow.

Greetings,

Wilmer van der Gaast.

+-------- .''`.     - -- ---+  +        - -- --- ---- ----- ------+
| lintux : :'  :  lintux.cx |  | OSS Programmer   www.bitlbee.org |
|   at   `. `~'  debian.org |  | www.algoritme.nl   www.lintux.cx |
+--- -- -  ` ---------------+  +------ ----- ---- --- -- -        +

I remember problem with masquerade on 2.4.22 and there was included attached diff on Slackware 9.1

==========================================================================
# This patch is needed in 2.4.22 or else NAT (masquerade) will not work. # It fixes the "Rusty's broken brain" error/failure.

===== net/ipv4/netfilter/ipt_MASQUERADE.c 1.6 vs edited =====

@@ -90,6 +90,7 @@
 #ifdef CONFIG_IP_ROUTE_FWMARK
        key.fwmark = (*pskb)->nfmark;
 #endif
+       key.oif = 0;
        if (ip_route_output_key(&rt, &key) != 0) {
                 /* Funky routing can do this. */
                 if (net_ratelimit())

==========================================================================

interesting is that 2.4.23 is pached BUT

==========================================================================

        key.tos = RT_TOS((*pskb)->nh.iph->tos)|RTO_CONN;
        key.oif = 0;
#ifdef CONFIG_IP_ROUTE_FWMARK
        key.fwmark = (*pskb)->nfmark;
#endif
        if (ip_route_output_key(&rt, &key) != 0) {
                /* Funky routing can do this. */

==========================================================================

So problem is there perhaps, anyway edit file or patch it with att, patch ipt_MASQ.diff

Zoran Davidovac

# This patch is needed in 2.4.22 or else NAT (masquerade) will not work. # It fixes the "Rusty's broken brain" error/failure.

===== net/ipv4/netfilter/ipt_MASQUERADE.c 1.6 vs edited =====

@@ -90,6 +90,7 @@
 #ifdef CONFIG_IP_ROUTE_FWMARK
 	key.fwmark = (*pskb)->nfmark;
 #endif
+	key.oif = 0;
 	if (ip_route_output_key(&rt, &key) != 0) {
                 /* Funky routing can do this. */
                 if (net_ratelimit())

@@ -87,10 +87,10 @@
 	key.dst = (*pskb)->nh.iph->daddr;
 	key.src = 0; /* Unknown: that's what we're trying to establish */
 	key.tos = RT_TOS((*pskb)->nh.iph->tos)|RTO_CONN;
-	key.oif = 0;
 #ifdef CONFIG_IP_ROUTE_FWMARK
 	key.fwmark = (*pskb)->nfmark;
 #endif
+	key.oif = 0;
 	if (ip_route_output_key(&rt, &key) != 0) {
                 /* Funky routing can do this. */
                 if (net_ratelimit())

This seems to be the same as
http://www.ussg.iu.edu/hypermail/linux/kernel/0312.0/0465.html
and https://bugzilla.netfilter.org/cgi-bin/bugzilla/show_bug.cgi?id=144

I've committed the proposed fix (from #144) into patch-o-matic/pending.

Comments?

Patrick,

- Harald Welte <[email protected]>             http://www.netfilter.org/
============================================================================
  "Fragmentation is like classful addressing -- an interesting early
   architectural error that shows how much experimentation was going
   on while IP was being designed."                    -- Paul Vixie

I don't know if reverting to 2.4.22 is the correct fix, the change was made after this mail from Alexey http://marc.theaimsgroup.com/?l=linux-net&m=105915597804604&w=2 , he states that giving out ifindex is a bug. I don't understand the problem yet but I'm looking into it.

BTW: Why do we need a route lookup at all ? Couldn't we just use the first address on dev->in_dev->ifa_list ?

Best regards,
Patrick

I've attached two patches (2.4+2.6) to the bugtracker which change MASQUERADE to use indev->ifa_list->ifa_local. The 2.6 version is running here without problems so far. Please have a look at
https://bugzilla.netfilter.org/cgi-bin/bugzilla/show_bug.cgi?id=144 .

Best regards,
Patrick