still can't route using fwmark

Hello,

I am still having trouble doing load balancing over ppp0 and ppp1 for a Debian Etch PC (kernel 2.6.20.3, iptables 1.3.8). I used patch-o-matic to update my kernel sources and recompiled the kernel, and compiled and installed iptables 1.3.8. Also all the netfilter-related kernel configuration options I need are set, I believe.

Quoted 3 lines · Lloyd Standish

Hello,

I am still having trouble doing load balancing over ppp0 and ppp1 for a Debian Etch PC (kernel 2.6.20.3, iptables 1.3.8). I used patch-o-matic to update my kernel sources and recompiled the kernel, and compiled and installed iptables 1.3.8. Also all the netfilter-related kernel configuration options I need are set, I believe.

Why the strange versions? There are neither the ones that comes with Etch nor are they anything current? Anyway, Etch's stock kernel and iptables (2.6.18/v1.3.6 based) are fine for what you want to do.

You need the iptables (conn)marking stuff, and advanced router + policy/fw routing, and that's all part of stock packages.

/boot/config-2.6.18-6-686

CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_ASK_IP_FIB_HASH=y
# CONFIG_IP_FIB_TRIE is not set
CONFIG_IP_FIB_HASH=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_ROUTE_FWMARK=y
CONFIG_IP_ROUTE_MULTIPATH=y
# CONFIG_IP_ROUTE_MULTIPATH_CACHED is not set
CONFIG_IP_ROUTE_VERBOSE=y

dpkg -L iptables | grep -i mark
/lib/iptables/libipt_connmark.so
/lib/iptables/libipt_mark.so
/lib/iptables/libipt_CONNMARK.so
/lib/iptables/libipt_MARK.so
/lib/iptables/libipt_SECMARK.so
/lib/iptables/libipt_CONNSECMARK.so
/lib/iptables/libip6t_connmark.so
/lib/iptables/libip6t_mark.so
/lib/iptables/libip6t_CONNMARK.so
/lib/iptables/libip6t_MARK.so
/lib/iptables/libip6t_SECMARK.so
/lib/iptables/libip6t_CONNSECMARK.so

Here's what I used some time ago to do something similar (either with Etch or maybe even Sarge), sending outgoing packets back to the router the corresponding incoming packets of a connection came from, for a single machine.

#!/bin/sh

iptables=iptables

$iptables -t mangle -A INPUT -m mac --mac <ROUTERMAC1> -j CONNMARK --set-mark 1 $iptables -t mangle -A INPUT -m mac --mac <ROUTERMAC2> -j CONNMARK --set-mark 2

$iptables -t mangle -N TONET
$iptables -t mangle -F TONET
$iptables -t mangle -A OUTPUT -o <DEVICE> -d ! <LOCAL_NET> -j TONET

$iptables -t mangle -A TONET -m connmark ! --mark 0 -j CONNMARK --restore-mark

The route/rule stuff was more or less what you've got. If you want to forward packets for other hosts, this need to go into PREROUTING of course.

Thomas

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

!DSPAM:49e992ac62591343511188!

Thanks for the reply, Thomas. I've got iptables 1.3.8 because I downloaded it (and the patch-o-matic stuff) quite a while back when it was the most current version. I'm using it because my kernel is not the stock Etch kernel (it is in fact a Ubuntu kernel, necessary to get a module that Debian removed over licensing issues). I recompiled my kernel and installed itables 1.3.8 out of desperation, since I have studied this load-balancing stuff pretty thoroughly and it seems what I have ought to work.

My kernel 2.6.20.3 does not have a CONFIG_IP_ROUTE_FWMARK configuration setting! Has this been absorbed into another configuration setting?

How can I be sure fwmark is working? Judging by my setup, which is very simple, packets and connections are being marked, but routing is not affected by fwmark.

I have the kernel config options IP_NF_CONNTRACK_SUPPORT and XFRM_SUB_POLICY both set OFF. Might this be a problem?

Quoted 1 line

How can I be sure fwmark is working? Judging by my setup, which is very simple, packets and connections are being marked, but routing is not affected by fwmark.

You can check the routing cache using "ip route show cache", and see what routing decisions were taken.

Maybe you could post your connmark related iptables rules as well.

Also you could try to remove those two routes in the default table, which I gather from your description are routes for $gw0=$gw1=10.60.255.254 (btw, the second will normally never be used anyway), I remember having problems before when routes matched after the policy route tables in the main table.

Why are you using the same IP (10.60.255.254) on different links? Possibly you could try different IPs for that. I've haven't used PPP links for policy routing so far, could be that they behave differently in some significant way wrt. to all this, compared to broadcast networks.

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

!DSPAM:49ea206b166987818312239!

Quoted 4 lines · Thomas Jacob

Quoted 1 line

How can I be sure fwmark is working? Judging by my setup, which is very simple, packets and connections are being marked, but routing is not affected by fwmark.

You can check the routing cache using "ip route show cache", and see what routing decisions were taken.

Should there be signs here of fwmark-related decisions?

Quoted 1 line

Maybe you could post your connmark related iptables rules as well.

I am logging the CONNMARK stuff now. The logs show the connection marking taking place. Here are the rules from my shell script, with a few comments I just now added.

# define CONNMARK1
iptables -t mangle -N CONNMARK1
iptables -t mangle -A CONNMARK1 -j MARK --set-mark 1
iptables -t mangle -A CONNMARK1 -j CONNMARK --save-mark
iptables -t mangle -A CONNMARK1 -j LOG --log-prefix 'iptables-mark1: ' --log-level info

# define CONNMARK1
iptables -t mangle -N CONNMARK2
iptables -t mangle -A CONNMARK2 -j MARK --set-mark 2
iptables -t mangle -A CONNMARK2 -j CONNMARK --save-mark
iptables -t mangle -A CONNMARK2 -j LOG --log-prefix 'iptables-mark2: ' --log-level info

# define RESTOREMARK
iptables -t mangle -N RESTOREMARK
iptables -t mangle -A RESTOREMARK -j CONNMARK --restore-mark
iptables -t mangle -A RESTOREMARK -j LOG --log-prefix 'restore-mark: ' --log-level info

# define SNAT1
iptables -t nat -N SNAT1
iptables -t nat -A SNAT1 -j LOG --log-prefix "SNAT $src0: " --log-level info
iptables -t nat -A SNAT1 -j SNAT --to-source $src0

# define SNAT2
iptables -t nat -N SNAT2
iptables -t nat -A SNAT2 -j LOG --log-prefix "SNAT $src1: " --log-level info
iptables -t nat -A SNAT2 -j SNAT --to-source $src1

# restore the fwmark on packets that belong to an existing connection
iptables -A PREROUTING -i eth0  -t mangle -m state --state ESTABLISHED,RELATED -j RESTOREMARK
# if there is a mark, can quit
iptables -A PREROUTING -t mangle -m mark ! --mark 0 -j RETURN

# mark all packets with fwmark 1 (A)
iptables -A PREROUTING -t mangle -j CONNMARK1

# mark every other packet with fwmark 2 (B)
iptables -A PREROUTING -t mangle -m statistic --mode nth --every 2 --packet 0 -j CONNMARK2

# fix source IPs to match interface IP
iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2

# masquerade eth0
iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE

Quoted 4 lines

Also you could try to remove those two routes in the default table, which I gather from your description are routes for $gw0=$gw1=10.60.255.254 (btw, the second will normally never be used anyway), I remember having problems before when routes matched after the policy route tables in the main table.

Do you mean the routes in the rt_link1 and rt_link2 tables?

I saw in a couple of articles on load balancing the suggestion that only the default route is necessary in each of the user-defined tables. Of course, the only reason I use these tables is to be able to route through a different interface.

Quoted 4 lines

Why are you using the same IP (10.60.255.254) on different links? Possibly you could try different IPs for that. I've haven't used PPP links for policy routing so far, could be that they behave differently in some significant way wrt. to all this, compared to broadcast networks.

10.60.255.254 is the gateway for each of the links ppp0 and ppp1. I cannot change that, and I don't see why the gateway should have to be different.

I ought to point out that I am connecting to my ISP using GPRS (ppp tunneled over GSM telephone connection). The ISP assigns a private IP to me, and the gateway IP is also private. The ISP is natting to a public IP, and that public IP changes constantly! I assume this is part of some load-balancing that my ISP is doing!

Are you forwarding packets via this box, or do you want to loadbalance packets from the local machine? In the latter case the PREROUTING stuff needs to go into INPUT/OUTPUT.

Quoted 6 lines

Quoted 4 lines

Also you could try to remove those two routes in the default table, which I gather from your description are routes for $gw0=$gw1=10.60.255.254 (btw, the second will normally never be used anyway), I remember having problems before when routes matched after the policy route tables in the main table.

Do you mean the routes in the rt_link1 and rt_link2 tables?

no, the ones in the "main" routing table (that you displayed using "ip route show [table main]).

Yes, if you have a way to find your gateway (which is sort of implicit when both gateway are on the same ethernet link with mask /24 for instance), but in your case you seem to need host routes to designate where your default gateways are (the first route in rt_link1/2),

and the two routes in main
are the same as the first route in rt_link1 and 2 combined, right?

But maybe one does not need gateways for ppp since there should be no one else on that link anyway (as you suggested with your "default dev pppX" routes).

I am running out of ideas, sorry, maybe someone else on the list can chime in, I've only ever done policy routing with public IPs and broadcast networks, and there it always worked without problems on stock Debian Etch, CentOS 5.X or custom built kernels.

Quoted 3 lines · Thomas Jacob

Are you forwarding packets via this box, or do you want to loadbalance packets from the local machine? In the latter case the PREROUTING stuff needs to go into INPUT/OUTPUT.

Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0). The local machine is 192.168.1.1 on the LAN.

When I remove the default route in the main routing table, I completely lose Internet connectivity. My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.

The only experience I have with iptables is simple firewall stuff for my Internet-connected server. My grasp of routing is weak, and this trouble is good experience in an area I would like to become expert in.

Quoted 3 lines

But maybe one does not need gateways for ppp since there should be no one else on that link anyway (as you suggested with your "default dev pppX" routes).

I don't think I understand your comment. I need a gateway IP to forward Internet queries to...

I really appreciate your advice. I'm a networking novice so my ideas are not rooted in experience, but I think it should not matter that the gateway IP is a private IP.

Quoted 1 line

Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0). The local machine is 192.168.1.1 on the LAN.

Then your current setup in PREROUTING is what you want to go for, just keep in mind that this does not give you load balancing for connections originating from your router box, just the ones from your LAN.

C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png

I'm not sure why you need NAT on your eth0 though then, what are you trying to achieve with this? But that should not be the cause of the load balancing failure.

Quoted 2 lines

When I remove the default route in the main routing table, I completely lose Internet connectivity. My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.

That's right, but if all your /proc/net/ip_conntrack entries contain mark values then there really must be something wrong with the fw mark <-> route interaction.

My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for instance, then maybe more people have a comparable setup to look at.

Do you know about LARTC? The best way to get started IMO: http://lartc.org/

Quoted 6 lines

Quoted 2 lines

Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0).  The local machine is 192.168.1.1 on the LAN.

Then your current setup in PREROUTING is what you want to go for, just keep in mind that this does not give you load balancing for connections originating from your router box, just the ones from your LAN.

I'm sorry, I don't understand.   According to what you are saying, I should not get any load balancing, since all my testing up until now has been with connections (to the Internet) originating on the router box.  (I haven't even tried connecting from the LAN.)

However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark, put there by my prerouting rules.  Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables?  (As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)

Quoted 6 lines

C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png

I'm not sure why you need NAT on your eth0 though then, what are you trying to achieve with this? But that should not be the cause of the load balancing failure.

I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, but I mistakenly specified "-o eth0" instead of Internet connected interface.

The lines:
iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).

Quoted 8 lines

Quoted 2 lines

When I remove the default route in the main routing table, I completely lose Internet connectivity.   My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.

That's right, but if all your /proc/net/ip_conntrack entries contain mark values then there really must be something wrong with the fw mark <-> route interaction.

My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for instance, then maybe more people have a comparable setup to look at.

That's good advice, although I can't use kernel 2.6.27 I'm afraid.  At some point after 2.6.21 the code for a USB serial driver changed.  I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work.  (I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.)  GPRS is my only Internet option in my remote area of Costa Rica.

My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need.  (It is a drag to be tied to an old kernel version due to hardware dependency.)

Quoted 2 lines

Do you know about LARTC? The best way to get started IMO: http://lartc.org/

Yes, I downloaded the tutorial a couple of days ago, thanks!

--
Lloyd
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Hi,

I've been trying something similar to Lloyd's configuration for a long time with no success. I want to reach a host (server) from my laptop with two WiFi interfaces attached to different subnetworks/routers; the server is in the public network, both routers have public IPs (the same domain than the server) while setting up two different private networks (192.168.0.0/24 and 192.168.1.0/24), so each WiFi interface in my laptop are attached to each one of them.

The problem is that I can't manage to dynamically configure which interface to be chosen to redirect my traffic. In my case, I want the packets with destination port 8554 to be routed to one or other interface as I want (so load one network or the other), so I tried to change MARK iptables rules accordingly with no success, just getting what Lloyd had: packets are routed through the default route in the main ip routing table, while if I have not this route configured I lose network access, despite having the corresponding tables and rules properly configured. What I ended figuring out is that using the TOS target it worked properly, but this is not what I'd like to preserve as TOS changes the type of service parameter and could affect network management beyond my laptop; I need to use MARK to manage my outgoing and incoming traffic.

This is the script I'm currently using (stripped):

ip route flush table 1
ip route flush table 2
ip rule del prio 1
ip rule del prio 1
iptables -F OUTPUT -t mangle
iptables -F POSTROUTING -t nat
iptables -F PREROUTING -t nat

ip route add default via 192.168.0.1 dev ra0 ---> I need a
configure a route in the main table! Why?
ip route add table 1 192.168.0.0/24 dev ra0
ip route add table 1 default via 192.168.0.1 dev ra0 ip route add table 2 192.168.1.0/24 dev ra1
ip route add table 2 default via 192.168.1.1 dev ra1

ip rule add from all fwmark 1 table 1 prio 1
ip rule add from all fwmark 2 table 2 prio 1

iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT
--to-source 192.168.0.2

    iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j MARK --set-mark 1
    iptables -t mangle -A PREROUTING -s 147.83.47.178 -j MARK --set-mark 1

So, when I want to change the marking rules:

	iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT
--to-source 192.168.1.2
	iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j MARK --set-mark 2
	iptables -t mangle -R PREROUTING 1 -s 147.83.47.178 -j MARK --set-mark 2

Anyway, I'm not able to get this working, but only when the selected IP matches the default IP in the main routing table, so... it doesn't work. Changing MARKing rules with TOS works.

So, I can't dynamically change the used interface for the selected packets.

I opened a bug (#589) in the Netfilter bugzilla system and Jan Engelhardt has kindly tried to help me but I haven't been able to solve it yet.

By the way, I'm using kernel 2.6.27-11 wit iptables 1.4.0.

Any help would be much appreciated, as I'm not sure if this is really a bug or just a problem with iptables rules understanding or misuse.

Regards,
Javi

2009/4/20 Lloyd Standish <[email protected]>

Quoted 41 lines · Thomas Jacob

Quoted 6 lines

Quoted 2 lines

Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0). The local machine is 192.168.1.1 on the LAN.

Then your current setup in PREROUTING is what you want to go for, just keep in mind that this does not give you load balancing for connections originating from your router box, just the ones from your LAN.

I'm sorry, I don't understand. According to what you are saying, I should not get any load balancing, since all my testing up until now has been with connections (to the Internet) originating on the router box. (I haven't even tried connecting from the LAN.)

However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark, put there by my prerouting rules. Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables? (As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)

Quoted 6 lines

C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png

I'm not sure why you need NAT on your eth0 though then, what are you trying to achieve with this? But that should not be the cause of the load balancing failure.

I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, but I mistakenly specified "-o eth0" instead of Internet connected interface.

The lines:
iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).

Quoted 8 lines

Quoted 2 lines

When I remove the default route in the main routing table, I completely lose Internet connectivity. My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.

That's right, but if all your /proc/net/ip_conntrack entries contain mark values then there really must be something wrong with the fw mark <-> route interaction.

My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for instance, then maybe more people have a comparable setup to look at.

That's good advice, although I can't use kernel 2.6.27 I'm afraid. At some point after 2.6.21 the code for a USB serial driver changed. I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work. (I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.) GPRS is my only Internet option in my remote area of Costa Rica.

My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need. (It is a drag to be tied to an old kernel version due to hardware dependency.)

Quoted 2 lines

Do you know about LARTC? The best way to get started IMO: http://lartc.org/

Quoted 8 lines

Yes, I downloaded the tutorial a couple of days ago, thanks!

--
Lloyd
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

I haven't read thru all this, and you also haven't posted your TOS variant, but a few observations:

You should use CONNMARK like Lloyd does, MARK does not affect the connection, just a single packet, and you have to reconstruct your MARK values for each packet.

Your TOS rules most likely change the packet TOS field, and that is sort of "sticky" if all the intermediate hosts play ball, according to the standards, i.e. does do something like CONNMARK does.

Also be aware of the different paths thru the various chains taken by packets from/to the local host and by forwarded packets.

A small ascii diagram for your setup would be very helpful.

Quoted 133 lines · Javier Gálvez Guerrero

Hi,

I've been trying something similar to Lloyd's configuration for a long time with no success. I want to reach a host (server) from my laptop with two WiFi interfaces attached to different subnetworks/routers; the server is in the public network, both routers have public IPs (the same domain than the server) while setting up two different private networks (192.168.0.0/24 and 192.168.1.0/24), so each WiFi interface in my laptop are attached to each one of them.

The problem is that I can't manage to dynamically configure which interface to be chosen to redirect my traffic. In my case, I want the packets with destination port 8554 to be routed to one or other interface as I want (so load one network or the other), so I tried to change MARK iptables rules accordingly with no success, just getting what Lloyd had: packets are routed through the default route in the main ip routing table, while if I have not this route configured I lose network access, despite having the corresponding tables and rules properly configured. What I ended figuring out is that using the TOS target it worked properly, but this is not what I'd like to preserve as TOS changes the type of service parameter and could affect network management beyond my laptop; I need to use MARK to manage my outgoing and incoming traffic.

This is the script I'm currently using (stripped):

    ip route flush table 1
    ip route flush table 2
    ip rule del prio 1
    ip rule del prio 1
    iptables -F OUTPUT -t mangle
    iptables -F POSTROUTING -t nat
    iptables -F PREROUTING -t nat

    ip route add default via 192.168.0.1 dev ra0 ---> I need a configure a route in the main table! Why?
    ip route add table 1 192.168.0.0/24 dev ra0
    ip route add table 1 default via 192.168.0.1 dev ra0 ip route add table 2 192.168.1.0/24 dev ra1
    ip route add table 2 default via 192.168.1.1 dev ra1

    ip rule add from all fwmark 1 table 1 prio 1
    ip rule add from all fwmark 2 table 2 prio 1

    iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT --to-source 192.168.0.2
    iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j MARK --set-mark 1 iptables -t mangle -A PREROUTING -s 147.83.47.178 -j MARK --set-mark 1

So, when I want to change the marking rules:

      iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT --to-source 192.168.1.2
      iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j MARK --set-mark 2 iptables -t mangle -R PREROUTING 1 -s 147.83.47.178 -j MARK --set-mark 2

Anyway, I'm not able to get this working, but only when the selected IP matches the default IP in the main routing table, so... it doesn't work. Changing MARKing rules with TOS works.

So, I can't dynamically change the used interface for the selected packets.

I opened a bug (#589) in the Netfilter bugzilla system and Jan Engelhardt has kindly tried to help me but I haven't been able to solve it yet.

By the way, I'm using kernel 2.6.27-11 wit iptables 1.4.0.

Any help would be much appreciated, as I'm not sure if this is really a bug or just a problem with iptables rules understanding or misuse.

Regards,
Javi

2009/4/20 Lloyd Standish <[email protected]>

Quoted 41 lines · Thomas Jacob

Quoted 6 lines

Quoted 2 lines

Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0).  The local machine is 192.168.1.1 on the LAN.

Then your current setup in PREROUTING is what you want to go for, just keep in mind that this does not give you load balancing for connections originating from your router box, just the ones from your LAN.

I'm sorry, I don't understand.   According to what you are saying, I should not get any load balancing, since all my testing up until now has been with connections (to the Internet) originating on the router box.  (I haven't even tried connecting from the LAN.)

However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark, put there by my prerouting rules.  Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables?  (As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)

Quoted 6 lines

C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png

I'm not sure why you need NAT on your eth0 though then, what are you trying to achieve with this? But that should not be the cause of the load balancing failure.

I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, but I mistakenly specified "-o eth0" instead of Internet connected interface.

The lines:
iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).

Quoted 8 lines

Quoted 2 lines

When I remove the default route in the main routing table, I completely lose Internet connectivity.   My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.

That's right, but if all your /proc/net/ip_conntrack entries contain mark values then there really must be something wrong with the fw mark <-> route interaction.

My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for instance, then maybe more people have a comparable setup to look at.

That's good advice, although I can't use kernel 2.6.27 I'm afraid.  At some point after 2.6.21 the code for a USB serial driver changed.  I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work.  (I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.)  GPRS is my only Internet option in my remote area of Costa Rica.

My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need.  (It is a drag to be tied to an old kernel version due to hardware dependency.)

Quoted 2 lines

Do you know about LARTC? The best way to get started IMO: http://lartc.org/

Quoted 8 lines

Yes, I downloaded the tutorial a couple of days ago, thanks!

--
Lloyd
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

Thanks for your answer, Thomas.

Once I explained to Jan what I had to implement, he pointed me to the item 5 in http://dev.medozas.de/NF-Cookbook.txt, so I've also used CONNTRACK with no success:

"I'll try to explain what I need to implement. I'm developing a daemon application that manages WiFi handovers between two private networks (192.168.0.0/24 and 192.168.1.0/24) using two different WiFi NICs. When the link RSSI becomes low enough, the application triggers the handover in order to keep a seamless video streaming service, so first I must attach to the new network with the inactive NIC, ask for the media content through it (the video session configuration is done through the server port 8554) and, once I'm receiving the content through the new link (the media content is actually sent from and to ports differents from 8554), detach de old one.

In order to do this in a seamless way I need to play with ip route, ip rule and iptables. Marking traffic with iptables is not enough as it is done in POSTROUTING, so the kernel previously decides which IP the packets will have according to the default route in the main ip routing table. Using MARK I send the packet through the interface I want in each specific moment, while with SNAT I change the source IP to the one attached to the used interface (thus changing the IP given by the default routing table in case the new interface is not the one configured as the default one). The received (answer) packets should come through the same interface with the corresponding IP but they just follow the route configured in the default ip routing table (then not matching the connection established according to my custom setup), so I need to setup more MARKing and ip rules so the local application receives and delivers packets accordingly. This is where MARK fails and TOS success, but I don't want to mess TOS network configuration and use a more proper packet mangling with MARK as the only device that should care about the handover is the client device."

Anyway, it didn't work: as soon as I change the default route in the main table the old connection is broken. Instead, connections should be established and maintained through the links pointed in the respective tables (not the main one), according with the marking rules (that's what I need, to have two connections established with different interfaces and to different networks and dynamically chose one of both to send and receive the video stream).

Time for ASCII art (quite "simple"):

                               ,''''''''''''''`.
                               |  Server |
                               L_______J  147.83.47.178
                                 _/__
                              ,-'       `-.
                            ,'               `.
                           |                   '.
                           |     Network  |
                          /\                   /
                         /  \                /.
                       ,'    `._        _,'  `.
                      /         `----'          `.
                 |---'--------Y           ....:.............
                 |              |          |                   |
                 |Router A|           |    Router B |
                 '------------            `.................'
         192.168.0.0/24             192.168.1.0/24


                                       ,'''''''''''''''''''|
                         ra0        |               |  ra1
                                      |     Client |
                  192.168.0.2  L________|  192.168.1.2

The rules with TOS are exactly the same just changing the marking parameters by the TOS ones:

ip rule add from all tos 0x10 table 1 prio 1
ip rule add from all tos 0x04 table 2 prio 1

iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT --to-source 192.168.0.2
iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j TOS --set-tos 0x10 iptables -A PREROUTING -t mangle -s 147.83.47.178 -j TOS --set-tos 0x10

iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT --to-source 192.168.1.2
iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j TOS --set-tos 0x04 iptables -R PREROUTING 1 -t mangle -s 147.83.47.178 -j TOS --set-tos 0x04

This works but, as said before, I don't what this TOS stuff to be used.

So... any idea why the previous "item 5" didn't work for my environment? What should I try?

Thank you for your help,
Javi

Quoted 154 lines · Thomas Jacob

I haven't read thru all this, and you also haven't posted your TOS variant, but a few observations:

You should use CONNMARK like Lloyd does, MARK does not affect the connection, just a single packet, and you have to reconstruct your MARK values for each packet.

Your TOS rules most likely change the packet TOS field, and that is sort of "sticky" if all the intermediate hosts play ball, according to the standards, i.e. does do something like CONNMARK does.

Also be aware of the different paths thru the various chains taken by packets from/to the local host and by forwarded packets.

A small ascii diagram for your setup would be very helpful.

Quoted 133 lines · Javier Gálvez Guerrero

Hi,

I've been trying something similar to Lloyd's configuration for a long time with no success. I want to reach a host (server) from my laptop with two WiFi interfaces attached to different subnetworks/routers; the server is in the public network, both routers have public IPs (the same domain than the server) while setting up two different private networks (192.168.0.0/24 and 192.168.1.0/24), so each WiFi interface in my laptop are attached to each one of them.

The problem is that I can't manage to dynamically configure which interface to be chosen to redirect my traffic. In my case, I want the packets with destination port 8554 to be routed to one or other interface as I want (so load one network or the other), so I tried to change MARK iptables rules accordingly with no success, just getting what Lloyd had: packets are routed through the default route in the main ip routing table, while if I have not this route configured I lose network access, despite having the corresponding tables and rules properly configured. What I ended figuring out is that using the TOS target it worked properly, but this is not what I'd like to preserve as TOS changes the type of service parameter and could affect network management beyond my laptop; I need to use MARK to manage my outgoing and incoming traffic.

This is the script I'm currently using (stripped):

ip route flush table 1
ip route flush table 2
ip rule del prio 1
ip rule del prio 1
iptables -F OUTPUT -t mangle
iptables -F POSTROUTING -t nat
iptables -F PREROUTING -t nat

ip route add default via 192.168.0.1 dev ra0 ---> I need a
configure a route in the main table! Why?
ip route add table 1 192.168.0.0/24 dev ra0
ip route add table 1 default via 192.168.0.1 dev ra0 ip route add table 2 192.168.1.0/24 dev ra1
ip route add table 2 default via 192.168.1.1 dev ra1

ip rule add from all fwmark 1 table 1 prio 1
ip rule add from all fwmark 2 table 2 prio 1

iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT
--to-source 192.168.0.2

    iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j MARK --set-mark 1
    iptables -t mangle -A PREROUTING -s 147.83.47.178 -j MARK --set-mark 1

So, when I want to change the marking rules:

      iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT
--to-source 192.168.1.2
      iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j MARK --set-mark 2
      iptables -t mangle -R PREROUTING 1 -s 147.83.47.178 -j MARK --set-mark 2

Anyway, I'm not able to get this working, but only when the selected IP matches the default IP in the main routing table, so... it doesn't work. Changing MARKing rules with TOS works.

So, I can't dynamically change the used interface for the selected packets.

I opened a bug (#589) in the Netfilter bugzilla system and Jan Engelhardt has kindly tried to help me but I haven't been able to solve it yet.

By the way, I'm using kernel 2.6.27-11 wit iptables 1.4.0.

Any help would be much appreciated, as I'm not sure if this is really a bug or just a problem with iptables rules understanding or misuse.

Regards,
Javi

2009/4/20 Lloyd Standish <[email protected]>

Quoted 41 lines · Thomas Jacob

Quoted 6 lines

Quoted 2 lines

Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0). The local machine is 192.168.1.1 on the LAN.

Then your current setup in PREROUTING is what you want to go for, just keep in mind that this does not give you load balancing for connections originating from your router box, just the ones from your LAN.

I'm sorry, I don't understand. According to what you are saying, I should not get any load balancing, since all my testing up until now has been with connections (to the Internet) originating on the router box. (I haven't even tried connecting from the LAN.)

However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark, put there by my prerouting rules. Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables? (As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)

Quoted 6 lines

C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png

I'm not sure why you need NAT on your eth0 though then, what are you trying to achieve with this? But that should not be the cause of the load balancing failure.

I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, but I mistakenly specified "-o eth0" instead of Internet connected interface.

The lines:
iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).

Quoted 8 lines

Quoted 2 lines

When I remove the default route in the main routing table, I completely lose Internet connectivity. My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.

That's right, but if all your /proc/net/ip_conntrack entries contain mark values then there really must be something wrong with the fw mark <-> route interaction.

My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for instance, then maybe more people have a comparable setup to look at.

That's good advice, although I can't use kernel 2.6.27 I'm afraid. At some point after 2.6.21 the code for a USB serial driver changed. I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work. (I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.) GPRS is my only Internet option in my remote area of Costa Rica.

My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need. (It is a drag to be tied to an old kernel version due to hardware dependency.)

Quoted 2 lines

Do you know about LARTC? The best way to get started IMO: http://lartc.org/

Quoted 8 lines

Yes, I downloaded the tutorial a couple of days ago, thanks!

--
Lloyd
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Nice diagram ;)

Well, it is as thought, your TOS rules set TOS fields in the packets and those are usually bounced back to you from the reply direction, so effectively you are using something like CONNMARK already.

Jan's item 5 description is definitely a good starting point.

Maybe you could try this, I am assuming that you really are talking only about connections originating from client here:

   iptables -F OUTPUT -t mangle
   iptables -F POSTROUTING -t nat
   iptables -F PREROUTING -t nat

   iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT --to-source 192.168.0.2

   iptables -t mangle -A INPUT   -j CONNMARK --restore-mark

   iptables -t mangle -A OUTPUT -m state --state NEW \
       -p tcp --dport 8554 -d 147.83.47.178 -j CONNMARK --set-mark 1 iptables -t mangle -A OUTPUT   -j CONNMARK --restore-mark

If you want to load balance connections from the local net you need to use PREROUTING like Jan does. See the chain traversal diagrams I posted in the reply to Lloyd, understanding which packets pass thru which chains is really important for stuff like this.

Quoted 256 lines · Javier Gálvez Guerrero

Thanks for your answer, Thomas.

Once I explained to Jan what I had to implement, he pointed me to the item 5 in http://dev.medozas.de/NF-Cookbook.txt, so I've also used CONNTRACK with no success:

"I'll try to explain what I need to implement. I'm developing a daemon application that manages WiFi handovers between two private networks (192.168.0.0/24 and 192.168.1.0/24) using two different WiFi NICs. When the link RSSI becomes low enough, the application triggers the handover in order to keep a seamless video streaming service, so first I must attach to the new network with the inactive NIC, ask for the media content through it (the video session configuration is done through the server port 8554) and, once I'm receiving the content through the new link (the media content is actually sent from and to ports differents from 8554), detach de old one.

In order to do this in a seamless way I need to play with ip route, ip rule and iptables. Marking traffic with iptables is not enough as it is done in POSTROUTING, so the kernel previously decides which IP the packets will have according to the default route in the main ip routing table. Using MARK I send the packet through the interface I want in each specific moment, while with SNAT I change the source IP to the one attached to the used interface (thus changing the IP given by the default routing table in case the new interface is not the one configured as the default one). The received (answer) packets should come through the same interface with the corresponding IP but they just follow the route configured in the default ip routing table (then not matching the connection established according to my custom setup), so I need to setup more MARKing and ip rules so the local application receives and delivers packets accordingly. This is where MARK fails and TOS success, but I don't want to mess TOS network configuration and use a more proper packet mangling with MARK as the only device that should care about the handover is the client device."

Anyway, it didn't work: as soon as I change the default route in the main table the old connection is broken. Instead, connections should be established and maintained through the links pointed in the respective tables (not the main one), according with the marking rules (that's what I need, to have two connections established with different interfaces and to different networks and dynamically chose one of both to send and receive the video stream).

Time for ASCII art (quite "simple"):

                               ,''''''''''''''`. |  Server | L_______J  147.83.47.178 _/__
                              ,-'       `-.
                            ,'               `. |                   '. |     Network  |
                          /\                   / /  \                /. ,'    ._        _,'  . /         ----'          .

                 |---'--------Y           ....:.............
                 |              |          |                   |
                 |Router A|           |    Router B |

                 '------------            `.................' 192.168.0.0/24             192.168.1.0/24

                                       ,'''''''''''''''''''| ra0        |               |  ra1

                                      |     Client |
                  192.168.0.2  L________|  192.168.1.2

The rules with TOS are exactly the same just changing the marking parameters by the TOS ones:

ip rule add from all tos 0x10 table 1 prio 1
ip rule add from all tos 0x04 table 2 prio 1

iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT --to-source 192.168.0.2
iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j TOS --set-tos 0x10 iptables -A PREROUTING -t mangle -s 147.83.47.178 -j TOS --set-tos 0x10

iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT --to-source 192.168.1.2
iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j TOS --set-tos 0x04 iptables -R PREROUTING 1 -t mangle -s 147.83.47.178 -j TOS --set-tos 0x04

This works but, as said before, I don't what this TOS stuff to be used.

So... any idea why the previous "item 5" didn't work for my environment? What should I try?

Thank you for your help,
Javi

Quoted 154 lines · Thomas Jacob

I haven't read thru all this, and you also haven't posted your TOS variant, but a few observations:

You should use CONNMARK like Lloyd does, MARK does not affect the connection, just a single packet, and you have to reconstruct your MARK values for each packet.

Your TOS rules most likely change the packet TOS field, and that is sort of "sticky" if all the intermediate hosts play ball, according to the standards, i.e. does do something like CONNMARK does.

Also be aware of the different paths thru the various chains taken by packets from/to the local host and by forwarded packets.

A small ascii diagram for your setup would be very helpful.

Quoted 133 lines · Javier Gálvez Guerrero

Hi,

I've been trying something similar to Lloyd's configuration for a long time with no success. I want to reach a host (server) from my laptop with two WiFi interfaces attached to different subnetworks/routers; the server is in the public network, both routers have public IPs (the same domain than the server) while setting up two different private networks (192.168.0.0/24 and 192.168.1.0/24), so each WiFi interface in my laptop are attached to each one of them.

The problem is that I can't manage to dynamically configure which interface to be chosen to redirect my traffic. In my case, I want the packets with destination port 8554 to be routed to one or other interface as I want (so load one network or the other), so I tried to change MARK iptables rules accordingly with no success, just getting what Lloyd had: packets are routed through the default route in the main ip routing table, while if I have not this route configured I lose network access, despite having the corresponding tables and rules properly configured. What I ended figuring out is that using the TOS target it worked properly, but this is not what I'd like to preserve as TOS changes the type of service parameter and could affect network management beyond my laptop; I need to use MARK to manage my outgoing and incoming traffic.

This is the script I'm currently using (stripped):

    ip route flush table 1
    ip route flush table 2
    ip rule del prio 1
    ip rule del prio 1
    iptables -F OUTPUT -t mangle
    iptables -F POSTROUTING -t nat
    iptables -F PREROUTING -t nat

    ip route add default via 192.168.0.1 dev ra0 ---> I need a configure a route in the main table! Why?
    ip route add table 1 192.168.0.0/24 dev ra0
    ip route add table 1 default via 192.168.0.1 dev ra0 ip route add table 2 192.168.1.0/24 dev ra1
    ip route add table 2 default via 192.168.1.1 dev ra1

    ip rule add from all fwmark 1 table 1 prio 1
    ip rule add from all fwmark 2 table 2 prio 1

    iptables -t nat -A POSTROUTING -p tcp --dport 8554 -j SNAT --to-source 192.168.0.2
    iptables -A OUTPUT -t mangle -p tcp --dport 8554 -j MARK --set-mark 1 iptables -t mangle -A PREROUTING -s 147.83.47.178 -j MARK --set-mark 1

So, when I want to change the marking rules:

      iptables -t nat -R POSTROUTING 1 -p tcp --dport 8554 -j SNAT --to-source 192.168.1.2
      iptables -R OUTPUT 1 -t mangle -p tcp --dport 8554 -j MARK --set-mark 2 iptables -t mangle -R PREROUTING 1 -s 147.83.47.178 -j MARK --set-mark 2

Anyway, I'm not able to get this working, but only when the selected IP matches the default IP in the main routing table, so... it doesn't work. Changing MARKing rules with TOS works.

So, I can't dynamically change the used interface for the selected packets.

I opened a bug (#589) in the Netfilter bugzilla system and Jan Engelhardt has kindly tried to help me but I haven't been able to solve it yet.

By the way, I'm using kernel 2.6.27-11 wit iptables 1.4.0.

Any help would be much appreciated, as I'm not sure if this is really a bug or just a problem with iptables rules understanding or misuse.

Regards,
Javi

2009/4/20 Lloyd Standish <[email protected]>

Quoted 41 lines · Thomas Jacob

Quoted 6 lines

Quoted 2 lines

Well, I want to load-balance packets from the local machine, which is serving as gateway for a home LAN (eth0).  The local machine is 192.168.1.1 on the LAN.

Then your current setup in PREROUTING is what you want to go for, just keep in mind that this does not give you load balancing for connections originating from your router box, just the ones from your LAN.

I'm sorry, I don't understand.   According to what you are saying, I should not get any load balancing, since all my testing up until now has been with connections (to the Internet) originating on the router box.  (I haven't even tried connecting from the LAN.)

However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark, put there by my prerouting rules.  Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables?  (As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)

Quoted 6 lines

C.f.: http://ebtables.sourceforge.net/br_fw_ia/bridge3b.png

I'm not sure why you need NAT on your eth0 though then, what are you trying to achieve with this? But that should not be the cause of the load balancing failure.

I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, but I mistakenly specified "-o eth0" instead of Internet connected interface.

The lines:
iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).

Quoted 8 lines

Quoted 2 lines

When I remove the default route in the main routing table, I completely lose Internet connectivity.   My logic tells me that a default "main" route should not be necessary at all if all packets are marked and sent to my 2 custom routing tables (rt_link1/2), each of which has a default route.

That's right, but if all your /proc/net/ip_conntrack entries contain mark values then there really must be something wrong with the fw mark <-> route interaction.

My suggestion is to try this with the lastest IPtables user space and 2.6.27.X for instance, then maybe more people have a comparable setup to look at.

That's good advice, although I can't use kernel 2.6.27 I'm afraid.  At some point after 2.6.21 the code for a USB serial driver changed.  I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work.  (I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.)  GPRS is my only Internet option in my remote area of Costa Rica.

My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need.  (It is a drag to be tied to an old kernel version due to hardware dependency.)

Quoted 2 lines

Do you know about LARTC? The best way to get started IMO: http://lartc.org/

Quoted 8 lines

Yes, I downloaded the tutorial a couple of days ago, thanks!

--
Lloyd
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

$ sudo iptables -L -t mangle
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
CONNMARK   tcp  --  anywhere             anywhere            state NEW
tcp spt:rtsp CONNMARK set 0x1

With this environment I get the same results. I send the first TCP packet (SYN, dport 8554) through the interface ra1 (OK) with the IP bound to this interface (SNAT OK) and I get the (SYN,ACK) to the same IP and through the same interface (OK!), but my application does not send the final acknowledgement to the TCP connection establishment (ACK), so the RTSP messages are not sent and the client retries over and over again the TCP session establishment.

messing up old connections with other videos. I can't understand how this can be so difficult to configure. I must be missing something in my rules...

Quoted 5 lines · Javier Gálvez Guerrero

$ sudo iptables -L -t mangle
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
CONNMARK   tcp  --  anywhere             anywhere            state NEW
tcp spt:rtsp CONNMARK set 0x1

If you are forwarding packets via this host you need the CONNMARK restore here as well, then you can also drop the CONNMARK restore from the INPUT chain, PREROUTING is also traversed for packets destined for the local host.

Quoted 7 lines

With this environment I get the same results. I send the first TCP packet (SYN, dport 8554) through the interface ra1 (OK) with the IP bound to this interface (SNAT OK) and I get the (SYN,ACK) to the same IP and through the same interface (OK!), but my application does not send the final acknowledgement to the TCP connection establishment (ACK), so the RTSP messages are not sent and the client retries over and over again the TCP session establishment.

Your application does not send the ACK in the 3 way handshake, the client kernel does. Somehow it doesn't receive the SYN,ACK or the ACK does not reach the point where you're tcpdumping packets. Try to sniff as  close to your client app as possible. Could be a NAT issue. Or maybe rp_filter or something else is breaking it for you, you could try to enable /proc/sys/net/ipv4/conf/*/log_martians to see any issues.

Quoted 3 lines

messing up old connections with other videos. I can't understand how this can be so difficult to configure. I must be missing something in my rules...

Find another general purpose OS where you can do this AT ALL without additional products. Then we'll talk about what is difficult or not ;=)

I've switched to Debian Lenny 5.01 (kernel 2.6.26-2, iptables 1.4.2) and Jan's suggestion has worked out-of-the-box: the old streams are stable when changing the default route in the main table =).

Anyway, I would like to know if you have any clue about what could be happening in Ubuntu 8.10 (kernel 2.6.27-11, iptables 1.4.0), since it would be very helpful in case I need to develop some iptables-related stuff in the future.

I'm sorry, I don't understand. According to what you are saying, I should not get any load balancing,
since all my testing up until now has been with connections (to the Internet) originating on the router box. (I haven't even tried connecting from the LAN.)

However, the packets originating on the router box *are* showing up in the conntrack table with the fwmark,
put there by my prerouting rules. Is there a reason why they should not be pushed out the interface specified by the rt_link1/2 tables?
(As far as I can tell, my user-defined routing tables are ignored, and the default route in the "main" table is always used.)

I was trying to masquerade any LAN-connected machine so it could connect to the Internet through the router box, but I mistakenly specified "-o eth0" instead of Internet connected interface.

The lines:
iptables -t nat -A POSTROUTING -o ppp0 -j SNAT1
iptables -t nat -A POSTROUTING -o ppp1 -j SNAT2
should do the masquerading I suppose, although the idea was not that, but rather to fix the source address of outgoing packets to coincide with the IP of the interface (ppp0 or ppp1).

That's good advice, although I can't use kernel 2.6.27 I'm afraid.
At some point after 2.6.21 the code for a USB serial driver changed.
I have to patch that driver to make my USB-connected GPRS (ppp over GSM cell phone) modem work.

(I already hacked the patch once, after the driver code changed between kernel 2.4 and 2.6, and I don't want to have to do it again.) GPRS is my only Internet option in my remote area of Costa Rica.

My idea was to download the 2.6.18.8 kernel and use it with iptables v1.3.6, which as you pointed out previously ought to have the functionality I need.
(It is a drag to be tied to an old kernel version due to hardware dependency.)

So you if you want to load balance local packets correctly as well, you need to put some rules into OUTPUT. Possibly that's really your basic problem here, but I don't have time to think about that at the moment.

Take a look at the packet traversal graph, outgoing packets from local processes do not pass thru PREROUTING, but incoming packets do, that's where your markings come from. But in your case, I'm sure you want to select a link when the outgoing connection is first established and then stay with that link. Selecting a different link with the second packet doesn't work with NATed connections, right?

So you if you want to load balance local packets correctly as well, you need to put some rules into OUTPUT. Possibly that's really your basic problem here, but I don't have time to think about that at the moment.

Thomas, somehow I had the erroneous idea that ALL packets, even those originating at the local machine, go through PREROUTING. I finally understand now why I need the OUTPUT statements. As soon as I added those, the load balancing started to work!

# restore the fwmark on packets that belong to an existing connection
# this prerouting stuff would only be for connections initiated on the LAN

iptables -A PREROUTING -i eth0 -t mangle -m state --state ESTABLISHED,RELATED -j RESTOREMARK

iptables -A PREROUTING -i eth0 -t mangle -m mark ! --mark 0 -j RETURN iptables -A PREROUTING -t mangle -j CONNMARK1
iptables -A PREROUTING -t mangle -m statistic --mode nth --every 2 --packet 0 -j CONNMARK2

Once this is working for LAN I want to try load balancing according the byte count of each interface, if that is possible.

Quoted 2 lines

iptables -A PREROUTING -i eth0 -t mangle -m state --state ESTABLISHED,RELATED -j RESTOREMARK

This is too restrictive, drop the -i eth0, your forwarded packets from the internet will not arrive thru eth0 but still need a CONNMARK -> MARK restore.

Quoted 5 lines · Thomas Jacob

Quoted 2 lines

iptables -A PREROUTING -i eth0 -t mangle -m state --state ESTABLISHED,RELATED -j RESTOREMARK

This is too restrictive, drop the -i eth0, your forwarded packets from the internet will not arrive thru eth0 but still need a CONNMARK -> MARK restore.

Thanks for your help.

I tightened up the policies and added rules to pass any packets, for new connections or otherwise, to or from eth0 (the LAN), and drop NEW connection packets except those from the local machine and from LAN. :

iptables -t filter -P INPUT DROP
iptables -t filter -P OUTPUT ACCEPT
iptables -t filter -P FORWARD DROP

iptables -t mangle -A INPUT -i eth0 -j ACCEPT
iptables -t mangle -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -t mangle -A FORWARD -i eth0 -j ACCEPT
iptables -t mangle -A FORWARD -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT

Quoted 7 lines

I tightened up the policies and added rules to pass any packets, for new connections or otherwise, to or from eth0 (the LAN), and drop NEW connection packets except those from the local machine and from LAN. :

iptables -t filter -P INPUT DROP
iptables -t filter -P OUTPUT ACCEPT
iptables -t filter -P FORWARD DROP

Quoted 4 lines

iptables -t mangle -A INPUT -i eth0 -j ACCEPT
iptables -t mangle -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT iptables -t mangle -A FORWARD -i eth0 -j ACCEPT
iptables -t mangle -A FORWARD -o eth0 -m state --state ESTABLISHED,RELATED -j ACCEPT

Do you have the ESTABLISHED stuff also in Output? Otherwise you will block direct access to that box (see the packet traversal graph again)....

Are you forwarding packets via this box, or do you want to loadbalance packets from the local machine? In the latter case the PREROUTING stuff needs to go into INPUT/OUTPUT.