Skip to contentSkip to conversation

Wierd Policy Routing Behaviour

LARTC Sellaro 3 participants 12 messages May 30, 2002 – May 31, 2002

lore
Share & tools
lore

Sellaro

author
#0 lore raw

Hi there

I'm trying to set up policy routing in a simple scenario without further success. My linux router is connected to two different cable ISPs. One of these links (hereafter named A) permits traffic to SMTP servers while the other (link B) doesn't.

What I am trying to set up is: all traffic should flow through link B and only SMTP traffic through link A.

What I am doing:

Marking all packets from my intranet with source port within the range 1024 to 65535 and with destination port 25 with mark 0x3. I am also marking packets from our internal SMTP server with source port 25 to any other port with the same mark. Marks are being made in the OUTPUT chain (using iptables) as follows:

iptables -t mangle -A OUTPUT -p tcp --sport 1024:65535 --dport 25 -j MARK --set-mark 3

iptables -t mangle -A OUTPUT -p tcp -s <MY SMTP SERVER> --sport 25 -j MARK --set-mark 3

Then, I have created a new routing table named SMTP-ROUTE with default gateway being the default gateway for link A as follows:

ip ro add default via <A's default GW> table SMTP-ROUTE

To complete the setup, I've added a rule stating that all packets marked with 0x3 should use SMTP-ROUTE with this command:

ip ru add fwmark 3 table SMTP-ROUTE

Yes, I've flushed the routing cache with:

ip ro flush cache

Default route in main routing table is B's gateway.

I don't know why, but SMTP traffic keeps going through B's gateway, instead of A's, as expected.

I've sniffed the network and, in fact, the packets are trying to go out through B's gateway.

Can anyone please point what I am doing wrong?

Thank you in advance

Signature · 10 lines
Sellaro

Agente Livre - Linux Community (www.agentelivre.org)

PGP Key ID: 3ADF8645
PGP Key Fingerprint: 6AB0 D60B 69B5 B3F9 4553  2242 A1D0 17C0 3ADF 8645

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#1 lore raw

Sellaro,

I'm not sure about this, (others on the list may confirm), but I think you want to use --set-tos instead of --set-mark.

My understanding is that the mark only exists while the packet is on the local machine, but when you set the ToS flags, then you are changing the packet.

Best of luck,

-Martin

: Hi there
:
: I'm trying to set up policy routing in a simple scenario without further : success. My linux router is connected to two different cable ISPs. One : of these links (hereafter named A) permits traffic to SMTP servers while : the other (link B) doesn't.
:
: What I am trying to set up is: all traffic should flow through link B : and only SMTP traffic through link A.
:
: What I am doing:
:
: Marking all packets from my intranet with source port within the range : 1024 to 65535 and with destination port 25 with mark 0x3. I am also : marking packets from our internal SMTP server with source port 25 to any : other port with the same mark. Marks are being made in the OUTPUT chain : (using iptables) as follows:
:
: iptables -t mangle -A OUTPUT -p tcp --sport 1024:65535 --dport 25 -j : MARK --set-mark 3
:
: iptables -t mangle -A OUTPUT -p tcp -s <MY SMTP SERVER> --sport 25 -j : MARK --set-mark 3
:
: Then, I have created a new routing table named SMTP-ROUTE with default : gateway being the default gateway for link A as follows:
:
: ip ro add default via <A's default GW> table SMTP-ROUTE
:
: To complete the setup, I've added a rule stating that all packets marked : with 0x3 should use SMTP-ROUTE with this command:
:
: ip ru add fwmark 3 table SMTP-ROUTE
:
: Yes, I've flushed the routing cache with:
:
: ip ro flush cache
:
: Default route in main routing table is B's gateway.
:
: I don't know why, but SMTP traffic keeps going through B's gateway, : instead of A's, as expected.
:
: I've sniffed the network and, in fact, the packets are trying to go out : through B's gateway.
:
: Can anyone please point what I am doing wrong?
:
: Thank you in advance
:

Signature · 6 lines
Martin A. Brown --- SecurePipe, Inc. --- [email protected]


_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
author
#2 lore raw
In reply to Sellaro
Quoted 2 lines · Martin A. Brown

I'm not sure about this, (others on the list may confirm), but I think you want to use --set-tos instead of --set-mark.

I have not tried to use ToS as key for classifying the packets. I'll try to, eventhough I think there's nothing wrong with --set-mark.

Quoted 3 lines

My understanding is that the mark only exists while the packet is on the local machine, but when you set the ToS flags, then you are changing the packet.

But the packets are being marked on the router itself. That's why I think there is nothing wrong (explicitlly, at least) with the solution I'm trying to use. Anyway, I'll wait for comments from others on the list.

Thank you for your help.

Signature · 10 lines
Sellaro

Agente Livre - Linux Community (www.agentelivre.org)

PGP Key ID: 3ADF8645
PGP Key Fingerprint: 6AB0 D60B 69B5 B3F9 4553  2242 A1D0 17C0 3ADF 8645

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#3 lore raw
In reply to Sellaro

Without iproute
iptables -t nat -A POSTROUTING -s $SMTP -p tcp --sport 1024: --dport 25 -j SNAT --to-source $IP_B

Quoted 43 lines · Sellaro

Hi there

I'm trying to set up policy routing in a simple scenario without further success. My linux router is connected to two different cable ISPs. One of these links (hereafter named A) permits traffic to SMTP servers while the other (link B) doesn't.

What I am trying to set up is: all traffic should flow through link B and only SMTP traffic through link A.

What I am doing:

Marking all packets from my intranet with source port within the range 1024 to 65535 and with destination port 25 with mark 0x3. I am also marking packets from our internal SMTP server with source port 25 to any other port with the same mark. Marks are being made in the OUTPUT chain (using iptables) as follows:

iptables -t mangle -A OUTPUT -p tcp --sport 1024:65535 --dport 25 -j MARK --set-mark 3

iptables -t mangle -A OUTPUT -p tcp -s <MY SMTP SERVER> --sport 25 -j MARK --set-mark 3

Then, I have created a new routing table named SMTP-ROUTE with default gateway being the default gateway for link A as follows:

ip ro add default via <A's default GW> table SMTP-ROUTE

To complete the setup, I've added a rule stating that all packets marked with 0x3 should use SMTP-ROUTE with this command:

ip ru add fwmark 3 table SMTP-ROUTE

Yes, I've flushed the routing cache with:

ip ro flush cache

Default route in main routing table is B's gateway.

I don't know why, but SMTP traffic keeps going through B's gateway, instead of A's, as expected.

Quoted 18 lines

I've sniffed the network and, in fact, the packets are trying to go out through B's gateway.

Can anyone please point what I am doing wrong?

Thank you in advance

Signature · 10 lines
Sellaro

Agente Livre - Linux Community (www.agentelivre.org)

PGP Key ID: 3ADF8645
PGP Key Fingerprint: 6AB0 D60B 69B5 B3F9 4553  2242 A1D0 17C0 3ADF 8645

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

-----------------------------------
mailto:[email protected]
BR
Alexey Talikov
FORTEK
-----------------------------------

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

author
#4 lore raw
In reply to Sellaro
Quoted 2 lines · Alexey Talikov

Without iproute
iptables -t nat -A POSTROUTING -s $SMTP -p tcp --sport 1024: --dport 25 -j SNAT --to-source $IP_B

Source NAT is not what I want to do. I really want to send SMTP packets through a different route.

Signature · 10 lines
Sellaro

Agente Livre - Linux Community (www.agentelivre.org)

PGP Key ID: 3ADF8645
PGP Key Fingerprint: 6AB0 D60B 69B5 B3F9 4553  2242 A1D0 17C0 3ADF 8645

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#5 lore raw
In reply to Sellaro

IP: advanced router (CONFIG_IP_ADVANCED_ROUTER) [Y/n/?]
IP: policy routing (CONFIG_IP_MULTIPLE_TABLES) [Y/n/?]
IP: use netfilter MARK value as routing key (CONFIG_IP_ROUTE_FWMARK) [Y/n/?]

iptables -t mangle -A PREROUTING -p tcp --dport 25 -j MARK --set-mark 1 echo 201 mail.out >> /etc/iproute2/rt_tables
ip rule add fwmark 1 table mail.out
ip route add default via $IP dev $DEV table mail.out

for more detail see Advanced-Routing HOWTO Chapter 11

Quoted 4 lines · Sellaro
Quoted 2 lines · Alexey Talikov

Without iproute
iptables -t nat -A POSTROUTING -s $SMTP -p tcp --sport 1024: --dport 25 -j SNAT --to-source

$IP_B

Quoted 12 lines

Source NAT is not what I want to do. I really want to send SMTP packets through a different route.

Signature · 6 lines
Sellaro

Agente Livre - Linux Community (www.agentelivre.org)

PGP Key ID: 3ADF8645
PGP Key Fingerprint: 6AB0 D60B 69B5 B3F9 4553  2242 A1D0 17C0 3ADF 8645

-----------------------------------
mailto:[email protected]
BR
Alexey Talikov
FORTEK
-----------------------------------

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

author
#6 lore raw
In reply to Sellaro
Quoted 3 lines

IP: advanced router (CONFIG_IP_ADVANCED_ROUTER) [Y/n/?]
IP: policy routing (CONFIG_IP_MULTIPLE_TABLES) [Y/n/?]
IP: use netfilter MARK value as routing key (CONFIG_IP_ROUTE_FWMARK)

[Y/n/?]

internet:/usr/src/linux# cat .config | grep ROUTER
CONFIG_IP_ADVANCED_ROUTER=y
internet:/usr/src/linux# cat .config | grep MULTIPLE
CONFIG_IP_MULTIPLE_TABLES=y
internet:/usr/src/linux# cat .config | grep FWMARK  
CONFIG_IP_ROUTE_FWMARK=y
Quoted 1 line

iptables -t mangle -A PREROUTING -p tcp --dport 25 -j MARK --set-mark 1

OK, I'm marking with 3, not 1, but I think this will not cause any problem:

iptables -t mangle -A PREROUTING -p tcp --dport 25 -j MARK --set-mark 3

Quoted 1 line

echo 201 mail.out >> /etc/iproute2/rt_tables

OK. But table is named only mail.

Quoted 1 line

ip rule add fwmark 1 table mail.out

ip ru add fwmark 3 table mail

ip ru ls
0:
from all lookup local
32765:
from all fwmark 3 lookup mail
32766:
from all lookup main
32767:
from all lookup default

Quoted 1 line

ip route add default via $IP dev $DEV table mail.out

Done.

You've forgoten to mention flushing route cahce. I did it with:
ip ro flush cache

And nothing happens as expected. That's why I am writting to the list.

Thank you for your help.

--
Sellaro

Network Management for the Masses

--
Prendi GRATIS l'email universale che... risparmia: http://www.email.it/f

Sponsor:
Tutti i bulbi in offerta... da Peraga risparmi il 30%! Clicca qui: http://adv2.email.it/cgi-bin/foclick.cgi?mid=448&d=31-5
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#7 lore raw
In reply to Sellaro

Your don't add $DEV
ip route add default via $IP dev $DEV table mail.out

Quoted 56 lines · Sellaro
Quoted 3 lines

IP: advanced router (CONFIG_IP_ADVANCED_ROUTER) [Y/n/?]
IP: policy routing (CONFIG_IP_MULTIPLE_TABLES) [Y/n/?]
IP: use netfilter MARK value as routing key (CONFIG_IP_ROUTE_FWMARK)

[Y/n/?]

internet:/usr/src/linux# cat .config | grep ROUTER
CONFIG_IP_ADVANCED_ROUTER=y
internet:/usr/src/linux# cat .config | grep MULTIPLE
CONFIG_IP_MULTIPLE_TABLES=y
internet:/usr/src/linux# cat .config | grep FWMARK  
CONFIG_IP_ROUTE_FWMARK=y
Quoted 1 line

iptables -t mangle -A PREROUTING -p tcp --dport 25 -j MARK --set-mark 1

OK, I'm marking with 3, not 1, but I think this will not cause any problem:

iptables -t mangle -A PREROUTING -p tcp --dport 25 -j MARK --set-mark 3

Quoted 1 line

echo 201 mail.out >> /etc/iproute2/rt_tables

OK. But table is named only mail.

Quoted 1 line

ip rule add fwmark 1 table mail.out

ip ru add fwmark 3 table mail

ip ru ls
0:
from all lookup local
32765:
from all fwmark 3 lookup mail
32766:
from all lookup main
32767:
from all lookup default

Quoted 1 line

ip route add default via $IP dev $DEV table mail.out

Done.

You've forgoten to mention flushing route cahce. I did it with:
ip ro flush cache

And nothing happens as expected. That's why I am writting to the list.

Thank you for your help.

--
Sellaro

Network Management for the Masses

Quoted 7 lines

--
Prendi GRATIS l'email universale che... risparmia: http://www.email.it/f

Sponsor:
Tutti i bulbi in offerta... da Peraga risparmi il 30%! Clicca qui: http://adv2.email.it/cgi-bin/foclick.cgi?midD8&d1-5

-----------------------------------
mailto:[email protected]
BR
Alexey Talikov
FORTEK
-----------------------------------

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

author
#8 lore raw
In reply to Sellaro
Quoted 2 lines · Alexey Talikov

Your don't add $DEV
ip route add default via $IP dev $DEV table mail.out

Done. But this is not the problem. I've found a post (reading Advanced Routing Howto again) from Rusty speaking about MASQ and fwmark problems. The post is available at this address:
http://lists.samba.org/pipermail/netfilter/2000-November/006089.html

OK. Now we have a little change on our scenario. All hosts from my INTRANET are able to connect to external SMTP servers properly. However routing itself is not. Sniffing the netowrk shows that from the router the packets are going out through the wrong interface.

I've also added the same same marking rule to OUTPUT table, but it didn't work. Any clues?

Thank you in advance.

Signature · 10 lines
Sellaro

Agente Livre - Linux Community (www.agentelivre.org)

PGP Key ID: 3ADF8645
PGP Key Fingerprint: 6AB0 D60B 69B5 B3F9 4553  2242 A1D0 17C0 3ADF 8645

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#9 lore raw
In reply to Sellaro

Version of kernel iptables and iproute , any patches ???

Quoted 15 lines · Sellaro
Quoted 2 lines · Alexey Talikov

Your don't add $DEV
ip route add default via $IP dev $DEV table mail.out

Done. But this is not the problem. I've found a post (reading Advanced Routing Howto again) from Rusty speaking about MASQ and fwmark problems. The post is available at this address:
http://lists.samba.org/pipermail/netfilter/2000-November/006089.html

OK. Now we have a little change on our scenario. All hosts from my INTRANET are able to connect to external SMTP servers properly. However routing itself is not. Sniffing the netowrk shows that from the router the packets are going out through the wrong interface.

Quoted 12 lines

I've also added the same same marking rule to OUTPUT table, but it didn't work. Any clues?

Thank you in advance.

Signature · 6 lines
Sellaro

Agente Livre - Linux Community (www.agentelivre.org)

PGP Key ID: 3ADF8645
PGP Key Fingerprint: 6AB0 D60B 69B5 B3F9 4553  2242 A1D0 17C0 3ADF 8645

-----------------------------------
mailto:[email protected]
BR
Alexey Talikov
FORTEK
-----------------------------------

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

author
#10 lore raw
In reply to Sellaro
Quoted 1 line · Alexey Talikov

Version of kernel iptables and iproute , any patches ???

Sorry for not mentioning before.

kernel: 2.4.18
iproute: 20010824-7
Patches: only HTB 3 into the kernel, of course.

Signature · 10 lines
Sellaro

Agente Livre - Linux Community (www.agentelivre.org)

PGP Key ID: 3ADF8645
PGP Key Fingerprint: 6AB0 D60B 69B5 B3F9 4553  2242 A1D0 17C0 3ADF 8645

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#11 lore raw
In reply to Sellaro

add input interface in rule because this table general for all interfaces also add ip if possible iptables -t mangle -A PREROUTING -i $DEV -p tcp --dport 25 -j MARK --set-mark 1 if don't work
try
iptables -t mangle -A FORWARD -i $DEV -p tcp --dport 25 -j MARK --set-mark 1 (don't work for old iptables and kernel, missing FORWARD and INPUT chain for mangle table)

if you compile netfilter as modules -- all necessary loaded ?

if don't work try to test mark with tc filter
try to use last iproute2-2.4.7-020116 (possible that the ip route can't see your marks and last htb-3.6 (Thanks Devik !)

Send me results !!

(Sorry for my pure English :(( )

Quoted 18 lines · Sellaro
Quoted 1 line · Alexey Talikov

Version of kernel iptables and iproute , any patches ???

Sorry for not mentioning before.

kernel: 2.4.18
iproute: 20010824-7
Patches: only HTB 3 into the kernel, of course.

Signature · 6 lines
Sellaro

Agente Livre - Linux Community (www.agentelivre.org)

PGP Key ID: 3ADF8645
PGP Key Fingerprint: 6AB0 D60B 69B5 B3F9 4553  2242 A1D0 17C0 3ADF 8645

-----------------------------------
mailto:[email protected]
BR
Alexey Talikov
FORTEK
-----------------------------------

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/