Skip to contentSkip to conversation

How to balance OUTBOUND traffic by packet if..

LARTC [email protected] 4 participants 11 messages Jan 7, 2005 – Jan 14, 2005 netfilter

lore
Share & tools
lore
#0 lore raw

how to use iproute/iptables to balance by packet OUTBOUND traffic between 2 PPP (pptp cable modem and pppoe adsl modem) links to the same ISP . also by using the fact that the ISP dosen't filter source IP address - meaning i can use either PPP's link assigned IP address as the source IP in the ip header.., it works for both devices.., what i want to achive is : when uploading a large file , i could use both devices to shoot data but bound only to one source IP.

Naturally i could expect all data to go back through the device with that corresponding IP , atleast that is what happens according to ethereal
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw
Quoted 10 lines · [email protected] wrote:

how to use iproute/iptables to balance by packet OUTBOUND traffic between 2 PPP (pptp cable modem and pppoe adsl modem) links to the same ISP . also by using the fact that the ISP dosen't filter source IP address - meaning i can use either PPP's link assigned IP address as the source IP in the ip header.., it works for both devices.., what i want to achive is : when uploading a large file , i could use both devices to shoot data but bound only to one source IP.

Naturally i could expect all data to go back through the device with that corresponding IP , atleast that is what happens according to ethereal

Quick copy n past of a reply I gave recently.

Another way you could look into is doing it at IP level using the netfilter patch Nth. You could just round robin packets over the links - not as nice as multilink as packet size isn't accounted for, but may be OK in practise.

Below is the help for it - You may not need to do the NAT but AIUI you could use it to mark and then route using the marks.

Andy.

Author: Fabrice MARIE <[email protected]>
Status: Works For Me.

This option adds CONFIG_IP_NF_MATCH_NTH, which supplies a match module that will allow you to match every Nth packet encountered. By default there are 16 different counters that can be used.

This match functions in one of two ways
1) Match ever Nth packet, and only the Nth packet. example:
iptables -t mangle -A PREROUTING -m nth --every 10 -j DROP
This rule will drop every 10th packet.
2) Unique rule for every packet. This is an easy and quick

     method to produce load-balancing for both inbound and outbound.
     example:
      iptables -t nat -A POSTROUTING -o eth0 -m nth --counter 7 \
               --every 3 --packet 0 -j SNAT --to-source 10.0.0.5
      iptables -t nat -A POSTROUTING -o eth0 -m nth --counter 7 \
               --every 3 --packet 1 -j SNAT --to-source 10.0.0.6
      iptables -t nat -A POSTROUTING -o eth0 -m nth --counter 7 \
               --every 3 --packet 2 -j SNAT --to-source 10.0.0.7
     This example evenly splits connections between the three SNAT
     addresses.

     By using the mangle table and iproute2, you can setup complex
     load-balanced routing.  There's lot of other uses.  Be creative!

Suppported options are:

     --every     Nth         Match every Nth packet
    [--counter]  num         Use counter 0-15 (default:0)
    [--start]    num         Initialize the counter at the number 'num'

instead of 0. Must be between 0 and Nth-1
[--packet] num Match on 'num' packet. Must be between 0
and Nth-1.
If --packet is used for a counter than there must be Nth number of --packet rules, covering all values between 0 and Nth-1 inclusively.

Andy.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#2 lore raw
In reply to [email protected]

I am still looking for a solution to this problem.......

Quoted 27 lines
Quoted 18 lines · [email protected] wrote:

Hi,
Yes i did give this a try a couple of times before with no success

/sbin/iptables -I OUTPUT -m nth --every 2 --packet 1 -t mangle -j MARK
--set-mark 0x2
/sbin/iptables -I OUTPUT -m nth --every 2 --packet 0 -t mangle -j MARK
--set-mark 0x1

ip rule :

0:      from all lookup local
201:    from all fwmark 0x2 lookup 202
201:    from all fwmark 0x1 lookup 201
32766:  from all lookup main

ip route show table 202
default via 212.199.28.244 dev ppp1 proto static src 80.178.89.120 ip route show table 201
default via 212.199.26.111 dev ppp0 proto static src 84.94.148.214 ip route show table main

Quoted 5 lines
212.199.26.111 dev ppp0  proto kernel  scope link  src 84.94.148.214
212.199.28.244 dev ppp1  proto kernel  scope link  src 80.178.89.120
default  proto static equalize
        nexthop via 212.199.26.111  dev ppp0 weight 1
        nexthop via 212.199.28.244  dev ppp1 weight 1
Quoted 8 lines

I must admit I am more into tc than ip so I've never tried it.

My guess is that you need to get rid of equalize/weights as these load balance per connection and the routes get cached.

Andy.

how to use iproute/iptables to balance by packet OUTBOUND traffic between 2 PPP (pptp cable modem and pppoe adsl modem) links to the same ISP . also by using the fact that the ISP dosen't filter source IP address - meaning i can use either PPP's link assigned IP address as the source IP in the ip header.., it works for both devices.., what i want to achive is : when uploading a large file , i could use both devices to shoot data but bound only to one source IP.

Naturally i could expect all data to go back through the device with that corresponding IP , atleast that is what happens according to ethereal

how to use iproute/iptables to balance by packet OUTBOUND traffic between 2 PPP (pptp cable modem and pppoe adsl modem) links to the same ISP . also by using the fact that the ISP dosen't filter source IP address - meaning i can use either PPP's link assigned IP address as the source IP in the ip header.., it works for both devices.., what i want to achive is : when uploading a large file , i could use both devices to shoot data but bound only to one source IP.

Naturally i could expect all data to go back through the device with that corresponding IP , atleast that is what happens according to ethereal

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#3 lore raw
In reply to [email protected]
Quoted 3 lines · [email protected] wrote:

I am still looking for a solution to this problem.......

Quoted 1 line

[email protected] wrote:

I realize this is the opposite of what you try to accomplish, but perhaps it will help?

Google
"LARTC load balance a file download across two connections - success"

gypsy
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#4 lore raw
In reply to [email protected]

Hey thanks for replying , i got this from ethereal ..:
tethereal -i ppp0
Capturing on ppp0
0.000000 80.178.69.24 -> 67.15.162.64 TCP 31604 > 53880 [ACK] Seq=0
Ack=0 WinV80 Len60
0.201730 80.178.69.24 -> 67.15.162.64 TCP [TCP Previous segment lost]
31604 > 53880 [ACK] Seq'20 Ack=0 WinV80 Len60
0.301142 80.178.69.24 -> 67.15.162.64 TCP [TCP Previous segment lost]
31604 > 53880 [PSH, ACK] SeqT40 Ack=0 WinV80 Len60
0.417813 80.178.69.24 -> 67.15.162.64 TCP [TCP Previous segment lost]
31604 > 53880 [ACK] Seq�60 Ack=0 WinV80 Len60
0.516307 80.178.69.24 -> 67.15.162.64 TCP [TCP Previous segment lost]
31604 > 53880 [ACK] Seq880 Ack=0 WinV80 Len60
0.599234 80.178.69.24 -> 67.15.162.64 TCP [TCP Previous segment lost]
31604 > 53880 [ACK] Seq600 Ack=0 WinV80 Len60
0.647668 80.178.69.24 -> 67.15.162.64 TCP [TCP Previous segment lost]
31604 > 53880 [ACK] Seq320 Ack=0 WinV80 Len60
0.699314 80.178.69.24 -> 67.15.162.64 TCP [TCP Previous segment lost]
....
tethereal -i ppp1
Capturing on ppp1
0.000000 67.15.162.64 -> 80.178.69.24 TCP 53880 > 31604 [ACK] Seq=0
Ack=0 WinD880 Len=0 SLE114001847 SRE114003207 SLE113999127 SRE114000487 SLE113996407 SRE113997767 SLE113993687
SRE113995047
0.008278 80.178.69.24 -> 67.15.162.64 TCP [TCP Previous segment lost]
31604 > 53880 [PSH, ACK] SeqC520 Ack=0 WinV80 Len60
0.014025 67.15.162.64 -> 80.178.69.24 TCP [TCP Dup ACK 1#1] 53880 >
31604 [ACK] Seq=0 Ack=0 WinF240 Len=0 SLE114004567 SRE114005927 SLE114001847 SRE114003207 SLE113999127 SRE114000487
SLE113996407 SRE113997767
0.091375 67.15.162.64 -> 80.178.69.24 TCP [TCP Dup ACK 1#2] 53880 >
31604 [ACK] Seq=0 Ack=0 WinF240 Len=0 SLE114007287 SRE114008647 SLE114004567 SRE114005927 SLE114001847 SRE114003207
SLE113999127 SRE114000487
0.098127 67.15.162.64 -> 80.178.69.24 TCP 53880 > 31604 [ACK] Seq=0
Ack'20 WinC520 Len=0 SLE114007287 SRE114008647 SLE114004567 SRE114005927 SLE114001847 SRE114003207 SLE113999127
SRE114000487
0.175701 67.15.162.64 -> 80.178.69.24 TCP [TCP Dup ACK 5#1] 53880 >
31604 [ACK] Seq=0 Ack'20 WinD880 Len=0 SLE114010007 SRE114011367 SLE114007287 SRE114008647 SLE114004567 SRE114005927
SLE114001847 SRE114003207
0.175892 80.178.69.24 -> 67.15.162.64 TCP [TCP Previous segment lost]
31604 > 53880 [ACK] SeqF240 Ack=0 WinV80 Len60
.....
Combine nth match with ROUTE target and you will get it.
ip ro re default dev ppp0
iptables -A POSTROUTING -t mangle -o ppp+ -m nth --every 2 --packet 0 -j ROUTE --oif ppp0
iptables -A POSTROUTING -t mangle -o ppp+ -m nth --every 2 --packet 1 -j ROUTE --oif ppp1
iptables -A POSTROUTING -t nat -o ppp+ -j SNAT --to $ppp0_ip_addr I just guess that it'll work, I don't have IP spoofing enabled lines to test I am still looking for a solution to this problem.......

Quoted 9 lines
Quoted 8 lines · [email protected] wrote:

Hi,
Yes i did give this a try a couple of times before with no success

/sbin/iptables -I OUTPUT -m nth --every 2 --packet 1 -t mangle -j MARK
--set-mark 0x2
/sbin/iptables -I OUTPUT -m nth --every 2 --packet 0 -t mangle -j MARK
--set-mark 0x1

Quoted 7 lines
Quoted 7 lines

ip rule :

0:      from all lookup local
201:    from all fwmark 0x2 lookup 202
201:    from all fwmark 0x1 lookup 201
32766:  from all lookup main

ip route show table 202
default via 212.199.28.244 dev ppp1 proto static src 80.178.89.120 ip

route show table 201

Quoted 1 line
Quoted 1 line

default via 212.199.26.111 dev ppp0 proto static src 84.94.148.214 ip

route show table main

Quoted 1 line
Quoted 1 line

212.199.26.111 dev ppp0 proto kernel scope link src 84.94.148.214

212.199.28.244 dev ppp1 proto kernel scope link src 80.178.89.120 default proto static equalize

Quoted 8 lines
Quoted 2 lines
        nexthop via 212.199.26.111  dev ppp0 weight 1
        nexthop via 212.199.28.244  dev ppp1 weight 1

I must admit I am more into tc than ip so I've never tried it.

My guess is that you need to get rid of equalize/weights as these load

balance per connection and the routes get cached.

Quoted 4 lines

Andy.

how to use iproute/iptables to balance by packet OUTBOUND traffic between 2 PPP (pptp cable modem and pppoe adsl modem) links to the same ISP . also by using the fact that the ISP dosen't filter source IP address - meaning i can use either PPP's link assigned IP address as the source IP in the ip header.., it works for both devices.., what i want to achive is : when uploading a large file , i could use both devices to shoot data but bound only to one source IP.

Naturally i could expect all data to go back through the device with that corresponding IP , atleast that is what happens according to ethereal
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#5 lore raw
In reply to [email protected]
Quoted 36 lines · [email protected] wrote:

I am still looking for a solution to this problem.......

Quoted 34 lines
Quoted 18 lines · [email protected] wrote:

Hi,
Yes i did give this a try a couple of times before with no success

/sbin/iptables -I OUTPUT -m nth --every 2 --packet 1 -t mangle -j MARK --set-mark 0x2
/sbin/iptables -I OUTPUT -m nth --every 2 --packet 0 -t mangle -j MARK --set-mark 0x1

ip rule :

0:      from all lookup local
201:    from all fwmark 0x2 lookup 202
201:    from all fwmark 0x1 lookup 201
32766:  from all lookup main

ip route show table 202
default via 212.199.28.244 dev ppp1 proto static src 80.178.89.120 ip route show table 201
default via 212.199.26.111 dev ppp0 proto static src 84.94.148.214 ip route show table main

Quoted 5 lines
212.199.26.111 dev ppp0  proto kernel  scope link  src 84.94.148.214
212.199.28.244 dev ppp1  proto kernel  scope link  src 80.178.89.120
default  proto static equalize
       nexthop via 212.199.26.111  dev ppp0 weight 1
       nexthop via 212.199.28.244  dev ppp1 weight 1

I must admit I am more into tc than ip so I've never tried it.

My guess is that you need to get rid of equalize/weights as these load balance per connection and the routes get cached.

Andy.

More things I would try - mark in postrouting, snat to the adress of downlink you want return to use. Get rid of the src adresses in in tables.

Andy.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#7 lore raw
In reply to [email protected]
Quoted 11 lines · [email protected] wrote:

I am still looking for a solution to this problem.......

Quoted 9 lines
Quoted 7 lines · [email protected] wrote:

Hi,
Yes i did give this a try a couple of times before with no success

/sbin/iptables -I OUTPUT -m nth --every 2 --packet 1 -t mangle -j MARK --set-mark 0x2
/sbin/iptables -I OUTPUT -m nth --every 2 --packet 0 -t mangle -j MARK --set-mark 0x1

Had a go at this on 2.6.9 / iptables 1.2.11 and it seeems nth is broken.

I'll try again tomorrow.

Andy.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#8 lore raw
Quoted 15 lines · Andy Furniss
Quoted 11 lines · [email protected] wrote:

I am still looking for a solution to this problem.......

Quoted 9 lines
Quoted 7 lines · [email protected] wrote:

Hi,
Yes i did give this a try a couple of times before with no success

/sbin/iptables -I OUTPUT -m nth --every 2 --packet 1 -t mangle -j MARK --set-mark 0x2
/sbin/iptables -I OUTPUT -m nth --every 2 --packet 0 -t mangle -j MARK --set-mark 0x1

Quoted 3 lines

Had a go at this on 2.6.9 / iptables 1.2.11 and it seeems nth is broken.

I'll try again tomorrow.

Seems still broken.

I built vanilla(apart from nth) 2.6.10, new iptables 1.2.11 + pom-200400621 with runme extra only said y to nth.

I see -

Chain OUTPUT (policy ACCEPT 817 packets, 103K bytes)
pkts bytes target prot opt in out source
destination

    15  1260 MARK       all  --  *      eth0    0.0.0.0/0 
0.0.0.0/0           every 2th packet #0 MARK set 0x1
    15  1260 MARK       all  --  *      eth0    0.0.0.0/0 

0.0.0.0/0 every 2th packet #1 MARK set 0x2

Chain POSTROUTING (policy ACCEPT 817 packets, 103K bytes)
pkts bytes target prot opt in out source
destination

     0     0            all  --  *      eth0    0.0.0.0/0 
0.0.0.0/0           MARK match 0x1
    15  1260            all  --  *      eth0    0.0.0.0/0 

0.0.0.0/0 MARK match 0x2
[root@amd /home/andy]# cat ./nth-test
iptables -t mangle -A OUTPUT -o eth0 -m nth --counter 1 --every 2 --packet 0 -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -o eth0 -m nth --counter 1 --every 2 --packet 1 -j MARK --set-mark 2
iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 1 iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 2

Any Ideas anyone?

TIA

Andy.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#9 lore raw
Quoted 28 lines · Andy Furniss

Seems still broken.

I built vanilla(apart from nth) 2.6.10, new iptables 1.2.11 + pom-200400621 with runme extra only said y to nth.

I see -

Chain OUTPUT (policy ACCEPT 817 packets, 103K bytes)

 pkts bytes target     prot opt in     out     source destination
   15  1260 MARK       all  --  *      eth0    0.0.0.0/0 

0.0.0.0/0 every 2th packet #0 MARK set 0x1
15 1260 MARK all -- * eth0 0.0.0.0/0
0.0.0.0/0 every 2th packet #1 MARK set 0x2

Chain POSTROUTING (policy ACCEPT 817 packets, 103K bytes)

 pkts bytes target     prot opt in     out     source destination
    0     0            all  --  *      eth0    0.0.0.0/0 

0.0.0.0/0 MARK match 0x1
15 1260 all -- * eth0 0.0.0.0/0
0.0.0.0/0 MARK match 0x2
[root@amd /home/andy]# cat ./nth-test
iptables -t mangle -A OUTPUT -o eth0 -m nth --counter 1 --every 2 --packet 0 -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -o eth0 -m nth --counter 1 --every 2 --packet 1 -j MARK --set-mark 2
iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 1 iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 2

Quoted 1 line

Any Ideas anyone?

Try adding "-m mark --mark X -j RETURN" rules after the MARK rules. The MARK target is non-terminal, so you are overwriting the mark in the second rule. Alternatively you can use "--start 1" in the one of the nth matches.

Regards
Patrick

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#10 lore raw
Quoted 30 lines · Patrick McHardy
Quoted 28 lines · Andy Furniss

Seems still broken.

I built vanilla(apart from nth) 2.6.10, new iptables 1.2.11 + pom-200400621 with runme extra only said y to nth.

I see -

Chain OUTPUT (policy ACCEPT 817 packets, 103K bytes)

 pkts bytes target     prot opt in     out     source destination
   15  1260 MARK       all  --  *      eth0    0.0.0.0/0 

0.0.0.0/0 every 2th packet #0 MARK set 0x1
15 1260 MARK all -- * eth0 0.0.0.0/0
0.0.0.0/0 every 2th packet #1 MARK set 0x2

Chain POSTROUTING (policy ACCEPT 817 packets, 103K bytes)

 pkts bytes target     prot opt in     out     source destination
    0     0            all  --  *      eth0    0.0.0.0/0 

0.0.0.0/0 MARK match 0x1
15 1260 all -- * eth0 0.0.0.0/0
0.0.0.0/0 MARK match 0x2
[root@amd /home/andy]# cat ./nth-test
iptables -t mangle -A OUTPUT -o eth0 -m nth --counter 1 --every 2 --packet 0 -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -o eth0 -m nth --counter 1 --every 2 --packet 1 -j MARK --set-mark 2
iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 1 iptables -t mangle -A POSTROUTING -o eth0 -m mark --mark 2

Quoted 8 lines
Quoted 1 line

Any Ideas anyone?

Try adding "-m mark --mark X -j RETURN" rules after the MARK rules. The MARK
target is non-terminal, so you are overwriting the mark in the second rule. Alternatively you can use "--start 1" in the one of the nth matches.

Thanks Patrick - I assumed that with the same counter and packet 0/1 that nth would only actually match alternate packets (start didn't work).

So this works -

iptables -t mangle -A OUTPUT -o eth0 -m nth --counter 1 --every 2 --packet 0 -j MARK --set-mark 1
iptables -t mangle -A OUTPUT -o eth0 -m mark --mark 1 -j RETURN iptables -t mangle -A OUTPUT -o eth0 -m nth --counter 1 --every 2 --packet 1 -j MARK --set-mark 2
iptables -t mangle -A OUTPUT -o eth0 -m mark --mark 2 -j RETURN

but is over complicated really - Why I didn't do it like below in the first place I don't know <slaps self>

iptables -t mangle -A OUTPUT -o eth0 -j MARK --set-mark 1 iptables -t mangle -A OUTPUT -o eth0 -m nth --every 2 -j MARK --set-mark 2

Andy.

PS had to remove [email protected] from CC as my ISP server threw a domain not found - can someone else CC him please.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/