UDP stream load balancing across multiple uplinks
I've configured a kubuntu 8.04 PC as router with one LAN interface and three Internet uplinks. A UDP stream is being produced on the LAN. I'd like the UPD packets from the stream to be load balanced across the uplinks. Currently, the UDP stream is tracked as a single connection and all the UPD packets are sent to the same uplink. What's the best way to accomplish this?
Bonding is not required in this case. The system that receives the UDP stream does not care if the UDP packets have different source IP addresses. NAT would not be required in this case, but the ISP might drop packets if the source address does not match the uplink address.
Given those assumptions, what's the best way to load balance across the uplinks? If netfilter cannot be configured to do this, can I change ip_conntrack to ignore UDP packets?
Quoted 19 lines
Why would it need to be tracked? Are you using NAT? NAT necessarily must assign one public address to the outgoing connection as it traverses the router. Therefore, unless all three uplinks can be bonded together to have the same IP address, you can not do that.
Quoted 12 lines · Barry A Rich
I've configured a kubuntu 8.04 PC as router with one LAN interface and three Internet uplinks. A UDP stream is being produced on the LAN. I'd like the UPD packets from the stream to be load balanced across the uplinks. Currently, the UDP stream is tracked as a single connection and all the UPD packets are sent to the same uplink. What's the best way to accomplish this?
--
To unsubscribe from this list: send the line "unsubscribe netfilter" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 3 lines · Barry A Rich
Bonding is not required in this case. The system that receives the UDP stream does not care if the UDP packets have different source IP addresses. NAT would not be required in this case, but the ISP might drop packets if the source address does not match the uplink address.
Given those assumptions, what's the best way to load balance across the uplinks? If netfilter cannot be configured to do this, can I change
ip_conntrackto ignore UDP packets?
Given that the source IP of your packets can change mid stream, you could use the "nth" match extension. Using "nth" you would match packets to decide how to mark them and then use the mark to determine what routing table to use which would ultimately decide which outbound path to use.
Grant. . . .
My research came to the same conclusion. Conntrack tracks the UDP stream and puts all related packets to the same interface. The UDP stream shows up in /proc/net/ip_conntrack.
Unfortunately, I cannot test without loading conntrack. The UDP stream producer uses TCP connections to coordinate with the other end. The TCP connections require conntrack.
It appears my only option is to modify conntrack. Any other opinions?
Quoted 34 lines
I'm not an expert in this area, but i believe the per-stream affinity to a given interface is because of conntrack. You could test this before expending a great deal of energy by not loading conntrack, and seeing it it still sticks to one nic.
Sofar as dynamically bypassing a loaded module for some traffic (UDP) but not all, good luck. I've never even thought of that before just now.
Quoted 1 line · Barry A Rich
Bonding is not required in this case. The system that receives the UDP
stream does not care if the UDP packets have different source IP addresses. NAT would not be required in this case, but the ISP might drop packets if the source address does not match the uplink address.
Quoted 2 lines
Given those assumptions, what's the best way to load balance across the
uplinks? If netfilter cannot be configured to do this, can I change
ip_conntrackto ignore UDP packets?Quoted 9 lines
Quoted 9 lines
Why would it need to be tracked? Are you using NAT? NAT necessarily must assign one public address to the outgoing connection as it traverses the router. Therefore, unless all three uplinks can be bonded together to have the same IP address, you can not do that.
Quoted 1 line · Barry A Rich
I've configured a kubuntu 8.04 PC as router with one LAN interface and
three
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Internet uplinks. A UDP stream is being produced on the LAN. I'd like the UPD packets from the stream to be load balanced across the uplinks. Currently, the UDP stream is tracked as a single connection and all the
UPD
Quoted 1 line
Quoted 1 line
Quoted 1 line
packets are sent to the same uplink. What's the best way to accomplish
this?
Quoted 2 lines
-----Original Message-----
From: [email protected]
[mailto:[email protected]]
Quoted 8 lines
On Behalf Of Grant Taylor
Sent: Friday, July 25, 2008 11:10 AMGiven that the source IP of your packets can change mid stream, you could use the "nth" match extension. Using "nth" you would match packets to decide how to mark them and then use the mark to determine what routing table to use which would ultimately decide which outbound path to use.
The nth match looks promising. Given that I want to avoid connection tracking, I tried the following commands:
iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -m statistic --mode nth --every 3 --packet 0 -j SNAT --to-source $UPLINK1IP iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -m statistic --mode nth --every 3 --packet 1 -j SNAT --to-source $UPLINK2IP iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -m statistic --mode nth --every 3 --packet 2 -j SNAT --to-source $UPLINK3IP
Each of these produce the following error:
iptables: Invalid argument
Any suggestions on what is wrong? Is this the correct way to bypass connection tracking?
Thanks.
Quoted 11 lines
-----Original Message-----
From: [email protected]
[mailto:[email protected]]
On Behalf Of Grant Taylor
Sent: Friday, July 25, 2008 11:10 AMGiven that the source IP of your packets can change mid stream, you could use the "nth" match extension. Using "nth" you would match packets to decide how to mark them and then use the mark to determine what routing table to use which would ultimately decide which outbound path to use.
The following setup does load balancing across the uplinks. However, the source address in the outgoing packets is not being SNAT'ed. Any suggestions?
iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -j NOTRACK
iptables -t mangle -A PREROUTING -p udp --sport $SPORT -m statistic --mode nth --every 3 --packet 1 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p udp --sport $SPORT -m statistic --mode nth --every 3 --packet 2 -j MARK --set-mark 2
iptables -t mangle -A PREROUTING -p udp --sport $SPORT -m statistic --mode nth --every 3 --packet 3 -j MARK --set-mark 3
iptables -t nat -A POSTROUTING -o $UPLINK1IF -j SNAT --to $UPLINK1IP iptables -t nat -A POSTROUTING -o $UPLINK2IF -j SNAT --to $UPLINK2IP iptables -t nat -A POSTROUTING -o $UPLINK3IF -j SNAT --to $UPLINK3IP
ip route add $UPLINK1NET dev $UPLINK1IF src $UPLINK1IP table uplink1 ip route add default via $UPLINK1GW table uplink1
ip rule add fwmark 1 table uplink1
ip route add $UPLINK2NET dev $UPLINK2IF src $UPLINK2IP table uplink2 ip route add default via $UPLINK2GW table uplink2
ip rule add fwmark 2 table uplink2
ip route add $UPLINK3NET dev $UPLINK3IF src $UPLINK23P table uplink3 ip route add default via $UPLINK3GW table uplink3
ip rule add fwmark 3 table uplink3
В Пнд, 28/07/2008 в 09:33 -0400, Barry A Rich пишет:
Quoted 12 lines
Quoted 11 lines
-----Original Message-----
From: [email protected]
[mailto:[email protected]]
On Behalf Of Grant Taylor
Sent: Friday, July 25, 2008 11:10 AMGiven that the source IP of your packets can change mid stream, you could use the "nth" match extension. Using "nth" you would match packets to decide how to mark them and then use the mark to determine what routing table to use which would ultimately decide which outbound path to use.
Quoted 3 lines
The following setup does load balancing across the uplinks. However, the source address in the outgoing packets is not being SNAT'ed. Any suggestions?
Only first packet of udp "connection" is reaching the rule in the nat table and does the job, since nat occurs for all packets in this connection rest of rules would not match.
Signature · 1 line
Покотиленко Костик <[email protected]>
Quoted 2 lines
-----Original Message-----
From: [email protected]
[mailto:[email protected]]
Quoted 19 lines
On Behalf Of ??????????? ??????
Sent: Monday, August 04, 2008 3:26 AM
To: [email protected]
Cc: 'Mail List - Netfilter'
Subject: RE: UDP stream load balancing across multiple uplinksВ Пнд, 28/07/2008 в 09:33 -0400, Barry A Rich пишет:
Quoted 12 lines
Quoted 11 lines
-----Original Message-----
From: [email protected]
[mailto:[email protected]]
On Behalf Of Grant Taylor
Sent: Friday, July 25, 2008 11:10 AMGiven that the source IP of your packets can change mid stream, you could use the "nth" match extension. Using "nth" you would match packets to decide how to mark them and then use the mark to determine what routing table to use which would ultimately decide which outbound path to use.
Quoted 7 lines
Quoted 3 lines
The following setup does load balancing across the uplinks. However, the source address in the outgoing packets is not being SNAT'ed. Any suggestions?
Only first packet of udp "connection" is reaching the rule in the nat table and does the job, since nat occurs for all packets in this connection rest of rules would not match.
I understand better. I removed the following rule:
#iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -j NOTRACK
The connection is tracked and the outgoing UDP packets are SNAT-ed. However, the source address for all UDP packets is the IP address of the uplink assigned to the connection (regardless of the uplink they actually are sent).
Is there a way to SNAT without connection tracking?
В Пнд, 04/08/2008 в 09:57 -0400, Barry A Rich пишет:
Quoted 32 lines
Quoted 2 lines
-----Original Message-----
From: [email protected][mailto:[email protected]]
Quoted 19 lines
On Behalf Of ??????????? ??????
Sent: Monday, August 04, 2008 3:26 AM
To: [email protected]
Cc: 'Mail List - Netfilter'
Subject: RE: UDP stream load balancing across multiple uplinksВ Пнд, 28/07/2008 в 09:33 -0400, Barry A Rich пишет:
Quoted 12 lines
Quoted 11 lines
-----Original Message-----
From: [email protected]
[mailto:[email protected]]
On Behalf Of Grant Taylor
Sent: Friday, July 25, 2008 11:10 AMGiven that the source IP of your packets can change mid stream, you could use the "nth" match extension. Using "nth" you would match packets to decide how to mark them and then use the mark to determine what routing table to use which would ultimately decide which outbound path to use.
Quoted 7 lines
Quoted 3 lines
The following setup does load balancing across the uplinks. However, the source address in the outgoing packets is not being SNAT'ed. Any suggestions?
Only first packet of udp "connection" is reaching the rule in the nat table and does the job, since nat occurs for all packets in this connection rest of rules would not match.
I understand better. I removed the following rule:
Quoted 8 lines
#iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -j NOTRACK
The connection is tracked and the outgoing UDP packets are SNAT-ed. However, the source address for all UDP packets is the IP address of the uplink assigned to the connection (regardless of the uplink they actually are sent).
Is there a way to SNAT without connection tracking?
Sure:
http://linux-ip.net/html/nat-stateless.html
Signature · 1 line
Покотиленко Костик <[email protected]>
Quoted 7 lines
-----Original Message-----
From: Покотиленко Костик [mailto:[email protected]]
Sent: Monday, August 04, 2008 10:41 AM
To: [email protected]
Cc: 'Mail List - Netfilter'
Subject: RE: UDP stream load balancing across multiple uplinks
Quoted 18 lines
В Пнд, 04/08/2008 в 09:57 -0400, Barry A Rich пишет:
Quoted 17 lines
Quoted 2 lines
-----Original Message-----
From: [email protected][mailto:[email protected]]
Quoted 14 lines
On Behalf Of ??????????? ??????
Sent: Monday, August 04, 2008 3:26 AM
To: [email protected]
Cc: 'Mail List - Netfilter'
Subject: RE: UDP stream load balancing across multiple uplinksВ Пнд, 28/07/2008 в 09:33 -0400, Barry A Rich пишет:
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: [email protected]
[mailto:[email protected]]
On Behalf Of Grant Taylor
Sent: Friday, July 25, 2008 11:10 AMGiven that the source IP of your packets can change mid stream,
you
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
could use the "nth" match extension. Using "nth" you would match packets to decide how to mark them and then use the mark to
determine
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
what routing table to use which would ultimately decide which
outbound
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
path to use.
The following setup does load balancing across the uplinks. However,
the
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
source address in the outgoing packets is not being SNAT'ed. Any suggestions?
Only first packet of udp "connection" is reaching the rule in the nat table and does the job, since nat occurs for all packets in this connection rest of rules would not match.
Quoted 5 lines
Quoted 5 lines
I understand better. I removed the following rule:
#iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -j NOTRACK
The connection is tracked and the outgoing UDP packets are SNAT-ed.
However,
Quoted 9 lines
Quoted 5 lines
the source address for all UDP packets is the IP address of the uplink assigned to the connection (regardless of the uplink they actually are sent).
Is there a way to SNAT without connection tracking?
Sure:
Stateless NAT with IPROUTE2 does not appear to be selective. I only want to SNAT UDP packets with a specific source port.
В Пнд, 04/08/2008 в 11:48 -0400, Barry A Rich пишет:
Quoted 52 lines
Quoted 7 lines
-----Original Message-----
From: Покотиленко Костик [mailto:[email protected]]
Sent: Monday, August 04, 2008 10:41 AM
To: [email protected]
Cc: 'Mail List - Netfilter'
Subject: RE: UDP stream load balancing across multiple uplinksQuoted 18 lines
В Пнд, 04/08/2008 в 09:57 -0400, Barry A Rich пишет:
Quoted 17 lines
Quoted 2 lines
-----Original Message-----
From: [email protected][mailto:[email protected]]
Quoted 14 lines
On Behalf Of ??????????? ??????
Sent: Monday, August 04, 2008 3:26 AM
To: [email protected]
Cc: 'Mail List - Netfilter'
Subject: RE: UDP stream load balancing across multiple uplinksВ Пнд, 28/07/2008 в 09:33 -0400, Barry A Rich пишет:
Quoted 7 lines
Quoted 7 lines
-----Original Message-----
From: [email protected]
[mailto:[email protected]]
On Behalf Of Grant Taylor
Sent: Friday, July 25, 2008 11:10 AMGiven that the source IP of your packets can change mid stream,
you
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
could use the "nth" match extension. Using "nth" you would match packets to decide how to mark them and then use the mark to
determine
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
what routing table to use which would ultimately decide which
outbound
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
Quoted 1 line
path to use.
The following setup does load balancing across the uplinks. However,
the
Quoted 7 lines
Quoted 7 lines
Quoted 6 lines
Quoted 2 lines
source address in the outgoing packets is not being SNAT'ed. Any suggestions?
Only first packet of udp "connection" is reaching the rule in the nat table and does the job, since nat occurs for all packets in this connection rest of rules would not match.
Quoted 5 lines
Quoted 5 lines
I understand better. I removed the following rule:
#iptables -t raw -A PREROUTING -i eth0 -p udp --sport $SPORT -j NOTRACK
The connection is tracked and the outgoing UDP packets are SNAT-ed.
However,
Quoted 4 lines
Quoted 4 lines
the source address for all UDP packets is the IP address of the uplink assigned to the connection (regardless of the uplink they actually are sent).
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Is there a way to SNAT without connection tracking?
Sure:
Stateless NAT with IPROUTE2 does not appear to be selective. I only want to SNAT UDP packets with a specific source port.
Maybe it is possible to use FWMARK, not sure...
Signature · 1 line
Покотиленко Костик <[email protected]>