How to reset everything
We use Netfilter to load balance UDP packets across multiple uplinks (ppp0, ppp1, ppp2, ppp3). Uplinks can be added or removed on the fly. When this happens, we reset everything and run the firewall/routing script that matches the new uplink configuration. The reset looks like this:
######################### Begin reset #########################
iptables -F INPUT
iptables -P INPUT DROP
iptables -F OUTPUT
iptables -P OUTPUT DROP
iptables -F FORWARD
iptables -P FORWARD DROP
iptables -F -t raw
iptables -F -t nat
iptables -F -t mangle
ip route del default
ip route flush table uplink1
ip route flush table uplink2
ip route flush table uplink3
ip route flush table uplink4
ip route flush dev ppp0
ip route flush dev ppp1
ip route flush dev ppp2
ip route flush dev ppp3
tc qdisc del dev ppp0 root
tc qdisc del dev ppp1 root
tc qdisc del dev ppp2 root
tc qdisc del dev ppp3 root
ip route flush cache
######################### End reset #########################
For two uplinks, the setup looks like this:
######################### Begin setup #########################
iptables -t raw -A PREROUTING -i eth0 -p udp --sport 6970 -j NOTRACK
iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode nth --every 2 --packet 0 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode nth --every 2 --packet 1 -j MARK --set-mark 2
tc qdisc add dev ppp0 root handle 1: prio
tc qdisc add dev ppp1 root handle 1: prio
tc filter add dev ppp0 parent 1:0 protocol ip prio 1 \
handle 1 fw flowid 1:1 action nat egress x.x.x.x/32 y.y.y.y
tc filter add dev ppp1 parent 1:0 protocol ip prio 1 \
handle 2 fw flowid 1:1 action nat egress x.x.x.x/32 z.z.z.z
######################### End setup #########################
The UDP stream continues to be received on the LAN interface during the reset/setup. The reset/setup works most of the time, but occasionally the packets going out ppp0 do not get NAT'd after a reset/setup. Repeating the setup/reset sequence a second time seems to make it work, but I'd rather understand what's wrong and fix it.
All help is appreciated.
Thanks.
В Срд, 06/05/2009 в 18:03 -0400, Barry A Rich пишет:
Quoted 66 lines
We use Netfilter to load balance UDP packets across multiple uplinks (ppp0, ppp1, ppp2, ppp3). Uplinks can be added or removed on the fly. When this happens, we reset everything and run the firewall/routing script that matches the new uplink configuration. The reset looks like this:
######################### Begin reset #########################
iptables -F INPUT
iptables -P INPUT DROP
iptables -F OUTPUT
iptables -P OUTPUT DROP
iptables -F FORWARD
iptables -P FORWARD DROP
iptables -F -t raw
iptables -F -t nat
iptables -F -t mangleip route del default
ip route flush table uplink1
ip route flush table uplink2
ip route flush table uplink3
ip route flush table uplink4
ip route flush dev ppp0
ip route flush dev ppp1
ip route flush dev ppp2
ip route flush dev ppp3tc qdisc del dev ppp0 root
tc qdisc del dev ppp1 root
tc qdisc del dev ppp2 root
tc qdisc del dev ppp3 rootip route flush cache
######################### End reset #########################
For two uplinks, the setup looks like this:
######################### Begin setup #########################
iptables -t raw -A PREROUTING -i eth0 -p udp --sport 6970 -j NOTRACK
iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode nth --every 2 --packet 0 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode nth --every 2 --packet 1 -j MARK --set-mark 2
tc qdisc add dev ppp0 root handle 1: prio
tc qdisc add dev ppp1 root handle 1: prio
tc filter add dev ppp0 parent 1:0 protocol ip prio 1 \
handle 1 fw flowid 1:1 action nat egress x.x.x.x/32 y.y.y.ytc filter add dev ppp1 parent 1:0 protocol ip prio 1 \
handle 2 fw flowid 1:1 action nat egress x.x.x.x/32 z.z.z.z######################### End setup #########################
The UDP stream continues to be received on the LAN interface during the reset/setup. The reset/setup works most of the time, but occasionally the packets going out ppp0 do not get NAT'd after a reset/setup. Repeating the setup/reset sequence a second time seems to make it work, but I'd rather understand what's wrong and fix it.
Quoted 3 lines
All help is appreciated.
Thanks.
Try adding to your reset script:
conntrack -F conntrack
conntrack -F expect
Signature · 1 line
Покотиленко Костик <[email protected]>
Quoted 9 lines
-----Original Message-----
From: Покотиленко Костик [mailto:[email protected]]
Sent: Thursday, May 07, 2009 6:04 AM
To: [email protected]
Cc: [email protected]
Subject: Re: How to reset everythingВ Срд, 06/05/2009 в 18:03 -0400, Barry A Rich пишет:
Quoted 1 line
We use Netfilter to load balance UDP packets across multiple uplinks
(ppp0,
Quoted 35 lines
Quoted 35 lines
ppp1, ppp2, ppp3). Uplinks can be added or removed on the fly. When this happens, we reset everything and run the firewall/routing script that matches the new uplink configuration. The reset looks like this:
######################### Begin reset #########################
iptables -F INPUT
iptables -P INPUT DROP
iptables -F OUTPUT
iptables -P OUTPUT DROP
iptables -F FORWARD
iptables -P FORWARD DROP
iptables -F -t raw
iptables -F -t nat
iptables -F -t mangleip route del default
ip route flush table uplink1
ip route flush table uplink2
ip route flush table uplink3
ip route flush table uplink4
ip route flush dev ppp0
ip route flush dev ppp1
ip route flush dev ppp2
ip route flush dev ppp3tc qdisc del dev ppp0 root
tc qdisc del dev ppp1 root
tc qdisc del dev ppp2 root
tc qdisc del dev ppp3 rootip route flush cache
######################### End reset #########################
Quoted 7 lines
Quoted 7 lines
For two uplinks, the setup looks like this:
######################### Begin setup #########################
iptables -t raw -A PREROUTING -i eth0 -p udp --sport 6970 -j NOTRACK
iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode
nth
Quoted 3 lines
Quoted 3 lines
--every 2 --packet 0 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode
nth
Quoted 9 lines
Quoted 9 lines
--every 2 --packet 1 -j MARK --set-mark 2
tc qdisc add dev ppp0 root handle 1: prio
tc qdisc add dev ppp1 root handle 1: prio
tc filter add dev ppp0 parent 1:0 protocol ip prio 1 \
handle 1 fw flowid 1:1 action nat egress x.x.x.x/32 y.y.y.y
Quoted 7 lines
Quoted 7 lines
tc filter add dev ppp1 parent 1:0 protocol ip prio 1 \
handle 2 fw flowid 1:1 action nat egress x.x.x.x/32 z.z.z.z######################### End setup #########################
The UDP stream continues to be received on the LAN interface during the reset/setup. The reset/setup works most of the time, but occasionally
the
Quoted 1 line
Quoted 1 line
packets going out ppp0 do not get NAT'd after a reset/setup. Repeating
the
Quoted 7 lines
Quoted 6 lines
setup/reset sequence a second time seems to make it work, but I'd rather understand what's wrong and fix it.
All help is appreciated.
Thanks.
Quoted 7 lines
Try adding to your reset script:
conntrack -F conntrack
conntrack -F expect--
Покотиленко Костик <[email protected]>
That did not solve the problem. Could it be packets that get queued up while the reset is in progress? Should the uplink queues get flushed? If so, how is that done?
В Чтв, 07/05/2009 в 09:49 -0400, Barry A Rich пишет:
Quoted 85 lines
Quoted 9 lines
-----Original Message-----
From: Покотиленко Костик [mailto:[email protected]]
Sent: Thursday, May 07, 2009 6:04 AM
To: [email protected]
Cc: [email protected]
Subject: Re: How to reset everythingВ Срд, 06/05/2009 в 18:03 -0400, Barry A Rich пишет:
Quoted 1 line
We use Netfilter to load balance UDP packets across multiple uplinks
(ppp0,
Quoted 35 lines
Quoted 35 lines
ppp1, ppp2, ppp3). Uplinks can be added or removed on the fly. When this happens, we reset everything and run the firewall/routing script that matches the new uplink configuration. The reset looks like this:
######################### Begin reset #########################
iptables -F INPUT
iptables -P INPUT DROP
iptables -F OUTPUT
iptables -P OUTPUT DROP
iptables -F FORWARD
iptables -P FORWARD DROP
iptables -F -t raw
iptables -F -t nat
iptables -F -t mangleip route del default
ip route flush table uplink1
ip route flush table uplink2
ip route flush table uplink3
ip route flush table uplink4
ip route flush dev ppp0
ip route flush dev ppp1
ip route flush dev ppp2
ip route flush dev ppp3tc qdisc del dev ppp0 root
tc qdisc del dev ppp1 root
tc qdisc del dev ppp2 root
tc qdisc del dev ppp3 rootip route flush cache
######################### End reset #########################
Quoted 7 lines
Quoted 7 lines
For two uplinks, the setup looks like this:
######################### Begin setup #########################
iptables -t raw -A PREROUTING -i eth0 -p udp --sport 6970 -j NOTRACK
iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode
nth
Quoted 3 lines
Quoted 3 lines
--every 2 --packet 0 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic --mode
nth
Quoted 9 lines
Quoted 9 lines
--every 2 --packet 1 -j MARK --set-mark 2
tc qdisc add dev ppp0 root handle 1: prio
tc qdisc add dev ppp1 root handle 1: prio
tc filter add dev ppp0 parent 1:0 protocol ip prio 1 \
handle 1 fw flowid 1:1 action nat egress x.x.x.x/32 y.y.y.yQuoted 7 lines
Quoted 7 lines
tc filter add dev ppp1 parent 1:0 protocol ip prio 1 \
handle 2 fw flowid 1:1 action nat egress x.x.x.x/32 z.z.z.z######################### End setup #########################
The UDP stream continues to be received on the LAN interface during the reset/setup. The reset/setup works most of the time, but occasionally
the
Quoted 1 line
Quoted 1 line
packets going out ppp0 do not get NAT'd after a reset/setup. Repeating
the
Quoted 9 lines
Quoted 6 lines
setup/reset sequence a second time seems to make it work, but I'd rather understand what's wrong and fix it.
All help is appreciated.
Thanks.
Try adding to your reset script:
Quoted 8 lines
Quoted 5 lines
conntrack -F conntrack
conntrack -F expect--
Покотиленко Костик <[email protected]>That did not solve the problem. Could it be packets that get queued up while the reset is in progress?
If this is the case the delay shouldn't be long. Also, if this is the case next packets should behave as you expected and you wouldn't be reseting second time.
Quoted 2 lines
Should the uplink queues get flushed? If so, how
is that done?
tc qdics del && tc qdisc add(???)
Signature · 1 line
Покотиленко Костик <[email protected]>
Quoted 2 lines
-----Original Message-----
From: [email protected]
[mailto:[email protected]]
Quoted 6 lines
On Behalf Of ??????????? ??????
Sent: Thursday, May 07, 2009 11:52 AM
To: [email protected]
Cc: [email protected]
Subject: RE: How to reset everything
Quoted 12 lines
В Чтв, 07/05/2009 в 09:49 -0400, Barry A Rich пишет:
Quoted 11 lines
Quoted 9 lines
-----Original Message-----
From: Покотиленко Костик [mailto:[email protected]]
Sent: Thursday, May 07, 2009 6:04 AM
To: [email protected]
Cc: [email protected]
Subject: Re: How to reset everythingВ Срд, 06/05/2009 в 18:03 -0400, Barry A Rich пишет:
Quoted 1 line
We use Netfilter to load balance UDP packets across multiple uplinks
(ppp0,
Quoted 1 line
Quoted 1 line
ppp1, ppp2, ppp3). Uplinks can be added or removed on the fly. When
this
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
happens, we reset everything and run the firewall/routing script
that
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
Quoted 33 lines
matches the new uplink configuration. The reset looks like this:
######################### Begin reset #########################
iptables -F INPUT
iptables -P INPUT DROP
iptables -F OUTPUT
iptables -P OUTPUT DROP
iptables -F FORWARD
iptables -P FORWARD DROP
iptables -F -t raw
iptables -F -t nat
iptables -F -t mangleip route del default
ip route flush table uplink1
ip route flush table uplink2
ip route flush table uplink3
ip route flush table uplink4
ip route flush dev ppp0
ip route flush dev ppp1
ip route flush dev ppp2
ip route flush dev ppp3tc qdisc del dev ppp0 root
tc qdisc del dev ppp1 root
tc qdisc del dev ppp2 root
tc qdisc del dev ppp3 rootip route flush cache
######################### End reset #########################
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
For two uplinks, the setup looks like this:
######################### Begin setup #########################
iptables -t raw -A PREROUTING -i eth0 -p udp --sport 6970 -j NOTRACK
iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic
--mode
Quoted 4 lines
Quoted 4 lines
nth
Quoted 3 lines
Quoted 3 lines
--every 2 --packet 0 -j MARK --set-mark 1
iptables -t mangle -A PREROUTING -p udp --sport 6970 -m statistic
--mode
Quoted 10 lines
Quoted 10 lines
nth
Quoted 9 lines
Quoted 9 lines
--every 2 --packet 1 -j MARK --set-mark 2
tc qdisc add dev ppp0 root handle 1: prio
tc qdisc add dev ppp1 root handle 1: prio
tc filter add dev ppp0 parent 1:0 protocol ip prio 1 \
handle 1 fw flowid 1:1 action nat egress x.x.x.x/32 y.y.y.y
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
tc filter add dev ppp1 parent 1:0 protocol ip prio 1 \
handle 2 fw flowid 1:1 action nat egress x.x.x.x/32 z.z.z.z######################### End setup #########################
The UDP stream continues to be received on the LAN interface during
the
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
reset/setup. The reset/setup works most of the time, but
occasionally
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
packets going out ppp0 do not get NAT'd after a reset/setup.
Repeating
Quoted 2 lines
Quoted 2 lines
the
Quoted 1 line
Quoted 1 line
setup/reset sequence a second time seems to make it work, but I'd
rather
Quoted 8 lines
Quoted 8 lines
Quoted 8 lines
Quoted 5 lines
understand what's wrong and fix it.
All help is appreciated.
Thanks.
Try adding to your reset script:
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
conntrack -F conntrack
conntrack -F expect--
Покотиленко Костик <[email protected]>That did not solve the problem. Could it be packets that get queued up
while
Quoted 6 lines
Quoted 1 line
the reset is in progress?
If this is the case the delay shouldn't be long. Also, if this is the case next packets should behave as you expected and you wouldn't be reseting second time.
Quoted 7 lines
Quoted 2 lines
Should the uplink queues get flushed? If so, how
is that done?tc qdics del && tc qdisc add(???)
--
Покотиленко Костик <[email protected]>
When this problem occurs after a reset/setup, the UDP packets are still routed to the both ppp0 and ppp1. They just are not NAT'd on ppp0.
This seems to indicate:
- The statistic module is correctly marking the packets, and
- The packets are being routed correctly, and
- The traffic control filter is not NAT'ing the packets.
Is there a way to log the packets (with the marks) as they exit traffic control to verify this?