Qeues seem to stop working after an hour, and it collapses, no idea why!
Hello, this is my first message to the board, so please excuse me if i am omiting any prestablished rule ;)
Let me explain you my problem..
I have 300 users, studens, wich automatically means Edonkey,KaZaa,and most of this peer to peer sharing programs.
I actually have 2 x 2Mbit ADSL's. My solution includes load balancing for those 2 inet access with a linux firewall separing both routers and my LAN.
Right know i have a stateful firewall doing connection tracking (required for load balancing) DROPING all connections but web/smtp/pop3/https/imap etc.... That works fine, but people want to use p2p programs. After doing my research, i found out about QoS and ingress/egress queues.
My idea was to priorize critical traffic like http/smtp/pop3/ssh/etc.., granting bandwith and leaving the rest to downloads (having these the posibility of ceiling bw as well).
I have two egress and two ingress queues attached to the ethernets connected to each of the ethernet connected to the routers.
Ingress is done with IMQ, and egress with HTB.
At first it seems to work really well, but after half an hour having it working, INTERACTIVE TRAFFIC COLLAPSES, and looks as if there is no queues, hence interactive traffic is no interactive anymore.. :)
I red somewhere that to decrease latency i had to set the txqueuelen from 100 to 30, but still collapses...
I have noticed using iptraf, that the number of packets arriving to the LAN-eth (eth3 in my little sketch) is about 30% bigger than the sum of eth0 and eth2, is that normal?
I attach the configuration files of both the firewall, and the queues...
And a little graphical description of my topology:
Obviously doing SNAT and LB :)
______
<hub1> <-------> |Switch| ______
| | | | eth0
<hub2> <-------> | | |Linux | <----------> Router ADSL1
. | | eth3 | |192.168.3.5 192.168.3.6
. | | <-----> | Box |
. | |192.169.1.1| | eth2
. | | | | <----------> Router ADSL2
<hub24> <------> |______| |______|192.168.4.2 192.168.4.1I am really desperate, because it has been a month right now, and i am begging to suspect i am a bit of a fool ;) Thank you very much!!
_________________________________________________________________
Charla con tus amigos en línea mediante MSN Messenger: http://messenger.yupimsn.com/
#killall adslrunning
#/usr/local/bin/adslrunning &
#/usr/local/bin/adsl/shaper/1
#/usr/local/bin/adsl/shaper/2
ip rule del from 192.168.2.0/24
ip rule del prio 50
ip rule del from 192.168.3.0/24 prio 201
ip rule del from 192.168.4.0/24 prio 202
ip rule del prio 222
ip rule del from 192.168.2.252
ip rule del from 192.168.2.20
ip rule del from 192.168.0.0/24
ip route del default table main
ip rule add prio 50 table main
ip rule add prio 201 from 192.168.3.0/24 table 201
ip route add default via 192.168.3.3 dev eth2 src 192.168.3.5 proto static table 201
ip route append prohibit default table 201 metric 1 proto static
ip rule add prio 202 from 192.168.4.0/24 table 202
ip route add default via 192.168.4.1 dev eth0 src 192.168.4.2 proto static table 202
ip route append prohibit default table 202 metric 1 proto static
ip rule add prio 222 table 222
ip route add default table 222 proto static nexthop via 192.168.3.3 dev eth2 nexthop via 192.168.4.1 dev eth0
##################################################
## Firewall para CMU CHAMINADE
##
##
VLAN0=192.168.0.0/255.255.255.0
VLAN1=192.168.1.0/255.255.255.0
VLAN2=192.168.2.0/255.255.255.0
VLAN3=192.168.3.0/255.255.255.0
VLAN4=192.168.4.0/255.255.255.0##################################################
## Primero limpiamos todas las reglas
##
##
/sbin/iptables -t filter -F
/sbin/iptables -t nat -F
/sbin/iptables -t mangle -F
echo Limpiando reglas...
##################################################
## Aceptamos todas las conexiones## excepto las de nuestra red ##
/sbin/iptables -t filter -P FORWARD ACCEPT
##################################################
## SNAT
##
/sbin/iptables -t nat -A POSTROUTING -o eth2 -s 192.168.1.0/255.255.0.0
-j SNAT --to 192.168.3.5
/sbin/iptables -t nat -A POSTROUTING -o eth0 -s 192.168.1.0/255.255.0.0
-j SNAT --to 192.168.4.2
###################################################
## Forzamos a que los usuarios usen el proxy
##
##
# /sbin/iptables -t nat -A PREROUTING -s $VLAN2 -i eth3 -p tcp --dport 80 -j REDIRECT --to-ports 8080
# /sbin/iptables -t nat -A PREROUTING -s $VLAN0 -i eth3 -p tcp --dport 80 -j REDIRECT --to-ports 8080
# /sbin/iptables -t nat -A PREROUTING -s $VLAN1 -i eth3 -p tcp --dport 80 -j REDIRECT --to-ports 8080
###################################################
## No permitimos que utilicen otro proxy
##
##
# /sbin/iptables -A FORWARD -s $VLAN0 -d ! 192.168.1.1 -p tcp --dport 3128
-j DROP
# /sbin/iptables -A FORWARD -s $VLAN0 -d ! 192.168.1.1 -p tcp --dport 8080
-j DROP
# /sbin/iptables -A FORWARD -s $VLAN1 -d ! 192.168.1.1 -p tcp --dport 3128
-j DROP
# /sbin/iptables -A FORWARD -s $VLAN1 -d ! 192.168.1.1 -p tcp --dport 8080
-j DROP
# /sbin/iptables -A FORWARD -s $VLAN2 -d ! 192.168.1.1 -p tcp --dport 3128
-j DROP
# /sbin/iptables -A FORWARD -s $VLAN2 -d ! 192.168.1.1 -p tcp --dport 8080
-j DROP
####################################################
## Stateful Firewall
##
##
##
iptables -t filter -N keep_state iptables -t filter -A keep_state -m state --state RELATED,ESTABLISHED -j
ACCEPT
iptables -t filter -A keep_state -j RETURN
iptables -t nat -N keep_state
iptables -t nat -A keep_state -m state --state RELATED,ESTABLISHED -j
ACCEPT
iptables -t nat -A keep_state -j RETURN
iptables -t nat -A PREROUTING -j keep_state iptables -t nat -A POSTROUTING -j keep_state iptables -t nat -A OUTPUT -j keep_state
iptables -t filter -A INPUT -j keep_state iptables -t filter -A OUTPUT -j keep_state iptables -t filter -A FORWARD -j keep_state
DEV=eth2
IDEV=imq1
IQDEV=1
tc qdisc add dev ${IDEV} handle 1: root htb default 1 tc class add dev ${IDEV} parent 1: classid 1:1 htb rate 1900kbit
tc qdisc add dev ${IDEV} parent 1:1 handle 10: htb default 2
tc class add dev ${IDEV} parent 10: classid 10:1 htb rate 1500kbit burst 6k prio 1
tc class add dev ${IDEV} parent 10: classid 10:2 htb rate 512kbit ceil 1900kbit burst 6k prio 2
tc qdisc add dev ${IDEV} parent 10:1 handle 21:0 pfifo tc qdisc add dev ${IDEV} parent 10:2 handle 22:0 sfq
tc filter add dev ${IDEV} protocol ip pref 1 parent 10: handle 1 fw classid 10:1
tc filter add dev ${IDEV} protocol ip pref 2 parent 10: handle 2 fw classid 10:2
#iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
#iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128
iptables -t mangle -N IMQ${IQDEV}
iptables -t mangle -A IMQ${IQDEV} -p icmp -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 23 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 25 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 53 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 80 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 110 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 443 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 21 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 22 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 6667 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 20 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -j RETURN
iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ${IQDEV} iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ --todev ${IQDEV}
ip link set ${IDEV} up
DEV=eth0
IDEV=imq0
IQDEV=0
tc qdisc add dev ${IDEV} handle 1: root htb default 1 tc class add dev ${IDEV} parent 1: classid 1:1 htb rate 1900kbit
tc qdisc add dev ${IDEV} parent 1:1 handle 10: htb default 2
tc class add dev ${IDEV} parent 10: classid 10:1 htb rate 1500kbit burst 6k prio 1
tc class add dev ${IDEV} parent 10: classid 10:2 htb rate 512kbit ceil 1900kbit burst 6k prio 2
tc qdisc add dev ${IDEV} parent 10:1 handle 21:0 pfifo tc qdisc add dev ${IDEV} parent 10:2 handle 22:0 sfq
tc filter add dev ${IDEV} protocol ip pref 1 parent 10: handle 1 fw classid 10:1
tc filter add dev ${IDEV} protocol ip pref 2 parent 10: handle 2 fw classid 10:2
#iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
#iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss 128
iptables -t mangle -N IMQ${IQDEV}
iptables -t mangle -A IMQ${IQDEV} -p icmp -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 23 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 25 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 53 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 80 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 110 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 443 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 21 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 22 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 6667 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -p tcp --sport 20 -j MARK --set-mark 1 iptables -t mangle -A IMQ${IQDEV} -j RETURN
iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ${IQDEV} iptables -t mangle -A PREROUTING -i ${DEV} -j IMQ --todev ${IQDEV}
ip link set ${IDEV} up
DOWNLINK=1800
UPLINK=250
DEV=eth2
#tc qdisc del dev $DEV root
#tc qdisc del dev $DEV ingress
#####################
## UPLINK
##
tc qdisc add dev $DEV root handle 1: htb default 20
tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit ceil ${UPLINK}kbit
tc class add dev $DEV parent 1:1 classid 1:10 htb rate 100kbit ceil 150kbit prio 0
tc class add dev $DEV parent 1:1 classid 1:20 htb rate 100kbit ceil ${UPLINK}kbit prio 1
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10 tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
#####################
## DOWNLINK
##
#tc qdisc add dev $DEV handle ffff: ingress
#tc filter add dev $DEV parent fffff: protocol ip u32 match ip dst 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 30k drop flowid :1
############################
## Filtros de prioridades
##
tc filter add dev $DEV protocol ip parent 1:0 prio 0 handle 1 fw classid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip protocol 1 0xff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 80 0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 25 0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 110 0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 22 0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 8080 0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 53 0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 6667 0xffff flowid 1:10
DOWNLINK=1800
UPLINK=250
DEV=eth0
#tc qdisc del dev $DEV root
#tc qdisc del dev $DEV ingress
#####################
## UPLINK
##
tc qdisc add dev $DEV root handle 1: htb default 20
tc class add dev $DEV parent 1: classid 1:1 htb rate ${UPLINK}kbit ceil ${UPLINK}kbit
tc class add dev $DEV parent 1:1 classid 1:10 htb rate 200kbit ceil 150kbit prio 0
tc class add dev $DEV parent 1:1 classid 1:20 htb rate 100kbit ceil ${UPLINK}kbit prio 1
tc qdisc add dev $DEV parent 1:10 handle 10: sfq perturb 10 tc qdisc add dev $DEV parent 1:20 handle 20: sfq perturb 10
#####################
## DOWNLINK
##
#tc qdisc add dev $DEV handle ffff: ingress
#tc filter add dev $DEV parent fffff: protocol ip u32 match ip dst 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 30k drop flowid :1
############################
## Filtros de prioridades
##
tc filter add dev $DEV protocol ip parent 1:0 prio 0 handle 1 fw classid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip protocol 1 0xff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 80 0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 25 0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 110 0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 22 0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 8080 0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 53 0xffff flowid 1:10
tc filter add dev $DEV protocol ip parent 1:0 prio 1 u32 match ip dport 6667 0xffff flowid 1:10
Quoted 52 lines · GoMi .
Hello, this is my first message to the board, so please excuse me if i am omiting any prestablished rule ;)
Let me explain you my problem..
I have 300 users, studens, wich automatically means Edonkey,KaZaa,and most of this peer to peer sharing programs.
I actually have 2 x 2Mbit ADSL's. My solution includes load balancing for those 2 inet access with a linux firewall separing both routers and my LAN.
Right know i have a stateful firewall doing connection tracking (required for load balancing) DROPING all connections but web/smtp/pop3/https/imap etc.... That works fine, but people want to use p2p programs. After doing my research, i found out about QoS and ingress/egress queues.
My idea was to priorize critical traffic like http/smtp/pop3/ssh/etc.., granting bandwith and leaving the rest to downloads (having these the posibility of ceiling bw as well).
I have two egress and two ingress queues attached to the ethernets connected to each of the ethernet connected to the routers.
Ingress is done with IMQ, and egress with HTB.
At first it seems to work really well, but after half an hour having it working, INTERACTIVE TRAFFIC COLLAPSES, and looks as if there is no queues, hence interactive traffic is no interactive anymore.. :)
I red somewhere that to decrease latency i had to set the txqueuelen from 100 to 30, but still collapses...
I have noticed using iptraf, that the number of packets arriving to the LAN-eth (eth3 in my little sketch) is about 30% bigger than the sum of eth0 and eth2, is that normal?
I attach the configuration files of both the firewall, and the queues...
And a little graphical description of my topology:
Obviously doing SNAT and LB :)______<hub1> <-------> |Switch| ______
| | | | eth0
<hub2> <-------> | | |Linux | <----------> Router ADSL1 . | | eth3 | |192.168.3.5 192.168.3.6 . | | <-----> | Box | . | |192.169.1.1| | eth2 . | | | | <----------> Router ADSL2 <hub24> <------> |______| |______|192.168.4.2 192.168.4.1
Quoted 2 lines
I am really desperate, because it has been a month right now, and i am begging to suspect i am a bit of a fool ;) Thank you very much!!
I don't know the solution to you problem. But I have some remarks.
The burst you configured, is too low. If you remove the burst parameter, htb will calculate the lowest prossible for you.
Also, just as test, give each class the same prio.
Pref (or prio) if you add a filter is only used to determine the order the filters are checked. In your case, the pref doesn't mather so you take the same pref for the filters.
And is it each time after the same second/minutes that you have problems with interactive traffic? And/or does it depends on the load of the link?
Stef
Signature · 8 lines
[email protected] "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.oftc.net _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/