Dynamic Ratelimiting
I've been using linux routing (htb qdisc) for almost a year now to try and manage the network here in a college environment. One of the major problems that I faced when I started this "little" project was P2P upload/downloads. At times the network would slow down so much one couldn't even load a webpage. I've tried the ratelimiting of certain ports, prioritizing certain blocks of IP, but all of it seems to be "less than ideal." We had continued to have problems with legitimate traffic being limited, our VoIP network was degraded (even after prioritizing), and our mirroring of slackware.com and cpan.org was less than glorious. It was workable but it was no way a good scene. After analyzing traffic, I thought it would have been inefficient to try and look into the data portion of the datagram but what I did notice about the traffic we had here was that the P2P machines had an unusually high number of connections. For out network, the number of connections was something that could
easily be monitored. So, I've created a few scripts that used
iptables, tc, and a sniffer that dynamically ratelimits machines(IPs). I've been using this script for awhile and it has done wonders for our network. A side effect of the scripts has been a ratelimiting of new Windows(tm) worm scans, port scans, and anything else that makes an unusually high number of connections. The VoIP traffic finally is usable (ideal?), and our mirrors work great. The project (I've called in 'pacemaker') is pretty configurable in that you can ignore certain hosts, networks, or ports if you know you would never want to ratelimit those resources based on number of connections. Seeing that it work so well here, I thought I'd offer it to the open source community and see if they could give me any pointers on making pacemaker better.
You can find the network statistics pages here:
http://mrtg.saintjoe.edu/
and pacemaker specifically here:
http://mrtg.saintjoe.edu/mrtg/ratelimit/pacemaker/
peace
Signature · 14 lines
David DeLauro Computer Systems Analyst Saint Joseph's College Rensselaer, IN 47978 Do not handicap your children by making their lives easy. - Robert Heinlein Hata ukinichukia la kweli nitakwambia - Kanga Proverb I have often regretted my speech, never my silence. - Xenocrates _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 7 lines
It was workable but it was no way a good scene. After analyzing traffic, I thought it would have been inefficient to try and look into the data portion of the datagram but what I did notice about the traffic we had here was that the P2P machines had an unusually high number of connections. For out network, the number of connections was something that could
easily be monitored. So, I've created a few scripts that used
iptables, tc, and a sniffer that dynamically ratelimits machines(IPs).
Very interesting, I'll look more in depth to your scripts ... do you think it would be easy to change the decission of who to ratelimit, from the number of connections to the bandwidth they are using?
Something like, if this user is using 512kb for 5 minutes ratelimit him?
Signature · 5 lines
Damjan Georgievski jabberID: [email protected] _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Damjan,
I am working in a script to do something like that, can you post your script for us?
Thanks in Advance,
Anderson
----- Original Message -----
From: "Damjan" <[email protected]>
To: <[email protected]>
Cc: "David DeLauro" <[email protected]>
Sent: Tuesday, December 23, 2003 6:28 PM
Subject: Re: [LARTC] Dynamic Ratelimiting
Quoted 1 line
Quoted 1 line
It was workable but it was no way a good scene. After analyzing
traffic,
Quoted 3 lines
Quoted 3 lines
I thought it would have been inefficient to try and look into the data portion of the datagram but what I did notice about the traffic we had here was that the P2P machines had an unusually high number of
connections.
Quoted 8 lines
Quoted 3 lines
For out network, the number of connections was something that could
easily be monitored. So, I've created a few scripts that used
iptables, tc, and a sniffer that dynamically ratelimits machines(IPs).Very interesting, I'll look more in depth to your scripts ... do you think it would be easy to change the decission of who to ratelimit, from the number of connections to the bandwidth they are using?
Quoted 8 lines
Something like, if this user is using 512kb for 5 minutes ratelimit him?
--
Damjan Georgievski
jabberID: [email protected]
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 3 lines
I am working in a script to do something like that, can you post your script
for us?
I have my own hard-stuff for solve that problem.
I measure all of my clients for long time (tc+parser+sql)
After some period of time, 90% of my LAN clients get large autmated bandwidth speeds , thus the rest about 10% has much worst speeds. My system increase/decrese client speeds based on per client policy (kept in DB):
It is Per User Policy :
Array ( "LBS_TBS_MAX_ANDOR" => Array ( "T1"=>"OR", "NIGHT"=>"OR", "COMB"=>"OR", ), "LBS_TBS_MIN_ANDOR" => Array ( "T1"=>"AND", "NIGHT"=>"AND", "COMB"=>"AND", ), "LBS_MIN_OK" => Array ( "T1"=>"2000", "NIGHT"=>"1500", "COMB"=>"4000", ), "LBS_MAX_OK" => Array ( "T1"=>"4000", "NIGHT"=>"5000", "COMB"=>"9000", ), "TBS_MIN_OK" => Array ( "T1"=>"1800", "NIGHT"=>"2000", "COMB"=>"4000", ), "TBS_MAX_OK" => Array ( "T1"=>"3000", "NIGHT"=>"2500", "COMB"=>"8000", ), "INC_STEP" => Array ( "T1"=>"10", "NIGHT"=>"0", "COMB"=>"0", ), "DEC_STEP" => Array ( "T1"=>"30", "NIGHT"=>"30", "COMB"=>"3", ), "MAX_SPD" => Array ( "T1"=>"150", "NIGHT"=>"150", "COMB"=>"150", ), "MIN_SPD" => Array ( "T1"=>"50", "NIGHT"=>"50", "COMB"=>"50", ), "LBS_DELTA_MIN" => Array ( "T1"=>"3600", "NIGHT"=>"3600", "COMB"=>"3600", ), )
The speeds are collected that way in MYSQL:
Array ( "TB" => Array ( "COMB"=>"22739678964", "T1"=>"3339908691", "COMBNIGHT"=>"33319656215", "NIGHT"=>"5217145438", "COMBLNIGHT"=>"6016054440", "LNIGHT"=>"1541492392", "COMBT1"=>"0", ), "LB" => Array ( "COMB"=>"416211349", "T1"=>"201458741", "COMBNIGHT"=>"395545975", "NIGHT"=>"228162616", "COMBLNIGHT"=>"2270334036", "LNIGHT"=>"3614076", "COMBT1"=>"88073956", ), "TT" => Array ( "COMB"=>"7290365", "T1"=>"10701292", "COMBNIGHT"=>"6348749", "NIGHT"=>"8714890", "COMBLNIGHT"=>"4005954", "LNIGHT"=>"7909108", "COMBT1"=>"0", ), "LD" => Array ( "COMB"=>"33763", "T1"=>"33763", "COMBNIGHT"=>"25194", "NIGHT"=>"25194", "COMBLNIGHT"=>"27538", "LNIGHT"=>"27538", "COMBT1"=>"8880", ), "TS" => Array ( "COMB"=>"1072420867", "T1"=>"1072420867", "COMBNIGHT"=>"1072454577", "NIGHT"=>"1072454577", "COMBLNIGHT"=>"1072393354", "LNIGHT"=>"1072393354", "COMBT1"=>"1056198868", ), )
Which i can see from user-level managment appz:
hub3:~# abo "inder Ark" all
192.168.190.122; Binder Arkadiusz * sqix * [email protected] *
xxxxxx87,3xxxx82 * hub3.xxxxx.pl * 00:50:xxxx:51:65 * CI50/24 [CI50B-I] * SPD87 * FIXED
(as you can see currently i have EIR‡ Kbits), yesterday i had over 130 Kbits, just because i downloaded too-much as T1 policy (201.46Mb). But tommorrow i will have it back !
PRECIOSION-INFORMATIONS:
* T_WHEN_CONNECTED= 2002-06-30
* T_WHO_CONNECTED= Szarmach
* A_RECORD_CREATE_DATE= 2002-06-14-10-56-32
* A_RECORD_CREATE_AUTHOR= bzyk
* N_AVG_TRAFFIC = {Total_BYTES(COMB) 22.74_Gbytes, during 2.81_Months AVG_T=3.12_kbps Last Bytes(COMB) 416.21_Mbytes, during 9.38_Hours L_AVG_T.33_kbps
Updated 2003-12-26 17:3.50Total_BYTES(T1) 3.34_Gbytes, during 4.13_Months AVG_T12.10_bps Last Bytes(T1) 201.46_Mbytes, during 9.38_Hours L_AVG_T=5.97_kbps
Updated 2003-12-26 17:3.50Total_BYTES(COMBNIGHT) 33.32_Gbytes, during 2.45_Months AVG_T=5.25_kbps Last Bytes(COMBNIGHT) 395.55_Mbytes, during 7.00_Hours L_AVG_T.70_kbps
Updated 2003-12-27 0:2.51Total_BYTES(NIGHT) 5.22_Gbytes, during 3.36_Months AVG_TY8.65_bps Last Bytes(NIGHT) 228.16_Mbytes, during 7.00_Hours L_AVG_T=9.06_kbps
Updated 2003-12-27 0:2.51Total_BYTES(COMBLNIGHT) 6.02_Gbytes, during 1.55_Months AVG_T=1.50_kbps Last Bytes(COMBLNIGHT) 2.27_Gbytes, during 7.65_Hours L_AVG_T‚.44_kbps
Updated 2003-12-26 7:41.32Total_BYTES(LNIGHT) 1.54_Gbytes, during 3.05_Months AVG_T4.90_bps Last Bytes(LNIGHT) 3.61_Mbytes, during 7.65_Hours L_AVG_T1.24_bps
Updated 2003-12-26 7:41.32Total_BYTES(COMBT1) 0.00_bytes, during 1.00_secs AVG_T=0.00_bps Last Bytes(COMBT1) 88.07_Mbytes, during 2.47_Hours L_AVG_T=9.92_kbps
Updated 2003-6-21 17:2.28
} = TOTAL_AVERAGE 1.68_k_bps
hub3:~#
If anyone is interested in such stuff
visit http://nsm.pl/~arek/superedit -for other shots and contact me.
I can't put it free, just because i've been writing that for 2 years, but i can share with other such projects/communities.
A.Binder
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Thanks David for your post.
[]'s
Anderson O. Muniz
----- Original Message -----
From: "David DeLauro" <[email protected]>
To: "Anderson O Muniz" <[email protected]>
Cc: <[email protected]>
Sent: Monday, December 29, 2003 1:11 PM
Subject: Re: [LARTC] Dynamic Ratelimiting
Quoted 5 lines
On Fri, 26 Dec 2003 At 13:18 -0200, [email protected] Articulated:
Quoted 3 lines
Damjan,
I am working in a script to do something like that, can you post your
script
Quoted 11 lines
Quoted 1 line
for us?
http://mrtg.saintjoe.edu/mrtg/ratelimit/pacemaker/
--
David DeLauro
Computer Systems Analyst
Saint Joseph's College
Rensselaer, IN 47978I do this really moronic thing that the government doesn't want me to do.
It is called thinking. - George Carlin
Quoted 2 lines
Do not handicap your children by making their lives easy. - Robert
Heinlein
Quoted 2 lines
To many, total abstinence is easier than perfect moderation. - St.
Augustine
Quoted 3 lines
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Tue, 23 Dec 2003 At 21:28 +0100, [email protected] Articulated:
Quoted 6 lines
Very interesting, I'll look more in depth to your scripts ... do you think it would be easy to change the decission of who to ratelimit, from the number of connections to the bandwidth they are using?
Something like, if this user is using 512kb for 5 minutes ratelimit him?
Right now the scripts are just using a standard sniffer (tcpdump or tethereal) to gather information about the connections. For sure adding the ability watch bandwidth as well would be something I'm looking into adding... I haven't figured an efficient way to do it yet without parsing datagrams myself for HLEN and TOTAL LENGTH.
Quoted 2 lines
Signature · 11 lines
David DeLauro Do not handicap your children by making their lives easy. - Robert Heinlein If the soul could know God without the world, the world would never have been created. - Meister Eckhart Hata ukinichukia la kweli nitakwambia - Kanga Proverb _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Fri, 26 Dec 2003 At 13:18 -0200, [email protected] Articulated:
Quoted 4 lines
Damjan,
I am working in a script to do something like that, can you post your script for us?
http://mrtg.saintjoe.edu/mrtg/ratelimit/pacemaker/
Signature · 13 lines
David DeLauro Computer Systems Analyst Saint Joseph's College Rensselaer, IN 47978 I do this really moronic thing that the government doesn't want me to do. It is called thinking. - George Carlin Do not handicap your children by making their lives easy. - Robert Heinlein To many, total abstinence is easier than perfect moderation. - St. Augustine _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/