Skip to contentSkip to conversation

Dynamic Ratelimiting

LARTC David DeLauro 4 participants 7 messages Dec 23, 2003 – Dec 29, 2003

lore
Share & tools
lore
#0 lore raw

I've been using linux routing (htb qdisc) for almost a year now to try and manage the network here in a college environment. One of the major problems that I faced when I started this "little" project was P2P upload/downloads. At times the network would slow down so much one couldn't even load a webpage. I've tried the ratelimiting of certain ports, prioritizing certain blocks of IP, but all of it seems to be "less than ideal." We had continued to have problems with legitimate traffic being limited, our VoIP network was degraded (even after prioritizing), and our mirroring of slackware.com and cpan.org was less than glorious. It was workable but it was no way a good scene. After analyzing traffic, I thought it would have been inefficient to try and look into the data portion of the datagram but what I did notice about the traffic we had here was that the P2P machines had an unusually high number of connections. For out network, the number of connections was something that could
easily be monitored. So, I've created a few scripts that used
iptables, tc, and a sniffer that dynamically ratelimits machines(IPs). I've been using this script for awhile and it has done wonders for our network. A side effect of the scripts has been a ratelimiting of new Windows(tm) worm scans, port scans, and anything else that makes an unusually high number of connections. The VoIP traffic finally is usable (ideal?), and our mirrors work great. The project (I've called in 'pacemaker') is pretty configurable in that you can ignore certain hosts, networks, or ports if you know you would never want to ratelimit those resources based on number of connections. Seeing that it work so well here, I thought I'd offer it to the open source community and see if they could give me any pointers on making pacemaker better.

You can find the network statistics pages here:
http://mrtg.saintjoe.edu/

and pacemaker specifically here:
http://mrtg.saintjoe.edu/mrtg/ratelimit/pacemaker/

peace

Signature · 14 lines
David DeLauro
Computer Systems Analyst
Saint Joseph's College
Rensselaer, IN 47978


Do not handicap your children by making their lives easy. - Robert Heinlein

Hata ukinichukia la kweli nitakwambia - Kanga Proverb

I have often regretted my speech, never my silence. - Xenocrates
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#1 lore raw
Quoted 7 lines

It was workable but it was no way a good scene. After analyzing traffic, I thought it would have been inefficient to try and look into the data portion of the datagram but what I did notice about the traffic we had here was that the P2P machines had an unusually high number of connections. For out network, the number of connections was something that could
easily be monitored. So, I've created a few scripts that used
iptables, tc, and a sniffer that dynamically ratelimits machines(IPs).

Very interesting, I'll look more in depth to your scripts ... do you think it would be easy to change the decission of who to ratelimit, from the number of connections to the bandwidth they are using?

Something like, if this user is using 512kb for 5 minutes ratelimit him?

Signature · 5 lines
Damjan Georgievski
jabberID: [email protected]
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#2 lore raw
In reply to David DeLauro

Damjan,

I am working in a script to do something like that, can you post your script for us?

Thanks in Advance,
Anderson

----- Original Message -----
From: "Damjan" <[email protected]>
To: <[email protected]>
Cc: "David DeLauro" <[email protected]>
Sent: Tuesday, December 23, 2003 6:28 PM
Subject: Re: [LARTC] Dynamic Ratelimiting

Quoted 1 line
Quoted 1 line

It was workable but it was no way a good scene. After analyzing

traffic,

Quoted 3 lines
Quoted 3 lines

I thought it would have been inefficient to try and look into the data portion of the datagram but what I did notice about the traffic we had here was that the P2P machines had an unusually high number of

connections.

Quoted 8 lines
Quoted 3 lines

For out network, the number of connections was something that could
easily be monitored. So, I've created a few scripts that used
iptables, tc, and a sniffer that dynamically ratelimits machines(IPs).

Very interesting, I'll look more in depth to your scripts ... do you think it would be easy to change the decission of who to ratelimit, from the number of connections to the bandwidth they are using?

Quoted 8 lines

Something like, if this user is using 512kb for 5 minutes ratelimit him?

--
Damjan Georgievski
jabberID: [email protected]
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#3 lore raw
In reply to David DeLauro
Quoted 3 lines

I am working in a script to do something like that, can you post your script
for us?

I have my own hard-stuff for solve that problem.

I measure all of my clients for long time (tc+parser+sql)
After some period of time, 90% of my LAN clients get large autmated bandwidth speeds , thus the rest about 10% has much worst speeds. My system increase/decrese client speeds based on per client policy (kept in DB):

It is Per User Policy :
Array ( "LBS_TBS_MAX_ANDOR" => Array ( "T1"=>"OR", "NIGHT"=>"OR", "COMB"=>"OR", ), "LBS_TBS_MIN_ANDOR" => Array ( "T1"=>"AND", "NIGHT"=>"AND", "COMB"=>"AND", ), "LBS_MIN_OK" => Array ( "T1"=>"2000", "NIGHT"=>"1500", "COMB"=>"4000", ), "LBS_MAX_OK" => Array ( "T1"=>"4000", "NIGHT"=>"5000", "COMB"=>"9000", ), "TBS_MIN_OK" => Array ( "T1"=>"1800", "NIGHT"=>"2000", "COMB"=>"4000", ), "TBS_MAX_OK" => Array ( "T1"=>"3000", "NIGHT"=>"2500", "COMB"=>"8000", ), "INC_STEP" => Array ( "T1"=>"10", "NIGHT"=>"0", "COMB"=>"0", ), "DEC_STEP" => Array ( "T1"=>"30", "NIGHT"=>"30", "COMB"=>"3", ), "MAX_SPD" => Array ( "T1"=>"150", "NIGHT"=>"150", "COMB"=>"150", ), "MIN_SPD" => Array ( "T1"=>"50", "NIGHT"=>"50", "COMB"=>"50", ), "LBS_DELTA_MIN" => Array ( "T1"=>"3600", "NIGHT"=>"3600", "COMB"=>"3600", ), )

The speeds are collected that way in MYSQL:
Array ( "TB" => Array ( "COMB"=>"22739678964", "T1"=>"3339908691", "COMBNIGHT"=>"33319656215", "NIGHT"=>"5217145438", "COMBLNIGHT"=>"6016054440", "LNIGHT"=>"1541492392", "COMBT1"=>"0", ), "LB" => Array ( "COMB"=>"416211349", "T1"=>"201458741", "COMBNIGHT"=>"395545975", "NIGHT"=>"228162616", "COMBLNIGHT"=>"2270334036", "LNIGHT"=>"3614076", "COMBT1"=>"88073956", ), "TT" => Array ( "COMB"=>"7290365", "T1"=>"10701292", "COMBNIGHT"=>"6348749", "NIGHT"=>"8714890", "COMBLNIGHT"=>"4005954", "LNIGHT"=>"7909108", "COMBT1"=>"0", ), "LD" => Array ( "COMB"=>"33763", "T1"=>"33763", "COMBNIGHT"=>"25194", "NIGHT"=>"25194", "COMBLNIGHT"=>"27538", "LNIGHT"=>"27538", "COMBT1"=>"8880", ), "TS" => Array ( "COMB"=>"1072420867", "T1"=>"1072420867", "COMBNIGHT"=>"1072454577", "NIGHT"=>"1072454577", "COMBLNIGHT"=>"1072393354", "LNIGHT"=>"1072393354", "COMBT1"=>"1056198868", ), )

Which i can see from user-level managment appz:

hub3:~# abo "inder Ark" all

192.168.190.122; Binder Arkadiusz * sqix * [email protected] *
xxxxxx87,3xxxx82 * hub3.xxxxx.pl * 00:50:xxxx:51:65 * CI50/24 [CI50B-I] * SPD87 * FIXED
(as you can see currently i have EIR‡ Kbits), yesterday i had over 130 Kbits, just because i downloaded too-much as T1 policy (201.46Mb). But tommorrow i will have it back !

PRECIOSION-INFORMATIONS:
* T_WHEN_CONNECTED= 2002-06-30
* T_WHO_CONNECTED= Szarmach
* A_RECORD_CREATE_DATE= 2002-06-14-10-56-32
* A_RECORD_CREATE_AUTHOR= bzyk
* N_AVG_TRAFFIC = {
Total_BYTES(COMB) 22.74_Gbytes, during 2.81_Months AVG_T=3.12_kbps Last Bytes(COMB) 416.21_Mbytes, during 9.38_Hours L_AVG_T.33_kbps
Updated 2003-12-26 17:3.50
Total_BYTES(T1) 3.34_Gbytes, during 4.13_Months AVG_T12.10_bps Last Bytes(T1) 201.46_Mbytes, during 9.38_Hours L_AVG_T=5.97_kbps
Updated 2003-12-26 17:3.50
Total_BYTES(COMBNIGHT) 33.32_Gbytes, during 2.45_Months AVG_T=5.25_kbps Last Bytes(COMBNIGHT) 395.55_Mbytes, during 7.00_Hours L_AVG_T.70_kbps
Updated 2003-12-27 0:2.51
Total_BYTES(NIGHT) 5.22_Gbytes, during 3.36_Months AVG_TY8.65_bps Last Bytes(NIGHT) 228.16_Mbytes, during 7.00_Hours L_AVG_T=9.06_kbps
Updated 2003-12-27 0:2.51
Total_BYTES(COMBLNIGHT) 6.02_Gbytes, during 1.55_Months AVG_T=1.50_kbps Last Bytes(COMBLNIGHT) 2.27_Gbytes, during 7.65_Hours L_AVG_T‚.44_kbps
Updated 2003-12-26 7:41.32
Total_BYTES(LNIGHT) 1.54_Gbytes, during 3.05_Months AVG_T4.90_bps Last Bytes(LNIGHT) 3.61_Mbytes, during 7.65_Hours L_AVG_T1.24_bps
Updated 2003-12-26 7:41.32
Total_BYTES(COMBT1) 0.00_bytes, during 1.00_secs AVG_T=0.00_bps Last Bytes(COMBT1) 88.07_Mbytes, during 2.47_Hours L_AVG_T=9.92_kbps
Updated 2003-6-21 17:2.28

                  } = TOTAL_AVERAGE 1.68_k_bps

hub3:~#

If anyone is interested in such stuff
visit http://nsm.pl/~arek/superedit -for other shots and contact me.

I can't put it free, just because i've been writing that for 2 years, but i can share with other such projects/communities.

A.Binder

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#4 lore raw
In reply to David DeLauro

Thanks David for your post.

[]'s
Anderson O. Muniz

----- Original Message -----
From: "David DeLauro" <[email protected]>
To: "Anderson O Muniz" <[email protected]>
Cc: <[email protected]>
Sent: Monday, December 29, 2003 1:11 PM
Subject: Re: [LARTC] Dynamic Ratelimiting

Quoted 5 lines

On Fri, 26 Dec 2003 At 13:18 -0200, [email protected] Articulated:

Quoted 3 lines

Damjan,

I am working in a script to do something like that, can you post your

script

Quoted 11 lines
Quoted 1 line

for us?

http://mrtg.saintjoe.edu/mrtg/ratelimit/pacemaker/

--
David DeLauro
Computer Systems Analyst
Saint Joseph's College
Rensselaer, IN 47978

I do this really moronic thing that the government doesn't want me to do.

It is called thinking. - George Carlin

Quoted 2 lines

Do not handicap your children by making their lives easy. - Robert

Heinlein

Quoted 2 lines

To many, total abstinence is easier than perfect moderation. - St.

Augustine

Quoted 3 lines

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#5 lore raw
In reply to David DeLauro

On Tue, 23 Dec 2003 At 21:28 +0100, [email protected] Articulated:

Quoted 6 lines

Very interesting, I'll look more in depth to your scripts ... do you think it would be easy to change the decission of who to ratelimit, from the number of connections to the bandwidth they are using?

Something like, if this user is using 512kb for 5 minutes ratelimit him?

Right now the scripts are just using a standard sniffer (tcpdump or tethereal) to gather information about the connections. For sure adding the ability watch bandwidth as well would be something I'm looking into adding... I haven't figured an efficient way to do it yet without parsing datagrams myself for HLEN and TOTAL LENGTH.

Quoted 2 lines
Signature · 11 lines
David DeLauro

Do not handicap your children by making their lives easy. - Robert Heinlein

If the soul could know God without the world, the world would never have been created. - Meister Eckhart

Hata ukinichukia la kweli nitakwambia - Kanga Proverb

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#6 lore raw
In reply to David DeLauro

On Fri, 26 Dec 2003 At 13:18 -0200, [email protected] Articulated:

Quoted 4 lines

Damjan,

I am working in a script to do something like that, can you post your script for us?

http://mrtg.saintjoe.edu/mrtg/ratelimit/pacemaker/

Signature · 13 lines
David DeLauro
Computer Systems Analyst
Saint Joseph's College
Rensselaer, IN 47978

I do this really moronic thing that the government doesn't want me to do.  It is called thinking. - George Carlin

Do not handicap your children by making their lives easy. - Robert Heinlein

To many, total abstinence is easier than perfect moderation. - St. Augustine
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/