Skip to contentSkip to conversation

New HTB-derived qdisc for accounting?

LARTC Carl-Daniel Hailfinger 3 participants 7 messages Jun 4, 2005 – Jun 6, 2005

lore
Share & tools
lore
#0 lore raw

Hi,

at my local university network, I have to make sure no student uploads more than x GB/day. So far, I give them unlimited bandwidth until they have more than y GB (y<x) upload. After that, I use the u32 filter to associate the affected IP address with a HTB class limited to the rate (remainingtraffic/remainingtime).

Since the accounting is done with ipt_ACCOUNT right now and the netfilter framework obviously can't know whether a packet was dropped because the queue was full, dropped packets are counted as traffic, but that is somewhat unfair.

Can I do the accounting with the traffic control framework in the kernel instead of using ipt_ACCOUNT? Giving every IP its own htb class and setting that to an incredibly high limit to do the accounting seems overkill (there is no point in forcing HTB to do all the calculations for shaping if I only want to count) especially because I'd have to do that for a sparsely populated /16 network (about 1500-2800 hosts).

One (not that sophisticted) idea is to leave the accounting "as is"
and add a "dropped bytes" count to htb. This could be subtracted
from the numbers ipt_ACCOUNT gives me. That would modify the output
of "tc class ls" from
Sent 86136037 bytes 103963 pkt (dropped 283998, overlimits 0 requeues 0)
to
Sent 86136037 bytes 103963 pkt (dropped 385953282 bytes 283998 pkt, overlimits 0 requeues 0)

Proposal:
----------------------------------
Another idea would be to create a qdisc HTBQ (HTB with quota) derived from HTB with the following characteristics:

htb_rate=min(htbq_rate, (alreadysent=>htbq_squota)?((htbq_quota-alreadysent)/remtime):htbq_rate) htb_ceil=htbq_ceil //this is just passed on
htb_burst=htbq_burst
htb_cburst=htbq_cburst
htb_prio=htbq_prio
htb_quantum=htbq_quantum //should be set automatically

htbq_interval seconds //time after which quota is reset htbq_starttime seconds //time when the first interval starts htbq_quota bytes //maximum allowed bytes htbq_squota bytes //unshaped quota

htbq_rate is optional, unlimited if not set
htbq_ceil is optional
htbq_burst is optional
htbq_cburst is optional
htbq_prio is mandantory
htbq_quantum is optional

htbq_interval is mandantory
htbq_starttime is optional, defaults to unixtime 0 htbq_quota is mandantory
htbq_squota is optional, defaults to 0 (shape from beginning)

Basic description:
Case 1: already sent bytes are less than htbq_squota
if htbq_{rate,ceil,burst,cburst,quantum} is set then
do normal htb shaping with copied parameters
else
pass on packets directly to network device
Case 2: already sent bytes are greater or equal than htbq_squota
if htbq_{rate,ceil,burst,cburst,quantum} is set then
do htb shaping with copied parameters except htb_rate=min(htbq_rate, (htbq_quota-alreadysent)/remtime) htb_ceil=min(htbq_ceil, (htbq_quota-alreadysent)/remtime)
else
do htb shaping with following parameters htb_rate=(htbq_quota-alreadysent)/remtime htb_ceil=(htbq_quota-alreadysent)/remtime

This would surely be helpful for some admins who have to limit users to a certain quota without constantly shaping their network traffic or pulling the plug once the quota is full.

Thoughts?

Regards,
Carl-Daniel

Signature · 5 lines
http://www.hailfinger.org/
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
#1 lore raw
Quoted 1 line · On Sat, 04 Jun 2005 14:45:30 +0200 Carl-Daniel Hailfinger…

Hi,

Hi.

Quoted 4 lines

Since the accounting is done with ipt_ACCOUNT right now and the netfilter framework obviously can't know whether a packet was dropped because the queue was full, dropped packets are counted as traffic, but that is somewhat unfair.

I did some tests about a year ago and arrived to the conclusion that this difference can be ignored for practical purposes (~2%).

Yours sincerely,
Peter
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#2 lore raw
In reply to Carl-Daniel Hailfinger

Hi,

Quoted 9 lines · Peter Surda

On Sat, 04 Jun 2005 14:45:30 +0200 Carl-Daniel Hailfinger

Quoted 4 lines

Since the accounting is done with ipt_ACCOUNT right now and the netfilter framework obviously can't know whether a packet was dropped because the queue was full, dropped packets are counted as traffic, but that is somewhat unfair.

I did some tests about a year ago and arrived to the conclusion that this difference can be ignored for practical purposes (~2%).

I know that it can be ignored most of the time, but certain types of communication using UDP packets don't adjust their rates like a TCP connection would. One pathological case was >70% dropped packets and the netfilter counters were complete garbage.

Regards,
Carl-Daniel

Signature · 5 lines
http://www.hailfinger.org/
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
#3 lore raw
In reply to Carl-Daniel Hailfinger
Quoted 6 lines · Carl-Daniel Hailfinger

Proposal:
----------------------------------
Another idea would be to create a qdisc HTBQ (HTB with quota) derived from HTB with the following characteristics:

htb_rate=min(htbq_rate, (alreadysent=>htbq_squota)?((htbq_quota-alreadysent)/remtime):htbq_rate)

Why do you want to decrease speed as the quota is approached? Wouldn't a simple per-class quota or a quota-ematch work as well?

Regards
Patrick
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#4 lore raw
In reply to Carl-Daniel Hailfinger
Quoted 1 line · On Sat, 04 Jun 2005 15:21:36 +0200 Carl-Daniel Hailfinger…

Hi,

Hi,

Quoted 4 lines

I know that it can be ignored most of the time, but certain types of communication using UDP packets don't adjust their rates like a TCP connection would. One pathological case was >70% dropped packets and the netfilter counters were complete garbage.

In that case, on the same website as wrr ( http://wipl-wrr.sf.net/ ), there is another program, wipl. It is supposed to do IP-accounting based on tc (I don't know if it depends on wrr, I never used it, it just says it was designed for the purpose you require).

Quoted 2 lines

Regards,
Carl-Daniel

Yours sincerely,
Peter
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

#5 lore raw
In reply to Carl-Daniel Hailfinger
Quoted 11 lines · Patrick McHardy
Quoted 6 lines · Carl-Daniel Hailfinger

Proposal:
----------------------------------
Another idea would be to create a qdisc HTBQ (HTB with quota) derived from HTB with the following characteristics:

htb_rate=min(htbq_rate, (alreadysent=>htbq_squota)?((htbq_quota-alreadysent)/remtime):htbq_rate)

Why do you want to decrease speed as the quota is approached?

We have two phases (simplified):
1. Already sent traffic is less than htbq_squota

     -> Do not limit anything.

2. Already sent traffic is more than htbq_squota -> Limit the rate to something which allows the quota to be filled completely in the remainig time.

Most people stay below htbq_squota and do not notice anything at all, i.e. they get full wire speed. Power users will cause more traffic than htb_squota and be limited so they can't get over htbq_quota. Since it is impossible to send faster than htb_rate, htb_rate will stay constant if it is fully utilized. If htb_rate is not fully utilized, the speed will actually *increase* over time.

Quoted 2 lines

Wouldn't a simple per-class quota or a quota-ematch work as well?

That would be easier, but I can't see how to realize the process above with a quota-ematch. Especially the dynamic rate adaption needs something more than just quota. Probably one could combine quota-ematch with some userspace hackery to achieve what I want, but it would require to statically set up 65536 classes for a /16 network. Hm. The only difference to my solution would be that we need a quota ematch instead of a htb_quota qdisc and rate control would be done in userspace.

So where can I find code doing a quota ematch?

Regards,
Carl-Daniel

Signature · 5 lines
http://www.hailfinger.org/
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
#6 lore raw
In reply to Carl-Daniel Hailfinger
Quoted 12 lines · Carl-Daniel Hailfinger
Quoted 1 line · Patrick McHardy

Why do you want to decrease speed as the quota is approached?

We have two phases (simplified):
1. Already sent traffic is less than htbq_squota

     -> Do not limit anything.

2. Already sent traffic is more than htbq_squota -> Limit the rate to something which allows the quota to be filled completely in the remainig time.

Quoted 7 lines

Most people stay below htbq_squota and do not notice anything at all, i.e. they get full wire speed. Power users will cause more traffic than htb_squota and be limited so they can't get over htbq_quota. Since it is impossible to send faster than htb_rate, htb_rate will stay constant if it is fully utilized. If htb_rate is not fully utilized, the speed will actually *increase* over time.

Ok, so why do you want to prevent people from sending from htbq_squota to htbq_quota at full speed? Isn't the important point that noone sends more than their quota? Dynamic rate adaption is not easy, HTB needs pre-calculated rate tables, with HFSC it is even more complicated and can cause short periods of unfairness.

Quoted 13 lines
Quoted 2 lines

Wouldn't a simple per-class quota or a quota-ematch work as well?

That would be easier, but I can't see how to realize the process above with a quota-ematch. Especially the dynamic rate adaption needs something more than just quota. Probably one could combine quota-ematch with some userspace hackery to achieve what I want, but it would require to statically set up 65536 classes for a /16 network. Hm. The only difference to my solution would be that we need a quota ematch instead of a htb_quota qdisc and rate control would be done in userspace.

Quoted 1 line

So where can I find code doing a quota ematch?

Nowhere I guess, but shouldn't take more than a couple of minutes to write one. It would have the same problems as with iptables however, it doesn't get feedback about dropped packets. A per-class quota inside the qdisc is probably the best way.

Regards
Patrick
_______________________________________________
LARTC mailing list
[email protected]
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc