Skip to contentSkip to conversation

HTB doesn't respect rate values

LARTC Sergiusz Brzeziński 6 participants 14 messages Jul 5, 2003 – Jul 8, 2003

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Hi,

machine:            AMD K6 200 MHz
Linux distribution: Mandrake 8.1
kernel:             compiled 2.4.21

applied this:
#define PSCHED_CLOCK_SOURCE PSCHED_CPU in file linux/include/net/pkt_sched.h bevore compiled the kernel (described on http://www.docum.org/stef.coene/qos/faq/cache/40.html)
bandwitch on eth0: 128kbit

The most simple configuration - 122kbit guaranted for WWW (sport 80) and 6kbit for the rest.

tc qdisc del root dev eth0

tc qdisc add dev eth0 root handle 1:0 htb default 3

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit tc class add dev eth0 parent 1:1 classid 1:2 htb rate 122kbit ceil 128kbit quantum 30500
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 6kbit ceil
128kbit quantum 1500

tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80 0xffff flowid 1:2

I make a test:
I send an email - it goes to default class 1:3. Then (during email is sent) I get e big file through www. What happen? WWW rate is 30-70kbit. So it doesn't keep his guaranted rate 122kbit. It lends his rate for SMTP. When SMTP stops sending his packets, WWW gets 100%.

In real, I've got the more complicated configuration (64 kbit for www, 40kbit for pop3, 6kbit for smtp 12kbit for others), but the problem
remains: www doesn't get hit minmal rate. Extremally it gets < 30kbit!

It works only "ceil". If I set ceil for a class (for example 1:3) - it is respected. 1:3 doesn't get more than set in ceil.

So, HTB doesn't work for me at all!

WHAT'S UP?

Could it be, that my rates are small? (6kbit). Does HTB work good for bandwitch <8kbit?

Did I something wrong?

Some sugestions?

thanks in advance

Sergiusz

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#1 lore raw

1) 6kbit is really too small it should be at least 10 ..
2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.

devik

Quoted 52 lines · Sergiusz Brzeziñski

Hi,

machine:            AMD K6 200 MHz
Linux distribution: Mandrake 8.1
kernel:             compiled 2.4.21

applied this:
#define PSCHED_CLOCK_SOURCE PSCHED_CPU in file linux/include/net/pkt_sched.h bevore compiled the kernel (described on http://www.docum.org/stef.coene/qos/faq/cache/40.html)
bandwitch on eth0: 128kbit

The most simple configuration - 122kbit guaranted for WWW (sport 80) and 6kbit for the rest.

tc qdisc del root dev eth0

tc qdisc add dev eth0 root handle 1:0 htb default 3

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit tc class add dev eth0 parent 1:1 classid 1:2 htb rate 122kbit ceil 128kbit quantum 30500
tc class add dev eth0 parent 1:1 classid 1:3 htb rate 6kbit ceil
128kbit quantum 1500

tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80 0xffff flowid 1:2

I make a test:
I send an email - it goes to default class 1:3. Then (during email is sent) I get e big file through www. What happen? WWW rate is 30-70kbit. So it doesn't keep his guaranted rate 122kbit. It lends his rate for SMTP. When SMTP stops sending his packets, WWW gets 100%.

In real, I've got the more complicated configuration (64 kbit for www, 40kbit for pop3, 6kbit for smtp 12kbit for others), but the problem
remains: www doesn't get hit minmal rate. Extremally it gets < 30kbit!

It works only "ceil". If I set ceil for a class (for example 1:3) - it is respected. 1:3 doesn't get more than set in ceil.

So, HTB doesn't work for me at all!

WHAT'S UP?

Could it be, that my rates are small? (6kbit). Does HTB work good for bandwitch <8kbit?

Did I something wrong?

Some sugestions?

Quoted 10 lines

thanks in advance

Sergiusz

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#2 lore raw
In reply to Sergiusz Brzeziński

Thak you for your hints!

Quoted 1 line

1) 6kbit is really too small it should be at least 10 ..

I tried with 12, 20 and even with 30kbit for 1:3

I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.

Quoted 5 lines

2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.

I made a test with settings:
---------------------------------

tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500

tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80 0xffff flowid 1:2

Bevore test: (reseted htb)
--------------------------------

# tc -s class show dev eth0

class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131

After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151

Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.

1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?

Sergiusz

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#3 lore raw
In reply to Sergiusz Brzeziński

Interestingly from what I see HTB didn't come into play. All drop and overlimits counters are zero. It seems that www server haven't managed to send more.
Please try to add pfifo with limit 10 under both classes. Because you are sending from the same computer, your TCP stack uses send queue management which counts packets in qdisc and backs off. It MIGHT cause problems ...

-------------------------------
Martin Devera aka devik
Linux kernel QoS/HTB maintainer
http://luxik.cdi.cz/~devik/

Quoted 86 lines · Sergiusz Brzeziñski

Thak you for your hints!

Quoted 1 line

1) 6kbit is really too small it should be at least 10 ..

I tried with 12, 20 and even with 30kbit for 1:3

I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.

Quoted 5 lines

2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.

I made a test with settings:
---------------------------------

tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500

tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80 0xffff flowid 1:2

Bevore test: (reseted htb)
--------------------------------

# tc -s class show dev eth0

class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131

After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151

Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.

1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?

Quoted 10 lines

Sergiusz

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#4 lore raw
In reply to Sergiusz Brzeziński

That's it!

I did pfifo with limit 10 and HTB started to work. I noticed drops and rate was OK. Sometimes (for 10-40 seconds) but seldom it worked bad (1:2 got less, than it should) but there wasn't drops during this time. I tried this also with 12kbit at it was similar.

If I good understand, there is something (TCP stack or what) whitch works BEVORE htb and this makes some connections slower or faster, so HTB has later nothing to do.

The question for me is: how can I set it (the mechanism bevore HTB) to give HTB full control over the bandwitch? I don't wont to use pfifo (not everywhere). I would like to use sfq for some classes.

Should I set specific parameter somewhere or set something in network configuration, or maybe recompile the kernel with a specific option?

Sergiusz

U¿ytkownik devik napisa³:

Quoted 8 lines

Interestingly from what I see HTB didn't come into play. All drop and overlimits counters are zero. It seems that www server haven't managed to send more.
Please try to add pfifo with limit 10 under both classes. Because you are sending from the same computer, your TCP stack uses send queue management which counts packets in qdisc and backs off. It MIGHT cause problems ...

Quoted 96 lines

-------------------------------
Martin Devera aka devik
Linux kernel QoS/HTB maintainer
http://luxik.cdi.cz/~devik/

Quoted 88 lines · Sergiusz Brzeziñski

Thak you for your hints!

Quoted 1 line

1) 6kbit is really too small it should be at least 10 ..

I tried with 12, 20 and even with 30kbit for 1:3

I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.

Quoted 5 lines

2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.

I made a test with settings:
---------------------------------

tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500

tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80 0xffff flowid 1:2

Bevore test: (reseted htb)
--------------------------------

# tc -s class show dev eth0

class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131

After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151

Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.

1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?

Sergiusz

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#5 lore raw
In reply to Sergiusz Brzeziński

At 01:05 a.m. 06/07/03 +0200, you wrote:

Hi, Sergiusz:

Quoted 5 lines

I make a test:
I send an email - it goes to default class 1:3. Then (during email is sent) I get e big file through www. What happen? WWW rate is 30-70kbit. So it doesn't keep his guaranted rate 122kbit. It lends his rate for SMTP. When SMTP stops sending his packets, WWW gets 100%.

If your HTB configuration is working well (HTB works really very well) you have to wait some time that TCP flows get stable. How long your test last? How heavy is your e-mail? Did you leave enough time to www to reclaim its rights? How strong is your www flow? Are you measuring average or instantaneous rates?

Best regards,

Leonardo Balliache

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#6 lore raw
In reply to Sergiusz Brzeziński
Quoted 5 lines

I did pfifo with limit 10 and HTB started to work. I noticed drops and rate was OK. Sometimes (for 10-40 seconds) but seldom it worked bad (1:2 got less, than it should) but there wasn't drops during this time. I tried this also with 12kbit at it was similar.

Quoted 7 lines

If I good understand, there is something (TCP stack or what) whitch works BEVORE htb and this makes some connections slower or faster, so HTB has later nothing to do.

The question for me is: how can I set it (the mechanism bevore HTB) to give HTB full control over the bandwitch? I don't wont to use pfifo (not everywhere). I would like to use sfq for some classes.

you can try to play with /proc/sys/net/ipv4/tcp_{w,r,}mem. If wmem is smaller than space in qdisc (15kB for 10 pfifo, approx 200kB for SFQ) then TCP will back off before it fill the qdisc...
Also with sfq it should work - it is like pfifo with limit 128 - you might need to increase wmem.
Note that the problem occur only when you have qdisc on the same machine as sending app.

devik

Quoted 113 lines

Should I set specific parameter somewhere or set something in network configuration, or maybe recompile the kernel with a specific option?

Sergiusz

U¿ytkownik devik napisa³:

Quoted 104 lines

Interestingly from what I see HTB didn't come into play. All drop and overlimits counters are zero. It seems that www server haven't managed to send more.
Please try to add pfifo with limit 10 under both classes. Because you are sending from the same computer, your TCP stack uses send queue management which counts packets in qdisc and backs off. It MIGHT cause problems ...

-------------------------------
Martin Devera aka devik
Linux kernel QoS/HTB maintainer
http://luxik.cdi.cz/~devik/

Quoted 88 lines · Sergiusz Brzeziñski

Thak you for your hints!

Quoted 1 line

1) 6kbit is really too small it should be at least 10 ..

I tried with 12, 20 and even with 30kbit for 1:3

I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.

Quoted 5 lines

2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.

I made a test with settings:
---------------------------------

tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500

tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80 0xffff flowid 1:2

Bevore test: (reseted htb)
--------------------------------

# tc -s class show dev eth0

class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131

After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151

Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.

1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?

Sergiusz

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#7 lore raw
In reply to Sergiusz Brzeziński
Quoted 9 lines · Leonardo Balliache

At 01:05 a.m. 06/07/03 +0200, you wrote:

Hi, Sergiusz:

Quoted 5 lines

I make a test:
I send an email - it goes to default class 1:3. Then (during email is sent) I get e big file through www. What happen? WWW rate is 30-70kbit. So it doesn't keep his guaranted rate 122kbit. It lends his rate for SMTP. When SMTP stops sending his packets, WWW gets 100%.

Quoted 5 lines

If your HTB configuration is working well (HTB works really very well) you have to wait some time that TCP flows get stable. How long your test last? How heavy is your e-mail? Did you leave enough time to www to reclaim its rights? How strong is your www flow? Are you measuring average or instantaneous rates?

Also, check out my tests on www.docum.org. Just to prove that it's working. You can even use my test scripts and adapt them to your needs.

Stef

Signature · 8 lines
[email protected]
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#8 lore raw
In reply to Sergiusz Brzeziński

U¿ytkownik devik napisa³:

Quoted 14 lines
Quoted 5 lines

I did pfifo with limit 10 and HTB started to work. I noticed drops and rate was OK. Sometimes (for 10-40 seconds) but seldom it worked bad (1:2 got less, than it should) but there wasn't drops during this time. I tried this also with 12kbit at it was similar.

Quoted 7 lines

If I good understand, there is something (TCP stack or what) whitch works BEVORE htb and this makes some connections slower or faster, so HTB has later nothing to do.

The question for me is: how can I set it (the mechanism bevore HTB) to give HTB full control over the bandwitch? I don't wont to use pfifo (not everywhere). I would like to use sfq for some classes.

Quoted 7 lines

you can try to play with /proc/sys/net/ipv4/tcp_{w,r,}mem. If wmem is smaller than space in qdisc (15kB for 10 pfifo, approx 200kB for SFQ) then TCP will back off before it fill the qdisc...
Also with sfq it should work - it is like pfifo with limit 128 - you might need to increase wmem.
Note that the problem occur only when you have qdisc on the same machine as sending app.

- I did this:

# echo "4096 2048000 5120000" > /proc/sys/net/ipv4/tcp_wmem
than I tried if in /proc/sys/net/ipv4/tcp_wmem there are really the new 
values (i set so high values because i wanted really be sure that the 
amount of memory will be OK :) - defaults where: 4096 16384 131072)

Well, it helped in 80%. Why only in 80? I repeated my test with SFQ and:

- it worked better than bevore, there where long time periods (15-20sec.) with right rate-values
- but class whitch should become 98kbit still got sometimes only 38kbit; it happen seldom and short but it was a fact!
- what was very strange: there were still no drops and no overlimits (!!!) in stats ("tc -s class show dev eth0"); in a test with "pfifo limit 10" I could see: when there were drops - the rate was OK, when there were no drops - the rate got lower than it should; with SFQ there were NO DROPS AT ALL so the question is: who (or what) really makes the whole work? It doesn't look like HTB-work.(have I right?)

Should I also do something with "/proc/sys/net/ipv4/tcp_mem"? Is the min value in tcp_wmem (4096) OK?
Do you have some more ideas?

I would make some experiments but I'm really not familiar with this theme. So the only thing I can do is to ask YOU or someone else from the group.

Sergiusz

Quoted 104 lines
Quoted 104 lines

U¿ytkownik devik napisa³:

Quoted 102 lines

Interestingly from what I see HTB didn't come into play. All drop and overlimits counters are zero. It seems that www server haven't managed to send more.
Please try to add pfifo with limit 10 under both classes. Because you are sending from the same computer, your TCP stack uses send queue management which counts packets in qdisc and backs off. It MIGHT cause problems ...

-------------------------------
Martin Devera aka devik
Linux kernel QoS/HTB maintainer
http://luxik.cdi.cz/~devik/

Quoted 85 lines · Sergiusz Brzeziñski

Thak you for your hints!

Quoted 1 line

1) 6kbit is really too small it should be at least 10 ..

I tried with 12, 20 and even with 30kbit for 1:3

I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.

Quoted 5 lines

2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.

I made a test with settings:
---------------------------------

tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500

tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80 0xffff flowid 1:2

Bevore test: (reseted htb)
--------------------------------

# tc -s class show dev eth0

class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131

After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151

Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.

Quoted 10 lines
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines

1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?

Sergiusz

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#9 lore raw
In reply to Sergiusz Brzeziński

Well. The right way for your case would be to limit single subqueue in SFQ. See line 24 of attached patch - and try patch itself.
devik

Quoted 46 lines · Sergiusz Brzeziński
Quoted 14 lines · Użytkownik devik
Quoted 5 lines

I did pfifo with limit 10 and HTB started to work. I noticed drops and rate was OK. Sometimes (for 10-40 seconds) but seldom it worked bad (1:2 got less, than it should) but there wasn't drops during this time. I tried this also with 12kbit at it was similar.

Quoted 7 lines

If I good understand, there is something (TCP stack or what) whitch works BEVORE htb and this makes some connections slower or faster, so HTB has later nothing to do.

The question for me is: how can I set it (the mechanism bevore HTB) to give HTB full control over the bandwitch? I don't wont to use pfifo (not everywhere). I would like to use sfq for some classes.

Quoted 7 lines

you can try to play with /proc/sys/net/ipv4/tcp_{w,r,}mem. If wmem is smaller than space in qdisc (15kB for 10 pfifo, approx 200kB for SFQ) then TCP will back off before it fill the qdisc...
Also with sfq it should work - it is like pfifo with limit 128 - you might need to increase wmem.
Note that the problem occur only when you have qdisc on the same machine as sending app.

- I did this:

# echo "4096 2048000 5120000" > /proc/sys/net/ipv4/tcp_wmem
than I tried if in /proc/sys/net/ipv4/tcp_wmem there are really the new
values (i set so high values because i wanted really be sure that the
amount of memory will be OK :) - defaults where: 4096 16384 131072)

Well, it helped in 80%. Why only in 80? I repeated my test with SFQ and:

- it worked better than bevore, there where long time periods (15-20sec.) with right rate-values
- but class whitch should become 98kbit still got sometimes only 38kbit; it happen seldom and short but it was a fact!
- what was very strange: there were still no drops and no overlimits (!!!) in stats ("tc -s class show dev eth0"); in a test with "pfifo limit 10" I could see: when there were drops - the rate was OK, when there were no drops - the rate got lower than it should; with SFQ there were NO DROPS AT ALL so the question is: who (or what) really makes the whole work? It doesn't look like HTB-work.(have I right?)

Should I also do something with "/proc/sys/net/ipv4/tcp_mem"? Is the min value in tcp_wmem (4096) OK?
Do you have some more ideas?

Quoted 124 lines

I would make some experiments but I'm really not familiar with this theme. So the only thing I can do is to ask YOU or someone else from the group.

Sergiusz

Quoted 114 lines
Quoted 112 lines
Quoted 107 lines · Użytkownik devik

Interestingly from what I see HTB didn't come into play. All drop and overlimits counters are zero. It seems that www server haven't managed to send more.
Please try to add pfifo with limit 10 under both classes. Because you are sending from the same computer, your TCP stack uses send queue management which counts packets in qdisc and backs off. It MIGHT cause problems ...

-------------------------------
Martin Devera aka devik
Linux kernel QoS/HTB maintainer
http://luxik.cdi.cz/~devik/

Quoted 90 lines · Sergiusz Brzeziński

Thak you for your hints!

Quoted 1 line

1) 6kbit is really too small it should be at least 10 ..

I tried with 12, 20 and even with 30kbit for 1:3

I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.

Quoted 5 lines

2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.

I made a test with settings:
---------------------------------

tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500

tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80 0xffff flowid 1:2

Bevore test: (reseted htb)
--------------------------------

# tc -s class show dev eth0

class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131

After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151

Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.

1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?

Sergiusz

net/sched/sch_sfq.c renamed from usr/src/linux-2.2.16/net/sched/sch_sfq.c +12 -3
diff -rubB /usr/src/linux-2.2.16/net/sched/sch_sfq.c gatek/net/sched/sch_sfq.c
@@ -111,6 +111,8 @@
 	int		perturbation;
 	sfq_index	tail;		/* Index of current slot in round */
 	sfq_index	max_depth;	/* Maximal depth */
+	short		flows;		/* Num of flows */
+	short		max_flows;	/* Max num of flows */
 
 	sfq_index	ht[SFQ_HASH_DIVISOR];	/* Hash table */
 	sfq_index	next[SFQ_DEPTH];	/* Active slots link */
@@ -265,6 +267,7 @@
 	__skb_queue_tail(&q->qs[x], skb);
 	sfq_inc(q, x);
 	if (q->qs[x].qlen == 1) {		/* The flow is new */
+	    	if (++q->flows > q->max_flows) q->max_flows = q->flows;
 		if (q->tail == SFQ_DEPTH) {	/* It is the first flow */
 			q->tail = x;
 			q->next[x] = x;
@@ -276,11 +279,13 @@
 		}
 	}
 	if (++sch->q.qlen < SFQ_DEPTH-1) {
+		// arbitrary hack to limit maximal flow's backlog
+		if (q->qs[x].qlen > 5) goto drop;
 		sch->stats.bytes += skb->len;
 		sch->stats.packets++;
 		return 1;
 	}
-
+drop:
 	sfq_drop(sch);
 	return 0;
 }
@@ -300,6 +305,7 @@
 	__skb_queue_head(&q->qs[x], skb);
 	sfq_inc(q, x);
 	if (q->qs[x].qlen == 1) {		/* The flow is new */
+	    	if (++q->flows > q->max_flows) q->max_flows = q->flows;
 		if (q->tail == SFQ_DEPTH) {	/* It is the first flow */
 			q->tail = x;
 			q->next[x] = x;
@@ -341,6 +347,7 @@
 
 	/* Is the slot empty? */
 	if (q->qs[a].qlen == 0) {
+	    	q->flows--;
 		a = q->next[a];
 		if (a == old_a) {
 			q->tail = SFQ_DEPTH;
@@ -417,6 +424,8 @@
 		q->dep[i+SFQ_DEPTH].prev = i+SFQ_DEPTH;
 	}
 	q->max_depth = 0;
+	q->max_flows = 0;
+	q->flows = 0;
 	q->tail = SFQ_DEPTH;
 	if (opt == NULL) {
 		q->quantum = psched_mtu(sch->dev);
@@ -450,8 +459,8 @@
 	opt.perturb_period = q->perturb_period/HZ;
 
 	opt.limit = SFQ_DEPTH;
-	opt.divisor = SFQ_HASH_DIVISOR;
-	opt.flows = SFQ_DEPTH;
+	opt.divisor = q->max_flows;
+	opt.flows = q->flows;
 
 	RTA_PUT(skb, TCA_OPTIONS, sizeof(opt), &opt);
 
#10 lore raw
In reply to Sergiusz Brzeziński

U¿ytkownik devik napisa³:

Quoted 4 lines

Well. The right way for your case would be to limit single subqueue in SFQ. See line 24 of attached patch - and try patch itself.
devik

hmm.. with pfifo_fast is thesame problem - no drops... and unstable. It doesn't look like a SFQ-specific problem. With fifo (without setting limit) there were only 2 drops. Only "fifo limit 10" generates drops (but it has also periods with no drops and with bad rates). Everything of course with tcp_wmem = "4096 2048000 5120000"

Sergiusz

Quoted 166 lines · Sergiusz Brzeziñski
Quoted 164 lines

U¿ytkownik devik napisa³:

Quoted 14 lines
Quoted 5 lines

I did pfifo with limit 10 and HTB started to work. I noticed drops and rate was OK. Sometimes (for 10-40 seconds) but seldom it worked bad (1:2 got less, than it should) but there wasn't drops during this time. I tried this also with 12kbit at it was similar.

Quoted 7 lines

If I good understand, there is something (TCP stack or what) whitch works BEVORE htb and this makes some connections slower or faster, so HTB has later nothing to do.

The question for me is: how can I set it (the mechanism bevore HTB) to give HTB full control over the bandwitch? I don't wont to use pfifo (not everywhere). I would like to use sfq for some classes.

Quoted 7 lines

you can try to play with /proc/sys/net/ipv4/tcp_{w,r,}mem. If wmem is smaller than space in qdisc (15kB for 10 pfifo, approx 200kB for SFQ) then TCP will back off before it fill the qdisc...
Also with sfq it should work - it is like pfifo with limit 128 - you might need to increase wmem.
Note that the problem occur only when you have qdisc on the same machine as sending app.

- I did this:

# echo "4096 2048000 5120000" > /proc/sys/net/ipv4/tcp_wmem
than I tried if in /proc/sys/net/ipv4/tcp_wmem there are really the new
values (i set so high values because i wanted really be sure that the
amount of memory will be OK :) - defaults where: 4096 16384 131072)

Well, it helped in 80%. Why only in 80? I repeated my test with SFQ and:

- it worked better than bevore, there where long time periods (15-20sec.) with right rate-values
- but class whitch should become 98kbit still got sometimes only 38kbit; it happen seldom and short but it was a fact!
- what was very strange: there were still no drops and no overlimits (!!!) in stats ("tc -s class show dev eth0"); in a test with "pfifo limit 10" I could see: when there were drops - the rate was OK, when there were no drops - the rate got lower than it should; with SFQ there were NO DROPS AT ALL so the question is: who (or what) really makes the whole work? It doesn't look like HTB-work.(have I right?)

Should I also do something with "/proc/sys/net/ipv4/tcp_mem"? Is the min value in tcp_wmem (4096) OK?
Do you have some more ideas?

I would make some experiments but I'm really not familiar with this theme. So the only thing I can do is to ask YOU or someone else from the group.

Sergiusz

Quoted 108 lines
Quoted 108 lines

U¿ytkownik devik napisa³:

Quoted 105 lines

Interestingly from what I see HTB didn't come into play. All drop and overlimits counters are zero. It seems that www server haven't managed to send more.
Please try to add pfifo with limit 10 under both classes. Because you are sending from the same computer, your TCP stack uses send queue management which counts packets in qdisc and backs off. It MIGHT cause problems ...

-------------------------------
Martin Devera aka devik
Linux kernel QoS/HTB maintainer
http://luxik.cdi.cz/~devik/

Quoted 87 lines · Sergiusz Brzeziñski

Thak you for your hints!

Quoted 1 line

1) 6kbit is really too small it should be at least 10 ..

I tried with 12, 20 and even with 30kbit for 1:3

I noticed, that it work for some seconds (or 1-2 minutes) but than the 1:3 class gets more then it should get :(.

Quoted 5 lines

2) it should workeven with 6k:
- look at stats (tc -s class show dev eth0) before and after the test - you are interested in drops. Also try it during the test to look whether queues are build up.

I made a test with settings:
---------------------------------

tc qdisc del root dev eth0
tc qdisc add dev eth0 root handle 1:0 htb default 3

tc class add dev eth0 parent 1:0 classid 1:1 htb rate 128kbit ceil 128kbit burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:2 htb rate 98kbit ceil 128kbit quantum 4900 burst 20kbit

tc class add dev eth0 parent 1:1 classid 1:3 htb rate 30kbit ceil 128kbit quantum 1500

tc filter add dev eth0 protocol ip parent 1:0 prio 1 u32 match ip sport 80 0xffff flowid 1:2

Bevore test: (reseted htb)
--------------------------------

# tc -s class show dev eth0

class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 244140 ctokens: 168131

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 318876 ctokens: 168131

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 0 bytes 0 pkts (dropped 0, overlimits 0)
lended: 0 borrowed: 0 giants: 0
tokens: 666503 ctokens: 168131

After test:
------------
class htb 1:1 root rate 128Kbit ceil 128Kbit burst 2559b cburst 1762b Sent 5843869 bytes 4715 pkts (dropped 0, overlimits 0) rate 15427bps 12pps
lended: 1461 borrowed: 0 giants: 0
tokens: -21142 ctokens: -97151

class htb 1:2 parent 1:1 prio 0 rate 98Kbit ceil 128Kbit burst 2559b cburst 1762b
Sent 2735702 bytes 1811 pkts (dropped 0, overlimits 0) rate 6397bps 4pps
lended: 1802 borrowed: 9 giants: 0
tokens: 312898 ctokens: 163555

class htb 1:3 parent 1:1 prio 0 rate 30Kbit ceil 128Kbit burst 1637b cburst 1762b
Sent 3108167 bytes 2904 pkts (dropped 0, overlimits 0) rate 9488bps 8pps
lended: 1452 borrowed: 1452 giants: 0
tokens: -561135 ctokens: -97151

Description of the test:
------------------------
On the beginning it was everything OK, after 1 min, 1:2 lost his 98kbit. Than he got sometimes his 98kbit again and sometimes he got even 30kbit.

Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

1. Can I do something more to find out what happen?
2. What does mean: "queues are build up" ?

Sergiusz

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#11 lore raw
In reply to Sergiusz Brzeziński

Original Message:
-----------------
From: Sergiusz Brzeziñski [email protected]

Quoted 4 lines

Well, it helped in 80%. Why only in 80? I repeated my test with SFQ and:
- it worked better than bevore, there where long time periods (15-20sec.) with right rate-values
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

I didnt follow this thread from beginning, sorry if i didnt solve the problem. From your last post to this list i assumed you have set correct rate values to each class, but the distribution is still unfair. Am i right ?

At first i applied HTB for my bandwidth manager i came into the same problem. When my users at one class start to open several tcp streams, perhaps you can see such as Kazaa, DAP, FlashGET. These can consume high bandwidth. Also they opened several tcp streams to the same destination host. The fact is those class get high rate while the other class is suffer.

I applied HTB with SFQ as leaf qdisc. I played with the quantum values and it started to work well.
Try it..

Regards,
Rio Martin.

--------------------------------------------------------------------
mail2web - Check your email from the web at
http://mail2web.com/ .

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#12 lore raw
In reply to Sergiusz Brzeziński
Quoted 7 lines

U¿ytkownik devik napisa³:

Quoted 4 lines

Well. The right way for your case would be to limit single subqueue in SFQ. See line 24 of attached patch - and try patch itself.
devik

Quoted 5 lines

hmm.. with pfifo_fast is thesame problem - no drops... and unstable. It doesn't look like a SFQ-specific problem. With fifo (without setting limit) there were only 2 drops. Only "fifo limit 10" generates drops (but it has also periods with no drops and with bad rates). Everything of course with tcp_wmem = "4096 2048000 5120000"

because pfifo_fast uses default queue size for eth = 100. It is almost the same as SFQ uses. Did you try my patch ?

Regarding wmem, try also increasing /proc/sys/net/core/wmem_max. Maybe it is blocking TCP's settings.

devik

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#13 lore raw
In reply to Sergiusz Brzeziński
Quoted 9 lines · Sergiusz Brzezi?ski
machine:            AMD K6 200 MHz
Linux distribution: Mandrake 8.1
kernel:             compiled 2.4.21

applied this:
#define PSCHED_CLOCK_SOURCE PSCHED_CPU in file linux/include/net/pkt_sched.h bevore compiled the kernel (described on http://www.docum.org/stef.coene/qos/faq/cache/40.html)
bandwitch on eth0: 128kbit

Could this have something to do with this known bug in linux kernels 2.5.x ?
http://bugme.osdl.org/show_bug.cgi?ide7

Regards.

Signature · 5 lines
Jose Luis Domingo Lopez
Linux Registered User #189436     Debian Linux Sid (Linux 2.5.73)
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/