Skip to contentSkip to conversation

Making tcp start transfers slow

LARTC Patrick Petersen 3 participants 9 messages Apr 15, 2004 – Apr 24, 2004

lore
Share & tools
lore
#0 lore raw

Hey list
I have almost gotten my shaping setup up and running as planned. The last barrier seems to be tcp overshooting availible bandwidth when its starting a transfer, and thereby bursting the line, so ping rises for a moment. At least this is my best guess at the problem :) There is a possibility that its just plain old traffic being bursty for some reason.. I am using bittorrent to test this, as it seems to be what stresses the line the most.
Would it be possible to lower the default window size, and thereby making tcp start up slower, or would this just lower the overall speed? My efforts so far can be seen here:
http://tc.schmakk.dk/betashaper

Signature · 5 lines
Patrick Petersen <[email protected]>

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#1 lore raw

It is possible to slow down tcp start and it realy helped for me to get better pings,
but expense is too high, so I think it do not work as it should,

if you start 5 coonections at once you receive 5 1.5kbyte packets, what fills your queue at isp side
the more connections you start at once the worse delay will be. this can be partialy fixed if you give some reserve using about 80-90% of link like everybody usualy do

now I am working on new driver which could help to solve this, the only way probably is predict new coonecions and reduce speed of exsisting ones before new ones start.

----- Original Message -----
From: "Patrick Petersen" <[email protected]>
To: <[email protected]>
Sent: Thursday, April 15, 2004 1:26 PM
Subject: [LARTC] Making tcp start transfers slow

Quoted 4 lines

Hey list
I have almost gotten my shaping setup up and running as planned. The last barrier seems to be tcp overshooting availible bandwidth when its starting a transfer, and thereby bursting the line, so ping rises for a

Quoted 15 lines

moment. At least this is my best guess at the problem :) There is a possibility that its just plain old traffic being bursty for some reason.. I am using bittorrent to test this, as it seems to be what stresses the line the most.
Would it be possible to lower the default window size, and thereby making tcp start up slower, or would this just lower the overall speed? My efforts so far can be seen here:
http://tc.schmakk.dk/betashaper

Signature · 5 lines
Patrick Petersen <[email protected]>

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#2 lore raw
In reply to Patrick Petersen
Quoted 5 lines · Patrick Petersen

Hey list
I have almost gotten my shaping setup up and running as planned. The last barrier seems to be tcp overshooting availible bandwidth when its starting a transfer, and thereby bursting the line, so ping rises for a moment. At least this is my best guess at the problem :)

I sortof workaround this using the connbytes netfilter patch to make the first 80k of new connections go into a short queue limited to 1/3 - 1/2 of my downstream bandwidth. It works well in the case where the link is empty apart from a gamestream and someone is browsing "heavy .jpg" type web paged. It also helps a bit if there is other traffic - but if there are enough tcp connections on the go there will be higher latency bursts caused by new connections as HTB can't throttle until it's a bit too late.

Quoted 3 lines

There is a possibility that its just plain old traffic being bursty for some reason.. I am using bittorrent to test this, as it seems to be what stresses the line the most.

Ahh bittorrent - this is a special case. It uses full duplex tcp - so may break some upstream shapers, you can assume that a fair number of your peers have flooded modem buffers - so there could be quite a few "unstoppable" packets headed your way. The worse thing about it, though is that it runs it's own algorithm on allready open tcps - so existing connections may go back into slowstart.

Quoted 2 lines

Would it be possible to lower the default window size, and thereby making tcp start up slower, or would this just lower the overall speed?

It could help, but may also give you less of a share of a given uploaders bandwidth. Reducing MTU may also help (if you run BT on linux it may reduce rwin for you aswell).

Quoted 2 lines

My efforts so far can be seen here:
http://tc.schmakk.dk/betashaper

Had a quick look - Some thoughts:

I don't think you can catch all BT traffic by marking the BT ports, I see ipp2p - can you do it with this or maybe do per IP fairness for bulk traffic?

Be carefull about priorotising acks - don't all TCP packets after syn have ack set. Being lazy on a home setup I get away with giving small packets priority - saves alot of marking :-)

For ingress shaping - I find it nicer to shape per IP with htb and use esfq classic to get per tcp fairness rather than esfq on dst which is going to effectively make many bittorrent connections go into a FIFO, which could make for more burstiness.

Andy.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#3 lore raw
In reply to Patrick Petersen

I just had one idea, about this.

what if make iptables module which will make something like enlarged copy of syn packet and send it back to the sender?
(another option would be to kill 1 or 2 ack packets for one syn packet this whould force server to reduce speed)

that way htb could count upcoming packet and prepare by reducing other connetions speed?
of course that synthetic packet will have higest possible priority since it supposed to be appear in future so cant be shaped anyway

I will try to add this functionality to my imq module next week probably.
------------------------
connbytes solution is not good for this, it slows down small picture loading in web pages very much, and big downloads get even more "unused" bandwitch. so effect is not good. expecialy that looks bad on network, when pages become incredibly slow, but big downloads fast.

----- Original Message -----
From: "Andy Furniss" <[email protected]>
To: "Patrick Petersen" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, April 18, 2004 2:13 AM
Subject: Re: [LARTC] Making tcp start transfers slow

Quoted 7 lines
Quoted 2 lines · Patrick Petersen

Hey list
I have almost gotten my shaping setup up and running as

planned. The

Quoted 1 line

last barrier seems to be tcp overshooting availible bandwidth

when its

Quoted 1 line

starting a transfer, and thereby bursting the line, so ping

Quoted 8 lines

rises for a

Quoted 1 line

moment. At least this is my best guess at the problem

:)

I sortof workaround this using the connbytes netfilter patch to make the first 80k of new connections go into a short queue limited to 1/3 - 1/2 of my downstream bandwidth. It works well in the case where the link is empty apart from a gamestream and someone is browsing '"'heavy .jpg'"'

type

Quoted 7 lines

web paged. It also helps a bit if there is other traffic - but if there are enough tcp connections on the go there will be higher latency bursts caused by new connections as HTB can't throttle until it's a bit too late.

Quoted 1 line

There is a possibility that its just plain old traffic being

bursty for

Quoted 8 lines
Quoted 1 line

some reason.. I am using bittorrent to test this, as it seems

to be what

Quoted 1 line

stresses the line the most.

Ahh bittorrent - this is a special case. It uses full duplex tcp - so may break some upstream shapers, you can assume that a fair number of your peers have flooded modem buffers - so there could be quite a few '"'unstoppable'"' packets headed your way. The worse thing about it,

though

Quoted 30 lines

is that it runs it's own algorithm on allready open tcps - so existing connections may go back into slowstart.

Quoted 1 line

Would it be possible to lower the default window size, and

thereby

Quoted 1 line

making tcp start up slower, or would this just lower the

overall speed?

It could help, but may also give you less of a share of a given uploaders bandwidth. Reducing MTU may also help (if you run BT on linux it may reduce rwin for you aswell).

Quoted 2 lines

My efforts so far can be seen here:
http://tc.schmakk.dk/betashaper

Had a quick look - Some thoughts:

I don't think you can catch all BT traffic by marking the BT ports, I see ipp2p - can you do it with this or maybe do per IP fairness for bulk traffic?

Be carefull about priorotising acks - don't all TCP packets after syn have ack set. Being lazy on a home setup I get away with giving small packets priority - saves alot of marking :-)

For ingress shaping - I find it nicer to shape per IP with htb and use esfq classic to get per tcp fairness rather than esfq on dst which is going to effectively make many bittorrent connections go into a FIFO, which could make for more burstiness.

Quoted 9 lines

Andy.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#4 lore raw
In reply to Patrick Petersen
Quoted 8 lines · Roy

I just had one idea, about this.

what if make iptables module which will make something like enlarged copy of syn packet and send it back to the sender?
(another option would be to kill 1 or 2 ack packets for one syn packet this whould force server to reduce speed)

Quoted 5 lines

that way htb could count upcoming packet and prepare by reducing other connetions speed?
of course that synthetic packet will have higest possible priority since it supposed to be appear in future so cant be shaped anyway

I don't really get this :-)

Quoted 7 lines

I will try to add this functionality to my imq module next week probably.
------------------------
connbytes solution is not good for this, it slows down small picture loading in web pages very much, and big downloads get even more "unused" bandwitch. so effect is not good. expecialy that looks bad on network, when pages become incredibly slow, but big downloads fast.

Depends on lots of things I suppose - the way I have it set new connections get 256kbit - not that bad for browsing. ISTR seeing one of your scripts that did similar, IIRC using sfq with low rates. I don't quite do it like that - for a start sfq 128 queue length is too much and if you use it on ingress sfq will hash the ~4 simoultaneous connections your browser makes into one slot. I guess yours simulated a drop with the reordering when they swapped queues rather than really dropping with a short queue to get out of slowstart.

SFQ causes instability every time it rehashes on ingress because of this
- there is a todo in the code somewhere. I like to set perturb high.

This ingress shaping with stuff made for egress is a bit tricky - but it can be tweaked a bit.

Andy.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#5 lore raw
In reply to Patrick Petersen
Quoted 13 lines
Quoted 3 lines · Roy

I just had one idea, about this.

what if make iptables module which will make something like

enlarged copy

Quoted 2 lines

of syn packet and send it back to the sender? (another option would be to kill 1 or 2 ack packets for one syn

packet this

Quoted 4 lines

whould force server to reduce speed)

that way htb could count upcoming packet and prepare by

reducing other

Quoted 9 lines
Quoted 2 lines

connetions speed?
of course that synthetic packet will have higest possible

priority since it

Quoted 1 line

supposed to be appear in future so cant be shaped

anyway

Quoted 1 line

I don't really get this :-)

-----------------------------------------------
Ok, probably I was not able to explain it quite well, but basicaly it is how to predict incomming connections and decrease speed of exsisting one before new connections will start this prediction method should be simple and easy to implement

(all concept of slow start is wrong, we exactly need fast start and slow download later)

Quoted 21 lines
Quoted 2 lines

I will try to add this functionality to my imq module next week

probably.

Quoted 2 lines

------------------------
connbytes solution is not good for this, it slows down small

picture loading

Quoted 1 line

in web pages very much, and big downloads get even more

'"'unused'"' bandwitch.

Quoted 1 line

so effect is not good. expecialy that looks bad on network,

when pages

Quoted 1 line

become incredibly slow, but big downloads fast.

Depends on lots of things I suppose - the way I have it set new connections get 256kbit - not that bad for browsing. ISTR seeing one of your scripts that did similar, IIRC using sfq with low rates. I don't quite do it like that - for a start sfq 128 queue length is too much and if you use it on ingress sfq will hash the ~4 simoultaneous connections your browser makes into one slot. I guess yours simulated a drop with the reordering when they swapped queues rather than really dropping with a short queue to get out of slowstart.

Quoted 8 lines

SFQ causes instability every time it rehashes on ingress because of this
- there is a todo in the code somewhere. I like to set perturb high.

This ingress shaping with stuff made for egress is a bit tricky - but it can be tweaked a bit.

Andy.

all this work well, on small number of new coonections at once , but try 20 or more at once and you will see that it is not good at all
I am also using this way now, and that why I say it is not good.

I dont think sfq may create any prolems, because it is basicaly same as few random fifo's

Quoted 4 lines

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#6 lore raw
In reply to Patrick Petersen

Sorry for the long response time.

Quoted 3 lines · Andy Furniss

I sortof workaround this using the connbytes netfilter patch to make the first 80k of new connections go into a short queue limited to 1/3 - 1/2 of my downstream bandwidth.

I will try this soon.. I recently screwed my kernel over bad with patch-o-matic, so its fixing time at the moment :)

Quoted 6 lines

Ahh bittorrent - this is a special case. It uses full duplex tcp - so may break some upstream shapers, you can assume that a fair number of your peers have flooded modem buffers - so there could be quite a few "unstoppable" packets headed your way. The worse thing about it, though is that it runs it's own algorithm on allready open tcps - so existing connections may go back into slowstart.

Thats pretty bad.. But if i get this working, even with bittorrent, i guess that means it will probably work with anything! :)

Quoted 10 lines
Quoted 2 lines

Would it be possible to lower the default window size, and thereby making tcp start up slower, or would this just lower the overall speed?

It could help, but may also give you less of a share of a given uploaders bandwidth. Reducing MTU may also help (if you run BT on linux it may reduce rwin for you aswell).

I don't think you can catch all BT traffic by marking the BT ports, I see ipp2p - can you do it with this or maybe do per IP fairness for bulk traffic?

Im fortunate enough to be able to run a small network with only a few people, so theres no real bad things with people doing whatever they can to cheat. Also, ipp2p should catch torrent connections, and seems to be doing so.

Quoted 8 lines

Be carefull about priorotising acks - don't all TCP packets after syn have ack set. Being lazy on a home setup I get away with giving small packets priority - saves alot of marking :-)

For ingress shaping - I find it nicer to shape per IP with htb and use esfq classic to get per tcp fairness rather than esfq on dst which is going to effectively make many bittorrent connections go into a FIFO, which could make for more burstiness.

I'll make a note, but shouldnt esfq make things more or less fair no matter how much traffic one client has over another?

Signature · 5 lines
Patrick Petersen <[email protected]>

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
#7 lore raw
In reply to Patrick Petersen
Quoted 5 lines · Patrick Petersen

Sorry for the long response time.

On Sun, 18 Apr 2004 00:13:49 +0100, Andy Furniss <[email protected]> wrote:

Quoted 7 lines
Quoted 3 lines

I sortof workaround this using the connbytes netfilter patch to make the first 80k of new connections go into a short queue limited to 1/3 - 1/2 of my downstream bandwidth.

I will try this soon.. I recently screwed my kernel over bad with patch-o-matic, so its fixing time at the moment :)

One possible problem - I don't think connbytes will patch with connmark (which IIRC ipp2p uses). I have been told that it is possible to get them to work, but patch will fail as they both change the same struct, so it's a do it by hand job...

Quoted 10 lines
Quoted 8 lines

Ahh bittorrent - this is a special case. It uses full duplex tcp - so may break some upstream shapers, you can assume that a fair number of your peers have flooded modem buffers - so there could be quite a few "unstoppable" packets headed your way. The worse thing about it, though is that it runs it's own algorithm on allready open tcps - so existing connections may go back into slowstart.

Quoted 2 lines

Thats pretty bad.. But if i get this working, even with bittorrent, i guess that means it will probably work with anything! :)

LOL - I think your idea about reducing rwin may help - easy on a small setup where you have control.

Quoted 14 lines
Quoted 5 lines
Quoted 4 lines

Would it be possible to lower the default window size, and thereby making tcp start up slower, or would this just lower the overall speed?

Quoted 7 lines

It could help, but may also give you less of a share of a given uploaders bandwidth. Reducing MTU may also help (if you run BT on linux it may reduce rwin for you aswell).

I don't think you can catch all BT traffic by marking the BT ports, I see ipp2p - can you do it with this or maybe do per IP fairness for bulk traffic?

Quoted 4 lines

Im fortunate enough to be able to run a small network with only a few people, so theres no real bad things with people doing whatever they can to cheat. Also, ipp2p should catch torrent connections, and seems to be doing so.

Yes - so you don't need to mark all those ports then.

Quoted 6 lines
Quoted 6 lines

Be carefull about priorotising acks - don't all TCP packets after syn have ack set. Being lazy on a home setup I get away with giving small packets priority - saves alot of marking :-)

Quoted 8 lines
Quoted 4 lines

For ingress shaping - I find it nicer to shape per IP with htb and use esfq classic to get per tcp fairness rather than esfq on dst which is going to effectively make many bittorrent connections go into a FIFO, which could make for more burstiness.

I'll make a note, but shouldnt esfq make things more or less fair no matter how much traffic one client has over another?

Yes esfq on dst will make things fair per IP address - but in the case of ingress with the many tcps that BT uses dropping is important to get the sender to back off. Esfq on dst is not really a fairness queue any more WRT individual tcp connections, it may not make much difference, but if you use htb for per IP fairness and esfq classic, then the drops are a bit more likely to get at the most active connections.

I modified esfq to drop from the head of a slot rather than the end - which in theory should make things slightly nicer and changed hash to use dst port. I really ought to make them into a switches (when I work out how) I also intend to patch with the change to sfq that was recently talked about on here. If it looks OK I'll put it up on my webspace sometime - I don't really know what I am doing though :-)

For us small bandwidth users a R(real)FQ would be nice - (e)sfq is OK but it often hashes into the same slot and perturb causes packet reordering which hurts more when used for ingress.

Andy.

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

#8 lore raw
In reply to Patrick Petersen
Quoted 4 lines · Andy Furniss

One possible problem - I don't think connbytes will patch with connmark (which IIRC ipp2p uses). I have been told that it is possible to get them to work, but patch will fail as they both change the same struct, so it's a do it by hand job...

Im usually all for the "do it by hand" thing, but im not really up for screwing with this yet i think :)
Its correct that ipp2p uses connmark... That way it is only necessary to tag a connection once, then leave it. Great for those huge setups, where the ipp2p usage should be kept at a minimum.. Afaik its a bit heavy on ressources.

[SNIP - torrent talk]

Quoted 2 lines

LOL - I think your idea about reducing rwin may help - easy on a small setup where you have control.

That was my idea, but so far ive only found out how to alter the window size on a per route basis. If it could just be extended a little by making new connections have a small size, then putting them back to normal after a while. But if i went all the way out there, i might just as well start a complete dynamic window chaning thing.

[SNIP - more torrent and ipp2p]

Quoted 1 line

Yes - so you don't need to mark all those ports then.

Probably not, but i dont think the extra overhead will do any harm as long as its a puny adsl im dealing with :) I guess im not trusting ipp2p enough.

Quoted 7 lines

Yes esfq on dst will make things fair per IP address - but in the case of ingress with the many tcps that BT uses dropping is important to get the sender to back off. Esfq on dst is not really a fairness queue any more WRT individual tcp connections, it may not make much difference, but if you use htb for per IP fairness and esfq classic, then the drops are a bit more likely to get at the most active connections.

Quoted 6 lines

I modified esfq to drop from the head of a slot rather than the end - which in theory should make things slightly nicer and changed hash to use dst port. I really ought to make them into a switches (when I work out how) I also intend to patch with the change to sfq that was recently talked about on here. If it looks OK I'll put it up on my webspace sometime - I don't really know what I am doing though :-)

Sounds nice, share when you think its stable. Im all for betatesting too if you need any.

Quoted 3 lines

For us small bandwidth users a R(real)FQ would be nice - (e)sfq is OK but it often hashes into the same slot and perturb causes packet reordering which hurts more when used for ingress.

I will try fiddling without esfq with fingers crossed - thanks.

Signature · 5 lines
Patrick Petersen <[email protected]>

_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/