Making tcp start transfers slow
Hey list
I have almost gotten my shaping setup up and running as planned. The last barrier seems to be tcp overshooting availible bandwidth when its starting a transfer, and thereby bursting the line, so ping rises for a moment. At least this is my best guess at the problem :) There is a possibility that its just plain old traffic being bursty for some reason.. I am using bittorrent to test this, as it seems to be what stresses the line the most.
Would it be possible to lower the default window size, and thereby making tcp start up slower, or would this just lower the overall speed? My efforts so far can be seen here:
http://tc.schmakk.dk/betashaper
Signature · 5 lines
Patrick Petersen <[email protected]> _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
It is possible to slow down tcp start and it realy helped for me to get better pings,
but expense is too high, so I think it do not work as it should,
if you start 5 coonections at once you receive 5 1.5kbyte packets, what fills your queue at isp side
the more connections you start at once the worse delay will be. this can be partialy fixed if you give some reserve using about 80-90% of link like everybody usualy do
now I am working on new driver which could help to solve this, the only way probably is predict new coonecions and reduce speed of exsisting ones before new ones start.
----- Original Message -----
From: "Patrick Petersen" <[email protected]>
To: <[email protected]>
Sent: Thursday, April 15, 2004 1:26 PM
Subject: [LARTC] Making tcp start transfers slow
Quoted 4 lines
Hey list
I have almost gotten my shaping setup up and running as planned. The last barrier seems to be tcp overshooting availible bandwidth when its starting a transfer, and thereby bursting the line, so ping rises for a
Quoted 15 lines
moment. At least this is my best guess at the problem :) There is a possibility that its just plain old traffic being bursty for some reason.. I am using bittorrent to test this, as it seems to be what stresses the line the most.
Would it be possible to lower the default window size, and thereby making tcp start up slower, or would this just lower the overall speed? My efforts so far can be seen here:
http://tc.schmakk.dk/betashaperSignature · 5 lines
Patrick Petersen <[email protected]> _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 5 lines · Patrick Petersen
Hey list
I have almost gotten my shaping setup up and running as planned. The last barrier seems to be tcp overshooting availible bandwidth when its starting a transfer, and thereby bursting the line, so ping rises for a moment. At least this is my best guess at the problem :)
I sortof workaround this using the connbytes netfilter patch to make the first 80k of new connections go into a short queue limited to 1/3 - 1/2 of my downstream bandwidth. It works well in the case where the link is empty apart from a gamestream and someone is browsing "heavy .jpg" type web paged. It also helps a bit if there is other traffic - but if there are enough tcp connections on the go there will be higher latency bursts caused by new connections as HTB can't throttle until it's a bit too late.
Quoted 3 lines
There is a possibility that its just plain old traffic being bursty for some reason.. I am using bittorrent to test this, as it seems to be what stresses the line the most.
Ahh bittorrent - this is a special case. It uses full duplex tcp - so may break some upstream shapers, you can assume that a fair number of your peers have flooded modem buffers - so there could be quite a few "unstoppable" packets headed your way. The worse thing about it, though is that it runs it's own algorithm on allready open tcps - so existing connections may go back into slowstart.
Quoted 2 lines
Would it be possible to lower the default window size, and thereby making tcp start up slower, or would this just lower the overall speed?
It could help, but may also give you less of a share of a given uploaders bandwidth. Reducing MTU may also help (if you run BT on linux it may reduce rwin for you aswell).
Quoted 2 lines
My efforts so far can be seen here:
http://tc.schmakk.dk/betashaper
Had a quick look - Some thoughts:
I don't think you can catch all BT traffic by marking the BT ports, I see ipp2p - can you do it with this or maybe do per IP fairness for bulk traffic?
Be carefull about priorotising acks - don't all TCP packets after syn have ack set. Being lazy on a home setup I get away with giving small packets priority - saves alot of marking :-)
For ingress shaping - I find it nicer to shape per IP with htb and use esfq classic to get per tcp fairness rather than esfq on dst which is going to effectively make many bittorrent connections go into a FIFO, which could make for more burstiness.
Andy.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
I just had one idea, about this.
what if make iptables module which will make something like enlarged copy of syn packet and send it back to the sender?
(another option would be to kill 1 or 2 ack packets for one syn packet this whould force server to reduce speed)
that way htb could count upcoming packet and prepare by reducing other connetions speed?
of course that synthetic packet will have higest possible priority since it supposed to be appear in future so cant be shaped anyway
I will try to add this functionality to my imq module next week probably.
------------------------
connbytes solution is not good for this, it slows down small picture loading in web pages very much, and big downloads get even more "unused" bandwitch. so effect is not good. expecialy that looks bad on network, when pages become incredibly slow, but big downloads fast.
----- Original Message -----
From: "Andy Furniss" <[email protected]>
To: "Patrick Petersen" <[email protected]>
Cc: <[email protected]>
Sent: Sunday, April 18, 2004 2:13 AM
Subject: Re: [LARTC] Making tcp start transfers slow
Quoted 7 lines
Quoted 2 lines · Patrick Petersen
Hey list
I have almost gotten my shaping setup up and running asplanned. The
Quoted 1 line
last barrier seems to be tcp overshooting availible bandwidth
when its
Quoted 1 line
starting a transfer, and thereby bursting the line, so ping
Quoted 8 lines
rises for a
Quoted 1 line
moment. At least this is my best guess at the problem
:)
I sortof workaround this using the connbytes netfilter patch to make the first 80k of new connections go into a short queue limited to 1/3 - 1/2 of my downstream bandwidth. It works well in the case where the link is empty apart from a gamestream and someone is browsing '"'heavy .jpg'"'
type
Quoted 7 lines
web paged. It also helps a bit if there is other traffic - but if there are enough tcp connections on the go there will be higher latency bursts caused by new connections as HTB can't throttle until it's a bit too late.
Quoted 1 line
There is a possibility that its just plain old traffic being
bursty for
Quoted 8 lines
Quoted 1 line
some reason.. I am using bittorrent to test this, as it seems
to be what
Quoted 1 line
stresses the line the most.
Ahh bittorrent - this is a special case. It uses full duplex tcp - so may break some upstream shapers, you can assume that a fair number of your peers have flooded modem buffers - so there could be quite a few '"'unstoppable'"' packets headed your way. The worse thing about it,
though
Quoted 30 lines
is that it runs it's own algorithm on allready open tcps - so existing connections may go back into slowstart.
Quoted 1 line
Would it be possible to lower the default window size, and
thereby
Quoted 1 line
making tcp start up slower, or would this just lower the
overall speed?
It could help, but may also give you less of a share of a given uploaders bandwidth. Reducing MTU may also help (if you run BT on linux it may reduce rwin for you aswell).
Quoted 2 lines
My efforts so far can be seen here:
http://tc.schmakk.dk/betashaperHad a quick look - Some thoughts:
I don't think you can catch all BT traffic by marking the BT ports, I see ipp2p - can you do it with this or maybe do per IP fairness for bulk traffic?
Be carefull about priorotising acks - don't all TCP packets after syn have ack set. Being lazy on a home setup I get away with giving small packets priority - saves alot of marking :-)
For ingress shaping - I find it nicer to shape per IP with htb and use esfq classic to get per tcp fairness rather than esfq on dst which is going to effectively make many bittorrent connections go into a FIFO, which could make for more burstiness.
Quoted 9 lines
Andy.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 8 lines · Roy
I just had one idea, about this.
what if make iptables module which will make something like enlarged copy of syn packet and send it back to the sender?
(another option would be to kill 1 or 2 ack packets for one syn packet this whould force server to reduce speed)
Quoted 5 lines
that way htb could count upcoming packet and prepare by reducing other connetions speed?
of course that synthetic packet will have higest possible priority since it supposed to be appear in future so cant be shaped anyway
I don't really get this :-)
Quoted 7 lines
I will try to add this functionality to my imq module next week probably.
------------------------
connbytes solution is not good for this, it slows down small picture loading in web pages very much, and big downloads get even more "unused" bandwitch. so effect is not good. expecialy that looks bad on network, when pages become incredibly slow, but big downloads fast.
Depends on lots of things I suppose - the way I have it set new connections get 256kbit - not that bad for browsing. ISTR seeing one of your scripts that did similar, IIRC using sfq with low rates. I don't quite do it like that - for a start sfq 128 queue length is too much and if you use it on ingress sfq will hash the ~4 simoultaneous connections your browser makes into one slot. I guess yours simulated a drop with the reordering when they swapped queues rather than really dropping with a short queue to get out of slowstart.
SFQ causes instability every time it rehashes on ingress because of this
- there is a todo in the code somewhere. I like to set perturb high.
This ingress shaping with stuff made for egress is a bit tricky - but it can be tweaked a bit.
Andy.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 13 lines
Quoted 3 lines · Roy
I just had one idea, about this.
what if make iptables module which will make something like
enlarged copy
Quoted 2 lines
of syn packet and send it back to the sender? (another option would be to kill 1 or 2 ack packets for one syn
packet this
Quoted 4 lines
whould force server to reduce speed)
that way htb could count upcoming packet and prepare by
reducing other
Quoted 9 lines
Quoted 2 lines
connetions speed?
of course that synthetic packet will have higest possiblepriority since it
Quoted 1 line
supposed to be appear in future so cant be shaped
anyway
Quoted 1 line
I don't really get this :-)
-----------------------------------------------
Ok, probably I was not able to explain it quite well, but basicaly it is how to predict incomming connections and decrease speed of exsisting one before new connections will start this prediction method should be simple and easy to implement
(all concept of slow start is wrong, we exactly need fast start and slow download later)
Quoted 21 lines
Quoted 2 lines
I will try to add this functionality to my imq module next week
probably.
Quoted 2 lines
------------------------
connbytes solution is not good for this, it slows down smallpicture loading
Quoted 1 line
in web pages very much, and big downloads get even more
'"'unused'"' bandwitch.
Quoted 1 line
so effect is not good. expecialy that looks bad on network,
when pages
Quoted 1 line
become incredibly slow, but big downloads fast.
Depends on lots of things I suppose - the way I have it set new connections get 256kbit - not that bad for browsing. ISTR seeing one of your scripts that did similar, IIRC using sfq with low rates. I don't quite do it like that - for a start sfq 128 queue length is too much and if you use it on ingress sfq will hash the ~4 simoultaneous connections your browser makes into one slot. I guess yours simulated a drop with the reordering when they swapped queues rather than really dropping with a short queue to get out of slowstart.
Quoted 8 lines
SFQ causes instability every time it rehashes on ingress because of this
- there is a todo in the code somewhere. I like to set perturb high.This ingress shaping with stuff made for egress is a bit tricky - but it can be tweaked a bit.
Andy.
all this work well, on small number of new coonections at once , but try 20 or more at once and you will see that it is not good at all
I am also using this way now, and that why I say it is not good.
I dont think sfq may create any prolems, because it is basicaly same as few random fifo's
Quoted 4 lines
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Sorry for the long response time.
Quoted 3 lines · Andy Furniss
I sortof workaround this using the connbytes netfilter patch to make the first 80k of new connections go into a short queue limited to 1/3 - 1/2 of my downstream bandwidth.
I will try this soon.. I recently screwed my kernel over bad with patch-o-matic, so its fixing time at the moment :)
Quoted 6 lines
Ahh bittorrent - this is a special case. It uses full duplex tcp - so may break some upstream shapers, you can assume that a fair number of your peers have flooded modem buffers - so there could be quite a few "unstoppable" packets headed your way. The worse thing about it, though is that it runs it's own algorithm on allready open tcps - so existing connections may go back into slowstart.
Thats pretty bad.. But if i get this working, even with bittorrent, i guess that means it will probably work with anything! :)
Quoted 10 lines
Quoted 2 lines
Would it be possible to lower the default window size, and thereby making tcp start up slower, or would this just lower the overall speed?
It could help, but may also give you less of a share of a given uploaders bandwidth. Reducing MTU may also help (if you run BT on linux it may reduce rwin for you aswell).
I don't think you can catch all BT traffic by marking the BT ports, I see ipp2p - can you do it with this or maybe do per IP fairness for bulk traffic?
Im fortunate enough to be able to run a small network with only a few people, so theres no real bad things with people doing whatever they can to cheat. Also, ipp2p should catch torrent connections, and seems to be doing so.
Quoted 8 lines
Be carefull about priorotising acks - don't all TCP packets after syn have ack set. Being lazy on a home setup I get away with giving small packets priority - saves alot of marking :-)
For ingress shaping - I find it nicer to shape per IP with htb and use esfq classic to get per tcp fairness rather than esfq on dst which is going to effectively make many bittorrent connections go into a FIFO, which could make for more burstiness.
I'll make a note, but shouldnt esfq make things more or less fair no matter how much traffic one client has over another?
Signature · 5 lines
Patrick Petersen <[email protected]> _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 5 lines · Patrick Petersen
Sorry for the long response time.
On Sun, 18 Apr 2004 00:13:49 +0100, Andy Furniss <[email protected]> wrote:
Quoted 7 lines
Quoted 3 lines
I sortof workaround this using the connbytes netfilter patch to make the first 80k of new connections go into a short queue limited to 1/3 - 1/2 of my downstream bandwidth.
I will try this soon.. I recently screwed my kernel over bad with patch-o-matic, so its fixing time at the moment :)
One possible problem - I don't think connbytes will patch with connmark (which IIRC ipp2p uses). I have been told that it is possible to get them to work, but patch will fail as they both change the same struct, so it's a do it by hand job...
Quoted 10 lines
Quoted 8 lines
Ahh bittorrent - this is a special case. It uses full duplex tcp - so may break some upstream shapers, you can assume that a fair number of your peers have flooded modem buffers - so there could be quite a few "unstoppable" packets headed your way. The worse thing about it, though is that it runs it's own algorithm on allready open tcps - so existing connections may go back into slowstart.
Quoted 2 lines
Thats pretty bad.. But if i get this working, even with bittorrent, i guess that means it will probably work with anything! :)
LOL - I think your idea about reducing rwin may help - easy on a small setup where you have control.
Quoted 14 lines
Quoted 5 lines
Quoted 4 lines
Would it be possible to lower the default window size, and thereby making tcp start up slower, or would this just lower the overall speed?
Quoted 7 lines
It could help, but may also give you less of a share of a given uploaders bandwidth. Reducing MTU may also help (if you run BT on linux it may reduce rwin for you aswell).
I don't think you can catch all BT traffic by marking the BT ports, I see ipp2p - can you do it with this or maybe do per IP fairness for bulk traffic?
Quoted 4 lines
Im fortunate enough to be able to run a small network with only a few people, so theres no real bad things with people doing whatever they can to cheat. Also, ipp2p should catch torrent connections, and seems to be doing so.
Yes - so you don't need to mark all those ports then.
Quoted 6 lines
Quoted 6 lines
Be carefull about priorotising acks - don't all TCP packets after syn have ack set. Being lazy on a home setup I get away with giving small packets priority - saves alot of marking :-)
Quoted 8 lines
Quoted 4 lines
For ingress shaping - I find it nicer to shape per IP with htb and use esfq classic to get per tcp fairness rather than esfq on dst which is going to effectively make many bittorrent connections go into a FIFO, which could make for more burstiness.
I'll make a note, but shouldnt esfq make things more or less fair no matter how much traffic one client has over another?
Yes esfq on dst will make things fair per IP address - but in the case of ingress with the many tcps that BT uses dropping is important to get the sender to back off. Esfq on dst is not really a fairness queue any more WRT individual tcp connections, it may not make much difference, but if you use htb for per IP fairness and esfq classic, then the drops are a bit more likely to get at the most active connections.
I modified esfq to drop from the head of a slot rather than the end - which in theory should make things slightly nicer and changed hash to use dst port. I really ought to make them into a switches (when I work out how) I also intend to patch with the change to sfq that was recently talked about on here. If it looks OK I'll put it up on my webspace sometime - I don't really know what I am doing though :-)
For us small bandwidth users a R(real)FQ would be nice - (e)sfq is OK but it often hashes into the same slot and perturb causes packet reordering which hurts more when used for ingress.
Andy.
_______________________________________________
LARTC mailing list / [email protected]
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Quoted 4 lines · Andy Furniss
One possible problem - I don't think connbytes will patch with connmark (which IIRC ipp2p uses). I have been told that it is possible to get them to work, but patch will fail as they both change the same struct, so it's a do it by hand job...
Im usually all for the "do it by hand" thing, but im not really up for screwing with this yet i think :)
Its correct that ipp2p uses connmark... That way it is only necessary to tag a connection once, then leave it. Great for those huge setups, where the ipp2p usage should be kept at a minimum.. Afaik its a bit heavy on ressources.
[SNIP - torrent talk]
Quoted 2 lines
LOL - I think your idea about reducing rwin may help - easy on a small setup where you have control.
That was my idea, but so far ive only found out how to alter the window size on a per route basis. If it could just be extended a little by making new connections have a small size, then putting them back to normal after a while. But if i went all the way out there, i might just as well start a complete dynamic window chaning thing.
[SNIP - more torrent and ipp2p]
Quoted 1 line
Yes - so you don't need to mark all those ports then.
Probably not, but i dont think the extra overhead will do any harm as long as its a puny adsl im dealing with :) I guess im not trusting ipp2p enough.
Quoted 7 lines
Yes esfq on dst will make things fair per IP address - but in the case of ingress with the many tcps that BT uses dropping is important to get the sender to back off. Esfq on dst is not really a fairness queue any more WRT individual tcp connections, it may not make much difference, but if you use htb for per IP fairness and esfq classic, then the drops are a bit more likely to get at the most active connections.
Quoted 6 lines
I modified esfq to drop from the head of a slot rather than the end - which in theory should make things slightly nicer and changed hash to use dst port. I really ought to make them into a switches (when I work out how) I also intend to patch with the change to sfq that was recently talked about on here. If it looks OK I'll put it up on my webspace sometime - I don't really know what I am doing though :-)
Sounds nice, share when you think its stable. Im all for betatesting too if you need any.
Quoted 3 lines
For us small bandwidth users a R(real)FQ would be nice - (e)sfq is OK but it often hashes into the same slot and perturb causes packet reordering which hurts more when used for ingress.
I will try fiddling without esfq with fingers crossed - thanks.
Signature · 5 lines
Patrick Petersen <[email protected]> _______________________________________________ LARTC mailing list / [email protected] http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/