Skip to contentSkip to conversation

bandwidth for bkbits.net (good news)

DISCUSSION Pascal Schmidt 12 participants 66 messages Aug 30, 2003 – Sep 8, 2003 3 messages not in the archive

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): [email protected]

#3 lore raw

On Sat, 30 Aug 2003 18:20:11 +0200, you wrote in linux.kernel:

Quoted 6 lines

In order to control incoming traffic, it is easiest to look at tcp. udp can work similarly, but it doesn't have to. To throttle the stream of tcp packets, you could simply throw away the acks and the sending side will reduce it's speed. So you have to measure one stream and control a related but different one. Maybe this is possible, not sure.

All you have to do is drop the incoming packets if they exceed a certain bandwidth. That will stop the corresponding ack automatically since your TCP stack won't even see the packet.

I'm doing this on my ISDN line to limit the rest of the house to one third of the bandwidth even if they're all busy downloading tons of warez. I'm paying, I want the bandwidth when I need it. They can get it all when there's no traffic for my machine.

No problem with the HTB queueing discipline.

[...]

Quoted 3 lines

Third, there is the problem transition from continuous streams to discrete packets, when bandwidth is low. It doesn't take a huge amount of large packets to create enough latency for your sad VOIP.

Yes, latency is a problem if you want to saturate your bandwidth. It is easy to guarantee some bandwith for latency critical stuff - just don't give out that piece of bandwith to something else. Of course, then most of the time that piece is wasted... and it doesn't help with problems somewhere along the net.

Quoted 3 lines

And fourth, the possibility of resonance frequency. If measurement and/or shaping intervals are too long and match nicely to the travel time to one of your peers, you are back at point three.

Dunno about that one.

Signature · 2 lines
Ciao,
Pascal
#4 lore raw
Quoted 11 lines · Pascal Schmidt

On Sat, 30 Aug 2003 18:20:11 +0200, you wrote in linux.kernel:

Quoted 6 lines

In order to control incoming traffic, it is easiest to look at tcp. udp can work similarly, but it doesn't have to. To throttle the stream of tcp packets, you could simply throw away the acks and the sending side will reduce it's speed. So you have to measure one stream and control a related but different one. Maybe this is possible, not sure.

All you have to do is drop the incoming packets if they exceed a certain bandwidth.

If you think we haven't done that, think again. We're at the wrong end of the pipe to do that, I'm pretty sure that what you are describing simply won't work. We've already tried rate limiting all traffic except for the phone traffic to 1/2 the T1 line and it doesn't work.

Remember, we're talking vioce here. Retransmits don't work and aren't even attempted. I've got people in 5 or 6 states spread out from one end of the USA to the other. So any disruption means things don't work.

All of the traffic shaping tends to be focussed on "good" versus "bad" bandwidth utilization. That's nice but it means that you are tolerant of a "settling time" during which everyone is fighting for bandwidth.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#5 lore raw
Quoted 6 lines · Larry McVoy
Quoted 2 lines

All you have to do is drop the incoming packets if they exceed a certain bandwidth.

If you think we haven't done that, think again.

We're at the wrong end of the pipe to do that, I'm pretty sure that what you are describing simply won't work.

In a way, you're on the right end of the pipe because the system that does your traffic shaping is part of the general network, viewed from the machines behind the shaper.

Dropping the packets means that the sending side, at least if we're talking TCP, will throttle its sending rate. But, depending on the distance in hops to the sender, it may take up to a few seconds for this to kick in. So I guess that's why it doesn't work for your VoIP case - the senders don't notice fast enough that they should slow down.

When my ISDN line is saturated with downloads from the rest of the people here and data starts to come in for MY machine, it takes about 5 seconds until I get my bandwidth. Too slow for VoIP.

Traffic shaping on the side of your ISP will probably help since they have a bigger incoming pipe. If they hadn't, they'd have the same problem, though.

Signature · 2 lines
Ciao,
Pascal
#6 lore raw
Quoted 13 lines · Pascal Schmidt
Quoted 6 lines · Larry McVoy
Quoted 2 lines

All you have to do is drop the incoming packets if they exceed a certain bandwidth.

If you think we haven't done that, think again.

We're at the wrong end of the pipe to do that, I'm pretty sure that what you are describing simply won't work.

In a way, you're on the right end of the pipe because the system that does your traffic shaping is part of the general network, viewed from the machines behind the shaper.

Quoted 6 lines

Dropping the packets means that the sending side, at least if we're talking TCP, will throttle its sending rate. But, depending on the distance in hops to the sender, it may take up to a few seconds for this to kick in. So I guess that's why it doesn't work for your VoIP case - the senders don't notice fast enough that they should slow down.

that's because you don't limit the bkbits.net to a fixed rate. If you want to give priorities, it won't work well because it takes time to be effective, but if you rate limit hard both ways it has to work, unless you're under syn-flood ;) The downside is that you will waste bandwith (i.e. you will hurt the bkbits.net service even when you don't use voip), but it will work.

This is what I use normally to limit my brother downloads, and it works fine for me (though I don't often place calls through adsl myself, it's basically useless since people only uses cellphones here, and last time I chekced voip wasn't free for cellphones with my isp).

this is the script:

INTERNAL_NET=eth0
EXTERNAL_NET=ppp0

# External net, prioritize with TOS and low prio traffic in MARK chan 6 tc qdisc add dev $EXTERNAL_NET root handle 1: prio bands 5
tc qdisc add dev $EXTERNAL_NET parent 1:1 handle 10: sfq
tc qdisc add dev $EXTERNAL_NET parent 1:2 handle 20: sfq
tc qdisc add dev $EXTERNAL_NET parent 1:3 handle 30: sfq
tc qdisc add dev $EXTERNAL_NET parent 1:4 tbf rate 16kbit buffer 1600 limit 3000 tc qdisc add dev $EXTERNAL_NET parent 1:5 tbf rate 16kbit buffer 1600 limit 3000

tc filter add dev $EXTERNAL_NET protocol ip parent 1:0 prio 4 handle 6 fw flowid 1:4 tc filter add dev $EXTERNAL_NET protocol ip parent 1:0 prio 5 handle 7 fw flowid 1:5

tc qdisc add dev $INTERNAL_NET root handle 1: prio bands 5
tc qdisc add dev $INTERNAL_NET parent 1:1 handle 10: sfq
tc qdisc add dev $INTERNAL_NET parent 1:2 handle 20: sfq
tc qdisc add dev $INTERNAL_NET parent 1:3 handle 30: sfq
tc qdisc add dev $INTERNAL_NET parent 1:4 tbf rate 64kbit buffer 1600 limit 3000 tc qdisc add dev $INTERNAL_NET parent 1:5 tbf rate 64kbit buffer 1600 limit 3000

tc filter add dev $INTERNAL_NET protocol ip parent 1:0 prio 4 handle 8 fw flowid 1:4 tc filter add dev $INTERNAL_NET protocol ip parent 1:0 prio 5 handle 9 fw flowid 1:5

#tc qdisc add dev $EXTERNAL_NET root sfq perturb 30

btw, the sfq is a great idea to have anyways, to provide fariness to the traffing going out of your network.

then the below scripts will turn it off/on (so he can get full bandwith while I'm not at my desk or when he's not doing anything very network intensive ;)

Basically you should run the below with 'start' before placing the call, and running 'stop' after the call is complete. It may take a few seconds to rate limit all the connections but it should work fine.

EXTERNAL_NET=ppp0
INTERNAL_NET=eth0

case "$1" in

	start)
		iptables -t mangle -A PREROUTING -t mangle -j MARK -s 192.168.1.9 --set-mark 6 -i $INTERNAL_NET
		iptables -t mangle -A POSTROUTING -t mangle -j MARK -d 192.168.1.9 --set-mark 8
		;;
	stop)
		iptables -t mangle -D PREROUTING -t mangle -j MARK -s 192.168.1.9 --set-mark 6 -i $INTERNAL_NET
		iptables -t mangle -D POSTROUTING -t mangle -j MARK -d 192.168.1.9 --set-mark 8
		;;
	status)
		iptables -t mangle -L -v
		;;
	*)
		echo "Usage: $0 {start|stop}"
		exit 1
		;;

esac

(he's IP 192.168.1.9 of course, you should replace it with bkbits.net and tweak the 64kbit output/16k input with the input/output bandwith you want to give to bkbits)

BTW, I still wonder how to rsync the bkcvs coherently? (and I wonder how can Peter sync it synchronous too if you don't give him a pair of sequence numbers or a fcntl lock, personally I prefer the sequence numbers so it works for pure read only too like the rsync on kernel.org)

Andrea

#7 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 21 lines · Andrea Arcangeli
Quoted 13 lines · Pascal Schmidt
Quoted 7 lines · Larry McVoy
Quoted 2 lines

All you have to do is drop the incoming packets if they exceed a certain bandwidth.

If you think we haven't done that, think again.

We're at the wrong end of the pipe to do that, I'm pretty sure that what you are describing simply won't work.

In a way, you're on the right end of the pipe because the system that does your traffic shaping is part of the general network, viewed from the machines behind the shaper.

Quoted 6 lines

Dropping the packets means that the sending side, at least if we're talking TCP, will throttle its sending rate. But, depending on the distance in hops to the sender, it may take up to a few seconds for this to kick in. So I guess that's why it doesn't work for your VoIP case - the senders don't notice fast enough that they should slow down.

Quoted 6 lines

that's because you don't limit the bkbits.net to a fixed rate. If you want to give priorities, it won't work well because it takes time to be effective, but if you rate limit hard both ways it has to work, unless you're under syn-flood ;) The downside is that you will waste bandwith (i.e. you will hurt the bkbits.net service even when you don't use voip), but it will work.

How about giving something to voip as a hard limit and then using some shaper to give it more if needed.

Jeff.

- --
*NOTE: This message is ROT-13 encrypted twice for extra protection* -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.3 (GNU/Linux)

iD8DBQE/UVsBwFP0+seVj/4RAkmpAJ9YwjdPLZZsdD7fCRDRHyS+JrJnkgCdG/sc sr5Mqx5Qii//AQPepCqHDcw=
=RoPR
-----END PGP SIGNATURE-----

#8 lore raw
Quoted 32 lines · Jeff Sipek

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 21 lines · Andrea Arcangeli
Quoted 13 lines · Pascal Schmidt
Quoted 7 lines · Larry McVoy
Quoted 2 lines

All you have to do is drop the incoming packets if they exceed a certain bandwidth.

If you think we haven't done that, think again.

We're at the wrong end of the pipe to do that, I'm pretty sure that what you are describing simply won't work.

In a way, you're on the right end of the pipe because the system that does your traffic shaping is part of the general network, viewed from the machines behind the shaper.

Quoted 6 lines

Dropping the packets means that the sending side, at least if we're talking TCP, will throttle its sending rate. But, depending on the distance in hops to the sender, it may take up to a few seconds for this to kick in. So I guess that's why it doesn't work for your VoIP case - the senders don't notice fast enough that they should slow down.

Quoted 6 lines

that's because you don't limit the bkbits.net to a fixed rate. If you want to give priorities, it won't work well because it takes time to be effective, but if you rate limit hard both ways it has to work, unless you're under syn-flood ;) The downside is that you will waste bandwith (i.e. you will hurt the bkbits.net service even when you don't use voip), but it will work.

Quoted 2 lines

How about giving something to voip as a hard limit and then using some shaper to give it more if needed.

it may take a few secs to rate limit the rest, the old tcp connections will keep sending packets that will get rejected when voip kicks in, that's the problem described in the email I answered to with the solution (i.e. limit bkbits.net hard). I'm pretty sure that rate limting bkbits.net hard (like we do with my brother's computer when he generates heavy traffic) will let voip to work flawlessy, but it has the downside of limiting the bandwith available to bkbits users even when nobody place phone calls. Upgrading the current line and using the software rate limiting with the solution I proposed, or adding a new line dedicated to bkbits.net, wouldn't be much different in practice (if it was me I'd choose it only in function of what's more cost effective, after double checking that my algorithm infact works fine for higher bandwidth too, I only tested it on 640kbit).

Andrea

#9 lore raw
In reply to Andrea Arcangeli
Quoted 8 lines · Andrea Arcangeli
Quoted 6 lines

Dropping the packets means that the sending side, at least if we're talking TCP, will throttle its sending rate. But, depending on the distance in hops to the sender, it may take up to a few seconds for this to kick in. So I guess that's why it doesn't work for your VoIP case - the senders don't notice fast enough that they should slow down.

that's because you don't limit the bkbits.net to a fixed rate.

We not only limited bkbits.net to a fixed rate, we limited all TCP based traffic to a fixed rate and it still didn't work.

I'm pretty convinced we can't solve the problem at our end. Maybe we can but I'm voting for throwing another T1 at the problem, we'll try working with the ISP suggested solution of trunking them and rate limiting at their end and if that doesn't work then we'll split them and use one for bkbits.net and other bitmover related TCP traffic and use the other one for phones.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#10 lore raw
Quoted 1 line · Larry McVoy

I'm pretty convinced we can't solve the problem at our end. Maybe we can

For bursts of traffic you can't.

Quoted 1 line

but I'm voting for throwing another T1 at the problem, we'll try working

or switch to ATM ;)

Quoted 4 lines

with the ISP suggested solution of trunking them and rate limiting at their end and if that doesn't work then we'll split them and use one for bkbits.net and other bitmover related TCP traffic and use the other one for phones.

Fractioning the line is also doable but less flexible with some kit.

#11 lore raw
Quoted 4 lines · Alan Cox
Quoted 1 line · Larry McVoy

I'm pretty convinced we can't solve the problem at our end. Maybe we can

For bursts of traffic you can't.

what's the difference of rejecting packets in software, or because the link can't handle them? Assume the guaranteed bandwidth is much lower than what the link can provide (very common here during the day), the isp will have to do the software thing anyways to balance the bandwidth across the different ip.

it works flawlessy for me and it's the same problem (I also use streaming services), and they're unusable until I turn the shaping on, I'm sure that if you use the script and you change 1kbyte/sec to everything but voip it'll work fine for you too, since basically everything else won't pass anymore, it will take ages to open an html page and all the bkbits.net users will hang, and the link will be idle 99% of the time, so voip will take it over as much as it can. I don't think it's a matter of "if it works or not", I think it's a matter of how much you're ok to lose in terms of global bandwith for all the other services but voip.

the only annoying problem I run into, is that tc is missing a flush operation (like iptables -F) but I never need to tweak it anymore so I don't mind too much.

Andrea

#12 lore raw
Quoted 8 lines · Andrea Arcangeli
Quoted 4 lines · Alan Cox
Quoted 1 line · Larry McVoy

I'm pretty convinced we can't solve the problem at our end. Maybe we can

For bursts of traffic you can't.

what's the difference of rejecting packets in software, or because the link can't handle them? Assume the guaranteed bandwidth is much lower

It doesn't work when you dont control incoming. As a simple extreme example if I pingflood you from a fast site then no amount of shaping your end of the link will help, it has to be shaped at the ISP end.

#13 lore raw
Quoted 5 lines · Alan Cox
Quoted 4 lines · Andrea Arcangeli
Quoted 3 lines · Alan Cox
Quoted 1 line · Larry McVoy

I'm pretty convinced we can't solve the problem at our end. Maybe we can

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

For bursts of traffic you can't.

what's the difference of rejecting packets in software, or because the link can't handle them? Assume the guaranteed bandwidth is much lower

It doesn't work when you dont control incoming. As a simple extreme example if I pingflood you from a fast site then no amount of shaping your end of the link will help, it has to be shaped at the ISP end.

If someone wants to DOS you, he can. Full stop.

iirc, Larry was worried about well behaved traffic still doing bad things to his connection.

Jörn

Signature · 4 lines
Fancy algorithms are slow when n is small, and n is usually small.
Fancy algorithms have big constants. Until you know that n is
frequently going to be big, don't get fancy.
-- Rob Pike
#14 lore raw
Quoted 8 lines · Jörn Engel
Quoted 7 lines · Alan Cox
Quoted 6 lines · Andrea Arcangeli
Quoted 4 lines · Alan Cox
Quoted 1 line · Larry McVoy

I'm pretty convinced we can't solve the problem at our end. Maybe we can

For bursts of traffic you can't.

Quoted 8 lines
Quoted 6 lines
Quoted 2 lines

what's the difference of rejecting packets in software, or because the link can't handle them? Assume the guaranteed bandwidth is much lower

It doesn't work when you dont control incoming. As a simple extreme example if I pingflood you from a fast site then no amount of shaping your end of the link will help, it has to be shaped at the ISP end.

If someone wants to DOS you, he can. Full stop.

yes, that's unfixable at Larry's end, and normaly it's unfixable for the ISP too.

Quoted 2 lines

iirc, Larry was worried about well behaved traffic still doing bad things to his connection.

I also understood the problem were the legimate clones/checkouts happening in established state with a single tcp connection generating a burst of traffic.

If this is not the case and Larry is under attack when the voip doesn't work, of course nothing can help him, most probably not even his ISP, but I assumed this wasn't the case.

Andrea

#15 lore raw
Quoted 2 lines · Andrea Arcangeli

yes, that's unfixable at Larry's end, and normaly it's unfixable for the ISP too.

The ISP can do several things to prioritize production traffic or drop malicious traffic. However, this isn't trivial and requires careful planning, and it's unlikely that anyone who is able to would want to do this for a T1 customer (typically, it requires "unusual" configuration of vital production routers with the fat pipes).

Quoted 2 lines
Quoted 2 lines

iirc, Larry was worried about well behaved traffic still doing bad things to his connection.

In this case, it's a bit easier for the ISP because the tweaking can be done on edge routers (which typically die during DoS attacks as well, so countermeasures cannot be applied there). Especially with low-bandwidth links, it shouldn't be too hard to find a smallish, geeky ISP who is willing to try some fiddling on the edge router to improve performance (at least that's true in Germany, don't know about the U.S.).

However, why can't bkbits.net distributed around the world, at least for read access? This would eliminate the choking point once and for all.

#16 lore raw
Quoted 3 lines · Florian Weimer

Andrea Arcangeli <[email protected]> writes:
However, why can't bkbits.net distributed around the world, at least for read access?

It's not just the read access that is the problem but there is absolutely nothing stopping you or anyone else from setting up bkbackup.net as a cache.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#17 lore raw
In reply to Florian Weimer
Quoted 2 lines · Florian Weimer

do this for a T1 customer (typically, it requires "unusual" configuration of vital production routers with the fat pipes).

You need a shaper connected to the ISP backbone which shapes the outgoing traffic for you and a border router which talks to the T1 (C17xx or C26xx). Normally, if your ISP has some sort of clue, you will also need a bastion router which can handle backbone <-> 100 MBit traffic and does dynamic routing updates (EGP or OSPF) to the ISP backbone (A C26xx or C37xx).

This isn't your run-of-the-mill setup but I know of plenty ISPs here that will happily sell you this and also the consulting needed to plan, setup and operate the link. However, you won't get it for $199/month. :-)

Regards
Henning

Signature · 8 lines
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
[email protected]        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)
#18 lore raw
Quoted 11 lines · Henning P. Schmiedehausen
Quoted 2 lines · Florian Weimer

do this for a T1 customer (typically, it requires "unusual" configuration of vital production routers with the fat pipes).

You need a shaper connected to the ISP backbone which shapes the outgoing traffic for you and a border router which talks to the T1 (C17xx or C26xx). Normally, if your ISP has some sort of clue, you will also need a bastion router which can handle backbone <-> 100 MBit traffic and does dynamic routing updates (EGP or OSPF) to the ISP backbone (A C26xx or C37xx).

C37xx can handle a maximum load of 225 kpps (data sheet number, i.e. this value cannot be exceeded even under most favorable conditions), the others handle even less. Such routers are of no help during a DoS attack.

Yes, I snipped the DoS context, and your approach would work in a benign environment. 8-)

#19 lore raw
Quoted 7 lines · Florian Weimer
Quoted 6 lines

You need a shaper connected to the ISP backbone which shapes the outgoing traffic for you and a border router which talks to the T1 (C17xx or C26xx). Normally, if your ISP has some sort of clue, you will also need a bastion router which can handle backbone <-> 100 MBit traffic and does dynamic routing updates (EGP or OSPF) to the ISP backbone (A C26xx or C37xx).

Quoted 7 lines

C37xx can handle a maximum load of 225 kpps (data sheet number, i.e. this value cannot be exceeded even under most favorable conditions), the others handle even less. Such routers are of no help during a DoS attack.

Yes, I snipped the DoS context, and your approach would work in a benign environment. 8-)

225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec

100 MBit / 8 bit = 12,5 MBytes / sec

So, IMHO even with a small packet saturated 100 MBit link you won't reach 225kpps. AFAIK this was Ciscos intention to publish this number. It basically says "you will have filled your link before you fill our router".

I'm pretty sure that your 37xx won't do any routing updates anymore at this point. And if you do _anything_ that forces the packets down the slow path from the routing engine, you're toast anyway.

But I'm pretty sure that a C37xx would handle full 100 MBit traffic to a busy website without any problems. In fact, I know that it does. ;-) (We did switch to a C12000 shortly after, mainly because we went Gigabit).

Regards
Henning

Signature · 8 lines
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
[email protected]        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)
#20 lore raw
Quoted 11 lines · Henning Schmiedehausen
Quoted 2 lines

Yes, I snipped the DoS context, and your approach would work in a benign environment. 8-)

225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec

100 MBit / 8 bit = 12,5 MBytes / sec

So, IMHO even with a small packet saturated 100 MBit link you won't reach 225kpps. AFAIK this was Ciscos intention to publish this number. It basically says "you will have filled your link before you fill our router".

Cisco sells Gigabit Ethernet linecards for this router, and the situation is quite different in this case. And with most attacks, it's a futile exercise to try to combat them on a Fast Ethernet link, anyway. 8-(

Quoted 3 lines

But I'm pretty sure that a C37xx would handle full 100 MBit traffic to a busy website without any problems. In fact, I know that it does. ;-) (We did switch to a C12000 shortly after, mainly because we went Gigabit).

Of course it does. After all, web traffic is somewhat different from pure DoS traffic. 8-)

#21 lore raw
In reply to Henning Schmiedehausen
Quoted 17 lines · Henning Schmiedehausen
Quoted 7 lines · Florian Weimer
Quoted 6 lines

You need a shaper connected to the ISP backbone which shapes the outgoing traffic for you and a border router which talks to the T1 (C17xx or C26xx). Normally, if your ISP has some sort of clue, you will also need a bastion router which can handle backbone <-> 100 MBit traffic and does dynamic routing updates (EGP or OSPF) to the ISP backbone (A C26xx or C37xx).

Quoted 7 lines

C37xx can handle a maximum load of 225 kpps (data sheet number, i.e. this value cannot be exceeded even under most favorable conditions), the others handle even less. Such routers are of no help during a DoS attack.

Yes, I snipped the DoS context, and your approach would work in a benign environment. 8-)

Quoted 8 lines

225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec

100 MBit / 8 bit = 12,5 MBytes / sec

So, IMHO even with a small packet saturated 100 MBit link you won't reach 225kpps. AFAIK this was Ciscos intention to publish this number. It basically says "you will have filled your link before you fill our router".

100Mb/s LAN @ 64 byte packets is 148500 pps half duplex from my testing => 257Kpps full duplex.

Here's some interesting results I've got from the latest intel e100 3.0.0_dev13 driver, showing how the receive rate degrades at higher packet rates.

------------------------
send Kpps recv Kpps
------------------------

126           126
126.5         118.4
128           115.7
130           102.7
135            99.1
140            90.6
148            88.2

------------------------

It's not a CPU issue as it can receive at
the same rate on another interface.
renicing ksoftirqd has no effect.
system is PIII 1.2GHz, i815, 2.4.20
NAPI is turned on in the driver.

Pádraig.

#22 lore raw
In reply to Henning Schmiedehausen
Quoted 8 lines · Henning Schmiedehausen

225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec

100 MBit / 8 bit = 12,5 MBytes / sec

So, IMHO even with a small packet saturated 100 MBit link you won't reach 225kpps. AFAIK this was Ciscos intention to publish this number. It basically says "you will have filled your link before you fill our router".

64B is the minimum ETHERNET frame size. That isn't true for PPP, HDLC, Frame relay, ATM, etc.

Quoted 3 lines

I'm pretty sure that your 37xx won't do any routing updates anymore at this point. And if you do _anything_ that forces the packets down the slow path from the routing engine, you're toast anyway.

Sure it can. Yeah, it'll be slow_er_, but not stopped. CEF/PXF doesn't require much CPU to switch a packet.

At process switching, the router is rated to a few K pps. Process switching SUCKS.

I've seen a 7206/200 _attempt_ to move 150kpps @ 64B each. (misbehaving software and a misconfigured firewall...) The router was chuggin' right along -- discarding UDP traffic like a mad man. From the console, it was working fine. *grin* It's rather hard for telnet to compete (and even harder for ssh.)

All those machines are behind a 7401 now. And it doesn't even blink at such things. (That thing's worth every penny we paid for it.)

--Ricky

#23 lore raw

On Fri, 2003-09-05 at 18:43, Ricky Beam wrote:

Quoted 11 lines · Henning Schmiedehausen
Quoted 8 lines

225kpps * 64 Bytes (minimum packet len) = 13,7 MBytes / sec

100 MBit / 8 bit = 12,5 MBytes / sec

So, IMHO even with a small packet saturated 100 MBit link you won't reach 225kpps. AFAIK this was Ciscos intention to publish this number. It basically says "you will have filled your link before you fill our router".

64B is the minimum ETHERNET frame size. That isn't true for PPP, HDLC, Frame relay, ATM, etc.

We were talking 100 MBit Ethernet, weren't we? ;-)

Regards
Henning

#24 lore raw
Quoted 4 lines · Henning Schmiedehausen
Quoted 2 lines

64B is the minimum ETHERNET frame size. That isn't true for PPP, HDLC, Frame relay, ATM, etc.

We were talking 100 MBit Ethernet, weren't we? ;-)

Actually, we're talking about VoIP between remote offices. We already know this is not accomplished via fast ethernet WAN links. The WAN link is, in fact, a T1. T1's do not have minimum frame sizes like ethernet.

--Ricky

#25 lore raw

That is where the thread started. ;-) As we reached the max kpps on a C37xx point of discussion, we were already on 100 MBit Ethernet. ;-)

Let's end this thread.

Regards
Henning

Quoted 12 lines · Ricky Beam
Quoted 4 lines · Henning Schmiedehausen
Quoted 2 lines

64B is the minimum ETHERNET frame size. That isn't true for PPP, HDLC, Frame relay, ATM, etc.

We were talking 100 MBit Ethernet, weren't we? ;-)

Actually, we're talking about VoIP between remote offices. We already know this is not accomplished via fast ethernet WAN links. The WAN link is, in fact, a T1. T1's do not have minimum frame sizes like ethernet.

--Ricky

Signature · 8 lines
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
[email protected]        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)
#26 lore raw
In reply to Florian Weimer
Quoted 5 lines · Florian Weimer

The ISP can do several things to prioritize production traffic or drop malicious traffic. However, this isn't trivial and requires careful planning, and it's unlikely that anyone who is able to would want to do this for a T1 customer (typically, it requires "unusual" configuration of vital production routers with the fat pipes).

In the cisco world, all it takes is:
interface <foo>
fair-queue

That's it. That's all it takes on the ISPs part to provide a minimal amount of QoS. Cisco's documentation clearly lists how IOS handles the queuing. It's not full policy based traffic shaping with RSVP and all that crap, but it's enough for a many situations.

Larry's situation is highly complicated by "the internet". I've never known anyone to be happy with VoIP across the internet. It's hard enough to make it work properly within a single ISP's network _with_ their assistance. Across the wastelands, forget about it. (Why do you think GRIC built their own global network for moving VoIP traffic?)

--Ricky

#27 lore raw
Quoted 10 lines · Ricky Beam
Quoted 5 lines · Florian Weimer

The ISP can do several things to prioritize production traffic or drop malicious traffic. However, this isn't trivial and requires careful planning, and it's unlikely that anyone who is able to would want to do this for a T1 customer (typically, it requires "unusual" configuration of vital production routers with the fat pipes).

In the cisco world, all it takes is:
interface <foo>
fair-queue

WFQ is already the default for low-bandwidth links (and it obviously can't work on high-end routers). 8-/

#28 lore raw
In reply to Alan Cox
Quoted 7 lines · Alan Cox
Quoted 6 lines · Andrea Arcangeli
Quoted 4 lines · Alan Cox
Quoted 1 line · Larry McVoy

I'm pretty convinced we can't solve the problem at our end. Maybe we can

For bursts of traffic you can't.

Quoted 7 lines
Quoted 2 lines

what's the difference of rejecting packets in software, or because the link can't handle them? Assume the guaranteed bandwidth is much lower

It doesn't work when you dont control incoming. As a simple extreme example if I pingflood you from a fast site then no amount of shaping your end of the link will help, it has to be shaped at the ISP end.

sure, that's why I said it won't work with synflood. I just doubt the ping/syn floods distributed denial of services are an high percentage of the traffic passing through on bkbits.net. I though it was legitimate traffic, and I assume bitkeeer is somehow efficient in handling the transfer, by using a single tcp connection for the whole transfer of the data, just like pserver/cvs-ssh do. For example if bitkeeper would open a new tcp connection for each file (similar to cvsps -p -g w/o the --cvs-direct option that I asked for), it would be much harder to shape that traffic. But I understood the traffic that hurts is all in established state for several seconds, so it should be technically possible to stop it to around 1kbyte/sec globally to give an huge margin to voip.

Andrea

#29 lore raw
Quoted 5 lines · Andrea Arcangeli
Quoted 3 lines

It doesn't work when you dont control incoming. As a simple extreme example if I pingflood you from a fast site then no amount of shaping your end of the link will help, it has to be shaped at the ISP end.

sure, that's why I said it won't work with synflood.

Someone syncs w/ bkbits every 19 seconds 24x7. We also run our web server
here. All traffic to from bitmover.com/bitkeeper.com/bkbits.net goes
through that T1 line.

You guys who are saying it can work are thinking (a) one connection of long duration (think about all the web hits on bkbits.net, those are all short and new TCP connections) and (b) that a little settling time is OK.

There is a reason that the phone networks don't work like IP networks. The bandwidth is allocated whether you use it or not and your phone works. Doing optimistic allocation and then backing off means that the phones don't work.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#30 lore raw
Quoted 8 lines · Larry McVoy
Quoted 5 lines · Andrea Arcangeli
Quoted 3 lines

It doesn't work when you dont control incoming. As a simple extreme example if I pingflood you from a fast site then no amount of shaping your end of the link will help, it has to be shaped at the ISP end.

sure, that's why I said it won't work with synflood.

Someone syncs w/ bkbits every 19 seconds 24x7. We also run our web server

1 syn every 19 seconds is nothing.

Quoted 2 lines

You guys who are saying it can work are thinking (a) one connection of long duration (think about all the web hits on bkbits.net, those are all

it doesn't need to be long duration, just longer than a syn or a ping. If it goes in established for a few packets is should be enough to throttle it just fine.

Andrea

#31 lore raw
Quoted 11 lines · Andrea Arcangeli
Quoted 8 lines · Larry McVoy
Quoted 5 lines · Andrea Arcangeli
Quoted 3 lines

It doesn't work when you dont control incoming. As a simple extreme example if I pingflood you from a fast site then no amount of shaping your end of the link will help, it has to be shaped at the ISP end.

sure, that's why I said it won't work with synflood.

Someone syncs w/ bkbits every 19 seconds 24x7. We also run our web server

1 syn every 19 seconds is nothing.

A sync != one connection. And that doesn't include the auto backup that we do to another server, doesn't include all the HTTP traffic, doesn't include our web site traffic.

Quoted 6 lines
Quoted 2 lines

You guys who are saying it can work are thinking (a) one connection of long duration (think about all the web hits on bkbits.net, those are all

it doesn't need to be long duration, just longer than a syn or a ping. If it goes in established for a few packets is should be enough to throttle it just fine.

You are welcome to *demonstrate* something that works but telling me that it works when we've tried what you said to try isn't very compelling. I know this doesn't work from both theory and practice.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#32 lore raw
Quoted 2 lines · Larry McVoy

works when we've tried what you said to try isn't very compelling. I know this doesn't work from both theory and practice.

Post your configure scripts so we can point you what you did wrong.

I already posted a perfectly working configuration that I depend totally on, and it never failed once to throttle the traffic. If you don't care to try it, at least post whatever you tried so far. Any other claim or email that doesn't include scripts will go to /dev/null.

Andrea

#33 lore raw
Quoted 5 lines · Andrea Arcangeli
Quoted 2 lines · Larry McVoy

works when we've tried what you said to try isn't very compelling. I know this doesn't work from both theory and practice.

Post your configure scripts so we can point you what you did wrong.

They are Cisco configuration, it won't do you much good. All the traffic goes in/out through a Cisco 2610, we have a full T1 and we clamped all TCP traffic at .75Mbit. Still didn't help even though we verified that it was indeed clamping down on the traffic.

I'm not sure why you are arguing this, if you have a fat pipe feeding into a small pipe and you are trying to throttle at far end of the small pipe isn't it obvious that you can't make that work? It's not the packets we send, it's the packets you send. And all the flow control stuff is per connection, not per pipe.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#34 lore raw
Quoted 8 lines · Larry McVoy
Quoted 5 lines · Andrea Arcangeli
Quoted 2 lines · Larry McVoy

works when we've tried what you said to try isn't very compelling. I know this doesn't work from both theory and practice.

Post your configure scripts so we can point you what you did wrong.

They are Cisco configuration, it won't do you much good. All the traffic

I don't trust anything but the sourcecode I can read, so please try again with linux.

Quoted 1 line

I'm not sure why you are arguing this, if you have a fat pipe feeding into

you never tried with linux, how can you claim you know it doesn't work in practice? The fact is that you never tried it, while we used it all the time.

Quoted 3 lines

a small pipe and you are trying to throttle at far end of the small pipe isn't it obvious that you can't make that work? It's not the packets we send, it's the packets you send. And all the flow control stuff is per

it's tcp, it's trivial to rate limit the packets we send, as far as you go into established somehow.

Andrea

#35 lore raw
Quoted 5 lines · Andrea Arcangeli
Quoted 1 line

I'm not sure why you are arguing this, if you have a fat pipe feeding into

you never tried with linux, how can you claim you know it doesn't work in practice? The fact is that you never tried it, while we used it all the time.

How about because any undergraduate can do the mathematical proof its not possible. Unless he controls the ISP end of the link random bursts of traffic, pingfloods, anything not respecting requests to slow down will lose voice traffic.

Since he doesn't appear to control the ISP end (either directly or via RSVP) he's lost.

#36 lore raw
Quoted 11 lines · Alan Cox
Quoted 5 lines · Andrea Arcangeli
Quoted 1 line

I'm not sure why you are arguing this, if you have a fat pipe feeding into

you never tried with linux, how can you claim you know it doesn't work in practice? The fact is that you never tried it, while we used it all the time.

How about because any undergraduate can do the mathematical proof its not possible. Unless he controls the ISP end of the link random bursts of traffic, pingfloods, anything not respecting requests to slow down will lose voice traffic.

Whoops, if I had read this I wouldn't have sent the other reply. What Alan said is what I've been trying to say.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#37 lore raw
In reply to Alan Cox
Quoted 11 lines · Alan Cox
Quoted 5 lines · Andrea Arcangeli
Quoted 1 line

I'm not sure why you are arguing this, if you have a fat pipe feeding into

you never tried with linux, how can you claim you know it doesn't work in practice? The fact is that you never tried it, while we used it all the time.

How about because any undergraduate can do the mathematical proof its not possible. Unless he controls the ISP end of the link random bursts of traffic, pingfloods, anything not respecting requests to slow down will lose voice traffic.

they are legitimate tcp connections, not udp or icmp. I'm not saying you can control pingfloods or udp floods or syn floods.

Andrea

#38 lore raw
Quoted 8 lines · Andrea Arcangeli
Quoted 4 lines · Alan Cox

How about because any undergraduate can do the mathematical proof its not possible. Unless he controls the ISP end of the link random bursts of traffic, pingfloods, anything not respecting requests to slow down will lose voice traffic.

they are legitimate tcp connections, not udp or icmp. I'm not saying you can control pingfloods or udp floods or syn floods.

I'll try this one more time and then I'm giving up because as far as I can tell you haven't tried this with a busy server, I think you said you did shaping on your home machine or something. Hardly the same thing.

How about a series of tiny HTTP requests? All 1.0, no connection reuse.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#39 lore raw
Quoted 7 lines · Larry McVoy
Quoted 6 lines · Andrea Arcangeli
Quoted 4 lines · Alan Cox

How about because any undergraduate can do the mathematical proof its not possible. Unless he controls the ISP end of the link random bursts of traffic, pingfloods, anything not respecting requests to slow down will lose voice traffic.

Quoted 6 lines
Quoted 2 lines

they are legitimate tcp connections, not udp or icmp. I'm not saying you can control pingfloods or udp floods or syn floods.

I'll try this one more time and then I'm giving up because as far as I can tell you haven't tried this with a busy server, I think you said you did shaping on your home machine or something. Hardly the same thing.

the only difference I believe is that the connections are originated by my end, but that only changes the syn packet that normally accounts for a non significant amount of the bandwidth. And while this is located in a house, this isn't what I would call an home user usage.

Quoted 1 line

How about a series of tiny HTTP requests? All 1.0, no connection reuse.

on the long run, the way I recommend you to act, is to have a script that turns the traffic shaping on on demand before/after using voip, so you won't hurt the bkbits.net userbase when you're not at the phone. So you will fix the unprofessional thing during conf calls completely and the community should be ok with reasonably short slowdowns while you're at the phone (if only voip would run under linux too you could automate it).

Just try for a test with my script changing it to give 1kbyte/sec both ways (not 64k/16k) to everything but voip. I think I already posted all the needed information you need to make it work. You will need to rework the iptables rules carefully though, to include all the traffic but voip.

Hope this helps you too (I couldn't live without it),

Andrea

#40 lore raw
Quoted 2 lines · Andrea Arcangeli

on the long run, the way I recommend you to act, is to have a script that turns the traffic shaping on on demand before/after using voip

That's a cool idea but unfortunately we are using the 3com NBX stuff which is based on Wind River and we don't have any way to know that a voice conversation is going on. I suppose we could build a way based on snooping the network though. Have to think about that.

By the way, I'd love to replace the 3com stuff with something based on Linux. I strongly suspect that the phones could do better than they are doing and we're getting the short end of the stick by going with some closed answer. But all the open answers don't come close to offering what 3com does.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#41 lore raw
Quoted 1 line · Larry McVoy

But all the open answers don't come close to offering what 3com does.

Have you looked at what Asterisk and Bayonne do? If so, is an important feature missing which you use?

Cheers,
-- Jamie

#42 lore raw
Quoted 5 lines · Jamie Lokier
Quoted 1 line · Larry McVoy

But all the open answers don't come close to offering what 3com does.

Have you looked at what Asterisk and Bayonne do? If so, is an important feature missing which you use?

Hunt groups. I want incoming sales and support calls to go to group of people. All the "virtual PBX" solutions do a lame hunt where they ring the first guy 3 times, then the next guy 3 times, etc. The 3com rings everyone at the same time. Believe it or not, if you guys call for support I want someone to pick up the phone *right now*. If you or our commercial customers are having nasty BK problems I don't want you to wait while the stupid phone system goes hunting around looking for someone who isn't out to lunch or whatever.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#43 lore raw
In reply to Andrea Arcangeli
Quoted 4 lines · Andrea Arcangeli

the only difference I believe is that the connections are originated by my end, but that only changes the syn packet that normally accounts for a non significant amount of the bandwidth. And while this is located in a house, this isn't what I would call an home user usage.

Each ACK that has caused previous delays generally opens up a 64K window so you get bursts of data incoming. A sequence of acks can cause the incoming pipe to fill enough to screw up voip very easily.

Window bending stuff does help and certainly the packeteer boxes use it to good effect. I've never tried that on Linux but even then Im dubious that just cutting the routes down to an 8K window would help

#44 lore raw
Quoted 8 lines · Alan Cox
Quoted 4 lines · Andrea Arcangeli

the only difference I believe is that the connections are originated by my end, but that only changes the syn packet that normally accounts for a non significant amount of the bandwidth. And while this is located in a house, this isn't what I would call an home user usage.

Each ACK that has caused previous delays generally opens up a 64K window so you get bursts of data incoming. A sequence of acks can cause the

the congestion avoidance shouldn't allow what you say. It sends a few packets immediatly (cwnd starts > 1 recently), and that's why non-keepalive connections are bad, but after that the congestion window will remain low if we drop the packets.

Quoted 5 lines

incoming pipe to fill enough to screw up voip very easily.

Window bending stuff does help and certainly the packeteer boxes use it to good effect. I've never tried that on Linux but even then Im dubious that just cutting the routes down to an 8K window would help

The only object of my posts was to try to correct the misinformation that was spreading from Larry's posts that implied you had to buy a speparate connection to shape the bitkeeper connections doing clones of the tree. That's what I and everybody else understood. Now apparently he was wrong and the hurting factor aren't the bitkeeper clients cloning the tree but the webserver. So I'm glad to have reached my object.

Andrea

#45 lore raw
Quoted 6 lines · Andrea Arcangeli

The only object of my posts was to try to correct the misinformation that was spreading from Larry's posts that implied you had to buy a speparate connection to shape the bitkeeper connections doing clones of the tree. That's what I and everybody else understood. Now apparently he was wrong and the hurting factor aren't the bitkeeper clients cloning the tree but the webserver. So I'm glad to have reached my object.

I don't know what axe you are trying to grind but that's nonsense. Yes, I said when you start doing a clone the phones don't work but I never said "the only problem we have is that an out going clone causes problems". That's clearly not true, there are incoming clones, pushes, outgoing pulls, all sorts of HTTP traffic, etc.

The problem is that you heard clone and you assumed that clone was the ONLY source of traffic. That's your problem, not mine, so don't go accusing me of spreading misinformation because you weren't thinking or listening, please.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#46 lore raw
In reply to Andrea Arcangeli
Quoted 7 lines · Andrea Arcangeli
Quoted 2 lines

Each ACK that has caused previous delays generally opens up a 64K window so you get bursts of data incoming. A sequence of acks can cause the

the congestion avoidance shouldn't allow what you say. It sends a few packets immediatly (cwnd starts > 1 recently), and that's why non-keepalive connections are bad, but after that the congestion window will remain low if we drop the packets.

You may trigger fast retransmit patterns. Thats why you have to bend the window.

#47 lore raw
Quoted 11 lines · Alan Cox
Quoted 7 lines · Andrea Arcangeli
Quoted 2 lines

Each ACK that has caused previous delays generally opens up a 64K window so you get bursts of data incoming. A sequence of acks can cause the

the congestion avoidance shouldn't allow what you say. It sends a few packets immediatly (cwnd starts > 1 recently), and that's why non-keepalive connections are bad, but after that the congestion window will remain low if we drop the packets.

You may trigger fast retransmit patterns. Thats why you have to bend the window.

fast restransmit are good to trigger, they don't arrive in a flood like if we had a huge cwnd, if we can send packets out of the network, they're right to be sent. During congestions the outgoing acks will be rejected too (it's limiting both ways).

there is no difference between artificial congestion generated by a shaper and a true congestion. If tcp is correct it has to slowdown immediatly when it notices congestion.

I also don't see anything specific to a busy http server non support keepalive in this ack-fast-retransmit matter.

Andrea

#48 lore raw
In reply to Alan Cox

Hi,

Quoted 4 lines · Alan Cox

How about because any undergraduate can do the mathematical proof its not possible. Unless he controls the ISP end of the link random bursts of traffic, pingfloods, anything not respecting requests to slow down will lose voice traffic.

At first Larry wasn't talking about incoming bursts: "We do VOIP phones and when you guys clone a repo our phones don't work".

bye, Roman

#49 lore raw
Quoted 11 lines · Roman Zippel

Hi,

Quoted 4 lines · Alan Cox

How about because any undergraduate can do the mathematical proof its not possible. Unless he controls the ISP end of the link random bursts of traffic, pingfloods, anything not respecting requests to slow down will lose voice traffic.

At first Larry wasn't talking about incoming bursts: "We do VOIP phones and when you guys clone a repo our phones don't work".

Hey, let me make something clear in case it isn't. This isn't your problem, you have every right to clone away as fast as you want. So I am definitely *not* asking you to ease up on bkbits.net. In fact, I'd be unhappy if you did, we get a fair amount of stress testing of new releases on bkbits.net. So have at it.

But getting back to the problem space, it's incoming traffic, outgoing traffic, all the web hits (Google crawls the web interface to bkbits for example, even though we've tried to turn that off through robots.txt people have linkages into specific csets and then google will follow those down and up), etc.

I'd love for shaping to be something that would help here but I agree with Alan and others who say that this isn't something we can control at our end.

The good news is that I'm 99% sure we have a deal for 2 T1 lines at $500/each and we're currently paying $800/month for one T1 so I view this as a slight increase in dollars for double the bandwidth. The ISP is willing to do the shaping at their end, where it will help, if that works that means that everyone wins, you guys get more bandwidth. If it doesn't work then you guys get a dedicated T1 line for bkbits which is more than what you have now.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#50 lore raw
Quoted 6 lines · Larry McVoy
Quoted 2 lines · Roman Zippel

At first Larry wasn't talking about incoming bursts: "We do VOIP phones and when you guys clone a repo our phones don't work".

Hey, let me make something clear in case it isn't. This isn't your problem, you have every right to clone away as fast as you want.

I forgot to add that once we have this sorted out we'll do two other things that you'll like:

a) give you BK URL's that don't change (the current URL's are unstable, they are based on revisions and revision numbers are unstable)
b) make every changeset (or range of changesets) be something you can grab as a regular diff -Nur style patch. So all the BK users can post to the kernel list and include a URL that you all can wget and there is the patch. No need to use BK at all unless you want to, the data is right there as a patch.

All I'm trying to do is to underscore the point that none of you should "be nice" and not beat up on bkbits.net. It's a service, we get at least some benefit from you using the service so bang on it all you want. We'll solve the bandwidth problems.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#51 lore raw
In reply to Andrea Arcangeli
Quoted 4 lines · Andrea Arcangeli
Quoted 1 line

They are Cisco configuration, it won't do you much good. All the traffic

I don't trust anything but the sourcecode I can read, so please try again with linux.

We did that as well. We ran wondershaper on bkbits.net AND capped the Cisco.

Quoted 5 lines
Quoted 1 line

I'm not sure why you are arguing this, if you have a fat pipe feeding into

you never tried with linux, how can you claim you know it doesn't work in practice? The fact is that you never tried it, while we used it all the time.

Not for VOIP with a busy server you don't.

Quoted 6 lines
Quoted 3 lines

a small pipe and you are trying to throttle at far end of the small pipe isn't it obvious that you can't make that work? It's not the packets we send, it's the packets you send. And all the flow control stuff is per

it's tcp, it's trivial to rate limit the packets we send, as far as you go into established somehow.

Let's see. We've tried Linux. It didn't work. We tried again w/ the Cisco. Didn't work. Tried with both. Didn't work. Stopped hacking and starting thinking and realized it's impossible to make it work at just one end, especially at the wrong end. Several people who I trust (like Alan for example) pointed it that the theory matches the practice, it can't work.

Yet you keep insisting it will work. Why? What is the theory that says you can keep the other end of the T1 line from being congested when you don't have control over that router? And that router has several 100Mbit links all of which can point at my 1.5Mbit link. Explain to me how anything you do can make that work.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#52 lore raw
Quoted 3 lines · Larry McVoy

Yet you keep insisting it will work. Why? What is the theory that says you can keep the other end of the T1 line from being congested when you don't have control over that router? And that router has several 100Mbit

it's absolutely trivial, your end only needs to drop 99% of your outgoing acks and their incoming packets for every connection but voip while you are at the phone, you won't kill the connections but everybody but your voip will work. the exponential backoff and sstrash on the other and will rate limit everything immediatly.

you will definitely control the other end.

Anyways, I was only trying to help, I know how it works, it works perfectly for me, and the reason everything makes perfect sense is that this directory exists in Alan's tree too:

	2.4.22ac1/net/sched/

All assuming you're not under attack, if that's the case you should contact your ISP since that's unfixable on your end.

Andrea

#53 lore raw
Quoted 10 lines · Andrea Arcangeli
Quoted 3 lines · Larry McVoy

Yet you keep insisting it will work. Why? What is the theory that says you can keep the other end of the T1 line from being congested when you don't have control over that router? And that router has several 100Mbit

it's absolutely trivial, your end only needs to drop 99% of your outgoing acks and their incoming packets for every connection but voip while you are at the phone, you won't kill the connections but everybody but your voip will work. the exponential backoff and sstrash on the other and will rate limit everything immediatly.

Let's work it out. We assume 99% means drop virtually everything:

Every 19 seconds on average, 24x7, a new HTTP connection.

Rate is not uniform throughout the day. Let's take a wild guess and say it is 10 times higher at peak times.

That's one connection every 1.9 seconds.

Let's assume you drop 99% of outgoing ACKs.

Then all connecting remote clients will retry their SYNs until they get a connection or a timeout. Default tcp_syn_retries (assuming Linux clients) is 5.

That's one SYN every 0.38 seconds. -> bad but not awful.

Plus existing connections. Let's pretend each connection take 100 seconds.

That's 100/1.9 or 52 concurrent connections, roughly.

Each of those will retry with an ACK if it has any pending ACK or data.

When you start using the phone, that's 100 ACKs total,
approximately, with exponential backoff. --> bad but not awful.

These calculations are horrendously inaccurate, but should be ok within a couple of orders of magnitude.

So, near-total annihilation of bkbits.net when Larry or any of his team are on the phone should work. You can either integrate the phone system with netfilter so it is automatic when a customer calls. Trivial ;) Or disable bkbits.net during Larry's working day. ;)

-- Jamie

#54 lore raw
Quoted 1 line · Jamie Lokier

Every 19 seconds on average, 24x7, a new HTTP connection.

Wrong. Every 19 seconds is a new BK connection. The HTTP connections are at least an order of magnitude more frequent. And increasing, more and more people are becoming aware that the source is there and you can point to it.

Quoted 1 line

That's one connection every 1.9 seconds.

Ha. Don't I wish. Peak connection rates are a lot higher than that.

Quoted 1 line

Trivial ;) Or disable bkbits.net during Larry's working day. ;)

It's not my working day or the other people who work locally, all our phones are on a 100Mbit switched net to the 3com before it goes out on POTS. It's the remote people. And half our company is currently remote. I could write a book on how well remote does and doesn't work.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#55 lore raw
In reply to Jamie Lokier
Quoted 2 lines · Jamie Lokier

So, near-total annihilation of bkbits.net when Larry or any of his team are on the phone should work. You can either integrate the phone

this is *exactly* my point.

Of course near-total annihilation of bkbits.net may not be acceptable to the community, but still it should work as a proof of concept.

Depending on the connect/sec of the http server (not bkbits, for the largest part of the conversation I couldn't know about the http server, Larry only mentioned the bkbits.net clone until recently), the "reservation" margin will have to change: the less connect/sec the smaller margin Larry will need to reserve, the more connect/sec the bigger marging will be necessary.

Andrea

#56 lore raw
Quoted 6 lines · Andrea Arcangeli

Depending on the connect/sec of the http server (not bkbits, for the largest part of the conversation I couldn't know about the http server, Larry only mentioned the bkbits.net clone until recently), the "reservation" margin will have to change: the less connect/sec the smaller margin Larry will need to reserve, the more connect/sec the bigger marging will be necessary.

Hi Andrea,

Above a certain connection rate, no amount of margin is enough. The connections are a SYN flood.

You need to take into account the peak rate, which varies from second to second, due to simple statistics plus the tendency of the net to make traffic burstier than it started.

Above a certain number of packets/sec, competing VoIP traffic degrades
- a little added latency is equivalent to dropping with VoIP. That translates to dropouts in the audio. Acceptable for hackers talking over a flakey line, but not corporate telephony.

At Larry's server, inbound connection rate resembles a low-level SYN flood, which is enough to poke holes in VoIP latency.

Of course the real problem is lack of end to end QoS between Larry's nodes. Two T1s is just a workaround :)

-- Jamie

#57 lore raw
Quoted 17 lines · Jamie Lokier
Quoted 6 lines · Andrea Arcangeli

Depending on the connect/sec of the http server (not bkbits, for the largest part of the conversation I couldn't know about the http server, Larry only mentioned the bkbits.net clone until recently), the "reservation" margin will have to change: the less connect/sec the smaller margin Larry will need to reserve, the more connect/sec the bigger marging will be necessary.

Hi Andrea,

Above a certain connection rate, no amount of margin is enough. The connections are a SYN flood.

You need to take into account the peak rate, which varies from second to second, due to simple statistics plus the tendency of the net to make traffic burstier than it started.

Quoted 7 lines

Above a certain number of packets/sec, competing VoIP traffic degrades
- a little added latency is equivalent to dropping with VoIP. That translates to dropouts in the audio. Acceptable for hackers talking over a flakey line, but not corporate telephony.

At Larry's server, inbound connection rate resembles a low-level SYN flood, which is enough to poke holes in VoIP latency.

Yes. if you check my very first email in this thread, you will see I said "it has to work, unless you're under syn-flood ;)".

this is obvious, I'm not arguing about that. From Larry's description of the problem it couldn't be a syn flood, but it was a bkbits.net clone thing or checkout or whatever non syn intensive.

now apparently bkbits.net has nothing to do with it, and it's all about the http server. Then of course the syn overhead may become noticeable but things have changed totally from the original description of the problem (the bkbits.net clone thing).

however keep in mind you will somehow throttle the number of syns too, unless every single syn arrives to the webserver from a different user (unlikely).

Andrea

#58 lore raw
Quoted 2 lines · Andrea Arcangeli

now apparently bkbits.net has nothing to do with it, and it's all about the http server.

BK _is_ an http server. The same daemon you talk to for clones is the HTTP server. That's one and the same machine.

Quoted 3 lines

however keep in mind you will somehow throttle the number of syns too, unless every single syn arrives to the webserver from a different user (unlikely).

That's exactly the situation that any busy server has. Which is why I kept saying "tell me how you made this work for a busy server". "Busy server" by definition in this context at least means a server that is getting lots of connection requests from lots of different users.

Why that isn't obvious to you I don't understand. This is bkbits.net. Yeah, it's not slashdot or anything but there are something like 400 branches of the Linux kernel on it and that's ignoring all other projects. It's the web server which provides insight into the Linux kernel source base, of course it is busy.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#59 lore raw
In reply to Andrea Arcangeli
Quoted 4 lines · Andrea Arcangeli
Quoted 1 line

They are Cisco configuration, it won't do you much good. All the traffic

I don't trust anything but the sourcecode I can read, so please try again with linux.

Andrea, you have no clue at all. Please stop this. You might be terrific at VM and linux kernel internals but you suck at network topology and serious internet traffic routing.

You want to try rate limiting a DDoS attack from the wrong side of a small pipe. You can't do this. Not with Linux, with BSD, with a C64 or a Cisco.

People have been trying this on the internet since 1995 and know what they're talking about. This is not a case of "I have no clue, so I didn't know that it was impossible and I did it". There are people out there that got Ph.D.'s for proving that it's impossible to do what you suggest and other people that got rich building devices that can do this (from the right end of the pipe).

Regards
Henning

Signature · 8 lines
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
[email protected]        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)
#60 lore raw
In reply to Andrea Arcangeli
Quoted 6 lines · Andrea Arcangeli
Quoted 3 lines

It doesn't work when you dont control incoming. As a simple extreme example if I pingflood you from a fast site then no amount of shaping your end of the link will help, it has to be shaped at the ISP end.

sure, that's why I said it won't work with synflood. I just doubt the ping/syn floods distributed denial of services are an high percentage of

The traffic on sites like www.microsoft.com, www.kernel.org or I'd say even www.bkbits.net isn't distinguishable from a DDoS Synflood on a regular day. A bazillion of different IP addresses from all over the net trying to establish a TCP connection at the same time. And the other sides are dog-slow modem/isdn/dsl connections which means high latency, low bandwith, many retransmits and still they're legal connections that you want to serve.

Forget it. You can't compare your homegrown ISDN/DSL line with the traffic of a serious internet service. If you try, you get burned.

Regards
Henning

Signature · 8 lines
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
[email protected]        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)
#61 lore raw
In reply to Alan Cox
Quoted 6 lines · Alan Cox
Quoted 2 lines

what's the difference of rejecting packets in software, or because the link can't handle them? Assume the guaranteed bandwidth is much lower

It doesn't work when you dont control incoming. As a simple extreme example if I pingflood you from a fast site then no amount of shaping your end of the link will help, it has to be shaped at the ISP end.

HTTP traffic is enough to simulate this, the connections are all small, short lived, and there are a lot of them.

Signature · 2 lines
---
Larry McVoy              lm at bitmover.com          http://www.bitmover.com/lm
#62 lore raw
Quoted 10 lines · Larry McVoy
Quoted 6 lines · Alan Cox
Quoted 2 lines

what's the difference of rejecting packets in software, or because the link can't handle them? Assume the guaranteed bandwidth is much lower

It doesn't work when you dont control incoming. As a simple extreme example if I pingflood you from a fast site then no amount of shaping your end of the link will help, it has to be shaped at the ISP end.

HTTP traffic is enough to simulate this, the connections are all small, short lived, and there are a lot of them.

it's much harder to throttle http, but it should work too. you may need bigger margin due the higher percentage of unthrottable packets like syns.

Andrea

#63 lore raw
In reply to Andrea Arcangeli
Quoted 8 lines · Andrea Arcangeli
Quoted 4 lines · Alan Cox
Quoted 1 line · Larry McVoy

I'm pretty convinced we can't solve the problem at our end. Maybe we can

For bursts of traffic you can't.

what's the difference of rejecting packets in software, or because the link can't handle them?

Nothing, packets are always rejected in software. :)

Quoted 10 lines

it works flawlessy for me and it's the same problem (I also use streaming services), and they're unusable until I turn the shaping on, I'm sure that if you use the script and you change 1kbyte/sec to everything but voip it'll work fine for you too, since basically everything else won't pass anymore, it will take ages to open an html page and all the bkbits.net users will hang, and the link will be idle 99% of the time, so voip will take it over as much as it can. I don't think it's a matter of "if it works or not", I think it's a matter of how much you're ok to lose in terms of global bandwith for all the other services but voip.

Yes, that should work for well behaved traffic. You have to leave more breathing room compared to your provider doing the traffic shaping for you, but it is also more flexible than one line for each VOIP and the rest.

Still, I don't like the idea of throwing away those packets that have made it through the bottleneck. :)

Jörn

Signature · 2 lines
The strong give up and move away, while the weak give up and stay.
-- unknown
#64 lore raw
In reply to Larry McVoy
Quoted 1 line · Larry McVoy

I'm pretty convinced we can't solve the problem at our end. Maybe we can

You're right. You can't no matter what all the "network experts" around here say. It's either placing a shaper at the ISP or getting a second line.

I still don't understand why you don't simply co-lo the bkbits.net box. There should be plenty of housing centers in the Bay Area within a 30 minute drive of your office. ;-)

Regards
Henning

Signature · 8 lines
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
[email protected]        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)
#65 lore raw
In reply to Andrea Arcangeli
Quoted 6 lines · Andrea Arcangeli

This is what I use normally to limit my brother downloads, and it works fine for me (though I don't often place calls through adsl myself, it's basically useless since people only uses cellphones here, and last time I chekced voip wasn't free for cellphones with my isp).

this is the script:

Mine is similar, though I use tc filters instead of firewalling rules (my machine is on 192.168.2.0/24, rest of the house is on 192.168.3.0/24) I'm using the imq device on a 2.2 kernel to have all traffic go through that device for shaping:

#!/bin/bash

# traffic shaping startup script
#
# (c) 2003 Pascal Schmidt <[email protected]>

. /etc/rc.d/rc.functions

PATH=/bin:/sbin:/usr/bin:/usr/sbin

case "$1" in
start)
echo -n "Setting up traffic shaper: "

# remove other qdisc if any
tc qdisc del dev imq root &> /dev/null

# use HTB as root
tc qdisc add dev imq root handle 1:0 htb default 3

# complete LAN
tc class add dev imq parent 1:0 classid 1:1 htb \
rate 10mbit burst 15k

# ISDN connection, split up into two "halves" tc class add dev imq parent 1:1 classid 1:2 htb \
rate 64kbit burst 15k
tc class add dev imq parent 1:2 classid 1:21 htb \
rate 40kbit ceil 64kbit burst 15k
tc class add dev imq parent 1:2 classid 1:22 htb \
rate 24kbit ceil 59kbit burst 15k

# rest of LAN bandwidth for internal traffic tc class add dev imq parent 1:1 classid 1:3 htb \
rate 9mbit ceil 10mbit burst 15k

# use SFQ on leaf classes
tc qdisc add dev imq parent 1:21 handle 21:0 sfq perturb 10 tc qdisc add dev imq parent 1:22 handle 22:0 sfq perturb 10 tc qdisc add dev imq parent 1:3 handle 3:0 sfq perturb 10

# LAN to LAN traffic goes to fast class
tc filter add dev imq protocol ip parent 1:0 prio 1 u32 \
match ip src 192.168.0.0/16 \
match ip dst 192.168.0.0/16 flowid 1:3

# rest of traffic to LAN #1 (must be from Internet) # goes to first ISDN class (half ISDN channel) tc filter add dev imq protocol ip parent 1:0 prio 1 u32 \
match ip dst 192.168.2.0/24 flowid 1:21

# rest of traffic to LAN #2 (must be from Internet) # goes to second ISDN class (half ISDN channel) tc filter add dev imq protocol ip parent 1:0 prio 1 u32 \
match ip dst 192.168.3.0/24 flowid 1:22

check_status

# bring up intermediate queue interface echo -n "Bringing up intermediate queue: " ifconfig imq up
check_status
;;
stop)

	# bring down intermediate queue interface
	echo -n "Taking down intermediate queue: "
	ifconfig imq down
	check_status
	# remove root qdisc
	echo -n "Disabling traffic shaper: "
	tc qdisc del dev imq root
	check_status
	;;
status)
	tc -d -s class show dev imq
	;;
restart)
	$0 stop
	$0 start
	;;
*)
	echo "Usage: $0 {start|stop|status|restart}"
	exit 1
esac
Signature · 2 lines
Ciao,
Pascal
#66 lore raw
Quoted 7 lines · Pascal Schmidt
Quoted 5 lines · Andrea Arcangeli

This is what I use normally to limit my brother downloads, and it works fine for me (though I don't often place calls through adsl myself, it's basically useless since people only uses cellphones here, and last time I chekced voip wasn't free for cellphones with my isp).

Quoted 6 lines
Quoted 1 line

this is the script:

Mine is similar, though I use tc filters instead of firewalling rules (my machine is on 192.168.2.0/24, rest of the house is on 192.168.3.0/24) I'm using the imq device on a 2.2 kernel to have all traffic go through that device for shaping:

the main reason I choosen to mark the packets through netfilter is that I don't find tc very intuitive (the most annoying thing is that it misses a flush command), so it'll be quicker to add future tweaks this way (like adding more ips or port/ip combinations into the "shaped" channel)

thanks,
Andrea

#67 lore raw
In reply to Pascal Schmidt
Quoted 3 lines · Pascal Schmidt

In a way, you're on the right end of the pipe because the system that does your traffic shaping is part of the general network, viewed from the machines behind the shaper.

Stop argueing. You can shape only outgoing packets, which means that the shaper must have a big uplink pipe and then shape that into a thin downlink pipe. In Larrys' case, the shaper must be on the ISP network before the T1.

Once the packets hit the T1 (or your ISDN link), you've already lost. No matter how much shaping, dropping, non-acking you're doing on your side of the link.

Regards
Henning

Signature · 8 lines
Dipl.-Inf. (Univ.) Henning P. Schmiedehausen          INTERMETA GmbH
[email protected]        +49 9131 50 654 0   http://www.intermeta.de/

Java, perl, Solaris, Linux, xSP Consulting, Web Services 
freelance consultant -- Jakarta Turbine Development  -- hero for hire

"Dominate!! Dominate!! Eat your young and aggregate! I have grotty silicon!" 
      -- AOL CD when played backwards  (User Friendly - 200-10-15)
#68 lore raw
In reply to Pascal Schmidt
Quoted 4 lines · Pascal Schmidt

All you have to do is drop the incoming packets if they exceed a certain bandwidth. That will stop the corresponding ack automatically since your TCP stack won't even see the packet.

You've cut away the part where I explained that you don't want to throw away the incoming packet. Yes, the stupid approach works, but it is still stupid.

Quoted 6 lines

I'm doing this on my ISDN line to limit the rest of the house to one third of the bandwidth even if they're all busy downloading tons of warez. I'm paying, I want the bandwidth when I need it. They can get it all when there's no traffic for my machine.

No problem with the HTB queueing discipline.

But latency just went boom. On your side, your packets will come out quickly because the queue is short. But on your ISP's side, there is a single queue for both your and the warez' traffic. Your packets will get thrown away just as much, as theirs.

Quoted 4 lines

Yes, latency is a problem if you want to saturate your bandwidth. It is easy to guarantee some bandwith for latency critical stuff - just don't give out that piece of bandwith to something else. Of course, then most of the time that piece is wasted...

This works if your latency requirements are moderate or the pipe is big. Assume 5ms and ISDN and you have a window of 400 Bytes roughly. Each time you happen to receive 400 Bytes of packets at the same time, you hear a pause. A single large packet is more than that. Oops!

For Larry's T1 line, the numbers naturally go up, but it still doesn't take a huge amount of packets.

Quoted 1 line

and it doesn't help with problems somewhere along the net.

Compared to ISDN or T1, the net usually has big pipes and people tend to care about low latency, so that problem doesn't even exist compared with the receiving end.

Jörn

Signature · 2 lines
Simplicity is prerequisite for reliability.
-- Edsger W. Dijkstra