: Test results with latest CCID3 patch set

As promised, here are some test results using the latest (2.6.25-backported) version of the test tree.

These are sanity checks and by no means "statistically significant". For that it would necessary to run more tests, take averages and calculate standard deviation / error bars.

Longer test runs (> 1/3 Min) would also be good.

Please therefore feel encouraged to run similar tests against this; on possibly different pieces of hardware.

All runs were for 20 seconds. The figures in square brackets are the theoretical values using the throughput equation from RFC 3448, 3.1, which ignores IP/DCCP header sizes.

I) First run: laptop to double-CPU server via NetEm router
----------------------------------------------------------
This run was done before the latest revision of patches today, but already including the patch set as per Friday last week. Setup for these tests:
* client: Pentium M 1.7 GHz laptop
* server: 2 x Pentium III 700 MHz SMP server
* router: 2.4 GHz PIV desktop with 2.6.24 NetEm

RTT 40ms, loss 1%

	* DCCPv6, s20:        3.3  Mbps   [3.3 Mbps]
	*         s%6:         658  kbps   [602 kbps]
	*         s2:           63  kbps   [ 75 kbps]
	* DCCPv4, s40:        3.5  Mbps   [3.4 Mbps]
	*         s%6:         638  kbps   [602 kbps]
	*         s2:           73  kbps   [ 75 kbps]

RTT 150ms, loss 1%

	* DCCPv6, s20:         826 kbps   [891 kbps]
	*         s%6:          120 kbps   [161 kbps]
	*         s2:            18 kbps   [ 20 kbps]
	* DCCPv4, s40:         714 kbps   [904 kbps]
	*         s%6:          230 kbps   [161 kbps]
	*         s2:            20 kbps   [ 20 kbps]

RTT 150ms, loss 10%

	* DCCPv6, s20:        130  kbps   [132 kbps]
	*         s%6:          23  kbps   [ 24 kbps]
	*         s2:           3.8 kbps   [  3 kbps]
	* DCCPv4, s40:        221  kbps   [134 kbps]
	*         s%6:          25  kbps   [ 24 kbps]
	*         s2:            3  kbps   [  3 kbps]

II) Second run: desktop-to-desktop via NetEm bridge
---------------------------------------------------
This run was done with the latest revision of patches today. Setup for these tests:
* client: 2.4 GHz PIV desktop
* server: 2.8 GHz Pentium dual-core desktop
* bridge: 2.4 GHz PIV desktop with 2.6.18-4 NetEm

RTT 40ms, loss 1%

	* DCCPv6, s20:        3.07 Mbps   [3.3 Mbps]
	*         s%6:          644 kbps   [602 kbps]
	*         s2:            63 kbps   [ 75 kbps]
	* DCCPv4, s40:        3.26 Mbps   [3.4 Mbps]
	*         s%6:          600 kbps   [602 kbps]
	*         s2:            67 kbps   [ 75 kbps]

RTT 150ms, loss 1%

	* DCCPv6, s20:         756 kbps   [891 kbps]
	*         s%6:          148 kbps   [161 kbps]
	*         s2:            16 kbps   [ 20 kbps]
	* DCCPv4, s40:         947 kbps   [904 kbps]
	*         s%6:          131 kbps   [161 kbps]
	*         s2:            23 kbps   [ 20 kbps]

RTT 150ms, loss 10%

	* DCCPv6, s20:         117 kbps   [132 kbps]
	*         s%6:           28 kbps   [ 24 kbps]
	*         s2:           3.6 kbps   [  3 kbps]
	* DCCPv4, s40:         187 kbps   [134 kbps]
	*         s%6:           50 kbps   [ 24 kbps]
	*         s2:             4 kbps   [  3 kbps]

I think this encouraging and asks for more challenging tests.

Hi all,

I am new of this mailing list and I am really interested in the measurements you are performing with DCCP.
Which tool are you using ? Are you using Iperf for such measurements ? In the past I used it, currently I'm working on D-ITG.
Have you ever heard about D-ITG ?
It is both a flexible traffic generator and a measurement platform which also supports DCCP.
More precisely, as regards transport protocols, it
supports TCP, UDP, SCTP, and DCCP. It is also possible
to generate ICMP traffic.
You can find more information here:
http://www.grid.unina.it/software/ITG

I am one of the authors of such platform and I have also performed some very preliminary tests with DCCP.

I would be very glad to have your opinion on that and I'm very interested in improving its features, also with specific regard to the support of transport protocols.

Best Regards.

AB

Gerrit Renker ha scritto:

Alessio Botta, PhD Student
Dipartimento di Informatica e Sistemistica
Università degli Studi di Napoli Federico II
Via Claudio 21 -- 80125 Napoli (Italy)
Phone: +390817683821 - Fax: +390817683816
Email: [email protected]
         [email protected]
WWW: http://wpage.unina.it/a.botta

| I am new of this mailing list and I am really interested in the | measurements you are performing with DCCP.
This was more of a regression test, as there had been recent changes in the test tree, to see that the kernel (not userspace) still performs in a predictable way.

| Which tool are you using ? Are you using Iperf for such measurements ? The setup is the one from
http://www.linux-foundation.org/en/Net:DCCP_Testing#Regression_testing
and, yes, it uses iperf.

| Have you ever heard about D-ITG ?

| You can find more information here:
| http://www.grid.unina.it/software/ITG
| 
| I am one of the authors of such platform and I have also
| performed some very preliminary tests with DCCP.
| 
| I would be very glad to have your opinion on that and I'm very interested
| in improving its features, also with specific regard to the support of
| transport protocols.
| 

It is a very nice tool with many features. I only ran simple tests with it (version 2.6), again only as basic sanity tests -- the throughput result was similar to the one tested with iperf.

I think that the tool has more to offer and can help improve/extend DCCP testing. Here is my list of points, hoping that the others will add theirs, too:

* would be good to have a standardised set of scripts, for comparison/benchmarking

* the built-in VoIP module only works for UDP -- is it possible to port this to DCCP?

* as per previous email, more complex traffic scenarios would be good, in particular
- switching on/off background traffic at times to observe TCP/flow-friendliness
- running multiple DCCP flows in parallel and at overlapping times

Gerrit

Em Tue, Dec 11, 2007 at 02:21:28PM +0000, Gerrit Renker escreveu:

Does this tool records results in a database keyed by kernel version/buildid for us to use it as a regression tool?

Something that would produce results around these lines:

"WARNING: test #23 counter #3 variance bigger than specified since the last kernel tested (git cset 55ed793afb4a8025d33a8e6a5f2f89d5ac4d8432)!"

- Arnaldo

comparison/benchmarking

port this to DCCP?

in particular

TCP/flow-friendliness

The scenario that I mostly use is limiting the bandwidth with a middlebox running TBF. However, all of the recent trees except 2.6.20final_dccp (2.6.20 patched with Ian's modifications) that I have tested fail to achieve acceptable transfer rates. Test setup is as follows, unless otherwise mentioned:

- Two Linux boxes are connected through another Linux box (middlebox) running netem. The boxes at the edge are identical machines with P4 2.4GHz and 512MB memory. Middlebox is a P4 2.6GHz with 512MB memory.
- TBF is used to limit the bandwidth. TBF buffer is set as 10000 bytes and the limit is 30000 bytes.
- 20ms constant delay is present in both sender->receiver and receiver->sender directions.
- iperf in bytestream mode is used in these tests, and the streaming duration is set as 60 seconds.
- DCCPv4 is used in the tests.

Below are the results of the two different trees:

Kernel: 2.6.20final_dccp (2.6.20 davem tree with Ian's patches - except for the best_packet_next patch. The patches applied are not the latest ones which are updated at 2nd of December, but the ones before the latest.) Results:
Bottleneck00Kbps, tx_qlen=5: 0.0-60.3 sec 6.86 MBytes 955 Kbits/sec

Bottleneck 00Kbps, tx_qlen=5: 0.0-60.1 sec 13.3 MBytes 1.86 Kbits/sec

BottleneckP00Kbps, tx_qlen=5: 0.0-60.0 sec 27.2 MBytes 3.80 Kbits/sec

Bottleneck000Kbps, limitP000bytes, tx_qlen=5: 0.0-60.0 sec 43.6 MBytes 6.09 Kbits/sec

Kernel: 2.6.24-rc4 (Gerrit's latest tree):
Results (Bottleneck bandwidth, tx_qlen: iperf server output):
Bottleneck00Kbps, tx_qlen=5: 0.0-168.2 sec 107 KBytes 5.21 Kbits/sec

Bottleneck00Kbps, tx_qlen=0: 0.0-147.8 sec 157 KBytes 8.71 Kbits/sec

Bottleneck 00Kbps, tx_qlen=5: 0.0-89.0 sec 138 KBytes 12.7 Kbits/sec

Bottleneck 00Kbps, tx_qlen=0: 0.0-103.9 sec 203 KBytes 16.0 Kbits/sec

BottleneckP00Kbps, tx_qlen=5: 0.0-66.6 sec 222 KBytes 27.3 Kbits/sec

BottleneckP00Kbps, tx_qlen=0: 0.0-100.2 sec 314 KBytes 25.7 Kbits/sec

Bottleneck000Kbps, tx_qlen=5: 0.0-60.5 sec 1.30 MBytes 181 Kbits/sec

Bottleneck000Kbps, tx_qlen=0: 0.0-65.3 sec 922 KBytes 116 Kbits/sec

While writing this mail, I noticed that Ian has updated his patch set for 2.6.20. I will use this set and repeat the tests this week, hopefully. Moreover, I also tested Ian's patches for other trees (2.6.22 and 2.6.24) but the results were not as good as 2.6.20final_dccp results, if I remember correctly. I can go over them again, if necessary.

Burak

| The scenario that I mostly use is limiting the bandwidth with a middlebox running TBF. However, all of the recent trees | except 2.6.20final_dccp (2.6.20 patched with Ian's modifications) that I have tested fail to achieve acceptable transfer rates. Thank you for the report, but the material as-is does not help us very much. What is missing is the link speed of the ethernet cards that lead to the middlebox, and have you monitored the loss rate p?
Using a TBF in this setting means slowing down the link speed by a factor of 10 .. 20, and unless you are using large FIFOs in addition to the TBF, the loss rate will very soon reach high values.

Therefore, can you please clarify what you mean by "acceptable transfer rates": maybe the scenario is not supposed to warrant any high transfer rates at all. Which would mean expecting something where not much can be expected - as said, without more detailed knowledge about how p reacts, these figures don't tell us very much.

Didn't you have a web page with further information?

Furthermore, I don't think that this is a question of "Ian's patches" or "Gerrit's patches": between 2.6.20 and 2.6.25-rc4 there is a huge number of changes, among these the conversion to the ktime_t interface.

| Test setup is as follows, unless otherwise mentioned:
|

| - Two Linux boxes are connected through another Linux box (middlebox) running netem. 
|   The boxes at the edge are identical machines with P4 2.4GHz and 512MB memory. Middlebox is a P4 2.6GHz with 512MB memory.
| - TBF is used to limit the bandwidth. TBF buffer is set as 10000 bytes and the limit is 30000 bytes.
| - 20ms constant delay is present in both sender->receiver and receiver->sender directions.
| - iperf in bytestream mode is used in these tests, and the streaming duration is set as 60 seconds.
| - DCCPv4 is used in the tests.
| 
| 
| Below are the results of the two different trees:
| 
| Kernel: 2.6.20final_dccp (2.6.20 davem tree with Ian's patches - except for the best_packet_next patch. The patches applied are not the latest ones which are updated at 2nd of December, but the ones before the latest.)
<snip>
| 
| While writing this mail, I noticed that Ian has updated his patch set for 2.6.20. I will use this set and repeat the tests this week, hopefully.i
| Moreover, I also tested Ian's patches for other trees (2.6.22 and 2.6.24) but the results were not as good as 2.6.20final_dccp results, if I remember correctly.
| I can go over them again, if necessary.
| 

Yes, please: the cleanest comparison would be to take a 2.6.24 tree, and compare the patch sets on the same basis. I almost expected that the results are not as good as the 2.6.20final_dccp -- it is an almost sure indication that the difference is not due to the patches, and that there are other factors at work. By carefully looking at these differences, we will be able to see clearer what is happening in the above.

Again thanks a lot for posting results, hope you will be back with further information soon,

Gerrit

middlebox running TBF. However, all of the recent trees

that I have tested fail to achieve acceptable transfer rates.

loss rate p?

factor of 10 .. 20, and unless you are using large

values.

Link speed is 100 Mbps, and I tested DCCP under various bottleneck bandwidths, like 1000, 2000, 5000 and 10000 Mbps. I will repeat the tests with dccp_probe enabled, and show the results in a website, as soon as I have time.

rates": maybe the scenario is not supposed to

something where not much can be expected - as said,

tell us very much.

By acceptable transfer rates I was referring to the rates achieved by 2.6.20final_dccp. But you are right, we cannot be sure of the goodness of the results by comparing two DCCP stacks, so I am giving TCP-Reno streaming results below under same conditions, which should be a solid benchmark:

Bottleneck00Kbps: 0.0-60.6 sec 6.91 MBytes 958
Kbits/sec
Bottleneck 00Kbps: 0.0-60.1 sec 13.7 MBytes 1.91 Mbits/sec

BottleneckP00Kbps: 0.0-60.1 sec 33.6 MBytes 4.69 Mbits/sec

Bottleneck000Kbps,limitP000bytes: 0.0-60.1 sec 65.4 MBytes 9.13 Mbits/sec

The average transfer rates in the rightmost column show that the configured bottleneck rates are achievable, hence I think the transfer rates that 2.6.20final_dccp reaches seem acceptable, whereas the rates of other trees are not.

I will have soon, hopefully.

and compare the patch sets on the same basis. I almost expected that the

indication that the difference is not due to the patches, and that

differences, we will be able to see clearer what is happening in the above.

I agree with you. 2 or 3 weeks ago I applied Ian's patches to

more recent trees (2.6.22 and 2.6.24) and the results were

not as good as 2.6.20.

I am planning to repeat the tests focusing on 2.6.24 and post the results - with dccp_probe figures in my website.

Burak

Gerrit Renker ha scritto:

I can provide you the necessary support to verify the scripts and also to start to
set up them. However, I need to know what you would like to generate.

I have seen on the web page reported above the kind of tests you are performing
with iperf, and I would like to provide you some some pieces of useful information
regarding D-ITG. This tool allows to set two random variables that control the characteristics of traffic you generate. One of these variables models the Inter
Departure Time (IDT) and the other one models the Size (PS) of the Packets. As of now,
we support 6 random variables that are exponential, gamma, normal, pareto, cauchy,
and poisson. Clearly also constant IDT and PS can be set.

From the web page reported above I have seen that with iperf you
generate CBR
traffic but I could not find the rate.

Moreover, in order to standardize the tests I need to know how you measure the
obtained bit-rate. D-ITG can log at both sender and receiver sides different information
for each sent and received packet respectively. Therefore after the tests you can
obtain different information by analyzing the log files (with ITGDec). Regarding the
bit-rate, you can have an average value related to the complete generation period as
well as related to smaller time intervals.

The VoIP traffic is produced by setting an appropriate IDT and PS which are basically
constant and printed out on standard output when a VoIP traffic flows is about to be
generated. Therefore, you can already perform VoIP tests with DCCP. For more information you can have a look at D-ITG manual:
http://www.grid.unina.it/software/ITG/codice/D-ITG2.6-manual.pdf

If you can not find all the necessary information, just ask me.

D-ITG supports multi-flow operation mode. In this case it reads the input information
from a script file instead of the standard input. It uses a thread for each requested flow
therefore the flows can have completely different characteristics in terms of kind of
traffic (IDT and PS models), duration, start time, and transport protocols.

This is already possible thanks to the script mode.

Alessio

Arnaldo Carvalho de Melo ha scritto:

No, it does not record the results in a database. This feature has to be added by using
some perl-like scripts.

Yes, we can work on building some scripts for this.

Alessio

Em Thu, Dec 13, 2007 at 12:21:20AM +0100, Alessio Botta escreveu:

That would be very nice of you and of great help to the DCCP effort. Being able to compare metrics automatically as we go evolving the codebase would be very, very interesting.

- Arnaldo

Burak

I have tried to replay your results but can not find anything wrong with the current test kernel. Please have a look at the results below, I repeated the same experiment on two different testbeds.

Incidentally, these results agree with the values you observed for TCP Reno. It seems that the setup is not right, I am happy to help debugging this but I don't think that there is a problem with the test kernel.

Can you please describe your setup - maybe something went wrong.

The first thing to check is whether you are really using the test tree - it is necessary to check out the dccp' subtree, if you are on the master' instead of the `dccp' subtree, then you are using a vanilla 2.6.24-rc5. A detailed HowTo is on
http://www.linux-foundation.org/en/Net:DCCP_Testing#Experimental_DCCP_source_tree
The difference can also easily be told in the Request/Response handshake, the CCIDs are now fully negotiated; the first value of the ConfirmL/R in the DCCP-Response says which CCIDs are used (but they are still nonetheless set via net.dccp.default.{rx,tx}_ccid).

With regard to Ian's patches: quite a few of these are earlier patches which are either already in the kernel or part of the test tree. The test tree in addition fixes several bugs/omissions in the loss detection / loss computation code, so in a way this means comparing apples with pears - it will not give you a clear result. What it can help with, and I am keen to investigate that, is to make sure that the test tree does not trade old bugs for new ones.

Similar results had been posted by Patrick Andrieux, who used TBF+delay and TBF+RED (posted this summer). I tried similar experiments on the test tree, and they produced sane results; but this was done earlier and not posted.

1) Two hosts via NetEm router
-----------------------------
All runs for 60 seconds from 1.7 Ghz Pentium M laptop onto 2x700Mhz Pentium-III Xeon SMP server, connected via 2.4GHz PIV acting as router with TBF script as below. The laptop had a e1000 interface, the PIV 2x100Mbps via crossover (one e100, one sc92301 cheapo RTL8139 clone), the server had an e100, also connected via crossover. All kernels were 2.6.24-rc5.

tx_qlen = 5:

 1000kbps -  951 kbps   (loss initially up to 5%, going down to ~0.8% after 10 seconds)
 2000kbps - 1.90 Mbps   (loss initially up to 3%, going down to ~0.6% after circa 5 seconds)
 5000kbps - 4.72 Mbps   (loss initially peaking at 1%, converging to ~0.15% after 10 seconds)
10000kbps - 7.73 Mbps   (loss initially at 1%, quickly going down to ~0.2%)

For comparison, TCP throughput with TBF bottleneck of 10Mbps was 7.95 Mbps.

tx_qlen = 0, same setup:

 1000kbps -  950 kbps   (loss initially up to ~6%, then going down to ~0.8% after 10 seconds)
 2000kbps - 1.91 Mbps   (similar loss pattern)
 5000kbps - 4.75 Mbps   (loss initially up to 1%, then going down to ~0.3% after 2 seconds)
10000kbps - 8.90 Mbps   (loss initially up to 1%, quickly going down to ~0.1% after 1..2 seconds)

2) Two hosts via NetEm bridge
-----------------------------
Sender: PIV 2.4GHz with e100 NIC
Receiver: Pentium D 2.8 Ghz with 3com 3c905 (Typhoon)
Bridge: PIV 2.4GHz using e100 and 3c905, using NetEm based on 2.6.18-4

All tests likewise for 60 seconds, detailed analysis of p omitted (likely very similar to the above).

tx_qlen = 5:
1000kbps - 908 kbps
2000kbps - 1.93 Mbps
5000kbps - 4.72 Mbps
10000kbps - 8.68 Mbps

For comparison, TCP (cubic) throughput with TBF bottleneck of 10Mbps was 7.85 Mbps.

tx_qlen = 0, same setup:
1000kbps - 951 kbps
2000kbps - 1.88 Mbps
5000kbps - 4.75 Mbps
10000kbps - 9.15 Mbps

3) Script I used (with RTT@ms, delay=RTT/2):
-------------------------------------------
#!/bin/sh
#

# tc script for rate-control plus delay
#
# Usage: $0 <RTT in ms> <TBF-rate in kbps>
#
set -e

case "$1" in
start)

        RTT=${2:?}; rate=${3:?}; delay=$(($RTT / 2))
	echo "Using ${delay}ms one-way delay each and a TBF rate of ${rate} Kbps "
	set -x
        tc qdisc add dev eth0 root handle 1: netem delay ${delay}ms
        tc qdisc add dev eth1 root handle 1: netem delay ${delay}ms
	# rate:   rate of incoming tokens, in bytes
	# buffer: size of the bucket, in bytes
	# limit:  number of bytes that can be queued waiting for tokens to become available
        tc qdisc add dev eth0 parent 1:1 tbf rate ${rate}kbit buffer 10000 limit 30000
        tc qdisc add dev eth1 parent 1:1 tbf rate ${rate}kbit buffer 10000 limit 30000
        ;;

stop)
tc qdisc del dev eth0 root
tc qdisc del dev eth1 root
;;
restart) $0 stop; shift; $0 start $@
;;
show|stat*)
tc -s qdisc ls dev eth0
tc -s qdisc ls dev eth1
;;
*) echo "Usage: $0 {start|stop|restart|show} <RTT in ms> <TBF-rate in kbps>"
exit 1;;
esac

Gerrit,

I am cloning your tree with the following command:

git-clone git://eden-feed.erg.abdn.ac.uk/dccp_exp my_dccp

But I did not check out the DCCP sub-branch, by:

    git-checkout --track -b dccp origin/dccp

However, as I type this command above, git complains saying:

git checkout: branch dccp already exists

It this OK to say so?

--
Burak

----- Original Message ----

with the current test kernel.

on two different testbeds.

Reno. It seems that the

that there is a problem

- it is necessary to check

`dccp' subtree, then you are

http://www.linux-foundation.org/en/Net:DCCP_Testing#Experimental_DCCP_source_tr

handshake, the CCIDs are now

DCCP-Response says which CCIDs are

net.dccp.default.{rx,tx}_ccid).

<snip>

|     git-clone git://eden-feed.erg.abdn.ac.uk/dccp_exp my_dccp
| 
| 
| But I did not check out the DCCP sub-branch, by:
| 
|     git-checkout --track -b dccp origin/dccp
| 
| 
| However, as I type this command above, git complains saying:
| 
|     git checkout: branch dccp already exists
| 
| 
| It this OK to say so?

The `--track' command is only necessary to instruct git to always pull from the given remote destination.

Apparently the dccp subtree got cloned as well. If this succeeded, you can check out the sub branch via "git-checkout dccp" (no -b); check with "git-branch" on which branch you are on.

I mostly use the other method, i.e. for my work I pull the test tree after creating a branch master/dccp and then clone:
git checkout -b dccp master
git pull git://eden-feed.erg.abdn.ac.uk/dccp_exp dccp

Sorry in the previous reply I forgot to mention that there is a much easier way to doi all this - the entire test tree is available as a single patch, for most recent releases on
http://www.erg.abdn.ac.uk/users/gerrit/dccp/testing_dccp/test-tree/
This can be applied via "zcat test-tree_*.diff.gz| patch -p1 [--dry-run]"

Coming in very late in this thread I'd just make a couple of observations:
- I'm still using 2.6.20 for myself but only because for my PhD I need one baseline and if I were to upgrade I'd have to rerun every single test for my PhD. A daunting task. I would recommend anyone else use more modern code. In particular the 2.6.20 code often crashes when running at full speed and also is missing a whole lot of fixes/improvements
- I know for a while there were issues like Burak experienced but I have no idea if there still is. It might be as Gerrit says because he is not using latest code which has now been explained
- when I did have issues like Burak I tracked it down to the send credit side of things. I could never quite prove exactly what was going on though. If problems are still there that is where I would look first
- I don't have time to really help :-( Burak - if you want to help hack DCCP code as well that would be awesome.

Ian