Skip to contentSkip to conversation

AIM9 regression

DISCUSSION Christoph Lameter 7 participants 20 messages Sep 23, 2008 – Oct 31, 2008

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

Christoph Lameter

author
#0 lore raw

I just dont seem to be able to get 2.6.27 to behave in a speedy way network wise. Configured out various components (netfilter, etc etc) but I still keep getting these aim9 result against 2.6.22:

47 misc_rtns_1 448038.00 430118.00 -17920.00 -4.00% Auxiliary Loops/second
48 dir_rtns_1 2412587.41 2723000.00 310412.59 12.87% Directory
Operations/second
49 shell_rtns_1 364.30 345.80 -18.50 -5.08% Shell Scripts/second
50 shell_rtns_2 364.20 355.34 -8.86 -2.43% Shell Scripts/second
51 shell_rtns_3 363.30 353.60 -9.70 -2.67% Shell Scripts/second
52 series_1 6694290.00 6706690.00 12400.00 0.19% Series Evaluations/second
53 shared_memory 1042900.00 1080630.00 37730.00 3.62% Shared Memory
Operations/second

54 tcp_test     352035.00 278442.00  -73593.00 -20.91% TCP/IP Messages/second
55 udp_test     640940.00 585570.00  -55370.00 -8.64% UDP/IP DataGrams/second
56 fifo_test    772440.00 932330.00  159890.00 20.70% FIFO Messages/second
57 stream_pipe  1222870.00 1230140.00    7270.00  0.59% Stream Pipe

Messages/second
58 dgram_pipe 1143106.89 1152730.00 9623.11 0.84% DataGram Pipe
Messages/second
59 pipe_cpy 867850.00 1065430.00 197580.00 22.77% Pipe Messages/second

#1 lore raw
Quoted 17 lines · On Tue, 23 Sep 2008 13:14:27 -0500 Christoph Lameter <[email protected]>…

I just dont seem to be able to get 2.6.27 to behave in a speedy way network wise. Configured out various components (netfilter, etc etc) but I still keep getting these aim9 result against 2.6.22:

47 misc_rtns_1 448038.00 430118.00 -17920.00 -4.00% Auxiliary Loops/second
48 dir_rtns_1 2412587.41 2723000.00 310412.59 12.87% Directory
Operations/second
49 shell_rtns_1 364.30 345.80 -18.50 -5.08% Shell Scripts/second
50 shell_rtns_2 364.20 355.34 -8.86 -2.43% Shell Scripts/second
51 shell_rtns_3 363.30 353.60 -9.70 -2.67% Shell Scripts/second
52 series_1 6694290.00 6706690.00 12400.00 0.19% Series Evaluations/second
53 shared_memory 1042900.00 1080630.00 37730.00 3.62% Shared Memory
Operations/second

54 tcp_test     352035.00 278442.00  -73593.00 -20.91% TCP/IP Messages/second
55 udp_test     640940.00 585570.00  -55370.00 -8.64% UDP/IP DataGrams/second
56 fifo_test    772440.00 932330.00  159890.00 20.70% FIFO Messages/second
57 stream_pipe  1222870.00 1230140.00    7270.00  0.59% Stream Pipe
Quoted 8 lines

Messages/second
58 dgram_pipe 1143106.89 1152730.00 9623.11 0.84% DataGram Pipe
Messages/second
59 pipe_cpy 867850.00 1065430.00 197580.00 22.77% Pipe Messages/second
--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Hardware configuration please?

author
#2 lore raw
Quoted 1 line · Stephen Hemminger

Hardware configuration please?

Dual Processor 4 core 8G, Xeon X5460 @ 3.16GHz, 667Mhz memory.

author
#3 lore raw
Quoted 1 line · Christoph Lameter

Dual Processor 4 core 8G, Xeon X5460 @ 3.16GHz, 667Mhz memory.

So 8G Ram, 8 processors total.

#4 lore raw
In reply to Christoph Lameter
Quoted 5 lines · Christoph Lameter
Quoted 1 line · Stephen Hemminger

Hardware configuration please?

Dual Processor 4 core 8G, Xeon X5460 @ 3.16GHz, 667Mhz memory.

Network hardware configuration?

Or is the TCP test over loopback?

maintainer
#5 lore raw

From: Jeff Garzik <[email protected]>
Date: Tue, 23 Sep 2008 21:20:40 -0400

Quoted 9 lines
Quoted 4 lines · Christoph Lameter
Quoted 1 line · Stephen Hemminger

Hardware configuration please?

Dual Processor 4 core 8G, Xeon X5460 @ 3.16GHz, 667Mhz memory.

Network hardware configuration?

Or is the TCP test over loopback?

I'm pretty sure it's over loopback :-)

author
#6 lore raw
Quoted 6 lines · David Miller

From: Jeff Garzik <[email protected]>
Date: Tue, 23 Sep 2008 21:20:40 -0400

Quoted 3 lines
Quoted 2 lines · Christoph Lameter

Stephen Hemminger wrote:

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
Quoted 1 line

Hardware configuration please?

Dual Processor 4 core 8G, Xeon X5460 @ 3.16GHz, 667Mhz memory.

Network hardware configuration?

Or is the TCP test over loopback?

I'm pretty sure it's over loopback :-)

Correct.

#7 lore raw
In reply to Christoph Lameter
Quoted 3 lines · Christoph Lameter

I just dont seem to be able to get 2.6.27 to behave in a speedy way network wise. Configured out various components (netfilter, etc etc) but I still keep getting these aim9 result against 2.6.22:

Could you please compare this against something less ancient, like 2.6.26 perhaps?

Thanks,

Signature · 4 lines
Visit Openswan at http://www.openswan.org/
Email: Herbert Xu ~{PmV>HI~} <[email protected]>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
maintainer
#8 lore raw

From: Herbert Xu <[email protected]>
Date: Wed, 24 Sep 2008 13:12:37 +0800

Quoted 7 lines · Christoph Lameter
Quoted 3 lines

I just dont seem to be able to get 2.6.27 to behave in a speedy way network wise. Configured out various components (netfilter, etc etc) but I still keep getting these aim9 result against 2.6.22:

Could you please compare this against something less ancient, like 2.6.26 perhaps?

Herbert, this is part of the tbench regression issues. Christoph took tbench from 2.6.22 until 2.6.27 and at basically every release tbench performance suffered noticably.

Now, he's taking the AIM9 benchmark networking numbers and showing that the same exact effect is seen there too.

It really behooves us to start doing something proactive about this blindingly obvious set of networking performance regressions through the past 6 or so releases instead of barking at the reporters saying things like "try this, try that, what's your config" etc.

:-)

#9 lore raw
Quoted 18 lines · On Tue, 23 Sep 2008 22:18:31 -0700 (PDT) David Miller <[email protected]>…

From: Herbert Xu <[email protected]>
Date: Wed, 24 Sep 2008 13:12:37 +0800

Quoted 7 lines · Christoph Lameter
Quoted 3 lines

I just dont seem to be able to get 2.6.27 to behave in a speedy way network wise. Configured out various components (netfilter, etc etc) but I still keep getting these aim9 result against 2.6.22:

Could you please compare this against something less ancient, like 2.6.26 perhaps?

Herbert, this is part of the tbench regression issues. Christoph took tbench from 2.6.22 until 2.6.27 and at basically every release tbench performance suffered noticably.

Now, he's taking the AIM9 benchmark networking numbers and showing that the same exact effect is seen there too.

Quoted 6 lines

It really behooves us to start doing something proactive about this blindingly obvious set of networking performance regressions through the past 6 or so releases instead of barking at the reporters saying things like "try this, try that, what's your config" etc.

:-)

These loopback benchmarks are often more sensitive to scheduler than networking changes.

author
#10 lore raw
Quoted 2 lines · Stephen Hemminger

These loopback benchmarks are often more sensitive to scheduler than networking changes.

Just ran a test with real NICs which show the same issues. I guess I need to get familiar with the network stack and start hacking on it. Sigh.

maintainer
#11 lore raw

From: Christoph Lameter <[email protected]>
Date: Wed, 24 Sep 2008 14:10:54 -0500

Quoted 7 lines
Quoted 2 lines · Stephen Hemminger

These loopback benchmarks are often more sensitive to scheduler than networking changes.

Just ran a test with real NICs which show the same issues. I guess I need to get familiar with the network stack and start hacking on it. Sigh.

I feel your pain, I think people are being very unreasonable in their analysis of your numbers, and for this I want to personally apologize.

It's clearly a networking issue in my eyes, and I wish my co-developers in networking would treat it as such instead of pushing the blame under the carpet and saying "scheduler", "SLUB", and all kinds of other bullshit without any facts on this specific case to back up such accusations.

#12 lore raw
Quoted 11 lines · On Wed, 24 Sep 2008 12:53:46 -0700 (PDT) David Miller <[email protected]>…

From: Christoph Lameter <[email protected]>
Date: Wed, 24 Sep 2008 14:10:54 -0500

Quoted 7 lines
Quoted 2 lines · Stephen Hemminger

These loopback benchmarks are often more sensitive to scheduler than networking changes.

Just ran a test with real NICs which show the same issues. I guess I need to get familiar with the network stack and start hacking on it. Sigh.

Quoted 7 lines

I feel your pain, I think people are being very unreasonable in their analysis of your numbers, and for this I want to personally apologize.

It's clearly a networking issue in my eyes, and I wish my co-developers in networking would treat it as such instead of pushing the blame under the carpet and saying "scheduler", "SLUB", and all kinds of other bullshit without any facts on this specific case to back up such accusations.

Is this a one time change, or has networking been getting slower over time?

maintainer
#13 lore raw

From: Stephen Hemminger <[email protected]>
Date: Wed, 24 Sep 2008 14:34:19 -0700

Quoted 17 lines
Quoted 14 lines · On Wed, 24 Sep 2008 12:53:46 -0700 (PDT) David Miller <[email protected]>…

From: Christoph Lameter <[email protected]>
Date: Wed, 24 Sep 2008 14:10:54 -0500

Quoted 7 lines
Quoted 2 lines · Stephen Hemminger

These loopback benchmarks are often more sensitive to scheduler than networking changes.

Just ran a test with real NICs which show the same issues. I guess I need to get familiar with the network stack and start hacking on it. Sigh.

I feel your pain, I think people are being very unreasonable in their analysis of your numbers, and for this I want to personally apologize.

Quoted 6 lines
Quoted 4 lines

It's clearly a networking issue in my eyes, and I wish my co-developers in networking would treat it as such instead of pushing the blame under the carpet and saying "scheduler", "SLUB", and all kinds of other bullshit without any facts on this specific case to back up such accusations.

Is this a one time change, or has networking been getting slower over time?

As per the tbench thread, it's been getting slower and slower, every single release, since as far back as people have tested, which seems to be 2.6.22 or thereabouts.

maintainer
#14 lore raw
In reply to Stephen Hemminger

From: Stephen Hemminger <[email protected]>
Date: Wed, 24 Sep 2008 08:16:03 -0700

Quoted 26 lines
Quoted 23 lines · On Tue, 23 Sep 2008 22:18:31 -0700 (PDT) David Miller <[email protected]>…

From: Herbert Xu <[email protected]>
Date: Wed, 24 Sep 2008 13:12:37 +0800

Quoted 7 lines · Christoph Lameter
Quoted 3 lines

I just dont seem to be able to get 2.6.27 to behave in a speedy way network wise. Configured out various components (netfilter, etc etc) but I still keep getting these aim9 result against 2.6.22:

Could you please compare this against something less ancient, like 2.6.26 perhaps?

Herbert, this is part of the tbench regression issues. Christoph took tbench from 2.6.22 until 2.6.27 and at basically every release tbench performance suffered noticably.

Now, he's taking the AIM9 benchmark networking numbers and showing that the same exact effect is seen there too.

It really behooves us to start doing something proactive about this blindingly obvious set of networking performance regressions through the past 6 or so releases instead of barking at the reporters saying things like "try this, try that, what's your config" etc.

Quoted 4 lines
Quoted 1 line

:-)

These loopback benchmarks are often more sensitive to scheduler than networking changes.

When it gets to %20, I strong start to doubt that, and this is exactly what's happening here.

What is it going to take to actually get someone to start profiling and analyzing this? :-)

#15 lore raw
Quoted 31 lines · David Miller

From: Stephen Hemminger <[email protected]>
Date: Wed, 24 Sep 2008 08:16:03 -0700

Quoted 28 lines
Quoted 24 lines · On Tue, 23 Sep 2008 22:18:31 -0700 (PDT) David Miller <[email protected]>…

From: Herbert Xu <[email protected]>
Date: Wed, 24 Sep 2008 13:12:37 +0800

Quoted 7 lines · Christoph Lameter
Quoted 3 lines

I just dont seem to be able to get 2.6.27 to behave in a speedy way network wise. Configured out various components (netfilter, etc etc) but I still keep getting these aim9 result against 2.6.22:

Could you please compare this against something less ancient, like 2.6.26 perhaps?

Herbert, this is part of the tbench regression issues. Christoph took tbench from 2.6.22 until 2.6.27 and at basically every release tbench performance suffered noticably.

Now, he's taking the AIM9 benchmark networking numbers and showing that the same exact effect is seen there too.

It really behooves us to start doing something proactive about this blindingly obvious set of networking performance regressions through the past 6 or so releases instead of barking at the reporters saying things like "try this, try that, what's your config" etc.

:-)

Quoted 8 lines
Quoted 2 lines

These loopback benchmarks are often more sensitive to scheduler than networking changes.

When it gets to %20, I strong start to doubt that, and this is exactly what's happening here.

What is it going to take to actually get someone to start profiling and analyzing this? :-)

...I was thinking earlier to answer "time?", but now once been there, it seems that more time is more appropriate... So far I haven't been able to find a way to create a reproducable serie of result numbers with aim9 tcp_test... it seems that the results vary within that (at least) 20% margin. Can Christoph actually get stable numbers out of it with 27-rcs (I haven't extensively tested .22 yet with long test durations but it seems that same problem occurs with it as well if short tests were used)?

...And what I've learned, I couldn't even finish a testrun with conntrack and default settings as ipv4 conntrack run out of entries :-).

Ow, almost forgot, I got some stable regression with lockdep though, I hope we've gotten some more power to its detection in return for the lost performance.

I got these top variations (in absolute numbers) between three consecutive runs of 1000 seconds aim9 tcp_test (3xoprof(abs,%), func, max-min, (max-min)/min), aim9+its data on tmpfs (with nodebug-nonf config):

266288 1.0221 420190 1.6457 614494 2.4039 vfs_read 348206 1.30763
233649 0.8968 317763 1.2446 508838 1.9906 vfs_write 275189 1.17779
228732 0.8779 496359 1.9440 324747 1.2704 dnotify_parent 267627 1.17005
671548 2.5776 592604 2.3210 445792 1.7440 inet_csk_get_port 225756 0.506416
392960 1.5083 362665 1.4204 491234 1.9217 netif_rx 128569 0.354512
121337 0.4657 208314 0.8159 249783 0.9772 do_sync_write 128446 1.05859
164951 0.6331 168276 0.6591 285451 1.1167 loopback_xmit 120500 0.73052
359659 1.3805 242133 0.9483 256785 1.0046 __tcp_select_window 117526 0.485378
876319 3.3636 762690 2.9872 772554 3.0223 tcp_sendmsg 113629 0.148985
266895 1.0244 199204 0.7802 176985 0.6924 tcp_established_options 89910 0.508009
689652 2.6471 647962 2.5378 608943 2.3822 dev_queue_xmit 80709 0.132539
206754 0.7936 265523 1.0400 284087 1.1114 __kmalloc_track_caller 77333 0.374034
544026 2.0882 496654 1.9452 571982 2.2376 tcp_recvmsg 75328 0.151671
600414 2.3046 525704 2.0590 567588 2.2204 ip_queue_xmit 74710 0.142114
131820 0.5060 59259 0.2321 121586 0.4757 getnstimeofday 72561 1.22447
67061 0.2574 132155 0.5176 137914 0.5395 rw_verify_area 70853 1.05655
129676 0.4977 60652 0.2376 98307 0.3846 sock_rfree 69024 1.13803
535701 2.0562 586248 2.2961 517563 2.0247 ip_finish_output 68685 0.132708
692187 2.6568 634962 2.4869 623888 2.4407 tcp_rcv_established 68299 0.109473
949233 3.6435 900741 3.5279 882256 3.4514 tcp_transmit_skb 66977 0.0759156

...like said, the variation in the aim9 results were ~20% at most.

Signature · 1 line
 i.
author
#16 lore raw
Quoted 7 lines · Ilpo Järvinen

...I was thinking earlier to answer "time?", but now once been there, it seems that more time is more appropriate... So far I haven't been able to find a way to create a reproducable serie of result numbers with aim9 tcp_test... it seems that the results vary within that (at least) 20% margin. Can Christoph actually get stable numbers out of it with 27-rcs (I haven't extensively tested .22 yet with long test durations but it seems that same problem occurs with it as well if short tests were used)?

Results fluctuate between 10 - 25%. The problem occurs with the short durations as well. If this is due to the additional code complexity in later kernels as we suspect then it may be an issue with cpu cache effectiveness.

Going to 64 bit binaries also yields a significant hit (as high as 30%) which also indicates caching issues.

Both 64 bit kernels and later kernels cause the variability of results to increase. 64 bit has double the effect than a 2.6.27 kernel. All indications of cpu caching issues. The L1 cache may become ineffective due to the increased cache footprint.

#17 lore raw
Quoted 9 lines · Christoph Lameter
Quoted 6 lines · Ilpo Järvinen

So far I haven't been able to
find a way to create a reproducable serie of result numbers with aim9 tcp_test... it seems that the results vary within that (at least) 20% margin. Can Christoph actually get stable numbers out of it with 27-rcs (I haven't extensively tested .22 yet with long test durations but it seems that same problem occurs with it as well if short tests were used)?

Quoted 3 lines

Results fluctuate between 10 - 25%. The problem occurs with the short durations as well. If this is due to the additional code complexity in later kernels as we suspect then it may be an issue with cpu cache effectiveness.

Hmm... I'll try to extract some very raw (and possible somewhat skewed) numbers out of that based on the profiles I have and some acme's tools.

Quoted 7 lines

Going to 64 bit binaries also yields a significant hit (as high as 30%) which also indicates caching issues.

Both 64 bit kernels and later kernels cause the variability of results to increase. 64 bit has double the effect than a 2.6.27 kernel. All indications of cpu caching issues. The L1 cache may become ineffective due to the increased cache footprint.

Ok. I was testing 64bit only... I'll probably try next some very short tests in vast numbers to see if the results converge after enough samples are taken, lets hope I don't need many days to get to such point... :-)

Signature · 1 line
 i.
#18 lore raw
In reply to Christoph Lameter

----- Original Message -----
From: "Christoph Lameter" <[email protected]>
To: "Ilpo Järvinen" <[email protected]>
Cc: "David Miller" <[email protected]>, [email protected], [email protected], [email protected]
Sent: Monday, September 29, 2008 4:54:11 PM GMT +01:00 Amsterdam / Berlin / Bern / Rome / Stockholm / Vienna
Subject: Re: AIM9 regression

Quoted 7 lines · Ilpo Järvinen

...I was thinking earlier to answer "time?", but now once been there, it seems that more time is more appropriate... So far I haven't been able to find a way to create a reproducable serie of result numbers with aim9 tcp_test... it seems that the results vary within that (at least) 20% margin. Can Christoph actually get stable numbers out of it with 27-rcs (I haven't extensively tested .22 yet with long test durations but it seems that same problem occurs with it as well if short tests were used)?

Results fluctuate between 10 - 25%. The problem occurs with the short durations as well. If this is due to the additional code complexity in later kernels as we suspect then it may be an issue with cpu cache effectiveness.

Going to 64 bit binaries also yields a significant hit (as high as 30%) which also indicates caching issues.

Both 64 bit kernels and later kernels cause the variability of results to increase. 64 bit has double the effect than a 2.6.27 kernel. All indications of cpu caching issues. The L1 cache may become ineffective due to the increased cache footprint.

-------------
One of the items showing up in the profile is the local side port allocation. Is the ephemeral port range getting full? If it is then the random port scan could take a long time to find the next free slot, especially now that source ports are randomized.

#19 lore raw
In reply to Christoph Lameter
Quoted 14 lines · Christoph Lameter
Quoted 7 lines · Ilpo Järvinen

...I was thinking earlier to answer "time?", but now once been there, it seems that more time is more appropriate... So far I haven't been able to find a way to create a reproducable serie of result numbers with aim9 tcp_test... it seems that the results vary within that (at least) 20% margin. Can Christoph actually get stable numbers out of it with 27-rcs (I haven't extensively tested .22 yet with long test durations but it seems that same problem occurs with it as well if short tests were used)?

Results fluctuate between 10 - 25%. The problem occurs with the short durations as well. If this is due to the additional code complexity in later kernels as we suspect then it may be an issue with cpu cache effectiveness.

Quoted 7 lines

Going to 64 bit binaries also yields a significant hit (as high as 30%) which also indicates caching issues.

Both 64 bit kernels and later kernels cause the variability of results to increase. 64 bit has double the effect than a 2.6.27 kernel. All indications of cpu caching issues. The L1 cache may become ineffective due to the increased cache footprint.

I experimented with it some and changed tcp_test to bind into supplied port instead of relying on the port allocator randomness, both server and client port were do like that. However, I had to turn tcp_tw_recycle on to get the test to actually return instead of -ESOMETHING. In addition I did sync & drop_caches before each run (I'm not sure if it did actually reduce variantion a bit or did I just imagine, I'd expect it to damp test harness caused artifacts if it did something) + sleep 20 before each 20 seconds test.

Port allocator could be benchmarked separately if so desired.

Here are my current numbers with 64-bit (nodebug & nonf):

.22 .28-rc2-gsmthg
GSO/TSO
off on
240700 232398 224194
241187 236722 227610
243940 237388 229472
244367 237469 229576
246134 238569 229680
246211 238680 229999
246400 238693 230262
248761 239076 230404
250934 239107 231404
251203 239152 231562
251572 239215 231912
254158 239863 232744
256407 239912 234017
257329 240022 -EINTR
259560 241352 -EINTR

http://www.cs.helsinki.fi/u/ijjarvin/aim9/res.png

TSO/GSO does modulos every so often but Dave is currently evaluating how to get rid of that, discussed here:
http://marc.info/?t=122411618000004&r=1&w=2
...Still some uncertainty where the remaining of Evgeniy's G&TSO off/on difference comes from.

2.6.27-rc7 has basically the same numbers as 2.6.28-rc2 though I accidently had there ftrace on so some extra nops were present.

Still some regression to attack, but there seems to considerably less than 20% when testing for net_random()'s output is removed.

Signature · 1 line
 i.