Timeline of IPoIB performance
Subsystems (2)
A message that is not in the archive (sent off-list or not yet received): 1128672413.13948.326.camel@localhost
A message that is not in the archive (sent off-list or not yet received): [email protected]
I'm adding netdev to this thread to see if they can help.
I'm seeing an IPoIB (IP over InfiniBand) netperf performance drop off, of up to 90 MB/s, when using kernels newer than 2.6.11. This doesn't appear to be an OpenIB IPoIB issue since the older in-kernel IB for 2.6.11 and a recent svn3687 snapshot both have the same performance (464 MB/s) with 2.6.11. I used the same kernel config file as a starting point for each of these kernel builds. Have there been any changes in Linux that would explain these results?
Here is the hardware setup and netperf results using 'netperf -f -M -c -C -H IPoIB_ADDRESS
All benchmarks are with RHEL4 x86_64 with HCA FW v4.7.0 dual EM64T 3.2 GHz PCIe IB HCA (memfull)
Kernel OpenIB msi_x netperf (MB/s)
2.6.14-rc3 in-kernel 1 374 2.6.13.2 svn3627 1 386 2.6.13.2 in-kernel 1 394 2.6.12.5-lustre in-kernel 1 399 2.6.12.5 in-kernel 1 402 2.6.12 in-kernel 1 406 2.6.12-rc6 in-kernel 1 407 2.6.12-rc5 in-kernel 1 405 <<<<< 2.6.12-rc4 in-kernel 1 470 <<<<< 2.6.12-rc3 in-kernel 1 466 2.6.12-rc2 in-kernel 1 469 2.6.12-rc1 in-kernel 1 466 2.6.11 in-kernel 1 464 2.6.11 svn3687 1 464 2.6.9-11.ELsmp svn3513 1 425 (Woody's results, 3.6Ghz EM64T)
Thanks,
- Matt
> 2.6.12-rc5 in-kernel 1 405 <<<<<
> 2.6.12-rc4 in-kernel 1 470 <<<<<I was optimistic when I saw this, because the changeover to git occurred with 2.6.12-rc2, so I thought I could use git bisect to track down exactly when the performance regression happened.
However, I haven't been able to get numbers that are stable enough to track this down. I have two systems, both HP DL145s with dual Opteron 875s and two-port mem-free PCI Express HCAs. I use MSI-X with the completion interrupt affinity set to CPU 0, and "taskset 2" to run netserver and netperf on CPU 1.
With default netperf parameters (just "-H otherguy") I get numbers between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5. The numbers are quite consistent between reboots, but if I reboot the system (even keeping the kernel identical), I see large performance changes. Presumably something is happening like the cache coloring of some hot data structures changing semi-randomly depending on the timing of various initialations.
Matt, how stable are your numbers?
- R.
Hi Roland,
Quoting r. Roland Dreier <[email protected]>:
Quoted 2 lines
However, I haven't been able to get numbers that are stable enough to track this down.
Disabling irq balancing sometimes helps me make the numbers more stable. Hope this helps,
Signature · 1 line
MST
Michael> Disabling irq balancing sometimes helps me make the Michael> numbers more stable.
I don't think that's an issue. I'm running on x86_64, which I don't think has the kernel irq balancer, and I'm not running a userspace IRQ balancer. I can see all the mthca interrupts going to the CPU I set through the smp_affinity file.
- R.
Quoted 13 lines · Roland Dreier
> 2.6.12-rc5 in-kernel 1 405 <<<<< > 2.6.12-rc4 in-kernel 1 470 <<<<<I was optimistic when I saw this, because the changeover to git occurred with 2.6.12-rc2, so I thought I could use git bisect to track down exactly when the performance regression happened.
However, I haven't been able to get numbers that are stable enough to track this down. I have two systems, both HP DL145s with dual Opteron 875s and two-port mem-free PCI Express HCAs. I use MSI-X with the completion interrupt affinity set to CPU 0, and "taskset 2" to run netserver and netperf on CPU 1.
Quoted 7 lines
With default netperf parameters (just "-H otherguy") I get numbers between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5. The numbers are quite consistent between reboots, but if I reboot the system (even keeping the kernel identical), I see large performance changes. Presumably something is happening like the cache coloring of some hot data structures changing semi-randomly depending on the timing of various initialations.
Which rev of netperf are you using, and areyou using the "confidence intervals" options (-i, -I)? for a long time, the linux-unique behaviour of returning the overhead bytes for SO_[SND|RCV]BUF and them being 2X what one gives in setsockopt() gave netperf some trouble - the socket buffer would double in size each iteration on a confidence interval run. Later netperf versions (late 2.3, and 2.4.X) have a kludge for this.
Slightly related to that, IIRC, the linux receiver code adjusts the advertised window as the connection goes along - how far the receive code opens the window may change from run to run - might that have an effect? If there is a way to get the linux receiver to simply advertise the full window from the beginning that might help minimize the number of variables.
Are there large changes in service demand along with the large performance changes?
FWIW, on later netperfs the -T option should allow you to specify the CPU on which netperf and/or netserver run, although I've had some trouble reliably detecting the right sched_setaffinity syntax among the releases.
rick jones
Rick> Which rev of netperf are you using, and areyou using the Rick> "confidence intervals" options (-i, -I)? for a long time, Rick> the linux-unique behaviour of returning the overhead bytes Rick> for SO_[SND|RCV]BUF and them being 2X what one gives in Rick> setsockopt() gave netperf some trouble - the socket buffer Rick> would double in size each iteration on a confidence interval Rick> run. Later netperf versions (late 2.3, and 2.4.X) have a Rick> kludge for this.
I believe it's netperf 2.2.
I'm not using any confidence interval stuff. However, the variation is not between single runs of netperf -- if I do 5 runs of netperf in a row, I get roughly the same number from each run. For example, I might see something like
TCP STREAM TEST to 192.168.145.2 : histogram
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.00 3869.82
and then
TCP STREAM TEST to 192.168.145.2 : histogram
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.00 3862.41
for two successive runs. However, if I reboot the system into the same kernel (ie everything set up exactly the same), the same invocation of netperf might give
TCP STREAM TEST to 192.168.145.2 : histogram
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
87380 16384 16384 10.00 4389.20
Rick> Are there large changes in service demand along with the Rick> large performance changes?
Not sure. How do I have netperf report service demand?
- R.
Quoted 10 lines · Roland Dreier
Rick> Which rev of netperf are you using, and areyou using the Rick> "confidence intervals" options (-i, -I)? for a long time, Rick> the linux-unique behaviour of returning the overhead bytes Rick> for SO_[SND|RCV]BUF and them being 2X what one gives in Rick>
setsockopt()gave netperf some trouble - the socket buffer Rick> would double in size each iteration on a confidence interval Rick> run. Later netperf versions (late 2.3, and 2.4.X) have a Rick> kludge for this.I believe it's netperf 2.2.
That's rather old. I literally just put 2.4.1 out on ftp.cup.hp.com - probably better to use that if possible. Not that it will change the variability just that I like it when people are up-to-date on the versions :) If nothing else, the 2.4.X version(s) have a much improved (hopefully) manual in doc/
[If you are really maschochistic, the very first release of netperf 4.0.0 source has happened. I can make no guarantees as to its actually working at the moment though :) Netperf4 is going to be the stream for the multiple-connection, multiple system tests rather than the single-connection nature of netperf2]
Quoted 33 lines
I'm not using any confidence interval stuff. However, the variation is not between single runs of netperf -- if I do 5 runs of netperf in a row, I get roughly the same number from each run. For example, I might see something like
TCP STREAM TEST to 192.168.145.2 : histogram
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec87380 16384 16384 10.00 3869.82
and then
TCP STREAM TEST to 192.168.145.2 : histogram
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec87380 16384 16384 10.00 3862.41
for two successive runs. However, if I reboot the system into the same kernel (ie everything set up exactly the same), the same invocation of netperf might give
TCP STREAM TEST to 192.168.145.2 : histogram
Recv Send Send
Socket Socket Message Elapsed
Size Size Size Time Throughput
bytes bytes bytes secs. 10^6bits/sec
Quoted 6 lines
87380 16384 16384 10.00 4389.20
Rick> Are there large changes in service demand along with the Rick> large performance changes?
Not sure. How do I have netperf report service demand?
Ask for CPU utilization with -c (local) and -C (remote). The /proc/stat stuff used by Linux does not need calibration (IIRC) so you don't have to worry about that.
If cache effects are involved, you can make netperf "harder" or "easier" on the caches by altering the size of the send and/or recv buffer rings. By default they are one more than the socket buffer size divided by the send size, but you can make them larger or smaller with the -W option.
These days I use a 128K socket buffer and 32K send for the "canonical" (although
not default :) netperf TCP_STREAM test:
netperf -H remote -c -C -- -s 128K -S 128K -m 32K
In netperf-speak K == 1024, k == 1000, M == 2^20, m == 10^6, G == 2^40, g == 10^9... rick jones
Quoted 7 lines · Roland Dreier
> 2.6.12-rc5 in-kernel 1 405 <<<<< > 2.6.12-rc4 in-kernel 1 470 <<<<<I was optimistic when I saw this, because the changeover to git occurred with 2.6.12-rc2, so I thought I could use git bisect to track down exactly when the performance regression happened.
Quoted 5 lines
However, I haven't been able to get numbers that are stable enough to track this down. I have two systems, both HP DL145s with dual Opteron 875s and two-port mem-free PCI Express HCAs. I use MSI-X with the completion interrupt affinity set to CPU 0, and "taskset 2" to run netserver and netperf on CPU 1.
As you know, opteron boxes are NUMA. I think you want MSI-X interrupt bound to the same CPU that's connected to the IO. Is CPU 0 closer to IO? I would bind netperf to CPU0 and netserver to CPU 1 on each box respectively. Or just try all 4 combinations to see which combinations are CPU bound vs memory/IO bound.
Quoted 5 lines
With default netperf parameters (just "-H otherguy") I get numbers between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5. The numbers are quite consistent between reboots, but if I reboot the system (even keeping the kernel identical), I see large performance changes.
I gather you meant "tests" in the first phrase? (vs reboot).
Quoted 3 lines
Presumably something is happening like the cache coloring of some hot data structures changing semi-randomly depending on the timing of various initialations.
My guess is based on the same premise.
The mem-free card will be very sensitive to were it's control data is allocated. Is either box configured to interleave memory from both CPUs?
If it's interleaving, every other cacheline will be "local". Can you disable interleave and try different netperf/server bindings as suggested above?
hth,
grant
On Mon, Oct 10, 2005 at 02:26:52PM -0700, Grant Grundler wrote:
...
Quoted 1 line
If it's interleaving, every other cacheline will be "local".
ISTR AMD64 was page-interleaved but then got confused by documents describing "128-bit" 2-way interleave. I now realize the 128bit is refering to interleave between two "banks" of memory behind each memory controller. ie 2 * 128-bit provides in the 32-byte cacheline size that most x86 programs expect.
Anyway, I'm hoping that we'll see a consistent result if node interleave is turned off.
sorry for the confusion,
grant
Quoted 10 lines · Grant Grundler
On Mon, Oct 10, 2005 at 02:26:52PM -0700, Grant Grundler wrote:
...Quoted 1 line
If it's interleaving, every other cacheline will be "local".
ISTR AMD64 was page-interleaved but then got confused by documents describing "128-bit" 2-way interleave. I now realize the 128bit is refering to interleave between two "banks" of memory behind each memory controller. ie 2 * 128-bit provides in the 32-byte cacheline size that most x86 programs expect.
The cache line size on K7 and K8 is 64 bytes.
Quoted 2 lines
Anyway, I'm hoping that we'll see a consistent result if node interleave is turned off.
Yes usually a good idea.
-Andi
Quoted 21 lines · Roland Dreier
> 2.6.12-rc5 in-kernel 1 405 <<<<< > 2.6.12-rc4 in-kernel 1 470 <<<<<I was optimistic when I saw this, because the changeover to git occurred with 2.6.12-rc2, so I thought I could use git bisect to track down exactly when the performance regression happened.
However, I haven't been able to get numbers that are stable enough to track this down. I have two systems, both HP DL145s with dual Opteron 875s and two-port mem-free PCI Express HCAs. I use MSI-X with the completion interrupt affinity set to CPU 0, and "taskset 2" to run netserver and netperf on CPU 1.
With default netperf parameters (just "-H otherguy") I get numbers between ~490 MB/sec and ~550 MB/sec for 2.6.12-rc4 and 2.6.12-rc5. The numbers are quite consistent between reboots, but if I reboot the system (even keeping the kernel identical), I see large performance changes. Presumably something is happening like the cache coloring of some hot data structures changing semi-randomly depending on the timing of various initialations.
Quoted 1 line
Matt, how stable are your numbers?
Pretty consistent. Here are a few runs with 2.6.12-rc5 with reboots
in between each run. I'm using netperf-2.3pl1.
Run 1:
TCP STREAM TEST to 10.128.20.6
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send
Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. KBytes /s % T % T us/KB
us/KB
87380 16384 16384 10.00 410302.39 99.89 92.09 4.869
4.489
Run 2: (after another reboot)
TCP STREAM TEST to 10.128.20.6
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send
Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. KBytes /s % T % T us/KB
us/KB
87380 16384 16384 10.00 409510.33 99.89 91.59 4.879
4.473
Run 3: (after reboot)
TCP STREAM TEST to 10.128.20.6
Recv Send Send Utilization Service
Demand
Socket Socket Message Elapsed Send Recv Send
Recv
Size Size Size Time Throughput local remote local
remote
bytes bytes bytes secs. KBytes /s % T % T us/KB
us/KB
87380 16384 16384 10.00 404354.11 99.89 91.39 4.941
4.520
I see the same variance in netperf results if I don't reboot between runs.
- Matt
Quoted 1 line
Matt> Pretty consistent. Here are a few runs with 2.6.12-rc5
Matt> with reboots in between each run. I'm using netperf-2.3pl1.
That's interesting. I'm guessing you're using mem-ful HCAs?
Given that your results are more stable than mine, if you're up for it, you could install git, clone Linus's tree, and then do a git bisect between 2.6.12-rc4 and 2.6.12-rc5 to narrow down the regression to a single commit (if in fact that's possible).
- R.
Quoted 4 lines · Roland Dreier
Matt> Pretty consistent. Here are a few runs with 2.6.12-rc5
Matt> with reboots in between each run. I'm using netperf-2.3pl1.That's interesting. I'm guessing you're using mem-ful HCAs?
Yes, I'm using mem-full HCAs. I could try reflashing the firmware for
memfree if that's of interest.
Quoted 5 lines
Given that your results are more stable than mine, if you're up for it, you could install git, clone Linus's tree, and then do a git bisect between 2.6.12-rc4 and 2.6.12-rc5 to narrow down the regression to a single commit (if in fact that's possible).
I was hoping someone else would do this. :)
I'll start working on it tomorrow if no one else gets to it.
Thanks,
- Matt
Matt> Yes, I'm using mem-full HCAs. I could try reflashing the
Matt> firmware for memfree if that's of interest.
No, probably not. If I get a chance I'll do the opposite (flash mem-free -> mem-full, since my HCAs do have memory) and see if it makes my results stable.
Matt> I was hoping someone else would do this. :) I'll start Matt> working on it tomorrow if no one else gets to it.
I might get a chance to do it tonight... I'll post if I do.
- R.
Roland> I might get a chance to do it tonight... I'll post if I do.
I'm giving it a shot but I just can't reproduce this well on my systems. I do see a pretty big regression between 2.6.12-rc4 and 2.6.14-rc2, but 2.6.12-rc5 looks OK on my systems.
I reflashed to FW 4.7.0 (mem-ful) and built netperf 2.4.1.
With 2.6.12-rc4 I've seen runs as slow as:
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.145.2 (192.168.145.2) port 0 AF_INET
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % S % U us/KB us/KB
87380 16384 16384 10.00 553.71 37.46 -1.00 2.642 -1.000
and with 2.6.12-rc5 I've seen runs as fast as:
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.145.2 (192.168.145.2) port 0 AF_INET
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % S % U us/KB us/KB
87380 16384 16384 10.00 581.82 39.58 -1.00 2.657 -1.000
so not much difference there. With 2.6.14-rc2, the best of 10 runs was:
TCP STREAM TEST from 0.0.0.0 (0.0.0.0) port 0 AF_INET to 192.168.145.2 (192.168.145.2) port 0 AF_INET
Recv Send Send Utilization Service Demand
Socket Socket Message Elapsed Send Recv Send Recv
Size Size Size Time Throughput local remote local remote
bytes bytes bytes secs. MBytes /s % S % U us/KB us/KB
87380 16384 16384 10.01 497.00 39.71 -1.00 3.121 -1.000
so we've definitely lost something there.
Time to do some more bisecting...
- R.