gianfar: low gigabit throughput
Subsystems (2)
RFC 1/1 gianfar: low gigabit throughput
Hi all,
Down here few question regarding networking throughput, I would appreciate any thoughts or ideas.
On the MPC8315E-RDB board (CPU at 400MHz, CSB at 133 MHz) I'm observing relatively low TCP throughput using gianfar driver...
The maximum value I've seen with the current kernels is 142 Mb/s of TCP and 354 Mb/s of UDP (NAPI and interrupts coalescing enabled):
root@b1:~# netperf -l 10 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 10.0.1.1 #Cpu utilization 0.10 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec
206848 212992 32768 10.00 142.40
root@b1:~# netperf -l 10 -H 10.0.1.1 -t UDP_STREAM -- -m 32768 -s 157344 -S 157344 UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1 #Cpu utilization 100.00 Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec
212992 32768 10.00 13539 0 354.84 206848 10.00 13539 354.84
Is this normal?
netperf running in loopback gives me 329 Mb/s of TCP throughput:
root@b1:~# netperf -l 10 -H 127.0.0.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 127.0.0.1 #Cpu utilization 100.00 #Cpu utilization 100.00 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec
212992 212992 32768 10.00 329.60
May I consider this as a something that is close to the Linux' theoretical maximum for this setup? Or this isn't reliable test?
I can compare with teh MPC8377E-RDB (very similar board - exactly the same ethernet phy, the same drivers are used, i.e. everything is the same from the ethernet stand point), but running at 666 MHz, CSB at 333MHz:
|CPU MHz|BUS MHz|UDP Mb/s|TCP Mb/s|
------------------------------------------
MPC8377| 666| 333| 646| 264| MPC8315| 400| 133| 354| 142|
------------------------------------------
RATIO | 1.6| 2.5| 1.8| 1.8|
It seems that things are really dependant on the CPU/CSB speed.
I've tried to tune gianfar driver in various ways... and it gave some positive results with this patch:
drivers/net/gianfar.h +2 -2
@@ -123,8 +123,8 @@ extern const char gfar_driver_version[];#define GFAR_10_TIME 25600#define DEFAULT_TX_COALESCE 1 -#define DEFAULT_TXCOUNT 16 -#define DEFAULT_TXTIME 21 +#define DEFAULT_TXCOUNT 80 +#define DEFAULT_TXTIME 105#define DEFAULT_RXTIME 21
Basically this raises the tx interrupts coalescing threshold (raising it more didn't help, as well as didn't help raising rx thresholds). Now:
root@b1:~# netperf -l 3 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 10.0.1.1 #Cpu utilization 100.00 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec
206848 212992 32768 3.00 163.04
That is +21 Mb/s (14% up). Not fantastic, but good anyway.
As expected, the latency increased too:
Before the patch:
--- 10.0.1.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 18997ms rtt min/avg/max/mdev = 0.108/0.124/0.173/0.022 ms
After:
--- 10.0.1.1 ping statistics ---
22 packets transmitted, 22 received, 0% packet loss, time 20997ms rtt min/avg/max/mdev = 0.158/0.167/0.182/0.004 ms
34% up... heh. Should we sacrifice the latency in favour of throughput? Is 34% latency growth bad thing? What is worse, lose 21 Mb/s or 34% of latency? ;-)
Thanks in advance,
p.s. Btw, the patch above helps even better on the on the -rt kernels, since on the -rt kernels the throughput is near 100 Mb/s, with the patch the throughput is close to 140 Mb/s.
Signature · 3 lines
Anton Vorontsov email: [email protected] irc://irc.freenode.net/bd2
Quoted 7 lines · Anton Vorontsov
Hi all,
Down here few question regarding networking throughput, I would appreciate any thoughts or ideas.
On the MPC8315E-RDB board (CPU at 400MHz, CSB at 133 MHz) I'm observing relatively low TCP throughput using gianfar driver...
What is the "target" of the test - is it another of those boards, or something else?
Quoted 21 lines
The maximum value I've seen with the current kernels is 142 Mb/s of TCP and 354 Mb/s of UDP (NAPI and interrupts coalescing enabled):
root@b1:~# netperf -l 10 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 10.0.1.1 #Cpu utilization 0.10 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec206848 212992 32768 10.00 142.40
root@b1:~# netperf -l 10 -H 10.0.1.1 -t UDP_STREAM -- -m 32768 -s 157344 -S 157344 UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1 #Cpu utilization 100.00 Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec
Quoted 3 lines
212992 32768 10.00 13539 0 354.84 206848 10.00 13539 354.84
I have _got_ to make CPU utilization enabled by default one of these days :) At least for mechanisms which don't require calibration.
Quoted 1 line
Is this normal?
Does gianfar do TSO? If not, what happens when you tell UDP_STREAM to send 1472 byte messages to bypass IP fragmentation?
While stock netperf won't report what the socket buffer size becomes when you allow autotuning to rear its head, you can take the top of trunk and enable the "omni" tests (./configure --enable-omni) and those versions of *_STREAM etc can report what the socket buffer size was at the beginning and at the end of the test. You can let the stack autotune and see if anything changes there. You can do the same with stock netperf, just it will only report the initial socket buffer sizes...
Quoted 11 lines
netperf running in loopback gives me 329 Mb/s of TCP throughput:
root@b1:~# netperf -l 10 -H 127.0.0.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 127.0.0.1 #Cpu utilization 100.00 #Cpu utilization 100.00 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec
Quoted 5 lines
212992 212992 32768 10.00 329.60
May I consider this as a something that is close to the Linux' theoretical maximum for this setup? Or this isn't reliable test?
I'm always leery of using a loopback number. It excercises both send and receive at the same time, but without the driver. Also, lo tends to have a much larger MTU than a "standard" NIC and if that NIC doesn't to TSO and LRO that can be a big difference in the number of times up and down the stack per KB transferred.
Quoted 4 lines
I can compare with teh MPC8377E-RDB (very similar board - exactly the same ethernet phy, the same drivers are used, i.e. everything is the same from the ethernet stand point), but running at 666 MHz, CSB at 333MHz:
Quoted 8 lines
|CPU MHz|BUS MHz|UDP Mb/s|TCP Mb/s|
------------------------------------------MPC8377| 666| 333| 646| 264| MPC8315| 400| 133| 354| 142|------------------------------------------
RATIO | 1.6| 2.5| 1.8| 1.8|It seems that things are really dependant on the CPU/CSB speed.
What is the nature of the DMA stream between the two tests? I find it interesting that the TCP Mb/s went up by more than the CPU MHz and wonder how much the Bus MHz came into play there - perhaps there were more DMA's to setup or across a broader memory footprint for TCP than for UDP.
Quoted 9 lines
I've tried to tune gianfar driver in various ways... and it gave some positive results with this patch:
drivers/net/gianfar.h +0
-0
drivers/net/gianfar.h extern const char gfar_driver_version[]; · line 129
Quoted 9 lines
drivers/net/gianfar.h +2
-2@@ -123,8 +123,8 @@ extern const char gfar_driver_version[];#define GFAR_10_TIME 25600#define DEFAULT_TX_COALESCE 1 -#define DEFAULT_TXCOUNT 16 -#define DEFAULT_TXTIME 21 +#define DEFAULT_TXCOUNT 80 +#define DEFAULT_TXTIME 105#define DEFAULT_RXTIME 21
No ethtool coalescing tuning support for gianfar?-)
Quoted 27 lines
Basically this raises the tx interrupts coalescing threshold (raising it more didn't help, as well as didn't help raising rx thresholds). Now:
root@b1:~# netperf -l 3 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 10.0.1.1 #Cpu utilization 100.00 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec206848 212992 32768 3.00 163.04
That is +21 Mb/s (14% up). Not fantastic, but good anyway.
As expected, the latency increased too:
Before the patch:
--- 10.0.1.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 18997ms rtt min/avg/max/mdev = 0.108/0.124/0.173/0.022 msAfter:
Quoted 8 lines
--- 10.0.1.1 ping statistics ---
22 packets transmitted, 22 received, 0% packet loss, time 20997ms rtt min/avg/max/mdev = 0.158/0.167/0.182/0.004 ms34% up... heh. Should we sacrifice the latency in favour of throughput? Is 34% latency growth bad thing? What is worse, lose 21 Mb/s or 34% of latency? ;-)
Well, I'm not always fond of that sort of trade-off:
ftp://ftp.cup.hp.com/dist/networking/briefs/
there should be a nic latency vs tput writeup there.
Quoted 8 lines
Thanks in advance,
p.s. Btw, the patch above helps even better on the on the -rt kernels, since on the -rt kernels the throughput is near 100 Mb/s, with the patch the throughput is close to 140 Mb/s.
Quoted 17 lines
Quoted 8 lines
I've tried to tune gianfar driver in various ways... and it gave some positive results with this patch:
drivers/net/gianfar.h +0
-0drivers/net/gianfar.h extern const char gfar_driver_version[]; · line 127
Quoted 7 lines
drivers/net/gianfar.h +2
-2@@ -123,8 +123,8 @@ extern const char gfar_driver_version[];#define GFAR_10_TIME 25600#define DEFAULT_TX_COALESCE 1 -#define DEFAULT_TXCOUNT 16 -#define DEFAULT_TXTIME 21 +#define DEFAULT_TXCOUNT 80 +#define DEFAULT_TXTIME 105#define DEFAULT_RXTIME 21No ethtool coalescing tuning support for gianfar?-)
Yeah, there's coalescing tuning in gianfar.
Anton, those numbers aren't too surprising on a 400 MHz machine, I think. But I'd be happy to see any analysis on performance bottlenecks in the driver. And patches to fix those bottlenecks are even better! :)
Andy
Quoted 11 lines · Rick Jones
Quoted 7 lines · Anton Vorontsov
Hi all,
Down here few question regarding networking throughput, I would appreciate any thoughts or ideas.
On the MPC8315E-RDB board (CPU at 400MHz, CSB at 133 MHz) I'm observing relatively low TCP throughput using gianfar driver...
What is the "target" of the test - is it another of those boards, or something else?
I've tried in various ways (except for another of these boards, I don't have any), for example really fast machine with gbit ethernet, but most of the time I'm testing with MPC8315 and MPC8377 interconnected.
The other interesting thing is that when netserver is running on the MPC8315 (slow) target, and netperf on the MPC8377 (fast), the TCP and UDP throughput increases.
root@mpc837x_rdb:~# netperf -l 3 -H 10.0.1.2 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 10.0.1.2 #Cpu utilization 40.66 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec
212992 206848 32768 3.00 309.87
So the slow target can receive TCP packets with 300 Mb/s, but TCP packet generation (or transmitting) is slower.
Quoted 21 lines
Quoted 21 lines
The maximum value I've seen with the current kernels is 142 Mb/s of TCP and 354 Mb/s of UDP (NAPI and interrupts coalescing enabled):
root@b1:~# netperf -l 10 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 10.0.1.1 #Cpu utilization 0.10 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec206848 212992 32768 10.00 142.40
root@b1:~# netperf -l 10 -H 10.0.1.1 -t UDP_STREAM -- -m 32768 -s 157344 -S 157344 UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1 #Cpu utilization 100.00 Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec
Quoted 6 lines
Quoted 3 lines
212992 32768 10.00 13539 0 354.84 206848 10.00 13539 354.84I have _got_ to make CPU utilization enabled by default one of these days :) At least for mechanisms which don't require calibration.
Heh, I've skipped the calibration chapter in the netperf manual. :-D Should revert to it.
Quoted 3 lines
Quoted 1 line
Is this normal?
Does gianfar do TSO?
Afaik, it doesn't. The hardware can do header recognition/verification including checksums, and also could generate checksums for the TCP/IP.
But IP fragmentation and re-assembly on the software's shoulders.
Quoted 2 lines
If not, what happens when you tell
UDP_STREAMto send 1472 byte messages to bypass IP fragmentation?
A bit worse throughput:
root@b1:~# netperf -l 3 -H 10.0.1.1 -t UDP_STREAM -- -m 1472 -s 157344 -S 157344 UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1 #Cpu utilization 100.00 Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec
212992 1472 3.00 73377 0 287.86 206848 3.00 73377 287.86
And 32 * 1472:
root@b1:~# netperf -l 3 -H 10.0.1.1 -t UDP_STREAM -- -m 47104 -s 157344 -S 157344 UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1 #Cpu utilization 100.00 Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec
212992 47104 3.00 3124 0 392.13 206848 3.00 3124 392.13
So things becomes much better when the message size increases (I think the netperf then eating less cpu, and gives some precessing time to the kernel?).
The same for TCP packets with mlen of 1448.
Quoted 7 lines
While stock netperf won't report what the socket buffer size becomes when you allow autotuning to rear its head, you can take the top of trunk and enable the "omni" tests (./configure --enable-omni) and those versions of
*_STREAMetc can report what the socket buffer size was at the beginning and at the end of the test. You can let the stack autotune and see if anything changes there. You can do the same with stock netperf, just it will only report the initial socket buffer sizes...
Thanks, will try.
Quoted 14 lines
Quoted 14 lines
netperf running in loopback gives me 329 Mb/s of TCP throughput:
root@b1:~# netperf -l 10 -H 127.0.0.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 127.0.0.1 #Cpu utilization 100.00 #Cpu utilization 100.00 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec212992 212992 32768 10.00 329.60
Quoted 8 lines
Quoted 2 lines
May I consider this as a something that is close to the Linux' theoretical maximum for this setup? Or this isn't reliable test?
I'm always leery of using a loopback number. It excercises both send and receive at the same time, but without the driver. Also, lo tends to have a much larger MTU than a "standard" NIC and if that NIC doesn't to TSO and LRO that can be a big difference in the number of times up and down the stack per KB transferred.
I see. Will be cautious with it, too. ;-)
Quoted 11 lines
Quoted 11 lines
I can compare with teh MPC8377E-RDB (very similar board - exactly the same ethernet phy, the same drivers are used, i.e. everything is the same from the ethernet stand point), but running at 666 MHz, CSB at 333MHz:
|CPU MHz|BUS MHz|UDP Mb/s|TCP Mb/s|
------------------------------------------MPC8377| 666| 333| 646| 264| MPC8315| 400| 133| 354| 142|------------------------------------------
RATIO | 1.6| 2.5| 1.8| 1.8|
Quoted 7 lines
Quoted 1 line
It seems that things are really dependant on the CPU/CSB speed.
What is the nature of the DMA stream between the two tests? I find it interesting that the TCP Mb/s went up by more than the CPU MHz and wonder how much the Bus MHz came into play there - perhaps there were more DMA's to setup or across a broader memory footprint for TCP than for UDP.
The gianfar indeed does a lot of dma on the "buffer descriptors", so probably the bus speed matters a lot. And combination of CPU and Bus gives the final result.
Quoted 4 lines
Quoted 4 lines
I've tried to tune gianfar driver in various ways... and it gave some positive results with this patch:
Quoted 14 lines
drivers/net/gianfar.h extern const char gfar_driver_version[]; · line 127
Quoted 12 lines
drivers/net/gianfar.h +2
-2@@ -123,8 +123,8 @@ extern const char gfar_driver_version[];#define GFAR_10_TIME 25600#define DEFAULT_TX_COALESCE 1 -#define DEFAULT_TXCOUNT 16 -#define DEFAULT_TXTIME 21 +#define DEFAULT_TXCOUNT 80 +#define DEFAULT_TXTIME 105#define DEFAULT_RXTIME 21No ethtool coalescing tuning support for gianfar?-)
Heh. :-) I should have looked into gianfar_ethtool.c before editing anything. Yes, there is.
Quoted 36 lines
Quoted 27 lines
Basically this raises the tx interrupts coalescing threshold (raising it more didn't help, as well as didn't help raising rx thresholds). Now:
root@b1:~# netperf -l 3 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 10.0.1.1 #Cpu utilization 100.00 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec206848 212992 32768 3.00 163.04
That is +21 Mb/s (14% up). Not fantastic, but good anyway.
As expected, the latency increased too:
Before the patch:
--- 10.0.1.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 18997ms rtt min/avg/max/mdev = 0.108/0.124/0.173/0.022 msAfter:
Quoted 8 lines
--- 10.0.1.1 ping statistics ---
22 packets transmitted, 22 received, 0% packet loss, time 20997ms rtt min/avg/max/mdev = 0.158/0.167/0.182/0.004 ms34% up... heh. Should we sacrifice the latency in favour of throughput? Is 34% latency growth bad thing? What is worse, lose 21 Mb/s or 34% of latency? ;-)
Quoted 5 lines
Well, I'm not always fond of that sort of trade-off:
ftp://ftp.cup.hp.com/dist/networking/briefs/
there should be a nic latency vs tput writeup there.
Thanks!
Signature · 3 lines
Anton Vorontsov email: [email protected] irc://irc.freenode.net/bd2
Quoted 18 lines · Andy Fleming
Quoted 17 lines
Quoted 8 lines
I've tried to tune gianfar driver in various ways... and it gave some positive results with this patch:
drivers/net/gianfar.h +0
-0drivers/net/gianfar.h extern const char gfar_driver_version[]; · line 127
Quoted 7 lines
drivers/net/gianfar.h +2
-2@@ -123,8 +123,8 @@ extern const char gfar_driver_version[];#define GFAR_10_TIME 25600#define DEFAULT_TX_COALESCE 1 -#define DEFAULT_TXCOUNT 16 -#define DEFAULT_TXTIME 21 +#define DEFAULT_TXCOUNT 80 +#define DEFAULT_TXTIME 105#define DEFAULT_RXTIME 21No ethtool coalescing tuning support for gianfar?-)
Quoted 4 lines
Yeah, there's coalescing tuning in gianfar.
Anton, those numbers aren't too surprising on a 400 MHz machine, I think.
This is my thinking too, actually. I've seen the MPC885 board with CPU running at 133 MHz and CSB at 66MHz. It was using the completely different driver (fs_enet's FEC), and it failed to produce even 60 Mb/s (with a 100 Mb/s PHY). 50-55 Mb/s was quite normal for that board.
Which means, I think, that ~160 Mb/s is quite good for the 400/133 MHz board.
Quoted 2 lines
But I'd be happy to see any analysis on performance bottlenecks in the driver. And patches to fix those bottlenecks are even better! :)
The thing is that I don't see obvious bottlenecks, so I'm afraid we'll not able to do x2 boost or something... Just fine tuning the driver here and there... Interrupts coalescing was very obvious step to boost the troughtput, but there are its drawbacks...
Thanks,
Signature · 3 lines
Anton Vorontsov email: [email protected] irc://irc.freenode.net/bd2
Anton,
we've just built a digital GigEVision camera based on a MPC8343 running at 266/400 csb/core speed.
Transmission is done from a kernel module that allocates skb into which the image data is DMA'd by an external PCI master. As soon as the image data is complete all buffers are sent out via dev->hard_start_xmit ...
Bandwidth is currently 1.3MPixel @ 50Hz which give 65MBytes/sec (~520MBit/s). Of course it's UDP _without_ checksumming ....
Actually I have no sensor available that gives higher bandwidth ... but having a look at transmission time I'm sure the MPC8343 can easily go up to +800MBit.
Obviously your cpu time is consumed on a higher level.
Cheers,
André
Quoted 121 lines · Anton Vorontsov
Hi all,
Down here few question regarding networking throughput, I would appreciate any thoughts or ideas.
On the MPC8315E-RDB board (CPU at 400MHz, CSB at 133 MHz) I'm observing relatively low TCP throughput using gianfar driver...
The maximum value I've seen with the current kernels is 142 Mb/s of TCP and 354 Mb/s of UDP (NAPI and interrupts coalescing enabled):
root@b1:~# netperf -l 10 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 10.0.1.1 #Cpu utilization 0.10 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec206848 212992 32768 10.00 142.40
root@b1:~# netperf -l 10 -H 10.0.1.1 -t UDP_STREAM -- -m 32768 -s 157344 -S 157344 UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1 #Cpu utilization 100.00 Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec212992 32768 10.00 13539 0 354.84 206848 10.00 13539 354.84Is this normal?
netperf running in loopback gives me 329 Mb/s of TCP throughput:
root@b1:~# netperf -l 10 -H 127.0.0.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 127.0.0.1 #Cpu utilization 100.00 #Cpu utilization 100.00 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec212992 212992 32768 10.00 329.60
May I consider this as a something that is close to the Linux' theoretical maximum for this setup? Or this isn't reliable test?
I can compare with teh MPC8377E-RDB (very similar board - exactly the same ethernet phy, the same drivers are used, i.e. everything is the same from the ethernet stand point), but running at 666 MHz, CSB at 333MHz:
|CPU MHz|BUS MHz|UDP Mb/s|TCP Mb/s|
------------------------------------------MPC8377| 666| 333| 646| 264| MPC8315| 400| 133| 354| 142|------------------------------------------
RATIO | 1.6| 2.5| 1.8| 1.8|It seems that things are really dependant on the CPU/CSB speed.
I've tried to tune gianfar driver in various ways... and it gave some positive results with this patch:
drivers/net/gianfar.h +2
-2@@ -123,8 +123,8 @@ extern const char gfar_driver_version[];#define GFAR_10_TIME 25600#define DEFAULT_TX_COALESCE 1 -#define DEFAULT_TXCOUNT 16 -#define DEFAULT_TXTIME 21 +#define DEFAULT_TXCOUNT 80 +#define DEFAULT_TXTIME 105#define DEFAULT_RXTIME 21Basically this raises the tx interrupts coalescing threshold (raising it more didn't help, as well as didn't help raising rx thresholds). Now:
root@b1:~# netperf -l 3 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 10.0.1.1 #Cpu utilization 100.00 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec206848 212992 32768 3.00 163.04
That is +21 Mb/s (14% up). Not fantastic, but good anyway.
As expected, the latency increased too:
Before the patch:
--- 10.0.1.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 18997ms rtt min/avg/max/mdev = 0.108/0.124/0.173/0.022 msAfter:
--- 10.0.1.1 ping statistics ---
22 packets transmitted, 22 received, 0% packet loss, time 20997ms rtt min/avg/max/mdev = 0.158/0.167/0.182/0.004 ms34% up... heh. Should we sacrifice the latency in favour of throughput? Is 34% latency growth bad thing? What is worse, lose 21 Mb/s or 34% of latency? ;-)
Quoted 7 lines
Thanks in advance,
p.s. Btw, the patch above helps even better on the on the -rt kernels, since on the -rt kernels the throughput is near 100 Mb/s, with the patch the throughput is close to 140 Mb/s.
MATRIX VISION GmbH, Talstraße 16, DE-71570 Oppenweiler - Registergericht: Amtsgericht Stuttgart, HRB 271090 Geschäftsführer: Gerhard Thullner, Werner Armingeon, Uwe Furtner
Quoted 2 lines · André Schwarz
Anton,
Much thanks for the information!
Quoted 7 lines
we've just built a digital GigEVision camera based on a MPC8343 running at 266/400 csb/core speed.
Transmission is done from a kernel module that allocates skb into which the image data is DMA'd by an external PCI master.
As soon as the image data is complete all buffers are sent out viadev->hard_start_xmit...
Ah. So no userspace and packet generation expenses.. I see. This should be definitely faster than netperf. But generally this fits into the
picture: MPC8315 running at 266 CSB would probably give better UDP througput (300 Mb/s currenty).
Quoted 146 lines
Bandwidth is currently 1.3MPixel @ 50Hz which give 65MBytes/sec (~520MBit/s).
Of course it's UDP _without_ checksumming ....Actually I have no sensor available that gives higher bandwidth ... but having a look at transmission time I'm sure the MPC8343 can easily go up to +800MBit.
Obviously your cpu time is consumed on a higher level.
Cheers,
AndréQuoted 119 lines · Anton Vorontsov
Hi all,
Down here few question regarding networking throughput, I would appreciate any thoughts or ideas.
On the MPC8315E-RDB board (CPU at 400MHz, CSB at 133 MHz) I'm observing relatively low TCP throughput using gianfar driver...
The maximum value I've seen with the current kernels is 142 Mb/s of TCP and 354 Mb/s of UDP (NAPI and interrupts coalescing enabled):
root@b1:~# netperf -l 10 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 10.0.1.1 #Cpu utilization 0.10 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec206848 212992 32768 10.00 142.40
root@b1:~# netperf -l 10 -H 10.0.1.1 -t UDP_STREAM -- -m 32768 -s 157344 -S 157344 UDP UNIDIRECTIONAL SEND TEST to 10.0.1.1 #Cpu utilization 100.00 Socket Message Elapsed Messages Size Size Time Okay Errors Throughput bytes bytes secs # # 10^6bits/sec212992 32768 10.00 13539 0 354.84 206848 10.00 13539 354.84Is this normal?
netperf running in loopback gives me 329 Mb/s of TCP throughput:
root@b1:~# netperf -l 10 -H 127.0.0.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 127.0.0.1 #Cpu utilization 100.00 #Cpu utilization 100.00 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec212992 212992 32768 10.00 329.60
May I consider this as a something that is close to the Linux' theoretical maximum for this setup? Or this isn't reliable test?
I can compare with teh MPC8377E-RDB (very similar board - exactly the same ethernet phy, the same drivers are used, i.e. everything is the same from the ethernet stand point), but running at 666 MHz, CSB at 333MHz:
|CPU MHz|BUS MHz|UDP Mb/s|TCP Mb/s|
------------------------------------------MPC8377| 666| 333| 646| 264| MPC8315| 400| 133| 354| 142|------------------------------------------
RATIO | 1.6| 2.5| 1.8| 1.8|It seems that things are really dependant on the CPU/CSB speed.
I've tried to tune gianfar driver in various ways... and it gave some positive results with this patch:
drivers/net/gianfar.h +2
-2@@ -123,8 +123,8 @@ extern const char gfar_driver_version[];#define GFAR_10_TIME 25600#define DEFAULT_TX_COALESCE 1 -#define DEFAULT_TXCOUNT 16 -#define DEFAULT_TXTIME 21 +#define DEFAULT_TXCOUNT 80 +#define DEFAULT_TXTIME 105#define DEFAULT_RXTIME 21Basically this raises the tx interrupts coalescing threshold (raising it more didn't help, as well as didn't help raising rx thresholds). Now:
root@b1:~# netperf -l 3 -H 10.0.1.1 -t TCP_STREAM -- -m 32768 -s 157344 -S 157344 TCP STREAM TEST to 10.0.1.1 #Cpu utilization 100.00 Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec206848 212992 32768 3.00 163.04
That is +21 Mb/s (14% up). Not fantastic, but good anyway.
As expected, the latency increased too:
Before the patch:
--- 10.0.1.1 ping statistics ---
20 packets transmitted, 20 received, 0% packet loss, time 18997ms rtt min/avg/max/mdev = 0.108/0.124/0.173/0.022 msAfter:
--- 10.0.1.1 ping statistics ---
22 packets transmitted, 22 received, 0% packet loss, time 20997ms rtt min/avg/max/mdev = 0.158/0.167/0.182/0.004 ms34% up... heh. Should we sacrifice the latency in favour of throughput? Is 34% latency growth bad thing? What is worse, lose 21 Mb/s or 34% of latency? ;-)
Quoted 7 lines
Thanks in advance,
p.s. Btw, the patch above helps even better on the on the -rt kernels, since on the -rt kernels the throughput is near 100 Mb/s, with the patch the throughput is close to 140 Mb/s.
MATRIX VISION GmbH, Talstraße 16, DE-71570 Oppenweiler - Registergericht: Amtsgericht Stuttgart, HRB 271090 Geschäftsführer: Gerhard Thullner, Werner Armingeon, Uwe Furtner
Signature · 3 lines
Anton Vorontsov email: [email protected] irc://irc.freenode.net/bd2
Quoted 5 lines
Quoted 1 line
I have _got_ to make CPU utilization enabled by default one of these days :) At least for mechanisms which don't require calibration.
Heh, I've skipped the calibration chapter in the netperf manual. :-D Should revert to it.
Under linux, the CPU utilization mechanism in netperf does not require calibration, so you can add a -c (and -C if the remote is also linux) to the global command line options. Netperf will the report CPU util and calculate the "service demand" which will be the quantity of CPU consumed per unit of work.
Quoted 3 lines
So things becomes much better when the message size increases (I think the netperf then eating less cpu, and gives some precessing time to the kernel?).
Unless the compiler isn't doing a very good job, or perhaps if you've enabled histograms (./configure --enable-histogram) and set verbosity to 2 or more (not the case here), netperf itself shouldn't be consuming very much CPU at all up in user space. Now, as the size of the buffers passed to the transport increases, it does mean fewer system calls per KB transferred, which should indeed be more efficient.
Quoted 6 lines
Quoted 1 line
What is the nature of the DMA stream between the two tests? I find it interesting that the TCP Mb/s went up by more than the CPU MHz and wonder how much the Bus MHz came into play there - perhaps there were more DMA's to setup or across a broader memory footprint for TCP than for UDP.
The gianfar indeed does a lot of dma on the "buffer descriptors", so probably the bus speed matters a lot. And combination of CPU and Bus gives the final result.
Do you have any way to calculate bus utilization - logic analyzer, or perhaps some performance counters in the hardware?
If you have an HP-UX or Solaris system handy to act as a receiver, you might give that a try - it would be interesting to see the effect of their ACK avoidance heuristics on TCP throughput. One non-trivial difference I keep forgetting to mention is that TCP will have that returning ACK stream that UDP will not.
rick jones