Skip to contentSkip to conversation

Network performance - iperf

DISCUSSION Michal Simek 4 participants 11 messages Mar 29, 2010 – Mar 30, 2010

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi All,

I am doing several network benchmarks on Microblaze cpu with MMU.
I am seeing one issue which is weird and I would like know where the problem is.
I am using the same hw design and the same Linux kernel. I have done only change in memory size (in DTS).

32MB: 18.3Mb/s
64MB: 15.2Mb/s
128MB: 10.6Mb/s
256MB: 3.8Mb/s

There is huge difference between systems with 32MB and 256MB ram.

I am running iperf TCP tests with these commands.
On x86: iperf -c 192.168.0.105 -i 5 -t 50
On microblaze: iperf -s

I look at pte misses which are the same on all configurations which means that the number of do_page_fault exceptions is the same on all configurations.
I added some hooks to low level kernel code to be able to see number of tlb misses. There is big differences between number of misses on system with 256MB and 32MB. I measured two kernel settings. First column is kernel with asm optimized memcpy/memmove function and the second is without optimization. (Kernel with asm optimized lib functions is 30% faster than system without optimization)

32MB: 12703 13641
64MB: 1021750 655644
128MB: 1031644 531879
256MB: 1011322 430027

Most of them are data tlb misses. Microblaze MMU doesn't use any LRU mechanism to find TLB victim that's why we there is naive TLB replacement strategy based on incrementing counter. We using 2 tlbs for kernel itself which are not updated that's why we can use "only" 62 TLBs from 64.

I am using two LL_TEMAC driver which use dma and I observe the same results on both that's why I think that the problem is in kernel itself.

It could be connection with memory management or with cache behavior.

Have you ever met with this system behavior?
Do you know about tests which I can do?

I also done several tests to identify weak kernel places via Qemu and this is the most called functions.

Unknown label means functions outside kernel. Numbers are in %

TCP
31.47 - memcpy
15.00 - do_csum
11.93 - unknown
5.62 - __copy_tofrom_user
2.94 - memset
2.49 - default idle
1.66 - __invalidate_dcache_range
1.57 - __kmalloc
1.32 - skb_copy_bits
1.23 - __alloc_skb

UDP
51.86 - unknown
9.31 - default_idle
6.01 - __copy_tofrom_user
4.00 - do_csum
2.05 - schedule
1.92 - __muldi3
1.39 - update_curr
1.20 - __invalidate_dcache_range
1.12 - __enqueue_entity

I optimized copy_tofrom_user function to support word-copying. (Just cover aligned cases because the most copying is aligned.) Also uaccess unification was done.

Do you have any idea howto improve TCP/UDP performance in general? Or tests which can point me on weak places.

I am using microblaze-next branch. The same code is in linux-next tree.

Thanks,
Michal

Signature · 3 lines
Michal Simek, Ing. (M.Eng)
PetaLogix - Linux Solutions for a Reconfigurable World
w: www.petalogix.com p: +61-7-30090663,+42-0-721842854 f: +61-7-30090663
#1 lore raw
Quoted 2 lines · Michal Simek

Do you have any idea howto improve TCP/UDP performance in general? Or tests which can point me on weak places.

Could you post "netstat -s" on your receiver, after fresh boot and your iperf session, for 32 MB and 256 MB ram case ?

#2 lore raw
Quoted 8 lines · Eric Dumazet
Quoted 2 lines · Michal Simek

Do you have any idea howto improve TCP/UDP performance in general? Or tests which can point me on weak places.

Could you post "netstat -s" on your receiver, after fresh boot and your iperf session, for 32 MB and 256 MB ram case ?

I am not sure if is helpful but look below.

Thanks,
Michal

~ # ./netstat -s
Ip:
0 total packets received
0 forwarded
0 incoming packets discarded
0 incoming packets delivered
0 requests sent out
Icmp:
0 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
Tcp:
0 active connections openings
0 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
0 segments received
0 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
0 packets received
0 packets to unknown port received.
0 packet receive errors
0 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
error parsing /proc/net/snmp: Success

Signature · 3 lines
Michal Simek, Ing. (M.Eng)
PetaLogix - Linux Solutions for a Reconfigurable World
w: www.petalogix.com p: +61-7-30090663,+42-0-721842854 f: +61-7-30090663
#3 lore raw
Quoted 12 lines · Michal Simek
Quoted 8 lines · Eric Dumazet
Quoted 2 lines · Michal Simek

Do you have any idea howto improve TCP/UDP performance in general? Or tests which can point me on weak places.

Could you post "netstat -s" on your receiver, after fresh boot and your iperf session, for 32 MB and 256 MB ram case ?

I am not sure if is helpful but look below.

Sorry I forget to c&p that second part. :-(

Look below.
Michal

32MB

~ # cat /proc/meminfo | head -n 1
MemTotal: 30024 kB
~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.0.10 port 5001 connected with 192.168.0.101 port 43577

[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-50.0 sec  78.0 MBytes  13.1 Mbits/sec

~ # ./netstat -s
Ip:
56596 total packets received
0 forwarded
0 incoming packets discarded
56596 incoming packets delivered
15752 requests sent out
Icmp:
0 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
Tcp:
0 active connections openings
1 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
56596 segments received
15752 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
0 packets received
0 packets to unknown port received.
0 packet receive errors
0 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
error parsing /proc/net/snmp: Success

256MB

~ # cat /proc/meminfo | head -n 1
MemTotal: 257212 kB
~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.0.10 port 5001 connected with 192.168.0.101 port 46069

[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-50.2 sec  19.5 MBytes  3.26 Mbits/sec

~ # ./netstat -s
Ip:
14163 total packets received
0 forwarded
0 incoming packets discarded
14163 incoming packets delivered
5209 requests sent out
Icmp:
0 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
Tcp:
0 active connections openings
1 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
14163 segments received
5209 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
0 packets received
0 packets to unknown port received.
0 packet receive errors
0 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
error parsing /proc/net/snmp: Success

Signature · 3 lines
Michal Simek, Ing. (M.Eng)
PetaLogix - Linux Solutions for a Reconfigurable World
w: www.petalogix.com p: +61-7-30090663,+42-0-721842854 f: +61-7-30090663
#8 lore raw
In reply to Michal Simek

I don't know how to set fixed socket buffer sizes in iperf, if you were running netperf though I would suggest fixing the socket buffer sizes with the test-specific -s (affects local) and -S (affects remote) options:

netperf -t TCP_STREAM -H <remote> -l 30 -- -s 32K -S 32K -m 32K

to test the hypothesis that the autotuning of the socket buffers/window size is allowing the windows to grow in the larger memory cases beyond what the TLB in your processor is comfortable with.

Particularly if you didn't see much degredation as RAM is increased on something like:

netperf -t TCP_RR -H <remote> -l 30 -- -r 1

which is a simple request/response test that will never try to have more than one packet in flight at a time, regardless of how large the window gets.

happy benchmarking,

rick jones
http://www.netperf.org/

#9 lore raw
Quoted 5 lines · Rick Jones

I don't know how to set fixed socket buffer sizes in iperf, if you were running netperf though I would suggest fixing the socket buffer sizes with the test-specific -s (affects local) and -S (affects remote) options:

netperf -t TCP_STREAM -H <remote> -l 30 -- -s 32K -S 32K -m 32K

to test the hypothesis that the autotuning of the socket buffers/window size is allowing the windows to grow in the larger memory cases beyond what the TLB in your processor is comfortable with.

BTW, by default, netperf will allocate a "ring" of send buffers - the number allocated will be one more than the socket buffer size divided by the send size - so in the example above, there will be two 32KB buffers allocated in netperf's send ring. A similar calculation may happen on the receive side.

That can be controlled via the global (before the "--") -W option.

-W send,recv Set the number of send,recv buffers

So, you might make the netperf command:

netperf -t TCP_STREAM -H <remote> -l 30 -W 1,1 -- -s 32K -S 32K -m 32K

happy benchmarking,

rick jones

#4 lore raw
In reply to Michal Simek
Quoted 7 lines · Michal Simek
Quoted 6 lines · Michal Simek
Quoted 5 lines · Eric Dumazet
Quoted 2 lines · Michal Simek

Do you have any idea howto improve TCP/UDP performance in general? Or tests which can point me on weak places.

Quoted 8 lines
Quoted 6 lines
Quoted 3 lines

Could you post "netstat -s" on your receiver, after fresh boot and your iperf session, for 32 MB and 256 MB ram case ?

I am not sure if is helpful but look below.

Sorry I forget to c&p that second part. :-(

Sorry, your netstat is not up2date.

If you cannot correct it to last version
[ net-tools 1.60 , netstat 1.42 ], please send

cat /proc/net/snmp
cat /proc/net/netstat

#10 lore raw
In reply to Michal Simek
Quoted 34 lines · Michal Simek

Hi All,

I am doing several network benchmarks on Microblaze cpu with MMU. I am seeing one issue which is weird and I would like know where the problem is.
I am using the same hw design and the same Linux kernel. I have done only change in memory size (in DTS).

32MB: 18.3Mb/s
64MB: 15.2Mb/s
128MB: 10.6Mb/s
256MB: 3.8Mb/s

There is huge difference between systems with 32MB and 256MB ram.

I am running iperf TCP tests with these commands.
On x86: iperf -c 192.168.0.105 -i 5 -t 50
On microblaze: iperf -s

I look at pte misses which are the same on all configurations which means that the number of do_page_fault exceptions is the same on all configurations.
I added some hooks to low level kernel code to be able to see number of tlb misses. There is big differences between number of misses on system with 256MB and 32MB. I measured two kernel settings. First column is kernel with asm optimized memcpy/memmove function and the second is without optimization. (Kernel with asm optimized lib functions is 30% faster than system without optimization)

32MB: 12703 13641
64MB: 1021750 655644
128MB: 1031644 531879
256MB: 1011322 430027

Quoted 6 lines

Most of them are data tlb misses. Microblaze MMU doesn't use any LRU mechanism to find TLB victim that's why we there is naive TLB replacement strategy based on incrementing counter. We using 2 tlbs for kernel itself which are not updated that's why we can use "only" 62 TLBs from 64.

This probably has nothing to do with tcp stack, but trashing tlb on some pathological cases (you have 62 entries, thats good for working size up to 248 Kbytes, all included (program stack, program static & dynamic data), given microblaze 4Kbytes page size.

You could try :

echo "4096 8192 32768" >/proc/sys/net/ipv4/tcp_rmem

to reduce memory footprint of iperf (or use iperf parameters)

Of course, I suppose kernel memory is 32 MB max, if you use only two tlbs (16 Mbytes each) for kernel...

#5 lore raw
In reply to Eric Dumazet
Quoted 10 lines · Eric Dumazet
Quoted 9 lines · Michal Simek
Quoted 8 lines · Michal Simek
Quoted 7 lines · Eric Dumazet
Quoted 2 lines · Michal Simek

Do you have any idea howto improve TCP/UDP performance in general? Or tests which can point me on weak places.

Could you post "netstat -s" on your receiver, after fresh boot and your iperf session, for 32 MB and 256 MB ram case ?

Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

I am not sure if is helpful but look below.

Sorry I forget to c&p that second part. :-(

Sorry, your netstat is not up2date.

I am afraid that is up2date.

Quoted 6 lines

If you cannot correct it to last version
[ net-tools 1.60 , netstat 1.42 ], please send

cat /proc/net/snmp
cat /proc/net/netstat

There is small buffer for parsing /proc/net/netstat.
There is necessary to extend buffer size because one line is greater than 1024 chars.

~ # head -n 1 /proc/net/netstat
TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps ArpFilter TW TWRecycled TWKilled PAWSPassive PAWSActive PAWSEstab DelayedACKs DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPPrequeued TCPDirectCopyFromBacklog TCPDirectCopyFromPrequeue TCPPrequeueDropped TCPHPHits TCPHPHitsToUser TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging TCPFACKReorder TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo TCPDSACKUndo TCPLossUndo TCPLoss TCPLostRetransmit TCPRenoFailures TCPSackFailures TCPLossFailures TCPFastRetrans TCPForwardRetrans TCPSlowStartRetrans TCPTimeouts TCPRenoRecoveryFail TCPSackRecoveryFail TCPSchedulerFailed TCPRcvCollapsed TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv TCPAbortOnSyn TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures TCPSACKDiscard TCPDSACKIgnoredOld TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected TCPSackShifted TCPSackMerged TCPSackShiftFallback TCPBacklogDrop TCPMinTTLDrop

Look at attached patch.
And updated results are below.

Thanks,
Michal

256M

~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.0.10 port 5001 connected with 192.168.0.101 port 33261

[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-50.2 sec  22.9 MBytes  3.83 Mbits/sec

~ # ./netstat -s
Ip:
16618 total packets received
0 forwarded
0 incoming packets discarded
16618 incoming packets delivered
6490 requests sent out
Icmp:
0 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
Tcp:
0 active connections openings
1 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
16618 segments received
6490 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
0 packets received
0 packets to unknown port received.
0 packet receive errors
0 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
TcpExt:
2233 packets pruned from receive queue because of socket buffer overrun
ArpFilter: 0
1 delayed acks sent
5519 packets header predicted
TCPPureAcks: 2
TCPHPAcks: 0
TCPRenoRecovery: 0
TCPSackRecovery: 0
TCPSACKReneging: 0
TCPFACKReorder: 0
TCPSACKReorder: 0
TCPRenoReorder: 0
TCPTSReorder: 0
TCPFullUndo: 0
TCPPartialUndo: 0
TCPDSACKUndo: 0
TCPLossUndo: 0
TCPLoss: 0
TCPLostRetransmit: 0
TCPRenoFailures: 0
TCPSackFailures: 0
TCPLossFailures: 0
TCPFastRetrans: 0
TCPForwardRetrans: 0
TCPSlowStartRetrans: 0
TCPTimeouts: 0
TCPRenoRecoveryFail: 0
TCPSackRecoveryFail: 0
TCPSchedulerFailed: 0
TCPRcvCollapsed: 207654
TCPDSACKOldSent: 0
TCPDSACKOfoSent: 0
TCPDSACKRecv: 0
TCPDSACKOfoRecv: 0
TCPAbortOnSyn: 0
TCPAbortOnData: 0
TCPAbortOnClose: 0
TCPAbortOnMemory: 0
TCPAbortOnTimeout: 0
TCPAbortOnLinger: 0
TCPAbortFailed: 0
TCPMemoryPressures: 0
TCPSACKDiscard: 0
TCPDSACKIgnoredOld: 0
TCPDSACKIgnoredNoUndo: 0
TCPSpuriousRTOs: 0
TCPMD5NotFound: 0
TCPMD5Unexpected: 0
TCPSackShifted: 0
TCPSackMerged: 0
TCPSackShiftFallback: 0
TCPBacklogDrop: 0
TCPMinTTLDrop: 0
IpExt:

    InNoRoutes: 0
    InTruncatedPkts: 0
    InMcastPkts: 0
    OutMcastPkts: 0
    InBcastPkts: 0
    OutBcastPkts: 0
    InOctets: 24915880
    OutOctets: 337488
    InMcastOctets: 0
    OutMcastOctets: 0
    InBcastOctets: 0
    OutBcastOctets: 0

~ # ./netstat --version
net-tools 1.60
netstat 1.42 (2001-04-15)
Fred Baumgarten, Alan Cox, Bernd Eckenfels, Phil Blundell, Tuan Hoang and others +NEW_ADDRT +RTF_IRTT +RTF_REJECT +FW_MASQUERADE -I18N
AF: (inet) +UNIX +INET -INET6 -IPX -AX25 -NETROM -X25 -ATALK -ECONET -ROSE
HW: +ETHER -ARC +SLIP +PPP -TUNNEL -TR -AX25 -NETROM -X25 -FR -ROSE -ASH -SIT -FDDI -HIPPI -HDLC/LAPB ~ # head -n 1 /proc/meminfo
MemTotal: 257108 kB

32MB

~ # head -n 1 /proc/meminfo
MemTotal: 29920 kB
~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.0.10 port 5001 connected with 192.168.0.101 port 50088
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-50.0 sec 109 MBytes 18.3 Mbits/sec
~ # ./netstat -s
Ip:
79040 total packets received
0 forwarded
0 incoming packets discarded
79040 incoming packets delivered
29655 requests sent out
Icmp:
0 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
Tcp:
0 active connections openings
1 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
79040 segments received
29655 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
0 packets received
0 packets to unknown port received.
0 packet receive errors
0 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
TcpExt:
9773 packets pruned from receive queue because of socket buffer overrun
ArpFilter: 0
1 delayed acks sent
101 packets directly queued to recvmsg prequeue.
558928 packets directly received from prequeue
33274 packets header predicted
378 packets header predicted and directly queued to user
TCPPureAcks: 2
TCPHPAcks: 0
TCPRenoRecovery: 0
TCPSackRecovery: 0
TCPSACKReneging: 0
TCPFACKReorder: 0
TCPSACKReorder: 0
TCPRenoReorder: 0
TCPTSReorder: 0
TCPFullUndo: 0
TCPPartialUndo: 0
TCPDSACKUndo: 0
TCPLossUndo: 0
TCPLoss: 0
TCPLostRetransmit: 0
TCPRenoFailures: 0
TCPSackFailures: 0
TCPLossFailures: 0
TCPFastRetrans: 0
TCPForwardRetrans: 0
TCPSlowStartRetrans: 0
TCPTimeouts: 0
TCPRenoRecoveryFail: 0
TCPSackRecoveryFail: 0
TCPSchedulerFailed: 0
TCPRcvCollapsed: 120195
TCPDSACKOldSent: 0
TCPDSACKOfoSent: 0
TCPDSACKRecv: 0
TCPDSACKOfoRecv: 0
TCPAbortOnSyn: 0
TCPAbortOnData: 0
TCPAbortOnClose: 0
TCPAbortOnMemory: 0
TCPAbortOnTimeout: 0
TCPAbortOnLinger: 0
TCPAbortFailed: 0
TCPMemoryPressures: 0
TCPSACKDiscard: 0
TCPDSACKIgnoredOld: 0
TCPDSACKIgnoredNoUndo: 0
TCPSpuriousRTOs: 0
TCPMD5NotFound: 0
TCPMD5Unexpected: 0
TCPSackShifted: 0
TCPSackMerged: 0
TCPSackShiftFallback: 0
TCPBacklogDrop: 0
TCPMinTTLDrop: 0
IpExt:

    InNoRoutes: 0
    InTruncatedPkts: 0
    InMcastPkts: 0
    OutMcastPkts: 0
    InBcastPkts: 0
    OutBcastPkts: 0
    InOctets: 118232864
    OutOctets: 1542068
    InMcastOctets: 0
    OutMcastOctets: 0
    InBcastOctets: 0
    OutBcastOctets: 0

~ #

Signature · 3 lines
Michal Simek, Ing. (M.Eng)
PetaLogix - Linux Solutions for a Reconfigurable World
w: www.petalogix.com p: +61-7-30090663,+42-0-721842854 f: +61-7-30090663
Quoted 1 line

From e2d160e2e235103af3e08b5bbbc451982bc0fed0 Mon Sep 17 00:00:00 2001

From: Michal Simek <[email protected]>
Date: Tue, 30 Mar 2010 10:45:06 +0200
Subject: [PATCH] Extend buffer size because line size is greater than 1024 chars

Error shown on console

UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
error parsing /proc/net/snmp: Success

It is easy to check size of line which is necessary to check.

1151

---
 statistics.c |    2 +-
 1 files changed, 1 insertions(+), 1 deletions(-)
statistics.c +1 -1
index a878df8..51beb8a 100644
@@ -291,7 +291,7 @@ struct tabtab *newtable(struct tabtab *tabs, char *title)
 
 void process_fd(FILE *f)
 {
-    char buf1[1024], buf2[1024];
+    char buf1[2048], buf2[2048];
     char *sp, *np, *p;
     while (fgets(buf1, sizeof buf1, f)) {
 	int endflag;
Signature · 1 line
1.5.5.1
#6 lore raw

Looks like a lot of time spent in tcp_collapse(). I think there is assumption that the memcpys involved in collapsing lead to better throughput than packet retransmit, which on our platforms may not be the case. If so is there a way to tune when tcp_collapse gets invoked? I haven't seen one.

Steve

-----Original Message-----
From: Michal Simek <[email protected]>
To: Eric Dumazet <[email protected]>
Cc: LKML <[email protected]>, John Williams <[email protected]>, [email protected], Grant Likely <[email protected]>, John Linn <[email protected]>, "Steven J. Magnani" <[email protected]>, Arnd Bergmann <[email protected]>, [email protected]
Date: Tue, 30 Mar 2010 11:34:29 +0200
Subject: Re: Network performance - iperf

Quoted 335 lines
Quoted 6 lines · Eric Dumazet
Quoted 5 lines · Michal Simek
Quoted 4 lines · Michal Simek
Quoted 3 lines · Eric Dumazet
Quoted 1 line · Michal Simek

Do you have any idea howto improve TCP/UDP performance in

general?

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 1 line

Or tests which can point me on weak places.

Could you post "netstat -s" on your receiver, after fresh boot and

your

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
Quoted 2 lines

iperf session, for 32 MB and 256 MB ram case ?

I am not sure if is helpful but look below.

Sorry I forget to c&p that second part. :-(

Sorry, your netstat is not up2date.

I am afraid that is up2date.

Quoted 6 lines

If you cannot correct it to last version
[ net-tools 1.60 , netstat 1.42 ], please send

cat /proc/net/snmp
cat /proc/net/netstat

There is small buffer for parsing /proc/net/netstat. There is necessary to extend buffer size because one line is greater than 1024 chars.

~ # head -n 1 /proc/net/netstat
TcpExt: SyncookiesSent SyncookiesRecv SyncookiesFailed EmbryonicRsts PruneCalled RcvPruned OfoPruned OutOfWindowIcmps LockDroppedIcmps ArpFilter TW TWRecycled TWKilled PAWSPassive PAWSActive PAWSEstab DelayedACKs DelayedACKLocked DelayedACKLost ListenOverflows ListenDrops TCPPrequeued TCPDirectCopyFromBacklog TCPDirectCopyFromPrequeue TCPPrequeueDropped TCPHPHits TCPHPHitsToUser TCPPureAcks TCPHPAcks TCPRenoRecovery TCPSackRecovery TCPSACKReneging TCPFACKReorder TCPSACKReorder TCPRenoReorder TCPTSReorder TCPFullUndo TCPPartialUndo TCPDSACKUndo TCPLossUndo TCPLoss TCPLostRetransmit TCPRenoFailures TCPSackFailures TCPLossFailures TCPFastRetrans TCPForwardRetrans TCPSlowStartRetrans TCPTimeouts TCPRenoRecoveryFail TCPSackRecoveryFail TCPSchedulerFailed TCPRcvCollapsed TCPDSACKOldSent TCPDSACKOfoSent TCPDSACKRecv TCPDSACKOfoRecv TCPAbortOnSyn TCPAbortOnData TCPAbortOnClose TCPAbortOnMemory TCPAbortOnTimeout TCPAbortOnLinger TCPAbortFailed TCPMemoryPressures TCPSACKDiscard TCPDSACKIgnoredOld TCPDSACKIgnoredNoUndo TCPSpuriousRTOs TCPMD5NotFound TCPMD5Unexpected TCPSackShifted TCPSackMerged TCPSackShiftFallback TCPBacklogDrop TCPMinTTLDrop

Look at attached patch.
And updated results are below.

Thanks,
Michal

256M

~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.0.10 port 5001 connected with 192.168.0.101 port 33261

[ ID] Interval       Transfer     Bandwidth
[  6]  0.0-50.2 sec  22.9 MBytes  3.83 Mbits/sec

~ # ./netstat -s
Ip:
16618 total packets received
0 forwarded
0 incoming packets discarded
16618 incoming packets delivered
6490 requests sent out
Icmp:
0 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
Tcp:
0 active connections openings
1 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
16618 segments received
6490 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
0 packets received
0 packets to unknown port received.
0 packet receive errors
0 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
TcpExt:
2233 packets pruned from receive queue because of socket buffer
overrun
ArpFilter: 0
1 delayed acks sent
5519 packets header predicted
TCPPureAcks: 2
TCPHPAcks: 0
TCPRenoRecovery: 0
TCPSackRecovery: 0
TCPSACKReneging: 0
TCPFACKReorder: 0
TCPSACKReorder: 0
TCPRenoReorder: 0
TCPTSReorder: 0
TCPFullUndo: 0
TCPPartialUndo: 0
TCPDSACKUndo: 0
TCPLossUndo: 0
TCPLoss: 0
TCPLostRetransmit: 0
TCPRenoFailures: 0
TCPSackFailures: 0
TCPLossFailures: 0
TCPFastRetrans: 0
TCPForwardRetrans: 0
TCPSlowStartRetrans: 0
TCPTimeouts: 0
TCPRenoRecoveryFail: 0
TCPSackRecoveryFail: 0
TCPSchedulerFailed: 0
TCPRcvCollapsed: 207654
TCPDSACKOldSent: 0
TCPDSACKOfoSent: 0
TCPDSACKRecv: 0
TCPDSACKOfoRecv: 0
TCPAbortOnSyn: 0
TCPAbortOnData: 0
TCPAbortOnClose: 0
TCPAbortOnMemory: 0
TCPAbortOnTimeout: 0
TCPAbortOnLinger: 0
TCPAbortFailed: 0
TCPMemoryPressures: 0
TCPSACKDiscard: 0
TCPDSACKIgnoredOld: 0
TCPDSACKIgnoredNoUndo: 0
TCPSpuriousRTOs: 0
TCPMD5NotFound: 0
TCPMD5Unexpected: 0
TCPSackShifted: 0
TCPSackMerged: 0
TCPSackShiftFallback: 0
TCPBacklogDrop: 0
TCPMinTTLDrop: 0
IpExt:

     InNoRoutes: 0
     InTruncatedPkts: 0
     InMcastPkts: 0
     OutMcastPkts: 0
     InBcastPkts: 0
     OutBcastPkts: 0
     InOctets: 24915880
     OutOctets: 337488
     InMcastOctets: 0
     OutMcastOctets: 0
     InBcastOctets: 0
     OutBcastOctets: 0

~ # ./netstat --version
net-tools 1.60
netstat 1.42 (2001-04-15)
Fred Baumgarten, Alan Cox, Bernd Eckenfels, Phil Blundell, Tuan Hoang and others
+NEW_ADDRT +RTF_IRTT +RTF_REJECT +FW_MASQUERADE -I18N
AF: (inet) +UNIX +INET -INET6 -IPX -AX25 -NETROM -X25 -ATALK -ECONET -ROSE
HW: +ETHER -ARC +SLIP +PPP -TUNNEL -TR -AX25 -NETROM -X25 -FR -ROSE -ASH -SIT -FDDI -HIPPI -HDLC/LAPB
~ # head -n 1 /proc/meminfo
MemTotal: 257108 kB

32MB

~ # head -n 1 /proc/meminfo
MemTotal: 29920 kB
~ # iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.0.10 port 5001 connected with 192.168.0.101 port 50088
[ ID] Interval Transfer Bandwidth
[ 6] 0.0-50.0 sec 109 MBytes 18.3 Mbits/sec
~ # ./netstat -s
Ip:
79040 total packets received
0 forwarded
0 incoming packets discarded
79040 incoming packets delivered
29655 requests sent out
Icmp:
0 ICMP messages received
0 input ICMP message failed.
ICMP input histogram:
0 ICMP messages sent
0 ICMP messages failed
ICMP output histogram:
Tcp:
0 active connections openings
1 passive connection openings
0 failed connection attempts
0 connection resets received
0 connections established
79040 segments received
29655 segments send out
0 segments retransmited
0 bad segments received.
0 resets sent
Udp:
0 packets received
0 packets to unknown port received.
0 packet receive errors
0 packets sent
RcvbufErrors: 0
SndbufErrors: 0
UdpLite:
InDatagrams: 0
NoPorts: 0
InErrors: 0
OutDatagrams: 0
RcvbufErrors: 0
SndbufErrors: 0
TcpExt:
9773 packets pruned from receive queue because of socket buffer
overrun
ArpFilter: 0
1 delayed acks sent
101 packets directly queued to recvmsg prequeue. 558928 packets directly received from prequeue
33274 packets header predicted
378 packets header predicted and directly queued to user
TCPPureAcks: 2
TCPHPAcks: 0
TCPRenoRecovery: 0
TCPSackRecovery: 0
TCPSACKReneging: 0
TCPFACKReorder: 0
TCPSACKReorder: 0
TCPRenoReorder: 0
TCPTSReorder: 0
TCPFullUndo: 0
TCPPartialUndo: 0
TCPDSACKUndo: 0
TCPLossUndo: 0
TCPLoss: 0
TCPLostRetransmit: 0
TCPRenoFailures: 0
TCPSackFailures: 0
TCPLossFailures: 0
TCPFastRetrans: 0
TCPForwardRetrans: 0
TCPSlowStartRetrans: 0
TCPTimeouts: 0
TCPRenoRecoveryFail: 0
TCPSackRecoveryFail: 0
TCPSchedulerFailed: 0
TCPRcvCollapsed: 120195
TCPDSACKOldSent: 0
TCPDSACKOfoSent: 0
TCPDSACKRecv: 0
TCPDSACKOfoRecv: 0
TCPAbortOnSyn: 0
TCPAbortOnData: 0
TCPAbortOnClose: 0
TCPAbortOnMemory: 0
TCPAbortOnTimeout: 0
TCPAbortOnLinger: 0
TCPAbortFailed: 0
TCPMemoryPressures: 0
TCPSACKDiscard: 0
TCPDSACKIgnoredOld: 0
TCPDSACKIgnoredNoUndo: 0
TCPSpuriousRTOs: 0
TCPMD5NotFound: 0
TCPMD5Unexpected: 0
TCPSackShifted: 0
TCPSackMerged: 0
TCPSackShiftFallback: 0
TCPBacklogDrop: 0
TCPMinTTLDrop: 0
IpExt:

     InNoRoutes: 0
     InTruncatedPkts: 0
     InMcastPkts: 0
     OutMcastPkts: 0
     InBcastPkts: 0
     OutBcastPkts: 0
     InOctets: 118232864
     OutOctets: 1542068
     InMcastOctets: 0
     OutMcastOctets: 0
     InBcastOctets: 0
     OutBcastOctets: 0

~ #

Signature · 4 lines
Michal Simek, Ing. (M.Eng)
PetaLogix - Linux Solutions for a Reconfigurable World
w: www.petalogix.com p: +61-7-30090663,+42-0-721842854 f:
+61-7-30090663
#7 lore raw
In reply to Michal Simek
Quoted 2 lines · Michal Simek

2233 packets pruned from receive queue because of socket buffer overrun
TCPRcvCollapsed: 207654

Thats a problem. A big one :(

If I remember, you use LL_TEMAC driver.

This drivers allocates big skbs for RX ring (more than 9000 bytes each skb). Given your 32 Mbytes kernel size, this seems plain wrong.

You might try to copybreak them before giving skb to network stack, consuming the minimum space.

This would also help this driver to survive in low memory conditions, avoiding death if high order pages are not available.

I cannot even compile this driver on my x86 platform, but here is a preliminar patch to give you the idea :

[PATCH] ll_temac: Fix some memory allocation problems

Driver use high order allocations that might fail after a while. When receiving a buffer from card, try to copy it to keep a pool of pre-allocated high order buffers.

---
Please please please note I did not test this patch.

 drivers/net/ll_temac_main.c |   48 +++++++++++++++-------------------
 1 file changed, 22 insertions(+), 26 deletions(-)
drivers/net/ll_temac_main.c +22 -26
index a18e348..412b72e 100644
@@ -155,8 +155,8 @@ static int temac_dma_bd_init(struct net_device *ndev)
 		lp->rx_bd_v[i].next = lp->rx_bd_p +
 				sizeof(*lp->rx_bd_v) * ((i + 1) % RX_BD_NUM);
 
-		skb = alloc_skb(XTE_MAX_JUMBO_FRAME_SIZE
-				+ XTE_ALIGN, GFP_ATOMIC);
+		skb = alloc_skb(XTE_MAX_JUMBO_FRAME_SIZE + XTE_ALIGN,
+				GFP_KERNEL);
 		if (skb == 0) {
 			dev_err(&ndev->dev, "alloc_skb error %d\n", i);
 			return -1;
@@ -625,34 +625,30 @@ static void ll_temac_recv(struct net_device *ndev)
 		skb = lp->rx_skb[lp->rx_bd_ci];
 		length = cur_p->app4 & 0x3FFF;
 
-		skb_vaddr = virt_to_bus(skb->data);
+		new_skb = netdev_alloc_skb_ip_align(length);
+		if (new_skb) {
+			skb_copy_to_linear_data(new_skb, skb->data, length);
+			skb_put(new_skb, length);
+			skb_vaddr = virt_to_bus(skb->data);
+			dma_sync_single_for_device(ndev->dev.parent,
+						   skb_vaddr,
+						   XTE_MAX_JUMBO_FRAME_SIZE,
+						   PCI_DMA_FROMDEVICE);
+			new_skb->dev = ndev;
+			new_skb->protocol = eth_type_trans(new_skb, ndev);
+			new_skb->ip_summed = CHECKSUM_NONE;
+
+			netif_rx(new_skb);
+
+			ndev->stats.rx_packets++;
+			ndev->stats.rx_bytes += length;
+		} else
+			ndev->stats.rx_dropped++;
+
 		dma_unmap_single(ndev->dev.parent, skb_vaddr, length,
 				 DMA_FROM_DEVICE);
 
-		skb_put(skb, length);
-		skb->dev = ndev;
-		skb->protocol = eth_type_trans(skb, ndev);
-		skb->ip_summed = CHECKSUM_NONE;
-
-		netif_rx(skb);
-
-		ndev->stats.rx_packets++;
-		ndev->stats.rx_bytes += length;
-
-		new_skb = alloc_skb(XTE_MAX_JUMBO_FRAME_SIZE + XTE_ALIGN,
-				GFP_ATOMIC);
-		if (new_skb == 0) {
-			dev_err(&ndev->dev, "no memory for new sk_buff\n");
-			spin_unlock_irqrestore(&lp->rx_lock, flags);
-			return;
-		}
-
-		skb_reserve(new_skb, BUFFER_ALIGN(new_skb->data));
-
 		cur_p->app0 = STS_CTRL_APP0_IRQONEND;
-		cur_p->phys = dma_map_single(ndev->dev.parent, new_skb->data,
-					     XTE_MAX_JUMBO_FRAME_SIZE,
-					     DMA_FROM_DEVICE);
 		cur_p->len = XTE_MAX_JUMBO_FRAME_SIZE;
 		lp->rx_skb[lp->rx_bd_ci] = new_skb;