Excessive ethernet interrupts on AM335x board

DISCUSSION Mark Jackson 3 participants 6 messages Mar 12, 2013 – Mar 13, 2013

Share & tools

Mark Jackson

author

2013-03-12 15:35 UTC #0 lore raw

I'm just fighting an issue with ethernet on our custom AM335x board:-

# uname -a
Linux nanobone 3.9.0-rc2-00113-gd60f039 #139 Tue Mar 12 15:14:01 GMT 2013 armv7l GNU/Linux

Every now and then, the whole unit slows to a crawl. The only indication of any problem is:-

(a) the serial tty port becomes much less responsive
(b) normal ping times jump from 1ms to >10sec (sometimes >20sec !!)
(c) the ethernet interrupt count rockets (see below)

I've tried to force the problem by flood pinging from my PC.

# while true

Quoted 2 lines

do grep "58:" /proc/interrupts; sleep 10
done

 58:       1291      INTC  4a100000.ethernet  <<< normal pinging (about 100 irqs per 10sec)
 58:       1333      INTC  4a100000.ethernet
 58:       1372      INTC  4a100000.ethernet
 58:       3979      INTC  4a100000.ethernet  <<< start flood ping (about 4k irqs per 10sec)
 58:       6540      INTC  4a100000.ethernet
 58:      17519      INTC  4a100000.ethernet  <<< big jump >>>
 58:      20169      INTC  4a100000.ethernet
 58:      22775      INTC  4a100000.ethernet
 58:      25368      INTC  4a100000.ethernet
 58:      34598      INTC  4a100000.ethernet  <<< big jump >>>
 58:      37182      INTC  4a100000.ethernet
 58:      39730      INTC  4a100000.ethernet
 58:     141220      INTC  4a100000.ethernet  <<< whoa !!! >>>
 58:     146080      INTC  4a100000.ethernet
 58:     149351      INTC  4a100000.ethernet
 58:     152922      INTC  4a100000.ethernet
 58:     156420      INTC  4a100000.ethernet
 58:     159538      INTC  4a100000.ethernet
 58:     162711      INTC  4a100000.ethernet
 58:     165746      INTC  4a100000.ethernet
 58:     168973      INTC  4a100000.ethernet
 58:     172128      INTC  4a100000.ethernet
 58:     175030      INTC  4a100000.ethernet
 58:     177957      INTC  4a100000.ethernet
 58:     180782      INTC  4a100000.ethernet
 58:     183618      INTC  4a100000.ethernet
 58:     186450      INTC  4a100000.ethernet
 58:     189242      INTC  4a100000.ethernet
 58:     191909      INTC  4a100000.ethernet
 58:     194565      INTC  4a100000.ethernet
 58:     197153      INTC  4a100000.ethernet
 58:     199730      INTC  4a100000.ethernet  <<< another big jump >>>
 58:     252629      INTC  4a100000.ethernet
 58:     262955      INTC  4a100000.ethernet
 58:     265557      INTC  4a100000.ethernet
 58:     268131      INTC  4a100000.ethernet
 58:     272586      INTC  4a100000.ethernet
 58:     275623      INTC  4a100000.ethernet  <<< here I stop flood pings >>>
[  631.727758] nfs: server 10.0.0.100 not responding, still trying
[  638.738864] nfs: server 10.0.0.100 OK
 58:     277694      INTC  4a100000.ethernet
 58:     277703      INTC  4a100000.ethernet
 58:     277709      INTC  4a100000.ethernet
 58:     277719      INTC  4a100000.ethernet
 58:     277725      INTC  4a100000.ethernet
 58:     277734      INTC  4a100000.ethernet
 58:     277745      INTC  4a100000.ethernet

As you can see, when I stop the flood pings, the nfs link is now reported as being lost.

Any ideas ?

Cheers
Mark J.

Mark Jackson

author

2013-03-12 15:54 UTC #1 lore raw

Quoted 16 lines · Mark Jackson

I'm just fighting an issue with ethernet on our custom AM335x board:-
# uname -a
Linux nanobone 3.9.0-rc2-00113-gd60f039 #139 Tue Mar 12 15:14:01 GMT 2013 armv7l GNU/Linux
Every now and then, the whole unit slows to a crawl. The only indication of any problem is:-
(a) the serial tty port becomes much less responsive
(b) normal ping times jump from 1ms to >10sec (sometimes >20sec !!)
(c) the ethernet interrupt count rockets (see below)
I've tried to force the problem by flood pinging from my PC.
# while true
Quoted 2 lines
do grep "58:" /proc/interrupts; sleep 10
done

Quoted 14 lines

 58:       1291      INTC  4a100000.ethernet  <<< normal pinging (about 100 irqs per 10sec)
 58:       1333      INTC  4a100000.ethernet
 58:       1372      INTC  4a100000.ethernet
 58:       3979      INTC  4a100000.ethernet  <<< start flood ping (about 4k irqs per 10sec)
 58:       6540      INTC  4a100000.ethernet
 58:      17519      INTC  4a100000.ethernet  <<< big jump >>>
 58:      20169      INTC  4a100000.ethernet
 58:      22775      INTC  4a100000.ethernet
 58:      25368      INTC  4a100000.ethernet
 58:      34598      INTC  4a100000.ethernet  <<< big jump >>>
 58:      37182      INTC  4a100000.ethernet
 58:      39730      INTC  4a100000.ethernet
 58:     141220      INTC  4a100000.ethernet  <<< whoa !!! >>>
 58:     146080      INTC  4a100000.ethernet

Doing the flood ping test on an old Beaglebone (running kernel 3.2.34 on an sdcard), I get:-

# while true

Quoted 1 line

do grep "94:" /proc/interrupts; sleep 10

Quoted 1 line

done

 94:     281353      INTC  cpsw.0
 94:     370782      INTC  cpsw.0
 94:     457537      INTC  cpsw.0
 94:     544876      INTC  cpsw.0
 94:     631795      INTC  cpsw.0
 94:     717747      INTC  cpsw.0
 94:     805974      INTC  cpsw.0
 94:     892961      INTC  cpsw.0
 94:     981490      INTC  cpsw.0
 94:    1070627      INTC  cpsw.0
 94:    1153086      INTC  cpsw.0
 94:    1242060      INTC  cpsw.0
 94:    1327734      INTC  cpsw.0
 94:    1413705      INTC  cpsw.0
 94:    1504494      INTC  cpsw.0
 94:    1591395      INTC  cpsw.0
 94:    1676769      INTC  cpsw.0

So these are going up by 90k irqs per 10sec ... meaning that the AM335x board seems to be *dropping* most of its ethernet irqs.

I'll try to get 3.9.0-rc2 on the BB and retest.

Mark J.

Koen Kooi

2013-03-13 08:44 UTC #2 lore raw

In reply to Mark Jackson

Op 12 mrt. 2013, om 16:35 heeft Mark Jackson <[email protected]> het volgende geschreven:

Quoted 10 lines

I'm just fighting an issue with ethernet on our custom AM335x board:-
# uname -a
Linux nanobone 3.9.0-rc2-00113-gd60f039 #139 Tue Mar 12 15:14:01 GMT 2013 armv7l GNU/Linux
Every now and then, the whole unit slows to a crawl. The only indication of any problem is:-
(a) the serial tty port becomes much less responsive
(b) normal ping times jump from 1ms to >10sec (sometimes >20sec !!)
(c) the ethernet interrupt count rockets (see below)

You probably have PG2.x silicon, have a look at this patch: https://github.com/beagleboard/kernel/blob/3.8/patches/net/0003-cpsw-Fix-interrupt-storm-among-other-things.patch

I saw some patches going into net-next today that might address this in a different way, but I haven't tried 3.9rc on an am335x yet.

Mark Jackson

author

2013-03-13 10:11 UTC #3 lore raw

Quoted 8 lines · Koen Kooi

Op 12 mrt. 2013, om 16:35 heeft Mark Jackson <[email protected]> het volgende geschreven:
Quoted 5 lines
I'm just fighting an issue with ethernet on our custom AM335x board:-
# uname -a
Linux nanobone 3.9.0-rc2-00113-gd60f039 #139 Tue Mar 12 15:14:01 GMT 2013 armv7l GNU/Linux

Quoted 7 lines

Quoted 5 lines
Every now and then, the whole unit slows to a crawl. The only indication of any problem is:-
(a) the serial tty port becomes much less responsive
(b) normal ping times jump from 1ms to >10sec (sometimes >20sec !!)
(c) the ethernet interrupt count rockets (see below)
You probably have PG2.x silicon, have a look at this patch: https://github.com/beagleboard/kernel/blob/3.8/patches/net/0003-cpsw-Fix-interrupt-storm-among-other-things.patch

No, it's 1.0 ...

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 3.9.0-rc2-00113-gd60f039-dirty (mpfj@mpfj-nanobone) (gcc version 4.5.4 (Buildroot 2012.11) ) #141 Wed Mar 13 09:14:03 GMT 2013
[    0.000000] CPU: ARMv7 Processor [413fc082] revision 2 (ARMv7), cr=10c53c7d
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] Machine: Generic AM33XX (Flattened Device Tree), model: Newflow AM335x NanoBone
[    0.000000] Memory policy: ECC disabled, Data cache writeback
[    0.000000] CPU: All CPU(s) started in SVC mode.
[    0.000000] AM335X ES1.0 (neon )

The patch certainly didn't fix things (and possibly made things worse i.e. my nfs root kept dropping off even more).

Quoted 1 line

I saw some patches going into net-next today that might address this in a different way, but I haven't tried 3.9rc on an am335x yet.

I might track those down and test them.

Cheers
Mark J.

Daniel Mack

2013-03-13 10:32 UTC #4 lore raw

In reply to Mark Jackson

Quoted 64 lines · Mark Jackson

I'm just fighting an issue with ethernet on our custom AM335x board:-

# uname -a
Linux nanobone 3.9.0-rc2-00113-gd60f039 #139 Tue Mar 12 15:14:01 GMT 2013 armv7l GNU/Linux

Every now and then, the whole unit slows to a crawl. The only indication of any problem is:-

(a) the serial tty port becomes much less responsive
(b) normal ping times jump from 1ms to >10sec (sometimes >20sec !!)
(c) the ethernet interrupt count rockets (see below)

I've tried to force the problem by flood pinging from my PC.

# while true

Quoted 2 lines

do grep "58:" /proc/interrupts; sleep 10
done

 58:       1291      INTC  4a100000.ethernet  <<< normal pinging (about 100 irqs per 10sec)
 58:       1333      INTC  4a100000.ethernet
 58:       1372      INTC  4a100000.ethernet
 58:       3979      INTC  4a100000.ethernet  <<< start flood ping (about 4k irqs per 10sec)
 58:       6540      INTC  4a100000.ethernet
 58:      17519      INTC  4a100000.ethernet  <<< big jump >>>
 58:      20169      INTC  4a100000.ethernet
 58:      22775      INTC  4a100000.ethernet
 58:      25368      INTC  4a100000.ethernet
 58:      34598      INTC  4a100000.ethernet  <<< big jump >>>
 58:      37182      INTC  4a100000.ethernet
 58:      39730      INTC  4a100000.ethernet
 58:     141220      INTC  4a100000.ethernet  <<< whoa !!! >>>
 58:     146080      INTC  4a100000.ethernet
 58:     149351      INTC  4a100000.ethernet
 58:     152922      INTC  4a100000.ethernet
 58:     156420      INTC  4a100000.ethernet
 58:     159538      INTC  4a100000.ethernet
 58:     162711      INTC  4a100000.ethernet
 58:     165746      INTC  4a100000.ethernet
 58:     168973      INTC  4a100000.ethernet
 58:     172128      INTC  4a100000.ethernet
 58:     175030      INTC  4a100000.ethernet
 58:     177957      INTC  4a100000.ethernet
 58:     180782      INTC  4a100000.ethernet
 58:     183618      INTC  4a100000.ethernet
 58:     186450      INTC  4a100000.ethernet
 58:     189242      INTC  4a100000.ethernet
 58:     191909      INTC  4a100000.ethernet
 58:     194565      INTC  4a100000.ethernet
 58:     197153      INTC  4a100000.ethernet
 58:     199730      INTC  4a100000.ethernet  <<< another big jump >>>
 58:     252629      INTC  4a100000.ethernet
 58:     262955      INTC  4a100000.ethernet
 58:     265557      INTC  4a100000.ethernet
 58:     268131      INTC  4a100000.ethernet
 58:     272586      INTC  4a100000.ethernet
 58:     275623      INTC  4a100000.ethernet  <<< here I stop flood pings >>>
[  631.727758] nfs: server 10.0.0.100 not responding, still trying
[  638.738864] nfs: server 10.0.0.100 OK
 58:     277694      INTC  4a100000.ethernet
 58:     277703      INTC  4a100000.ethernet
 58:     277709      INTC  4a100000.ethernet
 58:     277719      INTC  4a100000.ethernet
 58:     277725      INTC  4a100000.ethernet
 58:     277734      INTC  4a100000.ethernet
 58:     277745      INTC  4a100000.ethernet

Quoted 2 lines

As you can see, when I stop the flood pings, the nfs link is now reported as being lost.

I had the same problem. Please check this patch, I'm sure it will fix you issue:

https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=d35162f89b8f00537d7b240b76d2d0e8b8d29aa0

Daniel

Mark Jackson

author

2013-03-13 10:37 UTC #5 lore raw

Quoted 13 lines · Daniel Mack

Quoted 12 lines · Mark Jackson
I'm just fighting an issue with ethernet on our custom AM335x board:-
# uname -a
Linux nanobone 3.9.0-rc2-00113-gd60f039 #139 Tue Mar 12 15:14:01 GMT 2013 armv7l GNU/Linux
Every now and then, the whole unit slows to a crawl. The only indication of any problem is:-
(a) the serial tty port becomes much less responsive
(b) normal ping times jump from 1ms to >10sec (sometimes >20sec !!)
(c) the ethernet interrupt count rockets (see below)
I've tried to force the problem by flood pinging from my PC.

<snip>

Quoted 6 lines

Quoted 2 lines
As you can see, when I stop the flood pings, the nfs link is now reported as being lost.
I had the same problem. Please check this patch, I'm sure it will fix you issue:
https://git.kernel.org/cgit/linux/kernel/git/davem/net.git/commit/?id=d35162f89b8f00537d7b240b76d2d0e8b8d29aa0

Brilliant ... that's the one !!

Cheers
Mark J.