Skip to contentSkip to conversation

2.6.29 & network stack strangeness

DISCUSSION Matthew Lear 2 participants 8 messages Jun 5, 2009 – Jun 10, 2009 3 messages not in the archive Networking

lore
Share & tools
lore
#0 lore raw

Hello all,

I'm running a 2.6.29 kernel on an MMU enabled m68k coldfire mcf54455 platform and I'm having some throughput problems when running network tests.

The kernel boots and mounts its rootfs from flash (jffs2). udhcpc runs, obtains a lease from the dhcp server and configures eth0. Network connectivity is ok. I can ping the target from the host and vice versa.

1/
If I run ping -s 1500 -i 0.0001 <target ip address> on the host pc, after several mins, the kernel reports 'unexpected interrupt from 24' which is the vector for a spurious interrupt. This message will repeat randomly (from what I saw it appeared ~ 20 times when running the ping test above for 40 mins). The mcf54455 reference manual describes a possible cause for spurious interrupts. However, this test very rarely reports any packet loss, although the max time to receive a packet can be very large indeed.

2/
If I reboot, start again and run a ping flood test (ping -f) from host pc -> target, all icmp requests are acknowledged - for a while. Before the target begins to fail to respond to the icmp requests, running top shows that the ksoftirq daemon is running at ~ 5% cpu load. This is normal as it is involved in processing the deferred tasks of processing data fired up to the network stack. So when the target beings to stop responding to icmp, if I then stop the ping flood and try to ping the host from the target, there is no reply indicated by ping. However, if you do this with a packet sniffer running (eg wireshark) you can see that data is still being transmitted from the target -> host and you can see the icmp reply, only the reply from the host appears to be received ok by the fec driver but is processed by the network stack target.

When in this state, a proc entry that I added to the fec driver shows that the last return value from netif_rx() (called in the fec rx interrupt handling routine) is 1, indicating that the last packet was dropped by the network stack, e.g.

~ # cat /proc/driver/fec
total interrupts: 1421619
last interrupt type: 2 [1=tx, 2=rx, 3=mii]
total tx interrupts: 709148
total rx interrupts: 712472
total mii interrupts: 1
last interrupt event: 0x2000000
total eberr interrupts: 0
total hberr interrupts: 0
tx loop current count: 0
tx loop last count: 1
rx loop current count: 0
rx loop last count: 1
rx last cbd ctrl/status: 0x800
rx last cbd len: 346
rx last cbd buff addr: 0x40410000
rx last netif_rx status: 1

Strangely, wireshark still shows data being transmitted from the target -> host. I can see ARP requests and I can also see DHCP discovery packets being sent by the target when its DHCP lease expires. This all looks ok, only the reply from host -> target is never processed by the target as the network stack is in a state where it is dropping all incoming data provided to it by the driver.

I believe udhcpc utilises the network device directly, ie it does not require an intermediate network protocol being implemented in the kernel (tcpdump is similar).

The fec driver still seems to be running ok because I can see the ring buffer address changing when data is received. Everything seems to be ok apart from the network stack. Very strange indeed.

Running network throughput tests between host and target with netcat or netperf only run for a few seconds before activity ceases.

Has anybody experienced anything similar? Why does the network stack appear to be stuck and constantly dropping packets?

Any feedback appreciated.

Rgds,
-- Matt

#1 lore raw

Does the problem manifest only when the DHCP lease expires? Can you reproduce the problem with a static IP?

Finn

Quoted 83 lines · Matthew Lear

Hello all,

I'm running a 2.6.29 kernel on an MMU enabled m68k coldfire mcf54455 platform and I'm having some throughput problems when running network tests.

The kernel boots and mounts its rootfs from flash (jffs2). udhcpc runs, obtains a lease from the dhcp server and configures eth0. Network connectivity is ok. I can ping the target from the host and vice versa.

1/
If I run ping -s 1500 -i 0.0001 <target ip address> on the host pc, after several mins, the kernel reports 'unexpected interrupt from 24' which is the vector for a spurious interrupt. This message will repeat randomly (from what I saw it appeared ~ 20 times when running the ping test above for 40 mins). The mcf54455 reference manual describes a possible cause for spurious interrupts. However, this test very rarely reports any packet loss, although the max time to receive a packet can be very large indeed.

2/
If I reboot, start again and run a ping flood test (ping -f) from host pc -> target, all icmp requests are acknowledged - for a while. Before the target begins to fail to respond to the icmp requests, running top shows that the ksoftirq daemon is running at ~ 5% cpu load. This is normal as it is involved in processing the deferred tasks of processing data fired up to the network stack. So when the target beings to stop responding to icmp, if I then stop the ping flood and try to ping the host from the target, there is no reply indicated by ping. However, if you do this with a packet sniffer running (eg wireshark) you can see that data is still being transmitted from the target -> host and you can see the icmp reply, only the reply from the host appears to be received ok by the fec driver but is processed by the network stack target.

When in this state, a proc entry that I added to the fec driver shows that the last return value from netif_rx() (called in the fec rx interrupt handling routine) is 1, indicating that the last packet was dropped by the network stack, e.g.

~ # cat /proc/driver/fec
total interrupts: 1421619
last interrupt type: 2 [1=tx, 2=rx, 3=mii]
total tx interrupts: 709148
total rx interrupts: 712472
total mii interrupts: 1
last interrupt event: 0x2000000
total eberr interrupts: 0
total hberr interrupts: 0
tx loop current count: 0
tx loop last count: 1
rx loop current count: 0
rx loop last count: 1
rx last cbd ctrl/status: 0x800
rx last cbd len: 346
rx last cbd buff addr: 0x40410000
rx last netif_rx status: 1

Strangely, wireshark still shows data being transmitted from the target -> host. I can see ARP requests and I can also see DHCP discovery packets being sent by the target when its DHCP lease expires. This all looks ok, only the reply from host -> target is never processed by the target as the network stack is in a state where it is dropping all incoming data provided to it by the driver.

I believe udhcpc utilises the network device directly, ie it does not require an intermediate network protocol being implemented in the kernel (tcpdump is similar).

The fec driver still seems to be running ok because I can see the ring buffer address changing when data is received. Everything seems to be ok apart from the network stack. Very strange indeed.

Running network throughput tests between host and target with netcat or netperf only run for a few seconds before activity ceases.

Has anybody experienced anything similar? Why does the network stack appear to be stuck and constantly dropping packets?

Any feedback appreciated.

Rgds,
-- Matt
--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw

Hi - thanks for your reply.

The problem doesn't manifest only when the DHCP lease expires and I can still reproduce the problem with a static IP. With or without DHCP makes no difference.

It seems to effect socket comms quite seriously (and quickly). If I run a simple server program on the host that listens on a socket and writes a response string to the socket when it receives data, and on the target I run a simple client program which writes a string to the socket, reads and prints the response sent the server, I only have to send data from client to server with a delay of 1ms between transmissions for a few seconds and the client program hangs on calling read() on the socket fd.

If I run a simple netcat test, eg

on target: nc -l -p 3333 > /dev/null
on host: dd if=/dev/zero | nc <target-ip> 3333

...strangely, once activity on the ethernet link as a result of the netcat test ceases, running netstat -a on the target hangs for several seconds, eg:

~ # nc -l -p 3333 > /dev/null &
~ # netstat -a
Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 *:login                 *:*                     LISTEN
tcp        0      0 *:shell                 *:*                     LISTEN
tcp        0      0 *:sunrpc                *:*                     LISTEN
tcp        0      0 *:finger                *:*                     LISTEN
tcp        0      0 *:auth                  *:*                     LISTEN
tcp        0      0 *:ftp                   *:*                     LISTEN
tcp        0      0 *:telnet                *:*                     LISTEN

<system hangs for several seconds here>

tcp 0 0 192.168.0.11:3333 gateway0:45645
ESTABLISHED

udp        0      0 *:ntalk                 *:*
udp        0      0 *:sunrpc                *:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  4      [ ]         DGRAM                    111    /dev/log
unix  3      [ ]         STREAM     CONNECTED     123
unix  3      [ ]         STREAM     CONNECTED     122
unix  2      [ ]         DGRAM                    120
unix  2      [ ]         DGRAM                    114

~ #

I thought this was interesting. Also, after this, I have trouble entering characters over the serial port / console. It seems like interrupts may having trouble getting serviced but this may be a side-effect...

If you run the same netstat command with strace, you can see that the delay is caused by polling the socket following calling send:

...
...
gettimeofday({366, 470000}, NULL) = 0
poll([{fd=4, events=POLLOUT, revents=POLLOUT}], 1, 0) = 1 send(4, "lJ\1\0\0\1\0\0\0\0\0\0\00211\0010\003168\003192\7in-ad"..., 43, 0x4000) = 43
poll(

<delay is here>

[{fd=4, events=POLLIN}], 1, 5000) = 0
...
...

-- Matt

Quoted 92 lines · Finn Thain

Does the problem manifest only when the DHCP lease expires? Can you reproduce the problem with a static IP?

Finn

Quoted 83 lines · Matthew Lear

Hello all,

I'm running a 2.6.29 kernel on an MMU enabled m68k coldfire mcf54455 platform and I'm having some throughput problems when running network tests.

The kernel boots and mounts its rootfs from flash (jffs2). udhcpc runs, obtains a lease from the dhcp server and configures eth0. Network connectivity is ok. I can ping the target from the host and vice versa.

1/
If I run ping -s 1500 -i 0.0001 <target ip address> on the host pc, after several mins, the kernel reports 'unexpected interrupt from 24' which is the vector for a spurious interrupt. This message will repeat randomly (from what I saw it appeared ~ 20 times when running the ping test above for 40 mins). The mcf54455 reference manual describes a possible cause for spurious interrupts. However, this test very rarely reports any packet loss, although the max time to receive a packet can be very large indeed.

2/
If I reboot, start again and run a ping flood test (ping -f) from host pc -> target, all icmp requests are acknowledged - for a while. Before the target begins to fail to respond to the icmp requests, running top shows that the ksoftirq daemon is running at ~ 5% cpu load. This is normal as it is involved in processing the deferred tasks of processing data fired up to the network stack. So when the target beings to stop responding to icmp, if I then stop the ping flood and try to ping the host from the target, there is no reply indicated by ping. However, if you do this with a packet sniffer running (eg wireshark) you can see that data is still being transmitted from the target -> host and you can see the icmp reply, only the reply from the host appears to be received ok by the fec driver but is processed by the network stack target.

When in this state, a proc entry that I added to the fec driver shows that the last return value from netif_rx() (called in the fec rx interrupt handling routine) is 1, indicating that the last packet was dropped by the network stack, e.g.

~ # cat /proc/driver/fec
total interrupts: 1421619
last interrupt type: 2 [1=tx, 2=rx, 3=mii]
total tx interrupts: 709148
total rx interrupts: 712472
total mii interrupts: 1
last interrupt event: 0x2000000
total eberr interrupts: 0
total hberr interrupts: 0
tx loop current count: 0
tx loop last count: 1
rx loop current count: 0
rx loop last count: 1
rx last cbd ctrl/status: 0x800
rx last cbd len: 346
rx last cbd buff addr: 0x40410000
rx last netif_rx status: 1

Strangely, wireshark still shows data being transmitted from the target -> host. I can see ARP requests and I can also see DHCP discovery packets being sent by the target when its DHCP lease expires. This all looks ok, only the reply from host -> target is never processed by the target as the network stack is in a state where it is dropping all incoming data provided to it by the driver.

I believe udhcpc utilises the network device directly, ie it does not require an intermediate network protocol being implemented in the kernel (tcpdump is similar).

The fec driver still seems to be running ok because I can see the ring buffer address changing when data is received. Everything seems to be ok apart from the network stack. Very strange indeed.

Running network throughput tests between host and target with netcat or netperf only run for a few seconds before activity ceases.

Has anybody experienced anything similar? Why does the network stack appear to be stuck and constantly dropping packets?

Any feedback appreciated.

Rgds,
-- Matt
--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#3 lore raw

My only guess would be that the network stack delayed work queues depend upon working timer interrupts...

But since I have no knowledge of your hardware, I don't think I'll be a lot of help with that.

Finn

Quoted 174 lines · Matthew Lear

Hi - thanks for your reply.

The problem doesn't manifest only when the DHCP lease expires and I can still reproduce the problem with a static IP. With or without DHCP makes no difference.

It seems to effect socket comms quite seriously (and quickly). If I run a simple server program on the host that listens on a socket and writes a response string to the socket when it receives data, and on the target I run a simple client program which writes a string to the socket, reads and prints the response sent the server, I only have to send data from client to server with a delay of 1ms between transmissions for a few seconds and the client program hangs on calling read() on the socket fd.

If I run a simple netcat test, eg

on target: nc -l -p 3333 > /dev/null
on host: dd if=/dev/zero | nc <target-ip> 3333

...strangely, once activity on the ethernet link as a result of the netcat test ceases, running netstat -a on the target hangs for several seconds, eg:

~ # nc -l -p 3333 > /dev/null &
~ # netstat -a
Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 *:login                 *:*                     LISTEN
tcp        0      0 *:shell                 *:*                     LISTEN
tcp        0      0 *:sunrpc                *:*                     LISTEN
tcp        0      0 *:finger                *:*                     LISTEN
tcp        0      0 *:auth                  *:*                     LISTEN
tcp        0      0 *:ftp                   *:*                     LISTEN
tcp        0      0 *:telnet                *:*                     LISTEN

<system hangs for several seconds here>

tcp 0 0 192.168.0.11:3333 gateway0:45645
ESTABLISHED

udp        0      0 *:ntalk                 *:*
udp        0      0 *:sunrpc                *:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  4      [ ]         DGRAM                    111    /dev/log
unix  3      [ ]         STREAM     CONNECTED     123
unix  3      [ ]         STREAM     CONNECTED     122
unix  2      [ ]         DGRAM                    120
unix  2      [ ]         DGRAM                    114

~ #

I thought this was interesting. Also, after this, I have trouble entering characters over the serial port / console. It seems like interrupts may having trouble getting serviced but this may be a side-effect...

If you run the same netstat command with strace, you can see that the delay is caused by polling the socket following calling send:

...
...
gettimeofday({366, 470000}, NULL) = 0
poll([{fd=4, events=POLLOUT, revents=POLLOUT}], 1, 0) = 1 send(4, "lJ\1\0\0\1\0\0\0\0\0\0\00211\0010\003168\003192\7in-ad"..., 43, 0x4000) = 43
poll(

<delay is here>

[{fd=4, events=POLLIN}], 1, 5000) = 0
...
...

-- Matt

Quoted 92 lines · Finn Thain

Does the problem manifest only when the DHCP lease expires? Can you reproduce the problem with a static IP?

Finn

Quoted 83 lines · Matthew Lear

Hello all,

I'm running a 2.6.29 kernel on an MMU enabled m68k coldfire mcf54455 platform and I'm having some throughput problems when running network tests.

The kernel boots and mounts its rootfs from flash (jffs2). udhcpc runs, obtains a lease from the dhcp server and configures eth0. Network connectivity is ok. I can ping the target from the host and vice versa.

1/
If I run ping -s 1500 -i 0.0001 <target ip address> on the host pc, after several mins, the kernel reports 'unexpected interrupt from 24' which is the vector for a spurious interrupt. This message will repeat randomly (from what I saw it appeared ~ 20 times when running the ping test above for 40 mins). The mcf54455 reference manual describes a possible cause for spurious interrupts. However, this test very rarely reports any packet loss, although the max time to receive a packet can be very large indeed.

2/
If I reboot, start again and run a ping flood test (ping -f) from host pc -> target, all icmp requests are acknowledged - for a while. Before the target begins to fail to respond to the icmp requests, running top shows that the ksoftirq daemon is running at ~ 5% cpu load. This is normal as it is involved in processing the deferred tasks of processing data fired up to the network stack. So when the target beings to stop responding to icmp, if I then stop the ping flood and try to ping the host from the target, there is no reply indicated by ping. However, if you do this with a packet sniffer running (eg wireshark) you can see that data is still being transmitted from the target -> host and you can see the icmp reply, only the reply from the host appears to be received ok by the fec driver but is processed by the network stack target.

When in this state, a proc entry that I added to the fec driver shows that the last return value from netif_rx() (called in the fec rx interrupt handling routine) is 1, indicating that the last packet was dropped by the network stack, e.g.

~ # cat /proc/driver/fec
total interrupts: 1421619
last interrupt type: 2 [1=tx, 2=rx, 3=mii]
total tx interrupts: 709148
total rx interrupts: 712472
total mii interrupts: 1
last interrupt event: 0x2000000
total eberr interrupts: 0
total hberr interrupts: 0
tx loop current count: 0
tx loop last count: 1
rx loop current count: 0
rx loop last count: 1
rx last cbd ctrl/status: 0x800
rx last cbd len: 346
rx last cbd buff addr: 0x40410000
rx last netif_rx status: 1

Strangely, wireshark still shows data being transmitted from the target -> host. I can see ARP requests and I can also see DHCP discovery packets being sent by the target when its DHCP lease expires. This all looks ok, only the reply from host -> target is never processed by the target as the network stack is in a state where it is dropping all incoming data provided to it by the driver.

I believe udhcpc utilises the network device directly, ie it does not require an intermediate network protocol being implemented in the kernel (tcpdump is similar).

The fec driver still seems to be running ok because I can see the ring buffer address changing when data is received. Everything seems to be ok apart from the network stack. Very strange indeed.

Running network throughput tests between host and target with netcat or netperf only run for a few seconds before activity ceases.

Has anybody experienced anything similar? Why does the network stack appear to be stuck and constantly dropping packets?

Any feedback appreciated.

Rgds,
-- Matt
--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#4 lore raw

Yes. I was suspecting that all may not be well in that area... Current set up is a 10ms tick with CONFIG_HZ set to 100. Further investigation is required I think. -- Matt

Quoted 184 lines · Finn Thain

My only guess would be that the network stack delayed work queues depend upon working timer interrupts...

But since I have no knowledge of your hardware, I don't think I'll be a lot of help with that.

Finn

Quoted 172 lines · Matthew Lear

Hi - thanks for your reply.

The problem doesn't manifest only when the DHCP lease expires and I can still reproduce the problem with a static IP. With or without DHCP makes no difference.

It seems to effect socket comms quite seriously (and quickly). If I run a simple server program on the host that listens on a socket and writes a response string to the socket when it receives data, and on the target I run a simple client program which writes a string to the socket, reads and prints the response sent the server, I only have to send data from client to server with a delay of 1ms between transmissions for a few seconds and the client program hangs on calling read() on the socket fd.

If I run a simple netcat test, eg

on target: nc -l -p 3333 > /dev/null
on host: dd if=/dev/zero | nc <target-ip> 3333

...strangely, once activity on the ethernet link as a result of the netcat test ceases, running netstat -a on the target hangs for several seconds, eg:

~ # nc -l -p 3333 > /dev/null &
~ # netstat -a
Active Internet connections (servers and established)

Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 *:login                 *:*                     LISTEN
tcp        0      0 *:shell                 *:*                     LISTEN
tcp        0      0 *:sunrpc                *:*                     LISTEN
tcp        0      0 *:finger                *:*                     LISTEN
tcp        0      0 *:auth                  *:*                     LISTEN
tcp        0      0 *:ftp                   *:*                     LISTEN
tcp        0      0 *:telnet                *:*                     LISTEN

<system hangs for several seconds here>

tcp 0 0 192.168.0.11:3333 gateway0:45645
ESTABLISHED

udp        0      0 *:ntalk                 *:*
udp        0      0 *:sunrpc                *:*
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node Path
unix  4      [ ]         DGRAM                    111    /dev/log
unix  3      [ ]         STREAM     CONNECTED     123
unix  3      [ ]         STREAM     CONNECTED     122
unix  2      [ ]         DGRAM                    120
unix  2      [ ]         DGRAM                    114

~ #

I thought this was interesting. Also, after this, I have trouble entering characters over the serial port / console. It seems like interrupts may having trouble getting serviced but this may be a side-effect...

If you run the same netstat command with strace, you can see that the delay is caused by polling the socket following calling send:

...
...
gettimeofday({366, 470000}, NULL) = 0
poll([{fd=4, events=POLLOUT, revents=POLLOUT}], 1, 0) = 1 send(4, "lJ\1\0\0\1\0\0\0\0\0\0\00211\0010\003168\003192\7in-ad"..., 43, 0x4000) = 43
poll(

<delay is here>

[{fd=4, events=POLLIN}], 1, 5000) = 0
...
...

-- Matt

Quoted 91 lines · Finn Thain

Does the problem manifest only when the DHCP lease expires? Can you reproduce the problem with a static IP?

Finn

Quoted 83 lines · Matthew Lear

Hello all,

I'm running a 2.6.29 kernel on an MMU enabled m68k coldfire mcf54455 platform and I'm having some throughput problems when running network tests.

The kernel boots and mounts its rootfs from flash (jffs2). udhcpc runs, obtains a lease from the dhcp server and configures eth0. Network connectivity is ok. I can ping the target from the host and vice versa.

1/
If I run ping -s 1500 -i 0.0001 <target ip address> on the host pc, after several mins, the kernel reports 'unexpected interrupt from 24' which is the vector for a spurious interrupt. This message will repeat randomly (from what I saw it appeared ~ 20 times when running the ping test above for 40 mins). The mcf54455 reference manual describes a possible cause for spurious interrupts. However, this test very rarely reports any packet loss, although the max time to receive a packet can be very large indeed.

2/
If I reboot, start again and run a ping flood test (ping -f) from host pc -> target, all icmp requests are acknowledged - for a while. Before the target begins to fail to respond to the icmp requests, running top shows that the ksoftirq daemon is running at ~ 5% cpu load. This is normal as it is involved in processing the deferred tasks of processing data fired up to the network stack. So when the target beings to stop responding to icmp, if I then stop the ping flood and try to ping the host from the target, there is no reply indicated by ping. However, if you do this with a packet sniffer running (eg wireshark) you can see that data is still being transmitted from the target -> host and you can see the icmp reply, only the reply from the host appears to be received ok by the fec driver but is processed by the network stack target.

When in this state, a proc entry that I added to the fec driver shows that the last return value from netif_rx() (called in the fec rx interrupt handling routine) is 1, indicating that the last packet was dropped by the network stack, e.g.

~ # cat /proc/driver/fec
total interrupts: 1421619
last interrupt type: 2 [1=tx, 2=rx, 3=mii]
total tx interrupts: 709148
total rx interrupts: 712472
total mii interrupts: 1
last interrupt event: 0x2000000
total eberr interrupts: 0
total hberr interrupts: 0
tx loop current count: 0
tx loop last count: 1
rx loop current count: 0
rx loop last count: 1
rx last cbd ctrl/status: 0x800
rx last cbd len: 346
rx last cbd buff addr: 0x40410000
rx last netif_rx status: 1

Strangely, wireshark still shows data being transmitted from the target -> host. I can see ARP requests and I can also see DHCP discovery packets being sent by the target when its DHCP lease expires. This all looks ok, only the reply from host -> target is never processed by the target as the network stack is in a state where it is dropping all incoming data provided to it by the driver.

I believe udhcpc utilises the network device directly, ie it does not require an intermediate network protocol being implemented in the kernel (tcpdump is similar).

The fec driver still seems to be running ok because I can see the ring buffer address changing when data is received. Everything seems to be ok apart from the network stack. Very strange indeed.

Running network throughput tests between host and target with netcat or netperf only run for a few seconds before activity ceases.

Has anybody experienced anything similar? Why does the network stack appear to be stuck and constantly dropping packets?

Any feedback appreciated.

Rgds,
-- Matt
--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
To unsubscribe from this list: send the line "unsubscribe linux-m68k" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 4 lines

Could you run "top" command and print out its information? "cat /proc/stat" && "cat /proc/interrupts" information are also appreciated.

Sure. Info below.

Before running netcat test:

top - 00:02:37 up 2 min, 1 user, load average: 0.07, 0.07, 0.02

Tasks:  24 total,   1 running,  23 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.7%us,  0.0%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.7%hi,  0.0%si, 
0.0%st
Mem:    254912k total,     6680k used,   248232k free,        0k buffers
Swap:        0k total,        0k used,        0k free,     3008k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  359 root      20   0  2856 1160  944 R  1.0  0.5   0:01.49 top
    3 root      15  -5     0    0    0 S  0.3  0.0   0:00.11 ksoftirqd/0
    1 root      20   0  3280  776  656 S  0.0  0.3   0:01.46 init
    2 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kthreadd
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 events/0
    6 root      15  -5     0    0    0 S  0.0  0.0   0:00.06 khelper
   56 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/0
   62 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod
   80 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pdflush
   81 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pdflush
   82 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kswapd0
   83 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
   84 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 nfsiod
  215 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 mtdblockd
  227 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 hid_compat
  230 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 rpciod/0

~ # cat /proc/stat
cpu 209 0 0 16892 0 757 0 0 0
cpu0 209 0 0 16892 0 757 0 0 0
intr 79221 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 61353 0 0 0 0 0 17858 0 0 5 2 0 0 0 2 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 72511
btime 0
processes 360
procs_running 1
procs_blocked 0
~ #

~ # cat /proc/interrupts

M5445X    90:      62109 UART
M5445X    96:      19793 timer
M5445X    99:          6 timer
M5445X   100:          2 fec(TXF)
M5445X   101:          0 fec(TXB)
M5445X   102:          0 fec(TXFIFO)
M5445X   103:          0 fec(TXCR)
M5445X   104:          2 fec(RXF)
M5445X   105:          0 fec(RXB)
M5445X   106:          1 fec(MII)
M5445X   107:          0 fec(LC)
M5445X   108:          0 fec(HBERR)
M5445X   109:          0 fec(GRA)
M5445X   110:          0 fec(EBERR)
M5445X   111:          0 fec(BABT)
M5445X   112:          0 fec(BABR)

~ #

Run netcat test:

target: nc -l -p 3333 > /dev/null &
host: dd if=/dev/zero | nc <target-ip> 3333

Netcat test activity on ethernet ceases after only a few seconds. Wireshark concurs.

top - 00:05:04 up 5 min, 1 user, load average: 0.00, 0.04, 0.01

Tasks:  25 total,   1 running,  24 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.7%us,  0.0%sy,  0.0%ni, 98.7%id,  0.0%wa,  0.7%hi,  0.0%si, 
0.0%st
Mem:    254912k total,     6904k used,   248008k free,        0k buffers
Swap:        0k total,        0k used,        0k free,     3008k cached
  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
  363 root      20   0  2856 1160  944 R  1.0  0.5   0:00.27 top
  362 root      20   0  3280  688  576 S  0.3  0.3   0:00.05 nc
    1 root      20   0  3280  776  656 S  0.0  0.3   0:01.46 init
    2 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kthreadd
    3 root      15  -5     0    0    0 S  0.0  0.0   0:00.12 ksoftirqd/0
    4 root      RT  -5     0    0    0 S  0.0  0.0   0:00.00 watchdog/0
    5 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 events/0
    6 root      15  -5     0    0    0 S  0.0  0.0   0:00.06 khelper
   56 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kblockd/0
   62 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kseriod
   80 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pdflush
   81 root      20   0     0    0    0 S  0.0  0.0   0:00.00 pdflush
   82 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 kswapd0
   83 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 aio/0
   84 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 nfsiod
  215 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 mtdblockd
  227 root      15  -5     0    0    0 S  0.0  0.0   0:00.00 hid_compat

~ # cat /proc/stat
cpu 228 0 0 31924 0 778 0 0 0
cpu0 228 0 0 31924 0 778 0 0 0
intr 106918 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 73732 0 0 0 0 0 32930 0 0 10 96 0 0 0 149 0 1 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
ctxt 110292
btime 0
processes 364
procs_running 1
procs_blocked 0
~ #

~ # cat /proc/interrupts

M5445X    90:      74523 UART
M5445X    96:      34522 timer
M5445X    99:         10 timer
M5445X   100:        100 fec(TXF)
M5445X   101:          0 fec(TXB)
M5445X   102:          0 fec(TXFIFO)
M5445X   103:          0 fec(TXCR)
M5445X   104:        155 fec(RXF)
M5445X   105:          0 fec(RXB)
M5445X   106:          1 fec(MII)
M5445X   107:          0 fec(LC)
M5445X   108:          0 fec(HBERR)
M5445X   109:          0 fec(GRA)
M5445X   110:          0 fec(EBERR)
M5445X   111:          0 fec(BABT)
M5445X   112:          0 fec(BABR)

~ #

At this point, my proc entry in the fec driver does not indicate any problem:

~ # cat /proc/driver/fec
total interrupts: 284
last interrupt type: 2 [1=tx, 2=rx, 3=mii]
total tx interrupts: 111
total rx interrupts: 172
total mii interrupts: 1
last interrupt event: 0x2000000
total eberr interrupts: 0
total hberr interrupts: 0
tx loop current count: 0
tx loop last count: 1
rx loop current count: 0
rx loop last count: 1
rx last cbd ctrl/status: 0x800
rx last cbd len: 64
rx last cbd buff addr: 0x40407800
rx last netif_rx status: 0
~ #

When starting the netcat test, activity on the ethernet ceases after only a few seconds. After stopping the dd/nc on the host, I cannot ping the host from the target or vice versa and the rs-232 console running on the target does not respond to every key press (usually every second key press).

I instrumented some of the raw packet handling in the network stack to attempt to glean a little more info about ping. Running ping from target to host shows:

~ # ping 192.168.0.2
PING 192.168.0.2 (192.168.0.2): 56 data bytes

[  490.489407] ping:367 /mnt/nht/linux-2.6.29/net/core/datagram.c:99
wait_for_packet - interrupted
[  490.500812] ping:367 /mnt/nht/linux-2.6.29/net/core/datagram.c:110
wait_for_packet - timeo=2147483647, err=-512
[  490.513637] ping:367 /mnt/nht/linux-2.6.29/net/ipv4/raw.c:668
raw_recvmsg - skb_recv_datagram
[  490.524790] ping:367 /mnt/nht/linux-2.6.29/net/ipv4/raw.c:701
raw_recvmsg - returning err -512
[  490.536035] ping:367 /mnt/nht/linux-2.6.29/net/core/sock.c:1869
sock_common_recvmsg - recvmsg returned -512
[  490.548497] ping:367

/mnt/nht/linux-2.6.29/arch/m68k/coldfire/signal.c:804 ERESTART

Other tests which induce similar problems in the network stack include using wget to pull a large file onto the target.

I can also force netif_rx() to eventually constantly return 1 (pkt dropped) by using a flood ping at high frequency (see previous post). This is a different scenario that netcat/wget but possible related to same cause.

-- Matt

A message that is not in the archive (sent off-list or not yet received): [email protected]

#8 lore raw
Quoted 1 line · Lanttor

I met the interrupt issue before on mcf5445x platform, following is my fixed patch, but I am not sure it could help resolve your problem, just have a try :-)

Thanks but this makes no difference. From what I can tell, the patch to entry.S is essentially benign because ret_from_interrupt() is not invoked anywhere.

Also, the patch to ints.c adds invocations of irq_enter() and irq_exit() to process_int(). These macros really just force entry/exit to/from hard interrupt context and increment/decrement a count (used for tracking nested interrupts). Given that process_int() is called by inthandler() in entry.S, I think this is always going to be in interrupt context anyway, so I can't see the advantage of this. Maybe I'm missing something?

Quoted 2 lines

For your mcf54455 platform, if you use one ether net port, try to disable second ether net port (FEC2) on the menuconfig to see whether it would affect the net working.

Yes. I've already got the second net port disabled.

I have had some level of success by calling flush_cache_all() immediately after calling netif_rx() in the rx part of the fec interrupt handler - fec_enet_rx() in fec.c.

This plus a hack to netif_rx() seems to vastly improve things in that I can wget large files, run netcat tests although in the cast of netcat there is still a high number of rx overruns in the fec driver.

Hack:

net/core/dev.c 2009-06-10 10:26:47.000000000 +0100 renamed from net/core/dev.c 2009-05-12 07:02:24.000000000 +0100 +2 -0
@@ -1969,16 +1969,18 @@
                        __skb_queue_tail(&queue->input_pkt_queue, skb);
                        local_irq_restore(flags);
                        return NET_RX_SUCCESS;
                }
 
                napi_schedule(&queue->backlog);
                goto enqueue;
        }
+        else
+                napi_schedule(&queue->backlog);
 
        __get_cpu_var(netdev_rx_stat).dropped++;
        local_irq_restore(flags);
 
        kfree_skb(skb);
        return NET_RX_DROP;
 }
 

This effectively ignores the backlog check and schedules the receive function to run.

I think all this points to is that there is a timing problem in the system. That said, I can believe the cache coherency issue as I've read some postings on the net about similar issues with ethernet tx on ppc...

-- Matt

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw
Quoted 5 lines · Lanttor
Quoted 2 lines

Thanks but this makes no difference. From what I can tell, the patch to entry.S is essentially benign because ret_from_interrupt() is not invoked anywhere.

ret_from_interrupt() is the part of inthandler(), and it's called by inthandler() at end.

Indeed it is. My apologies.