Skip to contentSkip to conversation

at91sam9260 MACB problem with IP fragmentation

PATCH Applied Erwin Rol 3 participants 7 messages Dec 6, 2012 – Feb 13, 2013

Subsystems (3)
lore
Share & tools
lore

Erwin Rol

author
#0 lore raw

Hello Nicolas, Havard, all,

I have a very obscure problem with a at91sam9260 board (almost 1 to 1 copy of the Atmel EK).

The MACB seems to stall when I use large (>2 * MTU) UDP datagrams. The test case is that a udp echo client (PC) sends datagrams with increasing length to the AT91 until the max length of the UDP datagram is reached. When there is no IP fragmentation everything is fine, but when the datagrams are starting to get fragmented the AT91 will not reply anymore. But as soon as some network traffic happens it goes on again, and non of the data is lost.

With wireshark the effect can be easily seen (192.168.1.4 is the PC echo client, and 192.168.1.133 is the at91 echo server) After the first request there comes no reply. After a 5 second timeout the second request is send. And then both replies are returned.

When I enabled debugging output it all started to work. So I tried some udelays in the driver instead of printk and with a 1ms delay in the irq handler it started working. Of course that is an unacceptable fix, but it looks like that is some weird race condition that causes the sending to stall. The only difference with normal MTU sized datagrams I can think of is that the fragmented packets can be passed very quickly to the macb tx function, because the kernel has all 5 skb's ready.

I would be very interested to hear if someone else could reproduce this problem. Or even better, has seen this problem and has a fix for it.

I tried several kernels including the test version from Nicolas that he posted on LKML in October. They all show the same effect.

please CC me because I am currently not on the list.

- Erwin

The wireshark dump;

Quoted 21 lines
No.     Time           Source                Destination           Protocol Length Info
      1 0.000000000    192.168.1.4           192.168.1.133         IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=0, ID=0654) [Reassembled in #5]
      2 0.000123000    192.168.1.4           192.168.1.133         IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=1480, ID=0654) [Reassembled in #5]
      3 0.000113000    192.168.1.4           192.168.1.133         IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=2960, ID=0654) [Reassembled in #5]
      4 0.000147000    192.168.1.4           192.168.1.133         IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=4440, ID=0654) [Reassembled in #5]
      5 0.000114000    192.168.1.4           192.168.1.133         ECHO     1259   Request
      6 4.527395000    192.168.1.4           192.168.1.133         IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=0, ID=065d) [Reassembled in #10]
      7 0.000174000    192.168.1.4           192.168.1.133         IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=1480, ID=065d) [Reassembled in #10]
      8 0.000026000    192.168.1.4           192.168.1.133         IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=2960, ID=065d) [Reassembled in #10]
      9 0.000213000    192.168.1.4           192.168.1.133         IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=4440, ID=065d) [Reassembled in #10]
     10 0.000018000    192.168.1.4           192.168.1.133         ECHO     1260   Request
     11 0.001115000    192.168.1.133         192.168.1.4           IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=0, ID=c75d) [Reassembled in #15]
     12 0.000120000    192.168.1.133         192.168.1.4           IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=1480, ID=c75d) [Reassembled in #15]
     13 0.000205000    192.168.1.133         192.168.1.4           IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=2960, ID=c75d) [Reassembled in #15]
     14 0.000167000    192.168.1.133         192.168.1.4           IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=4440, ID=c75d) [Reassembled in #15]
     15 0.000006000    192.168.1.133         192.168.1.4           ECHO     1259   Response
     16 0.000396000    192.168.1.133         192.168.1.4           IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=0, ID=c75e) [Reassembled in #20]
     17 0.000224000    192.168.1.133         192.168.1.4           IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=1480, ID=c75e) [Reassembled in #20]
     18 0.000009000    192.168.1.133         192.168.1.4           IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=2960, ID=c75e) [Reassembled in #20]
     19 0.000237000    192.168.1.133         192.168.1.4           IPv4     1514   Fragmented IP protocol (proto=UDP 17, off=4440, ID=c75e) [Reassembled in #20]
     20 0.000009000    192.168.1.133         192.168.1.4           ECHO     1260   Response
#1 lore raw

Erwin,

On 12/06/2012 12:32 PM, Erwin Rol :

Quoted 26 lines

Hello Nicolas, Havard, all,

I have a very obscure problem with a at91sam9260 board (almost 1 to 1 copy of the Atmel EK).

The MACB seems to stall when I use large (>2 * MTU) UDP datagrams. The test case is that a udp echo client (PC) sends datagrams with increasing length to the AT91 until the max length of the UDP datagram is reached. When there is no IP fragmentation everything is fine, but when the datagrams are starting to get fragmented the AT91 will not reply anymore. But as soon as some network traffic happens it goes on again, and non of the data is lost.

With wireshark the effect can be easily seen (192.168.1.4 is the PC echo client, and 192.168.1.133 is the at91 echo server) After the first request there comes no reply. After a 5 second timeout the second request is send. And then both replies are returned.

When I enabled debugging output it all started to work. So I tried some udelays in the driver instead of printk and with a 1ms delay in the irq handler it started working. Of course that is an unacceptable fix, but it looks like that is some weird race condition that causes the sending to stall. The only difference with normal MTU sized datagrams I can think of is that the fragmented packets can be passed very quickly to the macb tx function, because the kernel has all 5 skb's ready.

Quoted 5 lines

I would be very interested to hear if someone else could reproduce this problem. Or even better, has seen this problem and has a fix for it.

I tried several kernels including the test version from Nicolas that he posted on LKML in October. They all show the same effect.

[..]

It seems that Matteo has the same behavior: check here:
http://www.spinics.net/lists/netdev/msg218951.html

I am working on the macb driver right now, so I will try to reproduce and track this issue on my side.

Best regards,

Signature · 1 line
Nicolas Ferre
author
#2 lore raw

Hey Nicolas,

Quoted 8 lines · Nicolas Ferre

Erwin,

On 12/06/2012 12:32 PM, Erwin Rol :

Quoted 5 lines

Hello Nicolas, Havard, all,

I have a very obscure problem with a at91sam9260 board (almost 1 to 1 copy of the Atmel EK).

Quoted 6 lines
Quoted 2 lines

<snip>

[..]

It seems that Matteo has the same behavior: check here:
http://www.spinics.net/lists/netdev/msg218951.html

The difference seems to be that in Matteo's case the receiving stalls. In my case it is the sending that stalls. I see the UDP datagram in userspace and the sendto call also returns without error, but the data does not end up on the network (until the next packet is send)

Quoted 2 lines

I am working on the macb driver right now, so I will try to reproduce and track this issue on my side.

That would be really great, and thank you for the quick reply. If you have anything that I should try or test on my hardware just let me know.

BTW: A quick check on a at91sam9263 board did not show the problem. I will try to verify if it really does work on a 9263, cause maybe it just more rare on a 9263.

- Erwin

author
#3 lore raw
In reply to Nicolas Ferre

Hallo Nicolas,

Quoted 15 lines · Nicolas Ferre

Erwin,

On 12/06/2012 12:32 PM, Erwin Rol :

Quoted 12 lines

Hello Nicolas, Havard, all,

I have a very obscure problem with a at91sam9260 board (almost 1 to 1 copy of the Atmel EK).

The MACB seems to stall when I use large (>2 * MTU) UDP datagrams. The test case is that a udp echo client (PC) sends datagrams with increasing length to the AT91 until the max length of the UDP datagram is reached. When there is no IP fragmentation everything is fine, but when the datagrams are starting to get fragmented the AT91 will not reply anymore. But as soon as some network traffic happens it goes on again, and non of the data is lost.

<snip>

Quoted 7 lines
Quoted 2 lines

I tried several kernels including the test version from Nicolas that he posted on LKML in October. They all show the same effect.

[..]

It seems that Matteo has the same behavior: check here:
http://www.spinics.net/lists/netdev/msg218951.html

I tried Matteo's patch and it seems to work. But I don't know if the patch is really the right solution. I checked again with wireshark and it really seems the sending that stalls not the receiving. But as soon as a ethernet frame is received the sending "un-stalls". So maybe the patch just causes an MACB IRQ at certain moments that causes the sending to continue?

Quoted 2 lines

I am working on the macb driver right now, so I will try to reproduce and track this issue on my side.

Any luck reproducing it ?

- Erwin

#4 lore raw

On 12/20/2012 10:17 AM, Erwin Rol :

Quoted 29 lines

Hallo Nicolas,

Quoted 15 lines · Nicolas Ferre

Erwin,

On 12/06/2012 12:32 PM, Erwin Rol :

Quoted 12 lines

Hello Nicolas, Havard, all,

I have a very obscure problem with a at91sam9260 board (almost 1 to 1 copy of the Atmel EK).

The MACB seems to stall when I use large (>2 * MTU) UDP datagrams. The test case is that a udp echo client (PC) sends datagrams with increasing length to the AT91 until the max length of the UDP datagram is reached. When there is no IP fragmentation everything is fine, but when the datagrams are starting to get fragmented the AT91 will not reply anymore. But as soon as some network traffic happens it goes on again, and non of the data is lost.

<snip>

Quoted 7 lines
Quoted 2 lines

I tried several kernels including the test version from Nicolas that he posted on LKML in October. They all show the same effect.

[..]

It seems that Matteo has the same behavior: check here:
http://www.spinics.net/lists/netdev/msg218951.html

Quoted 6 lines

I tried Matteo's patch and it seems to work. But I don't know if the patch is really the right solution. I checked again with wireshark and it really seems the sending that stalls not the receiving. But as soon as a ethernet frame is received the sending "un-stalls". So maybe the patch just causes an MACB IRQ at certain moments that causes the sending to continue?

Any digging is interesting for me.

Quoted 4 lines
Quoted 2 lines

I am working on the macb driver right now, so I will try to reproduce and track this issue on my side.

Any luck reproducing it ?

Yes, I see unexpected things happening but as I am connected to a whole company network so maybe some broadcast packets are unlocking the interface...
Anyway, I am continuing to investigate.

Best regards,--
Nicolas Ferre

#5 lore raw

PATCH 1/1 net/macb: fix race with RX interrupt while doing NAPI

In reply to Erwin Rol

When interrupts are disabled, an RX condition can occur but it is not reported when enabling interrupts again. We need to check RSR and use napi_reschedule() if condition is met.

---
 drivers/net/ethernet/cadence/macb.c | 5 +++++
 1 file changed, 5 insertions(+)
drivers/net/ethernet/cadence/macb.c +5 -0
index a9b0830..b9d4bb9 100644
@@ -693,6 +693,11 @@ static int macb_poll(struct napi_struct *napi, int budget)
 		 * get notified when new packets arrive.
 		 */
 		macb_writel(bp, IER, MACB_RX_INT_FLAGS);
+
+		/* Packets received while interrupts were disabled */
+		status = macb_readl(bp, RSR);
+		if (unlikely(status))
+			napi_reschedule(napi);
 	}
 
 	/* TODO: Handle errors */
Signature · 1 line
1.8.0
maintainer Applied
#6 lore raw

Re: [PATCH] net/macb: fix race with RX interrupt while doing NAPI

From: Nicolas Ferre <[email protected]>
Date: Tue, 12 Feb 2013 11:08:48 +0100

Quoted 5 lines

When interrupts are disabled, an RX condition can occur but it is not reported when enabling interrupts again. We need to check RSR and use napi_reschedule() if condition is met.

Applied.