at91sam9260 MACB problem with IP fragmentation
Subsystems (3)
Hello Nicolas, Havard, all,
I have a very obscure problem with a at91sam9260 board (almost 1 to 1 copy of the Atmel EK).
The MACB seems to stall when I use large (>2 * MTU) UDP datagrams. The test case is that a udp echo client (PC) sends datagrams with increasing length to the AT91 until the max length of the UDP datagram is reached. When there is no IP fragmentation everything is fine, but when the datagrams are starting to get fragmented the AT91 will not reply anymore. But as soon as some network traffic happens it goes on again, and non of the data is lost.
With wireshark the effect can be easily seen (192.168.1.4 is the PC echo client, and 192.168.1.133 is the at91 echo server) After the first request there comes no reply. After a 5 second timeout the second request is send. And then both replies are returned.
When I enabled debugging output it all started to work. So I tried some udelays in the driver instead of printk and with a 1ms delay in the irq handler it started working. Of course that is an unacceptable fix, but it looks like that is some weird race condition that causes the sending to stall. The only difference with normal MTU sized datagrams I can think of is that the fragmented packets can be passed very quickly to the macb tx function, because the kernel has all 5 skb's ready.
I would be very interested to hear if someone else could reproduce this problem. Or even better, has seen this problem and has a fix for it.
I tried several kernels including the test version from Nicolas that he posted on LKML in October. They all show the same effect.
please CC me because I am currently not on the list.
- Erwin
The wireshark dump;
Quoted 21 lines
No. Time Source Destination Protocol Length Info
1 0.000000000 192.168.1.4 192.168.1.133 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=0, ID=0654) [Reassembled in #5]
2 0.000123000 192.168.1.4 192.168.1.133 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=1480, ID=0654) [Reassembled in #5]
3 0.000113000 192.168.1.4 192.168.1.133 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=2960, ID=0654) [Reassembled in #5]
4 0.000147000 192.168.1.4 192.168.1.133 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=4440, ID=0654) [Reassembled in #5]
5 0.000114000 192.168.1.4 192.168.1.133 ECHO 1259 Request
6 4.527395000 192.168.1.4 192.168.1.133 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=0, ID=065d) [Reassembled in #10]
7 0.000174000 192.168.1.4 192.168.1.133 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=1480, ID=065d) [Reassembled in #10]
8 0.000026000 192.168.1.4 192.168.1.133 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=2960, ID=065d) [Reassembled in #10]
9 0.000213000 192.168.1.4 192.168.1.133 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=4440, ID=065d) [Reassembled in #10]
10 0.000018000 192.168.1.4 192.168.1.133 ECHO 1260 Request
11 0.001115000 192.168.1.133 192.168.1.4 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=0, ID=c75d) [Reassembled in #15]
12 0.000120000 192.168.1.133 192.168.1.4 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=1480, ID=c75d) [Reassembled in #15]
13 0.000205000 192.168.1.133 192.168.1.4 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=2960, ID=c75d) [Reassembled in #15]
14 0.000167000 192.168.1.133 192.168.1.4 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=4440, ID=c75d) [Reassembled in #15]
15 0.000006000 192.168.1.133 192.168.1.4 ECHO 1259 Response
16 0.000396000 192.168.1.133 192.168.1.4 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=0, ID=c75e) [Reassembled in #20]
17 0.000224000 192.168.1.133 192.168.1.4 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=1480, ID=c75e) [Reassembled in #20]
18 0.000009000 192.168.1.133 192.168.1.4 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=2960, ID=c75e) [Reassembled in #20]
19 0.000237000 192.168.1.133 192.168.1.4 IPv4 1514 Fragmented IP protocol (proto=UDP 17, off=4440, ID=c75e) [Reassembled in #20]
20 0.000009000 192.168.1.133 192.168.1.4 ECHO 1260 ResponseErwin,
On 12/06/2012 12:32 PM, Erwin Rol :
Quoted 26 lines
Hello Nicolas, Havard, all,
I have a very obscure problem with a at91sam9260 board (almost 1 to 1 copy of the Atmel EK).
The MACB seems to stall when I use large (>2 * MTU) UDP datagrams. The test case is that a udp echo client (PC) sends datagrams with increasing length to the AT91 until the max length of the UDP datagram is reached. When there is no IP fragmentation everything is fine, but when the datagrams are starting to get fragmented the AT91 will not reply anymore. But as soon as some network traffic happens it goes on again, and non of the data is lost.
With wireshark the effect can be easily seen (192.168.1.4 is the PC echo client, and 192.168.1.133 is the at91 echo server) After the first request there comes no reply. After a 5 second timeout the second request is send. And then both replies are returned.
When I enabled debugging output it all started to work. So I tried some udelays in the driver instead of printk and with a 1ms delay in the irq handler it started working. Of course that is an unacceptable fix, but it looks like that is some weird race condition that causes the sending to stall. The only difference with normal MTU sized datagrams I can think of is that the fragmented packets can be passed very quickly to the macb tx function, because the kernel has all 5 skb's ready.
Quoted 5 lines
I would be very interested to hear if someone else could reproduce this problem. Or even better, has seen this problem and has a fix for it.
I tried several kernels including the test version from Nicolas that he posted on LKML in October. They all show the same effect.
[..]
It seems that Matteo has the same behavior: check here:
http://www.spinics.net/lists/netdev/msg218951.html
I am working on the macb driver right now, so I will try to reproduce and track this issue on my side.
Best regards,
Signature · 1 line
Nicolas Ferre
Hey Nicolas,
Quoted 8 lines · Nicolas Ferre
Erwin,
On 12/06/2012 12:32 PM, Erwin Rol :
Quoted 5 lines
Hello Nicolas, Havard, all,
I have a very obscure problem with a at91sam9260 board (almost 1 to 1 copy of the Atmel EK).
Quoted 6 lines
Quoted 2 lines
<snip>
[..]
It seems that Matteo has the same behavior: check here:
http://www.spinics.net/lists/netdev/msg218951.html
The difference seems to be that in Matteo's case the receiving stalls. In my case it is the sending that stalls. I see the UDP datagram in userspace and the sendto call also returns without error, but the data does not end up on the network (until the next packet is send)
Quoted 2 lines
I am working on the macb driver right now, so I will try to reproduce and track this issue on my side.
That would be really great, and thank you for the quick reply. If you have anything that I should try or test on my hardware just let me know.
BTW: A quick check on a at91sam9263 board did not show the problem. I will try to verify if it really does work on a 9263, cause maybe it just more rare on a 9263.
- Erwin
Hallo Nicolas,
Quoted 15 lines · Nicolas Ferre
Erwin,
On 12/06/2012 12:32 PM, Erwin Rol :
Quoted 12 lines
Hello Nicolas, Havard, all,
I have a very obscure problem with a at91sam9260 board (almost 1 to 1 copy of the Atmel EK).
The MACB seems to stall when I use large (>2 * MTU) UDP datagrams. The test case is that a udp echo client (PC) sends datagrams with increasing length to the AT91 until the max length of the UDP datagram is reached. When there is no IP fragmentation everything is fine, but when the datagrams are starting to get fragmented the AT91 will not reply anymore. But as soon as some network traffic happens it goes on again, and non of the data is lost.
<snip>
Quoted 7 lines
Quoted 2 lines
I tried several kernels including the test version from Nicolas that he posted on LKML in October. They all show the same effect.
[..]
It seems that Matteo has the same behavior: check here:
http://www.spinics.net/lists/netdev/msg218951.html
I tried Matteo's patch and it seems to work. But I don't know if the patch is really the right solution. I checked again with wireshark and it really seems the sending that stalls not the receiving. But as soon as a ethernet frame is received the sending "un-stalls". So maybe the patch just causes an MACB IRQ at certain moments that causes the sending to continue?
Quoted 2 lines
I am working on the macb driver right now, so I will try to reproduce and track this issue on my side.
Any luck reproducing it ?
- Erwin
On 12/20/2012 10:17 AM, Erwin Rol :
Quoted 29 lines
Hallo Nicolas,
Quoted 15 lines · Nicolas Ferre
Erwin,
On 12/06/2012 12:32 PM, Erwin Rol :
Quoted 12 lines
Hello Nicolas, Havard, all,
I have a very obscure problem with a at91sam9260 board (almost 1 to 1 copy of the Atmel EK).
The MACB seems to stall when I use large (>2 * MTU) UDP datagrams. The test case is that a udp echo client (PC) sends datagrams with increasing length to the AT91 until the max length of the UDP datagram is reached. When there is no IP fragmentation everything is fine, but when the datagrams are starting to get fragmented the AT91 will not reply anymore. But as soon as some network traffic happens it goes on again, and non of the data is lost.
<snip>
Quoted 7 lines
Quoted 2 lines
I tried several kernels including the test version from Nicolas that he posted on LKML in October. They all show the same effect.
[..]
It seems that Matteo has the same behavior: check here:
http://www.spinics.net/lists/netdev/msg218951.html
Quoted 6 lines
I tried Matteo's patch and it seems to work. But I don't know if the patch is really the right solution. I checked again with wireshark and it really seems the sending that stalls not the receiving. But as soon as a ethernet frame is received the sending "un-stalls". So maybe the patch just causes an MACB IRQ at certain moments that causes the sending to continue?
Any digging is interesting for me.
Quoted 4 lines
Quoted 2 lines
I am working on the macb driver right now, so I will try to reproduce and track this issue on my side.
Any luck reproducing it ?
Yes, I see unexpected things happening but as I am connected to a whole company network so maybe some broadcast packets are unlocking the interface...
Anyway, I am continuing to investigate.
Best regards,--
Nicolas Ferre
PATCH 1/1 net/macb: fix race with RX interrupt while doing NAPI
In reply to Erwin RolWhen interrupts are disabled, an RX condition can occur but it is not reported when enabling interrupts again. We need to check RSR and use napi_reschedule() if condition is met.
--- drivers/net/ethernet/cadence/macb.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/net/ethernet/cadence/macb.c +5 -0
@@ -693,6 +693,11 @@ static int macb_poll(struct napi_struct *napi, int budget)* get notified when new packets arrive. */ macb_writel (bp,IER ,MACB_RX_INT_FLAGS ); + +/* Packets received while interrupts were disabled */ + status =macb_readl (bp,RSR ); +if (unlikely (status)) +napi_reschedule (napi); }/* TODO: Handle errors */
Signature · 1 line
1.8.0
Re: [PATCH] net/macb: fix race with RX interrupt while doing NAPI
From: Nicolas Ferre <[email protected]>
Date: Tue, 12 Feb 2013 11:08:48 +0100
Quoted 5 lines
When interrupts are disabled, an RX condition can occur but it is not reported when enabling interrupts again. We need to check RSR and use
napi_reschedule()if condition is met.Signed-off-by Nicolas Ferre <[email protected]>
Applied.