Skip to contentSkip to conversation

Bestcomm trouble with NAPI for MPC5200 FEC

DISCUSSION Wolfgang Grandegger 3 participants 6 messages Jul 9, 2009 – Jul 10, 2009 Network drivers

lore
Share & tools
lore
#0 lore raw

Hello,

I'm currently trying to implement NAPI for the FEC on the MPC5200 to solve the well known problem, that network packet storms can cause interrupt flooding, which may totally block the system. The NAPI implementation, in principle, is straight forward and works well under normal and moderate network load. It just calls disable_irq() in the receive interrupt handler to defer packet processing to the NAPI poll callback, which calls enable_irq() when it has processed all packets. Unfortunately, under heavy network load (packet storm), problems show up:

- With DENX 2.4.25, the Bestcomm RX task gets and remains stopped after a while under additional system load. I have no idea how and when Bestcom tasks are stopped. In the auto-start mode, the firmware should poll forever for the next free descriptor block.

- With 2.6.31-rc2, the RFIFO error occurs quickly which does reset the FEC and Bestcomm (unfortunately, this does trigger an oops because it's called from the interrupt context, but that's another issue).

I'm realized that working with Bestcomm is a pain :-( but so far I have little knowledge of the Bestcomm limitations and quirks. Any idea what might go wrong or how to implement NAPI for that FEC properly.

TIA,

Wolfgang.

#1 lore raw

On Thu, Jul 9, 2009 at 2:33 PM, Wolfgang Grandegger<[email protected]> wrot= e:

Quoted 5 lines

Hello,

I'm currently trying to implement NAPI for the FEC on the MPC5200 to solve the well known problem, that network packet storms can cause interrupt flooding, which may totally block the system.

Good to hear it! Thanks for this work.

Quoted 13 lines

The NAPI
implementation, in principle, is straight forward and works well under normal and moderate network load. It just calls disable_irq() in the receive interrupt handler to defer packet processing to the NAPI poll callback, which calls enable_irq() when it has processed all packets. Unfortunately, under heavy network load (packet storm), problems show up:

- With DENX 2.4.25, the Bestcomm RX task gets and remains stopped after =A0a while under additional system load. I have no idea how and when =A0Bestcom tasks are stopped. In the auto-start mode, the firmware should =A0poll forever for the next free descriptor block.

Quoted 7 lines

- With 2.6.31-rc2, the RFIFO error occurs quickly which does reset the =A0FEC and Bestcomm (unfortunately, this does trigger an oops because =A0it's called from the interrupt context, but that's another issue).

I'm realized that working with Bestcomm is a pain :-( but so far I have little knowledge of the Bestcomm limitations and quirks. Any idea what might go wrong or how to implement NAPI for that FEC properly.

Yes, I have a few ideas. First, I suspect that the FEC rx queue isn't big enough and I wouldn't be surprised if the RFIFO error is occurring because Bestcomm gets overrun. This scenario needs to be handled more gracefully.

Second, I think resetting the PHY should be removed from the reset path. The phy doesn't at all need to be reset and doing this would avoid the OOPS condition. Also, in the RFIFO error path needs to be audited to make sure that all the good received packets are processed correctly before resetting the BCOM engine and to make sure that skbufs are not getting leaked.

Essentially, I think that the RFIFO error condition is currently handled in far too heavy handed a manner and it should not be expensive to recover from.

Thanks for this work!
g..

--=20
Grant Likely, B.Sc., P.Eng.
Secret Lab Technologies Ltd.

#2 lore raw
Quoted 21 lines · Grant Likely
Quoted 5 lines · Wolfgang Grandegger

Hello,

I'm currently trying to implement NAPI for the FEC on the MPC5200 to solve the well known problem, that network packet storms can cause interrupt flooding, which may totally block the system.

Good to hear it! Thanks for this work.

Quoted 12 lines

The NAPI
implementation, in principle, is straight forward and works well under normal and moderate network load. It just calls disable_irq() in the receive interrupt handler to defer packet processing to the NAPI poll callback, which calls enable_irq() when it has processed all packets. Unfortunately, under heavy network load (packet storm), problems show up:

- With DENX 2.4.25, the Bestcomm RX task gets and remains stopped after a while under additional system load. I have no idea how and when Bestcom tasks are stopped. In the auto-start mode, the firmware should poll forever for the next free descriptor block.

Do you know when the Bestcomm firmware does stop the task? I have the impression that it happens when all buffer descriptors are used (RX queue full).

Quoted 4 lines
Quoted 4 lines

- With 2.6.31-rc2, the RFIFO error occurs quickly which does reset the FEC and Bestcomm (unfortunately, this does trigger an oops because it's called from the interrupt context, but that's another issue).

Quoted 8 lines
Quoted 3 lines

I'm realized that working with Bestcomm is a pain :-( but so far I have little knowledge of the Bestcomm limitations and quirks. Any idea what might go wrong or how to implement NAPI for that FEC properly.

Yes, I have a few ideas. First, I suspect that the FEC rx queue isn't big enough and I wouldn't be surprised if the RFIFO error is occurring because Bestcomm gets overrun. This scenario needs to be handled more gracefully.

The RFIFO error does not show up with DENX 2.4.25 and therefore I'm not sure if overruns are a real problem.

Quoted 6 lines

Second, I think resetting the PHY should be removed from the reset path. The phy doesn't at all need to be reset and doing this would avoid the OOPS condition. Also, in the RFIFO error path needs to be audited to make sure that all the good received packets are processed correctly before resetting the BCOM engine and to make sure that skbufs are not getting leaked.

Agreed, the manual says: "When this occurs, software must ensure both the FIFO Controller and BestComm are soft-reset."

Quoted 3 lines

Essentially, I think that the RFIFO error condition is currently handled in far too heavy handed a manner and it should not be expensive to recover from.

Yep, it looks like.

Wolfgang.

#3 lore raw
In reply to Grant Likely
Quoted 26 lines · Grant Likely
Quoted 5 lines · Wolfgang Grandegger

Hello,

I'm currently trying to implement NAPI for the FEC on the MPC5200 to solve the well known problem, that network packet storms can cause interrupt flooding, which may totally block the system.

Good to hear it! Thanks for this work.

Quoted 17 lines

The NAPI
implementation, in principle, is straight forward and works well under normal and moderate network load. It just calls disable_irq() in the receive interrupt handler to defer packet processing to the NAPI poll callback, which calls enable_irq() when it has processed all packets. Unfortunately, under heavy network load (packet storm), problems show up:

- With DENX 2.4.25, the Bestcomm RX task gets and remains stopped after a while under additional system load. I have no idea how and when Bestcom tasks are stopped. In the auto-start mode, the firmware should poll forever for the next free descriptor block.

- With 2.6.31-rc2, the RFIFO error occurs quickly which does reset the FEC and Bestcomm (unfortunately, this does trigger an oops because it's called from the interrupt context, but that's another issue).

Quoted 8 lines
Quoted 3 lines

I'm realized that working with Bestcomm is a pain :-( but so far I have little knowledge of the Bestcomm limitations and quirks. Any idea what might go wrong or how to implement NAPI for that FEC properly.

Yes, I have a few ideas. First, I suspect that the FEC rx queue isn't big enough and I wouldn't be surprised if the RFIFO error is occurring because Bestcomm gets overrun. This scenario needs to be handled more gracefully.

First some words concerning NAPI. NAPI is mainly used to improve network performance by processing network packets in the process context while reducing interrupt load at the same time. Thereby it also solves the problem of interrupt flooding, which may totally block the system. Most (maybe all?) Gigabit Ethernet drivers use NAPI, e.g. ucc_geth. Below I have attached my preliminary (and not yet complete or even correct) patch, which should demonstrate how NAPI is supposed to work. The old NAPI implementation of 2.4 is documented here:

http://lxr.linux.no/linux-old+v2.4.31/Documentation/networking/NAPI_HOWTO.txt.

As the NAPI polling competes with other task/processes, it's clear that a bigger queue only helps partially.

Wolfgang.

---
 drivers/net/Kconfig       |    7 ++++
 drivers/net/fec_mpc52xx.c |   76 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)
drivers/net/Kconfig +7 -0
Index: linux-2.6-denx/drivers/net/Kconfig
===================================================================
@@ -1896,6 +1896,13 @@ config FEC_MPC52xx
 	  Fast Ethernet Controller
 	  If compiled as module, it will be called fec_mpc52xx.
 
+config FEC_MPC52xx_NAPI
+	bool "Use NAPI for MPC52xx FEC driver"
+	depends on FEC_MPC52xx
+	---help---
+	  This option enables NAPI support for the MPC5200's on-chip
+	  Fast Ethernet Controller driver.
+
 config FEC_MPC52xx_MDIO
 	bool "MPC52xx FEC MDIO bus driver"
 	depends on FEC_MPC52xx
drivers/net/fec_mpc52xx.c +76 -0

151 lines not on this page · raw message

---
 drivers/net/Kconfig       |    7 ++++
 drivers/net/fec_mpc52xx.c |   76 ++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 83 insertions(+)
drivers/net/Kconfig +7 -0
Index: linux-2.6-denx/drivers/net/Kconfig
===================================================================
@@ -1896,6 +1896,13 @@ config FEC_MPC52xx
 	  Fast Ethernet Controller
 	  If compiled as module, it will be called fec_mpc52xx.
 
+config FEC_MPC52xx_NAPI
+	bool "Use NAPI for MPC52xx FEC driver"
+	depends on FEC_MPC52xx
+	---help---
+	  This option enables NAPI support for the MPC5200's on-chip
+	  Fast Ethernet Controller driver.
+
 config FEC_MPC52xx_MDIO
 	bool "MPC52xx FEC MDIO bus driver"
 	depends on FEC_MPC52xx
drivers/net/fec_mpc52xx.c +76 -0

151 lines not on this page · raw message

#4 lore raw
In reply to Wolfgang Grandegger

Hello Wolfgang,

Quoted 3 lines

I'm currently trying to implement NAPI for the FEC on the MPC5200 to solve the well known problem, that network packet storms can cause interrupt flooding, which may totally block the system. The NAPI

That's great news. Do you happen to have a scenario which reliably triggers the problem? Then I could do some testing as well. I haven't seen this on our boards for quite a while, but I'd like to verify.

Quoted 3 lines

- With 2.6.31-rc2, the RFIFO error occurs quickly which does reset the FEC and Bestcomm (unfortunately, this does trigger an oops because it's called from the interrupt context, but that's another issue).

+1 for "error handling resets too much"

Regards,

Wolfram

Signature · 2 lines
Pengutronix e.K.                           | Wolfram Sang                |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
#5 lore raw
Quoted 5 lines · Wolfram Sang

Hello Wolfgang,

Quoted 3 lines

I'm currently trying to implement NAPI for the FEC on the MPC5200 to solve the well known problem, that network packet storms can cause interrupt flooding, which may totally block the system. The NAPI

Just bombard the FEC with network packets. I use pktgen on the host and I could send you a script if you like.

Quoted 9 lines

That's great news. Do you happen to have a scenario which reliably triggers the problem? Then I could do some testing as well. I haven't seen this on our boards for quite a while, but I'd like to verify.

Quoted 3 lines

- With 2.6.31-rc2, the RFIFO error occurs quickly which does reset the FEC and Bestcomm (unfortunately, this does trigger an oops because it's called from the interrupt context, but that's another issue).

+1 for "error handling resets too much"

Yes, and furthermore it's leaking skb's :-(.

Wolfgang.