can: mcp251xfd: fix infinite loop when xmit fails

From: Vitor Soares <[email protected]>

When the mcp251xfd_start_xmit() function fails, the driver stops processing messages, and the interrupt routine does not return, running indefinitely even after killing the running application.

Error messages:
[  441.298819] mcp251xfd spi2.0 can0: ERROR in mcp251xfd_start_xmit:
-16
[  441.306498] mcp251xfd spi2.0 can0: Transmit Event FIFO buffer not empty. (seq=0x000017c7, tef_tail=0x000017cf, tef_head=0x000017d0, tx_head=0x000017d3).
... and repeat forever.

The issue can be triggered when multiple devices share the same SPI interface. And there is concurrent access to the bus.

The problem occurs because tx_ring->head increments even if mcp251xfd_start_xmit() fails. Consequently, the driver skips one TX package while still expecting a response in mcp251xfd_handle_tefif_one().

This patch resolves the issue by decreasing tx_ring->head if mcp251xfd_start_xmit() fails. With the fix, if we trigger the issue and
the err = -EBUSY, the driver returns NETDEV_TX_BUSY. The network stack
retries to transmit the message.
Otherwise, it prints an error and discards the message.

Fixes 55e5b97f003e ("can: mcp25xxfd: add driver for Microchip

MCP25xxFD SPI CAN")

Cc [email protected]

Signed-off-by Vitor Soares <[email protected]>

---

V2->V3:
  - Add tx_dropped stats.
  - netdev_sent_queue() only if can_put_echo_skb() succeed.

V1->V2:
  - Return NETDEV_TX_BUSY if mcp251xfd_tx_obj_write() == -EBUSY.
  - Rework the commit message to address the change above.
  - Change can_put_echo_skb() to be called after mcp251xfd_tx_obj_write() succeed.
    Otherwise, we get Kernel NULL pointer dereference error.

 drivers/net/can/spi/mcp251xfd/mcp251xfd-tx.c | 34 ++++++++++++------ --

 1 file changed, 21 insertions(+), 13 deletions(-)

drivers/net/can/spi/mcp251xfd/mcp251xfd-tx.c +1 -0

index 160528d3cc26..146c44e47c60 100644
@@ -166,6 +166,7 @@ netdev_tx_t mcp251xfd_start_xmit(struct sk_buff
 *skb,
                                  struct net_device *ndev)
  {
         struct mcp251xfd_priv *priv = netdev_priv(ndev);
+       struct net_device_stats *stats = &ndev->stats;
         struct mcp251xfd_tx_ring *tx_ring = priv->tx;
         struct mcp251xfd_tx_obj *tx_obj;

        unsigned int frame_len;

drivers/net/can/spi/mcp251xfd/mcp251xfd-tx.c +0 -0

@@ -181,25 +182,32 @@ netdev_tx_t mcp251xfd_start_xmit(struct sk_buff
 *skb,
         tx_obj = mcp251xfd_get_tx_obj_next(tx_ring);
         mcp251xfd_tx_obj_from_skb(priv, tx_obj, skb, tx_ring->head);

-       /* Stop queue if we occupy the complete TX FIFO */ tx_head = mcp251xfd_get_tx_head(tx_ring);
-       tx_ring->head++;
-       if (mcp251xfd_get_tx_free(tx_ring) == 0)
-               netif_stop_queue(ndev);
-
        frame_len = can_skb_get_frame_len(skb);
-       err = can_put_echo_skb(skb, ndev, tx_head, frame_len); -       if (!err)
-               netdev_sent_queue(priv->ndev, frame_len); +
+       tx_ring->head++;

        err = mcp251xfd_tx_obj_write(priv, tx_obj);
-       if (err)
-               goto out_err;
+       if (err) {
+               tx_ring->head--;

-       return NETDEV_TX_OK;
+               if (err == -EBUSY)
+                       return NETDEV_TX_BUSY;

- out_err:
-       netdev_err(priv->ndev, "ERROR in %s: %d\n", __func__, err); +               stats->tx_dropped++;
+
+               if (net_ratelimit())
+                       netdev_err(priv->ndev,
+                                  "ERROR in %s: %d\n", __func__, err);
+       } else {
+               err = can_put_echo_skb(skb, ndev, tx_head, frame_len);
+               if (!err)
+                       netdev_sent_queue(priv->ndev, frame_len); +
+               /* Stop queue if we occupy the complete TX FIFO */ +               if (mcp251xfd_get_tx_free(tx_ring) == 0) +                       netif_stop_queue(ndev);
+       }

        return NETDEV_TX_OK;
 }

From: Vitor Soares <[email protected]>

When the mcp251xfd_start_xmit() function fails, the driver stops processing messages, and the interrupt routine does not return, running indefinitely even after killing the running application.

Error messages:

[  441.298819] mcp251xfd spi2.0 can0: ERROR in mcp251xfd_start_xmit: -16
[  441.306498] mcp251xfd spi2.0 can0: Transmit Event FIFO buffer not empty. (seq=0x000017c7, tef_tail=0x000017cf, tef_head=0x000017d0, tx_head=0x000017d3).

... and repeat forever.

The issue can be triggered when multiple devices share the same SPI interface. And there is concurrent access to the bus.

The problem occurs because tx_ring->head increments even if mcp251xfd_start_xmit() fails. Consequently, the driver skips one TX package while still expecting a response in mcp251xfd_handle_tefif_one().

This patch resolves the issue by decreasing tx_ring->head if mcp251xfd_start_xmit() fails. With the fix, if we trigger the issue and the err = -EBUSY, the driver returns NETDEV_TX_BUSY. The network stack retries to transmit the message.
Otherwise, it prints an error and discards the message.

Fixes 55e5b97f003e ("can: mcp25xxfd: add driver for Microchip MCP25xxFD SPI CAN")

Cc [email protected]

Signed-off-by Vitor Soares <[email protected]>

From: Vitor Soares <[email protected]>

When the mcp251xfd_start_xmit() function fails, the driver stops processing messages, and the interrupt routine does not return, running indefinitely even after killing the running application.

Error messages:

[  441.298819] mcp251xfd spi2.0 can0: ERROR in mcp251xfd_start_xmit: -16
[  441.306498] mcp251xfd spi2.0 can0: Transmit Event FIFO buffer not empty. (seq=0x000017c7, tef_tail=0x000017cf, tef_head=0x000017d0, tx_head=0x000017d3).

... and repeat forever.

The issue can be triggered when multiple devices share the same SPI interface. And there is concurrent access to the bus.

The problem occurs because tx_ring->head increments even if mcp251xfd_start_xmit() fails. Consequently, the driver skips one TX package while still expecting a response in mcp251xfd_handle_tefif_one().

This patch resolves the issue by decreasing tx_ring->head if mcp251xfd_start_xmit() fails. With the fix, if we trigger the issue and the err = -EBUSY, the driver returns NETDEV_TX_BUSY. The network stack retries to transmit the message.
Otherwise, it prints an error and discards the message.

Fixes 55e5b97f003e ("can: mcp25xxfd: add driver for Microchip MCP25xxFD SPI CAN")

Cc [email protected]

Signed-off-by Vitor Soares <[email protected]>

---

V2->V3:
- Add tx_dropped stats.
- netdev_sent_queue() only if can_put_echo_skb() succeed.

V1->V2:
- Return NETDEV_TX_BUSY if mcp251xfd_tx_obj_write() == -EBUSY.
- Rework the commit message to address the change above.
- Change can_put_echo_skb() to be called after mcp251xfd_tx_obj_write() succeed. Otherwise, we get Kernel NULL pointer dereference error.

 drivers/net/can/spi/mcp251xfd/mcp251xfd-tx.c | 34 ++++++++++++--------
 1 file changed, 21 insertions(+), 13 deletions(-)

drivers/net/can/spi/mcp251xfd/mcp251xfd-tx.c +11 -13

index 160528d3cc26..146c44e47c60 100644
@@ -166,6 +166,7 @@ netdev_tx_t mcp251xfd_start_xmit(struct sk_buff *skb,
 				 struct net_device *ndev)
 {
 	struct mcp251xfd_priv *priv = netdev_priv(ndev);
+	struct net_device_stats *stats = &ndev->stats;
 	struct mcp251xfd_tx_ring *tx_ring = priv->tx;
 	struct mcp251xfd_tx_obj *tx_obj;
 	unsigned int frame_len;

@@ -181,25 +182,32 @@ netdev_tx_t mcp251xfd_start_xmit(struct sk_buff *skb,
 	tx_obj = mcp251xfd_get_tx_obj_next(tx_ring);
 	mcp251xfd_tx_obj_from_skb(priv, tx_obj, skb, tx_ring->head);
 
-	/* Stop queue if we occupy the complete TX FIFO */
 	tx_head = mcp251xfd_get_tx_head(tx_ring);
-	tx_ring->head++;
-	if (mcp251xfd_get_tx_free(tx_ring) == 0)
-		netif_stop_queue(ndev);
-
 	frame_len = can_skb_get_frame_len(skb);
-	err = can_put_echo_skb(skb, ndev, tx_head, frame_len);
-	if (!err)
-		netdev_sent_queue(priv->ndev, frame_len);
+
+	tx_ring->head++;
 
 	err = mcp251xfd_tx_obj_write(priv, tx_obj);
-	if (err)
-		goto out_err;
+	if (err) {
+		tx_ring->head--;
 
-	return NETDEV_TX_OK;
+		if (err == -EBUSY)
+			return NETDEV_TX_BUSY;
 
- out_err:
-	netdev_err(priv->ndev, "ERROR in %s: %d\n", __func__, err);
+		stats->tx_dropped++;
+
+		if (net_ratelimit())
+			netdev_err(priv->ndev,

Quoted 90 lines · Vitor Soares

From: Vitor Soares <[email protected]>

When the mcp251xfd_start_xmit() function fails, the driver stops processing messages, and the interrupt routine does not return, running indefinitely even after killing the running application.

Error messages:
[  441.298819] mcp251xfd spi2.0 can0: ERROR in mcp251xfd_start_xmit: -16
[  441.306498] mcp251xfd spi2.0 can0: Transmit Event FIFO buffer not empty. (seq=0x000017c7, tef_tail=0x000017cf, tef_head=0x000017d0, tx_head=0x000017d3).
... and repeat forever.

The issue can be triggered when multiple devices share the same SPI interface. And there is concurrent access to the bus.

The problem occurs because tx_ring->head increments even if mcp251xfd_start_xmit() fails. Consequently, the driver skips one TX package while still expecting a response in mcp251xfd_handle_tefif_one().

This patch resolves the issue by decreasing tx_ring->head if mcp251xfd_start_xmit() fails. With the fix, if we trigger the issue and the err = -EBUSY, the driver returns NETDEV_TX_BUSY. The network stack retries to transmit the message.
Otherwise, it prints an error and discards the message.

Fixes 55e5b97f003e ("can: mcp25xxfd: add driver for Microchip MCP25xxFD SPI CAN")

Cc [email protected]

Signed-off-by Vitor Soares <[email protected]>

---

V2->V3:
  - Add tx_dropped stats.
  - netdev_sent_queue() only if can_put_echo_skb() succeed.

V1->V2:
  - Return NETDEV_TX_BUSY if mcp251xfd_tx_obj_write() == -EBUSY.
  - Rework the commit message to address the change above.
  - Change can_put_echo_skb() to be called after mcp251xfd_tx_obj_write() succeed. Otherwise, we get Kernel NULL pointer dereference error.

 drivers/net/can/spi/mcp251xfd/mcp251xfd-tx.c | 34 ++++++++++++--------

 1 file changed, 21 insertions(+), 13 deletions(-)

drivers/net/can/spi/mcp251xfd/mcp251xfd-tx.c +1 -0

index 160528d3cc26..146c44e47c60 100644
@@ -166,6 +166,7 @@ netdev_tx_t mcp251xfd_start_xmit(struct sk_buff *skb,
                                  struct net_device *ndev)
  {
         struct mcp251xfd_priv *priv = netdev_priv(ndev);
+       struct net_device_stats *stats = &ndev->stats;
         struct mcp251xfd_tx_ring *tx_ring = priv->tx;
         struct mcp251xfd_tx_obj *tx_obj;
         unsigned int frame_len;

@@ -181,25 +182,32 @@ netdev_tx_t mcp251xfd_start_xmit(struct sk_buff *skb,
         tx_obj = mcp251xfd_get_tx_obj_next(tx_ring);
         mcp251xfd_tx_obj_from_skb(priv, tx_obj, skb, tx_ring->head);

-       /* Stop queue if we occupy the complete TX FIFO */ tx_head = mcp251xfd_get_tx_head(tx_ring);
-       tx_ring->head++;
-       if (mcp251xfd_get_tx_free(tx_ring) == 0)
-               netif_stop_queue(ndev);
-
        frame_len = can_skb_get_frame_len(skb);
-       err = can_put_echo_skb(skb, ndev, tx_head, frame_len); -       if (!err)
-               netdev_sent_queue(priv->ndev, frame_len); +
+       tx_ring->head++;

        err = mcp251xfd_tx_obj_write(priv, tx_obj);
-       if (err)
-               goto out_err;
+       if (err) {
+               tx_ring->head--;

-       return NETDEV_TX_OK;
+               if (err == -EBUSY)
+                       return NETDEV_TX_BUSY;

- out_err:
-       netdev_err(priv->ndev, "ERROR in %s: %d\n", __func__, err); +               stats->tx_dropped++;

drivers/net/can/spi/mcp251xfd/mcp251xfd-tx.c +3 -0

+
+               if (net_ratelimit())
+                       netdev_err(priv->ndev,

drivers/net/can/spi/mcp251xfd/mcp251xfd-tx.c netdev_tx_t mcp251xfd_start_xmit(struct sk_buff *skb, · line 210

Quoted 10 lines

drivers/net/can/spi/mcp251xfd/mcp251xfd-tx.c +10 -0

+                                  "ERROR in %s: %d\n", __func__, err);
+       } else {
+               err = can_put_echo_skb(skb, ndev, tx_head, frame_len);
+               if (!err)
+                       netdev_sent_queue(priv->ndev, frame_len);
+
+               /* Stop queue if we occupy the complete TX FIFO */
+               if (mcp251xfd_get_tx_free(tx_ring) == 0)
+                       netif_stop_queue(ndev);
+       }

Once you have sent the TX objects to the chip, it can trigger a TX complete IRQ. This is not very likely as the SPI sends asynchronously and the xmit handler cannot sleep.

This means you have to call can_put_echo_skb() and stop the queue if needed, _before_ you call mcp251xfd_tx_obj_write(). In case of an error, you have to roll-back these.

regards,
Marc