can: mcp251xfd: Always stop on BUS_OFF and call netif_stop_queue
A message that is not in the archive (sent off-list or not yet received): 1b153bce-a66a-45ee-a5c6-963ea6fb1c82.949ef384-8293-46b8-903f-40a477c056ae.c7fafffb-93a7-4f4c-9c1e-df959c3ed3bb@emailsignatures365.codetwo.com
PATCH 1/1 can: mcp251xfd: Always stop on BUS_OFF and call netif_stop_queue
When there's an error attempting to store the BER counter, don't abort but continue shutting down the chip as required.
After disabling communications, also stop the TX queue with a call to netif_stop_queue. When the interface restarts, mcp251xfd_set_mode will call netif_wake_queue and resume.
This fixes a hangup in either send() or poll() from userspace. To
reproduce: I ran "cansequence can0 -p" to flood the system with packets. While running that, I shorted the CAN signals, causing a bus error. Usually communications would resume after the CAN bus restarted, but sometimes the system got stuck and refused to send any more packets. The sending process would be in either poll() or send(), waiting for the queue to resume. To "unstuck" the process from send() it was sufficient to send any packet on the can interface from another process. To get it out of the poll() hang, only bringing the can interface down (and up) would work.
After adding the netif_stop_queue call, I was unable to reproduce the problem.
---
drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
drivers/net/can/spi/mcp251xfd/mcp251xfd-core.c +2 -1
@@ -1084,10 +1084,11 @@ static int mcp251xfd_handle_cerrif(struct mcp251xfd_priv *priv)*/ err =__mcp251xfd_get_berr_counter (priv->ndev, &priv->bec);if (err) -return err; +netdev_err (priv->ndev,"Failed to store BER counter\n" );mcp251xfd_chip_stop (priv,CAN_STATE_BUS_OFF );can_bus_off (priv->ndev); +netif_stop_queue (priv->ndev); }if (!skb)
Signature · 18 lines
2.17.1 Met vriendelijke groet / kind regards, Mike Looijmans System Expert TOPIC Embedded Products B.V. Materiaalweg 4, 5681 RJ Best The Netherlands T: +31 (0) 499 33 69 69 E: [email protected] W: www.topic.nl Please consider the environment before printing this e-mail
Marc Kleine-Budde
maintainerQuoted 2 lines · Mike Looijmans
When there's an error attempting to store the BER counter, don't abort but continue shutting down the chip as required.
If you refer to an error by __mcp251xfd_get_berr_counter(), it's not a store error, but a failure of regmap_read(priv->map_reg, MCP251XFD_REG_TREC, &trec). By default the SPI transfers are CRC enabled and no/wrong data from the chip will be detected and return an error here (after 3 retires). In out of memory conditions or other kernel errors might be possible here, too.
Have you seen a problem here? But as we shut down the chip here anyways, we can ignore the error here.
Quoted 2 lines
After disabling communications, also stop the TX queue with a call to
netif_stop_queue.
can_bus_off() will call netif_carrier_off(), isn't this sufficient? Have you enabled automatic restart in case of bus off? I think the netdev watchdog will kick you, if the interface has a stooped queue for too long (IIRC 5 seconds).
Quoted 14 lines
When the interface restarts,
mcp251xfd_set_modewill callnetif_wake_queueand resume.This fixes a hangup in either
send()orpoll()from userspace. To
reproduce: I ran "cansequence can0 -p" to flood the system with packets. While running that, I shorted the CAN signals, causing a bus error. Usually communications would resume after the CAN bus restarted, but sometimes the system got stuck and refused to send any more packets. The sending process would be in eitherpoll()orsend(), waiting for the queue to resume. To "unstuck" the process fromsend()it was sufficient to send any packet on the can interface from another process. To get it out of thepoll()hang, only bringing the can interface down (and up) would work.
Quoted 2 lines
After adding the
netif_stop_queuecall, I was unable to reproduce the problem.
The newly added netif_stop_queue() will cause the netif_wake_queue() in the mcp251xfd_set_mode() to actually wake the queue. If you observe a problem, I think it's a general problem, so all drivers would be effected.
Quoted 1 line
Signed-off-by Mike Looijmans <[email protected]>
Marc
Signature · 4 lines
Pengutronix e.K. | Marc Kleine-Budde | Embedded Linux | https://www.pengutronix.de | Vertretung Nürnberg | Phone: +49-5121-206917-129 | Amtsgericht Hildesheim, HRA 2686 | Fax: +49-5121-206917-9 |
Met vriendelijke groet / kind regards,
Mike Looijmans
System Expert
TOPIC Embedded Products B.V.
Materiaalweg 4, 5681 RJ Best
The Netherlands
T: +31 (0) 499 33 69 69
E: [email protected]
W: www.topic.nl
Please consider the environment before printing this e-mail
Quoted 11 lines · Marc Kleine-Budde
Quoted 2 lines · Mike Looijmans
When there's an error attempting to store the BER counter, don't abort but continue shutting down the chip as required.
If you refer to an error by
__mcp251xfd_get_berr_counter(), it's not a store error, but a failure ofregmap_read(priv->map_reg, MCP251XFD_REG_TREC, &trec). By default the SPI transfers are CRC enabled and no/wrong data from the chip will be detected and return an error here (after 3 retires). In out of memory conditions or other kernel errors might be possible here, too.
Quoted 2 lines
Have you seen a problem here? But as we shut down the chip here anyways, we can ignore the error here.
Excactly my point, the important task here is to report the bus off state.
I haven't seen the error trigger, so it wasn't the cause of our problems.
In retrospect, maybe I should have split this in two patches?
Quoted 7 lines
Quoted 2 lines
After disabling communications, also stop the TX queue with a call to
netif_stop_queue.
can_bus_off()will callnetif_carrier_off(), isn't this sufficient? Have you enabled automatic restart in case of bus off? I think the netdev watchdog will kick you, if the interface has a stooped queue for too long (IIRC 5 seconds).
Apparently not. The problem also occurred in the field, where the CAN communication got stuck for something like a full day.
Automatic restart was enabled, with a 100ms timeout. Setting a shorter timeout like 50 ms seemed to make the problem more likely to occur
What I also noticed, but didn't accurately measure, is that the time it took to recover used to be somewhat random, it could take up to about a second for packets to start going out again, as observed by watching LEDs. After adding netif_stop_queue the recovery was almost instantaneous.
Quoted 15 lines
Quoted 15 lines
When the interface restarts,
mcp251xfd_set_modewill callnetif_wake_queueand resume.This fixes a hangup in either
send()orpoll()from userspace. To
reproduce: I ran "cansequence can0 -p" to flood the system with packets. While running that, I shorted the CAN signals, causing a bus error. Usually communications would resume after the CAN bus restarted, but sometimes the system got stuck and refused to send any more packets. The sending process would be in eitherpoll()orsend(), waiting for the queue to resume. To "unstuck" the process fromsend()it was sufficient to send any packet on the can interface from another process. To get it out of thepoll()hang, only bringing the can interface down (and up) would work.
Quoted 7 lines
Quoted 2 lines
After adding the
netif_stop_queuecall, I was unable to reproduce the problem.The newly added
netif_stop_queue()will cause thenetif_wake_queue()in themcp251xfd_set_mode()to actually wake the queue. If you observe a problem, I think it's a general problem, so all drivers would be effected.
After two days digging, my conclusion was that "something" wasn't getting a proper "kick" when the CAN interface came back online.
One could argue the netif_stop_queue should be put into can_bus_off. But the netif_wake_queue call is in the driver, so that would break symmetry.
Quoted 5 lines
Quoted 2 lines
Signed-off-by Mike Looijmans <[email protected]>Marc