Skip to contentSkip to conversation

serial: 8250: clear a stuck RX-timeout interrupt with an empty FIFO

PATCH In review Ryan Wilbur 2 participants 3 messages Jul 20 – Jul 21

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 serial: 8250: clear a stuck RX-timeout interrupt with an empty FIFO

Some UARTs can latch an RX character-timeout interrupt while the RX FIFO is empty: IIR reports UART_IIR_RX_TIMEOUT (0x0c) but LSR.DR is clear. A character timeout is only cleared by reading RHR, but serial8250_rx_chars() reads RHR only when LSR.DR is set, so nothing ever clears the condition. The interrupt is level-triggered and re-fires immediately, so on a single-core ARM926 the resulting interrupt storm livelocks the CPU.

It is reproducible when userspace repeatedly opens the front-panel port (ttyS1): serial8250_do_set_termios() re-enables interrupts on unlock and the handler then spins forever with iir=0xcc lsr=0x60 ier=0x05, tripping the soft-lockup detector in serial8250_handle_irq_locked().

Fix this by doing one throwaway RHR read to clear the timeout when the FIFO is empty (LSR.DR clear), so no real received data is ever discarded, and it is a no-op on healthy UARTs which never report a timeout with DR==0.

This is the same class of bug already worked around in 8250 drivers;
see commit 424d79183af0 ("serial: 8250_dw: Avoid "too much work" from bogus rx timeout interrupt") which reports the identical iir=0xcc/lsr=0x60. See also
UART_RX_TIMEOUT_QUIRK in 8250_omap, and the note in 8250_bcm7271.

LPC32xx (PORT_LPC3220) has no dedicated 8250 glue driver, driven by generic 8250_of, so it's handled here.

---
Changes in v2:
- Drop the PORT_LPC3220 gate and handle the spurious RX timeout generically

v1: https://lore.kernel.org/linux-serial/[email protected]/

 drivers/tty/serial/8250/8250_port.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
drivers/tty/serial/8250/8250_port.c +15 -0
index 8c241ec7f4f2..c63c4165105a 100644
@@ -1803,6 +1803,21 @@ void serial8250_handle_irq_locked(struct uart_port *port, unsigned int iir)
 	if (!(status & UART_LSR_DR) && (status & UART_LSR_FIFOE))
 		serial8250_clear_and_reinit_fifos(up);
 
+	/*
+	 * A UART can raise an RX character-timeout interrupt while the RX
+	 * FIFO is already empty (IIR reports RX_TIMEOUT but LSR.DR is
+	 * clear). The timeout is only cleared by reading RHR, but the RX
+	 * path below is skipped when the FIFO is empty, so nothing clears
+	 * it. With a level-triggered IRQ it re-fires immediately and can
+	 * livelock a single-core. Observed on the NXP LPC32xx
+	 * UART (PORT_LPC3220). Do one throwaway RHR read to
+	 * clear it. A healthy 16550 UART never reports a timeout with DR clear,
+	 * so this is a no-op elsewhere.
+	 */
+	if ((iir & UART_IIR_RX_TIMEOUT) == UART_IIR_RX_TIMEOUT &&
+	    !(status & UART_LSR_DR))
+		serial_in(up, UART_RX);
+
 	/*
 	 * If port is stopped and there are no error conditions in the
 	 * FIFO, then don't drain the FIFO, as this may lead to TTY buffer

base-commit: da7b5fd4e17f8e44c5590f2d603c01d499f056e6

Signature · 1 line
2.25.1
#1 lore raw
Quoted 49 lines · Ryan Wilbur

Some UARTs can latch an RX character-timeout interrupt while the RX FIFO is empty: IIR reports UART_IIR_RX_TIMEOUT (0x0c) but LSR.DR is clear. A character timeout is only cleared by reading RHR, but serial8250_rx_chars() reads RHR only when LSR.DR is set, so nothing ever clears the condition. The interrupt is level-triggered and re-fires immediately, so on a single-core ARM926 the resulting interrupt storm livelocks the CPU.

It is reproducible when userspace repeatedly opens the front-panel port (ttyS1): serial8250_do_set_termios() re-enables interrupts on unlock and the handler then spins forever with iir=0xcc lsr=0x60 ier=0x05, tripping the soft-lockup detector in serial8250_handle_irq_locked().

Fix this by doing one throwaway RHR read to clear the timeout when the FIFO is empty (LSR.DR clear), so no real received data is ever discarded, and it is a no-op on healthy UARTs which never report a timeout with DR==0.

This is the same class of bug already worked around in 8250 drivers;
see commit 424d79183af0 ("serial: 8250_dw: Avoid "too much work" from bogus rx timeout interrupt") which reports the identical iir=0xcc/lsr=0x60. See also
UART_RX_TIMEOUT_QUIRK in 8250_omap, and the note in 8250_bcm7271.

LPC32xx (PORT_LPC3220) has no dedicated 8250 glue driver, driven by generic 8250_of, so it's handled here.

---
Changes in v2:
- Drop the PORT_LPC3220 gate and handle the spurious RX timeout generically

v1: https://lore.kernel.org/linux-serial/[email protected]/

 drivers/tty/serial/8250/8250_port.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)
drivers/tty/serial/8250/8250_port.c +4 -0
index 8c241ec7f4f2..c63c4165105a 100644
@@ -1803,6 +1803,21 @@ void serial8250_handle_irq_locked(struct uart_port *port, unsigned int iir)
 	if (!(status & UART_LSR_DR) && (status & UART_LSR_FIFOE))
 		serial8250_clear_and_reinit_fifos(up);
 
+	/*
+	 * A UART can raise an RX character-timeout interrupt while the RX
+	 * FIFO is already empty (IIR reports RX_TIMEOUT but LSR.DR is
+	 * clear). The timeout is only cleared by reading RHR, but the RX

drivers/tty/serial/8250/8250_port.c void serial8250_handle_irq_locked(struct uart_port *port, unsigned int iir) · line 1819

Quoted 10 lines
drivers/tty/serial/8250/8250_port.c +10 -0
+	 * path below is skipped when the FIFO is empty, so nothing clears
+	 * it. With a level-triggered IRQ it re-fires immediately and can
+	 * livelock a single-core. Observed on the NXP LPC32xx
+	 * UART (PORT_LPC3220). Do one throwaway RHR read to
+	 * clear it. A healthy 16550 UART never reports a timeout with DR clear,
+	 * so this is a no-op elsewhere.
+	 */
+	if ((iir & UART_IIR_RX_TIMEOUT) == UART_IIR_RX_TIMEOUT &&
+	    !(status & UART_LSR_DR))
+		serial_in(up, UART_RX);

Hi,

I suspect Andy meant you should put this into hw specific file into a hw-specific handle_irq. That could be setup in 8250_of.c, it already seems to do something like that for some specific type.

It might make sense to put this code into a separate function that is called from those drivers. However, it also looked there are minor variations how each 8250 variant deals with this so it might not be that practical (8250_dw include check for LSR_BI bit and this one doesn't seem include that bit, not sure if that's an oversight or intentional difference and omap + bcm7271 did something different to workaround it).

So 3rd alternative would be to add e.g. UART_BUG_SPURIOUSRXTO but again there are the per driver variations. To handle the variations, one option would be add a callback which is invoked if UART_BUG_SPURIOUSRXTO is set (that is probably acceptable to occur from 8250_port.c's handle_irq). It would be a bit more work and a few patches in a series but definitely cleaner approach than copy-pasting this code around.

Signature · 1 line
 i.
#2 lore raw
Quoted 3 lines

I suspect Andy meant you should put this into hw specific file into a hw-specific handle_irq. That could be setup in 8250_of.c, it already seems to do something like that for some specific type.

Thanks. I'll move the fix out of 8250_port.c into a PORT_LPC3220 handle_irq, wired up in 8250_of.c the same way fsl8250_handle_irq is, with the handler logic following dw8250_handle_irq's empty-FIFO RX-timeout read.

Quoted 3 lines

(8250_dw include check for LSR_BI bit and this one doesn't seem include that bit, not sure if that's an oversight or intentional difference [...])

The missing LSR_BI check was an oversight, I'll gate on !(LSR & (UART_LSR_DR | UART_LSR_BI)) to match dw.

Quoted 3 lines

So 3rd alternative would be to add e.g. UART_BUG_SPURIOUSRXTO but again there are the per driver variations. [...] definitely cleaner approach than copy-pasting this code around.

I'll go with the per device handler over the UART_BUG_SPURIOUSRXTO callback because I can only test LPC32xx and I'm not comfortable touching drivers I can't explicitly test against.