Skip to contentSkip to conversation

serial: stm32: Move stm32_usart_transmit_chars() to interrupt thread

RFC PATCH In review Marek Vasut 2 participants 4 messages Dec 7, 2022

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 serial: stm32: Move stm32_usart_transmit_chars() to interrupt thread

Avoid locking in hard interrupt context, move the stm32_usart_transmit_chars() into the threaded IRQ handler. This fixes the following splat with preempt-rt:

 BUG: scheduling while atomic: (mount)/1289/0x00010001
 Modules linked in:
 Preemption disabled at:
 [<c0119127>] irq_enter_rcu+0xb/0x42
 CPU: 0 PID: 1289 Comm: (mount) Not tainted 6.1.0-rc7-rt5-stable-standard-00006-gd70aeccb9f0f #17
 Hardware name: STM32 (Device Tree Support)
  unwind_backtrace from show_stack+0xb/0xc
  show_stack from dump_stack_lvl+0x2b/0x34
  dump_stack_lvl from __schedule_bug+0x53/0x80
  __schedule_bug from __schedule+0x47/0x404
  __schedule from schedule_rtlock+0x15/0x34
  schedule_rtlock from rtlock_slowlock_locked+0x1d7/0x57e
  rtlock_slowlock_locked from rt_spin_lock+0x29/0x3c
  rt_spin_lock from stm32_usart_interrupt+0xa9/0x110
  stm32_usart_interrupt from __handle_irq_event_percpu+0x73/0x14e
  __handle_irq_event_percpu from handle_irq_event_percpu+0x9/0x22
  handle_irq_event_percpu from handle_irq_event+0x53/0x76
  handle_irq_event from handle_fasteoi_irq+0x65/0xa8
  handle_fasteoi_irq from handle_irq_desc+0xf/0x18
  handle_irq_desc from gic_handle_irq+0x45/0x54
  gic_handle_irq from generic_handle_arch_irq+0x19/0x2c
  generic_handle_arch_irq from call_with_stack+0xd/0x10

---
Cc: Alexandre Torgue <[email protected]>
Cc: Erwan Le Ray <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Jean Philippe Romain <[email protected]>
Cc: Jiri Slaby <[email protected]>
Cc: Maxime Coquelin <[email protected]>
Cc: Sebastian Andrzej Siewior <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Valentin Caron <[email protected]>
Cc: [email protected]
Cc: [email protected]
To: [email protected]

---
 drivers/tty/serial/stm32-usart.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)
drivers/tty/serial/stm32-usart.c +13 -6
index a1490033aa164..56357a7962edc 100644
@@ -791,11 +791,8 @@ static irqreturn_t stm32_usart_interrupt(int irq, void *ptr)
 		}
 	}
 
-	if ((sr & USART_SR_TXE) && !(stm32_port->tx_ch)) {
-		spin_lock(&port->lock);
-		stm32_usart_transmit_chars(port);
-		spin_unlock(&port->lock);
-	}
+	if ((sr & USART_SR_TXE) && !(stm32_port->tx_ch))
+		return IRQ_WAKE_THREAD;
 
 	if (stm32_usart_rx_dma_enabled(port))
 		return IRQ_WAKE_THREAD;
@@ -808,8 +805,18 @@ static irqreturn_t stm32_usart_threaded_interrupt(int irq, void *ptr)
 	struct uart_port *port = ptr;
 	struct tty_port *tport = &port->state->port;
 	struct stm32_port *stm32_port = to_stm32_port(port);
-	unsigned int size;
+	const struct stm32_usart_offsets *ofs = &stm32_port->info->ofs;
 	unsigned long flags;
+	unsigned int size;
+	u32 sr;
+
+	sr = readl_relaxed(port->membase + ofs->isr);
+
+	if ((sr & USART_SR_TXE) && !(stm32_port->tx_ch)) {
+		spin_lock_irqsave(&port->lock, flags);
+		stm32_usart_transmit_chars(port);
+		spin_unlock_irqrestore(&port->lock, flags);
+	}
 
 	/* Receiver timeout irq for DMA RX */
 	if (!stm32_port->throttled) {
Signature · 1 line
2.35.1
#1 lore raw
Quoted 49 lines · Marek Vasut

Avoid locking in hard interrupt context, move the stm32_usart_transmit_chars() into the threaded IRQ handler. This fixes the following splat with preempt-rt:

 BUG: scheduling while atomic: (mount)/1289/0x00010001
 Modules linked in:
 Preemption disabled at:
 [<c0119127>] irq_enter_rcu+0xb/0x42
 CPU: 0 PID: 1289 Comm: (mount) Not tainted 6.1.0-rc7-rt5-stable-standard-00006-gd70aeccb9f0f #17
 Hardware name: STM32 (Device Tree Support)
  unwind_backtrace from show_stack+0xb/0xc
  show_stack from dump_stack_lvl+0x2b/0x34
  dump_stack_lvl from __schedule_bug+0x53/0x80
  __schedule_bug from __schedule+0x47/0x404
  __schedule from schedule_rtlock+0x15/0x34
  schedule_rtlock from rtlock_slowlock_locked+0x1d7/0x57e
  rtlock_slowlock_locked from rt_spin_lock+0x29/0x3c
  rt_spin_lock from stm32_usart_interrupt+0xa9/0x110
  stm32_usart_interrupt from __handle_irq_event_percpu+0x73/0x14e
  __handle_irq_event_percpu from handle_irq_event_percpu+0x9/0x22
  handle_irq_event_percpu from handle_irq_event+0x53/0x76
  handle_irq_event from handle_fasteoi_irq+0x65/0xa8
  handle_fasteoi_irq from handle_irq_desc+0xf/0x18
  handle_irq_desc from gic_handle_irq+0x45/0x54
  gic_handle_irq from generic_handle_arch_irq+0x19/0x2c
  generic_handle_arch_irq from call_with_stack+0xd/0x10

---
Cc: Alexandre Torgue <[email protected]>
Cc: Erwan Le Ray <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Jean Philippe Romain <[email protected]>
Cc: Jiri Slaby <[email protected]>
Cc: Maxime Coquelin <[email protected]>
Cc: Sebastian Andrzej Siewior <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Valentin Caron <[email protected]>
Cc: [email protected]
Cc: [email protected]
To: [email protected]

---
 drivers/tty/serial/stm32-usart.c | 19 +++++++++++++------
 1 file changed, 13 insertions(+), 6 deletions(-)
drivers/tty/serial/stm32-usart.c +0 -0
index a1490033aa164..56357a7962edc 100644

drivers/tty/serial/stm32-usart.c static irqreturn_t stm32_usart_interrupt(int irq, void *ptr) · line 792

Quoted 2 lines
drivers/tty/serial/stm32-usart.c +0 -0
@@ -791,11 +791,8 @@ static irqreturn_t stm32_usart_interrupt(int irq, void *ptr)
 		}
 	}

Why is this handler running as the primary handler to begin with? There is
stm32_usart_rs485_rts_disable()
-> mctrl_gpio_set()

    -> gpiod_set_array_value()
       -> gpiod_set_array_value_complex()
         -> gpio_chip_set_multiple()
	    -> gc->set_multiple() || gc->set() ?
         -> bitmap_alloc() boom

I don't know if the underlying gpiod is always using a raw_spinlock_t but that bitmap_alloc() (depending on FASTPATH_NGPIO()) is not going to work.
pm_wakeup_dev_event() is also using a spinlock_t.

drivers/tty/serial/stm32-usart.c static irqreturn_t stm32_usart_interrupt(int irq, void *ptr) · line 795

Quoted 7 lines
drivers/tty/serial/stm32-usart.c +2 -5
-	if ((sr & USART_SR_TXE) && !(stm32_port->tx_ch)) {
-		spin_lock(&port->lock);
-		stm32_usart_transmit_chars(port);
-		spin_unlock(&port->lock);
-	}
+	if ((sr & USART_SR_TXE) && !(stm32_port->tx_ch))
+		return IRQ_WAKE_THREAD;

Before that, there is a "stm32_port->throttled" check using the very same lock.

drivers/tty/serial/stm32-usart.c static irqreturn_t stm32_usart_interrupt(int irq, void *ptr) · line 798

Quoted 2 lines
drivers/tty/serial/stm32-usart.c +0 -0
 	if (stm32_usart_rx_dma_enabled(port))
 		return IRQ_WAKE_THREAD;

Sebastian

#2 lore raw
Quoted 69 lines · Sebastian Andrzej Siewior
Quoted 49 lines · Marek Vasut

Avoid locking in hard interrupt context, move the stm32_usart_transmit_chars() into the threaded IRQ handler. This fixes the following splat with preempt-rt:

  BUG: scheduling while atomic: (mount)/1289/0x00010001
  Modules linked in:
  Preemption disabled at:
  [<c0119127>] irq_enter_rcu+0xb/0x42
  CPU: 0 PID: 1289 Comm: (mount) Not tainted 6.1.0-rc7-rt5-stable-standard-00006-gd70aeccb9f0f #17
  Hardware name: STM32 (Device Tree Support)
   unwind_backtrace from show_stack+0xb/0xc
   show_stack from dump_stack_lvl+0x2b/0x34
   dump_stack_lvl from __schedule_bug+0x53/0x80
   __schedule_bug from __schedule+0x47/0x404
   __schedule from schedule_rtlock+0x15/0x34
   schedule_rtlock from rtlock_slowlock_locked+0x1d7/0x57e
   rtlock_slowlock_locked from rt_spin_lock+0x29/0x3c
   rt_spin_lock from stm32_usart_interrupt+0xa9/0x110
   stm32_usart_interrupt from __handle_irq_event_percpu+0x73/0x14e
   __handle_irq_event_percpu from handle_irq_event_percpu+0x9/0x22
   handle_irq_event_percpu from handle_irq_event+0x53/0x76
   handle_irq_event from handle_fasteoi_irq+0x65/0xa8
   handle_fasteoi_irq from handle_irq_desc+0xf/0x18
   handle_irq_desc from gic_handle_irq+0x45/0x54
   gic_handle_irq from generic_handle_arch_irq+0x19/0x2c
   generic_handle_arch_irq from call_with_stack+0xd/0x10

---
Cc: Alexandre Torgue <[email protected]>
Cc: Erwan Le Ray <[email protected]>
Cc: Greg Kroah-Hartman <[email protected]>
Cc: Jean Philippe Romain <[email protected]>
Cc: Jiri Slaby <[email protected]>
Cc: Maxime Coquelin <[email protected]>
Cc: Sebastian Andrzej Siewior <[email protected]>
Cc: Thomas Gleixner <[email protected]>
Cc: Valentin Caron <[email protected]>
Cc: [email protected]
Cc: [email protected]
To: [email protected]

---
  drivers/tty/serial/stm32-usart.c | 19 +++++++++++++------
  1 file changed, 13 insertions(+), 6 deletions(-)
drivers/tty/serial/stm32-usart.c +0 -0
index a1490033aa164..56357a7962edc 100644

drivers/tty/serial/stm32-usart.c static irqreturn_t stm32_usart_interrupt(int irq, void *ptr) · line 792

Quoted 2 lines
drivers/tty/serial/stm32-usart.c +0 -0
@@ -791,11 +791,8 @@ static irqreturn_t stm32_usart_interrupt(int irq, void *ptr)
  		}
  	}

Why is this handler running as the primary handler to begin with? There is
stm32_usart_rs485_rts_disable()
-> mctrl_gpio_set()

     -> gpiod_set_array_value()
        -> gpiod_set_array_value_complex()
          -> gpio_chip_set_multiple()
	    -> gc->set_multiple() || gc->set() ?
          -> bitmap_alloc() boom

I don't know if the underlying gpiod is always using a raw_spinlock_t but that bitmap_alloc() (depending on FASTPATH_NGPIO()) is not going to work.
pm_wakeup_dev_event() is also using a spinlock_t.

Quoted 13 lines

drivers/tty/serial/stm32-usart.c static irqreturn_t stm32_usart_interrupt(int irq, void *ptr) · line 795

Quoted 7 lines
drivers/tty/serial/stm32-usart.c +2 -5
-	if ((sr & USART_SR_TXE) && !(stm32_port->tx_ch)) {
-		spin_lock(&port->lock);
-		stm32_usart_transmit_chars(port);
-		spin_unlock(&port->lock);
-	}
+	if ((sr & USART_SR_TXE) && !(stm32_port->tx_ch))
+		return IRQ_WAKE_THREAD;

Before that, there is a "stm32_port->throttled" check using the very same lock.

Quoted 2 lines
drivers/tty/serial/stm32-usart.c +0 -0
  	if (stm32_usart_rx_dma_enabled(port))
  		return IRQ_WAKE_THREAD;

Is the suggestion therefore to completely remove the hard IRQ handler and move everything into the threaded IRQ handler ?

Are there any drawbacks of doing that to a serial port driver ?

#3 lore raw
Quoted 3 lines · Marek Vasut

Is the suggestion therefore to completely remove the hard IRQ handler and move everything into the threaded IRQ handler ?

Commit

	3489187204eb7 ("serial: stm32: adding dma support")

added the threaded mode with DMA support. It didn't say _why_ this is needed. So I would suggest to use request_irq() and merge the two handler. The part where it ignores the IRQ handler in the "stm32_port->throttled" case isn't nice since it returns IRQ_HANDLED while doing nothing.

Quoted 1 line

Are there any drawbacks of doing that to a serial port driver ?

The threaded handler runs with disabled interrupts (due to irq-save lock) so there shouldn't be any visible difference other than the thread is gone. I'm not sure what the benefit here actually is. With `threadirqs' and so on PREEMPT_RT the whole routine will be moved in the thread (as it should).

Looking at drivers/tty/serial/, the stm32-usart is the only one requesting threaded-interrupts with a primary handler. So I guess it was not needed to begin with.

Sebastian