char/ns16550: bound execution time of ns16550_interrupt()
PATCH 1/1 char/ns16550: bound execution time of ns16550_interrupt()
The current logic in ns16550_interrupt() will loop until the device sets the NOINT in IIR. At least on the Lenovo ThinkSystem SR630 V4 the flow control of the serial-over-lan emulated UART seems to be broken, as it doesn't set the NOINT bit consistently. The Transmitter Holding Register Empty in LSR also seems to not be properly signaled, as even with it set writes to the transmit register take ~6ms. This leads to the watchdog triggering very easily on such system.
Introduce an upper bound on the execution time of ns16550_interrupt(), this is currently set as 4x the polling interval, which is calculated as the time to fill RX FIFO and/or empty TX FIFO. The current maximum is 5ms. Once the timeout triggers the interrupt is disabled and the uart is switched to polling mode.
---
There's a possible alternative approach to solve this by moving the actual interrupt processing to a softirq tasklet and disabling the interrupt source until the processing is done, likely unifying the logic with the timer task. However that's a bigger change, and too risky for 4.22 at this point.
--- xen/drivers/char/ns16550.c | 33 ++++++++++++++++++++++++++++++++- 1 file changed, 32 insertions(+), 1 deletion(-)
xen/drivers/char/ns16550.c +32 -1
@@ -62,6 +62,7 @@ static struct ns16550 {#endif unsigned int timeout_ms;bool intr_works; +bool force_polling;bool dw_usr_bsy;#ifdef NS16550_PCI /* PCI card parameters. */
@@ -190,12 +191,41 @@ static void cf_check ns16550_interrupt(int irq, void *dev_id) {struct serial_port *port = dev_id;struct ns16550 *uart = port->uart; +/* Set quite arbitrarily as 4x the time to drain the TX or fill RX FIFOs. */ +const s_time_t timeout =NOW () +min (MILLISECS (uart->timeout_ms *4 ), +MILLISECS (5 )); + +if ( uart->force_polling ) +return ; uart->intr_works =1 ;while ( !(ns_read_reg (uart,UART_IIR ) &UART_IIR_NOINT ) ) {u8 lsr =ns_read_reg (uart,UART_LSR ); +s_time_t now =NOW (); + +/* Break out of the loop if spending too much time. */ +if ( now > timeout ) + { +struct irq_desc *desc =irq_to_desc (irq); + +/* Disable the interrupt source - it's never shared. */ +spin_lock_irq (&desc->lock); + desc->status |=IRQ_DISABLED ; +if ( desc->handler->disable ) + desc->handler->disable (desc); +spin_unlock_irq (&desc->lock); + +/* Disable interrupt generation on the device and arm the timer. */ + uart->force_polling = true; +ns_write_reg (uart,UART_IER ,0 ); +set_timer (&uart->timer, now +MILLISECS (uart->timeout_ms)); +printk (XENLOG_WARNING +"uart interrupt taking too long, switched to polling\n" ); + +return ; + }if ( (lsr & uart->lsr_mask) == uart->lsr_mask )serial_tx_interrupt (port);
@@ -223,7 +253,7 @@ static void cf_check __ns16550_poll(const struct cpu_user_regs *regs)struct ns16550 *uart = port->uart;const struct cpu_user_regs *old_regs; -if ( uart->intr_works ) +if ( uart->intr_works && !uart->force_polling )return ;/* Interrupts work - no more polling */ /* Mimic interrupt context. */
@@ -313,6 +343,7 @@ static void ns16550_setup_preirq(struct ns16550 *uart)unsigned int divisor; uart->intr_works =0 ; + uart->force_polling = false;pci_serial_early_init (uart);
Signature · 1 line
2.53.0
Quoted 8 lines · Roger Pau Monne
The current logic in
ns16550_interrupt()will loop until the device sets the NOINT in IIR. At least on the Lenovo ThinkSystem SR630 V4 the flow control of the serial-over-lan emulated UART seems to be broken, as it doesn't set the NOINT bit consistently. The Transmitter Holding Register Empty in LSR also seems to not be properly signaled, as even with it set writes to the transmit register take ~6ms. This leads to the watchdog triggering very easily on such system.
Quoted 6 lines
Introduce an upper bound on the execution time of
ns16550_interrupt(), this is currently set as 4x the polling interval, which is calculated as the time to fill RX FIFO and/or empty TX FIFO. The current maximum is 5ms. Once the timeout triggers the interrupt is disabled and the uart is switched to polling mode.
Don't you mmiss Fixes: tag?
Quoted 7 lines
Signed-off-by Roger Pau Monné <[email protected]>> ---There's a possible alternative approach to solve this by moving the actual interrupt processing to a softirq tasklet and disabling the interrupt source until the processing is done, likely unifying the logic with the timer task. However that's a bigger change, and too risky for 4.22 at this point.
---
Agree, it would be better to stick to the current solution:
Thanks.
~ Oleksii
Quoted 9 lines · Oleksii Kurochko
Quoted 8 lines · Roger Pau Monne
The current logic in
ns16550_interrupt()will loop until the device sets the NOINT in IIR. At least on the Lenovo ThinkSystem SR630 V4 the flow control of the serial-over-lan emulated UART seems to be broken, as it doesn't set the NOINT bit consistently. The Transmitter Holding Register Empty in LSR also seems to not be properly signaled, as even with it set writes to the transmit register take ~6ms. This leads to the watchdog triggering very easily on such system.
Quoted 7 lines
Quoted 5 lines
Introduce an upper bound on the execution time of
ns16550_interrupt(), this is currently set as 4x the polling interval, which is calculated as the time to fill RX FIFO and/or empty TX FIFO. The current maximum is 5ms. Once the timeout triggers the interrupt is disabled and the uart is switched to polling mode.Don't you mmiss Fixes: tag?
you mean? I think there's nothing wrong with our pre-existing code, and the changes here instead are a workaround for some (apparently) badly implemented SoL.
Jan
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Oleksii Kurochko
Quoted 13 lines · Roger Pau Monne
The current logic in
ns16550_interrupt()will loop until the device sets the NOINT in IIR. At least on the Lenovo ThinkSystem SR630 V4 the flow control of the serial-over-lan emulated UART seems to be broken, as it doesn't set the NOINT bit consistently. The Transmitter Holding Register Empty in LSR also seems to not be properly signaled, as even with it set writes to the transmit register take ~6ms. This leads to the watchdog triggering very easily on such system.Introduce an upper bound on the execution time of
ns16550_interrupt(), this is currently set as 4x the polling interval, which is calculated as the time to fill RX FIFO and/or empty TX FIFO. The current maximum is 5ms. Once the timeout triggers the interrupt is disabled and the uart is switched to polling mode.
Quoted 7 lines
Quoted 1 line
Don't you mmiss Fixes: tag?
Fixes "SoL on Lenovo ThinkSystem SR630 V4"you mean? I think there's nothing wrong with our pre-existing code, and the changes here instead are a workaround for some (apparently) badly implemented SoL.
It was on purpose that no Fixes tag was provided.
Xen code would be fine with well-behaved uarts, however most of the serial-over-lan emulated ones are not well behaved it seems.
There's a possible issue with the unbounded loop in ns16550_interrupt() as it's relying solely on hardware register values to terminate, which again would be OK if hardware was correctly implemented. I don't think this warrants a Fixes tag.
Thanks, Roger.
Quoted 14 lines · Roger Pau Monne
The current logic in
ns16550_interrupt()will loop until the device sets the NOINT in IIR. At least on the Lenovo ThinkSystem SR630 V4 the flow control of the serial-over-lan emulated UART seems to be broken, as it doesn't set the NOINT bit consistently. The Transmitter Holding Register Empty in LSR also seems to not be properly signaled, as even with it set writes to the transmit register take ~6ms. This leads to the watchdog triggering very easily on such system.Introduce an upper bound on the execution time of
ns16550_interrupt(), this is currently set as 4x the polling interval, which is calculated as the time to fill RX FIFO and/or empty TX FIFO. The current maximum is 5ms. Once the timeout triggers the interrupt is disabled and the uart is switched to polling mode.
Quoted 7 lines
Signed-off-by Roger Pau Monné <[email protected]>---
There's a possible alternative approach to solve this by moving the actual interrupt processing to a softirq tasklet and disabling the interrupt source until the processing is done, likely unifying the logic with the timer task. However that's a bigger change, and too risky for 4.22 at this point.
+1
Quoted 11 lines
xen/drivers/char/ns16550.c +1
-0@@ -62,6 +62,7 @@ static struct ns16550 {#endif unsigned int timeout_ms;bool intr_works; +bool force_polling;bool dw_usr_bsy;#ifdef NS16550_PCI /* PCI card parameters. */
xen/drivers/char/ns16550.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 194
Quoted 4 lines
xen/drivers/char/ns16550.c +1
-0@@ -190,12 +191,41 @@ static void cf_check ns16550_interrupt(int irq, void *dev_id) {struct serial_port *port = dev_id;struct ns16550 *uart = port->uart; +/* Set quite arbitrarily as 4x the time to drain the TX or fill RX FIFOs. */
Nit: I'd drop the latter of the two "the".
xen/drivers/char/ns16550.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 196
Quoted 2 lines
xen/drivers/char/ns16550.c +2
-0+const s_time_t timeout =NOW () +min (MILLISECS (uart->timeout_ms *4 ), +MILLISECS (5 ));
MILLISECS(min(uart->timeout_ms * 4, 5U)) ?
xen/drivers/char/ns16550.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 198
Quoted 2 lines
xen/drivers/char/ns16550.c +2
-0+if ( uart->force_polling ) +return ;
As the IRQ was disabled, is this even possible? I.e. should this be some kind of assertion or alike?
Quoted 4 lines
xen/drivers/char/ns16550.c +0
-0uart->intr_works =1 ;while ( !(ns_read_reg (uart,UART_IIR ) &UART_IIR_NOINT ) ) {
xen/drivers/char/ns16550.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 214
Quoted 10 lines
xen/drivers/char/ns16550.c +9
-0u8 lsr =ns_read_reg (uart,UART_LSR ); +s_time_t now =NOW (); + +/* Break out of the loop if spending too much time. */ +if ( now > timeout ) + { +struct irq_desc *desc =irq_to_desc (irq); + +/* Disable the interrupt source - it's never shared. */ +spin_lock_irq (&desc->lock);
This needs to be spin_lock_irqsave() - we may not rely on IRQs being on when we make it here. However, ...
xen/drivers/char/ns16550.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 218
Quoted 4 lines
xen/drivers/char/ns16550.c +4
-0+ desc->status |=IRQ_DISABLED ; +if ( desc->handler->disable ) + desc->handler->disable (desc); +spin_unlock_irq (&desc->lock);
... all of this open-coding is quite bad anyway. We should probably add a helper for this in IRQ handling code.
xen/drivers/char/ns16550.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 225
Quoted 6 lines
xen/drivers/char/ns16550.c +6
-0+/* Disable interrupt generation on the device and arm the timer. */ + uart->force_polling = true; +ns_write_reg (uart,UART_IER ,0 ); +set_timer (&uart->timer, now +MILLISECS (uart->timeout_ms)); +printk (XENLOG_WARNING +"uart interrupt taking too long, switched to polling\n" );
Probably it is indeed best to keep this simple, but: A single instance of this taking e.g. just over 5ms (perhaps with a low baud rate) may not be indicative of an actual issue. To alleviate this as least some, perhaps besides capping at 5ms we should also make sure that the timeout used isn't below ->timeout_ms?
Jan
Quoted 43 lines · Jan Beulich
Quoted 14 lines · Roger Pau Monne
The current logic in
ns16550_interrupt()will loop until the device sets the NOINT in IIR. At least on the Lenovo ThinkSystem SR630 V4 the flow control of the serial-over-lan emulated UART seems to be broken, as it doesn't set the NOINT bit consistently. The Transmitter Holding Register Empty in LSR also seems to not be properly signaled, as even with it set writes to the transmit register take ~6ms. This leads to the watchdog triggering very easily on such system.Introduce an upper bound on the execution time of
ns16550_interrupt(), this is currently set as 4x the polling interval, which is calculated as the time to fill RX FIFO and/or empty TX FIFO. The current maximum is 5ms. Once the timeout triggers the interrupt is disabled and the uart is switched to polling mode.Quoted 7 lines
Signed-off-by Roger Pau Monné <[email protected]>---
There's a possible alternative approach to solve this by moving the actual interrupt processing to a softirq tasklet and disabling the interrupt source until the processing is done, likely unifying the logic with the timer task. However that's a bigger change, and too risky for 4.22 at this point.+1
Quoted 11 lines
xen/drivers/char/ns16550.c +1
-0@@ -62,6 +62,7 @@ static struct ns16550 {#endif unsigned int timeout_ms;bool intr_works; +bool force_polling;bool dw_usr_bsy;#ifdef NS16550_PCI /* PCI card parameters. */ xen/drivers/char/ns16550.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 194
Quoted 4 lines
xen/drivers/char/ns16550.c +1
-0@@ -190,12 +191,41 @@ static void cf_check ns16550_interrupt(int irq, void *dev_id) {struct serial_port *port = dev_id;struct ns16550 *uart = port->uart; +/* Set quite arbitrarily as 4x the time to drain the TX or fill RX FIFOs. */ Nit: I'd drop the latter of the two "the".
Quoted 4 lines
xen/drivers/char/ns16550.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 196
Quoted 2 lines
xen/drivers/char/ns16550.c +2
-0+const s_time_t timeout =NOW () +min (MILLISECS (uart->timeout_ms *4 ), +MILLISECS (5 ));
MILLISECS(min(uart->timeout_ms * 4, 5U))?
Bah, yes, sorry, I've added the min() later and clearly didn't apply much brainpower about it's position.
Quoted 6 lines
xen/drivers/char/ns16550.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 198
Quoted 3 lines
xen/drivers/char/ns16550.c +2
-0+if ( uart->force_polling ) +return ;As the IRQ was disabled, is this even possible? I.e. should this be some kind of assertion or alike?
Hm, I wasn't setting IRQ_DISABLED before, and hence needed this guard. But now with IRQ_DISABLED being set in ->status do_IRQ() should filter any stray interrupts. I will attempt to add an ASSERT_UNREACHABLE() here.
Quoted 17 lines
Quoted 4 lines
xen/drivers/char/ns16550.c +0
-0uart->intr_works =1 ;while ( !(ns_read_reg (uart,UART_IIR ) &UART_IIR_NOINT ) ) {xen/drivers/char/ns16550.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 214
Quoted 10 lines
xen/drivers/char/ns16550.c +9
-0u8 lsr =ns_read_reg (uart,UART_LSR ); +s_time_t now =NOW (); + +/* Break out of the loop if spending too much time. */ +if ( now > timeout ) + { +struct irq_desc *desc =irq_to_desc (irq); + +/* Disable the interrupt source - it's never shared. */ +spin_lock_irq (&desc->lock);This needs to be
spin_lock_irqsave()- we may not rely on IRQs being on when we make it here. However, ...
I was relying on do_IRQ() unconditionally enabling IRQs before calling the handler, but it's safer to use the _irqsave() variant.
Quoted 7 lines
xen/drivers/char/ns16550.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 218
Quoted 4 lines
xen/drivers/char/ns16550.c +4
-0+ desc->status |=IRQ_DISABLED ; +if ( desc->handler->disable ) + desc->handler->disable (desc); +spin_unlock_irq (&desc->lock);... all of this open-coding is quite bad anyway. We should probably add a helper for this in IRQ handling code.
Can add, no problem.
Quoted 10 lines
xen/drivers/char/ns16550.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 225
Quoted 6 lines
xen/drivers/char/ns16550.c +6
-0+/* Disable interrupt generation on the device and arm the timer. */ + uart->force_polling = true; +ns_write_reg (uart,UART_IER ,0 ); +set_timer (&uart->timer, now +MILLISECS (uart->timeout_ms)); +printk (XENLOG_WARNING +"uart interrupt taking too long, switched to polling\n" );Probably it is indeed best to keep this simple, but: A single instance of this taking e.g. just over 5ms (perhaps with a low baud rate) may not be indicative of an actual issue.
It's all fiddly, yes, I could add a counter and maybe only disable if we have a certain amount of executions of ns16550_interrupt() exceeding the timeout, yet I wanted to keep this simple.
Quoted 3 lines
To alleviate this as least some, perhaps
besides capping at 5ms we should also make sure that the timeout used isn't below ->timeout_ms?
OK, let me see what I can do.
Thanks, Roger.
Quoted 5 lines · Roger Pau Monné
Quoted 4 lines · Jan Beulich
Quoted 2 lines · Roger Pau Monne
(quoted diff) +2
-0+ if ( uart->force_polling ) + return;
Quoted 7 lines
Quoted 2 lines
As the IRQ was disabled, is this even possible? I.e. should this be some kind of assertion or alike?
Hm, I wasn't setting
IRQ_DISABLEDbefore, and hence needed this guard. But now withIRQ_DISABLEDbeing set in ->statusdo_IRQ()should filter any stray interrupts. I will attempt to add anASSERT_UNREACHABLE()here.
Simply ASSERT(!uart->force_polling) should do here? It is not wrong to run the code below in release builds in such an event. If we kept getting interrupts (perhaps at a high frequency) we'd be in trouble anyway.
Jan
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 6 lines · Jan Beulich
Review
Quoted 2 lines · Roger Pau Monne
(quoted diff) +2
-0+ if ( uart->force_polling ) + return;As the IRQ was disabled, is this even possible? I.e. should this be some kind of assertion or alike?
Quoted 8 lines
Quoted 4 lines
Hm, I wasn't setting
IRQ_DISABLEDbefore, and hence needed this guard. But now withIRQ_DISABLEDbeing set in ->statusdo_IRQ()should filter any stray interrupts. I will attempt to add anASSERT_UNREACHABLE()here.Simply
ASSERT(!uart->force_polling)should do here? It is not wrong to run the code below in release builds in such an event. If we kept getting interrupts (perhaps at a high frequency) we'd be in trouble anyway.
No, I'm afraid I can't do it like that, I can't put an ASSERT there, because we can still get into ns16550_interrupt() after the interrupt has been disabled. In do_IRQ() we have the following loop:
while ( desc->status & IRQ_PENDING )
{
desc->status &= ~IRQ_PENDING;
spin_unlock_irq(&desc->lock);
tsc_in = tb_init_done ? get_cycles() : 0;
action->handler(irq, action->dev_id);
TRACE_TIME(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles());
spin_lock_irq(&desc->lock);
}So if the device is generating further interrupts in the window with IRQs enabled (while we execute the handler), we will keep looping around this, without taking into account the setting of IRQ_DISABLED.
This is something that we might want to fix, so that the loop is bound by IRQ_PENDING being set, and IRQ_DISABLED not, ie:
while ( (desc->status & (IRQ_PENDING | IRQ_DISABLED)) == IRQ_PENDING )
Regards, Roger.
Quoted 35 lines · Roger Pau Monné
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 6 lines · Jan Beulich
Review
Quoted 2 lines · Roger Pau Monne
(quoted diff) +2
-0+ if ( uart->force_polling ) + return;As the IRQ was disabled, is this even possible? I.e. should this be some kind of assertion or alike?
Quoted 8 lines
Quoted 4 lines
Hm, I wasn't setting
IRQ_DISABLEDbefore, and hence needed this guard. But now withIRQ_DISABLEDbeing set in ->statusdo_IRQ()should filter any stray interrupts. I will attempt to add anASSERT_UNREACHABLE()here.Simply
ASSERT(!uart->force_polling)should do here? It is not wrong to run the code below in release builds in such an event. If we kept getting interrupts (perhaps at a high frequency) we'd be in trouble anyway.No, I'm afraid I can't do it like that, I can't put an ASSERT there, because we can still get into
ns16550_interrupt()after the interrupt has been disabled. Indo_IRQ()we have the following loop:while ( desc->status & IRQ_PENDING ) { desc->status &= ~IRQ_PENDING; spin_unlock_irq(&desc->lock); tsc_in = tb_init_done ? get_cycles() : 0; action->handler(irq, action->dev_id); TRACE_TIME(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles()); spin_lock_irq(&desc->lock); }
Quoted 3 lines
So if the device is generating further interrupts in the window with IRQs enabled (while we execute the handler), we will keep looping around this, without taking into account the setting of
IRQ_DISABLED.
Ah yes.
Quoted 4 lines
This is something that we might want to fix, so that the loop is bound by
IRQ_PENDINGbeing set, andIRQ_DISABLEDnot, ie:while ( (desc->status & (IRQ_PENDING | IRQ_DISABLED)) == IRQ_PENDING )
Or perhaps ahead of the loop
desc->status &= ~IRQ_REPLAY;
if ( desc->status & IRQ_DISABLED )
goto out;
desc->status |= IRQ_PENDING;
/*
* Since we set PENDING, if another processor is handling a different
* instance of this same irq, the other processor will take care of it.
*/
if ( desc->status & IRQ_INPROGRESS )
goto out;
desc->status |= IRQ_INPROGRESS;thus also having the comment no longer describe only part of the conditional.
(While looking, I also notice that IRQ_REPLAY has been completely useless for perhaps a very long time. I'll make a patch to zap that.)
Jan
Quoted 65 lines · Jan Beulich
Quoted 35 lines · Roger Pau Monné
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 6 lines · Jan Beulich
Review
Quoted 2 lines · Roger Pau Monne
(quoted diff) +2
-0+ if ( uart->force_polling ) + return;As the IRQ was disabled, is this even possible? I.e. should this be some kind of assertion or alike?
Quoted 8 lines
Quoted 4 lines
Hm, I wasn't setting
IRQ_DISABLEDbefore, and hence needed this guard. But now withIRQ_DISABLEDbeing set in ->statusdo_IRQ()should filter any stray interrupts. I will attempt to add anASSERT_UNREACHABLE()here.Simply
ASSERT(!uart->force_polling)should do here? It is not wrong to run the code below in release builds in such an event. If we kept getting interrupts (perhaps at a high frequency) we'd be in trouble anyway.No, I'm afraid I can't do it like that, I can't put an ASSERT there, because we can still get into
ns16550_interrupt()after the interrupt has been disabled. Indo_IRQ()we have the following loop:while ( desc->status & IRQ_PENDING ) { desc->status &= ~IRQ_PENDING; spin_unlock_irq(&desc->lock); tsc_in = tb_init_done ? get_cycles() : 0; action->handler(irq, action->dev_id); TRACE_TIME(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles()); spin_lock_irq(&desc->lock); }Quoted 3 lines
So if the device is generating further interrupts in the window with IRQs enabled (while we execute the handler), we will keep looping around this, without taking into account the setting of
IRQ_DISABLED.Ah yes.
Quoted 4 lines
This is something that we might want to fix, so that the loop is bound by
IRQ_PENDINGbeing set, andIRQ_DISABLEDnot, ie:while ( (desc->status & (IRQ_PENDING | IRQ_DISABLED)) == IRQ_PENDING )Or perhaps ahead of the loop
desc->status &= ~IRQ_REPLAY; if ( desc->status & IRQ_DISABLED ) goto out; desc->status |= IRQ_PENDING; /* * Since we set PENDING, if another processor is handling a different * instance of this same irq, the other processor will take care of it. */ if ( desc->status & IRQ_INPROGRESS ) goto out; desc->status |= IRQ_INPROGRESS;
Quoted 1 line
thus also having the comment no longer describe only part of the conditional.
I think this is racy. An interrupt hitting in the window with interrupts enabled ahead of the handler having set IRQ_DISABLED will still set IRQ_PENDING, and thus the loop would get executed a further time, and the handler called after IRQ_DISABLED having been set.
I think we need an extra condition in the loop, I see no way this can be solved only by dealing with the concurrent setting of IRQ_PENDING.
Thanks, Roger.
Quoted 64 lines · Roger Pau Monné
Quoted 63 lines · Jan Beulich
Quoted 35 lines · Roger Pau Monné
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 6 lines · Jan Beulich
Review
Quoted 2 lines · Roger Pau Monne
(quoted diff) +2
-0+ if ( uart->force_polling ) + return;As the IRQ was disabled, is this even possible? I.e. should this be some kind of assertion or alike?
Quoted 8 lines
Quoted 4 lines
Hm, I wasn't setting
IRQ_DISABLEDbefore, and hence needed this guard. But now withIRQ_DISABLEDbeing set in ->statusdo_IRQ()should filter any stray interrupts. I will attempt to add anASSERT_UNREACHABLE()here.Simply
ASSERT(!uart->force_polling)should do here? It is not wrong to run the code below in release builds in such an event. If we kept getting interrupts (perhaps at a high frequency) we'd be in trouble anyway.No, I'm afraid I can't do it like that, I can't put an ASSERT there, because we can still get into
ns16550_interrupt()after the interrupt has been disabled. Indo_IRQ()we have the following loop:while ( desc->status & IRQ_PENDING ) { desc->status &= ~IRQ_PENDING; spin_unlock_irq(&desc->lock); tsc_in = tb_init_done ? get_cycles() : 0; action->handler(irq, action->dev_id); TRACE_TIME(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles()); spin_lock_irq(&desc->lock); }Quoted 3 lines
So if the device is generating further interrupts in the window with IRQs enabled (while we execute the handler), we will keep looping around this, without taking into account the setting of
IRQ_DISABLED.Ah yes.
Quoted 4 lines
This is something that we might want to fix, so that the loop is bound by
IRQ_PENDINGbeing set, andIRQ_DISABLEDnot, ie:while ( (desc->status & (IRQ_PENDING | IRQ_DISABLED)) == IRQ_PENDING )Or perhaps ahead of the loop
desc->status &= ~IRQ_REPLAY; if ( desc->status & IRQ_DISABLED ) goto out; desc->status |= IRQ_PENDING; /* * Since we set PENDING, if another processor is handling a different * instance of this same irq, the other processor will take care of it. */ if ( desc->status & IRQ_INPROGRESS ) goto out;
Quoted 8 lines
Quoted 3 lines
desc->status |= IRQ_INPROGRESS;thus also having the comment no longer describe only part of the conditional.
I think this is racy. An interrupt hitting in the window with interrupts enabled ahead of the handler having set
IRQ_DISABLEDwill still setIRQ_PENDING, and thus the loop would get executed a further time, and the handler called afterIRQ_DISABLEDhaving been set.
Hmm, I don't quite agree with how you put it, but I think I see what you mean. There's one question here, though: If PENDING is set first, and DISABLED only later, shouldn't that IRQ instance still be handled? If so, ...
Quoted 2 lines
I think we need an extra condition in the loop, I see no way this can be solved only by dealing with the concurrent setting of
IRQ_PENDING.
... such an extra condition would be wrong. If not, yes, I agree.
Jan
Quoted 74 lines · Jan Beulich
Quoted 64 lines · Roger Pau Monné
Quoted 63 lines · Jan Beulich
Quoted 35 lines · Roger Pau Monné
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 6 lines · Jan Beulich
Review
Quoted 2 lines · Roger Pau Monne
(quoted diff) +2
-0+ if ( uart->force_polling ) + return;As the IRQ was disabled, is this even possible? I.e. should this be some kind of assertion or alike?
Quoted 8 lines
Quoted 4 lines
Hm, I wasn't setting
IRQ_DISABLEDbefore, and hence needed this guard. But now withIRQ_DISABLEDbeing set in ->statusdo_IRQ()should filter any stray interrupts. I will attempt to add anASSERT_UNREACHABLE()here.Simply
ASSERT(!uart->force_polling)should do here? It is not wrong to run the code below in release builds in such an event. If we kept getting interrupts (perhaps at a high frequency) we'd be in trouble anyway.No, I'm afraid I can't do it like that, I can't put an ASSERT there, because we can still get into
ns16550_interrupt()after the interrupt has been disabled. Indo_IRQ()we have the following loop:while ( desc->status & IRQ_PENDING ) { desc->status &= ~IRQ_PENDING; spin_unlock_irq(&desc->lock); tsc_in = tb_init_done ? get_cycles() : 0; action->handler(irq, action->dev_id); TRACE_TIME(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles()); spin_lock_irq(&desc->lock); }Quoted 3 lines
So if the device is generating further interrupts in the window with IRQs enabled (while we execute the handler), we will keep looping around this, without taking into account the setting of
IRQ_DISABLED.Ah yes.
Quoted 4 lines
This is something that we might want to fix, so that the loop is bound by
IRQ_PENDINGbeing set, andIRQ_DISABLEDnot, ie:while ( (desc->status & (IRQ_PENDING | IRQ_DISABLED)) == IRQ_PENDING )Or perhaps ahead of the loop
desc->status &= ~IRQ_REPLAY; if ( desc->status & IRQ_DISABLED ) goto out; desc->status |= IRQ_PENDING; /* * Since we set PENDING, if another processor is handling a different * instance of this same irq, the other processor will take care of it. */ if ( desc->status & IRQ_INPROGRESS ) goto out;Quoted 8 lines
Quoted 3 lines
desc->status |= IRQ_INPROGRESS;thus also having the comment no longer describe only part of the conditional.
I think this is racy. An interrupt hitting in the window with interrupts enabled ahead of the handler having set
IRQ_DISABLEDwill still setIRQ_PENDING, and thus the loop would get executed a further time, and the handler called afterIRQ_DISABLEDhaving been set.
Quoted 8 lines
Hmm, I don't quite agree with how you put it, but I think I see what you mean. There's one question here, though: If PENDING is set first, and DISABLED only later, shouldn't that IRQ instance still be handled? If so, ...
Quoted 2 lines
I think we need an extra condition in the loop, I see no way this can be solved only by dealing with the concurrent setting of
IRQ_PENDING.... such an extra condition would be wrong. If not, yes, I agree.
But PENDING is always set, regardless of whether the IRQ is disabled, the normal flow in do_IRQ() is:
desc->status |= IRQ_PENDING;
/*
* Since we set PENDING, if another processor is handling a different
* instance of this same irq, the other processor will take care of it.
*/
if ( desc->status & (IRQ_DISABLED | IRQ_INPROGRESS) )
goto out;I think it's valid to have both PENDING and DISABLED set with the current logic. In fact, the code in ack_edge_ioapic_irq() relies on having both PENDING and DISABLED set to mask the source, as the ->disable hook for edge triggered IO-APIC pins is a no-op.
We could likely change all this to be more straight forward, but as with the serial interrupt handling I would rather not do that change during a code freeze.
Thanks, Roger.
Quoted 87 lines · Roger Pau Monné
Quoted 74 lines · Jan Beulich
Quoted 64 lines · Roger Pau Monné
Quoted 63 lines · Jan Beulich
Quoted 35 lines · Roger Pau Monné
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 6 lines · Jan Beulich
Review
Quoted 2 lines · Roger Pau Monne
(quoted diff) +2
-0+ if ( uart->force_polling ) + return;As the IRQ was disabled, is this even possible? I.e. should this be some kind of assertion or alike?
Quoted 8 lines
Quoted 4 lines
Hm, I wasn't setting
IRQ_DISABLEDbefore, and hence needed this guard. But now withIRQ_DISABLEDbeing set in ->statusdo_IRQ()should filter any stray interrupts. I will attempt to add anASSERT_UNREACHABLE()here.Simply
ASSERT(!uart->force_polling)should do here? It is not wrong to run the code below in release builds in such an event. If we kept getting interrupts (perhaps at a high frequency) we'd be in trouble anyway.No, I'm afraid I can't do it like that, I can't put an ASSERT there, because we can still get into
ns16550_interrupt()after the interrupt has been disabled. Indo_IRQ()we have the following loop:while ( desc->status & IRQ_PENDING ) { desc->status &= ~IRQ_PENDING; spin_unlock_irq(&desc->lock); tsc_in = tb_init_done ? get_cycles() : 0; action->handler(irq, action->dev_id); TRACE_TIME(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles()); spin_lock_irq(&desc->lock); }Quoted 3 lines
So if the device is generating further interrupts in the window with IRQs enabled (while we execute the handler), we will keep looping around this, without taking into account the setting of
IRQ_DISABLED.Ah yes.
Quoted 4 lines
This is something that we might want to fix, so that the loop is bound by
IRQ_PENDINGbeing set, andIRQ_DISABLEDnot, ie:while ( (desc->status & (IRQ_PENDING | IRQ_DISABLED)) == IRQ_PENDING )Or perhaps ahead of the loop
desc->status &= ~IRQ_REPLAY; if ( desc->status & IRQ_DISABLED ) goto out; desc->status |= IRQ_PENDING; /* * Since we set PENDING, if another processor is handling a different * instance of this same irq, the other processor will take care of it. */ if ( desc->status & IRQ_INPROGRESS ) goto out;Quoted 8 lines
Quoted 3 lines
desc->status |= IRQ_INPROGRESS;thus also having the comment no longer describe only part of the conditional.
I think this is racy. An interrupt hitting in the window with interrupts enabled ahead of the handler having set
IRQ_DISABLEDwill still setIRQ_PENDING, and thus the loop would get executed a further time, and the handler called afterIRQ_DISABLEDhaving been set.Quoted 8 lines
Hmm, I don't quite agree with how you put it, but I think I see what you mean. There's one question here, though: If PENDING is set first, and DISABLED only later, shouldn't that IRQ instance still be handled? If so, ...
Quoted 2 lines
I think we need an extra condition in the loop, I see no way this can be solved only by dealing with the concurrent setting of
IRQ_PENDING.... such an extra condition would be wrong. If not, yes, I agree.
But PENDING is always set, regardless of whether the IRQ is disabled, the normal flow in
do_IRQ()is:
Quoted 8 lines
desc->status |= IRQ_PENDING;
/*
* Since we set PENDING, if another processor is handling a different
* instance of this same irq, the other processor will take care of it.
*/
if ( desc->status & (IRQ_DISABLED | IRQ_INPROGRESS) )
goto out;Well, see the adjusted flow I did suggest earlier (still in context above).
Quoted 4 lines
I think it's valid to have both PENDING and DISABLED set with the current logic. In fact, the code in
ack_edge_ioapic_irq()relies on having both PENDING and DISABLED set to mask the source, as the ->disable hook for edge triggered IO-APIC pins is a no-op.
Yet this can be of use for a corner case only anyway, as we set PENDING only after having called ->ack(). That is, after setting PENDING _another_ IRQ has to fire. Which is possible, but likely can be dealt with differently.
Quoted 3 lines
We could likely change all this to be more straight forward, but as with the serial interrupt handling I would rather not do that change during a code freeze.
I definitely agree here. So perhaps indeed best to go with what you did proposed.
Jan
Quoted 90 lines · Jan Beulich
Quoted 89 lines · Roger Pau Monné
Quoted 74 lines · Jan Beulich
Quoted 64 lines · Roger Pau Monné
Quoted 63 lines · Jan Beulich
Quoted 35 lines · Roger Pau Monné
Quoted 9 lines · Jan Beulich
Quoted 8 lines · Roger Pau Monné
Quoted 6 lines · Jan Beulich
Review
Quoted 2 lines · Roger Pau Monne
(quoted diff) +2
-0+ if ( uart->force_polling ) + return;As the IRQ was disabled, is this even possible? I.e. should this be some kind of assertion or alike?
Quoted 8 lines
Quoted 4 lines
Hm, I wasn't setting
IRQ_DISABLEDbefore, and hence needed this guard. But now withIRQ_DISABLEDbeing set in ->statusdo_IRQ()should filter any stray interrupts. I will attempt to add anASSERT_UNREACHABLE()here.Simply
ASSERT(!uart->force_polling)should do here? It is not wrong to run the code below in release builds in such an event. If we kept getting interrupts (perhaps at a high frequency) we'd be in trouble anyway.No, I'm afraid I can't do it like that, I can't put an ASSERT there, because we can still get into
ns16550_interrupt()after the interrupt has been disabled. Indo_IRQ()we have the following loop:while ( desc->status & IRQ_PENDING ) { desc->status &= ~IRQ_PENDING; spin_unlock_irq(&desc->lock); tsc_in = tb_init_done ? get_cycles() : 0; action->handler(irq, action->dev_id); TRACE_TIME(TRC_HW_IRQ_HANDLED, irq, tsc_in, get_cycles()); spin_lock_irq(&desc->lock); }Quoted 3 lines
So if the device is generating further interrupts in the window with IRQs enabled (while we execute the handler), we will keep looping around this, without taking into account the setting of
IRQ_DISABLED.Ah yes.
Quoted 4 lines
This is something that we might want to fix, so that the loop is bound by
IRQ_PENDINGbeing set, andIRQ_DISABLEDnot, ie:while ( (desc->status & (IRQ_PENDING | IRQ_DISABLED)) == IRQ_PENDING )Or perhaps ahead of the loop
desc->status &= ~IRQ_REPLAY; if ( desc->status & IRQ_DISABLED ) goto out; desc->status |= IRQ_PENDING; /* * Since we set PENDING, if another processor is handling a different * instance of this same irq, the other processor will take care of it. */ if ( desc->status & IRQ_INPROGRESS ) goto out;Quoted 8 lines
Quoted 3 lines
desc->status |= IRQ_INPROGRESS;thus also having the comment no longer describe only part of the conditional.
I think this is racy. An interrupt hitting in the window with interrupts enabled ahead of the handler having set
IRQ_DISABLEDwill still setIRQ_PENDING, and thus the loop would get executed a further time, and the handler called afterIRQ_DISABLEDhaving been set.Quoted 8 lines
Hmm, I don't quite agree with how you put it, but I think I see what you mean. There's one question here, though: If PENDING is set first, and DISABLED only later, shouldn't that IRQ instance still be handled? If so, ...
Quoted 2 lines
I think we need an extra condition in the loop, I see no way this can be solved only by dealing with the concurrent setting of
IRQ_PENDING.... such an extra condition would be wrong. If not, yes, I agree.
But PENDING is always set, regardless of whether the IRQ is disabled, the normal flow in
do_IRQ()is:desc->status |= IRQ_PENDING;
Quoted 8 lines
Quoted 6 lines
/* * Since we set PENDING, if another processor is handling a different * instance of this same irq, the other processor will take care of it. */ if ( desc->status & (IRQ_DISABLED | IRQ_INPROGRESS) ) goto out;Well, see the adjusted flow I did suggest earlier (still in context above).
But that flow above could still set PENDING ahead of the handler having set DISABLED, and hence won't avoid at least an extra handler call despite the IRQ having been disabled? IOW: I think it's partially racy w.r.t setting DISABLED in the handler.
Quoted 8 lines
Quoted 4 lines
I think it's valid to have both PENDING and DISABLED set with the current logic. In fact, the code in
ack_edge_ioapic_irq()relies on having both PENDING and DISABLED set to mask the source, as the ->disable hook for edge triggered IO-APIC pins is a no-op.Yet this can be of use for a corner case only anyway, as we set PENDING only after having called ->ack(). That is, after setting PENDING _another_ IRQ has to fire. Which is possible, but likely can be dealt with differently.
Well, if the ->disable handler wasn't a no-op for edge triggered interrupts it wouldn't require this weird logic. However then IRQ migration will likely involve a mask/unmask for edge triggered interrupts, which we want to avoid in general. I would need to think about it.
Quoted 6 lines
Quoted 3 lines
We could likely change all this to be more straight forward, but as with the serial interrupt handling I would rather not do that change during a code freeze.
I definitely agree here. So perhaps indeed best to go with what you did proposed.
Thanks, let me see if I can sort out a minimally invasive solution.
Regards, Roger.