ns16550: bound interrupt handler execution time
v2 cover ns16550: bound interrupt handler execution time
Hello,
Same approach as v1, but the introduction of a generic disable_irq() handler requires two pre-patches for PowerPC and RISCV. There's also an extra fix for dealing with the interrupt being disabled while executing the handler in do_IRQ().
Thanks, Roger.
Roger Pau Monne (4):
riscv/irq: define a per-arch irq_to_desc()
xen/ppc: introduce a dummy irq_to_desc()
xen/irq: handle IRQ being disabled while executing its handler char/ns16550: bound execution time of ns16550_interrupt()
xen/arch/ppc/include/asm/irq.h | 6 ++++++ xen/arch/riscv/include/asm/irq.h | 4 ++++ xen/arch/riscv/irq.c | 5 +++++ xen/arch/x86/irq.c | 4 +++- xen/common/irq.c | 12 ++++++++++++ xen/drivers/char/ns16550.c | 30 +++++++++++++++++++++++++++++- xen/include/xen/irq.h | 1 + 7 files changed, 60 insertions(+), 2 deletions(-)
Signature · 1 line
2.53.0
PATCH v2 1/4 riscv/irq: define a per-arch irq_to_desc()
RISCV declares irq_desc as a static array, opposed to x86 that uses a pointer allocated at boot time. This creates issues when attempting to add an extern declaration for irq_desc, as asm/irq.h is included by xen/irq.h where the definition of struct irq_desc resides, and an empty forward declaration doesn't make the compiler happy because it doesn't know the type data-storage.
--- xen/arch/riscv/include/asm/irq.h | 4 ++++ xen/arch/riscv/irq.c | 5 +++++ 2 files changed, 9 insertions(+)
xen/arch/riscv/include/asm/irq.h +4 -0
@@ -35,6 +35,10 @@ struct arch_irq_desc {unsigned int type; }; +struct irq_desc ; +struct irq_desc *irq_to_desc (unsigned int irq); +#define irq_to_desc irq_to_desc +struct cpu_user_regs ;struct dt_device_node ;
xen/arch/riscv/irq.c +5 -0
@@ -19,6 +19,11 @@static irq_desc_t irq_desc[NR_IRQS ]; +struct irq_desc *irq_to_desc (unsigned int irq) +{ +return &irq_desc[irq]; +} +static bool irq_validate_new_type (unsigned int curr,unsigned int new) {return curr ==IRQ_TYPE_INVALID || curr == new;
Signature · 1 line
2.53.0
Re: [PATCH for-4.22 v2 1/4] riscv/irq: define a per-arch irq_to_desc()
Quoted 34 lines · Roger Pau Monne
RISCV declares
irq_descas a static array, opposed to x86 that uses a pointer allocated at boot time. This creates issues when attempting to add an extern declaration forirq_desc, asasm/irq.his included byxen/irq.hwhere the definition of structirq_descresides, and an empty forward declaration doesn't make the compiler happy because it doesn't know the type data-storage.Signed-off-by Roger Pau Monné <[email protected]>--- xen/arch/riscv/include/asm/irq.h | 4 ++++ xen/arch/riscv/irq.c | 5 +++++ 2 files changed, 9 insertions(+)xen/arch/riscv/include/asm/irq.h +4
-0@@ -35,6 +35,10 @@ struct arch_irq_desc {unsigned int type; }; +struct irq_desc ; +struct irq_desc *irq_to_desc (unsigned int irq); +#define irq_to_desc irq_to_desc +struct cpu_user_regs ;struct dt_device_node ;xen/arch/riscv/irq.c +0
-0@@ -19,6 +19,11 @@
xen/arch/riscv/irq.c line 29
Quoted 10 lines
xen/arch/riscv/irq.c +5
-0static irq_desc_t irq_desc[NR_IRQS ]; +struct irq_desc *irq_to_desc (unsigned int irq) +{ +return &irq_desc[irq]; +} +static bool irq_validate_new_type (unsigned int curr,unsigned int new) {return curr ==IRQ_TYPE_INVALID || curr == new;
Changes looks good to me.
There are similar changes in my dom0less enablement config for RISC-V but considering that likely this patch will be in staging faster then my patch series:
Thanks.
~ Oleksii
Re: [PATCH for-4.22 v2 1/4] riscv/irq: define a per-arch irq_to_desc()
In reply to Roger Pau Monnexen/arch/riscv/include/asm/irq.h struct arch_irq_desc { · line 38
Quoted 7 lines · Roger Pau Monne
xen/arch/riscv/include/asm/irq.h +1
-0@@ -35,6 +35,10 @@ struct arch_irq_desc {unsigned int type; }; +struct irq_desc ;
Nit: You don't need this here, as ...
xen/arch/riscv/include/asm/irq.h struct arch_irq_desc { · line 39
Quoted 1 line
xen/arch/riscv/include/asm/irq.h +1
-0+struct irq_desc *irq_to_desc (unsigned int irq);
... it's the function return type, not a parameter one.
xen/arch/riscv/irq.c line 25
Quoted 10 lines
xen/arch/riscv/irq.c +4
-0@@ -19,6 +19,11 @@static irq_desc_t irq_desc[NR_IRQS ]; +struct irq_desc *irq_to_desc (unsigned int irq) +{ +return &irq_desc[irq]; +}
Maybe better assert that the incoming number is in range? By this not (possibly, in the current arrangements) being an inline function, the compiler can't assist.
Jan
PATCH v2 2/4 xen/ppc: introduce a dummy irq_to_desc()
In reply to Roger Pau MonneIn preparation for irq_to_desc() being called by common IRQ code. PowerPC doesn't have an irq_desc array defined, so it cannot use the generic irq_to_desc macro in the common header.
--- xen/arch/ppc/include/asm/irq.h | 6 ++++++ 1 file changed, 6 insertions(+)
xen/arch/ppc/include/asm/irq.h +6 -0
@@ -30,4 +30,10 @@ static inline int platform_get_irq(const struct dt_device_node *device, int inde BUG_ON("unimplemented"); } +static inline void *irq_to_desc (unsigned int irq) +{ +BUG_ON ("unimplemented" ); +} +#define irq_to_desc irq_to_desc +#endif /* __ASM_PPC_IRQ_H__ */
Signature · 1 line
2.53.0
Re: [PATCH for-4.22 v2 2/4] xen/ppc: introduce a dummy irq_to_desc()
Quoted 5 lines · Roger Pau Monne
In preparation for
irq_to_desc()being called by common IRQ code. PowerPC doesn't have anirq_descarray defined, so it cannot use the genericirq_to_descmacro in the common header.Signed-off-by Roger Pau Monné <[email protected]>
Re: [PATCH for-4.22 v2 2/4] xen/ppc: introduce a dummy irq_to_desc()
In reply to Roger Pau MonneQuoted 14 lines · Roger Pau Monne
In preparation for
irq_to_desc()being called by common IRQ code. PowerPC doesn't have anirq_descarray defined, so it cannot use the genericirq_to_descmacro in the common header.Signed-off-by Roger Pau Monné <[email protected]>--- xen/arch/ppc/include/asm/irq.h | 6 ++++++ 1 file changed, 6 insertions(+)xen/arch/ppc/include/asm/irq.h +0
-0
xen/arch/ppc/include/asm/irq.h static inline int platform_get_irq(const struct dt_device_node *device, int inde · line 39
Quoted 10 lines
xen/arch/ppc/include/asm/irq.h +6
-0@@ -30,4 +30,10 @@ static inline int platform_get_irq(const struct dt_device_node *device, int inde BUG_ON("unimplemented"); } +static inline void *irq_to_desc (unsigned int irq) +{ +BUG_ON ("unimplemented" ); +} +#define irq_to_desc irq_to_desc +#endif /* __ASM_PPC_IRQ_H__ */
Thanks.
~ Oleksii
Re: [PATCH for-4.22 v2 2/4] xen/ppc: introduce a dummy irq_to_desc()
Quoted 21 lines · Oleksii Kurochko
Quoted 18 lines · Roger Pau Monne
In preparation for
irq_to_desc()being called by common IRQ code. PowerPC doesn't have anirq_descarray defined, so it cannot use the genericirq_to_descmacro in the common header.Signed-off-by Roger Pau Monné <[email protected]>---
xen/arch/ppc/include/asm/irq.h| 6 ++++++1 file changed, 6 insertions(+)xen/arch/ppc/include/asm/irq.h +0
-0@@ -30,4 +30,10 @@ static inline int platform_get_irq(const struct
dt_device_node*device, int indeBUG_ON("unimplemented");
}
+static inline void *irq_to_desc(unsigned int irq)
Quoted 8 lines
Quoted 6 lines
xen/arch/ppc/include/asm/irq.h +5
-0+{ + BUG_ON("unimplemented"); +} +#define irq_to_desc irq_to_desc +#endif /*
__ASM_PPC_IRQ_H__*/Reviewed-by Oleksii Kurochko <[email protected]>
~ Oleksii
PATCH v2 3/4 xen/irq: handle IRQ being disabled while executing its handler
In reply to Roger Pau MonneIt's possible for IRQ_DISABLED to be set while running the handler execution loop in do_IRQ(). Such disabling can be done by the handler itself, or from a remote CPU.
Check for IRQ_DISABLED not being set in the loop condition, as to not execute the handler if the IRQ has been disabled.
---
I wanted to add a fixes tag, but this has been broken since forever, and hence I should add:
Which is pointless IMO.
--- xen/arch/x86/irq.c | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-)
xen/arch/x86/irq.c +3 -1
@@ -2088,7 +2088,9 @@ void do_IRQ(struct cpu_user_regs *regs) desc->status |=IRQ_INPROGRESS ; action = desc->action; -while ( desc->status &IRQ_PENDING ) + +/* Deal with IRQ_DISABLED being set while inside the loop body. */ +while ( (desc->status & (IRQ_PENDING |IRQ_DISABLED )) ==IRQ_PENDING ) { desc->status &= ~IRQ_PENDING ;spin_unlock_irq (&desc->lock);
Signature · 1 line
2.53.0
Re: [PATCH for-4.22 v2 3/4] xen/irq: handle IRQ being disabled while executing its handler
Quoted 3 lines · Roger Pau Monne
It's possible for
IRQ_DISABLEDto be set while running the handler execution loop indo_IRQ(). Such disabling can be done by the handler itself, or from a remote CPU.
I don't think the handler itself can legitimately call any of the functions setting IRQ_DISABLED.
Quoted 4 lines
Check for
IRQ_DISABLEDnot being set in the loop condition, as to not execute the handler if the IRQ has been disabled.Signed-off-by Roger Pau Monné <[email protected]>
Preferably with the description adjusted (or it being pointed out what I may be overlooking):
Quoted 7 lines
---
I wanted to add a fixes tag, but this has been broken since forever, and hence I should add:Fixes 4676bbf96dc8 ("bitkeeper revision 1.2 (3ddb79c9KusG02eh7i-uXkgY0IksKA)")Which is pointless IMO.
+1
Jan
Re: [PATCH for-4.22 v2 3/4] xen/irq: handle IRQ being disabled while executing its handler
Quoted 7 lines · Jan Beulich
Quoted 3 lines · Roger Pau Monne
It's possible for
IRQ_DISABLEDto be set while running the handler execution loop indo_IRQ(). Such disabling can be done by the handler itself, or from a remote CPU.I don't think the handler itself can legitimately call any of the functions setting
IRQ_DISABLED.
Hm, yes, we still have no interrupt handlers that set IRQ_DISABLED, but we will gain one in the next patch, where ns16550_interrupt() will call disable_irq() that does set IRQ_DISABLED.
Quoted 8 lines
Quoted 4 lines
Check for
IRQ_DISABLEDnot being set in the loop condition, as to not execute the handler if the IRQ has been disabled.Signed-off-by Roger Pau Monné <[email protected]>Preferably with the description adjusted (or it being pointed out what I may be overlooking):
Reviewed-by Jan Beulich <[email protected]>
I'm happy to clarify the commit message to note that while we have no instances ATM, but that future changes might introduce some.
Thanks, Roger.
Re: [PATCH for-4.22 v2 3/4] xen/irq: handle IRQ being disabled while executing its handler
Quoted 16 lines · Roger Pau Monné
Quoted 7 lines · Jan Beulich
Quoted 3 lines · Roger Pau Monne
It's possible for
IRQ_DISABLEDto be set while running the handler execution loop indo_IRQ(). Such disabling can be done by the handler itself, or from a remote CPU.I don't think the handler itself can legitimately call any of the functions setting
IRQ_DISABLED.Hm, yes, we still have no interrupt handlers that set
IRQ_DISABLED, but we will gain one in the next patch, wherens16550_interrupt()will calldisable_irq()that does setIRQ_DISABLED.Quoted 3 lines
Quoted 3 lines
Check for
IRQ_DISABLEDnot being set in the loop condition, as to not execute the handler if the IRQ has been disabled.
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Signed-off-by Roger Pau Monné <[email protected]>Preferably with the description adjusted (or it being pointed out what I may be overlooking):
Reviewed-by Jan Beulich <[email protected]>I'm happy to clarify the commit message to note that while we have no instances ATM, but that future changes might introduce some.
Better leave as is.
Jan
Re: [PATCH for-4.22 v2 3/4] xen/irq: handle IRQ being disabled while executing its handler
In reply to Roger Pau MonneQuoted 8 lines · Roger Pau Monne
It's possible for
IRQ_DISABLEDto be set while running the handler execution loop indo_IRQ(). Such disabling can be done by the handler itself, or from a remote CPU.Check for
IRQ_DISABLEDnot being set in the loop condition, as to not execute the handler if the IRQ has been disabled.Signed-off-by Roger Pau Monné <[email protected]>
Quoted 8 lines
---
I wanted to add a fixes tag, but this has been broken since forever, and hence I should add:Fixes 4676bbf96dc8 ("bitkeeper revision 1.2 (3ddb79c9KusG02eh7i-uXkgY0IksKA)")Which is pointless IMO.
---
Thanks.
~ Oleksii
PATCH v2 4/4 char/ns16550: bound execution time of ns16550_interrupt()
In reply to Roger Pau MonneThe current logic in ns16550_interrupt() will loop until the device sets the NOINT in IIR. At least on the Lenovo ThinkSystem SR630 V4 the flow control of the serial-over-lan emulated UART seems to be broken, as it doesn't set the NOINT bit consistently. The Transmitter Holding Register Empty in LSR also seems to not be properly signaled, as even with it set writes to the transmit register take ~6ms. This leads to the watchdog triggering very easily on such system.
Introduce an upper bound on the execution time of ns16550_interrupt(), this is currently set as 4x the polling interval, which is calculated as the time to fill RX FIFO and/or empty TX FIFO. The current maximum is 5ms. Once the timeout triggers the interrupt is disabled and the uart is switched to polling mode.
---
Changes since v1:
- Move irq disabling to its own helper.
- Turn force_polling check in interrupt handler into an ASSERT().
- Improve logic for timeout calculation.
--- xen/common/irq.c | 12 ++++++++++++ xen/drivers/char/ns16550.c | 30 +++++++++++++++++++++++++++++- xen/include/xen/irq.h | 1 + 3 files changed, 42 insertions(+), 1 deletion(-)
xen/common/irq.c +12 -0
@@ -54,3 +54,15 @@ unsigned int cf_check irq_startup_none(struct irq_desc *desc) {return 0 ; } + +void disable_irq (unsigned int irq) +{ +struct irq_desc *desc =irq_to_desc (irq); +unsigned long flags; + +spin_lock_irqsave (&desc->lock, flags); + desc->status |=IRQ_DISABLED ; +if ( desc->handler->disable ) + desc->handler->disable (desc); +spin_unlock_irqrestore (&desc->lock, flags); +}
xen/drivers/char/ns16550.c +29 -1
@@ -62,6 +62,7 @@ static struct ns16550 {#endif unsigned int timeout_ms;bool intr_works; +bool force_polling;bool dw_usr_bsy;#ifdef NS16550_PCI /* PCI card parameters. */
@@ -190,12 +191,38 @@ static void cf_check ns16550_interrupt(int irq, void *dev_id) {struct serial_port *port = dev_id;struct ns16550 *uart = port->uart; +/* +* Set quite arbitrarily as 4x the time to drain the TX or fill RX FIFOs, +* set the upper bound as 5ms or the timeout_ms value, whatever is higher. +*/ +const unsigned int delta =min (uart->timeout_ms *4 , +max (5u , uart->timeout_ms)); +const s_time_t timeout =NOW () +MILLISECS (delta); +ASSERT (!uart->force_polling); uart->intr_works =1 ;while ( !(ns_read_reg (uart,UART_IIR ) &UART_IIR_NOINT ) ) {u8 lsr =ns_read_reg (uart,UART_LSR ); +s_time_t now =NOW (); + +/* Break out of the loop if spending too much time. */ +if ( now > timeout ) + { +/* Disable the interrupt source - it's never shared. */ +disable_irq (irq); + +/* Disable interrupt generation on the device and arm the timer. */ + uart->force_polling = true; +ns_write_reg (uart,UART_IER ,0 ); +set_timer (&uart->timer, now +MILLISECS (uart->timeout_ms)); +printk (XENLOG_WARNING +"uart interrupt taking more than %ums, switched to polling\n" , + delta); + +return ; + }if ( (lsr & uart->lsr_mask) == uart->lsr_mask )serial_tx_interrupt (port);
@@ -223,7 +250,7 @@ static void cf_check __ns16550_poll(const struct cpu_user_regs *regs)struct ns16550 *uart = port->uart;const struct cpu_user_regs *old_regs; -if ( uart->intr_works ) +if ( uart->intr_works && !uart->force_polling )return ;/* Interrupts work - no more polling */ /* Mimic interrupt context. */
@@ -313,6 +340,7 @@ static void ns16550_setup_preirq(struct ns16550 *uart)unsigned int divisor; uart->intr_works =0 ; + uart->force_polling = false;pci_serial_early_init (uart);
xen/include/xen/irq.h +1 -0
@@ -121,6 +121,7 @@ extern void release_irq(unsigned int irq, const void *dev_id);extern int request_irq (unsigned int irq,unsigned int irqflags,void (*handler)(int irq,void *dev_id),const char *devname,void *dev_id); +void disable_irq (unsigned int irq);extern const hw_irq_controller no_irq_type; void cf_check no_action(int cpl, void *dev_id);
Signature · 1 line
2.53.0
Re: [PATCH for-4.22 v2 4/4] char/ns16550: bound execution time of ns16550_interrupt()
Quoted 6 lines · Roger Pau Monne
xen/common/irq.c +0
-0@@ -54,3 +54,15 @@ unsigned int cf_check irq_startup_none(struct irq_desc *desc) {return 0 ; }
xen/common/irq.c unsigned int cf_check irq_startup_none(struct irq_desc *desc) · line 66
Quoted 10 lines
xen/common/irq.c +10
-0+ +void disable_irq (unsigned int irq) +{ +struct irq_desc *desc =irq_to_desc (irq); +unsigned long flags; + +spin_lock_irqsave (&desc->lock, flags); + desc->status |=IRQ_DISABLED ; +if ( desc->handler->disable ) + desc->handler->disable (desc);
I'd like to point out that __pirq_guest_unbind() has this the other way around:
Call ->disable(), then set flag. Similarly move_native_irq() only calls the hook with the flag clear. Whereas fixup_irqs() doesn't care about the flag at all. Also considering the wording "disable" vs "disabled", I think setting the flag afterwards is better.
xen/common/irq.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 199
Quoted 10 lines
xen/common/irq.c +6
-0@@ -190,12 +191,38 @@ static void cf_check ns16550_interrupt(int irq, void *dev_id) {struct serial_port *port = dev_id;struct ns16550 *uart = port->uart; +/* +* Set quite arbitrarily as 4x the time to drain the TX or fill RX FIFOs, +* set the upper bound as 5ms or the timeout_ms value, whatever is higher. +*/ +const unsigned int delta =min (uart->timeout_ms *4 , +max (5u , uart->timeout_ms));
You may want to also update the description accordingly.
Preferably with respective adjustments:
Jan
Re: [PATCH for-4.22 v2 4/4] char/ns16550: bound execution time of ns16550_interrupt()
Quoted 15 lines · Jan Beulich
Quoted 14 lines · Roger Pau Monne
xen/common/irq.c +8
-0@@ -54,3 +54,15 @@ unsigned int cf_check irq_startup_none(struct irq_desc *desc) {return 0 ; } + +void disable_irq (unsigned int irq) +{ +struct irq_desc *desc =irq_to_desc (irq); +unsigned long flags; + +spin_lock_irqsave (&desc->lock, flags); + desc->status |=IRQ_DISABLED ;
Quoted 8 lines
xen/common/irq.c unsigned int cf_check irq_startup_none(struct irq_desc *desc) · line 66
Quoted 2 lines
xen/common/irq.c +2
-0+if ( desc->handler->disable ) + desc->handler->disable (desc);I'd like to point out that
__pirq_guest_unbind()has this the other way around:
Call ->disable(), then set flag. Similarlymove_native_irq()only calls the hook with the flag clear. Whereasfixup_irqs()doesn't care about the flag at all. Also considering the wording "disable" vs "disabled", I think setting the flag afterwards is better.
As this was done inside of the locked region I didn't think it mattered much, but yes, the ->disable() handler might itself rely on the previous state.
Quoted 13 lines
xen/common/irq.c static void cf_check ns16550_interrupt(int irq, void *dev_id) · line 199
Quoted 10 lines
xen/common/irq.c +6
-0@@ -190,12 +191,38 @@ static void cf_check ns16550_interrupt(int irq, void *dev_id) {struct serial_port *port = dev_id;struct ns16550 *uart = port->uart; +/* +* Set quite arbitrarily as 4x the time to drain the TX or fill RX FIFOs, +* set the upper bound as 5ms or the timeout_ms value, whatever is higher. +*/ +const unsigned int delta =min (uart->timeout_ms *4 , +max (5u , uart->timeout_ms));You may want to also update the description accordingly.
Quoted 2 lines
Preferably with respective adjustments:
Reviewed-by Jan Beulich <[email protected]>
Thanks, I've applied the comments.
Regards, Roger.
Re: [PATCH for-4.22 v2 4/4] char/ns16550: bound execution time of ns16550_interrupt()
In reply to Roger Pau MonneQuoted 8 lines · Roger Pau Monne
The current logic in
ns16550_interrupt()will loop until the device sets the NOINT in IIR. At least on the Lenovo ThinkSystem SR630 V4 the flow control of the serial-over-lan emulated UART seems to be broken, as it doesn't set the NOINT bit consistently. The Transmitter Holding Register Empty in LSR also seems to not be properly signaled, as even with it set writes to the transmit register take ~6ms. This leads to the watchdog triggering very easily on such system.
Quoted 7 lines
Introduce an upper bound on the execution time of
ns16550_interrupt(), this is currently set as 4x the polling interval, which is calculated as the time to fill RX FIFO and/or empty TX FIFO. The current maximum is 5ms. Once the timeout triggers the interrupt is disabled and the uart is switched to polling mode.Signed-off-by Roger Pau Monné <[email protected]>
Thanks.
~ Oleksii