Skip to contentSkip to conversation

x86/apic: Update virtual irq base for DT/OF based system as well

PATCH Applied Rahul Tanwar 4 participants 9 messages Aug 21, 2019 – Aug 26, 2019 x86

lore
Share & tools
lore
#0 lore raw

PATCH 1/1 x86/apic: Update virtual irq base for DT/OF based system as well

'ioapic_dynirq_base' contains the virtual IRQ base number. Presently, it is updated to the end of hardware IRQ numbers but this is done only when IOAPIC configuration type is IOAPIC_DOMAIN_LEGACY or IOAPIC_DOMAIN_STRICT. There is a third type IOAPIC_DOMAIN_DYNAMIC which applies when IOAPIC configuration comes from devicetree.
Please see dtb_add_ioapic() in arch/x86/kernel/devicetree.c

In case of IOAPIC_DOMAIN_DYNAMIC (DT/OF based system), 'ioapic_dynirq_base' remains to zero initialized value. This means that for OF based systems, virtual IRQ base will get set to zero. Zero value for a virtual IRQ is a invalid value.
Please see https://yarchive.net/comp/linux/zero.html for more details.

Update 'ioapic_dynirq_base' for IOAPIC_DOMAIN_DYNAMIC case as well just like it is presently updated for IOAPIC_DOMAIN_LEGACY & IOAPIC_DOMAIN_STRICT i.e. set the virtual IRQ base to the end of hardware IRQ numbers when IOAPIC configuration comes from devicetree.

---
 arch/x86/kernel/apic/io_apic.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)
arch/x86/kernel/apic/io_apic.c +2 -1
index c7bb6c69f21c..fe50cd91122b 100644
@@ -2336,7 +2336,8 @@ static int mp_irqdomain_create(int ioapic)
 	ip->irqdomain->parent = parent;
 
 	if (cfg->type == IOAPIC_DOMAIN_LEGACY ||
-	    cfg->type == IOAPIC_DOMAIN_STRICT)
+	    cfg->type == IOAPIC_DOMAIN_STRICT ||
+	    cfg->type == IOAPIC_DOMAIN_DYNAMIC)
 		ioapic_dynirq_base = max(ioapic_dynirq_base,
 					 gsi_cfg->gsi_end + 1);
 
Signature · 1 line
2.11.0
#1 lore raw
Quoted 6 lines · Rahul Tanwar

'ioapic_dynirq_base' contains the virtual IRQ base number. Presently, it is updated to the end of hardware IRQ numbers but this is done only when IOAPIC configuration type is IOAPIC_DOMAIN_LEGACY or IOAPIC_DOMAIN_STRICT. There is a third type IOAPIC_DOMAIN_DYNAMIC which applies when IOAPIC configuration comes from devicetree.
Please see dtb_add_ioapic() in arch/x86/kernel/devicetree.c

We know how DT based ioapics are added. No need to point to it.

Quoted 5 lines

In case of IOAPIC_DOMAIN_DYNAMIC (DT/OF based system), 'ioapic_dynirq_base' remains to zero initialized value. This means that for OF based systems, virtual IRQ base will get set to zero. Zero value for a virtual IRQ is a invalid value.
Please see https://yarchive.net/comp/linux/zero.html for more details.

First of all, please do not use random archive links. See Documentation/process/ how links to LKML archives should look like

Secondly, this link is irrelevant. ioapic_dynirq_base has nothing to do with virtual IRQ number 0. It's a boundary for the dynamic allocation of virtual interrupt numbers so that the core allocator does not pick interrupts out of the IOAPIC's fixed interrupt number space.

This can be legitimately 0 when IOAPIC is not enabled at all.

Can you please explain what kind of problem you were seing and what this really fixes?

Thanks,

tglx

#2 lore raw
Quoted 9 lines · Thomas Gleixner

Secondly, this link is irrelevant. ioapic_dynirq_base has nothing to do with virtual IRQ number 0. It's a boundary for the dynamic allocation of virtual interrupt numbers so that the core allocator does not pick interrupts out of the IOAPIC's fixed interrupt number space.

This can be legitimately 0 when IOAPIC is not enabled at all.

Can you please explain what kind of problem you were seing and what this really fixes?

The problem is that device tree infrastructure considers 0 IRQ value as invalid/error

value whereas for ACPI, 0 is a valid value. Without this change, the problem that we

see is that the first driver using of_irq_get_xx() or its variants fails because of 0 IRQ

number. With this change, allocated IRQ number is never 0 so it works ok.

#3 lore raw
Quoted 7 lines · Rahul
Quoted 5 lines · Thomas Gleixner

Secondly, this link is irrelevant. ioapic_dynirq_base has nothing to do with virtual IRQ number 0. It's a boundary for the dynamic allocation of virtual interrupt numbers so that the core allocator does not pick interrupts out of the IOAPIC's fixed interrupt number space.

Quoted 7 lines
Quoted 4 lines

This can be legitimately 0 when IOAPIC is not enabled at all.

Can you please explain what kind of problem you were seing and what this really fixes?

The problem is that device tree infrastructure considers 0 IRQ value as invalid/error value whereas for ACPI, 0 is a valid value.

Sure.

Quoted 3 lines

Without this change, the problem that we see is that the first driver using of_irq_get_xx() or its variants fails because of 0 IRQ number. With this change, allocated IRQ number is never 0 so it works ok.

Well, this still is not a proper explanation. Just because it works does not make it correct in the first place.

ioapic_dynirq_base is pretty much irrelevant for a DT machine. The reason why it exists is that for regular BIOS the interrupt numbers are hard mapped to the IOAPIC pins. ioapic_dynirq_base is used to protect this hard mapped interrupt number space. The core allocator does not allocate from that space unless it is explicitely told to do so, which is the case for IOAPIC_DOMAIN_STRICT where the allocation tells the core to allocate the associated GSI number.

On DT the interrupt number is irrelevant as DT describes the irq controller and the pin to which a device is connected and does not make assumptions about the interrupt number. So the core can freely allocate any available interrupt number except 0. That's already prevented in the core code.

But x86 implements arch_dynirq_lower_bound() which overrides the core limit and because ioapic_dynirq_base is zero in the DT case it allows VIRQ 0 to be allocated which then causes of_irq*() to fail.

So your change prevents that by excluding the 'GSI' range from allocation, which means that the first irq number which is handed out is 24, assumed you have one IOAPIC with 24 pins.

That's fine as the interrupt number space is big enough, but it needs

- a coherent explanation in the changelog

- proper comments to that effect in the code

Also this is presumably a stable candidate and needs a Fixes: ... tag.

Thanks,

tglx

#4 lore raw
Quoted 38 lines · Thomas Gleixner
Quoted 7 lines · Rahul
Quoted 5 lines · Thomas Gleixner

Secondly, this link is irrelevant. ioapic_dynirq_base has nothing to do with virtual IRQ number 0. It's a boundary for the dynamic allocation of virtual interrupt numbers so that the core allocator does not pick interrupts out of the IOAPIC's fixed interrupt number space.

Quoted 7 lines
Quoted 4 lines

This can be legitimately 0 when IOAPIC is not enabled at all.

Can you please explain what kind of problem you were seing and what this really fixes?

The problem is that device tree infrastructure considers 0 IRQ value as invalid/error value whereas for ACPI, 0 is a valid value.

Sure.

Quoted 3 lines

Without this change, the problem that we see is that the first driver using of_irq_get_xx() or its variants fails because of 0 IRQ number. With this change, allocated IRQ number is never 0 so it works ok.

Well, this still is not a proper explanation. Just because it works does not make it correct in the first place.

ioapic_dynirq_base is pretty much irrelevant for a DT machine. The reason why it exists is that for regular BIOS the interrupt numbers are hard mapped to the IOAPIC pins. ioapic_dynirq_base is used to protect this hard mapped interrupt number space. The core allocator does not allocate from that space unless it is explicitely told to do so, which is the case for IOAPIC_DOMAIN_STRICT where the allocation tells the core to allocate the associated GSI number.

On DT the interrupt number is irrelevant as DT describes the irq controller and the pin to which a device is connected and does not make assumptions about the interrupt number. So the core can freely allocate any available interrupt number except 0. That's already prevented in the core code.

Quoted 7 lines

But x86 implements arch_dynirq_lower_bound() which overrides the core limit and because ioapic_dynirq_base is zero in the DT case it allows VIRQ 0 to be allocated which then causes of_irq*() to fail.

So your change prevents that by excluding the 'GSI' range from allocation, which means that the first irq number which is handed out is 24, assumed you have one IOAPIC with 24 pins.

I have tested this on the ACPI-based system where we have 55 lines of IOAPIC, no PIC, and some GPIO lines. Overall I see that nr_irqs is 512 and shifting by 55 freezes 10% of the space for nothing. Luckily we have SPARSE_IRQS selected for any X86, so, it wouldn't waste memory.

I think we may do slightly better if we just limit the change to the certain cases.

Signature · 2 lines
With Best Regards,
Andy Shevchenko
#5 lore raw
Quoted 9 lines · Andy Shevchenko
Quoted 7 lines · Thomas Gleixner

But x86 implements arch_dynirq_lower_bound() which overrides the core limit and because ioapic_dynirq_base is zero in the DT case it allows VIRQ 0 to be allocated which then causes of_irq*() to fail.

So your change prevents that by excluding the 'GSI' range from allocation, which means that the first irq number which is handed out is 24, assumed you have one IOAPIC with 24 pins.

Quoted 7 lines

I have tested this on the ACPI-based system where we have 55 lines of IOAPIC, no PIC, and some GPIO lines. Overall I see that nr_irqs is 512 and shifting by 55 freezes 10% of the space for nothing. Luckily we have SPARSE_IRQS selected for any X86, so, it wouldn't waste memory.

I think we may do slightly better if we just limit the change to the certain cases.

For DT we can actually avoid that completely. See below.

For ACPI not unfortunately as the stupid GSI mapping is hard coded.

Thanks,

tglx

8<-------------

arch/x86/kernel/apic/io_apic.c +7 -1
@@ -2438,7 +2438,13 @@ unsigned int arch_dynirq_lower_bound(uns
 	 * dmar_alloc_hwirq() may be called before setup_IO_APIC(), so use
 	 * gsi_top if ioapic_dynirq_base hasn't been initialized yet.
 	 */
-	return ioapic_initialized ? ioapic_dynirq_base : gsi_top;
+	if (!ioapic_initialized)
+		return gsi_top;
+	/*
+	 * For DT enabled machines ioapic_dynirq_base is irrelevant and not
+	 * updated. So simply return @from if ioapic_dynirq_base == 0.
+	 */
+	return ioapic_dynirq_base ? : from;
 }
 
 #ifdef CONFIG_X86_32
Tested-by
#6 lore raw
Quoted 15 lines · Thomas Gleixner
Quoted 14 lines · Andy Shevchenko
Quoted 7 lines · Thomas Gleixner

But x86 implements arch_dynirq_lower_bound() which overrides the core limit and because ioapic_dynirq_base is zero in the DT case it allows VIRQ 0 to be allocated which then causes of_irq*() to fail.

So your change prevents that by excluding the 'GSI' range from allocation, which means that the first irq number which is handed out is 24, assumed you have one IOAPIC with 24 pins.

I have tested this on the ACPI-based system where we have 55 lines of IOAPIC, no PIC, and some GPIO lines. Overall I see that nr_irqs is 512 and shifting by 55 freezes 10% of the space for nothing. Luckily we have SPARSE_IRQS selected for any X86, so, it wouldn't waste memory.

Quoted 6 lines
Quoted 2 lines

I think we may do slightly better if we just limit the change to the certain cases.

For DT we can actually avoid that completely. See below.

For ACPI not unfortunately as the stupid GSI mapping is hard coded.

The below works better for my case, so, if you are going with that

Quoted 23 lines

Thanks,

tglx

8<-------------

arch/x86/kernel/apic/io_apic.c +7 -1
@@ -2438,7 +2438,13 @@ unsigned int arch_dynirq_lower_bound(uns
 	 * dmar_alloc_hwirq() may be called before setup_IO_APIC(), so use
 	 * gsi_top if ioapic_dynirq_base hasn't been initialized yet.
 	 */
-	return ioapic_initialized ? ioapic_dynirq_base : gsi_top;
+	if (!ioapic_initialized)
+		return gsi_top;
+	/*
+	 * For DT enabled machines ioapic_dynirq_base is irrelevant and not
+	 * updated. So simply return @from if ioapic_dynirq_base == 0.
+	 */
+	return ioapic_dynirq_base ? : from;
 }
 
 #ifdef CONFIG_X86_32
Signature · 2 lines
With Best Regards,
Andy Shevchenko
#7 lore raw

Hi Thomas,

Quoted 15 lines · Andy Shevchenko
Quoted 3 lines

For DT we can actually avoid that completely. See below.

For ACPI not unfortunately as the stupid GSI mapping is hard coded.

The below works better for my case, so, if you are going with that

8<-------------

arch/x86/kernel/apic/io_apic.c +0 -1
@@ -2438,7 +2438,13 @@ unsigned int arch_dynirq_lower_bound(uns
  	 * dmar_alloc_hwirq() may be called before setup_IO_APIC(), so use
  	 * gsi_top if ioapic_dynirq_base hasn't been initialized yet.
  	 */
-	return ioapic_initialized ? ioapic_dynirq_base : gsi_top;

arch/x86/kernel/apic/io_apic.c unsigned int arch_dynirq_lower_bound(uns · line 2450

Quoted 10 lines
arch/x86/kernel/apic/io_apic.c +7 -0
+	if (!ioapic_initialized)
+		return gsi_top;
+	/*
+	 * For DT enabled machines ioapic_dynirq_base is irrelevant and not
+	 * updated. So simply return @from if ioapic_dynirq_base == 0.
+	 */
+	return ioapic_dynirq_base ? : from;
  }
 
  #ifdef CONFIG_X86_32

I have also tested above and it works fine. In fact, my first patch to

resolve it during internal review was exactly on similar lines. So if

you are going to add above then i will stop following up on this

topic further. Thanks.

Regards,

Rahul

#8 lore raw

[tip: x86/urgent] x86/apic: Fix arch_dynirq_lower_bound() bug for DT enabled machines

In reply to Rahul Tanwar
The following commit has been merged into the x86/urgent branch of tip:

The following commit has been merged into the x86/urgent branch of tip:

Commit-ID:     3e5bedc2c258341702ddffbd7688c5e6eb01eafa
Gitweb:        https://git.kernel.org/tip/3e5bedc2c258341702ddffbd7688c5e6eb01eafa
Author:        Thomas Gleixner <[email protected]>
AuthorDate:    Wed, 21 Aug 2019 15:16:31 +02:00
Committer:     Ingo Molnar <[email protected]>
CommitterDate: Mon, 26 Aug 2019 12:11:23 +02:00

x86/apic: Fix arch_dynirq_lower_bound() bug for DT enabled machines

Rahul Tanwar reported the following bug on DT systems:

Quoted 11 lines

'ioapic_dynirq_base' contains the virtual IRQ base number. Presently, it is updated to the end of hardware IRQ numbers but this is done only when IOAPIC configuration type is IOAPIC_DOMAIN_LEGACY or IOAPIC_DOMAIN_STRICT. There is a third type IOAPIC_DOMAIN_DYNAMIC which applies when IOAPIC configuration comes from devicetree.

See dtb_add_ioapic() in arch/x86/kernel/devicetree.c

In case of IOAPIC_DOMAIN_DYNAMIC (DT/OF based system), 'ioapic_dynirq_base' remains to zero initialized value. This means that for OF based systems, virtual IRQ base will get set to zero.

Such systems will very likely not even boot.

For DT enabled machines ioapic_dynirq_base is irrelevant and not updated, so simply map the IRQ base 1:1 instead.

---
 arch/x86/kernel/apic/io_apic.c | 8 +++++++-
 1 file changed, 7 insertions(+), 1 deletion(-)
arch/x86/kernel/apic/io_apic.c +7 -1
index c7bb6c6..d6af97f 100644
@@ -2438,7 +2438,13 @@ unsigned int arch_dynirq_lower_bound(unsigned int from)
 	 * dmar_alloc_hwirq() may be called before setup_IO_APIC(), so use
 	 * gsi_top if ioapic_dynirq_base hasn't been initialized yet.
 	 */
-	return ioapic_initialized ? ioapic_dynirq_base : gsi_top;
+	if (!ioapic_initialized)
+		return gsi_top;
+	/*
+	 * For DT enabled machines ioapic_dynirq_base is irrelevant and not
+	 * updated. So simply return @from if ioapic_dynirq_base == 0.
+	 */
+	return ioapic_dynirq_base ? : from;
 }
 
 #ifdef CONFIG_X86_32