Skip to contentSkip to conversation

Fix dmar fault interrupt problems

PATCH In review Kenji Kaneshige 3 participants 11 messages Nov 30, 2010 – Dec 1, 2010 x86

lore
Share & tools
lore
#0 lore raw

cover Fix dmar fault interrupt problems

Hi,

Here are patches to fix the following problems regarding dmar fault interrupt.

- No dmar fault event is notified in x2apic cluster mode
- Changing IRQ affinity of dmar fault interrupt causes "No irq handler for vector (irq XX)" message and dmar fault interrupts are never notified after that.

Regards,
Kenji Kaneshige

#1 lore raw

PATCH 1/2 dmar: fix fault interrupt setup

Fix the problem dmar fault event is not notified in x2apic cluster mode.

In the current code, dmar fault event is configured before setting up the x86_cpu_to_logical_apicid percpu variable in x2apic cluster mode. Because of this, invalid apic ID is used for dmar fault interrupt and this cuases the problem.

To fix the problem, do dmar fault event configuration after local apic setup (after end_local_APIC_setup()).

---
 arch/x86/kernel/apic/apic.c     |    9 +++++++++
 arch/x86/kernel/apic/probe_64.c |    7 -------
 2 files changed, 9 insertions(+), 7 deletions(-)
arch/x86/kernel/apic/apic.c +9 -0
Index: linux-next-20101125/arch/x86/kernel/apic/apic.c
===================================================================
@@ -1745,6 +1745,15 @@ int __init APIC_init_uniprocessor(void)
 
 	end_local_APIC_setup();
 
+#ifdef CONFIG_INTR_REMAP
+	/*
+	 * Now that local APIC setup is completed, configure the fault
+	 * handling for interrupt remapping.
+	 */
+	if (intr_remapping_enabled)
+		enable_drhd_fault_handling();
+#endif
+
 #ifdef CONFIG_X86_IO_APIC
 	if (smp_found_config && !skip_ioapic_setup && nr_ioapics)
 		setup_IO_APIC();
arch/x86/kernel/apic/probe_64.c +0 -7
Index: linux-next-20101125/arch/x86/kernel/apic/probe_64.c
===================================================================
@@ -79,13 +79,6 @@ void __init default_setup_apic_routing(v
 		/* need to update phys_pkg_id */
 		apic->phys_pkg_id = apicid_phys_pkg_id;
 	}
-
-	/*
-	 * Now that apic routing model is selected, configure the
-	 * fault handling for intr remapping.
-	 */
-	if (intr_remapping_enabled)
-		enable_drhd_fault_handling();
 }
 
 /* Same for both flat and physical. */
#4 lore raw

PATCH 2/2 dmar: Fix dmar interrupt affinity handling

In reply to Kenji Kaneshige

Fix the problem that changing IRQ affinity of dmar fault interrupt causes "No irq handler for vector (irq XX)" message and dmar fault interrupts are never notified after that.

The dmar_msi_set_affinity() must configure upper address register of remapping hardware interrupt in x2apic mode.

---
 arch/x86/kernel/apic/io_apic.c |    3 +++
 1 file changed, 3 insertions(+)
arch/x86/kernel/apic/io_apic.c +3 -0
Index: linux-next-20101125/arch/x86/kernel/apic/io_apic.c
===================================================================
@@ -3413,6 +3413,9 @@ dmar_msi_set_affinity(struct irq_data *d
 	msg.data |= MSI_DATA_VECTOR(cfg->vector);
 	msg.address_lo &= ~MSI_ADDR_DEST_ID_MASK;
 	msg.address_lo |= MSI_ADDR_DEST_ID(dest);
+	msg.address_hi = MSI_ADDR_BASE_HI;
+	if (x2apic_enabled())
+		msg.address_hi |= MSI_ADDR_EXT_DEST_ID(dest);
 
 	dmar_msi_write(irq, &msg);
 
#5 lore raw
In reply to Kenji Kaneshige
Quoted 8 lines · Kenji Kaneshige

Hi,

Here are patches to fix the following problems regarding dmar fault interrupt.

- No dmar fault event is notified in x2apic cluster mode
- Changing IRQ affinity of dmar fault interrupt causes "No irq handler for vector (irq XX)" message and dmar fault interrupts are never notified after that.

Kenji, In addition to these two, there are couple of more patches that are needed to fix the kexec/kdump issue that we are debugging on RH bugzilla.

I will post all these 4 patches shortly. I already noticed and queued the second patch (dmar irq migration) you mentioned.

thanks,
suresh

#6 lore raw
Quoted 10 lines · * Suresh Siddha ([email protected]) wrote:
Quoted 8 lines · Kenji Kaneshige

Hi,

Here are patches to fix the following problems regarding dmar fault interrupt.

- No dmar fault event is notified in x2apic cluster mode
- Changing IRQ affinity of dmar fault interrupt causes "No irq handler for vector (irq XX)" message and dmar fault interrupts are never notified after that.

Quoted 6 lines

Kenji, In addition to these two, there are couple of more patches that are needed to fix the kexec/kdump issue that we are debugging on RH bugzilla.

I will post all these 4 patches shortly. I already noticed and queued the second patch (dmar irq migration) you mentioned.

And one of them (the additional call to flushing existing dmar faults) is not needed. So the quirk plus ensuring ir faults show up as msi interrupts should be sufficient.

thanks,
-chris

#7 lore raw
Quoted 15 lines · Chris Wright
Quoted 14 lines · * Suresh Siddha ([email protected]) wrote:
Quoted 8 lines · Kenji Kaneshige

Hi,

Here are patches to fix the following problems regarding dmar fault interrupt.

- No dmar fault event is notified in x2apic cluster mode
- Changing IRQ affinity of dmar fault interrupt causes "No irq handler for vector (irq XX)" message and dmar fault interrupts are never notified after that.

Kenji, In addition to these two, there are couple of more patches that are needed to fix the kexec/kdump issue that we are debugging on RH bugzilla.

Quoted 5 lines
Quoted 2 lines

I will post all these 4 patches shortly. I already noticed and queued the second patch (dmar irq migration) you mentioned.

And one of them (the additional call to flushing existing dmar faults) is not needed.

yes, that patch is not needed for the current kexec/kdump issue. But nevertheless that is an issue that needs to be addressed aswell.

thanks,
suresh

#8 lore raw
Quoted 16 lines · * Suresh Siddha ([email protected]) wrote:
Quoted 15 lines · Chris Wright
Quoted 14 lines · * Suresh Siddha ([email protected]) wrote:
Quoted 8 lines · Kenji Kaneshige

Hi,

Here are patches to fix the following problems regarding dmar fault interrupt.

- No dmar fault event is notified in x2apic cluster mode
- Changing IRQ affinity of dmar fault interrupt causes "No irq handler for vector (irq XX)" message and dmar fault interrupts are never notified after that.

Kenji, In addition to these two, there are couple of more patches that are needed to fix the kexec/kdump issue that we are debugging on RH bugzilla.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

I will post all these 4 patches shortly. I already noticed and queued the second patch (dmar irq migration) you mentioned.

And one of them (the additional call to flushing existing dmar faults) is not needed.

yes, that patch is not needed for the current kexec/kdump issue. But nevertheless that is an issue that needs to be addressed aswell.

AFAICT, it just becomes a duplicate call in a narrow window.

native_smp_prepare_cpus() (or APIC_init_uniprocessor()):
enable_IR_x2apic()
enable_IR()
enable_intr_remapping() <-- clears faults
default_setup_apic_routing()
enable_drhd_fault_handling() <-- added call to clear faults

That's what I meant by not needed.

thanks,
-chris

#9 lore raw
Quoted 10 lines · Chris Wright

AFAICT, it just becomes a duplicate call in a narrow window.

native_smp_prepare_cpus() (or APIC_init_uniprocessor()):
enable_IR_x2apic()
enable_IR()
enable_intr_remapping() <-- clears faults
default_setup_apic_routing()
enable_drhd_fault_handling() <-- added call to clear faults

That's what I meant by not needed.

I agree that we will call clear faults twice.

With out the second clear fault, any fault in that narrow window of enabling intr-remapping and enabling fault handling will go unnoticed and block further faults getting reported.

I am not sure if the first fault is necessary. I can check and remove it if not needed.

Ideally we want to do something like:

Setup fault handling
Clear any faults
Enable intr-remapping

But fault handling interrupt configuration depends on the apic mode that gets selected based on whether intr-remapping was enabled successfully or not. And hence we were enabling fault handling after enabling the intr-remapping mode.

I can see if I can cleanup any of these flows for 2.6.38. For 2.6.37 and older kernels I would like to keep the patch simple.

thanks,
suresh

#2 lore raw

Re: [PATCH 1/2] dmar: fix fault interrupt setup

In reply to Kenji Kaneshige
Quoted 24 lines · Kenji Kaneshige

Fix the problem dmar fault event is not notified in x2apic cluster mode.

In the current code, dmar fault event is configured before setting up the x86_cpu_to_logical_apicid percpu variable in x2apic cluster mode. Because of this, invalid apic ID is used for dmar fault interrupt and this cuases the problem.

To fix the problem, do dmar fault event configuration after local apic setup (after end_local_APIC_setup()).

---
 arch/x86/kernel/apic/apic.c     |    9 +++++++++
 arch/x86/kernel/apic/probe_64.c |    7 -------
 2 files changed, 9 insertions(+), 7 deletions(-)

Index: linux-next-20101125/arch/x86/kernel/apic/apic.c
===================================================================

arch/x86/kernel/apic/apic.c +0 -0
@@ -1745,6 +1745,15 @@ int __init APIC_init_uniprocessor(void)
 

arch/x86/kernel/apic/apic.c int __init APIC_init_uniprocessor(void) · line 1755

Quoted 10 lines
arch/x86/kernel/apic/apic.c +8 -0
 	end_local_APIC_setup();
 
+#ifdef CONFIG_INTR_REMAP
+	/*
+	 * Now that local APIC setup is completed, configure the fault
+	 * handling for interrupt remapping.
+	 */
+	if (intr_remapping_enabled)
+		enable_drhd_fault_handling();
+#endif

BTW, this is wrong. APIC_init_uniprocessor() gets called only for UP kernel. I will move it to the end_local_APIC_setup()

Quoted 24 lines
arch/x86/kernel/apic/apic.c +1 -0
+
 #ifdef CONFIG_X86_IO_APIC
 	if (smp_found_config && !skip_ioapic_setup && nr_ioapics)
 		setup_IO_APIC();

Index: linux-next-20101125/arch/x86/kernel/apic/probe_64.c
===================================================================

arch/x86/kernel/apic/probe_64.c +0 -7
@@ -79,13 +79,6 @@ void __init default_setup_apic_routing(v
 		/* need to update phys_pkg_id */
 		apic->phys_pkg_id = apicid_phys_pkg_id;
 	}
-
-	/*
-	 * Now that apic routing model is selected, configure the
-	 * fault handling for intr remapping.
-	 */
-	if (intr_remapping_enabled)
-		enable_drhd_fault_handling();
 }
 
 /* Same for both flat and physical. */
#10 lore raw
In reply to Suresh Siddha
Quoted 18 lines · * Suresh Siddha ([email protected]) wrote:
Quoted 10 lines · Chris Wright

AFAICT, it just becomes a duplicate call in a narrow window.

native_smp_prepare_cpus() (or APIC_init_uniprocessor()):
enable_IR_x2apic()
enable_IR()
enable_intr_remapping() <-- clears faults
default_setup_apic_routing()
enable_drhd_fault_handling() <-- added call to clear faults

That's what I meant by not needed.

I agree that we will call clear faults twice.

With out the second clear fault, any fault in that narrow window of enabling intr-remapping and enabling fault handling will go unnoticed and block further faults getting reported.

Quoted 8 lines

I am not sure if the first fault is necessary. I can check and remove it if not needed.

Ideally we want to do something like:

Setup fault handling
Clear any faults
Enable intr-remapping

Agree, that's ideal.

Quoted 7 lines

But fault handling interrupt configuration depends on the apic mode that gets selected based on whether intr-remapping was enabled successfully or not. And hence we were enabling fault handling after enabling the intr-remapping mode.

I can see if I can cleanup any of these flows for 2.6.38. For 2.6.37 and older kernels I would like to keep the patch simple.

Yup, makes sense.

thanks,
-chris

#3 lore raw

Re: [PATCH 1/2] dmar: fix fault interrupt setup

In reply to Suresh Siddha
Quoted 27 lines · Suresh Siddha
Quoted 26 lines · Kenji Kaneshige

Fix the problem dmar fault event is not notified in x2apic cluster mode.

In the current code, dmar fault event is configured before setting up the x86_cpu_to_logical_apicid percpu variable in x2apic cluster mode. Because of this, invalid apic ID is used for dmar fault interrupt and this cuases the problem.

To fix the problem, do dmar fault event configuration after local apic setup (after end_local_APIC_setup()).

---
  arch/x86/kernel/apic/apic.c     |    9 +++++++++
  arch/x86/kernel/apic/probe_64.c |    7 -------
  2 files changed, 9 insertions(+), 7 deletions(-)

Index: linux-next-20101125/arch/x86/kernel/apic/apic.c
===================================================================

arch/x86/kernel/apic/apic.c +0 -0
@@ -1745,6 +1745,15 @@ int __init APIC_init_uniprocessor(void)
 
  	end_local_APIC_setup();
 
Quoted 12 lines

arch/x86/kernel/apic/apic.c int __init APIC_init_uniprocessor(void) · line 1755

Quoted 8 lines
arch/x86/kernel/apic/apic.c +8 -0
+#ifdef CONFIG_INTR_REMAP
+	/*
+	 * Now that local APIC setup is completed, configure the fault
+	 * handling for interrupt remapping.
+	 */
+	if (intr_remapping_enabled)
+		enable_drhd_fault_handling();
+#endif

BTW, this is wrong. APIC_init_uniprocessor() gets called only for UP kernel. I will move it to the end_local_APIC_setup()

Oh, it's totally wrong. I'm sorry about that.

Maybe even with my wrong patch, dmar interrupt is configured in the intel_iommu_init()->init_dmars()->dmar_set_interrupt() code path. I think this is why I could not noticed my badness in my test.

By the way, why do we need to try to configure dmar interrupt in the intel_iommu_init() code path again?

Regards,
Kenji Kaneshige