Skip to contentSkip to conversation

fix bugs of x2apic/intr-remap

PATCH In review Weidong Han 4 participants 28 messages Apr 17, 2009 – Apr 19, 2009

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

cover fix bugs of x2apic/intr-remap

interupt remapping was decoupled from x2apic already, but there are still some issues, such as doesn't ack remapped interrupts correctly, and doesn't remap ioapic interrupt when x2apic is not enabled.

This patchset fixes the ack for remapped interrupts, and alwasys enables interrupt remapping before ioapic setup to guarantee ioapic interrupts are remapped, adjusts the dependency of x2apic and interrupt remapping in lapic_resume. In addition, add option "nointremap" to disable interrupt remapping.

The patchset can be applied on linux head. Thanks.

Weidong Han (5):
docs: add nox2apic back to kernel-parameters.txt x86,intr-remap: fix ack for interrupt remapping x86, intr-remap: enable interrupt remapping early x86, intr-remap: add option to disable interrupt remapping
x86: fix x2apic/intr-remap resume

 Documentation/kernel-parameters.txt |    5 ++
 arch/x86/include/asm/apic.h         |   15 +-----
 arch/x86/kernel/apic/apic.c         |   91 +++++++++++++++++------------------
 arch/x86/kernel/apic/io_apic.c      |   32 ++----------
 drivers/pci/intel-iommu.c           |    9 ----
 drivers/pci/intr_remapping.c        |   39 ++++++++++-----
 include/linux/dmar.h                |    1 +
 7 files changed, 84 insertions(+), 108 deletions(-)
#1 lore raw

PATCH 1/5 docs: add nox2apic back to kernel-parameters.txt

"nox2apic" was removed from kernel-parameters.txt when Randy moved entries to be in alpha order (commit 0cb55ad2). But this early parameter is still there, add it back to kernel-parameters.txt.

---
 Documentation/kernel-parameters.txt |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +2 -0
index 6172e43..33989d2 100644
@@ -1588,6 +1588,8 @@ and is between 256 and 4096 characters. It is defined in the file
 
 	nowb		[ARM]
 
+	nox2apic	[X86-64,APIC] Do not enable x2APIC mode.
+
 	nptcg=		[IA64] Override max number of concurrent global TLB
 			purges which is reported from either PAL_VM_SUMMARY or
 			SAL PALO.
Signature · 1 line
1.6.0.4
#2 lore raw

Re: [PATCH 1/5] docs: add nox2apic back to kernel-parameters.txt

Quoted 3 lines · Weidong Han

"nox2apic" was removed from kernel-parameters.txt when Randy moved entries to be in alpha order (commit 0cb55ad2). But this early parameter is still there, add it back to kernel-parameters.txt.

Randy, this must have been some conflict resolution / patch confict merge related loss, right?

Mind double-checking 0cb55ad2 again to make sure no other boot parameters were lost during that reordering?

Thanks,

Ingo

#3 lore raw

[tip:x86/apic] docs, x86: add nox2apic back to kernel-parameters.txt

In reply to Weidong Han
Commit-ID:  66a2f36639a85d06e28819e2896efbea38e97ada
Gitweb:     http://git.kernel.org/tip/66a2f36639a85d06e28819e2896efbea38e97ada
Author:     Weidong Han <[email protected]>
AuthorDate: Fri, 17 Apr 2009 16:42:12 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 17 Apr 2009 16:23:44 +0200

docs, x86: add nox2apic back to kernel-parameters.txt

"nox2apic" was removed from kernel-parameters.txt by mistake, when entries were sorted in alpha order (commit 0cb55ad2). But this early parameter is still there, add it back to kernel-parameters.txt.

[ Impact: add boot parameter description ]

LKML-Reference: <[email protected]>

---
 Documentation/kernel-parameters.txt |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +2 -0
index 6172e43..33989d2 100644
@@ -1588,6 +1588,8 @@ and is between 256 and 4096 characters. It is defined in the file
 
 	nowb		[ARM]
 
+	nox2apic	[X86-64,APIC] Do not enable x2APIC mode.
+
 	nptcg=		[IA64] Override max number of concurrent global TLB
 			purges which is reported from either PAL_VM_SUMMARY or
 			SAL PALO.
#4 lore raw

[tip:x86/urgent] docs, x86: add nox2apic back to kernel-parameters.txt

In reply to Weidong Han
Commit-ID:  8b5b94e4e9813cdd77103827f48d58c806ab45c6
Gitweb:     http://git.kernel.org/tip/8b5b94e4e9813cdd77103827f48d58c806ab45c6
Author:     Weidong Han <[email protected]>
AuthorDate: Fri, 17 Apr 2009 16:42:12 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sat, 18 Apr 2009 09:29:16 +0200

docs, x86: add nox2apic back to kernel-parameters.txt

"nox2apic" was removed from kernel-parameters.txt by mistake, when entries were sorted in alpha order (commit 0cb55ad2). But this early parameter is still there, add it back to kernel-parameters.txt.

[ Impact: add boot parameter description ]

LKML-Reference: <[email protected]>

---
 Documentation/kernel-parameters.txt |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +2 -0
index a19f021..9e4fe72 100644
@@ -1588,6 +1588,8 @@ and is between 256 and 4096 characters. It is defined in the file
 
 	nowb		[ARM]
 
+	nox2apic	[X86-64,APIC] Do not enable x2APIC mode.
+
 	nptcg=		[IA64] Override max number of concurrent global TLB
 			purges which is reported from either PAL_VM_SUMMARY or
 			SAL PALO.
#5 lore raw

[tip:x86/apic] docs, x86: add nox2apic back to kernel-parameters.txt

In reply to Weidong Han
Commit-ID:  2b2fd87a6ef56ba7647a578e81bb8c8efda166b8
Gitweb:     http://git.kernel.org/tip/2b2fd87a6ef56ba7647a578e81bb8c8efda166b8
Author:     Weidong Han <[email protected]>
AuthorDate: Fri, 17 Apr 2009 16:42:12 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 19 Apr 2009 10:21:13 +0200

docs, x86: add nox2apic back to kernel-parameters.txt

"nox2apic" was removed from kernel-parameters.txt by mistake, when entries were sorted in alpha order (commit 0cb55ad2). But this early parameter is still there, add it back to kernel-parameters.txt.

[ Impact: add boot parameter description ]

LKML-Reference: <[email protected]>

---
 Documentation/kernel-parameters.txt |    2 ++
 1 files changed, 2 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +2 -0
index 6172e43..33989d2 100644
@@ -1588,6 +1588,8 @@ and is between 256 and 4096 characters. It is defined in the file
 
 	nowb		[ARM]
 
+	nox2apic	[X86-64,APIC] Do not enable x2APIC mode.
+
 	nptcg=		[IA64] Override max number of concurrent global TLB
 			purges which is reported from either PAL_VM_SUMMARY or
 			SAL PALO.
#6 lore raw

PATCH 2/5 x86,intr-remap: fix ack for interrupt remapping

In reply to Weidong Han

Shouldn't call ack_apic_edge in ir_ack_apic_edge, because ack_apic_edge does more than just ack, it also does irq migration in the non-interrupt-remapping case. But there is no such need for interrupt-remapping case. as irq migration is done in the process context.

Similarly, ir_ack_apic_level shouldn't call ack_apic_level, and instead do the local cpu's EOI + directed EOI to the io-apic.

ack_x2APIC_irq is not neccessary, becasue ack_APIC_irq will use MSR write for x2apic, and uncached write for non-x2apic.

---
 arch/x86/include/asm/apic.h    |    8 --------
 arch/x86/kernel/apic/io_apic.c |   32 +++++---------------------------
 2 files changed, 5 insertions(+), 35 deletions(-)
arch/x86/include/asm/apic.h +0 -8
index 42f2f83..c37af81 100644
@@ -204,14 +204,6 @@ static inline int x2apic_enabled(void)
 
 extern int get_physical_broadcast(void);
 
-#ifdef CONFIG_X86_X2APIC
-static inline void ack_x2APIC_irq(void)
-{
-	/* Docs say use 0 for future compatibility */
-	native_apic_msr_write(APIC_EOI, 0);
-}
-#endif
-
 extern int lapic_get_maxlvt(void);
 extern void clear_local_APIC(void);
 extern void connect_bsp_APIC(void);
arch/x86/kernel/apic/io_apic.c +5 -27
index a2789e4..149e1b1 100644
@@ -2557,20 +2557,6 @@ eoi_ioapic_irq(struct irq_desc *desc)
 	spin_unlock_irqrestore(&ioapic_lock, flags);
 }
 
-#ifdef CONFIG_X86_X2APIC
-static void ack_x2apic_level(unsigned int irq)
-{
-	struct irq_desc *desc = irq_to_desc(irq);
-	ack_x2APIC_irq();
-	eoi_ioapic_irq(desc);
-}
-
-static void ack_x2apic_edge(unsigned int irq)
-{
-	ack_x2APIC_irq();
-}
-#endif
-
 static void ack_apic_edge(unsigned int irq)
 {
 	struct irq_desc *desc = irq_to_desc(irq);
@@ -2634,9 +2620,6 @@ static void ack_apic_level(unsigned int irq)
 	 */
 	ack_APIC_irq();
 
-	if (irq_remapped(irq))
-		eoi_ioapic_irq(desc);
-
 	/* Now we can move and renable the irq */
 	if (unlikely(do_unmask_irq)) {
 		/* Only migrate the irq if the ack has been received.
@@ -2685,20 +2668,15 @@ static void ack_apic_level(unsigned int irq)
 #ifdef CONFIG_INTR_REMAP
 static void ir_ack_apic_edge(unsigned int irq)
 {
-#ifdef CONFIG_X86_X2APIC
-       if (x2apic_enabled())
-               return ack_x2apic_edge(irq);
-#endif
-       return ack_apic_edge(irq);
+	ack_APIC_irq();
 }
 
 static void ir_ack_apic_level(unsigned int irq)
 {
-#ifdef CONFIG_X86_X2APIC
-       if (x2apic_enabled())
-               return ack_x2apic_level(irq);
-#endif
-       return ack_apic_level(irq);
+	struct irq_desc *desc = irq_to_desc(irq);
+
+	ack_APIC_irq();
+	eoi_ioapic_irq(desc);
 }
 #endif /* CONFIG_INTR_REMAP */
 
Signature · 1 line
1.6.0.4
#7 lore raw

[tip:x86/apic] x86, intr-remap: fix ack for interrupt remapping

Commit-ID:  631dd7572e66ee524bb7dfe7dbc883f3484c7eee
Gitweb:     http://git.kernel.org/tip/631dd7572e66ee524bb7dfe7dbc883f3484c7eee
Author:     Weidong Han <[email protected]>
AuthorDate: Fri, 17 Apr 2009 16:42:13 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 17 Apr 2009 16:23:44 +0200

x86, intr-remap: fix ack for interrupt remapping

Shouldn't call ack_apic_edge() in ir_ack_apic_edge(), because ack_apic_edge() does more than just ack: it also does irq migration in the non-interrupt-remapping case. But there is no such need for interrupt-remapping case, as irq migration is done in the process context.

Similarly, ir_ack_apic_level() shouldn't call ack_apic_level, and instead should do the local cpu's EOI + directed EOI to the io-apic.

ack_x2APIC_irq() is not neccessary, because ack_APIC_irq() will use MSR write for x2apic, and uncached write for non-x2apic.

[ Impact: simplify/standardize intr-remap IRQ acking, fix on !x2apic ]

LKML-Reference: <[email protected]>

---
 arch/x86/include/asm/apic.h    |    8 --------
 arch/x86/kernel/apic/io_apic.c |   32 +++++---------------------------
 2 files changed, 5 insertions(+), 35 deletions(-)
arch/x86/include/asm/apic.h +0 -8
index 2bd5a46..d4cb7e5 100644
@@ -204,14 +204,6 @@ static inline int x2apic_enabled(void)
 
 extern int get_physical_broadcast(void);
 
-#ifdef CONFIG_X86_X2APIC
-static inline void ack_x2APIC_irq(void)
-{
-	/* Docs say use 0 for future compatibility */
-	native_apic_msr_write(APIC_EOI, 0);
-}
-#endif
-
 extern void apic_disable(void);
 extern int lapic_get_maxlvt(void);
 extern void clear_local_APIC(void);
arch/x86/kernel/apic/io_apic.c +5 -27
index 8499000..ea22a86 100644
@@ -2552,20 +2552,6 @@ eoi_ioapic_irq(struct irq_desc *desc)
 	spin_unlock_irqrestore(&ioapic_lock, flags);
 }
 
-#ifdef CONFIG_X86_X2APIC
-static void ack_x2apic_level(unsigned int irq)
-{
-	struct irq_desc *desc = irq_to_desc(irq);
-	ack_x2APIC_irq();
-	eoi_ioapic_irq(desc);
-}
-
-static void ack_x2apic_edge(unsigned int irq)
-{
-	ack_x2APIC_irq();
-}
-#endif
-
 static void ack_apic_edge(unsigned int irq)
 {
 	struct irq_desc *desc = irq_to_desc(irq);
@@ -2629,9 +2615,6 @@ static void ack_apic_level(unsigned int irq)
 	 */
 	ack_APIC_irq();
 
-	if (irq_remapped(irq))
-		eoi_ioapic_irq(desc);
-
 	/* Now we can move and renable the irq */
 	if (unlikely(do_unmask_irq)) {
 		/* Only migrate the irq if the ack has been received.
@@ -2680,20 +2663,15 @@ static void ack_apic_level(unsigned int irq)
 #ifdef CONFIG_INTR_REMAP
 static void ir_ack_apic_edge(unsigned int irq)
 {
-#ifdef CONFIG_X86_X2APIC
-       if (x2apic_enabled())
-               return ack_x2apic_edge(irq);
-#endif
-       return ack_apic_edge(irq);
+	ack_APIC_irq();
 }
 
 static void ir_ack_apic_level(unsigned int irq)
 {
-#ifdef CONFIG_X86_X2APIC
-       if (x2apic_enabled())
-               return ack_x2apic_level(irq);
-#endif
-       return ack_apic_level(irq);
+	struct irq_desc *desc = irq_to_desc(irq);
+
+	ack_APIC_irq();
+	eoi_ioapic_irq(desc);
 }
 #endif /* CONFIG_INTR_REMAP */
 
#8 lore raw

[tip:x86/apic] x86, intr-remap: fix ack for interrupt remapping

In reply to Weidong Han
Commit-ID:  5d0ae2db6deac4f15dac4f42f23bc56448fc8d4d
Gitweb:     http://git.kernel.org/tip/5d0ae2db6deac4f15dac4f42f23bc56448fc8d4d
Author:     Weidong Han <[email protected]>
AuthorDate: Fri, 17 Apr 2009 16:42:13 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 19 Apr 2009 10:21:33 +0200

x86, intr-remap: fix ack for interrupt remapping

Shouldn't call ack_apic_edge() in ir_ack_apic_edge(), because ack_apic_edge() does more than just ack: it also does irq migration in the non-interrupt-remapping case. But there is no such need for interrupt-remapping case, as irq migration is done in the process context.

Similarly, ir_ack_apic_level() shouldn't call ack_apic_level, and instead should do the local cpu's EOI + directed EOI to the io-apic.

ack_x2APIC_irq() is not neccessary, because ack_APIC_irq() will use MSR write for x2apic, and uncached write for non-x2apic.

[ Impact: simplify/standardize intr-remap IRQ acking, fix on !x2apic ]

LKML-Reference: <[email protected]>

---
 arch/x86/include/asm/apic.h    |    8 --------
 arch/x86/kernel/apic/io_apic.c |   32 +++++---------------------------
 2 files changed, 5 insertions(+), 35 deletions(-)
arch/x86/include/asm/apic.h +0 -8
index 2bd5a46..d4cb7e5 100644
@@ -204,14 +204,6 @@ static inline int x2apic_enabled(void)
 
 extern int get_physical_broadcast(void);
 
-#ifdef CONFIG_X86_X2APIC
-static inline void ack_x2APIC_irq(void)
-{
-	/* Docs say use 0 for future compatibility */
-	native_apic_msr_write(APIC_EOI, 0);
-}
-#endif
-
 extern void apic_disable(void);
 extern int lapic_get_maxlvt(void);
 extern void clear_local_APIC(void);
arch/x86/kernel/apic/io_apic.c +5 -27
index 8499000..ea22a86 100644
@@ -2552,20 +2552,6 @@ eoi_ioapic_irq(struct irq_desc *desc)
 	spin_unlock_irqrestore(&ioapic_lock, flags);
 }
 
-#ifdef CONFIG_X86_X2APIC
-static void ack_x2apic_level(unsigned int irq)
-{
-	struct irq_desc *desc = irq_to_desc(irq);
-	ack_x2APIC_irq();
-	eoi_ioapic_irq(desc);
-}
-
-static void ack_x2apic_edge(unsigned int irq)
-{
-	ack_x2APIC_irq();
-}
-#endif
-
 static void ack_apic_edge(unsigned int irq)
 {
 	struct irq_desc *desc = irq_to_desc(irq);
@@ -2629,9 +2615,6 @@ static void ack_apic_level(unsigned int irq)
 	 */
 	ack_APIC_irq();
 
-	if (irq_remapped(irq))
-		eoi_ioapic_irq(desc);
-
 	/* Now we can move and renable the irq */
 	if (unlikely(do_unmask_irq)) {
 		/* Only migrate the irq if the ack has been received.
@@ -2680,20 +2663,15 @@ static void ack_apic_level(unsigned int irq)
 #ifdef CONFIG_INTR_REMAP
 static void ir_ack_apic_edge(unsigned int irq)
 {
-#ifdef CONFIG_X86_X2APIC
-       if (x2apic_enabled())
-               return ack_x2apic_edge(irq);
-#endif
-       return ack_apic_edge(irq);
+	ack_APIC_irq();
 }
 
 static void ir_ack_apic_level(unsigned int irq)
 {
-#ifdef CONFIG_X86_X2APIC
-       if (x2apic_enabled())
-               return ack_x2apic_level(irq);
-#endif
-       return ack_apic_level(irq);
+	struct irq_desc *desc = irq_to_desc(irq);
+
+	ack_APIC_irq();
+	eoi_ioapic_irq(desc);
 }
 #endif /* CONFIG_INTR_REMAP */
 
#9 lore raw

PATCH 3/5 x86, intr-remap: enable interrupt remapping early

In reply to Weidong Han

Currently, when x2apic is not enabled, interrupt remapping will be enabled in init_dmars, where is too late to remap ioapic interrupts, that is, ioapic interrupts are really in compatibility mode, not remappable mode.

This patch always enables interrupt remapping before ioapic setup, it guarantees all interrupts will be remapped when interrupt remapping is enabled. Thus it needn't to set compatibility interrupt bit.

---
 arch/x86/include/asm/apic.h  |    7 ++--
 arch/x86/kernel/apic/apic.c  |   76 ++++++++++++++++++++----------------------
 drivers/pci/intel-iommu.c    |    9 -----
 drivers/pci/intr_remapping.c |   28 ++++++++--------
 include/linux/dmar.h         |    1 +
 5 files changed, 54 insertions(+), 67 deletions(-)
arch/x86/include/asm/apic.h +3 -4
index c37af81..4bff4cb 100644
@@ -169,7 +169,6 @@ static inline u64 native_x2apic_icr_read(void)
 extern int x2apic, x2apic_phys;
 extern void check_x2apic(void);
 extern void enable_x2apic(void);
-extern void enable_IR_x2apic(void);
 extern void x2apic_icr_write(u32 low, u32 id);
 static inline int x2apic_enabled(void)
 {
@@ -190,18 +189,18 @@ static inline void check_x2apic(void)
 static inline void enable_x2apic(void)
 {
 }
-static inline void enable_IR_x2apic(void)
-{
-}
 static inline int x2apic_enabled(void)
 {
 	return 0;
 }
 
 #define	x2apic	0
+#define	x2apic_preenabled 0
 
 #endif
 
+extern void enable_IR_x2apic(void);
+
 extern int get_physical_broadcast(void);
 
 extern int lapic_get_maxlvt(void);
arch/x86/kernel/apic/apic.c +36 -40

130 lines not on this page · raw message

drivers/pci/intel-iommu.c +0 -9
index 001b328..9ce8f07 100644
@@ -1968,15 +1968,6 @@ static int __init init_dmars(void)
 		}
 	}
 
-#ifdef CONFIG_INTR_REMAP
-	if (!intr_remapping_enabled) {
-		ret = enable_intr_remapping(0);
-		if (ret)
-			printk(KERN_ERR
-			       "IOMMU: enable interrupt remapping failed\n");
-	}
-#endif
-
 	/*
 	 * For each rmrr
 	 *   for each dev attached to rmrr
drivers/pci/intr_remapping.c +14 -14
index f5e0ea7..5c21426 100644
@@ -423,20 +423,6 @@ static void iommu_set_intr_remapping(struct intel_iommu *iommu, int mode)
 		      readl, (sts & DMA_GSTS_IRTPS), sts);
 	spin_unlock_irqrestore(&iommu->register_lock, flags);
 
-	if (mode == 0) {
-		spin_lock_irqsave(&iommu->register_lock, flags);
-
-		/* enable comaptiblity format interrupt pass through */
-		cmd = iommu->gcmd | DMA_GCMD_CFI;
-		iommu->gcmd |= DMA_GCMD_CFI;
-		writel(cmd, iommu->reg + DMAR_GCMD_REG);
-
-		IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG,
-			      readl, (sts & DMA_GSTS_CFIS), sts);
-
-		spin_unlock_irqrestore(&iommu->register_lock, flags);
-	}
-
 	/*
 	 * global invalidation of interrupt entry cache before enabling
 	 * interrupt-remapping.
@@ -516,6 +502,20 @@ end:
 	spin_unlock_irqrestore(&iommu->register_lock, flags);
 }
 
+int __init intr_remapping_supported(void)
+{
+	struct dmar_drhd_unit *drhd;
+
+	for_each_drhd_unit(drhd) {
+		struct intel_iommu *iommu = drhd->iommu;
+
+		if (!ecap_ir_support(iommu->ecap))
+			return 0;
+	}
+
+	return 1;
+}
+
 int __init enable_intr_remapping(int eim)
 {
 	struct dmar_drhd_unit *drhd;
include/linux/dmar.h +1 -0
index e397dc3..06f592a 100644
@@ -108,6 +108,7 @@ struct irte {
 };
 #ifdef CONFIG_INTR_REMAP
 extern int intr_remapping_enabled;
+extern int intr_remapping_supported(void);
 extern int enable_intr_remapping(int);
 extern void disable_intr_remapping(void);
 extern int reenable_intr_remapping(int);
Signature · 1 line
1.6.0.4
#10 lore raw

Re: [PATCH 3/5] x86, intr-remap: enable interrupt remapping early

arch/x86/kernel/apic/apic.c static int x2apic_preenabled; · line 122

Quoted 8 lines · Weidong Han
arch/x86/kernel/apic/apic.c +2 -0
@@ -118,6 +118,8 @@ static int x2apic_preenabled;
 static int disable_x2apic;
 static __init int setup_nox2apic(char *str)
 {
+	if (x2apic_enabled())
+		panic("Bios already enabled x2apic, can't enforce nox2apic");

Could you please turn that into something like:

		printk(KERN_WARNING "Bios already enabled x2apic, can't enforce nox2apic");
		return 1;

panic-ing the box just because we cannot meet a boot option is not good.

arch/x86/kernel/apic/apic.c void __init enable_IR_x2apic(void) · line 1357

Quoted 6 lines
arch/x86/kernel/apic/apic.c +6 -0
+#ifdef CONFIG_X86_X2APIC
+	if (cpu_has_x2apic)
+		ret = enable_intr_remapping(EIM_32BIT_APIC_ID);
+	else
+#endif
+		ret = enable_intr_remapping(EIM_8BIT_APIC_ID);

That construct looks rather ugly.

Why not clear x2apic from the CPU flags if CONFIG_X86_X2APIC is disabled. (and print a one-liner during bootup that we did so)

Then this could be written as:

	if (cpu_has_x2apic)
		ret = enable_intr_remapping(EIM_32BIT_APIC_ID);
	else
		ret = enable_intr_remapping(EIM_8BIT_APIC_ID);

which looks far more nice.

arch/x86/kernel/apic/apic.c void __init enable_IR_x2apic(void) · line 1370

Quoted 7 lines
arch/x86/kernel/apic/apic.c +4 -0
+#ifdef CONFIG_X86_X2APIC
+	if (cpu_has_x2apic && !x2apic) {
 		x2apic = 1;
 		enable_x2apic();
+		pr_info("Enabled x2apic\n");
 	}
+#endif

ditto - this #ifdef could go away with the cpuflags trick.

arch/x86/kernel/apic/apic.c end_restore: · line 1393

Quoted 3 lines
arch/x86/kernel/apic/apic.c +3 -0
+ir_failed:
+	if (x2apic_preenabled)
+		panic("x2apic enabled by bios. But IR enabling failed");

What is the likelyhood that we can continue in compat mode? If there's some chance, we should rather print a KERN_WARNING and should try to continue. If IRQs are not coming we'll hang shortly afterwards anyway.

arch/x86/kernel/apic/apic.c end_restore: · line 1403

Quoted 2 lines
arch/x86/kernel/apic/apic.c +0 -1
 		panic("x2apic enabled prior OS handover,"
-		      " enable CONFIG_INTR_REMAP");

ditto.

arch/x86/kernel/apic/apic.c static int __init init_dmars(void) · line 1978

Quoted 13 lines

+++ b/drivers/pci/intel-iommu.c

arch/x86/kernel/apic/apic.c +0 -8
@@ -1968,15 +1968,6 @@ static int __init init_dmars(void)
 		}
 	}
 
-#ifdef CONFIG_INTR_REMAP
-	if (!intr_remapping_enabled) {
-		ret = enable_intr_remapping(0);
-		if (ret)
-			printk(KERN_ERR
-			       "IOMMU: enable interrupt remapping failed\n");
-	}
-#endif

David, is this fine with you? Doing ir-remap setup in the ioapic code and early on is the obviously right thing to do.

drivers/pci/intr_remapping.c static void iommu_set_intr_remapping(struct intel_iommu *iommu, int mode) · line 429

Quoted 10 lines
drivers/pci/intr_remapping.c +0 -4
@@ -423,20 +423,6 @@ static void iommu_set_intr_remapping(struct intel_iommu *iommu, int mode)
 		      readl, (sts & DMA_GSTS_IRTPS), sts);
 	spin_unlock_irqrestore(&iommu->register_lock, flags);
 
-	if (mode == 0) {
-		spin_lock_irqsave(&iommu->register_lock, flags);
-
-		/* enable comaptiblity format interrupt pass through */

(this removal also fixes a typo ;-)

drivers/pci/intr_remapping.c static void iommu_set_intr_remapping(struct intel_iommu *iommu, int mode) · line 438

Quoted 9 lines
drivers/pci/intr_remapping.c +0 -9
-		cmd = iommu->gcmd | DMA_GCMD_CFI;
-		iommu->gcmd |= DMA_GCMD_CFI;
-		writel(cmd, iommu->reg + DMAR_GCMD_REG);
-
-		IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG,
-			      readl, (sts & DMA_GSTS_CFIS), sts);
-
-		spin_unlock_irqrestore(&iommu->register_lock, flags);
-	}

Btw., switching on compat mode might be worthwile to do in one of the failure paths? I.e. we try to switch to IR mode but fail - we should then try to switch to compat pass-through instead of leaving the controller in limbo. Does it matter in your opinion?

Quoted 11 lines
drivers/pci/intr_remapping.c +3 -1
-
 	/*
 	 * global invalidation of interrupt entry cache before enabling
 	 * interrupt-remapping.
@@ -516,6 +502,20 @@ end:
 	spin_unlock_irqrestore(&iommu->register_lock, flags);
 }
 
+int __init intr_remapping_supported(void)
+{
+	struct dmar_drhd_unit *drhd;

drivers/pci/intr_remapping.c end: · line 517

Quoted 10 lines
drivers/pci/intr_remapping.c +10 -0
+
+	for_each_drhd_unit(drhd) {
+		struct intel_iommu *iommu = drhd->iommu;
+
+		if (!ecap_ir_support(iommu->ecap))
+			return 0;
+	}
+
+	return 1;
+}

Jesse, are these bits fine with you? The layering is still a bit incestous but it's a marked improvement over what we had there before.

Ingo

#11 lore raw

Re: [PATCH 3/5] x86, intr-remap: enable interrupt remapping early

Quoted 12 lines · Ingo Molnar

arch/x86/kernel/apic/apic.c static int x2apic_preenabled; · line 122

Quoted 8 lines · Weidong Han
arch/x86/kernel/apic/apic.c +2 -0
@@ -118,6 +118,8 @@ static int x2apic_preenabled;
 static int disable_x2apic;
 static __init int setup_nox2apic(char *str)
 {
+	if (x2apic_enabled())
+		panic("Bios already enabled x2apic, can't enforce nox2apic");

Could you please turn that into something like:

I have couple of patches cleaning these up(including some warnings). I will post them shortly.

Quoted 8 lines

arch/x86/kernel/apic/apic.c end_restore: · line 1393

Quoted 3 lines
arch/x86/kernel/apic/apic.c +3 -0
+ir_failed:
+	if (x2apic_preenabled)
+		panic("x2apic enabled by bios. But IR enabling failed");

What is the likelyhood that we can continue in compat mode? If there's some chance, we should rather print a KERN_WARNING and should try to continue. If IRQs are not coming we'll hang shortly afterwards anyway.

Typically only on high end platforms we will see x2apic enabled by BIOS before OS handover. And in these cases, BIOS should have already enabled interrupt-remapping in extended interrupt mode, before handing over control to OS. If the extended interrupt mode is set, HW will block the compatibility format interrupts anyway. And I think its better to panic rather than hanging at random places(depending on which interrupts are already in remapped format and which are not). This way, it will be easy for anyone to figure out what is happening.

thanks,
suresh

#12 lore raw

Re: [PATCH 3/5] x86, intr-remap: enable interrupt remapping early

Quoted 14 lines · Suresh Siddha
Quoted 12 lines · Ingo Molnar

arch/x86/kernel/apic/apic.c static int x2apic_preenabled; · line 122

Quoted 8 lines · Weidong Han
arch/x86/kernel/apic/apic.c +2 -0
@@ -118,6 +118,8 @@ static int x2apic_preenabled;
 static int disable_x2apic;
 static __init int setup_nox2apic(char *str)
 {
+	if (x2apic_enabled())
+		panic("Bios already enabled x2apic, can't enforce nox2apic");

Could you please turn that into something like:

Quoted 2 lines

I have couple of patches cleaning these up(including some warnings). I will post them shortly.

ok!

Quoted 9 lines
Quoted 8 lines

arch/x86/kernel/apic/apic.c end_restore: · line 1393

Quoted 3 lines
arch/x86/kernel/apic/apic.c +3 -0
+ir_failed:
+	if (x2apic_preenabled)
+		panic("x2apic enabled by bios. But IR enabling failed");

What is the likelyhood that we can continue in compat mode? If there's some chance, we should rather print a KERN_WARNING and should try to continue. If IRQs are not coming we'll hang shortly afterwards anyway.

Quoted 9 lines

Typically only on high end platforms we will see x2apic enabled by BIOS before OS handover. And in these cases, BIOS should have already enabled interrupt-remapping in extended interrupt mode, before handing over control to OS. If the extended interrupt mode is set, HW will block the compatibility format interrupts anyway. And I think its better to panic rather than hanging at random places(depending on which interrupts are already in remapped format and which are not). This way, it will be easy for anyone to figure out what is happening.

fair enough.

Ingo

#13 lore raw

[tip:x86/apic] x86, intr-remap: enable interrupt remapping early

In reply to Weidong Han
Commit-ID:  dfc3c8a9ad3f633d0a3632d5bd165a2786def7c0
Gitweb:     http://git.kernel.org/tip/dfc3c8a9ad3f633d0a3632d5bd165a2786def7c0
Author:     Weidong Han <[email protected]>
AuthorDate: Fri, 17 Apr 2009 16:42:14 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 17 Apr 2009 16:23:45 +0200

x86, intr-remap: enable interrupt remapping early

Currently, when x2apic is not enabled, interrupt remapping will be enabled in init_dmars(), where it is too late to remap ioapic interrupts, that is, ioapic interrupts are really in compatibility mode, not remappable mode.

This patch always enables interrupt remapping before ioapic setup, it guarantees all interrupts will be remapped when interrupt remapping is enabled. Thus it doesn't need to set the compatibility interrupt bit.

[ Impact: refactor intr-remap init sequence, enable fuller remap mode ]

LKML-Reference: <[email protected]>

---
 arch/x86/include/asm/apic.h  |    7 ++--
 arch/x86/kernel/apic/apic.c  |   76 ++++++++++++++++++++----------------------
 drivers/pci/intel-iommu.c    |    9 -----
 drivers/pci/intr_remapping.c |   28 ++++++++--------
 include/linux/dmar.h         |    1 +
 5 files changed, 54 insertions(+), 67 deletions(-)
arch/x86/include/asm/apic.h +3 -4
index d4cb7e5..fbdd654 100644
@@ -169,7 +169,6 @@ static inline u64 native_x2apic_icr_read(void)
 extern int x2apic, x2apic_phys;
 extern void check_x2apic(void);
 extern void enable_x2apic(void);
-extern void enable_IR_x2apic(void);
 extern void x2apic_icr_write(u32 low, u32 id);
 static inline int x2apic_enabled(void)
 {
@@ -190,18 +189,18 @@ static inline void check_x2apic(void)
 static inline void enable_x2apic(void)
 {
 }
-static inline void enable_IR_x2apic(void)
-{
-}
 static inline int x2apic_enabled(void)
 {
 	return 0;
 }
 
 #define	x2apic	0
+#define	x2apic_preenabled 0
 
 #endif
 
+extern void enable_IR_x2apic(void);
+
 extern int get_physical_broadcast(void);
 
 extern void apic_disable(void);
arch/x86/kernel/apic/apic.c +36 -40

130 lines not on this page · raw message

drivers/pci/intel-iommu.c +0 -9
index 001b328..9ce8f07 100644
@@ -1968,15 +1968,6 @@ static int __init init_dmars(void)
 		}
 	}
 
-#ifdef CONFIG_INTR_REMAP
-	if (!intr_remapping_enabled) {
-		ret = enable_intr_remapping(0);
-		if (ret)
-			printk(KERN_ERR
-			       "IOMMU: enable interrupt remapping failed\n");
-	}
-#endif
-
 	/*
 	 * For each rmrr
 	 *   for each dev attached to rmrr
drivers/pci/intr_remapping.c +14 -14
index f5e0ea7..5c21426 100644
@@ -423,20 +423,6 @@ static void iommu_set_intr_remapping(struct intel_iommu *iommu, int mode)
 		      readl, (sts & DMA_GSTS_IRTPS), sts);
 	spin_unlock_irqrestore(&iommu->register_lock, flags);
 
-	if (mode == 0) {
-		spin_lock_irqsave(&iommu->register_lock, flags);
-
-		/* enable comaptiblity format interrupt pass through */
-		cmd = iommu->gcmd | DMA_GCMD_CFI;
-		iommu->gcmd |= DMA_GCMD_CFI;
-		writel(cmd, iommu->reg + DMAR_GCMD_REG);
-
-		IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG,
-			      readl, (sts & DMA_GSTS_CFIS), sts);
-
-		spin_unlock_irqrestore(&iommu->register_lock, flags);
-	}
-
 	/*
 	 * global invalidation of interrupt entry cache before enabling
 	 * interrupt-remapping.
@@ -516,6 +502,20 @@ end:
 	spin_unlock_irqrestore(&iommu->register_lock, flags);
 }
 
+int __init intr_remapping_supported(void)
+{
+	struct dmar_drhd_unit *drhd;
+
+	for_each_drhd_unit(drhd) {
+		struct intel_iommu *iommu = drhd->iommu;
+
+		if (!ecap_ir_support(iommu->ecap))
+			return 0;
+	}
+
+	return 1;
+}
+
 int __init enable_intr_remapping(int eim)
 {
 	struct dmar_drhd_unit *drhd;
include/linux/dmar.h +1 -0
index e397dc3..06f592a 100644
@@ -108,6 +108,7 @@ struct irte {
 };
 #ifdef CONFIG_INTR_REMAP
 extern int intr_remapping_enabled;
+extern int intr_remapping_supported(void);
 extern int enable_intr_remapping(int);
 extern void disable_intr_remapping(void);
 extern int reenable_intr_remapping(int);
#14 lore raw

[tip:x86/apic] x86, intr-remap: enable interrupt remapping early

In reply to Weidong Han
Commit-ID:  937582382c71b75b29fbb92615629494e1a05ac0
Gitweb:     http://git.kernel.org/tip/937582382c71b75b29fbb92615629494e1a05ac0
Author:     Weidong Han <[email protected]>
AuthorDate: Fri, 17 Apr 2009 16:42:14 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 19 Apr 2009 10:21:43 +0200

x86, intr-remap: enable interrupt remapping early

Currently, when x2apic is not enabled, interrupt remapping will be enabled in init_dmars(), where it is too late to remap ioapic interrupts, that is, ioapic interrupts are really in compatibility mode, not remappable mode.

This patch always enables interrupt remapping before ioapic setup, it guarantees all interrupts will be remapped when interrupt remapping is enabled. Thus it doesn't need to set the compatibility interrupt bit.

[ Impact: refactor intr-remap init sequence, enable fuller remap mode ]

LKML-Reference: <[email protected]>

---
 arch/x86/include/asm/apic.h  |    7 ++--
 arch/x86/kernel/apic/apic.c  |   76 ++++++++++++++++++++----------------------
 drivers/pci/intel-iommu.c    |    9 -----
 drivers/pci/intr_remapping.c |   28 ++++++++--------
 include/linux/dmar.h         |    1 +
 5 files changed, 54 insertions(+), 67 deletions(-)
arch/x86/include/asm/apic.h +3 -4
index d4cb7e5..fbdd654 100644
@@ -169,7 +169,6 @@ static inline u64 native_x2apic_icr_read(void)
 extern int x2apic, x2apic_phys;
 extern void check_x2apic(void);
 extern void enable_x2apic(void);
-extern void enable_IR_x2apic(void);
 extern void x2apic_icr_write(u32 low, u32 id);
 static inline int x2apic_enabled(void)
 {
@@ -190,18 +189,18 @@ static inline void check_x2apic(void)
 static inline void enable_x2apic(void)
 {
 }
-static inline void enable_IR_x2apic(void)
-{
-}
 static inline int x2apic_enabled(void)
 {
 	return 0;
 }
 
 #define	x2apic	0
+#define	x2apic_preenabled 0
 
 #endif
 
+extern void enable_IR_x2apic(void);
+
 extern int get_physical_broadcast(void);
 
 extern void apic_disable(void);
arch/x86/kernel/apic/apic.c +36 -40

130 lines not on this page · raw message

drivers/pci/intel-iommu.c +0 -9
index 001b328..9ce8f07 100644
@@ -1968,15 +1968,6 @@ static int __init init_dmars(void)
 		}
 	}
 
-#ifdef CONFIG_INTR_REMAP
-	if (!intr_remapping_enabled) {
-		ret = enable_intr_remapping(0);
-		if (ret)
-			printk(KERN_ERR
-			       "IOMMU: enable interrupt remapping failed\n");
-	}
-#endif
-
 	/*
 	 * For each rmrr
 	 *   for each dev attached to rmrr
drivers/pci/intr_remapping.c +14 -14
index f5e0ea7..5c21426 100644
@@ -423,20 +423,6 @@ static void iommu_set_intr_remapping(struct intel_iommu *iommu, int mode)
 		      readl, (sts & DMA_GSTS_IRTPS), sts);
 	spin_unlock_irqrestore(&iommu->register_lock, flags);
 
-	if (mode == 0) {
-		spin_lock_irqsave(&iommu->register_lock, flags);
-
-		/* enable comaptiblity format interrupt pass through */
-		cmd = iommu->gcmd | DMA_GCMD_CFI;
-		iommu->gcmd |= DMA_GCMD_CFI;
-		writel(cmd, iommu->reg + DMAR_GCMD_REG);
-
-		IOMMU_WAIT_OP(iommu, DMAR_GSTS_REG,
-			      readl, (sts & DMA_GSTS_CFIS), sts);
-
-		spin_unlock_irqrestore(&iommu->register_lock, flags);
-	}
-
 	/*
 	 * global invalidation of interrupt entry cache before enabling
 	 * interrupt-remapping.
@@ -516,6 +502,20 @@ end:
 	spin_unlock_irqrestore(&iommu->register_lock, flags);
 }
 
+int __init intr_remapping_supported(void)
+{
+	struct dmar_drhd_unit *drhd;
+
+	for_each_drhd_unit(drhd) {
+		struct intel_iommu *iommu = drhd->iommu;
+
+		if (!ecap_ir_support(iommu->ecap))
+			return 0;
+	}
+
+	return 1;
+}
+
 int __init enable_intr_remapping(int eim)
 {
 	struct dmar_drhd_unit *drhd;
include/linux/dmar.h +1 -0
index e397dc3..06f592a 100644
@@ -108,6 +108,7 @@ struct irte {
 };
 #ifdef CONFIG_INTR_REMAP
 extern int intr_remapping_enabled;
+extern int intr_remapping_supported(void);
 extern int enable_intr_remapping(int);
 extern void disable_intr_remapping(void);
 extern int reenable_intr_remapping(int);
#15 lore raw

PATCH 4/5 x86, intr-remap: add option to disable interrupt remapping

In reply to Weidong Han

Add option "nointremap" to disable interrupt remapping.

---
 Documentation/kernel-parameters.txt |    3 +++
 drivers/pci/intr_remapping.c        |   11 +++++++++++
 2 files changed, 14 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +3 -0
index 33989d2..843cb66 100644
@@ -1533,6 +1533,9 @@ and is between 256 and 4096 characters. It is defined in the file
 	noinitrd	[RAM] Tells the kernel not to load any configured
 			initial RAM disk.
 
+	nointremap	[X86-64, Intel-IOMMU] Do not enable interrupt
+			remapping.
+
 	nointroute	[IA-64]
 
 	nojitter	[IA64] Disables jitter checking for ITC timers.
drivers/pci/intr_remapping.c +11 -0
index 5c21426..842039e 100644
@@ -15,6 +15,14 @@ static struct ioapic_scope ir_ioapic[MAX_IO_APICS];
 static int ir_ioapic_num;
 int intr_remapping_enabled;
 
+static int disable_intremap;
+static __init int setup_nointremap(char *str)
+{
+	disable_intremap = 1;
+	return 0;
+}
+early_param("nointremap", setup_nointremap);
+
 struct irq_2_iommu {
 	struct intel_iommu *iommu;
 	u16 irte_index;
@@ -506,6 +514,9 @@ int __init intr_remapping_supported(void)
 {
 	struct dmar_drhd_unit *drhd;
 
+	if (disable_intremap)
+		return 0;
+
 	for_each_drhd_unit(drhd) {
 		struct intel_iommu *iommu = drhd->iommu;
 
Signature · 1 line
1.6.0.4
#16 lore raw

[tip:x86/apic] x86, intr-remap: add option to disable interrupt remapping

Commit-ID:  52549a96faf1852128c2a1850b29197f2d41a281
Gitweb:     http://git.kernel.org/tip/52549a96faf1852128c2a1850b29197f2d41a281
Author:     Weidong Han <[email protected]>
AuthorDate: Fri, 17 Apr 2009 16:42:15 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 17 Apr 2009 16:23:45 +0200

x86, intr-remap: add option to disable interrupt remapping

Add option "nointremap" to disable interrupt remapping.

[ Impact: add new boot option ]

LKML-Reference: <[email protected]>

---
 Documentation/kernel-parameters.txt |    3 +++
 drivers/pci/intr_remapping.c        |   11 +++++++++++
 2 files changed, 14 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +3 -0
index 33989d2..843cb66 100644
@@ -1533,6 +1533,9 @@ and is between 256 and 4096 characters. It is defined in the file
 	noinitrd	[RAM] Tells the kernel not to load any configured
 			initial RAM disk.
 
+	nointremap	[X86-64, Intel-IOMMU] Do not enable interrupt
+			remapping.
+
 	nointroute	[IA-64]
 
 	nojitter	[IA64] Disables jitter checking for ITC timers.
drivers/pci/intr_remapping.c +11 -0
index 5c21426..842039e 100644
@@ -15,6 +15,14 @@ static struct ioapic_scope ir_ioapic[MAX_IO_APICS];
 static int ir_ioapic_num;
 int intr_remapping_enabled;
 
+static int disable_intremap;
+static __init int setup_nointremap(char *str)
+{
+	disable_intremap = 1;
+	return 0;
+}
+early_param("nointremap", setup_nointremap);
+
 struct irq_2_iommu {
 	struct intel_iommu *iommu;
 	u16 irte_index;
@@ -506,6 +514,9 @@ int __init intr_remapping_supported(void)
 {
 	struct dmar_drhd_unit *drhd;
 
+	if (disable_intremap)
+		return 0;
+
 	for_each_drhd_unit(drhd) {
 		struct intel_iommu *iommu = drhd->iommu;
 
#17 lore raw

[tip:x86/apic] x86, intr-remap: add option to disable interrupt remapping

In reply to Weidong Han
Commit-ID:  03ea81550676296d94596e4337c771c6ba29f542
Gitweb:     http://git.kernel.org/tip/03ea81550676296d94596e4337c771c6ba29f542
Author:     Weidong Han <[email protected]>
AuthorDate: Fri, 17 Apr 2009 16:42:15 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 19 Apr 2009 10:21:54 +0200

x86, intr-remap: add option to disable interrupt remapping

Add option "nointremap" to disable interrupt remapping.

[ Impact: add new boot option ]

LKML-Reference: <[email protected]>

---
 Documentation/kernel-parameters.txt |    3 +++
 drivers/pci/intr_remapping.c        |   11 +++++++++++
 2 files changed, 14 insertions(+), 0 deletions(-)
Documentation/kernel-parameters.txt +3 -0
index 33989d2..843cb66 100644
@@ -1533,6 +1533,9 @@ and is between 256 and 4096 characters. It is defined in the file
 	noinitrd	[RAM] Tells the kernel not to load any configured
 			initial RAM disk.
 
+	nointremap	[X86-64, Intel-IOMMU] Do not enable interrupt
+			remapping.
+
 	nointroute	[IA-64]
 
 	nojitter	[IA64] Disables jitter checking for ITC timers.
drivers/pci/intr_remapping.c +11 -0
index 5c21426..842039e 100644
@@ -15,6 +15,14 @@ static struct ioapic_scope ir_ioapic[MAX_IO_APICS];
 static int ir_ioapic_num;
 int intr_remapping_enabled;
 
+static int disable_intremap;
+static __init int setup_nointremap(char *str)
+{
+	disable_intremap = 1;
+	return 0;
+}
+early_param("nointremap", setup_nointremap);
+
 struct irq_2_iommu {
 	struct intel_iommu *iommu;
 	u16 irte_index;
@@ -506,6 +514,9 @@ int __init intr_remapping_supported(void)
 {
 	struct dmar_drhd_unit *drhd;
 
+	if (disable_intremap)
+		return 0;
+
 	for_each_drhd_unit(drhd) {
 		struct intel_iommu *iommu = drhd->iommu;
 
#18 lore raw

PATCH 5/5 x86: fix x2apic/intr-remap resume

In reply to Weidong Han

Interrupt remapping was decoupled from x2apic. Shouldn't check x2apic before resume interrupt remapping. Otherwise, interrupt remapping won't be resumed when x2apic is not enabled.

---
 arch/x86/kernel/apic/apic.c |   15 +++++++++------
 1 files changed, 9 insertions(+), 6 deletions(-)
arch/x86/kernel/apic/apic.c +9 -6
index 39b621c..03b591e 100644
@@ -1987,7 +1987,7 @@ static int lapic_resume(struct sys_device *dev)
 		return 0;
 
 	local_irq_save(flags);
-	if (x2apic) {
+	if (intr_remapping_enabled) {
 		ioapic_entries = alloc_ioapic_entries();
 		if (!ioapic_entries) {
 			WARN(1, "Alloc ioapic_entries in lapic resume failed.");
@@ -2003,8 +2003,10 @@ static int lapic_resume(struct sys_device *dev)
 
 		mask_IO_APIC_setup(ioapic_entries);
 		mask_8259A();
-		enable_x2apic();
 	}
+
+	if (x2apic)
+		enable_x2apic();
 #else
 	if (!apic_pm_state.active)
 		return 0;
@@ -2052,10 +2054,12 @@ static int lapic_resume(struct sys_device *dev)
 	apic_read(APIC_ESR);
 
 #ifdef CONFIG_INTR_REMAP
-	if (intr_remapping_enabled)
-		reenable_intr_remapping(EIM_32BIT_APIC_ID);
+	if (intr_remapping_enabled) {
+		if (x2apic)
+			reenable_intr_remapping(EIM_32BIT_APIC_ID);
+		else
+			reenable_intr_remapping(EIM_8BIT_APIC_ID);
 
-	if (x2apic) {
 		unmask_8259A();
 		restore_IO_APIC_setup(ioapic_entries);
 		free_ioapic_entries(ioapic_entries);
@@ -2064,7 +2068,6 @@ static int lapic_resume(struct sys_device *dev)
 
 	local_irq_restore(flags);
 
-
 	return 0;
 }
 
Signature · 1 line
1.6.0.4
#19 lore raw

Re: [PATCH 5/5] x86: fix x2apic/intr-remap resume

arch/x86/kernel/apic/apic.c static int lapic_resume(struct sys_device *dev) · line 2061

Quoted 8 lines · Weidong Han

#ifdef CONFIG_INTR_REMAP

arch/x86/kernel/apic/apic.c +5 -2
-	if (intr_remapping_enabled)
-		reenable_intr_remapping(EIM_32BIT_APIC_ID);
+	if (intr_remapping_enabled) {
+		if (x2apic)
+			reenable_intr_remapping(EIM_32BIT_APIC_ID);
+		else
+			reenable_intr_remapping(EIM_8BIT_APIC_ID);

ok, we had this sequence before. Might be worthwile to push the x2apic condition into reenable_intr_remapping() itself? That would simplify the call sites.

Ingo

#20 lore raw

[tip:x86/apic] x86, intr-remap: fix x2apic/intr-remap resume

In reply to Weidong Han
Commit-ID:  752b065ceacda418a1f2c956353fd7b1d55020ea
Gitweb:     http://git.kernel.org/tip/752b065ceacda418a1f2c956353fd7b1d55020ea
Author:     Weidong Han <[email protected]>
AuthorDate: Fri, 17 Apr 2009 16:42:16 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Fri, 17 Apr 2009 16:23:46 +0200

x86, intr-remap: fix x2apic/intr-remap resume

Interrupt remapping was decoupled from x2apic. Shouldn't check x2apic before resume interrupt remapping. Otherwise, interrupt remapping won't be resumed when x2apic is not enabled.

[ Impact: fix potential intr-remap resume hang on !x2apic ]

LKML-Reference: <[email protected]>

---
 arch/x86/kernel/apic/apic.c |   15 +++++++++------
 1 files changed, 9 insertions(+), 6 deletions(-)
arch/x86/kernel/apic/apic.c +9 -6
index 0cf1eea..7b41a32 100644
@@ -2032,7 +2032,7 @@ static int lapic_resume(struct sys_device *dev)
 		return 0;
 
 	local_irq_save(flags);
-	if (x2apic) {
+	if (intr_remapping_enabled) {
 		ioapic_entries = alloc_ioapic_entries();
 		if (!ioapic_entries) {
 			WARN(1, "Alloc ioapic_entries in lapic resume failed.");
@@ -2048,8 +2048,10 @@ static int lapic_resume(struct sys_device *dev)
 
 		mask_IO_APIC_setup(ioapic_entries);
 		mask_8259A();
-		enable_x2apic();
 	}
+
+	if (x2apic)
+		enable_x2apic();
 #else
 	if (!apic_pm_state.active)
 		return 0;
@@ -2097,10 +2099,12 @@ static int lapic_resume(struct sys_device *dev)
 	apic_read(APIC_ESR);
 
 #ifdef CONFIG_INTR_REMAP
-	if (intr_remapping_enabled)
-		reenable_intr_remapping(EIM_32BIT_APIC_ID);
+	if (intr_remapping_enabled) {
+		if (x2apic)
+			reenable_intr_remapping(EIM_32BIT_APIC_ID);
+		else
+			reenable_intr_remapping(EIM_8BIT_APIC_ID);
 
-	if (x2apic) {
 		unmask_8259A();
 		restore_IO_APIC_setup(ioapic_entries);
 		free_ioapic_entries(ioapic_entries);
@@ -2109,7 +2113,6 @@ static int lapic_resume(struct sys_device *dev)
 
 	local_irq_restore(flags);
 
-
 	return 0;
 }
 
#21 lore raw

[tip:x86/apic] x86, intr-remap: fix x2apic/intr-remap resume

In reply to Weidong Han
Commit-ID:  9a2755c3569e4db92bd9b1daadeddb4045b0cccd
Gitweb:     http://git.kernel.org/tip/9a2755c3569e4db92bd9b1daadeddb4045b0cccd
Author:     Weidong Han <[email protected]>
AuthorDate: Fri, 17 Apr 2009 16:42:16 +0800
Committer:  Ingo Molnar <[email protected]>
CommitDate: Sun, 19 Apr 2009 10:22:05 +0200

x86, intr-remap: fix x2apic/intr-remap resume

Interrupt remapping was decoupled from x2apic. Shouldn't check x2apic before resume interrupt remapping. Otherwise, interrupt remapping won't be resumed when x2apic is not enabled.

[ Impact: fix potential intr-remap resume hang on !x2apic ]

LKML-Reference: <[email protected]>

---
 arch/x86/kernel/apic/apic.c |   15 +++++++++------
 1 files changed, 9 insertions(+), 6 deletions(-)
arch/x86/kernel/apic/apic.c +9 -6
index 0cf1eea..7b41a32 100644
@@ -2032,7 +2032,7 @@ static int lapic_resume(struct sys_device *dev)
 		return 0;
 
 	local_irq_save(flags);
-	if (x2apic) {
+	if (intr_remapping_enabled) {
 		ioapic_entries = alloc_ioapic_entries();
 		if (!ioapic_entries) {
 			WARN(1, "Alloc ioapic_entries in lapic resume failed.");
@@ -2048,8 +2048,10 @@ static int lapic_resume(struct sys_device *dev)
 
 		mask_IO_APIC_setup(ioapic_entries);
 		mask_8259A();
-		enable_x2apic();
 	}
+
+	if (x2apic)
+		enable_x2apic();
 #else
 	if (!apic_pm_state.active)
 		return 0;
@@ -2097,10 +2099,12 @@ static int lapic_resume(struct sys_device *dev)
 	apic_read(APIC_ESR);
 
 #ifdef CONFIG_INTR_REMAP
-	if (intr_remapping_enabled)
-		reenable_intr_remapping(EIM_32BIT_APIC_ID);
+	if (intr_remapping_enabled) {
+		if (x2apic)
+			reenable_intr_remapping(EIM_32BIT_APIC_ID);
+		else
+			reenable_intr_remapping(EIM_8BIT_APIC_ID);
 
-	if (x2apic) {
 		unmask_8259A();
 		restore_IO_APIC_setup(ioapic_entries);
 		free_ioapic_entries(ioapic_entries);
@@ -2109,7 +2113,6 @@ static int lapic_resume(struct sys_device *dev)
 
 	local_irq_restore(flags);
 
-
 	return 0;
 }
 
#22 lore raw
In reply to Weidong Han
Quoted 20 lines · Weidong Han

interupt remapping was decoupled from x2apic already, but there are still some issues, such as doesn't ack remapped interrupts correctly, and doesn't remap ioapic interrupt when x2apic is not enabled.

This patchset fixes the ack for remapped interrupts, and alwasys enables interrupt remapping before ioapic setup to guarantee ioapic interrupts are remapped, adjusts the dependency of x2apic and interrupt remapping in lapic_resume. In addition, add option "nointremap" to disable interrupt remapping.

The patchset can be applied on linux head. Thanks.

Weidong Han (5):
docs: add nox2apic back to kernel-parameters.txt x86,intr-remap: fix ack for interrupt remapping x86, intr-remap: enable interrupt remapping early x86, intr-remap: add option to disable interrupt remapping
x86: fix x2apic/intr-remap resume

Quoted 8 lines
 Documentation/kernel-parameters.txt |    5 ++
 arch/x86/include/asm/apic.h         |   15 +-----
 arch/x86/kernel/apic/apic.c         |   91 +++++++++++++++++------------------
 arch/x86/kernel/apic/io_apic.c      |   32 ++----------
 drivers/pci/intel-iommu.c           |    9 ----
 drivers/pci/intr_remapping.c        |   39 ++++++++++-----
 include/linux/dmar.h                |    1 +
 7 files changed, 84 insertions(+), 108 deletions(-)

Thanks, this series is exactly what the doctor ordered: it fixes real problems and still manages to shrink the code!

I've made a few smaller observations to specific patches but none seemed like a real showstopper to me so i've applied your series to tip:x86/apic (with minor edits to the changelogs) to get this tested some more - pending Ack/Nak from David and Jesse for the IOMMU and PCI interactions.

Please do the fixlets/cleanups as delta patches against tip:x86/apic:

http://people.redhat.com/mingo/tip.git/README

Thanks,

Ingo

#23 lore raw

Note, there's a new compiler warning caused by your patches:

arch/x86/kernel/apic/io_apic.c:2543: warning: ‘eoi_ioapic_irq’ defined but not used

most likely by:

631dd75: x86, intr-remap: fix ack for interrupt remapping

Ingo

#24 lore raw
Quoted 10 lines · Ingo Molnar

Note, there's a new compiler warning caused by your patches:

arch/x86/kernel/apic/io_apic.c:2543: warning: 'eoi_ioapic_irq'
defined but not used

most likely by:

631dd75: x86, intr-remap: fix ack for interrupt remapping

Ingo

eoi_ioapic_irq is only used for remapped interrupt, so we should define eoi_ioapic_irq and __eoi_ioapic_irq when #ifdef CONFIG_INTR_REMAP.

Regards,
Weidong

#25 lore raw
Quoted 8 lines · Han, Weidong
Quoted 7 lines · Ingo Molnar

Note, there's a new compiler warning caused by your patches:

arch/x86/kernel/apic/io_apic.c:2543: warning: 'eoi_ioapic_irq'
defined but not used

most likely by:

Quoted 7 lines
Quoted 3 lines

631dd75: x86, intr-remap: fix ack for interrupt remapping

Ingo

eoi_ioapic_irq is only used for remapped interrupt, so we should define eoi_ioapic_irq and __eoi_ioapic_irq when #ifdef CONFIG_INTR_REMAP.

Ok - please send a patch that fixes this.

Btw., it would be nice to reduce the number of #ifdef CONFIG_INTR_REMAP's in all .c files in general - or even eliminate them.

It might be feasible to move all the INTR_REMAP code into a separate .c file and only build it in the INTR_REMAP case. I have not tried to do that so i dont know - if the interfaces are too wide and unnatural to the rest of the ioapic code it might not improve the end result - but good layering generally does do the trick.

Ingo

#26 lore raw
In reply to Ingo Molnar
Quoted 6 lines · Ingo Molnar

I've made a few smaller observations to specific patches but none seemed like a real showstopper to me so i've applied your series to tip:x86/apic (with minor edits to the changelogs) to get this tested some more - pending Ack/Nak from David and Jesse for the IOMMU and PCI interactions.

For all five:

Signature · 1 line
dwmw2
#27 lore raw
Quoted 10 lines · David Woodhouse
Quoted 6 lines · Ingo Molnar

I've made a few smaller observations to specific patches but none seemed like a real showstopper to me so i've applied your series to tip:x86/apic (with minor edits to the changelogs) to get this tested some more - pending Ack/Nak from David and Jesse for the IOMMU and PCI interactions.

For all five:

Thanks!

Ingo