x86, intr-remapping patches for addressing kexec/kdump issues
cover x86, intr-remapping patches for addressing kexec/kdump issues
Following patches address/workaround the issues we have identified with the interrupt-remapping code flow while debugging hangs/spurious NMI's we have seen with different OEM platforms during kexec/kdump in the presence of interrupt-remapping (and x2apic in some cases).
All the patches are small and self-contained and are marked stable as it makes kexec/kdump functional on these platforms. While some of these patches touch pci files and self-contained, I would appreciate if all these patches get routed to Linus tree (for v2.6.37) through -tip tree.
thanks,
suresh
PATCH 1/4 vt-d: quirk for masking vtd spec errors to platform error handling logic
On platforms with Intel 7500 chipset, there were some reports of system hang/NMI's during kexec/kdump in the presence of interrupt-remapping enabled.
During kdump, there is a window where the devices might be still using old kernel's interrupt information, while the kdump kernel is coming up. This can cause vt-d faults as the interrupt configuration from the old kernel map to null IRTE entries in the new kernel etc. (with out interrupt-remapping enabled, we still have the same issue but in this case we will see benign spurious interrupt hit the new kernel).
Based on platform config settings, these platforms seem to generate NMI/SMI when a vt-d fault happens and there were reports that the resulting SMI causes the system to hang.
Fix it by masking vt-d spec defined errors to platform error reporting logic. VT-d spec related errors are already handled by the VT-d OS code, so need to report the same erorr through other channels.
--- drivers/pci/quirks.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)
drivers/pci/quirks.c +20 -0
@@ -2764,6 +2764,26 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_RIDECLARE_PCI_FIXUP_RESUME_EARLY (PCI_VENDOR_ID_RICOH ,PCI_DEVICE_ID_RICOH_R5C832 , ricoh_mmc_fixup_r5c832);#endif /*CONFIG_MMC_RICOH_MMC*/ +#if defined(CONFIG_DMAR ) || defined(CONFIG_INTR_REMAP ) +/* +* This is a quirk for masking vt-d spec defined errors to platform error +* handling logic. With out this, platforms seem to generate NMI/SMI (based +* on the RAS config settings of the platform) when a vt-d fault happens and +* there were reports that the resulting SMI causes system to hang. +* +* VT-d spec related errors are already handled by the VT-d OS code, so no +* need to report the same erorr through other channels. +*/ +static void vtd_mask_spec_errors (struct pci_dev *dev) +{ +u32 word; + +pci_read_config_dword (dev,0x1AC , &word); +pci_write_config_dword (dev,0x1AC , word | (1 <<31 )); +} +DECLARE_PCI_FIXUP_EARLY (PCI_VENDOR_ID_INTEL ,0x342e , vtd_mask_spec_errors); +DECLARE_PCI_FIXUP_EARLY (PCI_VENDOR_ID_INTEL ,0x3c28 , vtd_mask_spec_errors); +#endif static void pci_do_fixups (struct pci_dev *dev,struct pci_fixup *f,struct pci_fixup *end)
PATCH 4/4 vt-d: handle previous faults after enabling fault handling
In reply to Suresh SiddhaFault handling is getting enabled after enabling the interrupt-remapping (as the success of interrupt-remapping can affect the apic mode and hence the fault handling mode).
Hence there can potentially be some faults between the window of enabling interrupt-remapping in the vt-d and the fault-handling of the vt-d units.
Handle any previous faults after enabling the vt-d fault handling.
For v2.6.38 cleanup, need to check if we can remove the dmar_fault() in the enable_intr_remapping() and see if we can enable fault handling along with enabling intr-remapping.
--- drivers/pci/dmar.c | 5 +++++ 1 file changed, 5 insertions(+)
drivers/pci/dmar.c +5 -0
@@ -1417,6 +1417,11 @@ int __init enable_drhd_fault_handling(vo (unsigned long long)drhd->reg_base_addr, ret);return -1 ; } + +/* +* Clear any previous faults. +*/ +dmar_fault (iommu->irq, iommu); }return 0 ;
PATCH 3/4 x86: enable the intr-remap fault handling after local apic setup
In reply to Suresh SiddhaFrom: Kenji Kaneshige <[email protected]>
Subject: x86: enable the intr-remap fault handling after local apic setup
Interrupt-remapping gets enabled very early in the boot, as it determines the apic mode that the processor can use. And the current code enables the vt-d fault handling before the setup_local_APIC(). And hence the APIC LDR registers and data structure in the memory may not be initialized. So the vt-d fault handling in logical xapic/x2apic modes were broken.
Fix this by enabling the vt-d fault handling in the end_local_APIC_setup()
A cleaner fix of enabling fault handling while enabling intr-remapping will be addressed for v2.6.38. [ Enabling intr-remapping determines the usage of x2apic mode and the apic mode determines the fault-handling configuration. ]
--- arch/x86/kernel/apic/apic.c | 8 ++++++++ arch/x86/kernel/apic/probe_64.c | 7 ------- 2 files changed, 8 insertions(+), 7 deletions(-)
arch/x86/kernel/apic/apic.c +8 -0
@@ -1384,6 +1384,14 @@ void __cpuinit end_local_APIC_setup(void#endif apic_pm_activate (); + +/* +* Now that local APIC setup is completed for BP, configure the fault +* handling for interrupt remapping. +*/ +if (!smp_processor_id () && intr_remapping_enabled) +enable_drhd_fault_handling (); + }#ifdef CONFIG_X86_X2APIC
arch/x86/kernel/apic/probe_64.c +0 -7
@@ -79,13 +79,6 @@ void __init default_setup_apic_routing(v/* need to update phys_pkg_id */ apic->phys_pkg_id = apicid_phys_pkg_id; } - -/* -* Now that apic routing model is selected, configure the -* fault handling for intr remapping. -*/ -if (intr_remapping_enabled) -enable_drhd_fault_handling (); }/* Same for both flat and physical. */
PATCH 2/4 x86, vtd: fix the vt-d fault handling irq migration in the x2apic mode
In reply to Suresh SiddhaFrom: Kenji Kaneshige <[email protected]>
Subject: x86, vtd: fix the vt-d fault handling irq migration in the x2apic mode
In x2apic mode, we need to set the upper address register of the fault handling interrupt register of the vt-d hardware. Without this irq migration of the vt-d fault handling interrupt is broken.
--- arch/x86/kernel/apic/io_apic.c | 2 ++ 1 file changed, 2 insertions(+)
arch/x86/kernel/apic/io_apic.c +2 -0
@@ -3367,6 +3367,8 @@ dmar_msi_set_affinity(struct irq_data *d msg.data |=MSI_DATA_VECTOR (cfg->vector); msg.address_lo &= ~MSI_ADDR_DEST_ID_MASK ; msg.address_lo |=MSI_ADDR_DEST_ID (dest); +if (x2apic_mode) + msg.address_hi =MSI_ADDR_BASE_HI |MSI_ADDR_EXT_DEST_ID (dest);dmar_msi_write (irq, &msg);
Re: [patch 1/4] vt-d: quirk for masking vtd spec errors to platform error handling logic
In reply to Suresh SiddhaQuoted 14 lines · * Suresh Siddha ([email protected]) wrote:
On platforms with Intel 7500 chipset, there were some reports of system hang/NMI's during kexec/kdump in the presence of interrupt-remapping enabled.
During kdump, there is a window where the devices might be still using old kernel's interrupt information, while the kdump kernel is coming up. This can cause vt-d faults as the interrupt configuration from the old kernel map to null IRTE entries in the new kernel etc. (with out interrupt-remapping enabled, we still have the same issue but in this case we will see benign spurious interrupt hit the new kernel).
Based on platform config settings, these platforms seem to generate NMI/SMI when a vt-d fault happens and there were reports that the resulting SMI causes the system to hang.
Quoted 5 lines
Fix it by masking vt-d spec defined errors to platform error reporting logic. VT-d spec related errors are already handled by the VT-d OS code, so need to report the same erorr through other channels.
Signed-off-by Suresh Siddha <[email protected]>
Re: [patch 3/4] x86: enable the intr-remap fault handling after local apic setup
In reply to Suresh SiddhaQuoted 11 lines · * Suresh Siddha ([email protected]) wrote:
From: Kenji Kaneshige <[email protected]>
Subject: x86: enable the intr-remap fault handling after local apic setupInterrupt-remapping gets enabled very early in the boot, as it determines the apic mode that the processor can use. And the current code enables the vt-d fault handling before the
setup_local_APIC(). And hence the APIC LDR registers and data structure in the memory may not be initialized. So the vt-d fault handling in logical xapic/x2apic modes were broken.Fix this by enabling the vt-d fault handling in the
end_local_APIC_setup()
Quoted 7 lines
A cleaner fix of enabling fault handling while enabling intr-remapping will be addressed for v2.6.38. [ Enabling intr-remapping determines the usage of x2apic mode and the apic mode determines the fault-handling configuration. ]
Signed-off-by Kenji Kaneshige <[email protected]>Signed-off-by Suresh Siddha <[email protected]>
Re: [patch 2/4] x86, vtd: fix the vt-d fault handling irq migration in the x2apic mode
In reply to Suresh SiddhaQuoted 9 lines · * Suresh Siddha ([email protected]) wrote:
From: Kenji Kaneshige <[email protected]>
Subject: x86, vtd: fix the vt-d fault handling irq migration in the x2apic modeIn x2apic mode, we need to set the upper address register of the fault handling interrupt register of the vt-d hardware. Without this irq migration of the vt-d fault handling interrupt is broken.
Signed-off-by Kenji Kaneshige <[email protected]>Signed-off-by Suresh Siddha <[email protected]>
Looks correct, I didn't have a chance to test this patch.
Re: [patch 4/4] vt-d: handle previous faults after enabling fault handling
In reply to Suresh SiddhaQuoted 7 lines · * Suresh Siddha ([email protected]) wrote:
Fault handling is getting enabled after enabling the interrupt-remapping (as the success of interrupt-remapping can affect the apic mode and hence the fault handling mode).
Hence there can potentially be some faults between the window of enabling interrupt-remapping in the vt-d and the fault-handling of the vt-d units.
Quoted 7 lines
Handle any previous faults after enabling the vt-d fault handling.
For v2.6.38 cleanup, need to check if we can remove the
dmar_fault()in theenable_intr_remapping()and see if we can enable fault handling along with enabling intr-remapping.Signed-off-by Suresh Siddha <[email protected]>
Re: [patch 2/4] x86, vtd: fix the vt-d fault handling irq migration in the x2apic mode
In reply to Suresh SiddhaQuoted 19 lines · Suresh Siddha
From: Kenji Kaneshige <[email protected]>
Subject: x86, vtd: fix the vt-d fault handling irq migration in the x2apic modeIn x2apic mode, we need to set the upper address register of the fault handling interrupt register of the vt-d hardware. Without this irq migration of the vt-d fault handling interrupt is broken.
Signed-off-by Kenji Kaneshige <[email protected]>Signed-off-by Suresh Siddha <[email protected]>Cc [email protected] [v2.6.32+]--- arch/x86/kernel/apic/io_apic.c | 2 ++ 1 file changed, 2 insertions(+)Index:
tip/arch/x86/kernel/apic/io_apic.c
===================================================================
arch/x86/kernel/apic/io_apic.c dmar_msi_set_affinity(struct irq_data *d · line 3371
Quoted 5 lines
arch/x86/kernel/apic/io_apic.c +2
-0@@ -3367,6 +3367,8 @@ dmar_msi_set_affinity(struct irq_data *d msg.data |=MSI_DATA_VECTOR (cfg->vector); msg.address_lo &= ~MSI_ADDR_DEST_ID_MASK ; msg.address_lo |=MSI_ADDR_DEST_ID (dest); +if (x2apic_mode) + msg.address_hi =MSI_ADDR_BASE_HI |MSI_ADDR_EXT_DEST_ID (dest);
Is it necessary to test x2apic_mode here? It looks like MSI_ADDR_EXT_DEST_ID() gives you everything above the low 8 bits of the APIC ID. If those bits are always zero except in x2apic_mode, we might not need the test.
Does the ia64 dmar_msi_set_affinity() need the same fix?
Why do we have both x2apic_enabled() and x2apic_mode? They seem sort of redundant. (Not related to this patch, of course.)
Bjorn
Re: [patch 2/4] x86, vtd: fix the vt-d fault handling irq migration in the x2apic mode
Quoted 8 lines · Bjorn Helgaas
Review
Quoted 2 lines · Suresh Siddha
(quoted diff) +2
-0+ if (x2apic_mode) + msg.address_hi = MSI_ADDR_BASE_HI | MSI_ADDR_EXT_DEST_ID(dest);Is it necessary to test
x2apic_modehere? It looks likeMSI_ADDR_EXT_DEST_ID()gives you everything above the low 8 bits of the APIC ID. If those bits are always zero except inx2apic_mode, we might not need the test.
True. Appended the updated patch.
Quoted 1 line
Does the ia64
dmar_msi_set_affinity()need the same fix?
No.
Quoted 3 lines
Why do we have both
x2apic_enabled()andx2apic_mode? They seem sort of redundant. (Not related to this patch, of course.)
BIOS can handover to OS in x2apic mode in some cases. x2apic_enabled() is used to check for that and it reads the MSR to check the status. Some early portions of the kernel boot will use it.
For all others, we should be using x2apic_mode.
thanks,
suresh
---
From: Kenji Kaneshige <[email protected]>
Subject: x86, vtd: fix the vt-d fault handling irq migration in the x2apic mode
In x2apic mode, we need to set the upper address register of the fault handling interrupt register of the vt-d hardware. Without this irq migration of the vt-d fault handling interrupt is broken.
--- arch/x86/kernel/apic/io_apic.c | 1 + 1 file changed, 1 insertion(+)
arch/x86/kernel/apic/io_apic.c +1 -0
@@ -3367,6 +3367,7 @@ dmar_msi_set_affinity(struct irq_data *d msg.data |=MSI_DATA_VECTOR (cfg->vector); msg.address_lo &= ~MSI_ADDR_DEST_ID_MASK ; msg.address_lo |=MSI_ADDR_DEST_ID (dest); + msg.address_hi =MSI_ADDR_BASE_HI |MSI_ADDR_EXT_DEST_ID (dest);dmar_msi_write (irq, &msg);
Re: [patch 1/4] vt-d: quirk for masking vtd spec errors to platform error handling logic
In reply to Suresh SiddhaQuoted 45 lines · On Tue, 30 Nov 2010 22:22:26 -0800 Suresh Siddha <[email protected]>…
On platforms with Intel 7500 chipset, there were some reports of system hang/NMI's during kexec/kdump in the presence of interrupt-remapping enabled.
During kdump, there is a window where the devices might be still using old kernel's interrupt information, while the kdump kernel is coming up. This can cause vt-d faults as the interrupt configuration from the old kernel map to null IRTE entries in the new kernel etc. (with out interrupt-remapping enabled, we still have the same issue but in this case we will see benign spurious interrupt hit the new kernel).
Based on platform config settings, these platforms seem to generate NMI/SMI when a vt-d fault happens and there were reports that the resulting SMI causes the system to hang.
Fix it by masking vt-d spec defined errors to platform error reporting logic. VT-d spec related errors are already handled by the VT-d OS code, so need to report the same erorr through other channels.
Signed-off-by Suresh Siddha <[email protected]>Cc [email protected] [v2.6.32+]--- drivers/pci/quirks.c | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+)Index:
tip/drivers/pci/quirks.c
===================================================================drivers/pci/quirks.c +13
-0@@ -2764,6 +2764,26 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_RIDECLARE_PCI_FIXUP_RESUME_EARLY (PCI_VENDOR_ID_RICOH ,PCI_DEVICE_ID_RICOH_R5C832 , ricoh_mmc_fixup_r5c832);#endif /*CONFIG_MMC_RICOH_MMC*/ +#if defined(CONFIG_DMAR ) || defined(CONFIG_INTR_REMAP ) +/* +* This is a quirk for masking vt-d spec defined errors to platform error +* handling logic. With out this, platforms seem to generate NMI/SMI (based +* on the RAS config settings of the platform) when a vt-d fault happens and +* there were reports that the resulting SMI causes system to hang. +* +* VT-d spec related errors are already handled by the VT-d OS code, so no +* need to report the same erorr through other channels. +*/ +static void vtd_mask_spec_errors (struct pci_dev *dev) +{ +u32 word;
drivers/pci/quirks.c DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_RI · line 2789
Quoted 10 lines
drivers/pci/quirks.c +7
-0+ +pci_read_config_dword (dev,0x1AC , &word); + pci_write_config_dword(dev, 0x1AC, word | (1 << 31)); +} +DECLARE_PCI_FIXUP_EARLY (PCI_VENDOR_ID_INTEL ,0x342e , vtd_mask_spec_errors); +DECLARE_PCI_FIXUP_EARLY (PCI_VENDOR_ID_INTEL ,0x3c28 , vtd_mask_spec_errors); +#endif static void pci_do_fixups (struct pci_dev *dev,struct pci_fixup *f,struct pci_fixup *end)
Can we make these registers and bits a bit more self-documenting (i.e. #defines for both, maybe along with other useful bit definitions for this reg)? Also, "error" is misspelled as "erorr" above. :)
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Re: [patch 1/4] vt-d: quirk for masking vtd spec errors to platform error handling logic
Quoted 3 lines · Jesse Barnes
Can we make these registers and bits a bit more self-documenting (i.e. #defines for both, maybe along with other useful bit definitions for this reg)? Also, "error" is misspelled as "erorr" above. :)
Thanks for the review. Appended the updated patch. I haven't used #defines for the pci-id's, as the first one (IOH) is used by several chipsets and the second one is not named yet.
---
From: Suresh Siddha <[email protected]>
Subject: vt-d: quirk for masking vtd spec errors to platform error handling logic
On platforms with Intel 7500 chipset, there were some reports of system hang/NMI's during kexec/kdump in the presence of interrupt-remapping enabled.
During kdump, there is a window where the devices might be still using old kernel's interrupt information, while the kdump kernel is coming up. This can cause vt-d faults as the interrupt configuration from the old kernel map to null IRTE entries in the new kernel etc. (with out interrupt-remapping enabled, we still have the same issue but in this case we will see benign spurious interrupt hit the new kernel).
Based on platform config settings, these platforms seem to generate NMI/SMI when a vt-d fault happens and there were reports that the resulting SMI causes the system to hang.
Fix it by masking vt-d spec defined errors to platform error reporting logic. VT-d spec related errors are already handled by the VT-d OS code, so need to report the same error through other channels.
--- drivers/pci/quirks.c | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+)
drivers/pci/quirks.c +23 -0
@@ -2764,6 +2764,29 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_RIDECLARE_PCI_FIXUP_RESUME_EARLY (PCI_VENDOR_ID_RICOH ,PCI_DEVICE_ID_RICOH_R5C832 , ricoh_mmc_fixup_r5c832);#endif /*CONFIG_MMC_RICOH_MMC*/ +#if defined(CONFIG_DMAR ) || defined(CONFIG_INTR_REMAP ) +#define VTUNCERRMSK_REG 0x1ac +#define VTD_MSK_SPEC_ERRORS (1 << 31) +/* +* This is a quirk for masking vt-d spec defined errors to platform error +* handling logic. With out this, platforms using Intel 7500, 5500 chipsets +* (and the derivative chipsets like X58 etc) seem to generate NMI/SMI (based +* on the RAS config settings of the platform) when a vt-d fault happens. +* The resulting SMI caused the system to hang. +* +* VT-d spec related errors are already handled by the VT-d OS code, so no +* need to report the same error through other channels. +*/ +static void vtd_mask_spec_errors (struct pci_dev *dev) +{ +u32 word; + +pci_read_config_dword (dev,VTUNCERRMSK_REG , &word); +pci_write_config_dword (dev,VTUNCERRMSK_REG , word |VTD_MSK_SPEC_ERRORS ); +} +DECLARE_PCI_FIXUP_EARLY (PCI_VENDOR_ID_INTEL ,0x342e , vtd_mask_spec_errors); +DECLARE_PCI_FIXUP_EARLY (PCI_VENDOR_ID_INTEL ,0x3c28 , vtd_mask_spec_errors); +#endif static void pci_do_fixups (struct pci_dev *dev,struct pci_fixup *f,struct pci_fixup *end)
Re: [patch 1/4] vt-d: quirk for masking vtd spec errors to platform error handling logic
Quoted 8 lines · On Mon, 06 Dec 2010 12:26:30 -0800 Suresh Siddha <[email protected]>…
Quoted 3 lines · Jesse Barnes
Can we make these registers and bits a bit more self-documenting (i.e. #defines for both, maybe along with other useful bit definitions for this reg)? Also, "error" is misspelled as "erorr" above. :)
Thanks for the review. Appended the updated patch. I haven't used #defines for the pci-id's, as the first one (IOH) is used by several chipsets and the second one is not named yet.
Is there a bug # that should be referenced in the commit log? Any tested-bys to add?
Thanks,
Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
Re: [patch 1/4] vt-d: quirk for masking vtd spec errors to platform error handling logic
Quoted 8 lines · Jesse Barnes
Quoted 5 lines · On Mon, 06 Dec 2010 12:26:30 -0800 Suresh Siddha <[email protected]>…
Quoted 3 lines · Jesse Barnes
Can we make these registers and bits a bit more self-documenting (i.e. #defines for both, maybe along with other useful bit definitions for this reg)? Also, "error" is misspelled as "erorr" above. :)
Quoted 6 lines
Quoted 3 lines
Thanks for the review. Appended the updated patch. I haven't used #defines for the pci-id's, as the first one (IOH) is used by several chipsets and the second one is not named yet.
Is there a bug # that should be referenced in the commit log? Any tested-bys to add?
There is no kernel.org bug# but there are multiple bugs with different OSV's. And hence didn't care to mention to the bug #
Please add:
thanks,
suresh
Re: [patch 1/4] vt-d: quirk for masking vtd spec errors to platform error handling logic
Quoted 19 lines · Suresh Siddha
Quoted 8 lines · Jesse Barnes
Quoted 5 lines · On Mon, 06 Dec 2010 12:26:30 -0800 Suresh Siddha <[email protected]>…
Quoted 3 lines · Jesse Barnes
Can we make these registers and bits a bit more self-documenting (i.e. #defines for both, maybe along with other useful bit definitions for this reg)? Also, "error" is misspelled as "erorr" above. :)
Quoted 6 lines
Quoted 3 lines
Thanks for the review. Appended the updated patch. I haven't used #defines for the pci-id's, as the first one (IOH) is used by several chipsets and the second one is not named yet.
Is there a bug # that should be referenced in the commit log? Any tested-bys to add?
There is no kernel.org bug# but there are multiple bugs with different OSV's. And hence didn't care to mention to the bug #
Quoted 7 lines
Please add:
Reported-by Max Asbock <[email protected]>Reported-and-tested-by Takao Indoh <[email protected]>Acked-by Chris Wright <[email protected]>Acked-by Kenji Kaneshige <[email protected]>
I tested the patches on a system with a Tylersburg chipset. I used the patches against the 2.6.37-rc4 kernel and tested kdump. I still see the Vt-d errors but they no longer cause NMIs. It works as expected.
- Max
Takao Indoh
Tested-byRe: [patch 2/4] x86, vtd: fix the vt-d fault handling irq migration in the x2apic mode
In reply to Suresh SiddhaQuoted 11 lines · Suresh Siddha
Quoted 9 lines · Bjorn Helgaas
Quoted 3 lines · Suresh Siddha
(quoted diff) +2
-0+ if (x2apic_mode) + msg.address_hi = MSI_ADDR_BASE_HI | MSI_ADDR_EXT_DEST_ID(dest);
Is it necessary to test
x2apic_modehere? It looks likeMSI_ADDR_EXT_DEST_ID()gives you everything above the low 8 bits of the APIC ID. If those bits are always zero except inx2apic_mode, we might not need the test.
Quoted 1 line
True. Appended the updated patch.
I applied this patch against 2.6.36 and confirmed irq migration of vt-d fault worked.
Thanks,
Takao Indoh
Quoted 39 lines
Quoted 2 lines
Does the ia64
dmar_msi_set_affinity()need the same fix?No.
Quoted 3 lines
Why do we have both
x2apic_enabled()andx2apic_mode? They seem sort of redundant. (Not related to this patch, of course.)BIOS can handover to OS in x2apic mode in some cases.
x2apic_enabled()is used to check for that and it reads the MSR to check the status. Some early portions of the kernel boot will use it.For all others, we should be using
x2apic_mode.thanks,
suresh
---From: Kenji Kaneshige <[email protected]>
Subject: x86, vtd: fix the vt-d fault handling irq migration in the x2apic modeIn x2apic mode, we need to set the upper address register of the fault handling interrupt register of the vt-d hardware. Without this irq migration of the vt-d fault handling interrupt is broken.
Signed-off-by Kenji Kaneshige <[email protected]>Signed-off-by Suresh Siddha <[email protected]>Cc [email protected] [v2.6.32+]---
arch/x86/kernel/apic/io_apic.c| 1 +1 file changed, 1 insertion(+)Index:
tip/arch/x86/kernel/apic/io_apic.c
===================================================================
arch/x86/kernel/apic/io_apic.c dmar_msi_set_affinity(struct irq_data *d · line 3373
Quoted 7 lines
arch/x86/kernel/apic/io_apic.c +1
-0@@ -3367,6 +3367,7 @@ dmar_msi_set_affinity(struct irq_data *d msg.data |=MSI_DATA_VECTOR (cfg->vector); msg.address_lo &= ~MSI_ADDR_DEST_ID_MASK ; msg.address_lo |=MSI_ADDR_DEST_ID (dest); + msg.address_hi =MSI_ADDR_BASE_HI |MSI_ADDR_EXT_DEST_ID (dest);dmar_msi_write (irq, &msg);
---
印藤隆夫(INDOH Takao)
E-Mail : [email protected]
[tip:x86/urgent] x86, vt-d: Quirk for masking vtd spec errors to platform error handling logic
In reply to Suresh SiddhaCommit-ID: 254e42006c893f45bca48f313536fcba12206418 Gitweb: http://git.kernel.org/tip/254e42006c893f45bca48f313536fcba12206418 Author: Suresh Siddha <[email protected]> AuthorDate: Mon, 6 Dec 2010 12:26:30 -0800 Committer: H. Peter Anvin <[email protected]> CommitDate: Mon, 13 Dec 2010 16:51:51 -0800
x86, vt-d: Quirk for masking vtd spec errors to platform error handling logic
On platforms with Intel 7500 chipset, there were some reports of system hang/NMI's during kexec/kdump in the presence of interrupt-remapping enabled.
During kdump, there is a window where the devices might be still using old kernel's interrupt information, while the kdump kernel is coming up. This can cause vt-d faults as the interrupt configuration from the old kernel map to null IRTE entries in the new kernel etc. (with out interrupt-remapping enabled, we still have the same issue but in this case we will see benign spurious interrupt hit the new kernel).
Based on platform config settings, these platforms seem to generate NMI/SMI when a vt-d fault happens and there were reports that the resulting SMI causes the system to hang.
Fix it by masking vt-d spec defined errors to platform error reporting logic. VT-d spec related errors are already handled by the VT-d OS code, so need to report the same error through other channels.
LKML-Reference: <[email protected]>
--- drivers/pci/quirks.c | 23 +++++++++++++++++++++++ 1 files changed, 23 insertions(+), 0 deletions(-)
drivers/pci/quirks.c +23 -0
@@ -2764,6 +2764,29 @@ DECLARE_PCI_FIXUP_EARLY(PCI_VENDOR_ID_RICOH, PCI_DEVICE_ID_RICOH_R5C832, ricoh_mDECLARE_PCI_FIXUP_RESUME_EARLY (PCI_VENDOR_ID_RICOH ,PCI_DEVICE_ID_RICOH_R5C832 , ricoh_mmc_fixup_r5c832);#endif /*CONFIG_MMC_RICOH_MMC*/ +#if defined(CONFIG_DMAR ) || defined(CONFIG_INTR_REMAP ) +#define VTUNCERRMSK_REG 0x1ac +#define VTD_MSK_SPEC_ERRORS (1 << 31) +/* +* This is a quirk for masking vt-d spec defined errors to platform error +* handling logic. With out this, platforms using Intel 7500, 5500 chipsets +* (and the derivative chipsets like X58 etc) seem to generate NMI/SMI (based +* on the RAS config settings of the platform) when a vt-d fault happens. +* The resulting SMI caused the system to hang. +* +* VT-d spec related errors are already handled by the VT-d OS code, so no +* need to report the same error through other channels. +*/ +static void vtd_mask_spec_errors (struct pci_dev *dev) +{ +u32 word; + +pci_read_config_dword (dev,VTUNCERRMSK_REG , &word); +pci_write_config_dword (dev,VTUNCERRMSK_REG , word |VTD_MSK_SPEC_ERRORS ); +} +DECLARE_PCI_FIXUP_EARLY (PCI_VENDOR_ID_INTEL ,0x342e , vtd_mask_spec_errors); +DECLARE_PCI_FIXUP_EARLY (PCI_VENDOR_ID_INTEL ,0x3c28 , vtd_mask_spec_errors); +#endif static void pci_do_fixups (struct pci_dev *dev,struct pci_fixup *f,struct pci_fixup *end)
[tip:x86/urgent] x86, vt-d: Fix the vt-d fault handling irq migration in the x2apic mode
In reply to Suresh SiddhaCommit-ID: 086e8ced65d9bcc4a8e8f1cd39b09640f2883f90 Gitweb: http://git.kernel.org/tip/086e8ced65d9bcc4a8e8f1cd39b09640f2883f90 Author: Kenji Kaneshige <[email protected]> AuthorDate: Wed, 1 Dec 2010 09:40:32 -0800 Committer: H. Peter Anvin <[email protected]> CommitDate: Mon, 13 Dec 2010 16:52:52 -0800
x86, vt-d: Fix the vt-d fault handling irq migration in the x2apic mode
In x2apic mode, we need to set the upper address register of the fault handling interrupt register of the vt-d hardware. Without this irq migration of the vt-d fault handling interrupt is broken.
LKML-Reference: <1291225233.2648.39.camel@sbsiddha-MOBL3>
--- arch/x86/kernel/apic/io_apic.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-)
arch/x86/kernel/apic/io_apic.c +1 -0
@@ -3412,6 +3412,7 @@ dmar_msi_set_affinity(struct irq_data *data, const struct cpumask *mask, msg.data |=MSI_DATA_VECTOR (cfg->vector); msg.address_lo &= ~MSI_ADDR_DEST_ID_MASK ; msg.address_lo |=MSI_ADDR_DEST_ID (dest); + msg.address_hi =MSI_ADDR_BASE_HI |MSI_ADDR_EXT_DEST_ID (dest);dmar_msi_write (irq, &msg);
[tip:x86/urgent] x86: Enable the intr-remap fault handling after local APIC setup
In reply to Suresh SiddhaCommit-ID: 7f7fbf45c6b748074546f7f16b9488ca71de99c1 Gitweb: http://git.kernel.org/tip/7f7fbf45c6b748074546f7f16b9488ca71de99c1 Author: Kenji Kaneshige <[email protected]> AuthorDate: Tue, 30 Nov 2010 22:22:28 -0800 Committer: H. Peter Anvin <[email protected]> CommitDate: Mon, 13 Dec 2010 16:53:32 -0800
x86: Enable the intr-remap fault handling after local APIC setup
Interrupt-remapping gets enabled very early in the boot, as it determines the apic mode that the processor can use. And the current code enables the vt-d fault handling before the setup_local_APIC(). And hence the APIC LDR registers and data structure in the memory may not be initialized. So the vt-d fault handling in logical xapic/x2apic modes were broken.
Fix this by enabling the vt-d fault handling in the end_local_APIC_setup()
A cleaner fix of enabling fault handling while enabling intr-remapping will be addressed for v2.6.38. [ Enabling intr-remapping determines the usage of x2apic mode and the apic mode determines the fault-handling configuration. ]
LKML-Reference: <[email protected]>
--- arch/x86/kernel/apic/apic.c | 8 ++++++++ arch/x86/kernel/apic/probe_64.c | 7 ------- 2 files changed, 8 insertions(+), 7 deletions(-)
arch/x86/kernel/apic/apic.c +8 -0
@@ -1389,6 +1389,14 @@ void __cpuinit end_local_APIC_setup(void)setup_apic_nmi_watchdog (NULL );apic_pm_activate (); + +/* +* Now that local APIC setup is completed for BP, configure the fault +* handling for interrupt remapping. +*/ +if (!smp_processor_id () && intr_remapping_enabled) +enable_drhd_fault_handling (); + }#ifdef CONFIG_X86_X2APIC
arch/x86/kernel/apic/probe_64.c +0 -7
@@ -79,13 +79,6 @@ void __init default_setup_apic_routing(void)/* need to update phys_pkg_id */ apic->phys_pkg_id = apicid_phys_pkg_id; } - -/* -* Now that apic routing model is selected, configure the -* fault handling for intr remapping. -*/ -if (intr_remapping_enabled) -enable_drhd_fault_handling (); }/* Same for both flat and physical. */
[tip:x86/urgent] x86, vt-d: Handle previous faults after enabling fault handling
In reply to Suresh SiddhaCommit-ID: 7f99d946e71e71d484b7543b49e990508e70d0c0 Gitweb: http://git.kernel.org/tip/7f99d946e71e71d484b7543b49e990508e70d0c0 Author: Suresh Siddha <[email protected]> AuthorDate: Tue, 30 Nov 2010 22:22:29 -0800 Committer: H. Peter Anvin <[email protected]> CommitDate: Mon, 13 Dec 2010 16:53:57 -0800
x86, vt-d: Handle previous faults after enabling fault handling
Fault handling is getting enabled after enabling the interrupt-remapping (as the success of interrupt-remapping can affect the apic mode and hence the fault handling mode).
Hence there can potentially be some faults between the window of enabling interrupt-remapping in the vt-d and the fault-handling of the vt-d units.
Handle any previous faults after enabling the vt-d fault handling.
For v2.6.38 cleanup, need to check if we can remove the dmar_fault() in the enable_intr_remapping() and see if we can enable fault handling along with enabling intr-remapping.
LKML-Reference: <[email protected]>
--- drivers/pci/dmar.c | 5 +++++ 1 files changed, 5 insertions(+), 0 deletions(-)
drivers/pci/dmar.c +5 -0
@@ -1417,6 +1417,11 @@ int __init enable_drhd_fault_handling(void) (unsigned long long)drhd->reg_base_addr, ret);return -1 ; } + +/* +* Clear any previous faults. +*/ +dmar_fault (iommu->irq, iommu); }return 0 ;