Skip to contentSkip to conversation

New: Infinite loop on bootup while handling DMAR

Bugme-new In review Andrew Morton 8 participants 14 messages Aug 19, 2009 – Aug 31, 2009 1 message not in the archive PCI

lore
Share & tools
lore
#1 lore raw
In reply to a message not in the archive

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 3 lines · On Tue, 18 Aug 2009 14:54:22 GMT [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=14003

Summary: Infinite loop on bootup while handling DMAR

That's a box-killing post-2.6.30 regression.

Quoted 63 lines
           Product: Drivers
           Version: 2.5
    Kernel Version: 2.6.31-rc6
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: PCI
        AssignedTo: [email protected]
        ReportedBy: [email protected]
        Regression: Yes

Booting 64bit 2.6.31-rc6 on a hp xw4600 results in an infinite loop of

DMAR:[DMA READ] Request device [ff:1f.7] fault addr fffffffffffff000 DMAR:[fault reason 255] Unknown

even with iommu=soft (2.6.30 could boot with iommu=soft, but would result in the same error with hardware iommu enabled).

2.6.30 reports this about the DMAR ACPI tables:

ACPI: DMAR 00000000defc247f 00158 (v01 COMPAQ BEARLX38 00000001 00000000)
DMAR:Host address width 36
DMAR:DRHD (flags: 0x00000000)base: 0x00000000fed90000 DMAR:DRHD (flags: 0x00000001)base: 0x00000000fed93000 DMAR:RMRR base: 0x00000000defd0000 end: 0x00000000defd0fff DMAR:RMRR base: 0x00000000defd1000 end: 0x00000000defd1fff DMAR:RMRR base: 0x00000000defd2000 end: 0x00000000defd2fff DMAR:RMRR base: 0x00000000defd3000 end: 0x00000000defd3fff DMAR:RMRR base: 0x00000000defd4000 end: 0x00000000defd4fff DMAR:RMRR base: 0x00000000defd5000 end: 0x00000000defd5fff DMAR:RMRR base: 0x00000000defd6000 end: 0x00000000defd6fff DMAR:RMRR base: 0x00000000defd7000 end: 0x00000000defd7fff Not all IO-APIC's listed under remapping hardware

Device ff:1f.7 doesn't actually exist, lspci -n says

00:00.0 0600: 8086:29e0
00:01.0 0604: 8086:29e1
00:1a.0 0c03: 8086:2937 (rev 02)
00:1a.1 0c03: 8086:2938 (rev 02)
00:1a.2 0c03: 8086:2939 (rev 02)
00:1a.7 0c03: 8086:293c (rev 02)
00:1b.0 0403: 8086:293e (rev 02)
00:1c.0 0604: 8086:2940 (rev 02)
00:1c.4 0604: 8086:2948 (rev 02)
00:1c.5 0604: 8086:294a (rev 02)
00:1d.0 0c03: 8086:2934 (rev 02)
00:1d.1 0c03: 8086:2935 (rev 02)
00:1d.2 0c03: 8086:2936 (rev 02)
00:1d.7 0c03: 8086:293a (rev 02)
00:1e.0 0604: 8086:244e (rev 92)
00:1f.0 0601: 8086:2916 (rev 02)
00:1f.2 0106: 8086:2922 (rev 02)
01:00.0 0300: 1002:7187
01:00.1 0380: 1002:71a7
10:0b.0 0c00: 11c1:5811 (rev 61)
3f:00.0 0200: 14e4:167b (rev 02)

#2 lore raw
Quoted 16 lines · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 3 lines · On Tue, 18 Aug 2009 14:54:22 GMT [email protected] wrote:

http://bugzilla.kernel.org/show_bug.cgi?id=14003

Summary: Infinite loop on bootup while handling DMAR

That's a box-killing post-2.6.30 regression.

Quoted 5 lines

Booting 64bit 2.6.31-rc6 on a hp xw4600 results in an infinite loop of

DMAR:[DMA READ] Request device [ff:1f.7] fault addr fffffffffffff000 DMAR:[fault reason 255] Unknown

I think David has a fix queued up for this already. Please check
http://git.infradead.org/iommu-2.6.git/commit/0815565a

thanks,
suresh

#3 lore raw
In reply to Andrew Morton
Quoted 4 lines · Andrew Morton
Quoted 2 lines

http://bugzilla.kernel.org/show_bug.cgi?id=14003
Summary: Infinite loop on bootup while handling DMAR

That's a box-killing post-2.6.30 regression.

It's a BIOS bug -- the user's BIOS is written by idiots who obviously shipped it without any QA whatsoever.

As far as I can tell, 2.6.30 aborted early because of a _different_ BIOS bug, but now we cope with that particular bug and we fall over the next bug. Or just come across them in a different order.

The IOMMU on this board can _never_ have worked. Just disable it.

Or use a board with open source firmware available, and this kind of crap won't happen. (At least if it does, you'll be able to fix it).

Signature · 1 line
dwmw2
#4 lore raw

PATCH 1/1 intel-iommu: Work around yet another BIOS bug

Yet another reason why trusting this stuff to the BIOS was a bad idea.

There now seem to be a bunch of BIOSes which report an IOMMU at a physical address which just returns all ones. (Perhaps only when VT-d is actually _disabled_ in the BIOS?)

Well done, Dell and HP -- although I didn't think it was possible, you have _further_ lowered my already-unprintable opinion of closed source BIOSes and BIOS engineers.

This patch makes the kernel detect this particularly brokenness and abort early -- and fixes up the missing iounmap in the error paths which I noticed while I was poking at it.

This should fix kernel.org bug #14003, which was being called a 'regression' -- I think because the IOMMU code used to trip over _another_ BIOS bug earlier than this one, and that one _did_ cause it to abort.

---
 drivers/pci/dmar.c |   22 ++++++++++++++++++----
 1 files changed, 18 insertions(+), 4 deletions(-)
drivers/pci/dmar.c +18 -4
index 7b287cb..380b60e 100644
@@ -632,20 +632,31 @@ int alloc_iommu(struct dmar_drhd_unit *drhd)
 	iommu->cap = dmar_readq(iommu->reg + DMAR_CAP_REG);
 	iommu->ecap = dmar_readq(iommu->reg + DMAR_ECAP_REG);
 
+	if (iommu->cap == (uint64_t)-1 && iommu->ecap == (uint64_t)-1) {
+		/* Promote an attitude of violence to a BIOS engineer today */
+		WARN(1, "Your BIOS is broken; DMAR reported at address %llx returns all ones!\n"
+		     "BIOS vendor: %s; Ver: %s; Product Version: %s\n",
+		     drhd->reg_base_addr,
+		     dmi_get_system_info(DMI_BIOS_VENDOR),
+		     dmi_get_system_info(DMI_BIOS_VERSION),
+		     dmi_get_system_info(DMI_PRODUCT_VERSION));
+		goto err_unmap;
+	}
+
 #ifdef CONFIG_DMAR
 	agaw = iommu_calculate_agaw(iommu);
 	if (agaw < 0) {
 		printk(KERN_ERR
 		       "Cannot get a valid agaw for iommu (seq_id = %d)\n",
 		       iommu->seq_id);
-		goto error;
+		goto err_unmap;
 	}
 	msagaw = iommu_calculate_max_sagaw(iommu);
 	if (msagaw < 0) {
 		printk(KERN_ERR
 			"Cannot get a valid max agaw for iommu (seq_id = %d)\n",
 			iommu->seq_id);
-		goto error;
+		goto err_unmap;
 	}
 #endif
 	iommu->agaw = agaw;
@@ -665,7 +676,7 @@ int alloc_iommu(struct dmar_drhd_unit *drhd)
 	}
 
 	ver = readl(iommu->reg + DMAR_VER_REG);
-	pr_debug("IOMMU %llx: ver %d:%d cap %llx ecap %llx\n",
+	pr_info("IOMMU %llx: ver %d:%d cap %llx ecap %llx\n",
 		(unsigned long long)drhd->reg_base_addr,
 		DMAR_VER_MAJOR(ver), DMAR_VER_MINOR(ver),
 		(unsigned long long)iommu->cap,
@@ -675,7 +686,10 @@ int alloc_iommu(struct dmar_drhd_unit *drhd)
 
 	drhd->iommu = iommu;
 	return 0;
-error:
+
+ err_unmap:
+	iounmap(iommu->reg);
+ error:
 	kfree(iommu);
 	return -1;
 }
Signature · 5 lines
1.6.2.5


-- 
dwmw2
#14 lore raw
In reply to David Woodhouse
Quoted 7 lines · David Woodhouse
Quoted 3 lines · Andrew Morton
Quoted 2 lines

http://bugzilla.kernel.org/show_bug.cgi?id=14003
Summary: Infinite loop on bootup while handling DMAR

That's a box-killing post-2.6.30 regression.

It's a BIOS bug -- the user's BIOS is written by idiots who obviously shipped it without any QA whatsoever.

Matt may be wondering why he was addressed, since the bugzilla entry mentions only a bug in HP's BIOS.

I'm experiencing the same bug on a newly bought Dell Optiplex 760 with BIOS version A03, as explained in my mail in lkml, <[email protected]>, subject "[regression, bisected] fails to boot on Dell Optiplex 760 with VT-d enabled".

Thanks,
Faidon

#5 lore raw

Re: [PATCH] intel-iommu: Work around yet another BIOS bug

In reply to David Woodhouse

drivers/pci/dmar.c int alloc_iommu(struct dmar_drhd_unit *drhd) · line 639

Quoted 5 lines · On Thu, 20 Aug 2009 09:01:58 +0100 David Woodhouse <[email protected]> wrote:
drivers/pci/dmar.c +5 -0
+	if (iommu->cap == (uint64_t)-1 && iommu->ecap == (uint64_t)-1) {
+		/* Promote an attitude of violence to a BIOS engineer today */
+		WARN(1, "Your BIOS is broken; DMAR reported at address %llx returns all ones!\n"
+		     "BIOS vendor: %s; Ver: %s; Product Version: %s\n",
+		     drhd->reg_base_addr,

Printing a u64 with %ll will (still) generate a warning on four architectures.

Quoted 5 lines
drivers/pci/dmar.c +5 -0
+		     dmi_get_system_info(DMI_BIOS_VENDOR),
+		     dmi_get_system_info(DMI_BIOS_VERSION),
+		     dmi_get_system_info(DMI_PRODUCT_VERSION));
+		goto err_unmap;
+	}
#6 lore raw

Re: [PATCH] intel-iommu: Work around yet another BIOS bug

Quoted 2 lines · Andrew Morton

Printing a u64 with %ll will (still) generate a warning on four architectures.

Still that many? I thought we'd fixed them all already.

Certainly we fixed IA64, which is the last one I cared about in the context of this particular code.

Signature · 2 lines
David Woodhouse                            Open Source Technology Centre
[email protected]                              Intel Corporation
#8 lore raw

Re: [PATCH] intel-iommu: Work around yet another BIOS bug

In reply to Andrew Morton
Quoted 9 lines · Andrew Morton

drivers/pci/dmar.c int alloc_iommu(struct dmar_drhd_unit *drhd) · line 639

Quoted 5 lines · On Thu, 20 Aug 2009 09:01:58 +0100 David Woodhouse <[email protected]> wrote:
drivers/pci/dmar.c +5 -0
+	if (iommu->cap == (uint64_t)-1 && iommu->ecap == (uint64_t)-1) {
+		/* Promote an attitude of violence to a BIOS engineer today */
+		WARN(1, "Your BIOS is broken; DMAR reported at address %llx returns all ones!\n"
+		     "BIOS vendor: %s; Ver: %s; Product Version: %s\n",
+		     drhd->reg_base_addr,

Printing a u64 with %ll will (still) generate a warning on four architectures.

We've got them all now.

$ grep -l int-l64 arch/*/include/asm/types.h
arch/alpha/include/asm/types.h
arch/ia64/include/asm/types.h
arch/mips/include/asm/types.h
arch/powerpc/include/asm/types.h
$ grep -l int-ll64 $(grep -l int-l64 arch/*/include/asm/types.h)
arch/alpha/include/asm/types.h
arch/ia64/include/asm/types.h
arch/mips/include/asm/types.h
arch/powerpc/include/asm/types.h

ie all architectures which use int-l64 only do so for the benefit of userspace, and use int-ll64 within the kernel. I did check this by hand too ;-)

Signature · 4 lines
Matthew Wilcox				Intel Open Source Technology Centre
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."
#7 lore raw

Re: [PATCH] intel-iommu: Work around yet another BIOS bug

In reply to David Woodhouse
Quoted 5 lines · On Thu, 20 Aug 2009 12:14:37 +0100 David Woodhouse <[email protected]> wrote:
Quoted 2 lines · Andrew Morton

Printing a u64 with %ll will (still) generate a warning on four architectures.

Still that many? I thought we'd fixed them all already.

You're right, they all seem to be fixed. Nobody tells me nuttin. Whee.

#9 lore raw

Re: [PATCH] intel-iommu: Work around yet another BIOS bug

In reply to David Woodhouse
Quoted 3 lines · David Woodhouse

+       if (iommu->cap == (uint64_t)-1 && iommu->ecap == (uint64_t)-1) {
+               /* Promote an attitude of violence to a BIOS engineer today */
+               WARN(1, "Your BIOS is broken; DMAR reported at address %llx returns all ones!\n"

Think about changing this to a warning that "Your IOMMU appears to be disabled." All ones is, after all, the traditional hint that the device is turned off.

#10 lore raw

Re: [PATCH] intel-iommu: Work around yet another BIOS bug

Quoted 3 lines · Ray Lee

Think about changing this to a warning that "Your IOMMU appears to be disabled." All ones is, after all, the traditional hint that the device is turned off.

Hints are all very well, but the BIOS provided an ACPI table explicitly telling us that there was an active IOMMU at this location.

Signature · 2 lines
David Woodhouse                            Open Source Technology Centre
[email protected]                              Intel Corporation
#11 lore raw

Re: [PATCH] intel-iommu: Work around yet another BIOS bug

Quoted 7 lines · David Woodhouse
Quoted 3 lines · Ray Lee

Think about changing this to a warning that "Your IOMMU appears to be disabled." All ones is, after all, the traditional hint that the device is turned off.

Hints are all very well, but the BIOS provided an ACPI table explicitly telling us that there was an active IOMMU at this location.

Could that be to reserve address space that the "disabled" IOMMU might still be responding to?

Ie the BIOS hides the control registers so the OS won't talk to the device but the IOMMU might still attempt to lookup certain address ranges.

I'm more inclined to believe it's sloppiness on the part of the BIOS writers but thought this might be an alternative explanation.

thanks,
grant

#12 lore raw

Re: [PATCH] intel-iommu: Work around yet another BIOS bug

Quoted 10 lines · Grant Grundler

Could that be to reserve address space that the "disabled" IOMMU might still be responding to?

Ie the BIOS hides the control registers so the OS won't talk to the device but the IOMMU might still attempt to lookup certain address ranges.

I'm more inclined to believe it's sloppiness on the part of the BIOS writers but thought this might be an alternative explanation.

Nah, this is just the normal story: "We smoked too much crack and fried our brains, and we don't do any QA on the crap we write because that would leave fewer hours in the day for us to service our crack habit".

We _really_ need open source firmware.

Or at _least_ firmware written by competent engineers -- but I think we've all fairly much given up on that happening by now?

Signature · 1 line
dwmw2
#13 lore raw

Re: [PATCH] intel-iommu: Work around yet another BIOS bug

Quoted 6 lines · David Woodhouse
Quoted 3 lines · Grant Grundler

I'm more inclined to believe it's sloppiness on the part of the BIOS writers but thought this might be an alternative explanation.

Quoted 9 lines

Nah, this is just the normal story: "We smoked too much crack and fried our brains, and we don't do any QA on the crap we write because that would leave fewer hours in the day for us to service our crack habit".

We _really_ need open source firmware.

Or at _least_ firmware written by competent engineers -- but I think we've all fairly much given up on that happening by now?

coreboot is an open source (GPLv2) x86 lightweight firmware written in C (with optional BIOS and EFI compatibility modules if anyone cares). While it does not support all current x86 chipsets, it still does support quite a few mainboards.
The developers are friendly and try to fix any reported bugs. There are two big obstacles until x86 world domination, though: A dozen core developers are simply not enough for the huge task (there are more open specs than developers can keep up with), and some chipset vendors don't open their specs. Anyone interested is invited to contribute and make the world a better place.

Oh, and it supports packing a Linux kernel in the mainboard flash ROM for diskless/networkless booting (that coreboot+Linux combination was called LinuxBIOS some time ago).

More information at http://www.coreboot.org/

Regards,
Carl-Daniel