PCI: Add Broadcom 4331 reset quirk to prevent IRQ storm

Should also fix https://bugzilla.kernel.org/show_bug.cgi?id=111781 ? Given that this is a serious bug that can corrupt filesystems it would be good to see the fix in stable too.

Wow, this is horrible.

This patch should help a bit. But it's probably hard to workaround that DMA issue completely. There still is a chance of corruption, as soon as the kernel allocates and uses some memory that the DMA controller still uses from boot-up before the fixup is applied.

Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/b43-dev/attachments/20160401/1265c561/attachment.sig>

Hi Chris,

I cannot reproduce this particular issue on my MBP9,1 even though it is architecturally very similar to your MBP10,2. I tested it with "iommu=force intel_iommu=on", blacklisted b43 and stressed the machine a bit with kernel compiles. No issues.

Have you tested my patch? As Michael Büsch has pointed out correctly, it should mitigate that issue but will not solve it completely. You write in the bugzilla entry that you've seen the issue as early as 0.7 seconds into the boot. That's not a very precise method to measure boot progress but on my machine the card is reset around 1.2 seconds, so likely too late. It's possible to execute the PCI quirk slightly earlier (in the header fixup section rather than the late fixup section) but that wouldn't make a significant difference.

If you use the method described in Matthew Garrett's blog post, i.e. boot with "memmap=0xc0$0x1568a0", does the issue go away? If so, we could probably add a quirk in arch/x86/kernel/e820.c to mark that particular area reserved, or generally mark all EFI boot services areas as reserved on Macs. This memory could then be marked usable again once the card has been reset.

As to why the issue doesn't occur on my machine, my guess is that the wireless card only writes packets to memory if it is associated with a base station. Do you have an open WLAN around? Try moving out of its reach and test if the issue goes away. Alternatively, did you enter Wifi credentials in OS X? If so it has saved the credentials to EFI variables. To verify if it has, use this on OS X:

/usr/sbin/nvram 36C28AB5-6566-4C50-9EBD-CBB920F83843:current-network /usr/sbin/nvram 36C28AB5-6566-4C50-9EBD-CBB920F83843:preferred-networks /usr/sbin/nvram 36C28AB5-6566-4C50-9EBD-CBB920F83843:preferred-count

On Linux, invoke "mount -t efivarfs none /sys/firmware/efi/efivars" and delete the variables (use chattr -i first if you're on Linux 4.5+). Test if the issue goes away.

I have neither an open WLAN nearby nor Wifi credentials in my EFI variables, so the wireless card cannot associate with a base station in the pre-OS phase and that's probably the reason why the issue doesn't occur on my machine.

Best regards,

Lukas

I think you will have to stress the wireless, not the kernel. Enable iommu and let the wireless receive packets that go through the hw filters. If you have a calm network and nobody sends data to you, the card won't write anything to DMA. (This all depends on how the firmware configured the filters).

Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/b43-dev/attachments/20160402/751f8a10/attachment.sig>

Perhaps it can be hardware other than the Broadcom that causes this problem?

I do see an irq 17 problem on my macbook, but I thought grub is supposed to stop the boardcom wireless?

Thank-you for drawing my attention to this issue - I didn't know about this issue and that " IRQ 17 nobody cared" isue has been annoying a lot.
I have a MacBook10,1 :
dmesg | grep MacBook
DMI: Apple Inc. MacBookPro10,1/Mac-C3EC7CD22292981F, BIOS MBP101.88Z.00EE.B00.1205101839 05/10/2012

Investigating grub2 git://git.savannah.gnu.org/grub.git I see this patch rev 9d34bb8 which says it disables Broadcom wireless hardware on Apples:

* commit 9d34bb8

| Author: Matthew Garrett <[email protected]>
| Date:   Thu May 3 17:26:55 2012 +0200
|
|       Suspend broadcom cards in order to stop their DMA.
|
|       * grub-core/Makefile.am (KERNEL_HEADER_FILES): Add pci.h on x86 EFI.
|       * grub-core/Makefile.core.def (kernel): Add pci.c on x86 EFI.
|       (pci): Don't build on x86 EFI.
|       * grub-core/bus/pci.c (grub_pci_find_capability): New function.
|       * grub-core/kern/efi/mm.c (stop_broadcom) [__i386__ || __x86_64__]:
|       New function.
|       (grub_efi_finish_boot_services) [__i386__ || __x86_64__]: Call
|       stop_broadcom if running on EFI.
|       * include/grub/pci.h (GRUB_PCI_CLASS_NETWORK): New enum value.
|       (GRUB_PCI_CAP_POWER_MANAGEMENT): Likewise.
|       (GRUB_PCI_VENDOR_BROADCOM): Likewise.
|       (grub_pci_find_capability): New proto.
|
|       Also-By: Vladimir Serbinenko <[email protected]>
|
| M     ChangeLog
| M     grub-core/Makefile.am
| M     grub-core/Makefile.core.def
| M     grub-core/bus/pci.c
| M     grub-core/kern/efi/mm.c
| M     include/grub/pci.h

But I run debian grub2-common 2.02~beta2-22+deb8u1 which has this fix and I *still* get this irq issue

....

[  608.242849] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called
with disabled ep ffff88008a19df48
[  608.242851] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called
with disabled ep ffff88008a19df90
[  608.254975] irq 17: nobody cared (try booting with the "irqpoll" option)
[  608.254979] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C O
3.16.0-4-amd64 #1 Debian 3.16.7-ckt20-1+deb8u4
[  608.254981] Hardware name: Apple Inc.
MacBookPro10,1/Mac-C3EC7CD22292981F, BIOS
MBP101.88Z.00EE.B00.1205101839 05/10/2012
[  608.254985]  ffff88045a85bac4 ffffffff8150dcff ffff88045a85ba00
ffffffff810bd3ad
[  608.254987]  ffff88045a85ba00 0000000000000011 0000000000000000
ffffffff810bd8d1
[  608.254989]  0000000000000000 0000000000000000 0000000000000011
0000000000000000
[  608.254990] Call Trace:
[  608.254999]  <IRQ>  [<ffffffff8150dcff>] ? dump_stack+0x41/0x51
[  608.255006]  [<ffffffff810bd3ad>] ? __report_bad_irq+0x2d/0xc0
[  608.255010]  [<ffffffff810bd8d1>] ? note_interrupt+0x241/0x290
[  608.255013]  [<ffffffff810bb0f1>] ? handle_irq_event_percpu+0xa1/0x190
[  608.255017]  [<ffffffff810bb218>] ? handle_irq_event+0x38/0x60
[  608.255020]  [<ffffffff810be413>] ? handle_fasteoi_irq+0x83/0x150
[  608.255025]  [<ffffffff810150cd>] ? handle_irq+0x1d/0x30
[  608.255029]  [<ffffffff81516cd9>] ? do_IRQ+0x49/0xe0
[  608.255033]  [<ffffffff81514b2d>] ? common_interrupt+0x6d/0x6d
[  608.255037]  <EOI>  [<ffffffff8108aded>] ?
__hrtimer_start_range_ns+0x1cd/0x390
[  608.255041]  [<ffffffff813dfd42>] ? cpuidle_enter_state+0x52/0xc0
[  608.255044]  [<ffffffff813dfd38>] ? cpuidle_enter_state+0x48/0xc0
[  608.255047]  [<ffffffff810a80d8>] ? cpu_startup_entry+0x2f8/0x400
[  608.255051]  [<ffffffff81903076>] ? start_kernel+0x497/0x4a2
[  608.255054]  [<ffffffff81902a04>] ? set_init_arg+0x4e/0x4e
[  608.255057]  [<ffffffff81902120>] ? early_idt_handler_array+0x120/0x120
[  608.255060]  [<ffffffff8190271f>] ? x86_64_start_kernel+0x14d/0x15c
[  608.255061] handlers:
[  608.255088] [<ffffffffa0645730>] azx_interrupt [snd_hda_controller]
[  608.255089] Disabling IRQ #17
[  608.354706] usb 1-2: reset high-speed USB device number 3 using xhci_hcd

...

Hi Andrew,

Thanks for the pointer to the grub2 commit, I wasn't aware of that.

The commit puts the wireless card in power state D3hot but that doesn't stop it from sending interrupts. I have just tested that. So it's perfectly plausible that you're still seeing spurious interrupts despite using grub. Please test the patch I've posted, the spurious interrupts should disappear. If you "cat /proc/interrupts", you should then only see a few hundred interrupts on IRQ 17. Without the patch it should be in the 100000+ range.

Best regards,

Lukas

Sorry but testing the patch shows no difference.

I have just compiled debian jessie kernel 3.16.7-ckt25 and booted it and hibernated it twice, then did the same with your patch applied. There appeared to be no difference

On first boot I didn't get the nobody card disabling problem but after each hibernate I got the problem. But I did get 51130 IRQ 17 interrupts on the first boot but after the hibernate restore each time I got 100000 extra interrupts in /proc/interrupts and the irq 17:
nobody cared message.

I could not see any difference with or with out the patch. I boot with grub-efi using the linux/initrd commands

So perhaps the hibernate-restore needs the fix?

Andrew

Further testing Broadcom 4331 reset quirk to prevent IRQ storm patch testing reveals that:
1. quirk is run on initial boot up and this time appears to have
vastly reduced the interrupts (only 81 this time):
cat /proc/interrupts| grep 17
17: 81 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel

2. But it is apparently *NOT* run after a suspend/resume and we get
the problem:
17: 100084 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel

Rebooting a further nine times shows the low number (below 100) only happens around 1/3 of the times:
boot #2
17: 38706 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel
boot #3
17: 87 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel
LOC: 2494 2031 2094 1831 1157 1171
1573 1271 Local timer interrupts
boot #4
17: 50616 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel
boot#5
17: 26454 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel
boot#6
17: 34440 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel
boot#7
17: 79 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel
boot#8
17: 84 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel
boot#9
17: 37054 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel
boot#10
17: 24648 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel

Is there an easy setpci command to stop this we can add to grub?

Presently I have a grub work around for black screen as described here:
http://askubuntu.com/questions/264247/proprietary-nvidia-drivers-with-efi-on-mac-to-prevent-overheating/613573#613573

which basically involves adding a grub scriptlet to enable PCI-E bus mastering on graphics cards:

In /etc/grub.d/01_enable_vga.conf:

setpci -s "00:01.0" 3e.b=8
setpci -s "01:00.0" 04.b=7

Can we do some similar magic setpci commands to disable 04:00.0 which is my BCM4331

lspci | grep 4331
04:00.0 Network controller: Broadcom Corporation BCM4331 802.11a/b/g/n (rev 02)

....

Thanks for any suggestions

Andrew

Hi Andrew,

thank you for the extensive testing.

Something in the ballpark of 81 interrupt requests is fine.

The kernel will print the error message about spurious interrupts and switch to polling at 100000 requests. But even 20000 is way too much. This just means that b43 loaded quickly enough to stop the interrupts before the kernel limit of 100000 was reached, but the wireless card wasn't reset early on as it should have been.

It looks like the patch didn't work at all on your machine for some reason. Do you see a message "cannot iomap device, IRQ storm ahead" in dmesg?

Included below is a slightly different patch which is based on the 3.16.7-ckt tree (so it should apply cleanly to your kernel version). This version prints a message when the wireless card is reset. Please verify that you see the message "b43 quirk: resetting controller" in dmesg with this patch. This version of the patch also resets the card a bit earlier in the boot process than the previous version did. Please let me know if this improves the situation on your machine.

You might also want to try a newer kernel version, 3.16 is very old and I'm not sure if there have been any changes since which might negatively impact proper functioning of the patch. There are also official 4.4 kernels available for Jessie. Canonical ends support for the 3.16.y-ckt series anyway this month.

I think you need to upgrade to a newer version of the Nvidia driver, I have just verified that 352.63 will set the busmaster bit if it's cleared.

I think grub is also able to write to mmio space of a PCI device, which is what's necessary to reset the wireless card, so theoretically yes, but the right place to fix this is in the kernel since not everyone is using grub.

Best regards,

Lukas

-- >8 --
Subject: [PATCH] PCI: Add Broadcom 4331 reset quirk to prevent IRQ storm

Broadcom 4331 wireless cards built into Apple Macs unleash an IRQ storm on boot until they are reset, causing spurious interrupts if the IRQ is shared. Apparently the EFI bootloader enables the device and does not disable it before passing control to the OS. The bootloader contains a driver for the wireless card which allows it to phone home to Cupertino. This is used for Internet Recovery (download and install OS X images) and probably also for Back to My Mac (remote access, RFC 6281) and to discover stolen hardware.

The issue is most pronounced on 2011 and 2012 MacBook Pros where the IRQ is shared with 3 other devices (Light Ridge Thunderbolt controller, SDXC reader, HDA card on discrete GPU). As soon as an interrupt handler is installed for one of these devices, the ensuing storm of spurious IRQs causes the kernel to disable the IRQ and switch to polling. This lasts until the b43 driver loads and resets the device.

Loading the b43 driver first is not always an option, in particular with the Light Ridge Thunderbolt controller: The PCI hotplug IRQ handler gets installed early on because it is built in, unlike b43 which is usually a module.

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=79301
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=895951
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1009819
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1149632

---
 drivers/bcma/bcma_private.h |  2 --
 drivers/pci/quirks.c        | 28 ++++++++++++++++++++++++++++
 include/linux/bcma/bcma.h   |  1 +
 3 files changed, 29 insertions(+), 2 deletions(-)

index 09b632a..7ded994 100644
@@ -8,8 +8,6 @@
 #include <linux/bcma/bcma.h>
 #include <linux/delay.h>
 
-#define BCMA_CORE_SIZE		0x1000
-
 #define bcma_err(bus, fmt, ...) \
 	pr_err("bus%d: " fmt, (bus)->num, ##__VA_ARGS__)
 #define bcma_warn(bus, fmt, ...) \

index 1f5ea24..5a33924 100644
@@ -25,6 +25,8 @@
 #include <linux/sched.h>
 #include <linux/ktime.h>
 #include <linux/mm.h>
+#include <linux/bcma/bcma.h>
+#include <linux/bcma/bcma_regs.h>
 #include <asm/dma.h>	/* isa_dma_bridge_buggy */
 #include "pci.h"
 

@@ -3082,6 +3084,32 @@ static void quirk_no_bus_reset(struct pci_dev *dev)
  */
 DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_ATHEROS, 0x0030, quirk_no_bus_reset);
 
+/*
+ * Broadcom 4331 wireless cards built into Apple Macs unleash an IRQ storm
+ * on boot until they are reset, causing spurious interrupts if the IRQ is
+ * shared. Apparently the EFI bootloader enables the device to phone home
+ * to Cupertino and does not disable it before passing control to the OS.
+ */
+static void quirk_apple_b43_reset(struct pci_dev *dev)
+{
+	void __iomem *mmio;
+
+	if (!dmi_match(DMI_SYS_VENDOR, "Apple Inc.") || !dev->bus->self ||
+	    pci_pcie_type(dev->bus->self) != PCI_EXP_TYPE_ROOT_PORT)
+		return;
+
+	mmio = pci_iomap(dev, 0, 0);
+	if (!mmio) {
+		dev_err(&dev->dev, "b43 quirk: cannot iomap device, IRQ storm ahead\n");
+		return;
+	}
+	dev_info(&dev->dev, "b43 quirk: resetting controller\n");
+	iowrite32(BCMA_RESET_CTL_RESET,
+		  mmio + (1 * BCMA_CORE_SIZE) + BCMA_RESET_CTL);
+	pci_iounmap(dev, mmio);
+}
+DECLARE_PCI_FIXUP_HEADER(PCI_VENDOR_ID_BROADCOM, 0x4331, quirk_apple_b43_reset);
+
 static void pci_do_fixups(struct pci_dev *dev, struct pci_fixup *f,
 			  struct pci_fixup *end)
 {

index 0b3bb16..b19c7ed 100644
@@ -153,6 +153,7 @@ struct bcma_host_ops {
 #define BCMA_CORE_DEFAULT		0xFFF
 
 #define BCMA_MAX_NR_CORES		16
+#define BCMA_CORE_SIZE			0x1000
 
 /* Chip IDs of PCIe devices */
 #define BCMA_CHIP_ID_BCM4313	0x4313

1.8.5.2 (Apple Git-48)

Thank-you very much for your comments in your reply.

Actually the patch did work - I confirmed it was run and the iomap call was successful by adding a pr_info() after the pci_iomap() success branch.

The only time I am getting the IRQ 17 nobody cared message is on suspend / resume. A fresh boot always had below the 100k interrupt threshold level.

I tried your new patch and the number is even lower < 30,000 over two boots.

BUT on suspend resume again 126856.

Have you any insights on fixing suspend to disk / resume paths which presumably face the same issue of being passed live hardware on boot up?

Result from two reboots with my 3.16 kernel and your new patch

Three full boots (all below 30k interrupts):
17: 23978 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel
17: 30088 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel
17: 26853 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel

dmesg output showing quirk running
dmesg | grep -C 5 quirk

[    3.270315] pci 0000:00:1c.0: PCI bridge to [bus 03]
[    3.270323] pci 0000:00:1c.0:   bridge window [mem 0xc1a00000-0xc1afffff]
[    3.270331] pci 0000:00:1c.0:   bridge window [mem
0xc1800000-0xc18fffff 64bit pref]
[    3.270463] pci 0000:04:00.0: [14e4:4331] type 00 class 0x028000
[    3.270495] pci 0000:04:00.0: reg 0x10: [mem 0xc1900000-0xc1903fff 64bit]
[    3.270574] pci 0000:04:00.0: b43 quirk: resetting controller
[    3.270711] pci 0000:04:00.0: supports D1 D2
[    3.270712] pci 0000:04:00.0: PME# supported from D0 D3hot D3cold
[    3.270759] pci 0000:04:00.0: System wakeup disabled by ACPI
[    3.278239] pci 0000:00:1c.1: PCI bridge to [bus 04]
[    3.278251] pci 0000:00:1c.1:   bridge window [mem 0xc1900000-0xc19fffff]

Output after resume. Note: Some times it looks it can happen on the suspend to disk? But a new one is always present after the resume.

17: 126856 0 0 0 0 0
0 0 IO-APIC-fasteoi snd_hda_intel

[   53.404157] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called
with disabled ep ffff88045d495540
[   53.468249] irq 17: nobody cared (try booting with the "irqpoll" option)
[   53.468253] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G         C O
3.16.7-ckt25-3.16-bcm4331-patch2 #7
[   53.468254] Hardware name: Apple Inc.
MacBookPro10,1/Mac-C3EC7CD22292981F, BIOS
MBP101.88Z.00EE.B00.1205101839 05/10/2012
[   53.468259]  0000000000000000 ffffffff81520370 ffff88045a8a8c00
ffff88045a8a8cc4
[   53.468262]  ffffffff810bfe7d ffff88045a8a8c00 0000000000000000
0000000000000011
[   53.468264]  ffffffff810c022f 0000000000000000 0000000000000011
0000000000000000
[   53.468265] Call Trace:
[   53.468275]  <IRQ>  [<ffffffff81520370>] ? dump_stack+0x5d/0x78
[   53.468282]  [<ffffffff810bfe7d>] ? __report_bad_irq+0x2d/0xd0
[   53.468286]  [<ffffffff810c022f>] ? note_interrupt+0x25f/0x2b0
[   53.468290]  [<ffffffff810bd9c1>] ? handle_irq_event_percpu+0x121/0x190
[   53.468294]  [<ffffffff810bda68>] ? handle_irq_event+0x38/0x50
[   53.468296]  [<ffffffff810c0d5f>] ? handle_fasteoi_irq+0x7f/0x150
[   53.468302]  [<ffffffff810153ad>] ? handle_irq+0x1d/0x30
[   53.468307]  [<ffffffff81529118>] ? do_IRQ+0x48/0xe0
[   53.468311]  [<ffffffff81526f6d>] ? common_interrupt+0x6d/0x6d
[   53.468317]  <EOI>  [<ffffffff813ef54c>] ? cpuidle_enter_state+0x4c/0xc0
[   53.468320]  [<ffffffff813ef542>] ? cpuidle_enter_state+0x42/0xc0
[   53.468323]  [<ffffffff810aaa6a>] ? cpu_startup_entry+0x33a/0x460
[   53.468326]  [<ffffffff81911f34>] ? start_kernel+0x473/0x47b
[   53.468331]  [<ffffffff81911120>] ? early_idt_handler_array+0x120/0x120
[   53.468335]  [<ffffffff81911608>] ? x86_64_start_kernel+0x14d/0x15c
[   53.468336] handlers:
[   53.468367] [<ffffffffa02ce740>] azx_interrupt [snd_hda_controller]
[   53.468368] Disabling IRQ #17
[   53.513740] usb 3-1: reset high-speed USB device number 2 using xhci_hcd
[   53.633633] usb 1-1.1: reset high-speed USB device number 3 using ehci-pci
[   53.633646] usb 2-1.8: reset high-speed USB device number 3 using ehci-pci

Sorry for the old kernel - I want to run debian stable rather than
hand buit kernels so my other packages.

I don't see any newer kernels when I do
apt-cache search "^linux-source"

so perhaps I have to add backports or testing into my repository list? If you think it is worth it I can do that.

What other boot loaders do people use on a MacBook beside grub?

I think the setpci commands in grub might fix the problem for me for suspend/resume as well as boot. Can you can easily point me to how to translate the numbers from your patch:

Would it be:

setpci -s "04:00.0" 1800.l=1

Do you have another pointer to where to fix the suspend resume?

Thanks very much again

Andrew

Hi Andrew,

Hm, that's still way too much. You should have < 100 interrupts on IRQ 17 on *every* boot, anything else isn't satisfactory. I'm working on a patch that runs before EFI ExitBootServices() but will need a little longer to find an optimal solution.

Suspend to disk involves two kernels, the boot kernel and the image kernel. Perhaps these two are not the same on your machine and you've only compiled the patch into one of them?

Best regards,

Lukas

Hi Andrew,

The reason why my previous patches didn't work on your machine is because you're using grub, and grub contains a patch by Matthew Garrett which puts the wireless card into power state D3hot. The card's mmio space isn't accessible in D3hot. Included below is a new patch which transitions the card to D0 before resetting it. Please let me know if this fixes the issue for you.

@Chris Bainbridge: Please test this as well, this is no longer a plain vanilla PCI quirk but an early quirk, it should prevent any kind of memory corruption by DMAed packets.

Best regards,

Lukas

-- >8 --
Subject: [PATCH] x86: Add early quirk to reset Apple AirPort card
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

The EFI firmware on Macs contains a full-fledged network stack for downloading OS X images from osrecovery.apple.com. Unfortunately on Macs introduced 2011 and 2012, EFI brings up the Broadcom 4331 wireless card on every boot and leaves it enabled even after ExitBootServices has been called. The card continues to assert its IRQ, causing spurious interrupts if the IRQ is shared. It also corrupts memory by DMAing received packets.

The issue seems to be constrained to the Broadcom 4331. Chris Milsted has verified that the newer Broadcom 4360 built into the MacBookPro11,3 (2013/2014) does not exhibit this behaviour. The chances that Apple will ever supply a firmware fix for the older machines appear to be zero.

The solution is to reset the card on boot by writing to a reset bit in its mmio space. This must be done as an early quirk and not as a plain vanilla PCI quirk to prevent memory corruption by DMAed packets:
Matthew Garrett found out in 2012 that the packets are written to EfiBootServicesData memory (http://mjg59.dreamwidth.org/11235.html).
This type of memory is made available to the page allocator by efi_free_boot_services(). Plain vanilla PCI quirks run much later, in subsys initcall level. In-between a time window would be open for memory corruption. Random crashes occuring in this time window and attributed to DMAed packets have indeed been observed in the wild by Chris Bainbridge.

When Matthew Garrett analyzed the memory corruption issue in 2012, he sought to fix it with a grub quirk which transitions the card to D3hot:
http://git.savannah.gnu.org/cgit/grub.git/commit/?id=9d34bb85da56

While this may prevent DMAed packets, it does not cure the spurious interrupts emanating from the card. Unfortunately the card's mmio space is inaccessible during D3hot, so to reset it, we have to undo the effect of Matthew's patch and transition the card back to D0 if the system was booted with grub.

Since commit 8659c406ade3 ("x86: only scan the root bus in early PCI quirks"), early quirks can only be applied to devices on the root bus. However the Broadcom 4331 card is located on a secondary bus behind a PCIe root port. This commit therefore reintroduces scanning of secondary buses. The primary motivation of 8659c406ade3 was to prevent application of the nvidia_bugs() quirk on secondary buses. Amend the quirk to open code this requirement.

A secondary motivation was to speed up PCI scanning. The algorithm used prior to 8659c406ade3 was particularly time consuming because it scanned buses 0 to 31 brute force. The recursive algorithm used by this commit only scans buses that are actually reachable from the root bus and should thus be a bit faster. If this algorithm is found to significantly impact boot time, it would be possible to limit its recursion depth:
The Apple AirPort quirk applies at depth 1, all others at depth 0, so the bus need not be scanned deeper than that for now. An alternative approach would be to continue scanning only the root bus, and apply the AirPort quirk to the root ports 8086:1c12, 8086:1e12 and 8086:1e16. Apple always positioned the Broadcom 4331 behind one of these three ports (see model list below). The quirk would then check presence of the Broadcom 4331 in slot 0 below the root port and do its deed.

Note that the quirk takes a few shortcuts to reduce the amount of code:
The size of BAR 0 and the location of the PM capability is identical on all affected machines and therefore hardcoded. Only the address of BAR 0 differs between models. Also, it is assumed that the BCMA core currently mapped is the 802.11 core. The EFI driver seems to always take care of this.

Michael BÃŒsch and Bjorn Helgaas contributed feedback towards finding the best solution to this problem.

The following should be a comprehensive list of affected models:

    iMac13,1        2012  21.5"       [Root Port 00:1c.3 = 8086:1e16]
    iMac13,2        2012  27"         [Root Port 00:1c.3 = 8086:1e16]
    Macmini5,1      2011  i5 2.3 GHz  [Root Port 00:1c.1 = 8086:1c12]
    Macmini5,2      2011  i5 2.5 GHz  [Root Port 00:1c.1 = 8086:1c12]
    Macmini5,3      2011  i7 2.0 GHz  [Root Port 00:1c.1 = 8086:1c12]
    Macmini6,1      2012  i5 2.5 GHz  [Root Port 00:1c.1 = 8086:1e12]
    Macmini6,2      2012  i7 2.3 GHz  [Root Port 00:1c.1 = 8086:1e12]
    MacBookPro8,1   2011  13"         [Root Port 00:1c.1 = 8086:1c12]
    MacBookPro8,2   2011  15"         [Root Port 00:1c.1 = 8086:1c12]
    MacBookPro8,3   2011  17"         [Root Port 00:1c.1 = 8086:1c12]
    MacBookPro9,1   2012  15"         [Root Port 00:1c.1 = 8086:1e12]
    MacBookPro9,2   2012  13"         [Root Port 00:1c.1 = 8086:1e12]
    MacBookPro10,1  2012  15"         [Root Port 00:1c.1 = 8086:1e12]
    MacBookPro10,2  2012  13"         [Root Port 00:1c.1 = 8086:1e12]

For posterity, spurious interrupts caused by the Broadcom 4331 wireless card resulted in splats like this (stacktrace omitted):

    irq 17: nobody cared (try booting with the "irqpoll" option)
    handlers:
    [<ffffffff81374370>] pcie_isr
    [<ffffffffc0704550>] sdhci_irq [sdhci] threaded [<ffffffffc07013c0>] sdhci_thread_irq [sdhci]
    [<ffffffffc0a0b960>] azx_interrupt [snd_hda_codec]
    Disabling IRQ #17

Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=79301
Bugzilla: https://bugzilla.kernel.org/show_bug.cgi?id=111781
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=728916
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=895951#c16
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1009819
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1098621
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1149632#c5
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1279130
Bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=1332732

---
 arch/x86/kernel/early-quirks.c | 88 +++++++++++++++++++++++++++++++++++-------
 drivers/bcma/bcma_private.h    |  2 -
 include/linux/bcma/bcma.h      |  1 +
 3 files changed, 76 insertions(+), 15 deletions(-)

index db9a675..8aaab75 100644
@@ -11,7 +11,11 @@
 
 #include <linux/pci.h>
 #include <linux/acpi.h>
+#include <linux/delay.h>
+#include <linux/dmi.h>
 #include <linux/pci_ids.h>
+#include <linux/bcma/bcma.h>
+#include <linux/bcma/bcma_regs.h>
 #include <drm/i915_drm.h>
 #include <asm/pci-direct.h>
 #include <asm/dma.h>

@@ -21,6 +25,7 @@
 #include <asm/iommu.h>
 #include <asm/gart.h>
 #include <asm/irq_remapping.h>
+#include <asm/early_ioremap.h>
 
 static void __init fix_hypertransport_config(int num, int slot, int func)
 {

@@ -76,6 +81,13 @@ static void __init nvidia_bugs(int num, int slot, int func)
 #ifdef CONFIG_ACPI
 #ifdef CONFIG_X86_IO_APIC
 	/*
+	 * Only applies to Nvidia root ports (bus 0) and not to
+	 * Nvidia graphics cards on secondary buses with PCI ports.
+	 */
+	if (num)
+		return;
+
+	/*
 	 * All timer overrides on Nvidia are
 	 * wrong unless HPET is enabled.
 	 * Unfortunately that's not true on many Asus boards.

@@ -589,6 +601,47 @@ static void __init force_disable_hpet(int num, int slot, int func)
 #endif
 }
 
+#define BCM4331_MMIO_SIZE	16384
+#define BCM4331_PM_CAP		0x40
+
+static void __init apple_airport_reset(int bus, int slot, int func)
+{
+	void __iomem *mmio;
+	u16 pmcsr;
+	u64 addr;
+
+	if (!dmi_match(DMI_SYS_VENDOR, "Apple Inc."))
+		return;
+
+	/* Card may have been put into PCI_D3hot by grub quirk */
+	pmcsr = read_pci_config_16(bus, slot, func,
+				   BCM4331_PM_CAP + PCI_PM_CTRL);
+	if ((pmcsr & PCI_PM_CTRL_STATE_MASK) != PCI_D0) {
+		pmcsr &= ~PCI_PM_CTRL_STATE_MASK;
+		write_pci_config_16(bus, slot, func,
+				    BCM4331_PM_CAP + PCI_PM_CTRL, pmcsr);
+		mdelay(10);
+		pmcsr = read_pci_config_16(bus, slot, func,
+					   BCM4331_PM_CAP + PCI_PM_CTRL);
+		if ((pmcsr & PCI_PM_CTRL_STATE_MASK) != PCI_D0) {
+			printk(KERN_ERR "Cannot power up Apple AirPort card\n");
+			return;
+		}
+	}
+
+	addr  =      read_pci_config(bus, slot, func, PCI_BASE_ADDRESS_0);
+	addr |= (u64)read_pci_config(bus, slot, func, PCI_BASE_ADDRESS_1) << 32;
+	addr &= PCI_BASE_ADDRESS_MEM_MASK;
+	mmio = early_ioremap(addr, BCM4331_MMIO_SIZE);
+	if (!mmio) {
+		printk(KERN_ERR "Cannot iomap Apple AirPort card\n");
+		return;
+	}
+	printk(KERN_INFO "Resetting Apple AirPort card\n");
+	iowrite32(BCMA_RESET_CTL_RESET,
+		  mmio + (1 * BCMA_CORE_SIZE) + BCMA_RESET_CTL);
+	early_iounmap(mmio, BCM4331_MMIO_SIZE);
+}
 
 #define QFLAG_APPLY_ONCE 	0x1
 #define QFLAG_APPLIED		0x2

@@ -602,12 +655,6 @@ struct chipset {
 	void (*f)(int num, int slot, int func);
 };
 
-/*
- * Only works for devices on the root bus. If you add any devices
- * not on bus 0 readd another loop level in early_quirks(). But
- * be careful because at least the Nvidia quirk here relies on
- * only matching on bus 0.
- */
 static struct chipset early_qrk[] __initdata = {
 	{ PCI_VENDOR_ID_NVIDIA, PCI_ANY_ID,
 	  PCI_CLASS_BRIDGE_PCI, PCI_ANY_ID, QFLAG_APPLY_ONCE, nvidia_bugs },

@@ -637,9 +684,13 @@ static struct chipset early_qrk[] __initdata = {
 	 */
 	{ PCI_VENDOR_ID_INTEL, 0x0f00,
 		PCI_CLASS_BRIDGE_HOST, PCI_ANY_ID, 0, force_disable_hpet},
+	{ PCI_VENDOR_ID_BROADCOM, 0x4331,
+	  PCI_CLASS_NETWORK_OTHER, PCI_ANY_ID, 0, apple_airport_reset},
 	{}
 };
 
+static void __init early_pci_scan_bus(int bus);
+
 /**
  * check_dev_quirk - apply early quirks to a given PCI device
  * @num: bus number

@@ -648,7 +699,7 @@ static struct chipset early_qrk[] __initdata = {
  *
  * Check the vendor & device ID against the early quirks table.
  *
- * If the device is single function, let early_quirks() know so we don't
+ * If the device is single function, let early_pci_scan_bus() know so we don't
  * poke at this device again.
  */
 static int __init check_dev_quirk(int num, int slot, int func)

@@ -657,6 +708,7 @@ static int __init check_dev_quirk(int num, int slot, int func)
 	u16 vendor;
 	u16 device;
 	u8 type;
+	u8 sec;
 	int i;
 
 	class = read_pci_config_16(num, slot, func, PCI_CLASS_DEVICE);

@@ -684,25 +736,35 @@ static int __init check_dev_quirk(int num, int slot, int func)
 
 	type = read_pci_config_byte(num, slot, func,
 				    PCI_HEADER_TYPE);
+
+	if ((type & 0x7f) == PCI_HEADER_TYPE_BRIDGE) {
+		sec = read_pci_config_byte(num, slot, func, PCI_SECONDARY_BUS);
+		early_pci_scan_bus(sec);
+	}
+
 	if (!(type & 0x80))
 		return -1;
 
 	return 0;
 }
 
-void __init early_quirks(void)
+static void __init early_pci_scan_bus(int bus)
 {
 	int slot, func;
 
-	if (!early_pci_allowed())
-		return;
-
 	/* Poor man's PCI discovery */
-	/* Only scan the root bus */
 	for (slot = 0; slot < 32; slot++)
 		for (func = 0; func < 8; func++) {
 			/* Only probe function 0 on single fn devices */
-			if (check_dev_quirk(0, slot, func))
+			if (check_dev_quirk(bus, slot, func))
 				break;
 		}
 }
+
+void __init early_quirks(void)
+{
+	if (!early_pci_allowed())
+		return;
+
+	early_pci_scan_bus(0);
+}

index 38f1567..71df8f2 100644
@@ -8,8 +8,6 @@
 #include <linux/bcma/bcma.h>
 #include <linux/delay.h>
 
-#define BCMA_CORE_SIZE		0x1000
-
 #define bcma_err(bus, fmt, ...) \
 	pr_err("bus%d: " fmt, (bus)->num, ##__VA_ARGS__)
 #define bcma_warn(bus, fmt, ...) \

index 3feb1b2..14cd6f7 100644
@@ -156,6 +156,7 @@ struct bcma_host_ops {
 #define BCMA_CORE_DEFAULT		0xFFF
 
 #define BCMA_MAX_NR_CORES		16
+#define BCMA_CORE_SIZE			0x1000
 
 /* Chip IDs of PCIe devices */
 #define BCMA_CHIP_ID_BCM4313	0x4313

2.8.1

The grub patch is in Debian but didn't prevent the issue on my setup, presumably because of the Ubuntu "linuxefi" fallback that Matthew previously posted about.

I tested a "no b43" kernel for 10 boots with and without the patch.

Without the patch: 7 boots had potential memory corruption (caught by the IOMMU) and all had IRQ 17 disabled.

With the patch: no memory corruption or disabled IRQ 17.

With b43 built in to the kernel wifi worked as expected.

So the patch looks good to me. Thanks for the fix.