Unbreak MSI on ATI devices
PATCH 1/1 Unbreak MSI on ATI devices
Hello Jeff,
I'm using second patch below for couple of months to get MSI on all
devices present on my notebook which are MSI capable (except IDE - notebook uses IDE in legacy mode and seems unhappy with transition to native MSI-based mode; maybe I could try do the job with libata now when I switched; and VGA, I do not use dri...). All worked nicely until last sync, when I noticed that my USB devices suddenly stopped working (it took me few weeks as I do not use USB regulary).
After poking around I've found that problem is that at least ATI USB-HCDs apply INTX enable even for MSI, despite warning in the PCI specification that it should apply only to MSI (actually I have feeling that on these USB devices disabling INTX in MSI mode drives their INTA# line active as when ohci1394 module got loaded kernel complained about interrupt being continuously activated for no good reason (TI's 7421 is one of few MSI-incapable devices in my box).
So my question is - what is real reason for disabling INTX when in MSI mode? According to PCI spec it should not be needed, and it hurts at least chips listed below:
00:13.0 0c03: 1002:4374 USB Controller: ATI Technologies Inc IXP SB400 USB Host Controller 00:13.1 0c03: 1002:4375 USB Controller: ATI Technologies Inc IXP SB400 USB Host Controller 00:13.2 0c03: 1002:4373 USB Controller: ATI Technologies Inc IXP SB400 USB2 Host Controller
I believe that these devices from same vendor accept disabling INTX while in MSI fine (I did not notice problems with them even with INTX disabling code in msi.c):
00:14.5 0401: 1002:4370 (rev 02) Multimedia audio controller: ATI Technologies Inc IXP SB400 AC'97 Audio Controller (rev 02) 00:14.6 0703: 1002:4378 (rev 02) Modem: ATI Technologies Inc ATI SB400 - AC'97 Modem Controller (rev 02)
None of devices in the box assert INTX while in MSI even if INTX is enabled.
So I'd like to see first patch below accepted. If there are some devices which require INTX disabling, then apparently decision whether to disable it or no has to be moved to device drivers, or some blacklist/whitelist must be created...
I'm not sure about second one - I have it in my tree for months, but I run that kernel only on hardware mentioned above, so it is probably too dangerous until pci_enable_msi() gets answer whether MSI works or no always right.
Thanks,
Petr Vandrovec
/proc/interrupts after patch. Before patch *hci_hcd:usb* were at zero, IRQ21 was stuck with IRQ count at 10000, and HCD complained about "Unlink after no-IRQ?".
CPU0
0: 140326 local-APIC-edge-fasteio timer
1: 10773 IO-APIC-edge i8042 8: 1 IO-APIC-edge rtc 12: 34498 IO-APIC-edge i8042 14: 34714 IO-APIC-edge libata 15: 82 IO-APIC-edge libata 16: 1 IO-APIC-fasteoi tifm_7xx1, yenta, sdhci:slot0, sdhci:slot1, sdhci:slot2 21: 265 IO-APIC-fasteoi acpi, ohci1394 22: 28973 IO-APIC-fasteoi ipw2200 216: 1 PCI-MSI-edge ATI IXP Modem 217: 1 PCI-MSI-edge ATI IXP 218: 2 PCI-MSI-edge ehci_hcd:usb3 219: 88 PCI-MSI-edge ohci_hcd:usb2 220: 1 PCI-MSI-edge ohci_hcd:usb1 221: 1 PCI-MSI-edge eth0 NMI: 0 LOC: 140293 ERR: 0 MIS: 0
Do not disable INTX in MSI mode. It breaks ATI USB HCDs (both OHCI and EHCI).
drivers/pci/msi.c +1 -1
@@ -256,7 +256,7 @@ static void enable_msi_mode(struct pci_d dev->msix_enabled =1 ; } -pci_intx (dev,0 );/* disable intx */ +pci_intx (dev,1 );/* enable intx, on some devices it affects MSI as well */ }void disable_msi_mode (struct pci_dev *dev,int pos,int type)
Use MSI when available on USB HCDs and on ATI AC'97 hardware.
drivers/usb/core/hcd-pci.c +4 -0
@@ -69,6 +69,7 @@ int usb_hcd_pci_probe (struct pci_dev *dif (pci_enable_device (dev) <0 )return -ENODEV ; +pci_enable_msi (dev); dev->current_state =PCI_D0 ; dev->dev.power.power_state =PMSG_ON ;
@@ -139,6 +140,7 @@ int usb_hcd_pci_probe (struct pci_dev *drelease_region (hcd->rsrc_start, hcd->rsrc_len); err2:usb_put_hcd (hcd); +pci_disable_msi (dev); err1:pci_disable_device (dev);dev_err (&dev->dev,"init %s fail, %d\n" ,pci_name (dev), retval);
@@ -177,6 +179,7 @@ void usb_hcd_pci_remove (struct pci_devrelease_region (hcd->rsrc_start, hcd->rsrc_len); }usb_put_hcd (hcd); +pci_disable_msi (dev);pci_disable_device (dev); }EXPORT_SYMBOL (usb_hcd_pci_remove);
@@ -391,6 +394,7 @@ int usb_hcd_pci_resume (struct pci_dev * "can't re-enable after resume, %d!\n", retval);return retval; } +pci_enable_msi (dev);pci_set_master (dev);pci_restore_state (dev);
sound/pci/atiixp.c +25 -0
@@ -60,6 +60,8 @@ MODULE_PARM_DESC(spdif_aclink, "S/PDIF o/* just for backward compatibility */ static int enable;module_param (enable, bool,0444 ); +static int msi; +module_param (msi, bool,0444 );/*
@@ -279,6 +281,8 @@ struct atiixp {int spdif_over_aclink;/* passed from the module option */ struct mutex open_mutex;/* playback open mutex */ + +int have_msi; };
@@ -1442,6 +1446,11 @@ static int snd_atiixp_suspend(struct pcisnd_atiixp_aclink_down (chip);snd_atiixp_chip_stop (chip); +if (chip->have_msi) { +pci_disable_msi (pci); + }else { +pci_intx (pci,0 ); + }pci_disable_device (pci);pci_save_state (pci);pci_set_power_state (pci,pci_choose_state (pci, state));
@@ -1463,6 +1472,11 @@ static int snd_atiixp_resume(struct pci_return -EIO ; }pci_set_master (pci); +if (chip->have_msi) { +pci_enable_msi (pci); + }else { +pci_intx (pci,1 ); + }snd_atiixp_aclink_reset (chip);snd_atiixp_chip_start (chip);
@@ -1532,6 +1546,11 @@ static int snd_atiixp_free(struct atiixpif (chip->remap_addr)iounmap (chip->remap_addr);pci_release_regions (chip->pci); +if (chip->have_msi) { +pci_disable_msi (chip->pci); + }else { +pci_intx (chip->pci,0 ); + }pci_disable_device (chip->pci);kfree (chip);return 0 ;
@@ -1583,6 +1602,9 @@ static int __devinit snd_atiixp_create(sreturn -EIO ; } +if (msi &&pci_enable_msi (pci) ==0 ) { + chip->have_msi =1 ; + }if (request_irq (pci->irq, snd_atiixp_interrupt,IRQF_DISABLED |IRQF_SHARED , card->shortname, chip)) {snd_printk (KERN_ERR "unable to grab IRQ %d\n" , pci->irq);
@@ -1591,6 +1613,9 @@ static int __devinit snd_atiixp_create(s } chip->irq = pci->irq;pci_set_master (pci); +if (!chip->have_msi) { +pci_intx (pci,1 ); + }synchronize_irq (chip->irq);if ((err =snd_device_new (card,SNDRV_DEV_LOWLEVEL , chip, &ops)) <0 ) {
sound/pci/atiixp_modem.c +25 -1
@@ -54,6 +54,8 @@ MODULE_PARM_DESC(ac97_clock, "AC'97 code/* just for backward compatibility */ static int enable;module_param (enable, bool,0444 ); +static int msi; +module_param (msi, bool,0444 );/*
@@ -257,6 +259,8 @@ struct atiixp_modem {int spdif_over_aclink;/* passed from the module option */ struct mutex open_mutex;/* playback open mutex */ + +int have_msi; };
@@ -1128,6 +1132,11 @@ static int snd_atiixp_suspend(struct pcisnd_atiixp_aclink_down (chip);snd_atiixp_chip_stop (chip); +if (chip->have_msi) { +pci_disable_msi (pci); + }else { +pci_intx (pci,0 ); + }pci_disable_device (pci);pci_save_state (pci);pci_set_power_state (pci,pci_choose_state (pci, state));
@@ -1149,6 +1158,11 @@ static int snd_atiixp_resume(struct pci_return -EIO ; }pci_set_master (pci); +if (chip->have_msi) { +pci_enable_msi (pci); + }else { +pci_intx (pci,1 ); + }snd_atiixp_aclink_reset (chip);snd_atiixp_chip_start (chip);
@@ -1204,6 +1218,11 @@ static int snd_atiixp_free(struct atiixpfree_irq (chip->irq, chip);if (chip->remap_addr)iounmap (chip->remap_addr); +if (chip->have_msi) { +pci_disable_msi (chip->pci); + }else { +pci_intx (chip->pci,0 ); + }pci_release_regions (chip->pci);pci_disable_device (chip->pci);kfree (chip);
@@ -1255,7 +1274,9 @@ static int __devinit snd_atiixp_create(ssnd_atiixp_free (chip);return -EIO ; } - +if (msi &&pci_enable_msi (pci) ==0 ) { + chip->have_msi =1 ; + }if (request_irq (pci->irq, snd_atiixp_interrupt,IRQF_DISABLED |IRQF_SHARED , card->shortname, chip)) {snd_printk (KERN_ERR "unable to grab IRQ %d\n" , pci->irq);
@@ -1264,6 +1285,9 @@ static int __devinit snd_atiixp_create(s } chip->irq = pci->irq;pci_set_master (pci); +if (!chip->have_msi) { +pci_intx (pci,1 ); + }synchronize_irq (chip->irq);if ((err =snd_device_new (card,SNDRV_DEV_LOWLEVEL , chip, &ops)) <0 ) {
Quoted 17 lines · Petr Vandrovec
After poking around I've found that problem is that at least ATI USB-HCDs
apply INTX enable even for MSI, despite warning in the PCI specification that
it should apply only to MSI (actually I have feeling that on these USB devices disabling INTX in MSI mode drives their INTA# line active as when ohci1394 module got loaded kernel complained about interrupt being continuously activated for no good reason (TI's 7421 is one of few MSI-incapable devices in my box).So my question is - what is real reason for disabling INTX when in MSI mode? According to PCI spec it should not be needed, and it hurts at least chips listed below:
Do not disable INTX in MSI mode. It breaks ATI USB HCDs (both OHCI and EHCI).
Signed-off-by Petr Vandrovec <[email protected]>diff -uprdN
linux/drivers/pci/msi.clinux/drivers/pci/msi.c
drivers/pci/msi.c static void enable_msi_mode(struct pci_d · line 260
Quoted 6 lines
drivers/pci/msi.c +1
-1@@ -256,7 +256,7 @@ static void enable_msi_mode(struct pci_d dev->msix_enabled =1 ; } -pci_intx (dev,0 );/* disable intx */ +pci_intx (dev,1 );/* enable intx, on some devices it affects MSI as well */ }
I'm just going to CC Linus, and run ;-)
More seriously. Some other chips choke if you forget to disable INTx, before going into MSI mode.
Thus, turning off one irq source before turning on another is the most logical course of action.
I suppose we'll have to quirk ATI for being dumb...?
Jeff
Quoted 2 lines · Petr Vandrovec
So my question is - what is real reason for disabling INTX when in MSI mode? According to PCI spec it should not be needed.
The PCI spec is at least not clear enough on the matter to keep nVidia from thinking that it's the OS's responsibility to make legacy interrupts not happen, by disabling INTX.
Quoted 1 line
None of devices in the box assert INTX while in MSI even if INTX is enabled.
I've got a forcedeth-driven ethernet card that does, and people have reported that nVidia "Intel HDA" sound does as well.
Quoted 4 lines
So I'd like to see first patch below accepted. If there are some devices which require INTX disabling, then apparently decision whether to disable it or no has to be moved to device drivers, or some blacklist/whitelist must be created...
PCI Express (IIRC) had the pci_intx() calls already, so it's probably actually required by the spec (or at least common implementations) there.
I'd guess that it's more common for hardware to be unhappy with intx enabled than to be unhappy with intx disabled, since the hardware is supposed to not send legacy interrupts.
Quoted 3 lines
I'm not sure about second one - I have it in my tree for months, but I run that kernel only on hardware mentioned above, so it is probably too dangerous until
pci_enable_msi()gets answer whether MSI works or no always right.
I think it'd be better to add an module parameter, like in the later drivers in your patch. Figuring out how to get MSI working whenever it's available isn't going to move forward unless there's an easy way to test it, especially since (according to rumor) Windows doesn't use it at all.
Quoted 3 lines
/proc/interruptsafter patch. Before patch *hci_hcd:usb* were at zero, IRQ21 was stuck with IRQ count at 10000, and HCD complained about "Unlink after no-IRQ?".
Maybe the intx disable is just totally broken for your device? It certainly shouldn't cause the delivery of *more* legacy interrupts, and if it does with MSI enabled, I'd be surprised if it didn't without MSI. My guess is that that device should get a quirk to just leave the INTx disable bit alone (such that pci_intx doesn't do anything, regardless of context).
sound/pci/atiixp.c static int snd_atiixp_suspend(struct pci · line 1453
Quoted 12 lines
diff -uprdN
linux/sound/pci/atiixp.clinux/sound/pci/atiixp.csound/pci/atiixp.c +5
-0@@ -1442,6 +1446,11 @@ static int snd_atiixp_suspend(struct pcisnd_atiixp_aclink_down (chip);snd_atiixp_chip_stop (chip); +if (chip->have_msi) { +pci_disable_msi (pci); + }else { +pci_intx (pci,0 ); + }
This doesn't look right, at least for !chip->have_msi. Or is disabling intx desirable here for non-MSI? I'd guess that devices that freak out if you fiddle with intx are likely to be old, and therefore likely to not support MSI.
sound/pci/atiixp.c static int snd_atiixp_free(struct atiixp · line 1553
Quoted 9 lines
sound/pci/atiixp.c +5
-0@@ -1532,6 +1546,11 @@ static int snd_atiixp_free(struct atiixpif (chip->remap_addr)iounmap (chip->remap_addr);pci_release_regions (chip->pci); +if (chip->have_msi) { +pci_disable_msi (chip->pci); + }else { +pci_intx (chip->pci,0 ); + }
My playing with forcedeth trying to get my system working suggests that the expected intx state for a device with no driver is "not disabled". I think the else clause here would cause the device to not work if you used this driver, unloaded the module, and loaded a version without the patch (or kexeced an older kernel, or soft-rebooted into some operating system without MSI support).
-Daniel
*This .sig left intentionally blank*
Quoted 7 lines
So my question is - what is real reason for disabling INTX when in MSI mode? According to PCI spec it should not be needed, and it hurts at least chips listed below:
00:13.0 0c03: 1002:4374 USB Controller: ATI Technologies Inc IXP SB400 USB Host Controller 00:13.1 0c03: 1002:4375 USB Controller: ATI Technologies Inc IXP SB400 USB Host Controller 00:13.2 0c03: 1002:4373 USB Controller: ATI Technologies Inc IXP SB400 USB2 Host Controller
heh... I'm not gloating or anything... but I am glad that some ASIC designer was careless enough to prove me right when I said going beyond what the PCI spec requires is dangerous.
- R.
Quoted 11 lines · Roland Dreier
Quoted 7 lines
So my question is - what is real reason for disabling INTX when in MSI mode? According to PCI spec it should not be needed, and it hurts at least chips listed below:
00:13.0 0c03: 1002:4374 USB Controller: ATI Technologies Inc IXP SB400 USB Host Controller 00:13.1 0c03: 1002:4375 USB Controller: ATI Technologies Inc IXP SB400 USB Host Controller 00:13.2 0c03: 1002:4373 USB Controller: ATI Technologies Inc IXP SB400 USB2 Host Controller
heh... I'm not gloating or anything... but I am glad that some ASIC designer was careless enough to prove me right when I said going beyond what the PCI spec requires is dangerous.
Hi,
unfortunately it is not everything :-(
I cannot get MSI to work on IDE interface under any circumstances - in legacy mode it always uses IRQ14/15 regardless of whether MSI is enabled or not (that's probably correct), but in native mode as soon as I enable MSI it either does not deliver interrupts at all (definitely not through IRQ14/15, and, if I got routing right, also not through its INTA#), or it delivers them somewhere else than where programmed. As my boot device is connected to this adapter, and it is a notebook, it is not easy to debug what's really going on :-(
00:14.1 0101: 1002:4376 IDE interface: ATI Technologies Inc Standard Dual Channel PCI IDE Controller ATI (prog-if 8f [Master SecP SecO PriP PriO])
Subsystem: Rioworks Unknown device 2043
Flags: bus master, 66MHz, medium devsel, latency 64, IRQ 18
I/O ports at 01f0 [size=8]
I/O ports at 03f4 [size=4]
I/O ports at 0170 [size=8]
I/O ports at 0374 [size=4]
I/O ports at 8410 [size=16]
Capabilities: [70] Message Signalled Interrupts: Mask- 64bit- Queue=0/0 Enable-
Petr
Quoted 11 lines · Petr Vandrovec
Hi,
unfortunately it is not everything :-(I cannot get MSI to work on IDE interface under any circumstances - in legacy mode it always uses IRQ14/15 regardless of whether MSI is enabled or not (that's probably correct), but in native mode as soon as I enable MSI it either does not deliver interrupts at all (definitely not through IRQ14/15, and, if I got routing right, also not through its INTA#), or it delivers them somewhere else than where programmed. As my boot device is connected to this adapter, and it is a notebook, it is not easy to debug what's really going on :-(
Are you doing this with INTx left on or turned off? Have you determined whether turning off INTx does anything useful on these devices when you're not using MSI? (There are only a few places in the kernel which disable INTx, mostly associated with enabling MSI.)
It might be easier to test if you boot off a USB storage device of some sort.
-Daniel
*This .sig left intentionally blank*
Quoted 11 lines · Roland Dreier
Quoted 7 lines
So my question is - what is real reason for disabling INTX when in MSI mode? According to PCI spec it should not be needed, and it hurts at least chips listed below:
00:13.0 0c03: 1002:4374 USB Controller: ATI Technologies Inc IXP SB400 USB Host Controller 00:13.1 0c03: 1002:4375 USB Controller: ATI Technologies Inc IXP SB400 USB Host Controller 00:13.2 0c03: 1002:4373 USB Controller: ATI Technologies Inc IXP SB400 USB2 Host Controller
heh... I'm not gloating or anything... but I am glad that some ASIC designer was careless enough to prove me right when I said going beyond what the PCI spec requires is dangerous.
No more dangerous than expecting exactly following the PCI spec to be sufficient; at least some nVidia devices misbehave if you don't disable INTx when using MSI, while at least some ATI devices misehave if you do disable INTx. The only *safe* thing is to ignore the PCI spec and match the behavior of Windows. In this case, that's just don't use MSI yet.
Of course, this should be relatively easy to handle with quirks, especially if it's predictable which hardware bug you get from the vendor id.
-Daniel
*This .sig left intentionally blank*