Skip to contentSkip to conversation

dmesg: PCI interrupts are no longer routed automatically.........

DISCUSSION aryix 4 participants 11 messages Dec 30, 2004 – Jan 11, 2005

lore
Share & tools
lore
#1 lore raw

Do you have a device that doesn't work unless you specify "pci=routeirq"?

If all your devices work, you can ignore the note in dmesg.

#2 lore raw
Quoted 6 lines · Bjorn Helgaas

Do you have a device that doesn't work unless you specify "pci=routeirq"?

If all your devices work, you can ignore the note in dmesg.

-

I note that pci_enable_device() needs to be executed __before__ the IRQ is obtained on 2.6.10, otherwise you get the wrong IRQ (IRQ10 on this system)B.

This doesn't seem to be correct since the IRQ connection was set by the BIOS and certainly shouldn't be changed. On this system, interrupts that were not shared on 2.4.n and early 2.6.n end up being shared... See IRQ18 below.

CPU0

  0:    1135520    IO-APIC-edge  timer
  1:       3203    IO-APIC-edge  i8042
  7:          0    IO-APIC-edge  parport0
  8:          1    IO-APIC-edge  rtc
  9:          0   IO-APIC-level  acpi
 12:         66    IO-APIC-edge  i8042
 14:      10637    IO-APIC-edge  ide0
 16:          0   IO-APIC-level  uhci_hcd, uhci_hcd
 18:         81   IO-APIC-level  libata, uhci_hcd, Analogic Corp DLB
 19:          0   IO-APIC-level  uhci_hcd
 20:        801   IO-APIC-level  eth0
 21:        982   IO-APIC-level  aic7xxx
 23:          0   IO-APIC-level  ehci_hcd
NMI:          0 LOC:    1135484 ERR:          0
MIS:          0

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.

#3 lore raw
Quoted 3 lines · linux-os

I note that pci_enable_device() needs to be executed __before__ the IRQ is obtained on 2.6.10, otherwise you get the wrong IRQ (IRQ10 on this system)B.

Right.

Quoted 4 lines

This doesn't seem to be correct since the IRQ connection was set by the BIOS and certainly shouldn't be changed. On this system, interrupts that were not shared on 2.4.n and early 2.6.n end up being shared... See IRQ18 below.

It's not that we are changing the IRQ, it's just that we now do the ACPI routing at the time the driver claims the device, rather than doing all the ACPI routing at boot-time. The old strategy messed with IRQs that might never be used (which broke some things), and also didn't work for hot-plug PCI root bridges.

Back to my original question, do you have a device that only works when you use "pci=routeirq"? If so, what is it and what driver does it use?

#4 lore raw
Quoted 20 lines · Bjorn Helgaas
Quoted 3 lines · linux-os

I note that pci_enable_device() needs to be executed __before__ the IRQ is obtained on 2.6.10, otherwise you get the wrong IRQ (IRQ10 on this system)B.

Right.

Quoted 4 lines

This doesn't seem to be correct since the IRQ connection was set by the BIOS and certainly shouldn't be changed. On this system, interrupts that were not shared on 2.4.n and early 2.6.n end up being shared... See IRQ18 below.

It's not that we are changing the IRQ, it's just that we now do the ACPI routing at the time the driver claims the device, rather than doing all the ACPI routing at boot-time. The old strategy messed with IRQs that might never be used (which broke some things), and also didn't work for hot-plug PCI root bridges.

Quoted 4 lines

Back to my original question, do you have a device that only works when you use "pci=routeirq"? If so, what is it and what driver does it use?

No.
I modified our drivers to accommodate the new scheme. The problem is that I don't feel warm and fuzzy about enabling a device __before__ an IRQ handler is in place to handle the IRQ. For instance, Level interrupts from PLX chips on the PCI bus can (read do) generate interrupts when some of the BARS are being configured. Once you get an unhandled interrupt, you are dead because there's nothing to reset the line.

The new scheme requires that the device be enabled to get the correct IRQ. If you did this work, maybe it would be much better if you added a pci_set_irq() call instead of combining your routing with enabling the device?

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.

#5 lore raw
Quoted 5 lines · linux-os

For instance, Level interrupts from PLX chips on the PCI bus can (read do) generate interrupts when some of the BARS are being configured. Once you get an unhandled interrupt, you are dead because there's nothing to reset the line.

Why not unconditionally clear all interrupts after configuring the chip?

David Vrabel

#6 lore raw
Quoted 11 lines · David Vrabel
Quoted 5 lines · linux-os

For instance, Level interrupts from PLX chips on the PCI bus can (read do) generate interrupts when some of the BARS are being configured. Once you get an unhandled interrupt, you are dead because there's nothing to reset the line.

Why not unconditionally clear all interrupts after configuring the chip?

David Vrabel

You can't configure the chip until the BARS are set up for access. You _must_ know what the interrupt line is, before you touch any registers so that any "waiting" interrupt gets handled, i.e., cleared. Otherwise, the code that inspects the PCI bus, looking for a device to claim, finds the device, then (in order to make its IRQ correct) enables it ... BAM that's all she wrote. No messages, no nothing, a halted machine because the IRQ line is permanently TRUE with no code in place to reset it.

The temporary work-around is....

	pci_enable_device(pdev);
	save_irq = pdev->irq;
	pci_disable_device(pdev);	// Turn back off.

	init_bars(....);
	request_irq(save_irq,...)	// Put ISR in place

	pci_write_config_byte(pdev, PCI_CACHE_LINE_SIZE, 0x08);
	pci_write_config_byte(pdev, PCI_LATENCY_TIMER, 0x40);
	pci_set_dma_mask(pdev, 0x00000000ffffffffULL);
	pci_set_drvdata(pdev, NULL);
	pci_set_power_state(pdev, 0);
	pci_set_master(pdev);
	pci_set_mwi(pdev);
	pci_write_config_dword(pdev, PCI_COMMAND, PCI_CONFIG);

	.... configure chip-specific stuff, clear interrupts, etc.
	pci_enable_device(dev);

Now, the temporary work-around is a MACRO called ROUTE_IRQ(). If the guy who changed the PCI API adds a seperate callable procedure to do this routing without actually enabling the chip, I will substitute.

This work-round is a new Linux-2.6.10 bug-hack. It was never required before

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.

#7 lore raw
Quoted 9 lines · linux-os

The temporary work-around is....

(quoted diff) +0 -0
 	pci_enable_device(pdev);
 	save_irq = pdev->irq;
 	pci_disable_device(pdev);	// Turn back off.
 
 	init_bars(....);
 	request_irq(save_irq,...)	// Put ISR in place
 
 	pci_write_config_byte(pdev, PCI_CACHE_LINE_SIZE, 0x08);

Review

Quoted 10 lines
(quoted diff) +0 -0
 	pci_write_config_byte(pdev, PCI_LATENCY_TIMER, 0x40);
 	pci_set_dma_mask(pdev, 0x00000000ffffffffULL);
 	pci_set_drvdata(pdev, NULL);
 	pci_set_power_state(pdev, 0);
 	pci_set_master(pdev);
 	pci_set_mwi(pdev);
 	pci_write_config_dword(pdev, PCI_COMMAND, PCI_CONFIG);
 
 	.... configure chip-specific stuff, clear interrupts, etc.
 	pci_enable_device(dev);

So prior to 2.6.10, you did something like this?

	request_irq(pdev->irq, ...);
	pci_write_config_byte(pdev, PCI_CACHE_LINE_SIZE, 0x08);
	pci_write_config_byte(pdev, PCI_LATENCY_TIMER, 0x40);
	...
	pci_enable_device(pdev);

What sort of interrupts does the device generate before it's enabled? I can't find anything in the PCI spec that actually prohibits interrupts before the driver starts up the device, but it does seem strange.

You wouldn't want your ISR mucking around with a half-initialized device, so does it have to check a "device_configured" flag or something?

#8 lore raw
Quoted 21 lines · Bjorn Helgaas
Quoted 9 lines · linux-os

The temporary work-around is....

(quoted diff) +0 -0
 	pci_enable_device(pdev);
 	save_irq = pdev->irq;
 	pci_disable_device(pdev);	// Turn back off.
 
 	init_bars(....);
 	request_irq(save_irq,...)	// Put ISR in place
 
 	pci_write_config_byte(pdev, PCI_CACHE_LINE_SIZE, 0x08);

Review

Quoted 10 lines
(quoted diff) +0 -0
 	pci_write_config_byte(pdev, PCI_LATENCY_TIMER, 0x40);
 	pci_set_dma_mask(pdev, 0x00000000ffffffffULL);
 	pci_set_drvdata(pdev, NULL);
 	pci_set_power_state(pdev, 0);
 	pci_set_master(pdev);
 	pci_set_mwi(pdev);
 	pci_write_config_dword(pdev, PCI_COMMAND, PCI_CONFIG);
 
 	.... configure chip-specific stuff, clear interrupts, etc.
 	pci_enable_device(dev);
Quoted 8 lines

So prior to 2.6.10, you did something like this?

	request_irq(pdev->irq, ...);
	pci_write_config_byte(pdev, PCI_CACHE_LINE_SIZE, 0x08);
	pci_write_config_byte(pdev, PCI_LATENCY_TIMER, 0x40);
	...
	pci_enable_device(pdev);

Yes, exactly.

Quoted 5 lines

What sort of interrupts does the device generate before it's enabled? I can't find anything in the PCI spec that actually prohibits interrupts before the driver starts up the device, but it does seem strange.

The problem is that the PLX-9656BA INTCSR is not in configuration space, but runtime registers off a BAR. The interrupt source can be from a PLD that hasn't even had its microcode loaded yet!

FYI, the PLX or similar clone is the bus interface chip for many busmastering PCI boards.

Quoted 4 lines

You wouldn't want your ISR mucking around with a half-initialized device, so does it have to check a "device_configured" flag or something?

Yes. If the device isn't configured, the ISR reads all the INTCSR bits, then writes 0 to the register to prevent anything else.

If the PLX had been reset, then the INTCSR bits would all be masked off. However, reset is really only guaranteed from power OFF on some motherboards, in particuar the ones with so-called "hot-swap" capabilites fail. There is a software reset that, in fact, even reloads its serial EEPROM. However, the BAR needs to be accessible for this to be used.

So it would be wonderful if the correct IRQ could be made available before the chip could generate an interrupt.

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.

#9 lore raw
Quoted 8 lines · linux-os

The problem is that the PLX-9656BA INTCSR is not in configuration space, but runtime registers off a BAR. The interrupt source can be from a PLD that hasn't even had its microcode loaded yet!

FYI, the PLX or similar clone is the bus interface chip for many busmastering PCI boards.

Quoted 6 lines
Quoted 3 lines

You wouldn't want your ISR mucking around with a half-initialized device, so does it have to check a "device_configured" flag or something?

Yes. If the device isn't configured, the ISR reads all the INTCSR bits, then writes 0 to the register to prevent anything else.

The PLX might be a common device, but it sounds like this particular issue depends on the design of the rest of the board. And presumably, nobody who cared about performance would design a board with this property, right? I mean, to add a test in the ISR for a condition that exists only for a few milliseconds at driver startup-time seems sub-optimal.

Quoted 9 lines

If the PLX had been reset, then the INTCSR bits would all be masked off. However, reset is really only guaranteed from power OFF on some motherboards, in particuar the ones with so-called "hot-swap" capabilites fail. There is a software reset that, in fact, even reloads its serial EEPROM. However, the BAR needs to be accessible for this to be used.

So it would be wonderful if the correct IRQ could be made available before the chip could generate an interrupt.

If we exposed a new pcibios_route_irq() (to hide the arch- specific nature of IRQ routing via ACPI or other information), could you do what you need in a pci_fixup_early quirk?

#10 lore raw
Quoted 16 lines · Bjorn Helgaas
Quoted 8 lines · linux-os

The problem is that the PLX-9656BA INTCSR is not in configuration space, but runtime registers off a BAR. The interrupt source can be from a PLD that hasn't even had its microcode loaded yet!

FYI, the PLX or similar clone is the bus interface chip for many busmastering PCI boards.

Quoted 6 lines
Quoted 3 lines

You wouldn't want your ISR mucking around with a half-initialized device, so does it have to check a "device_configured" flag or something?

Yes. If the device isn't configured, the ISR reads all the INTCSR bits, then writes 0 to the register to prevent anything else.

Quoted 7 lines

The PLX might be a common device, but it sounds like this particular issue depends on the design of the rest of the board. And presumably, nobody who cared about performance would design a board with this property, right? I mean, to add a test in the ISR for a condition that exists only for a few milliseconds at driver startup-time seems sub-optimal.

I'm not so sure about that. There are many hardware registers to be read in the ISR before the ISR can "decide" what it is supposed to do. I'm not sure that the few hundred nanoseconds required to read/test a variable is all the consequential.

What I do know is that the board must recover from all known errors because you can't irradiate a patient with X-Rays and then decide to throw away the data.

Also, like most bus-mastering devices, you only get an interrupt to report something, basically one interrupt per transmitted packet and one per received. There are a few diagnostic thing that make more, but they are not common events. The 'packets' are 10 megabytes in length so the interrupts are really hidden in the noise. Even Ethernet drivers with their 1500 max byte packets that use this interface chip should really check for a "real" interrupt now that hot-swap is commonplace.

Quoted 7 lines
Quoted 7 lines

If the PLX had been reset, then the INTCSR bits would all be masked off. However, reset is really only guaranteed from power OFF on some motherboards, in particuar the ones with so-called "hot-swap" capabilites fail. There is a software reset that, in fact, even reloads its serial EEPROM. However, the BAR needs to be accessible for this to be used.

Quoted 7 lines
Quoted 2 lines

So it would be wonderful if the correct IRQ could be made available before the chip could generate an interrupt.

If we exposed a new pcibios_route_irq() (to hide the arch- specific nature of IRQ routing via ACPI or other information), could you do what you need in a pci_fixup_early quirk?

That would be wonderful. If you have a patch I will install it immediately and change the content of my macro (the source-code even compiles on 2.4.x so there are lots of macros).

Cheers,
Dick Johnson
Penguin : Linux version 2.6.10 on an i686 machine (5537.79 BogoMips).
Notice : All mail here is now cached for review by Dictator Bush.
98.36% of all statistics are fiction.