Skip to contentSkip to conversation

Trying to reset a PCIe device and scratching my head...

DISCUSSION Jimmie Mayfield 3 participants 4 messages Sep 28, 2010

lore
Share & tools
lore
#0 lore raw

Hello. I'm in the process of writing a device driver for a PCIe device where the bulk of the logic, including the PCIe interface, is implemented on a couple of large FPGAs. Current target platform is x86 Linux.

Having the PCIe interface implemented inside FPGA 'A' makes upgrading that particular FPGA rather troublesome. In a perfect world, one would be able to upgrade the FPGA without having to reboot the machine. The hardware guys have designed the card to reload that FPGA image upon a slot reset...either fundamental or hot. (If I could guarantee the card would always be plugged into a hotplug-capable slot, I suppose I could forge a hotplug event and toggle power to the slot. But I can't impose that requirement on customers.)

So I'd like to be able to send either a fundamental or hot reset to the device but so far I've had no success. Is there a recommended way of doing this on x86 Linux?

Jimmie

#1 lore raw
Quoted 8 lines · Jimmie Mayfield

Having the PCIe interface implemented inside FPGA 'A' makes upgrading that particular FPGA rather troublesome. In a perfect world, one would be able to upgrade the FPGA without having to reboot the machine. The hardware guys have designed the card to reload that FPGA image upon a slot reset...either fundamental or hot.

So I'd like to be able to send either a fundamental or hot reset to the device but so far I've had no success.

The PCIe AER driver (drivers/pci/pcie/aer/) sends a hot reset when it has received a fatal error. As far as I can tell, it just sets the PCI_BRIDGE_CTL_BUS_RESET bit of the upstream bridge; everything else is just infrastructure to handle error reporting and to notify the device driver about the reset.

Regards,
Clemens

#2 lore raw
Quoted 7 lines · On Tue, 28 Sep 2010 09:28:14 +0200 Clemens Ladisch <[email protected]> wrote:
Quoted 6 lines · Jimmie Mayfield

Having the PCIe interface implemented inside FPGA 'A' makes upgrading that particular FPGA rather troublesome. In a perfect world, one would be able to upgrade the FPGA without having to reboot the machine. The hardware guys have designed the card to reload that FPGA image upon a slot reset...either fundamental or hot.

Quoted 8 lines
Quoted 2 lines

So I'd like to be able to send either a fundamental or hot reset to the device but so far I've had no success.

The PCIe AER driver (drivers/pci/pcie/aer/) sends a hot reset when it has received a fatal error. As far as I can tell, it just sets the PCI_BRIDGE_CTL_BUS_RESET bit of the upstream bridge; everything else is just infrastructure to handle error reporting and to notify the device driver about the reset.

Right, if your system supports this, it may be the easiest way to go. But devices often have proprietary ways of resetting themselves too; maybe you could change your device to reset if a specific bit in config or MMIO space was flipped. You'd likely need some sort of delay before accessing the device again, but it should be bounded and fairly fixed, so probably not a big deal for the driver to handle.

Signature · 1 line
Jesse Barnes, Intel Open Source Technology Center
#3 lore raw
Quoted 1 line · Jesse Barnes

Right, if your system supports this, it may be the easiest way to go.

I was under the impression that PCIe requires support for at least fundamental reset and maybe hot reset (secondary bus reset) though I'd have to go back to the spec to verify the latter. Are you saying that not all PCIe chipsets support this? Sorry if this seems like a dumb question...I'm by no means a PCIe guru.

Quoted 5 lines

But devices often have proprietary ways of resetting themselves too; maybe you could change your device to reset if a specific bit in config or MMIO space was flipped. You'd likely need some sort of delay before accessing the device again, but it should be bounded and fairly fixed, so probably not a big deal for the driver to handle.

Indeed, the device does support a proprietary reset via MMIO but that reset only affects those components that lie behind the card's PCIe interface. So it's nearly a full device reset but not quite. The FPGA containing the PCIe interface does not get reloaded. Unfortunately, that behavior is likely set in stone.

Jimmie