MPC8536 PCI rescan to discover FPGA
Hi,
On a custom MPC8536 board running linux-2.6.31, I'd like to load FPGA code from linux and then rescan PCI-E bus to discover FPGA device. Is that possible ? When linux boots FPGA is not loaded, so initial PCI scan does not detect it.
I've tried playing with /sys/bus/pci/rescan and /sys/bus/pci/devices/.../rescan but didn't have much success.
Thanks.
Felix.
Hi Felix,
Quoted 8 lines
On a custom MPC8536 board running linux-2.6.31, I'd like to load FPGA code from linux and then rescan PCI-E bus to discover FPGA device. Is that possible ? When linux boots FPGA is not loaded, so initial PCI scan does not detect it.
I've tried playing with
/sys/bus/pci/rescanand/sys/bus/pci/devices/.../rescanbut didn't have much success.
This can be made to work using the kernel hot-swap interface. PCI devices have an ENUM# interrupt that they assert when inserted or extracted, and the host hot-swap driver can be hooked up to it. PCI-E may have a similar mechanism, if it does, then when your FPGA configures as a PCI-E device, it can assert that interrupt line (or send the appropriate PCI-E message to simulate that interrupt).
However, even if PCI-E does not have the concept of an ENUM# interrupt there is a way to generate a fake hot-swap event and generate a re-scan of the PCI bus.
I haven't tested the kernel hot-swap interface, but I know that Ira did, so I'll cc him on this mail, and he can let you know what he tested.
Cheers,
Dave
Quoted 9 lines · David Hawkins
This can be made to work using the kernel hot-swap interface. PCI devices have an ENUM# interrupt that they assert when inserted or extracted, and the host hot-swap driver can be hooked up to it. PCI-E may have a similar mechanism, if it does, then when your FPGA configures as a PCI-E device, it can assert that interrupt line (or send the appropriate PCI-E message to simulate that interrupt).
Quoted 7 lines
However, even if PCI-E does not have the concept of an ENUM# interrupt there is a way to generate a fake hot-swap event and generate a re-scan of the PCI bus.
I haven't tested the kernel hot-swap interface, but I know that Ira did, so I'll cc him on this mail, and he can let you know what he tested.
Right. However, in case it's a bit too much work to get hotswap implemented on the machine, you may still be able to do something simpler from your platform code, after you've finished loading the FPGA. I assume the FPGA doesn't contain a P2P bridge that would require probing further below the FPGA itself.
The basic idea is to call pci_scan_slot() on the devfn where the FPGA is supposed to respond.
Then you need to also do some fixup. First you need to call pcibios_setup_bus_devices(). This will wire up the device to an OF node if you have one, setup some default DMA ops, etc...
Note that this function will walk over all devices on that bus which is interesting since some of those may have already been fully setup initially. Hopefully that isn't a problem. If it was to become one, we would have to figure out a way to skip devices that have already been "setup".
And finally you call pcibios_finish_adding_to_bus() which will do the resource allocation pass on all new devices on the bus and register them with the core device layer.
Cheers,
Ben.
Hi, Benjamin
Quoted 21 lines · Benjamin Herrenschmidt
Right. However, in case it's a bit too much work to get hotswap implemented on the machine, you may still be able to do something simpler from your platform code, after you've finished loading the FPGA. I assume the FPGA doesn't contain a P2P bridge that would require probing further below the FPGA itself.
The basic idea is to call
pci_scan_slot()on the devfn where the FPGA is supposed to respond.Then you need to also do some fixup. First you need to call
pcibios_setup_bus_devices(). This will wire up the device to an OF node if you have one, setup some default DMA ops, etc...Note that this function will walk over all devices on that bus which is interesting since some of those may have already been fully setup initially. Hopefully that isn't a problem. If it was to become one, we would have to figure out a way to skip devices that have already been "setup".
Quoted 8 lines
And finally you call
pcibios_finish_adding_to_bus()which will do the resource allocation pass on all new devices on the bus and register them with the core device layer.Cheers,
Ben.
Thanks for the advice. This approach worked well for me, with some minor modifications. First I had to clear PPC_INDIRECT_TYPE_NO_PCIE_LINK in hose->indirect_type, otherwise reads from configuration space fail. And second, I had to call pci_assign_resource() to complete the assignment of FPGA memory. The memory range is 0x0 - 0x7fffff, and PCI code doesn't like resources starting at 0.
Again, thanks a lot for your help, much appreciated.
Felix.