FPGA access over PCI-E on MPC8536
Hi,
On my custom MPC8536 based board running 2.6.31 kernel FPGA is connected via x2 PCI-E lane. FPGA is identified during PCI scan and is visible via lspci.
0000:01:00.0 Class ff00: Altera Corporation Unknown device 0004 (rev 01)
Subsystem: Altera Corporation Unknown device 0004
Flags: fast devsel, IRQ 16
Memory at 98000000 (32-bit, non-prefetchable) [disabled] [size=1M]
Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
Queue=0/5 Enable-
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Endpoint IRQ 0
Capabilities: [100] Virtual Channel
However when I attempt to access FPGA memory my mmapping it in userspace the read hangs. The same happens in kernel space. Does it happen because FPGA memory is marked as disabled, or because FPGA code is doing something wrong ?
Another question is what can cause PCI device memory be marked as disabled.
Thanks a lot.
Felix.
Quoted 18 lines · Felix Radensky
Hi,
On my custom MPC8536 based board running 2.6.31 kernel FPGA is connected via x2 PCI-E lane. FPGA is identified during PCI scan and is visible via lspci.
0000:01:00.0 Class ff00: Altera Corporation Unknown device 0004 (rev 01)
Subsystem: Altera Corporation Unknown device 0004
Flags: fast devsel, IRQ 16
Memory at 98000000 (32-bit, non-prefetchable) [disabled]
[size=1M]
Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
Queue=0/5 Enable-
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Endpoint IRQ 0
Capabilities: [100] Virtual Channel
Quoted 1 line
However when I attempt to access FPGA memory my mmapping it in userspace the read hangs. The same happens in kernel space. Does it happen because FPGA memory is marked as disabled, or because FPGA code is doing something wrong ?
Can you access the device in u-boot? That would possible tell you if the HW is functioning or not.
Quoted 2 lines
Another question is what can cause PCI device memory be marked as disabled.
Good question, no idea how lspci decided to print [disabled]. Take a look at lspci source and see :)
- k
Quoted 6 lines
Quoted 5 lines
However when I attempt to access FPGA memory my mmapping it in userspace the read hangs. The same happens in kernel space. Does it happen because FPGA memory is marked as disabled, or because FPGA code is doing something wrong ?
Quoted 8 lines
Can you access the device in u-boot? That would possible tell you if the HW is functioning or not.
Quoted 2 lines
Another question is what can cause PCI device memory be marked as disabled.
Good question, no idea how lspci decided to print [disabled]. Take a look at lspci source and see :)
Maybe the memory enable in the PCI command register isn't set ?
If you don't have a kernel driver that sets it (by calling pci_enable_device) and u-boot doesn't set it then it's going to be off and you'll get hangs or machine checks trying to access the device...
Just an idea...
Cheers,
Ben.
Hi, Benjamin
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 6 lines
Quoted 1 line
However when I attempt to access FPGA memory my mmapping it in userspace the read hangs. The same happens in kernel space. Does it happen because FPGA memory is marked as disabled, or because FPGA code is doing something wrong ?
Can you access the device in u-boot? That would possible tell you if the HW is functioning or not.
Quoted 2 lines
Another question is what can cause PCI device memory be marked as disabled.
Good question, no idea how lspci decided to print [disabled]. Take a look at lspci source and see :)
Quoted 8 lines
Maybe the memory enable in the PCI command register isn't set ?
If you don't have a kernel driver that sets it (by calling
pci_enable_device) and u-boot doesn't set it then it's going to be off and you'll get hangs or machine checks trying to access the device...Just an idea...
Yes, that was my problem exactly ! Setting this bit via setpci fixed it.
Thanks a lot for your help.
Felix
Hi Felix,
Quoted 15 lines
Quoted 15 lines
On my custom MPC8536 based board running 2.6.31 kernel FPGA is connected via x2 PCI-E lane. FPGA is identified during PCI scan and is visible via lspci.
0000:01:00.0 Class ff00: Altera Corporation Unknown device 0004 (rev 01)
Subsystem: Altera Corporation Unknown device 0004
Flags: fast devsel, IRQ 16
Memory at 98000000 (32-bit, non-prefetchable) [disabled] [size=1M]
Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
Queue=0/5 Enable-
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Endpoint IRQ 0
Capabilities: [100] Virtual Channel
Quoted 8 lines
Quoted 1 line
However when I attempt to access FPGA memory my mmapping it in userspace the read hangs. The same happens in kernel space. Does it happen because FPGA memory is marked as disabled, or because FPGA code is doing something wrong ?
Can you access the device in u-boot? That would possible tell you if the HW is functioning or not.
Quoted 2 lines
Another question is what can cause PCI device memory be marked as disabled.
Good question, no idea how lspci decided to print [disabled]. Take a look at lspci source and see :)
lspci has a -x and -xxx options to print the bytes in the PCI configuration space header. You should use that to see what the settings are, and then you'll be able to interpret what disabled means.
For example, here's lspci run from an x86 host CPU looking at an MPC8349EA processor peripheral board:
# lspci -s 04:0d.0 -v
04:0d.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30)
Flags: 66Mhz, fast devsel, IRQ 10
Memory at fc300000 (32-bit, non-prefetchable) [size=1M]
Memory at fe400000 (32-bit, prefetchable) [size=4K]
Memory at fe300000 (64-bit, prefetchable) [size=1M]
Memory at fe200000 (64-bit, prefetchable) [size=1M]
Capabilities: [48] #06 [0080]# lspci -s 04:0d.0 -x 04:0d.0 Power PC: Freescale Semiconductor Inc MPC8349E (rev 30) 00: 57 19 80 00 42 01 b0 00 30 00 20 0b 08 40 00 00 10: 00 00 30 fc 08 00 40 fe 0c 00 30 fe 00 00 00 00 20: 0c 00 20 fe 00 00 00 00 00 00 00 00 00 00 00 00 30: 00 00 00 00 48 00 00 00 00 00 00 00 0a 01 00 00
The memory, non-prefetchable, and size are decoded from bits in the base address registers. I'm pretty sure there is an enabled/disabled bit in there too. Thats probably where your lspci output is getting it from.
I suspect that you could perform a configuration space write to enable that bit, or that the FPGA initialization code should be setting that bit. It really depends on what the PCIe component inside the FPGA was designed to do.
For example, on the MPC8349EA processor configured as a peripheral board, the PCI bridge can come up enabled, or disabled. If the bridge is disabled, then it is up to the PowerPC processor core (U-boot code) to enable the PCI interface prior to the host CPU BIOS configuring the base addresses in the interface. Its possible you've missed something like this when you configured the core, eg. an 'enable on boot option' (I haven't used the PCIe core, so have no idea if this is an option - but you could submit an Altera service request).
Since you're playing with an FPGA, you should be able to connect Altera's logic analyzer core to the PCI interface and capture traffic as well, so that could be used to debug your problem.
Cheers,
Dave
A message that is not in the archive (sent off-list or not yet received): [email protected]
Can you access fpga memory from u-boot?
Could very well be an fpga problem. Same thing can happen on xilinx parts when accessing an undefined register address via pci bridge --> plb bus --> ipif register block --> bad address. The pci core goes into some kind of retry loop with the plb bus (I'm wasn't the one to look at the pci analyzer, so I don't know the details), and it completely locks up.
g.
On Sep 16, 2009 11:02 PM, "Felix Radensky" <[email protected]> wrote:
Hi,
On my custom MPC8536 based board running 2.6.31 kernel FPGA is connected via x2 PCI-E lane. FPGA is identified during PCI scan and is visible via lspci.
0000:01:00.0 Class ff00: Altera Corporation Unknown device 0004 (rev 01)
Subsystem: Altera Corporation Unknown device 0004
Flags: fast devsel, IRQ 16
Memory at 98000000 (32-bit, non-prefetchable) [disabled] [size=1M]
Capabilities: [50] Message Signalled Interrupts: Mask+ 64bit+
Queue=0/5 Enable-
Capabilities: [78] Power Management version 3
Capabilities: [80] Express Endpoint IRQ 0
Capabilities: [100] Virtual Channel
However when I attempt to access FPGA memory my mmapping it in userspace the read hangs. The same happens in kernel space. Does it happen because FPGA memory is marked as disabled, or because FPGA code is doing something wrong ?
Another question is what can cause PCI device memory be marked as disabled.
Thanks a lot.
Felix.
_______________________________________________
Linuxppc-dev mailing list
[email protected]
https://lists.ozlabs.org/listinfo/linuxppc-dev
Hello Felix,
Quoted 4 lines · Felix Radensky
On my custom MPC8536 based board running 2.6.31 kernel FPGA is connected via x2 PCI-E lane. FPGA is identified during PCI scan and is visible via lspci.
I committed a PCI Express device driver for an Altera FPGA (chaining DMA reference) design upstream that resides in the upstream Linux kernel at drivers/staging/altpciechdma/
It can act as a reference for the generic part of your design.
Regards,
Signature · 1 line
Leon