Skip to contentSkip to conversation

Sharing PCIe MMIO with other Drivers

DISCUSSION Andrei Danaila 3 participants 7 messages Nov 10, 2018 – Nov 12, 2018 PCI

lore
Share & tools
lore
#0 lore raw

Hello,

I have a question about best practices in writing an PCIe driver for an FPGA. If this is not the best place to ask, please let me know.

I have an FPGA which is connected over PCIe to an x86 host. The FPGA has a variety of peripherals on it, I2C, UART, SPI etc. All of these peripherals can be accessed from the host by accessing different offsets from the BAR0 address.

I am running linux kernel 4.14 on the host and have written a PCIe device driver which probes off the device id manufacturer ID of the FPGA.

The device driver calls pci_iomap( to obtain the cookie used to access the BAR. This works fine and via this mechanism I can read/write to the FPGA address space after calling ioremap on the cookie.

What I am trying to do now however is create a I2C platform device representing the I2C bus on the FPGA and add to it, as a resource, the BAR0 address + the I2C offset, to get the host's i2c driver to probe off this new PCI device.

In addition I am also trying to add an IRQ number for the I2C driver to use which is an MSIX mapped interrupt number obtained via pci_irq_vector.

In essence, I am trying to get the x86 host to own this device exposed via io-remapped region in PCI land, and use its driver to manage it.

The problem I am having is that I am getting a EBUSY return code when I try to register the resource to the platform device, after the pci_iomap has taken place.

The resource type is IORESOURCE_SYSTEM_RAM | IORESOURCE_MUXED and the start of the resource is the BAR0 address as returned by pci_iomap + I2C_OFFSET.

In the I2C device driver, I am expecting to do an ioremap on the resource and be able to access it by de-refencing

A couple of questions:

1. Is this the correct software flow for managing multiple devices exposed by a PCIe BAR0 address space? If not, what is the correct flow?
If yes, any ideas on what may be going wrong?

Please feel free to point me to any examples, I have looked around quite a bit but did not manage to find enough detail to let me solve this problem.

Thank you

#1 lore raw
Quoted 5 lines · Andrei Danaila

A couple of questions:

1. Is this the correct software flow for managing multiple devices exposed by a PCIe BAR0 address space? If not, what is the correct flow?
If yes, any ideas on what may be going wrong?

General practice is to create a PCI Physical Function for each sub-functionality (I2C/UART/DMA etc.) so that individual drivers can claim their own PCI device and its own BAR space without sharing resources.

#2 lore raw

Thanks Sinan,

I thought of that as well, unfortunately that it is not possible since the FPGA IP block is proprietary from a 3P vendor and lumps everything under one function with no option of adding additional functions.

Quoted 10 lines · Sinan Kaya
Quoted 5 lines · Andrei Danaila

A couple of questions:

1. Is this the correct software flow for managing multiple devices exposed by a PCIe BAR0 address space? If not, what is the correct flow?
If yes, any ideas on what may be going wrong?

General practice is to create a PCI Physical Function for each sub-functionality (I2C/UART/DMA etc.) so that individual drivers can claim their own PCI device and its own BAR space without sharing resources.

#3 lore raw
Quoted 3 lines · Andrei Danaila

Thanks Sinan,

I thought of that as well, unfortunately that it is not possible since the FPGA IP block is proprietary from a 3P vendor and lumps everything under one function with no option of adding additional functions.

How about adding multiple BARs and then give each BAR to a children driver?

#4 lore raw

Unfortunately, I cannot change the IP on the FPGA to add more BARs either, the whole thing is locked down.

Quoted 6 lines · Sinan Kaya
Quoted 3 lines · Andrei Danaila

Thanks Sinan,

I thought of that as well, unfortunately that it is not possible since the FPGA IP block is proprietary from a 3P vendor and lumps everything under one function with no option of adding additional functions.

How about adding multiple BARs and then give each BAR to a children driver?

#5 lore raw
Quoted 1 line · Andrei Danaila

Unfortunately, I cannot change the IP on the FPGA to add more BARs either, the whole thing is locked down.

Not the best hardware.
OK. I am running out of ideas. It doesn't play nice with the driver development model.

One idea is to have a parent PCI driver to allocate BAR space and have children platform devices that you create at runtime. You can point to arbitrary address in the system memory map and an IRQ using the platform devices. Though, it might have some alignment requirement in order to be able to make MMU mappings.

#6 lore raw
In reply to Andrei Danaila
Quoted 11 lines · Andrei Danaila

Hello,

I have a question about best practices in writing an PCIe driver for an FPGA. If this is not the best place to ask, please let me know.

I have an FPGA which is connected over PCIe to an x86 host. The FPGA has a variety of peripherals on it, I2C, UART, SPI etc. All of these peripherals can be accessed from the host by accessing different offsets from the BAR0 address.

I am running linux kernel 4.14 on the host and have written a PCIe device driver which probes off the device id manufacturer ID of the FPGA.

The device driver calls pci_iomap( to obtain the cookie used to access the BAR. This works fine and via this mechanism I can read/write to the FPGA address space after calling ioremap on the cookie.

What I am trying to do now however is create a I2C platform device representing the I2C bus on the FPGA and add to it, as a resource, the BAR0 address + the I2C offset, to get the host's i2c driver to probe off this new PCI device.

Keep in mind the address you read out of BAR0 in config space is a PCI bus address which isn't necessarily usable as a system physical address. It might be fine on x86 though.

Quoted 2 lines

In addition I am also trying to add an IRQ number for the I2C driver to use which is an MSIX mapped interrupt number obtained via pci_irq_vector.

You should be able to do something with an irqchip to forward the interrupt to the platform device safely.

Quoted 6 lines

In essence, I am trying to get the x86 host to own this device exposed via io-remapped region in PCI land, and use its driver to manage it.

The problem I am having is that I am getting a EBUSY return code when I try to register the resource to the platform device, after the pci_iomap has taken place.

The resource type is IORESOURCE_SYSTEM_RAM | IORESOURCE_MUXED and the start of the resource is the BAR0 address as returned by pci_iomap + I2C_OFFSET.

IIRC IORESOURCE_SYSTEM_RAM marks the resource as being normal system RAM (i.e. cacheable) rather than MMIO space, so this probably not the right set of flags to use.

Anyway, trying to create resource from the result of pci_iomap() sounds a little fishy to me since memory resources generally contain physical addresses and pci_iomap() should be returning a virtual address. The -EBUSY is probably due to the resource you created conflicting with an existing resource that doesn't have the MUXED flag set. I don't remember the exact process for doing this properly, but If I were you I would:

a) convert your resource to use physical addresses
b) make the BAR resource the parent of your new resource.

and see if that fixes the problem.

Oliver