Skip to contentSkip to conversation

Exploring the PCI EP Framework

DISCUSSION Kishon Vijay Abraham I 2 participants 2 messages Nov 13, 2019 1 message not in the archive PCI

lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): 20191112114854.GA3478@kiki

#1 lore raw

+linux-pci

Hi,

Quoted 9 lines · Ismael Luceno Cortes

Hello Kishon and Lorenzo,

I'm in the process of evaluating options to implement virtual devices, and I have a doubt that I couldn't solve by a quick look at the code.

I have some prototype boards with general purpose cores that can synthesize devices (sadly a custom driver, but I plan to replace it if I get approval).

Quoted 6 lines

The cores are for the most part equal, even in function, so the problem is that for whichever takes the role of endpoint I need a lightweight mechanism to signal an incoming transfer.

I didn't see any way to implement doorbell registers; is there such a thing?

The endpoint relies on polling a shared memory to detect any events. However if the HW supports some sort of memory signaled interrupt (similar to what is usually available in systems with RC) then we could use it for interrupting the endpoint system. All we have to do is map the MSI address to a EP BAR region so that host can write to a mapped BAR in order to raise MSI interrupt.

Even if the HW supports, the EP framework doesn't support to use MSI. I'll have to see how that can be added.

Quoted 4 lines

I realize it may not exist because it would imply not yet existing mechanisms. I even don't know if such a thing is feasible with a standard IOMMU, but I'm trying to figure out just that.

Does the PCI EP controller also wired to IOMMU?

Quoted 3 lines

I'm tempted to try to glue together the EP framework with VFIO for the purpose.

That would help to create a user space endpoint function driver. Not sure if it'll help for the doorbell.

Quoted 2 lines

Are you aware of any efforts along those lines?

Most of the efforts in EP right now are towards binding the epf to virtio.
https://lore.kernel.org/linux-pci/[email protected]/T/#m4092f14a49852425a00a9a9afa80b4d3b1b836d1

Thanks
Kishon

#2 lore raw

I'm adding Patrik, who was interested in the thread.

On 13/Nov/2019 15:36, Kishon Vijay Abraham I wrote:
<...>

Quoted 5 lines

The endpoint relies on polling a shared memory to detect any events. However if the HW supports some sort of memory signaled interrupt (similar to what is usually available in systems with RC) then we could use it for interrupting the endpoint system. All we have to do is map the MSI address to a EP BAR region so that host can write to a mapped BAR in order to raise MSI interrupt.

I was proposing exactly that internally here, but I'm not 100% happy with having the RC-side driver trigger the interrupts this way as we need to expose a hint about "how", but it's probably the only safe bet (we know the endpoints in question will always support MSI-X).

Quoted 2 lines

Even if the HW supports, the EP framework doesn't support to use MSI. I'll have to see how that can be added.

Thanks for the confirmation. Where's the polling happening? I can take a look at that.

Patrik: Can you check if we could publish that part of our docs? It would involve only the signalling.

Quoted 5 lines
Quoted 3 lines

I realize it may not exist because it would imply not yet existing mechanisms. I even don't know if such a thing is feasible with a standard IOMMU, but I'm trying to figure out just that.

Does the PCI EP controller also wired to IOMMU?

No, I'm just daydreaming: a programmable IOMMU could implement magic pages which produce rewrites, hooked to the PPR mechanics. Even if it's not all hardwired, I don't think I could get to play with that.

Quoted 5 lines
Quoted 2 lines

I'm tempted to try to glue together the EP framework with VFIO for the purpose.

That would help to create a user space endpoint function driver. Not sure if it'll help for the doorbell.

Oh, now that I read it again, it's confusing, I meant I would like to tie EP Framework with VFIO somehow, to implement the EPFs in user-space; completely unrelated to the previous comments.

Quoted 5 lines
Quoted 2 lines

Are you aware of any efforts along those lines?

Most of the efforts in EP right now are towards binding the epf to virtio.
https://lore.kernel.org/linux-pci/[email protected]/T/#m4092f14a49852425a00a9a9afa80b4d3b1b836d1

Very interesting, I didn't hit any limitations because our EPC driver is a hack, but we're going to go with the EP Framework in the end, so good to know (plus we're using virtio already in pretty much the same way).

Thank you very much.

Best regards.

Signature · 5 lines
Ismael Luceno <[email protected]>
+34 934452260 ext. 139
Travessera de les Corts 302 Bajos, 08029 Barcelona, Spain

https://www.silicon-gears.com/ - https://www.tttech-auto.com/