Skip to contentSkip to conversation

PCI DMA to user mem on mpc83xx

DISCUSSION Andre Schwarz 3 participants 5 messages May 23, 2011 – May 24, 2011

lore
Share & tools
lore
#0 lore raw

Ira,

we have a pretty old PCI device driver here that needs some basic rework running on 2.6.27 on several MPC83xx. It's a simple char-device with "give me some data" implemented using read() resulting in zero-copy DMA to user mem.

There's get_user_pages() working under the hood along with SetPageDirty() and page_cache_release().

Main goal is to prepare a sg-list that gets fed into a DMA controller.

I wonder if there's a more up-to-date/efficient and future proof scheme of creating the mapping.

Could you provide some pointers or would you stick to the current scheme ?

Regards,
André

MATRIX VISION GmbH, Talstrasse 16, DE-71570 Oppenweiler
Registergericht: Amtsgericht Stuttgart, HRB 271090
Geschaeftsfuehrer: Gerhard Thullner, Werner Armingeon, Uwe Furtner

#1 lore raw
Quoted 10 lines · Andre Schwarz

Ira,

we have a pretty old PCI device driver here that needs some basic rework running on 2.6.27 on several MPC83xx.
It's a simple char-device with "give me some data" implemented using read() resulting in zero-copy DMA to user mem.

There's get_user_pages() working under the hood along with SetPageDirty() and page_cache_release().

Quoted 8 lines

Main goal is to prepare a sg-list that gets fed into a DMA controller.

I wonder if there's a more up-to-date/efficient and future proof scheme of creating the mapping.

Could you provide some pointers or would you stick to the current scheme ?

This scheme is the best you'll come up with for zero-copy IO. I used get_user_pages_fast(), but otherwise my implementation was the same. These interfaces should be fairly future proof.

In the end, I realized that most of my transfers were 4 bytes in length, and zero copy IO was a waste of effort. I decided to use mmap instead.

Ira

#2 lore raw

Ira,

Quoted 17 lines · Andre Schwarz
Quoted 16 lines

Ira,

we have a pretty old PCI device driver here that needs some basic rework running on 2.6.27 on several MPC83xx.
It's a simple char-device with "give me some data" implemented using read() resulting in zero-copy DMA to user mem.

There's get_user_pages() working under the hood along with SetPageDirty() and page_cache_release().

Main goal is to prepare a sg-list that gets fed into a DMA controller.

I wonder if there's a more up-to-date/efficient and future proof scheme of creating the mapping.

Quoted 5 lines
Quoted 2 lines

Could you provide some pointers or would you stick to the current scheme ?

This scheme is the best you'll come up with for zero-copy IO. I used get_user_pages_fast(), but otherwise my implementation was the same. These interfaces should be fairly future proof.

excellent - thanks.
Will stick to it then ...

Quoted 3 lines

In the end, I realized that most of my transfers were 4 bytes in length, and zero copy IO was a waste of effort. I decided to use mmap instead.

I'm using 98% page sized (4KiB) scatter gather transfers summing up to ~80MiB/sec sustained throughput.

Cheers,
André

MATRIX VISION GmbH, Talstrasse 16, DE-71570 Oppenweiler
Registergericht: Amtsgericht Stuttgart, HRB 271090
Geschaeftsfuehrer: Gerhard Thullner, Werner Armingeon, Uwe Furtner

#3 lore raw
In reply to Andre Schwarz

=20

Quoted 7 lines

we have a pretty old PCI device driver here that needs some basic rework running on 2.6.27 on several MPC83xx. It's a simple char-device with "give me some data" implemented using read() resulting in zero-copy DMA to user mem. =20
There's get_user_pages() working under the hood along with=20 SetPageDirty() and page_cache_release().

Does that dma use the userspace virtual address, or the physical address - or are you remapping the user memory into kernel address space.

If the memory is remapped into the kernel address space, the cost of the mmu and tlb operations (especially on MP systems) is such that a dma to kernel memory followed by copyout/copytouser may well be faster!

That may even be the case even if the dma is writing to the user virtual (or physical) addresses when it is only necessary to ensure the memory page is resident and that the caches are coherent.

In any case the second copy is probably far faster than the PCI one!

I've recently written driver that supports a pread/pwrite interface to the memory windows on a PCIe card. It was important to use dma for the PCIe transfers (to get a sensible transfer size). I overlapped the copyin/copyout with the next dma transfer. The dma's are fast enough that it is worth spinning waiting for completion - but slow enough to make the overlapped operation worthwhile (same speed as a single word pio transfer).

David

#4 lore raw

David,

Quoted 10 lines
Quoted 7 lines

we have a pretty old PCI device driver here that needs some basic rework running on 2.6.27 on several MPC83xx. It's a simple char-device with "give me some data" implemented using read() resulting in zero-copy DMA to user mem.

There's get_user_pages() working under the hood along with SetPageDirty() and page_cache_release().

Does that dma use the userspace virtual address, or the physical address - or are you remapping the user memory into kernel address space.

no mapping at all AFAIK.

I'm using get_user_pages() followed by allocation of a struct scatterlist being filled with sg_set_page(). After the transfer the pages are marked dirty using SetPageDirty().

Quoted 4 lines

If the memory is remapped into the kernel address space, the cost of the mmu and tlb operations (especially on MP systems) is such that a dma to kernel memory followed by copyout/copytouser may well be faster!

no mapping.

Quoted 4 lines

That may even be the case even if the dma is writing to the user virtual (or physical) addresses when it is only necessary to ensure the memory page is resident and that the caches are coherent.

All I need is physical addresses of user mem.
Since the allocating user driver is using mlock() and there's no swap I expect to be safe ... is this a stupid assumption ?

Quoted 2 lines

In any case the second copy is probably far faster than the PCI one!

huh - I observed memcpy() to be very expensive (at least on 83xx PowerPC).

Quoted 7 lines

I've recently written driver that supports a pread/pwrite interface to the memory windows on a PCIe card. It was important to use dma for the PCIe transfers (to get a sensible transfer size). I overlapped the copyin/copyout with the next dma transfer. The dma's are fast enough that it is worth spinning waiting for completion - but slow enough to make the overlapped operation worthwhile (same speed as a single word pio transfer).

Thanks for your feedback.

Cheers,
André

MATRIX VISION GmbH, Talstrasse 16, DE-71570 Oppenweiler
Registergericht: Amtsgericht Stuttgart, HRB 271090
Geschaeftsfuehrer: Gerhard Thullner, Werner Armingeon, Uwe Furtner