PCI DMA to user mem on mpc83xx
Ira,
we have a pretty old PCI device driver here that needs some basic rework running on 2.6.27 on several MPC83xx. It's a simple char-device with "give me some data" implemented using read() resulting in zero-copy DMA to user mem.
There's get_user_pages() working under the hood along with SetPageDirty() and page_cache_release().
Main goal is to prepare a sg-list that gets fed into a DMA controller.
I wonder if there's a more up-to-date/efficient and future proof scheme of creating the mapping.
Could you provide some pointers or would you stick to the current scheme ?
Regards,
André
MATRIX VISION GmbH, Talstrasse 16, DE-71570 Oppenweiler
Registergericht: Amtsgericht Stuttgart, HRB 271090
Geschaeftsfuehrer: Gerhard Thullner, Werner Armingeon, Uwe Furtner
Quoted 10 lines · Andre Schwarz
Ira,
we have a pretty old PCI device driver here that needs some basic rework running on 2.6.27 on several MPC83xx.
It's a simple char-device with "give me some data" implemented usingread()resulting in zero-copy DMA to user mem.There's
get_user_pages()working under the hood along withSetPageDirty()andpage_cache_release().
Quoted 8 lines
Main goal is to prepare a sg-list that gets fed into a DMA controller.
I wonder if there's a more up-to-date/efficient and future proof scheme of creating the mapping.
Could you provide some pointers or would you stick to the current scheme ?
This scheme is the best you'll come up with for zero-copy IO. I used get_user_pages_fast(), but otherwise my implementation was the same. These interfaces should be fairly future proof.
In the end, I realized that most of my transfers were 4 bytes in length, and zero copy IO was a waste of effort. I decided to use mmap instead.
Ira
Ira,
Quoted 17 lines · Andre Schwarz
Quoted 16 lines
Ira,
we have a pretty old PCI device driver here that needs some basic rework running on 2.6.27 on several MPC83xx.
It's a simple char-device with "give me some data" implemented usingread()resulting in zero-copy DMA to user mem.There's
get_user_pages()working under the hood along withSetPageDirty()andpage_cache_release().Main goal is to prepare a sg-list that gets fed into a DMA controller.
I wonder if there's a more up-to-date/efficient and future proof scheme of creating the mapping.
Quoted 5 lines
Quoted 2 lines
Could you provide some pointers or would you stick to the current scheme ?
This scheme is the best you'll come up with for zero-copy IO. I used
get_user_pages_fast(), but otherwise my implementation was the same. These interfaces should be fairly future proof.
excellent - thanks.
Will stick to it then ...
Quoted 3 lines
In the end, I realized that most of my transfers were 4 bytes in length, and zero copy IO was a waste of effort. I decided to use mmap instead.
I'm using 98% page sized (4KiB) scatter gather transfers summing up to ~80MiB/sec sustained throughput.
Cheers,
André
MATRIX VISION GmbH, Talstrasse 16, DE-71570 Oppenweiler
Registergericht: Amtsgericht Stuttgart, HRB 271090
Geschaeftsfuehrer: Gerhard Thullner, Werner Armingeon, Uwe Furtner
=20
Quoted 7 lines
we have a pretty old PCI device driver here that needs some basic rework running on 2.6.27 on several MPC83xx. It's a simple char-device with "give me some data" implemented using
read()resulting in zero-copy DMA to user mem. =20
There'sget_user_pages()working under the hood along with=20SetPageDirty()andpage_cache_release().
Does that dma use the userspace virtual address, or the physical address - or are you remapping the user memory into kernel address space.
If the memory is remapped into the kernel address space, the cost of the mmu and tlb operations (especially on MP systems) is such that a dma to kernel memory followed by copyout/copytouser may well be faster!
That may even be the case even if the dma is writing to the user virtual (or physical) addresses when it is only necessary to ensure the memory page is resident and that the caches are coherent.
In any case the second copy is probably far faster than the PCI one!
I've recently written driver that supports a pread/pwrite interface to the memory windows on a PCIe card. It was important to use dma for the PCIe transfers (to get a sensible transfer size). I overlapped the copyin/copyout with the next dma transfer. The dma's are fast enough that it is worth spinning waiting for completion - but slow enough to make the overlapped operation worthwhile (same speed as a single word pio transfer).
David
David,
Quoted 10 lines
Quoted 7 lines
we have a pretty old PCI device driver here that needs some basic rework running on 2.6.27 on several MPC83xx. It's a simple char-device with "give me some data" implemented using
read()resulting in zero-copy DMA to user mem.There's
get_user_pages()working under the hood along withSetPageDirty()andpage_cache_release().Does that dma use the userspace virtual address, or the physical address - or are you remapping the user memory into kernel address space.
no mapping at all AFAIK.
I'm using get_user_pages() followed by allocation of a struct scatterlist being filled with sg_set_page(). After the transfer the pages are marked dirty using SetPageDirty().
Quoted 4 lines
If the memory is remapped into the kernel address space, the cost of the mmu and tlb operations (especially on MP systems) is such that a dma to kernel memory followed by copyout/copytouser may well be faster!
no mapping.
Quoted 4 lines
That may even be the case even if the dma is writing to the user virtual (or physical) addresses when it is only necessary to ensure the memory page is resident and that the caches are coherent.
All I need is physical addresses of user mem.
Since the allocating user driver is using mlock() and there's no swap I expect to be safe ... is this a stupid assumption ?
Quoted 2 lines
In any case the second copy is probably far faster than the PCI one!
huh - I observed memcpy() to be very expensive (at least on 83xx PowerPC).
Quoted 7 lines
I've recently written driver that supports a pread/pwrite interface to the memory windows on a PCIe card. It was important to use dma for the PCIe transfers (to get a sensible transfer size). I overlapped the copyin/copyout with the next dma transfer. The dma's are fast enough that it is worth spinning waiting for completion - but slow enough to make the overlapped operation worthwhile (same speed as a single word pio transfer).
Thanks for your feedback.
Cheers,
André
MATRIX VISION GmbH, Talstrasse 16, DE-71570 Oppenweiler
Registergericht: Amtsgericht Stuttgart, HRB 271090
Geschaeftsfuehrer: Gerhard Thullner, Werner Armingeon, Uwe Furtner