DMA API issues
Hi.
This may have come up previously but I havent seen it, so...
My colleagues and I are encountering a number of difficulties with the DMA API, to which a generic solution is required (or risk multiple architectures, busses, and devices going their own way...)
Here is an example system that illustrates these problems:
I have a System On Chip device which, among other functions, contains an OHCI controller and 32K of SRAM.
heres the catch:- The OHCI controller has a different address space than the host bus, and worse, can *only* DMA data from its internal SRAM.
The architecture is not broken, merely unusual.
This causes the following problems:
1) The DMA API provides no methods to set up a mapping between the host memory map and the devices view of the space
example:
the OHCI controller above would see its 32K of SRAM as mapped from 0x10000 - 0x1ffff and not 0xXXX10000 - 0xXXX1ffff which is the address the CPU sees.
2) The DMA API assumes the device can access SDRAM
example:
the OHCI controller base is mapped at 0x10000000 on my platform. this is NOT is SDRAM, its in IO space.
If these points are possible to be addressed, it would allow at LEAST three chips *in use* in linux devices able to use mainline OHCI code directly - TC6393XB (in toshiba PDAs), SAMCOP (Ipaqs), and mediaQ (dell axims).
I am told HPPA has some similar problems also.
PS. please make use of CC: when replying
Quoted 10 lines · Ian Molton
I have a System On Chip device which, among other functions, contains an OHCI controller and 32K of SRAM.
heres the catch:- The OHCI controller has a different address space than the host bus, and worse, can *only* DMA data from its internal SRAM.
The architecture is not broken, merely unusual.
This causes the following problems:
Quoted 10 lines
1) The DMA API provides no methods to set up a mapping between the host memory map and the devices view of the space
example:
the OHCI controller above would see its 32K of SRAM as mapped from0x10000-0x1ffffand not 0xXXX10000 - 0xXXX1ffff which is the address the CPU sees.
2) The DMA API assumes the device can access SDRAM
example:
the OHCI controller base is mapped at0x10000000on my platform. this is NOT is SDRAM, its in IO space.
Can't you just implement an arch-specific allocator for your 32KB SRAM, then implement the DMA API streaming and dma_alloc/free APIs on top of that? Since this architecture is obviously not designed for performance, it doesn't seem to be a big deal to have the streaming APIs copy to/from the kmalloced (or whatever) buffer to/from the SRAM allocated memory and then have those APIs return the proper dma_addr_t for the embedded OHCI's address space view of the SRAM. Same thing goes for implementing dma_alloc/free (used by dma_pool*). I don't have the knowledge of USB subsystem buffer usage to know how quickly that little 32KB of SRAM is going to run out. But at a DMA API level, this seems doable, albeit with the greater possibility of negative retvals from these calls.
-Matt
Quoted 3 lines · On Fri, 18 Jun 2004 11:07:21 -0700 Matt Porter <[email protected]>…
Can't you just implement an arch-specific allocator for your 32KB SRAM, then implement the DMA API streaming and dma_alloc/free APIs on top of that?
Yes but thats not very generic is it? Im not the only one with this problem.
Quoted 2 lines
Since this architecture is obviously not designed
for performance
What makes you think writes to the 32K SRAM are any slower than to the SDRAM? the device is completely memory mapped.
Quoted 4 lines
, it doesn't seem to be a big deal to have the streaming APIs copy to/from the kmalloced (or whatever) buffer to/from the SRAM allocated memory and then have those APIs return the proper
dma_addr_tfor the embedded OHCI's address space view of the SRAM.
Again its a suboptimal solution, and on an architecture where the CPU isnt *that* fast in the first place it seems wrong to deliberately choose the slowest possible route...
Quoted 9 lines · Ian Molton
Quoted 3 lines · On Fri, 18 Jun 2004 11:07:21 -0700 Matt Porter <[email protected]>…
Can't you just implement an arch-specific allocator for your 32KB SRAM, then implement the DMA API streaming and dma_alloc/free APIs on top of that?
Yes but thats not very generic is it? Im not the only one with this problem.
Yes, it's suboptimal, but an option.
Quoted 5 lines
Quoted 2 lines
Since this architecture is obviously not designed
for performanceWhat makes you think writes to the 32K SRAM are any slower than to the SDRAM? the device is completely memory mapped.
I was referring to the small amount of space allowed for DMA operations, obviously not the speed of accessing SRAM.
Quoted 8 lines
Quoted 4 lines
, it doesn't seem to be a big deal to have the streaming APIs copy to/from the kmalloced (or whatever) buffer to/from the SRAM allocated memory and then have those APIs return the proper
dma_addr_tfor the embedded OHCI's address space view of the SRAM.Again its a suboptimal solution, and on an architecture where the CPU isnt *that* fast in the first place it seems wrong to deliberately choose the slowest possible route...
Ok, so you're looking for a complete change to the streaming DMA APIs, I guess. Possibly requiring another call to allocate streaming-capable memory since kmalloced buffers can't be used directly on your arch (or all arches). I agree it's suboptimal, it's one option to make it work in the current API.
-Matt
Quoted 26 lines · Matt Porter
Quoted 10 lines · Ian Molton
I have a System On Chip device which, among other functions, contains an OHCI controller and 32K of SRAM.
heres the catch:- The OHCI controller has a different address space than the host bus, and worse, can *only* DMA data from its internal SRAM.
The architecture is not broken, merely unusual.
This causes the following problems:
Quoted 12 lines
1) The DMA API provides no methods to set up a mapping between the host memory map and the devices view of the space
example:
the OHCI controller above would see its 32K of SRAM as mapped from0x10000-0x1ffffand not 0xXXX10000 - 0xXXX1ffff which is the address the CPU sees.
2) The DMA API assumes the device can access SDRAM
example:
the OHCI controller base is mapped at0x10000000on my platform. this is NOT is SDRAM, its in IO space.
Quoted 15 lines
Can't you just implement an arch-specific allocator for your 32KB SRAM, then implement the DMA API streaming and dma_alloc/free APIs on top of that? Since this architecture is obviously not designed for performance, it doesn't seem to be a big deal to have the streaming APIs copy to/from the kmalloced (or whatever) buffer to/from the SRAM allocated memory and then have those APIs return the proper
dma_addr_tfor the embedded OHCI's address space view of the SRAM. Same thing goes for implementing dma_alloc/free (used by dma_pool*). I don't have the knowledge of USB subsystem buffer usage to know how quickly that little 32KB of SRAM is going to run out. But at a DMA API level, this seems doable, albeit with the greater possibility of negative retvals from these calls.
Your analysis of what is needed is correct. However, we would prefer to fit this into the DMA API in a generic way, rather than having a specialized API that is not acceptable upstream. I'm more concerned with finding the best way to address this problem than with whether this is a 2.6 or 2.7 issue.
We do need a memory allocator for the pool of SRAM. If we're not going to have to build customized versions of the OHCI driver, we need that pool hooked into the dma_pool and dma_alloc_coherent interfaces. We could do that with a platform specific implementation of dma_alloc_coherent, but a pointer to dma_{alloc,free} from struct device seems like a cleaner solution. We also will need bounce buffers, as you described, but there is already a good start towards that support in 2.6. The other thing that might be needed is passing device to page_to_dma so that device specific dma addresses can be constructed.
Deepak Saxena wrote a pretty good summary as part if the discussion about this issue on the linux-arm-kernel mailing list:
http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-June/022796.html
I think I'm looking for something like the PARISC hppa_dma_ops but more generic:
http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-June/022813.html
Jamey Hicks
Quoted 32 lines · Jamey Hicks
Quoted 12 lines · Matt Porter
Quoted 10 lines · Ian Molton
1) The DMA API provides no methods to set up a mapping between the host memory map and the devices view of the space
example:
the OHCI controller above would see its 32K of SRAM as mapped from0x10000-0x1ffffand not 0xXXX10000 - 0xXXX1ffff which is the address the CPU sees.
2) The DMA API assumes the device can access SDRAM
example:
the OHCI controller base is mapped at0x10000000on my platform. this is NOT is SDRAM, its in IO space.Quoted 13 lines
Can't you just implement an arch-specific allocator for your 32KB SRAM, then implement the DMA API streaming and dma_alloc/free APIs on top of that? Since this architecture is obviously not designed for performance, it doesn't seem to be a big deal to have the streaming APIs copy to/from the kmalloced (or whatever) buffer to/from the SRAM allocated memory and then have those APIs return the proper
dma_addr_tfor the embedded OHCI's address space view of the SRAM. Same thing goes for implementing dma_alloc/free (used by dma_pool*). I don't have the knowledge of USB subsystem buffer usage to know how quickly that little 32KB of SRAM is going to run out. But at a DMA API level, this seems doable, albeit with the greater possibility of negative retvals from these calls.Your analysis of what is needed is correct. However, we would prefer to fit this into the DMA API in a generic way, rather than having a specialized API that is not acceptable upstream. I'm more concerned with finding the best way to address this problem than with whether this is a 2.6 or 2.7 issue.
Quoted 6 lines
We do need a memory allocator for the pool of SRAM. If we're not going to have to build customized versions of the OHCI driver, we need that pool hooked into the
dma_poolanddma_alloc_coherentinterfaces. We could do that with a platform specific implementation ofdma_alloc_coherent, but a pointer to dma_{alloc,free} from struct device seems like a cleaner solution. We also will need bounce buffers, as you
Yes, it can be cleaner, but it's not something I would say is completely broken with the DMA API. It does provide a way to make your device specific implementation, regardless of whether it's managed ideally via a struct device pointer. Migrating to dev->dma* sounds suspiciously like a 2.7ism.
Quoted 2 lines
described, but there is already a good start towards that support in 2.6. The other thing that might be needed is passing device to
Where's that code?
Quoted 1 line
page_to_dmaso that device specific dma addresses can be constructed.
A struct device argument to page_to_dma seems like a no brainer to be included.
Quoted 4 lines
Deepak Saxena wrote a pretty good summary as part if the discussion about this issue on the linux-arm-kernel mailing list:
http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-June/022796.html
Ahh, ok. Deepak and I have discussed this idea F2F on a several occassions, I recall he needed it for the small floating PCI window he has to manage on the IXP* ports. It may help in some embedded PPC areas as well.
Quoted 4 lines
I think I'm looking for something like the PARISC
hppa_dma_opsbut more generic:http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-June/022813.html
I see that's somewhat like what David Brownell suggested before...a single pointer to a set of dma ops from struct device. hppa_dma_ops translated into a generic dma_ops entity with fields corresponding to existing DMA API calls would be a good starting point. We can get rid of some address translation hacks in a lot of custom embedded PPC drivers with something like this.
-Matt
Quoted 4 lines · Matt Porter
Quoted 1 line
page_to_dmaso that device specific dma addresses can be constructed.A struct device argument to
page_to_dmaseems like a no brainer to be included.
Tony Lindgren recently submitted a patch for this:
http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=1931/1
which now pending for Linus. ARM platforms now have three macros to define if they want to override the default struct page to DMA address translation.
Quoted 6 lines
I see that's somewhat like what David Brownell suggested before...a single pointer to a set of dma ops from struct device.
hppa_dma_opstranslated into a genericdma_opsentity with fields corresponding to existing DMA API calls would be a good starting point. We can get rid of some address translation hacks in a lot of custom embedded PPC drivers with something like this.
I really don't think we need to go this far.
As I understand it, the issue seems to surround DMA coherent memory for USB descriptors, and what happens when we call the streaming DMA API.
We have the latter solved with Deepak's DMA bounce code (already merged) provided it's given the right information to determine when bounce buffers are needed.
The bounce code only needs a way to get at the "safe" DMA memory, and it uses DMA pools for that. DMA pools in turn take their memory from dma_alloc_coherent.
So, we just need a way to make dma_alloc_coherent do the right thing. One suggestion from James yesterday was to have a way to register device-private "coherent DMA" backing memory with dma_alloc_coherent, which it would use in preference to calling alloc_pages(). However, this memory would have no struct page pointer associated with it, and our dma_alloc_coherent implementation currently relies completely on that condition existing - so it would mean a complete rewrite of that.
Note also that some drivers (notably ALSA) assume that memory returned from dma_alloc_coherent() does have struct page pointers, so this would also break ALSA (which in turn provides much more of a justification for the DMA MMAP API I was trying (and failed) to propose a few months back.)
Also, we need to consider what the DMA mask means in this case? Should it be zero (to mean "can't DMA from system memory"?)
Signature · 4 lines
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 PCMCIA - http://pcmcia.arm.linux.org.uk/
2.6 Serial coreAt Fri, 18 Jun 2004 20:43:22 +0100,
Quoted 7 lines · Russell King
Quoted 4 lines · Matt Porter
Quoted 1 line
page_to_dmaso that device specific dma addresses can be constructed.A struct device argument to
page_to_dmaseems like a no brainer to be included.
Quoted 7 lines
Tony Lindgren recently submitted a patch for this:
http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=1931/1
which now pending for Linus. ARM platforms now have three macros to define if they want to override the default struct page to DMA address translation.
Wouldn't it be nicer to define a more generic style like
struct page *dma_to_page(struct device *, void *, dma_addr_t)
??
Quoted 30 lines
Quoted 7 lines
I see that's somewhat like what David Brownell suggested before...a single pointer to a set of dma ops from struct device.
hppa_dma_opstranslated into a genericdma_opsentity with fields corresponding to existing DMA API calls would be a good starting point. We can get rid of some address translation hacks in a lot of custom embedded PPC drivers with something like this.I really don't think we need to go this far.
As I understand it, the issue seems to surround DMA coherent memory for USB descriptors, and what happens when we call the streaming DMA API.
We have the latter solved with Deepak's DMA bounce code (already merged) provided it's given the right information to determine when bounce buffers are needed.
The bounce code only needs a way to get at the "safe" DMA memory, and it uses DMA pools for that. DMA pools in turn take their memory from
dma_alloc_coherent.So, we just need a way to make
dma_alloc_coherentdo the right thing. One suggestion from James yesterday was to have a way to register device-private "coherent DMA" backing memory withdma_alloc_coherent, which it would use in preference to callingalloc_pages(). However, this memory would have no struct page pointer associated with it, and ourdma_alloc_coherentimplementation currently relies completely on that condition existing - so it would mean a complete rewrite of that.
Quoted 5 lines
Note also that some drivers (notably ALSA) assume that memory returned from
dma_alloc_coherent()does have struct page pointers, so this would also break ALSA (which in turn provides much more of a justification for the DMA MMAP API I was trying (and failed) to propose a few months back.)
Yes, the struct page pointer is needed for vma_ops.nopage in mmap on ALSA. So far, this is broken on some architectures like ARM. We need a proper conversion from virtual/bus pointer to a page struct.
Of course, mmap is not mandatory, and we have a fallback via ioctl/read/write. But it's important to _know_ whether remapping is available...
--
Takashi Iwai <[email protected]> ALSA Developer - www.alsa-project.org
Quoted 13 lines · Takashi Iwai
At Fri, 18 Jun 2004 20:43:22 +0100,
Quoted 11 lines · Russell King
Quoted 4 lines · Matt Porter
Quoted 1 line
page_to_dmaso that device specific dma addresses can be constructed.A struct device argument to
page_to_dmaseems like a no brainer to be included.Tony Lindgren recently submitted a patch for this:
http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=1931/1
Quoted 7 lines
Quoted 3 lines
which now pending for Linus. ARM platforms now have three macros to define if they want to override the default struct page to DMA address translation.
Wouldn't it be nicer to define a more generic style like
struct page *dma_to_page(struct device *, void *,
dma_addr_t)
What's the 'void *' for? Hint: this has nothing to do with the virtual address and DMA address returned from dma_alloc_coherent().
page_to_dma - converts a struct page to a DMA address for a given device dma_to_virt - converts a DMA address to a CPU direct-mapped virtual address virt_to_dma - converts a CPU direct-mapped virtual address to a DMA address
Each one well defined for our _current_ interfaces.
Quoted 3 lines
Yes, the struct page pointer is needed for vma_ops.nopage in mmap on ALSA. So far, this is broken on some architectures like ARM. We need a proper conversion from virtual/bus pointer to a page struct.
Please get away from your nopage implementation. We've been around this before with Linus, and the conclusion was that it's up to architectures to provide a MMAP method for DMA memory, which _may_ use the nopage method if and only if it is appropriate for their implementation.
There's just no way I'm going to implement a half-baked "nopage" function for ALSA.
This is actually the same issue for some framebuffer devices as well - which also need DMA MMAP to be correct.
Signature · 4 lines
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 PCMCIA - http://pcmcia.arm.linux.org.uk/
2.6 Serial coreQuoted 10 lines · Russell King
Quoted 3 lines · Takashi Iwai
Yes, the struct page pointer is needed for vma_ops.nopage in mmap on ALSA. So far, this is broken on some architectures like ARM. We need a proper conversion from virtual/bus pointer to a page struct.
Please get away from your nopage implementation. We've been around this before with Linus, and the conclusion was that it's up to architectures to provide a MMAP method for DMA memory, which _may_ use the nopage method if and only if it is appropriate for their implementation.
Can you elaborate?
When I was writing sound/oss/via82cxxx_audio.c, Linus specifically recommended using ->nopage, which worked out quite well (modulo cleaning up VM_xxx flags).
Jeff
Quoted 4 lines · Jeff Garzik
When I was writing
sound/oss/via82cxxx_audio.c, Linus specifically recommended using ->nopage, which worked out quite well (modulo cleaning upVM_xxxflags).
It does need some architecture-specific support for getting the caching behaviour right, but I do believe that as long as there is some "struct page" that can be returned, together with a way of setting the vm_page_prot bits right, it should be ok.
The argument at some point was that some architectures may not even _have_ a "struct page" for DMA memory, since it's not "normal" memory (ie "slow memory" on m68k). However, I thought we all agreed that such a "struct page" could be furnished if that architecture wants so support mmap'ing.
Linus
Quoted 5 lines · Linus Torvalds
The argument at some point was that some architectures may not even _have_ a "struct page" for DMA memory, since it's not "normal" memory (ie "slow memory" on m68k). However, I thought we all agreed that such a "struct page" could be furnished if that architecture wants so support mmap'ing.
.. which is not to say that we shouldn't have a "pci_mmap_pages()" thing _too_. Pretty clearly the easiest interface often is to just map the pages at mmap() time, and then we should just have a helper function to do that.
I thought we did one already, but hey, maybe not.
Linus
At Mon, 21 Jun 2004 20:26:39 -0700 (PDT),
Quoted 8 lines · Linus Torvalds
Quoted 5 lines · Linus Torvalds
The argument at some point was that some architectures may not even _have_ a "struct page" for DMA memory, since it's not "normal" memory (ie "slow memory" on m68k). However, I thought we all agreed that such a "struct page" could be furnished if that architecture wants so support mmap'ing.
Quoted 5 lines
.. which is not to say that we shouldn't have a "
pci_mmap_pages()" thing _too_. Pretty clearly the easiest interface often is to just map the pages atmmap()time, and then we should just have a helper function to do that.I thought we did one already, but hey, maybe not.
I don't think we have such.
Russell has once proposed a similar one (but not pci-specific), and I believe it makes sense for many drivers. We can hide the architecture-specific cache handling inside the helper function, too.
Takashi
Quoted 14 lines · Takashi Iwai
At Mon, 21 Jun 2004 20:26:39 -0700 (PDT),
Quoted 12 lines · Linus Torvalds
Quoted 5 lines · Linus Torvalds
The argument at some point was that some architectures may not even _have_ a "struct page" for DMA memory, since it's not "normal" memory (ie "slow memory" on m68k). However, I thought we all agreed that such a "struct page" could be furnished if that architecture wants so support mmap'ing.
.. which is not to say that we shouldn't have a "
pci_mmap_pages()" thing _too_. Pretty clearly the easiest interface often is to just map the pages atmmap()time, and then we should just have a helper function to do that.
Quoted 7 lines
Quoted 1 line
I thought we did one already, but hey, maybe not.
I don't think we have such.
Russell has once proposed a similar one (but not pci-specific), and I believe it makes sense for many drivers. We can hide the architecture-specific cache handling inside the helper function, too.
Ok. So does anyone have any objections if I push the ARM DMA mmap interface upstream, which consists of:
/**
* dma_mmap_coherent - map a coherent DMA allocation into user space
* @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
* @vma: vm_area_struct describing requested user mapping
* @cpu_addr: kernel CPU-view address returned from dma_alloc_coherent
* @handle: device-view address returned from dma_alloc_coherent
* @size: size of memory originally requested in dma_alloc_coherent *
* Map a coherent DMA buffer previously allocated by dma_alloc_coherent
* into user space. The coherent DMA buffer must not be freed by the
* driver until the user space mapping has been released.
*/
int dma_mmap_coherent(struct device *dev, struct vm_area_struct *vma,
void *cpu_addr, dma_addr_t handle, size_t size);
and a similar one for the ARM-specific "write combining" case (for framebuffers utilising the DMA API)?
This was discussed on linux-arch, and the discussion died while trying to settle on a suitable interface through apparant lack of interest.
Signature · 4 lines
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 PCMCIA - http://pcmcia.arm.linux.org.uk/
2.6 Serial coreAt Wed, 23 Jun 2004 13:34:23 +0100,
Quoted 40 lines · Russell King
Quoted 14 lines · Takashi Iwai
At Mon, 21 Jun 2004 20:26:39 -0700 (PDT),
Quoted 12 lines · Linus Torvalds
Quoted 5 lines · Linus Torvalds
The argument at some point was that some architectures may not even _have_ a "struct page" for DMA memory, since it's not "normal" memory (ie "slow memory" on m68k). However, I thought we all agreed that such a "struct page" could be furnished if that architecture wants so support mmap'ing.
.. which is not to say that we shouldn't have a "
pci_mmap_pages()" thing _too_. Pretty clearly the easiest interface often is to just map the pages atmmap()time, and then we should just have a helper function to do that.Quoted 7 lines
Quoted 1 line
I thought we did one already, but hey, maybe not.
I don't think we have such.
Russell has once proposed a similar one (but not pci-specific), and I believe it makes sense for many drivers. We can hide the architecture-specific cache handling inside the helper function, too.
Ok. So does anyone have any objections if I push the ARM DMA mmap interface upstream, which consists of:
/**
*dma_mmap_coherent- map a coherent DMA allocation into user space
* @dev: valid struct device pointer, or NULL for ISA and EISA-like devices
* @vma:vm_area_structdescribing requested user mapping
* @cpu_addr: kernel CPU-view address returned fromdma_alloc_coherent
* @handle: device-view address returned fromdma_alloc_coherent
* @size: size of memory originally requested indma_alloc_coherent*
* Map a coherent DMA buffer previously allocated bydma_alloc_coherent
* into user space. The coherent DMA buffer must not be freed by the
* driver until the user space mapping has been released.
*/
int dma_mmap_coherent(struct device *dev, structvm_area_struct*vma,
Quoted 1 line
void *cpu_addr, dma_addr_t handle, size_t size);
This looks easy to use for me.
Quoted 2 lines
and a similar one for the ARM-specific "write combining" case (for framebuffers utilising the DMA API)?
pgprot_noncached() is used on many other architectures in fbmem.c (well, not really, but the result is identical).
Should it be provided as another one, or is it used as default in dma_mmap_coherent()?
Also, it would be nice to have a version for sg-buffer, too.
Takashi
Quoted 7 lines · Takashi Iwai
Quoted 2 lines
and a similar one for the ARM-specific "write combining" case (for framebuffers utilising the DMA API)?
pgprot_noncached()is used on many other architectures infbmem.c(well, not really, but the result is identical).
Should it be provided as another one, or is it used as default indma_mmap_coherent()?
The whole point is to kill the idea that drivers should have to know about page protection crap. That should be wholely contained within the architecture implementation.
Quoted 1 line
Also, it would be nice to have a version for sg-buffer, too.
Well, we don't have a sg-buffer version of dma_alloc_coherent(), so we don't have a sg-buffer version of dma_mmap_coherent().
Signature · 4 lines
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 PCMCIA - http://pcmcia.arm.linux.org.uk/
2.6 Serial coreAt Wed, 23 Jun 2004 16:44:19 +0100,
Quoted 5 lines · Russell King
Quoted 3 lines · Takashi Iwai
Quoted 2 lines
and a similar one for the ARM-specific "write combining" case (for framebuffers utilising the DMA API)?
Quoted 8 lines
Quoted 4 lines
pgprot_noncached()is used on many other architectures infbmem.c(well, not really, but the result is identical).
Should it be provided as another one, or is it used as default indma_mmap_coherent()?The whole point is to kill the idea that drivers should have to know about page protection crap. That should be wholely contained within the architecture implementation.
I agree. My question is whether we need to handle different cases with cached, non-cached and writecombine, according to the demand of the driver. In other words, do we always handle these mmap pages to be non-cached (except for writecombine version)?
Quoted 4 lines
Quoted 1 line
Also, it would be nice to have a version for sg-buffer, too.
Well, we don't have a sg-buffer version of
dma_alloc_coherent(), so we don't have a sg-buffer version ofdma_mmap_coherent().
That sound reasonable :)
Takashi
Quoted 12 lines · Takashi Iwai
At Wed, 23 Jun 2004 16:44:19 +0100,
Quoted 10 lines · Russell King
Quoted 7 lines · Takashi Iwai
Quoted 2 lines
and a similar one for the ARM-specific "write combining" case (for framebuffers utilising the DMA API)?
pgprot_noncached()is used on many other architectures infbmem.c(well, not really, but the result is identical).
Should it be provided as another one, or is it used as default indma_mmap_coherent()?
Quoted 8 lines
Quoted 3 lines
The whole point is to kill the idea that drivers should have to know about page protection crap. That should be wholely contained within the architecture implementation.
I agree. My question is whether we need to handle different cases with cached, non-cached and writecombine, according to the demand of the driver. In other words, do we always handle these mmap pages to be non-cached (except for writecombine version)?
No - the driver should have _zero_ visibility of this. The only time the cache property comes in is where the architecture needs to alter these properties to achieve the requirements of the interface - IOW to provide user space with a DMA-coherent mapping.
Remember that some architectures are unable to disable caching on a page by page basis, yet may be able to offer fully-cached DMA coherent mmaps, and therefore such driver interference is wrong.
Signature · 4 lines
Russell King
Linux kernel 2.6 ARM Linux - http://www.arm.linux.org.uk/
maintainer of: 2.6 PCMCIA - http://pcmcia.arm.linux.org.uk/
2.6 Serial coreAt Tue, 22 Jun 2004 00:08:38 +0100,
Quoted 21 lines · Russell King
Quoted 19 lines · Takashi Iwai
At Fri, 18 Jun 2004 20:43:22 +0100,
Quoted 7 lines · Russell King
Quoted 4 lines · Matt Porter
Quoted 1 line
page_to_dmaso that device specific dma addresses can be constructed.A struct device argument to
page_to_dmaseems like a no brainer to be included.Quoted 7 lines
Tony Lindgren recently submitted a patch for this:
http://www.arm.linux.org.uk/developer/patches/viewpatch.php?id=1931/1
which now pending for Linus. ARM platforms now have three macros to define if they want to override the default struct page to DMA address translation.
Wouldn't it be nicer to define a more generic style like
Quoted 8 lines
Quoted 1 line
struct page *dma_to_page(struct device *, void *,
dma_addr_t)What's the 'void *' for? Hint: this has nothing to do with the virtual address and DMA address returned from
dma_alloc_coherent().
page_to_dma- converts a struct page to a DMA address for a given devicedma_to_virt- converts a DMA address to a CPU direct-mapped virtual addressvirt_to_dma- converts a CPU direct-mapped virtual address to a DMA address
Well, I thought the address handling can be dependent on the device type, thus passing the both virtual and BUS addresses _might_ help to calculate the DMA address in some cases. But forget it, seems exaggerated.
Takashi
Quoted 10 lines · Matt Porter
On Fri, Jun 18, 2004 at 02:33:18PM -0400, Jamey Hicks wrote:
Yes, it can be cleaner, but it's not something I would say is completely broken with the DMA API. It does provide a way to make your device specific implementation, regardless of whether it's managed ideally via a struct device pointer. Migrating to dev->dma* sounds suspiciously like a 2.7ism.
The DMA API is certainly not completely broken. Except for page_to_dma needing struct device the interface could be implemented with per-device dma operation implementations.
Quoted 8 lines
Quoted 3 lines
described, but there is already a good start towards that support in 2.6. The other thing that might be needed is passing device to
Where's that code?
arch/arm/common/dmabounce.c
Quoted 28 lines
Quoted 3 lines
page_to_dmaso that device specific dma addresses can be constructed.A struct device argument to
page_to_dmaseems like a no brainer to be included.Quoted 5 lines
Deepak Saxena wrote a pretty good summary as part if the discussion about this issue on the linux-arm-kernel mailing list:
http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-June/022796.html
Ahh, ok. Deepak and I have discussed this idea F2F on a several occassions, I recall he needed it for the small floating PCI window he has to manage on the IXP* ports. It may help in some embedded PPC areas as well.
Quoted 5 lines
I think I'm looking for something like the PARISC
hppa_dma_opsbut more generic:http://lists.arm.linux.org.uk/pipermail/linux-arm-kernel/2004-June/022813.html
Quoted 8 lines
I see that's somewhat like what David Brownell suggested before...a single pointer to a set of dma ops from struct device.
hppa_dma_opstranslated into a genericdma_opsentity with fields corresponding to existing DMA API calls would be a good starting point. We can get rid of some address translation hacks in a lot of custom embedded PPC drivers with something like this.
Yes, I think this would be generally useful.
Jamey
Quoted 8 lines · Ian Molton
Hi.
This may have come up previously but I havent seen it, so...
My colleagues and I are encountering a number of difficulties with the DMA API, to which a generic solution is required (or risk multiple architectures, busses, and devices going their own way...)
Quoted 7 lines
Here is an example system that illustrates these problems:
I have a System On Chip device which, among other functions, contains an OHCI controller and 32K of SRAM.
heres the catch:- The OHCI controller has a different address space than the host bus, and worse, can *only* DMA data from its internal SRAM.
Unfortunately, I tend to think that you should not be using the DMA API for this, since it's not host RAM.
There are loads of devices with on-board SRAM/DRAM/... you really have to manage your own resources at that point.
I know that sucks, WRT driver re-use. Maybe you can wrap it inside the OHCI driver, ohci_dma_alloc_xxx() etc. For the normal case, the wrapper resolves directly to the DMA API. For the embedded case, the wrapper does SRAM-specific stuff.
You _might_ convince the kernel DMA gurus that this could be done by creating a driver-specific bus, and pointing struct device to that internal bus, but that seems like an awful lot of work as opposed to the wrappers.
Jeff
Quoted 28 lines · Jeff Garzik
Quoted 8 lines · Ian Molton
Hi.
This may have come up previously but I havent seen it, so...
My colleagues and I are encountering a number of difficulties with the DMA API, to which a generic solution is required (or risk multiple architectures, busses, and devices going their own way...)
Quoted 7 lines
Here is an example system that illustrates these problems:
I have a System On Chip device which, among other functions, contains an OHCI controller and 32K of SRAM.
heres the catch:- The OHCI controller has a different address space than the host bus, and worse, can *only* DMA data from its internal SRAM.
Unfortunately, I tend to think that you should not be using the DMA API for this, since it's not host RAM.
There are loads of devices with on-board SRAM/DRAM/... you really have to manage your own resources at that point.
I know that sucks, WRT driver re-use. Maybe you can wrap it inside the OHCI driver,
ohci_dma_alloc_xxx()etc. For the normal case, the wrapper resolves directly to the DMA API. For the embedded case, the wrapper does SRAM-specific stuff.
Quoted 7 lines
You _might_ convince the kernel DMA gurus that this could be done by creating a driver-specific bus, and pointing struct device to that internal bus, but that seems like an awful lot of work as opposed to the wrappers.
Jeff
I don't think that any DMA lends itself very well to a generic solution and I see that some have dedicated a lot of effort trying to make such an API.
Take the simple PLX chip that is pretty common, at least as a clone or macro-cell in many PCI/Bus devices. For its bus-master DMA, you need a place to build a scatter-list in physical RAM with a physical address that's known, plus you need kernel-mode virtual address pointers to be able to build the list. Then you need to get the physical address of some user buffer and lock it down during the transfer, or you need to have kernel buffers that you know the physical address of, plus have kernel mode pointers to that area. That area may not even be RAM! And, certainly the place that you get the data from inside the device may be a single longword with auto-incrementing hardware.
The fact that you write a 3 to some register somewhere to start a DMA for a lot of chips, doesn't mean that there will ever be a generic solution to DMA. Every driver is different.
Even the busmaster Ethernet boards fail to fully utilize the capability of DMA. The idea of allocating a DMA-able block, DMAing to it (waiting for the DMA to complete), then de-allocating that block is extremely wasteful.
I think the whole thing should be re-thought. Something like:
On startup, some list of every page of RAM should be built (once). A simple algorithm should be made that uses that list. With a few machine instructions one should be able to find:
(1) The physical page of the starting location. (2) The amount of space available in that page. (3) The physical page of the next location. (4) Etc.
Using that information a scatter-list can be built for the DMA transfer(s) and those pages need to be brought into memory and locked down only for the duration of the DMA.
This would allow any resident page anywhere to be the source or target of any DMA operation.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.26 on an i686 machine (5570.56 BogoMips).
Note 96.31% of all statistics are fiction.
Quoted 3 lines · On Fri, 18 Jun 2004 18:12:48 -0400 (EDT) "Richard B. Johnson"…
Take the simple PLX chip that is pretty common, at least as a clone or macro-cell in many PCI/Bus devices.
<cut>
But this *isnt* one of those. its a simple, straightforward, and fairly common architecture thats so close to the existing model support is near trivial.
Quoted 4 lines · On Fri, 18 Jun 2004 17:08:03 -0400 Jeff Garzik <[email protected]> wrote:
You _might_ convince the kernel DMA gurus that this could be done by creating a driver-specific bus, and pointing struct device to that internal bus, but that seems like an awful lot of work as opposed to the wrappers.
Its an awful lot less work than re-writing all those drivers!
Quoted 8 lines · Ian Molton
Quoted 4 lines · On Fri, 18 Jun 2004 17:08:03 -0400 Jeff Garzik <[email protected]> wrote:
You _might_ convince the kernel DMA gurus that this could be done by creating a driver-specific bus, and pointing struct device to that internal bus, but that seems like an awful lot of work as opposed to the wrappers.
Its an awful lot less work than re-writing all those drivers!
Every other driver bar this one already copes correctly with on chip memory using the ioremap methods. That's why we're all wondering if it isn't simpler to fix this driver.
James
Quoted 5 lines · James Bottomley
Quoted 3 lines · Ian Molton
On Fri, 18 Jun 2004 17:08:03 -0400
Jeff Garzik <[email protected]> wrote:
Quoted 8 lines
Quoted 3 lines
Quoted 1 line
You _might_ convince the kernel DMA gurus that this could be done by creating a driver-specific bus, and pointing struct device to that internal bus, but that seems like an awful lot of work as opposed to the wrappers.
Its an awful lot less work than re-writing all those drivers!
Every other driver bar this one already copes correctly with on chip memory using the ioremap methods. That's why we're all wondering if it isn't simpler to fix this driver.
Indeed... if the SRAM is accessible via ioremapped memory, DMA API shouldn't be needed.
Jeff
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · On Fri, 18 Jun 2004 20:22:07 -0400 Jeff Garzik <[email protected]> wrote:
Quoted 2 lines
my CPU can write directly to this 32K of SRAM. the chip can DMA from it.
Yes, write via MMIO. Non-local RAM is not DMA.
I fail to see your point. this is NOT MMIO. in MMIO you write and the device reads as you write. in DMA, you write and then tell the chip to read from the memory you wrote to.
this is exactly what Im talking about.
Heres what the DMA mapping code deals with:
CPU-----host bus------>RAM-----io bus---->device
and heres what I have:
CPU-----host bus----->RAM-----io bus----->device
the *only* difference is that the RAM in the first case is SDRAM and in the latter is SRAM. the type of RAM is irrelevant to the DMA system.
I *could* (at great expense) replace the SDRAM in a PC with SRAM. DMA would still work the same way.
sure, the SRAM is *on* the die of my OCHI/multi-io controller. whats that got to do with it?
and theres the other issue - if I made an ohci allocator for the SRAM I'd have to partition off a small ammount ONLY for OHCI. with only 32K to begin with thats a huge penalty for all the other devices in the multi IO chip. a bus level DMA allocator solves this AND makes drivers cleaner/more maintainable.
What more do you want?
Quoted 7 lines · Ian Molton
Quoted 4 lines · On Fri, 18 Jun 2004 20:22:07 -0400 Jeff Garzik <[email protected]> wrote:
Quoted 2 lines
my CPU can write directly to this 32K of SRAM. the chip can DMA from it.
Quoted 5 lines
Quoted 1 line
Yes, write via MMIO. Non-local RAM is not DMA.
I fail to see your point. this is NOT MMIO. in MMIO you write and the device reads as you write. in DMA, you write and then tell the chip to read from the memory you wrote to.
<snip>
I could possibly see some use for allowing multiple DMA domains in addition to the system memory. Maybe something to allow overriding the dma allocator per device? We could have optional dev->map() and unmap(), and then use the dma-mapping API only to keep track of what's allocated.
But I don't know how much sense that makes because there would be only one driver using that DMA buffer. The DMA area would not be shared with other drivers like the system memory is.
So maybe the only advantage would be to allow more portable generic device drivers.
Luckily I don't need this right now :)
Tony