dma_alloc_coherent versus streaming DMA, neither works satisfactory

Can anyone here offer some advise on this?

Quoted 2 lines · On Thursday 23 April 2015 13:52:34 Mike Looijmans wrote:

Can anyone here offer some advise on this?

The problem you are experiencing is a direct result of using hardware without cache-coherency from user space. There is no software workaround for this: If you want data to be cacheable *and* avoid doing manual cache flushes each time data is passed between user space and hardware, you have to use hardware that is cache-coherent.

You mentioned that you are using 'Zynq', which supports cache-coherent DMA using the 'accelerator coherency port'. If you are able to connect your device to that port, it should work, otherwise you should consider using a different platform.

Quoted 2 lines · On Thursday 23 April 2015 13:52:34 Mike Looijmans wrote:

Can anyone here offer some advise on this?

The problem you are experiencing is a direct result of using hardware without cache-coherency from user space. There is no software workaround for this: If you want data to be cacheable *and* avoid doing manual cache flushes each time data is passed between user space and hardware, you have to use hardware that is cache-coherent.

You mentioned that you are using 'Zynq', which supports cache-coherent DMA using the 'accelerator coherency port'. If you are able to connect your device to that port, it should work, otherwise you should consider using a different platform.

Quoted 10 lines · Arnd Bergmann

Quoted 2 lines · On Thursday 23 April 2015 13:52:34 Mike Looijmans wrote:

Can anyone here offer some advise on this?

The problem you are experiencing is a direct result of using hardware without cache-coherency from user space. There is no software workaround for this: If you want data to be cacheable *and* avoid doing manual cache flushes each time data is passed between user space and hardware, you have to use hardware that is cache-coherent.

Quoted 4 lines

You mentioned that you are using 'Zynq', which supports cache-coherent DMA using the 'accelerator coherency port'. If you are able to connect your device to that port, it should work, otherwise you should consider using a different platform.

I tried as you suggested, and used the ACP instead of the HP to connect the logic to the CPU.

Without any further changes, this passes all tests with the exact same performance numbers. The reason for that is that the driver/DMA framework is unaware of the "coherency" hardware and still uses the manual cache flush/invalidate routines, so I figured out that adding a "dma-coherent" property to the devicetree node changes that.

However, with "dma-coherent" set for my driver, the system locks up at random points in the tests. Simple memory transfer tests fail with data mismatches (probably stale cache results). Running DMA tests usually results in the system completely locking up at some point.
Normal register read/write access is done through the AXI bus directly, not using the ACP at all.

Is the ACP hardware broken, is there some extra things my driver needs to be aware of, or is there something else I need to do here?

You still need to synchronize MMIO register accesses with write buffers, as the readl() and writel() functions do in the kernel.

In particular, after you have written a buffer to memory from the CPU, you will need to do an outer_sync() before the MMIO write that triggers the DMA. This is still much cheaper than doing the cache flush though.

Another possible problem would be if the driver mmaps the buffer in uncached mode to user space. This is something your kernel driver has to get right, it won't be handled automatically by setting the "dma-coherent" property in DT.

Quoted 6 lines · Arnd Bergmann

You still need to synchronize MMIO register accesses with write buffers, as the readl() and writel() functions do in the kernel.

In particular, after you have written a buffer to memory from the CPU, you will need to do an outer_sync() before the MMIO write that triggers the DMA. This is still much cheaper than doing the cache flush though.

Note that outer_sync() is already done by readl/writel and/or the write memory barriers (mb()/wmb()).

Quoted 4 lines

Another possible problem would be if the driver mmaps the buffer in uncached mode to user space. This is something your kernel driver has to get right, it won't be handled automatically by setting the "dma-coherent" property in DT.

The buffer should also be mapped into userspace with the same memory type and cache attributes as the kernel side mapping. If using ACP, then you probably want "normal memory, cacheable, writeback, read allocate" or in the case of SMP, the same but "read/write allocate".

Quoted 10 lines · Russell King - ARM Linux

Quoted 6 lines · Arnd Bergmann

You still need to synchronize MMIO register accesses with write buffers, as the readl() and writel() functions do in the kernel.

In particular, after you have written a buffer to memory from the CPU, you will need to do an outer_sync() before the MMIO write that triggers the DMA. This is still much cheaper than doing the cache flush though.

Note that outer_sync() is already done by readl/writel and/or the write memory barriers (mb()/wmb()).

I initiate the DMA transfers using iowrite32() so if I understand correctly, I'm already doing the right thing here.

Just to be completely clear, there is no direct register access from user space, the driver does all MMIO. Userspace only gets an mmap for DMA buffers, and uses ioctl to initiate transfers.

Quoted 5 lines

Quoted 4 lines

Another possible problem would be if the driver mmaps the buffer in uncached mode to user space. This is something your kernel driver has to get right, it won't be handled automatically by setting the "dma-coherent" property in DT.

Quoted 4 lines

The buffer should also be mapped into userspace with the same memory type and cache attributes as the kernel side mapping. If using ACP, then you probably want "normal memory, cacheable, writeback, read allocate" or in the case of SMP, the same but "read/write allocate".

I currently use dma_alloc_coherent() to allocate buffers and dma_mmap_coherent() to map them to user space. I was under the assumption that these would do the right thing. Is that correct? If not, then what should I use?

Quoted 12 lines · On Wednesday 29 April 2015 11:47:37 Mike Looijmans wrote:

Quoted 10 lines · Russell King - ARM Linux

Quoted 6 lines · Arnd Bergmann

You still need to synchronize MMIO register accesses with write buffers, as the readl() and writel() functions do in the kernel.

In particular, after you have written a buffer to memory from the CPU, you will need to do an outer_sync() before the MMIO write that triggers the DMA. This is still much cheaper than doing the cache flush though.

Note that outer_sync() is already done by readl/writel and/or the write memory barriers (mb()/wmb()).

Quoted 6 lines

I initiate the DMA transfers using iowrite32() so if I understand correctly, I'm already doing the right thing here.

Just to be completely clear, there is no direct register access from user space, the driver does all MMIO. Userspace only gets an mmap for DMA buffers, and uses ioctl to initiate transfers.

Ok, that seems all fine then.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

Another possible problem would be if the driver mmaps the buffer in uncached mode to user space. This is something your kernel driver has to get right, it won't be handled automatically by setting the "dma-coherent" property in DT.

Quoted 8 lines

Quoted 4 lines

The buffer should also be mapped into userspace with the same memory type and cache attributes as the kernel side mapping. If using ACP, then you probably want "normal memory, cacheable, writeback, read allocate" or in the case of SMP, the same but "read/write allocate".

I currently use dma_alloc_coherent() to allocate buffers and dma_mmap_coherent() to map them to user space. I was under the assumption that these would do the right thing. Is that correct? If not, then what should I use?

dma_mmap_coherent() is the right interface, but I've just looked at the implementation of arm_dma_mmap() and I'm not sure that it actually uses the correct vma->vm_page_prot value here, because I don't see where it takes into account whether the device is coherent or not. Most ARM machines have only noncoherent devices, and dma_mmap_coherent() is used rarely by drivers, so it's quite possible that this interface got broken without anybody noticing.

If my suspicion is correct, we should either change arm_coherent_dma_ops() to refer to a different mmap() callback that does the right thing for coherent devices, or change arm_dma_mmap() to look at dev->is_coherent.

Quoted 10 lines · Arnd Bergmann

Quoted 5 lines · On Wednesday 29 April 2015 11:47:37 Mike Looijmans wrote:

Quoted 4 lines · Russell King - ARM Linux

The buffer should also be mapped into userspace with the same memory type and cache attributes as the kernel side mapping. If using ACP, then you probably want "normal memory, cacheable, writeback, read allocate" or in the case of SMP, the same but "read/write allocate".

If my suspicion is correct, we should either change arm_coherent_dma_ops() to refer to a different mmap() callback that does the right thing for coherent devices, or change arm_dma_mmap() to look at dev->is_coherent.

Following the route, arch/arm/mm/dma-mapping.c uses pgprot_dmacoherent() which is defined in arch/arm/include/asm/pgtable.h and that just returns uncached memory.

If you can give me some hints as to what the correct flags would be, I can patch my kernel and test it.

Quoted 12 lines · On Wednesday 29 April 2015 12:33:00 Mike Looijmans wrote:

Quoted 10 lines · Arnd Bergmann

Quoted 5 lines · On Wednesday 29 April 2015 11:47:37 Mike Looijmans wrote:

Quoted 4 lines · Russell King - ARM Linux

The buffer should also be mapped into userspace with the same memory type and cache attributes as the kernel side mapping. If using ACP, then you probably want "normal memory, cacheable, writeback, read allocate" or in the case of SMP, the same but "read/write allocate".

If my suspicion is correct, we should either change arm_coherent_dma_ops() to refer to a different mmap() callback that does the right thing for coherent devices, or change arm_dma_mmap() to look at dev->is_coherent.

Quoted 6 lines

Following the route, arch/arm/mm/dma-mapping.c uses pgprot_dmacoherent() which is defined in arch/arm/include/asm/pgtable.h and that just returns uncached memory.

If you can give me some hints as to what the correct flags would be, I can patch my kernel and test it.

Use the flags that Russell listed above.

Quoted 13 lines · Arnd Bergmann

Quoted 12 lines · On Wednesday 29 April 2015 12:33:00 Mike Looijmans wrote:

Quoted 10 lines · Arnd Bergmann

Quoted 5 lines · On Wednesday 29 April 2015 11:47:37 Mike Looijmans wrote:

Quoted 4 lines · Russell King - ARM Linux

The buffer should also be mapped into userspace with the same memory type and cache attributes as the kernel side mapping. If using ACP, then you probably want "normal memory, cacheable, writeback, read allocate" or in the case of SMP, the same but "read/write allocate".

If my suspicion is correct, we should either change arm_coherent_dma_ops() to refer to a different mmap() callback that does the right thing for coherent devices, or change arm_dma_mmap() to look at dev->is_coherent.

Quoted 8 lines

Quoted 6 lines

Following the route, arch/arm/mm/dma-mapping.c uses pgprot_dmacoherent() which is defined in arch/arm/include/asm/pgtable.h and that just returns uncached memory.

If you can give me some hints as to what the correct flags would be, I can patch my kernel and test it.

Use the flags that Russell listed above.

I would if I had a clue how to do that. For one thing, I don't understand all the L_PTE_... flag juggling that happens here.

I also tried just using kmalloc() to allocate the buffer, and then dma_map_single that. This is what the DMA documentation told me to do for non-coherent mappings. This works fine for the streaming-dma mode using the HP, but using dma-coherent this not only fails to work, it tends to completely lock the system.

Avoiding the streaming mapping and using only the coherent modes prevents the system locking up, it then doesn't do more harm than just corrupting data in the buffers.

Quoted 32 lines · On Wednesday 29 April 2015 14:49:26 Mike Looijmans wrote:

Quoted 13 lines · Arnd Bergmann

Quoted 12 lines · On Wednesday 29 April 2015 12:33:00 Mike Looijmans wrote:

Quoted 10 lines · Arnd Bergmann

Quoted 5 lines · On Wednesday 29 April 2015 11:47:37 Mike Looijmans wrote:

Quoted 4 lines · Russell King - ARM Linux

The buffer should also be mapped into userspace with the same memory type and cache attributes as the kernel side mapping. If using ACP, then you probably want "normal memory, cacheable, writeback, read allocate" or in the case of SMP, the same but "read/write allocate".

If my suspicion is correct, we should either change arm_coherent_dma_ops() to refer to a different mmap() callback that does the right thing for coherent devices, or change arm_dma_mmap() to look at dev->is_coherent.

Quoted 8 lines

Quoted 6 lines

Following the route, arch/arm/mm/dma-mapping.c uses pgprot_dmacoherent() which is defined in arch/arm/include/asm/pgtable.h and that just returns uncached memory.

If you can give me some hints as to what the correct flags would be, I can patch my kernel and test it.

Use the flags that Russell listed above.

I would if I had a clue how to do that. For one thing, I don't understand all the L_PTE_... flag juggling that happens here.

I also tried just using kmalloc() to allocate the buffer, and then dma_map_single that. This is what the DMA documentation told me to do for non-coherent mappings. This works fine for the streaming-dma mode using the HP, but using dma-coherent this not only fails to work, it tends to completely lock the system.

Quoted 3 lines

Avoiding the streaming mapping and using only the coherent modes prevents the system locking up, it then doesn't do more harm than just corrupting data in the buffers.

If I understand it right, you basically just skip the 'vma->vm_page_prot = __get_dma_pgprot(attrs, vma->vm_page_prot);' step and get the right mapping here, i.e. the pgprot value we use for all normal memory.

Just to give you a status update, I tried that too (by adding a dma_mmap_coherent variant that omits the "prot" change, and some printks to verify that it actually does as expected).

Current status is that the ACP behaves exactly like the HP port, which it should not do. If I send data from logic via the ACP port through the L2 cache, using a version of dma_sync that just invalidates the cache could (should?) result in data corruption. Instead, the data gets corrupted only if you do not invalidate the line. This is what the (non-coherent) HP port behaves like, as it writes directly to DDR.

Currently I'm assuming that the tools did something wrong in the bitstream, for example, wiring the "AWCACHE" and similar signals on the ACP to logic "0" instead of "1" while claiming to have wired them to "1" in the UI. A bug like that would also explain the behaviour I'm seeing now.

I'll let you know once I find out more.

Quoted 17 lines · On Thursday 30 April 2015 15:50:15 Mike Looijmans wrote:

Just to give you a status update, I tried that too (by adding a dma_mmap_coherent variant that omits the "prot" change, and some printks to verify that it actually does as expected).

Current status is that the ACP behaves exactly like the HP port, which it should not do. If I send data from logic via the ACP port through the L2 cache, using a version of dma_sync that just invalidates the cache could (should?) result in data corruption. Instead, the data gets corrupted only if you do not invalidate the line. This is what the (non-coherent) HP port behaves like, as it writes directly to DDR.

Currently I'm assuming that the tools did something wrong in the bitstream, for example, wiring the "AWCACHE" and similar signals on the ACP to logic "0" instead of "1" while claiming to have wired them to "1" in the UI. A bug like that would also explain the behaviour I'm seeing now.

Quoted 1 line

I'll let you know once I find out more.

Ok, maybe you have to configure the SCU to include the ACP in the cache coherency? That might not be done by default. I don't really know anything about the SCU or the ACP, so I'm just taking wild guesses here.

Quoted 20 lines · Arnd Bergmann

Quoted 18 lines · On Thursday 30 April 2015 15:50:15 Mike Looijmans wrote:

Just to give you a status update, I tried that too (by adding a dma_mmap_coherent variant that omits the "prot" change, and some printks to verify that it actually does as expected).

Current status is that the ACP behaves exactly like the HP port, which it should not do. If I send data from logic via the ACP port through the L2 cache, using a version of dma_sync that just invalidates the cache could (should?) result in data corruption. Instead, the data gets corrupted only if you do not invalidate the line. This is what the (non-coherent) HP port behaves like, as it writes directly to DDR.

Currently I'm assuming that the tools did something wrong in the bitstream, for example, wiring the "AWCACHE" and similar signals on the ACP to logic "0" instead of "1" while claiming to have wired them to "1" in the UI. A bug like that would also explain the behaviour I'm seeing now.

I'll let you know once I find out more.

Quoted 4 lines

Ok, maybe you have to configure the SCU to include the ACP in the cache coherency? That might not be done by default. I don't really know anything about the SCU or the ACP, so I'm just taking wild guesses here.

The issue seems to be in these signals, the tool falsely claimed to have them set high, but it actually only sets half of them, which thus has no effect at all. I'll be able to test that in an hour or so once the bitstream is ready.

Interestingly, the coherency is not a property of the port or of the device itself, it is a property of the DMA transaction. One single master on this post can do both coherent and non-coherent transfers.
Now there's an interesting challenge, since the kernel appears to assume that one device can only do one type. My 'hardware' actually has multiple masters, which could potentially be tied to different ports or busses.

Quoted 32 lines · On Wednesday 29 April 2015 14:49:26 Mike Looijmans wrote:

Quoted 13 lines · Arnd Bergmann

Quoted 12 lines · On Wednesday 29 April 2015 12:33:00 Mike Looijmans wrote:

Quoted 10 lines · Arnd Bergmann

Quoted 5 lines · On Wednesday 29 April 2015 11:47:37 Mike Looijmans wrote:

Quoted 4 lines · Russell King - ARM Linux

The buffer should also be mapped into userspace with the same memory type and cache attributes as the kernel side mapping. If using ACP, then you probably want "normal memory, cacheable, writeback, read allocate" or in the case of SMP, the same but "read/write allocate".

If my suspicion is correct, we should either change arm_coherent_dma_ops() to refer to a different mmap() callback that does the right thing for coherent devices, or change arm_dma_mmap() to look at dev->is_coherent.

Quoted 8 lines

Quoted 6 lines

Following the route, arch/arm/mm/dma-mapping.c uses pgprot_dmacoherent() which is defined in arch/arm/include/asm/pgtable.h and that just returns uncached memory.

If you can give me some hints as to what the correct flags would be, I can patch my kernel and test it.

Use the flags that Russell listed above.

I would if I had a clue how to do that. For one thing, I don't understand all the L_PTE_... flag juggling that happens here.

I also tried just using kmalloc() to allocate the buffer, and then dma_map_single that. This is what the DMA documentation told me to do for non-coherent mappings. This works fine for the streaming-dma mode using the HP, but using dma-coherent this not only fails to work, it tends to completely lock the system.

Quoted 3 lines

Avoiding the streaming mapping and using only the coherent modes prevents the system locking up, it then doesn't do more harm than just corrupting data in the buffers.

If I understand it right, you basically just skip the 'vma->vm_page_prot = __get_dma_pgprot(attrs, vma->vm_page_prot);' step and get the right mapping here, i.e. the pgprot value we use for all normal memory.

Quoted 8 lines · Arnd Bergmann

Quoted 3 lines · On Wednesday 29 April 2015 14:49:26 Mike Looijmans wrote:

Avoiding the streaming mapping and using only the coherent modes prevents the system locking up, it then doesn't do more harm than just corrupting data in the buffers.

If I understand it right, you basically just skip the 'vma->vm_page_prot = __get_dma_pgprot(attrs, vma->vm_page_prot);' step and get the right mapping here, i.e. the pgprot value we use for all normal memory.

I reverted all my patches and workarounds. Indeed, the kernel needs a "coherent" version of the dma_mmap routine, as the current version will map it as non-cachable, resulting in a big performance hit (and nullifying the whole idea behind it).

I'll test it further on my 'hardware' and cook up a patch that correctly maps the coherent pages.

I reverted all my patches and workarounds. Indeed, the kernel needs a "coherent" version of the dma_mmap routine, as the current version will map it as non-cachable, resulting in a big performance hit (and nullifying the whole idea behind it).

I'll test it further on my 'hardware' and cook up a patch that correctly maps the coherent pages.

Quoted 5 lines · Mike Looijmans

I reverted all my patches and workarounds. Indeed, the kernel needs a "coherent" version of the dma_mmap routine, as the current version will map it as non-cachable, resulting in a big performance hit (and nullifying the whole idea behind it).

Quoted 2 lines

I'll test it further on my 'hardware' and cook up a patch that correctly maps the coherent pages.

Sorry that I have only read this thread briefly, but I wonder if this is what you are looking for:
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-February/325489.html

Quoted 6 lines · Daniel Drake

Quoted 5 lines · Mike Looijmans

I reverted all my patches and workarounds. Indeed, the kernel needs a "coherent" version of the dma_mmap routine, as the current version will map it as non-cachable, resulting in a big performance hit (and nullifying the whole idea behind it).

Quoted 6 lines

Quoted 2 lines

I'll test it further on my 'hardware' and cook up a patch that correctly maps the coherent pages.

Sorry that I have only read this thread briefly, but I wonder if this is what you are looking for:
http://lists.infradead.org/pipermail/linux-arm-kernel/2015-February/325489.html

It's related, but targets another use case. This one does the same in case the driver requested non-consistent memory.

My use case was that I have hardware implemented coherency (through ACP) so the CPU's and device's view on memory is already consistent, regardless of the status of the cache.

The patches are complimentary, not overlapping.

Thanks for the link though, it's something I was also looking into, as I don't always need coherency.

I read the rest of the thread, apparently it was never integrated.

The patch for "non-consistent" is a BUG FIX, not some feature request or so. I was already wondering why my driver had to kalloc pages to get proper caching on it.

From https://www.kernel.org/doc/Documentation/DMA-attributes.txt:
"""
DMA_ATTR_NON_CONSISTENT ... lets the platform to choose to return either consistent or non-consistent memory as it sees fit. By using this API, you are guaranteeing to the platform that you have all the correct and necessary sync points for this memory in the driver.
"""

Quoted 5 lines · Mike Looijmans

I read the rest of the thread, apparently it was never integrated.

The patch for "non-consistent" is a BUG FIX, not some feature request or so. I was already wondering why my driver had to kalloc pages to get proper caching on it.

I disagree.

Quoted 7 lines

From https://www.kernel.org/doc/Documentation/DMA-attributes.txt:
"""
DMA_ATTR_NON_CONSISTENT ... lets the platform to choose to return either consistent or non-consistent memory as it sees fit. By using this API, you are guaranteeing to the platform that you have all the correct and necessary sync points for this memory in the driver.
"""

DMA attributes are something that came in _after_ the DMA API had been around for many years. It's a "new feature" that was added to an existing subsystem, and because there have been no need for it to be implemented on ARM, the new feature was never implemented.

More than that, the vast majority of ARM hardware can't provide this kind of memory, and there are _no_ kernel APIs to ensure that if

cacheable memory were to be returned, they could issue the necessary cache flushes to ensure that the device could see the data.

So it's _not_ a bug fix, and there have been very good reasons not to implement it.

Quoted 22 lines · Russell King - ARM Linux

Quoted 5 lines · Mike Looijmans

I read the rest of the thread, apparently it was never integrated.

The patch for "non-consistent" is a BUG FIX, not some feature request or so. I was already wondering why my driver had to kalloc pages to get proper caching on it.

I disagree.

Quoted 7 lines

From https://www.kernel.org/doc/Documentation/DMA-attributes.txt:
"""
DMA_ATTR_NON_CONSISTENT ... lets the platform to choose to return either consistent or non-consistent memory as it sees fit. By using this API, you are guaranteeing to the platform that you have all the correct and necessary sync points for this memory in the driver.
"""

DMA attributes are something that came in _after_ the DMA API had been around for many years. It's a "new feature" that was added to an existing subsystem, and because there have been no need for it to be implemented on ARM, the new feature was never implemented.

Quoted 2 lines

More than that, the vast majority of ARM hardware can't provide this kind of memory, and there are _no_ kernel APIs to ensure that if

By "non-coherent" memory I thought it meant the same kind of memory that kalloc would return. But from your answer it seems I am mistaken and this is something different?

Quoted 2 lines

cacheable memory were to be returned, they could issue the necessary cache flushes to ensure that the device could see the data.

Then what do the dma_sync_... methods do?

It has been my understanding that one can use dma_map... and dma_sync... methods to make memory ranges visible to the device.

Using dma_sync on coherent memory is just a waste of resources. So how do i allocate memory that I'm supposed to use with dma_sync?

Quoted 23 lines · On Friday 08 May 2015 07:55:26 Mike Looijmans wrote:

Quoted 22 lines · Russell King - ARM Linux

Quoted 5 lines · Mike Looijmans

I read the rest of the thread, apparently it was never integrated.

The patch for "non-consistent" is a BUG FIX, not some feature request or so. I was already wondering why my driver had to kalloc pages to get proper caching on it.

I disagree.

Quoted 7 lines

From https://www.kernel.org/doc/Documentation/DMA-attributes.txt:
"""
DMA_ATTR_NON_CONSISTENT ... lets the platform to choose to return either consistent or non-consistent memory as it sees fit. By using this API, you are guaranteeing to the platform that you have all the correct and necessary sync points for this memory in the driver.
"""

DMA attributes are something that came in _after_ the DMA API had been around for many years. It's a "new feature" that was added to an existing subsystem, and because there have been no need for it to be implemented on ARM, the new feature was never implemented.

Quoted 6 lines

Quoted 2 lines

More than that, the vast majority of ARM hardware can't provide this kind of memory, and there are _no_ kernel APIs to ensure that if

By "non-coherent" memory I thought it meant the same kind of memory that kalloc would return. But from your answer it seems I am mistaken and this is something different?

It depends: on a device that is actually cache-coherent, dma_alloc_coherent() and dma_alloc_noncoherent() both return normal memory.

On some architectures (not ARM) that are not fully coherent, dma_alloc_coherent() has to return uncached memory, while dma_alloc_noncoherent() is allowed to return cached memory but requires a dma_cache_sync() operation.

dma_alloc_attrs() with DMA_ATTR_NON_CONSISTENT is a variant of that, but I assume the idea is that you use dma_sync_single_fo_{cpu,device}() on that memory, which can actually work on ARM, unlike dma_cache_sync().

Quoted 7 lines

Quoted 2 lines

cacheable memory were to be returned, they could issue the necessary cache flushes to ensure that the device could see the data.

Then what do the dma_sync_... methods do?

It has been my understanding that one can use dma_map... and dma_sync... methods to make memory ranges visible to the device.

That is correct, but the DMA_ATTR_NON_CONSISTENT flag is not meaningful with dma_map_...(), as that memory is not assumed to be consistent unless you call dma_sync_...() to start with.

Quoted 2 lines

Using dma_sync on coherent memory is just a waste of resources. So how do i allocate memory that I'm supposed to use with dma_sync?

The traditional API (before the various attributes is):

dma_alloc_coherent()      --> never requires sync
dma_alloc_writecombine()  --> never requires sync, arch specific
dma_alloc_noncoherent()   --> dma_cache_sync(), arch specific

alloc_pages + dma_map_*() --> dma_sync_*

The dma_alloc_coherent() and dma_sync_*() interfaces are supposed to determine themselves whether they need to do any cache management based on whether the device is coherent already or not.

Quoted 35 lines · Arnd Bergmann

Quoted 23 lines · On Friday 08 May 2015 07:55:26 Mike Looijmans wrote:

Quoted 22 lines · Russell King - ARM Linux

Quoted 5 lines · Mike Looijmans

I read the rest of the thread, apparently it was never integrated.

The patch for "non-consistent" is a BUG FIX, not some feature request or so. I was already wondering why my driver had to kalloc pages to get proper caching on it.

I disagree.

Quoted 7 lines

From https://www.kernel.org/doc/Documentation/DMA-attributes.txt:
"""
DMA_ATTR_NON_CONSISTENT ... lets the platform to choose to return either consistent or non-consistent memory as it sees fit. By using this API, you are guaranteeing to the platform that you have all the correct and necessary sync points for this memory in the driver.
"""

DMA attributes are something that came in _after_ the DMA API had been around for many years. It's a "new feature" that was added to an existing subsystem, and because there have been no need for it to be implemented on ARM, the new feature was never implemented.

Quoted 6 lines

Quoted 2 lines

More than that, the vast majority of ARM hardware can't provide this kind of memory, and there are _no_ kernel APIs to ensure that if

By "non-coherent" memory I thought it meant the same kind of memory that kalloc would return. But from your answer it seems I am mistaken and this is something different?

It depends: on a device that is actually cache-coherent, dma_alloc_coherent() and dma_alloc_noncoherent() both return normal memory.

Quoted 8 lines

On some architectures (not ARM) that are not fully coherent, dma_alloc_coherent() has to return uncached memory, while dma_alloc_noncoherent() is allowed to return cached memory but requires a dma_cache_sync() operation.

dma_alloc_attrs() with DMA_ATTR_NON_CONSISTENT is a variant of that, but I assume the idea is that you use dma_sync_single_fo_{cpu,device}() on that memory, which can actually work on ARM, unlike dma_cache_sync().

Ah, okay, I was misled by the names. I was under the impression that memory would be either "coherent" or "non-coherent". But what is called "non-coherent" here is actually something like "less-coherent", it isn't normal memory as alloc_pages would return, but it also isn't completely coherent. Is that a correct summary?

In that case, I stand corrected.

I was looking for an interface that would allocate memory for access by my device, but that would be just alloc_pages style memory. If my DMA controller is limited to say only the first GB of RAM, I'd set the DMA mask to "30 bits". If I just allocate memory using alloc_pages, the kernel doesn't know that I'd want it to be in the lower 1GB range, and could allocate it in a spot my device could not map.

Hence I'd expect there to be some "dma_alloc_pages(struct device* ...)" style of call to get memory that my device could access (and I was under the false impression that dma_alloc_noncoherent was the one I was looking for).

Currently I can get away with just using alloc_pages or kmalloc since my DMA controller happens to be able to access all memory. But I also want my device driver to work on 64-bit platforms (e.g. arm64 for the MPSOC and x86-64 for the PCIe version of the board).

Quoted 51 lines · On Friday 08 May 2015 10:31:53 Mike Looijmans wrote:

Quoted 35 lines · Arnd Bergmann

Quoted 23 lines · On Friday 08 May 2015 07:55:26 Mike Looijmans wrote:

Quoted 22 lines · Russell King - ARM Linux

Quoted 5 lines · Mike Looijmans

I read the rest of the thread, apparently it was never integrated.

The patch for "non-consistent" is a BUG FIX, not some feature request or so. I was already wondering why my driver had to kalloc pages to get proper caching on it.

I disagree.

Quoted 7 lines

From https://www.kernel.org/doc/Documentation/DMA-attributes.txt:
"""
DMA_ATTR_NON_CONSISTENT ... lets the platform to choose to return either consistent or non-consistent memory as it sees fit. By using this API, you are guaranteeing to the platform that you have all the correct and necessary sync points for this memory in the driver.
"""

DMA attributes are something that came in _after_ the DMA API had been around for many years. It's a "new feature" that was added to an existing subsystem, and because there have been no need for it to be implemented on ARM, the new feature was never implemented.

Quoted 6 lines

Quoted 2 lines

More than that, the vast majority of ARM hardware can't provide this kind of memory, and there are _no_ kernel APIs to ensure that if

By "non-coherent" memory I thought it meant the same kind of memory that kalloc would return. But from your answer it seems I am mistaken and this is something different?

It depends: on a device that is actually cache-coherent, dma_alloc_coherent() and dma_alloc_noncoherent() both return normal memory.

Quoted 8 lines

On some architectures (not ARM) that are not fully coherent, dma_alloc_coherent() has to return uncached memory, while dma_alloc_noncoherent() is allowed to return cached memory but requires a dma_cache_sync() operation.

dma_alloc_attrs() with DMA_ATTR_NON_CONSISTENT is a variant of that, but I assume the idea is that you use dma_sync_single_fo_{cpu,device}() on that memory, which can actually work on ARM, unlike dma_cache_sync().

Ah, okay, I was misled by the names. I was under the impression that memory would be either "coherent" or "non-coherent". But what is called "non-coherent" here is actually something like "less-coherent", it isn't normal memory as alloc_pages would return, but it also isn't completely coherent. Is that a correct summary?

Quoted 1 line

In that case, I stand corrected.

Almost. I think the part that you are still missing is that memory itself it not coherent or non-coherent. It's the device access to that memory that can be coherent or not with regard to the CPU.

The memory that is returned by alloc_pages can be coherent with one device but non-coherent with another.

Quoted 6 lines

I was looking for an interface that would allocate memory for access by my device, but that would be just alloc_pages style memory. If my DMA controller is limited to say only the first GB of RAM, I'd set the DMA mask to "30 bits". If I just allocate memory using alloc_pages, the kernel doesn't know that I'd want it to be in the lower 1GB range, and could allocate it in a spot my device could not map.

If you have that low-memory restriction, you also need to ensure that there is a ZONE_DMA that is large enough. ZONE_DMA should be sized to match the common subset that all devices can access, so a GFP_DMA request returns memory that is guaranteed to be accessible by all devices.

Quoted 4 lines

Hence I'd expect there to be some "dma_alloc_pages(struct device* ...)" style of call to get memory that my device could access (and I was under the false impression that dma_alloc_noncoherent was the one I was looking for).

Quoted 4 lines

Currently I can get away with just using alloc_pages or kmalloc since my DMA controller happens to be able to access all memory. But I also want my device driver to work on 64-bit platforms (e.g. arm64 for the MPSOC and x86-64 for the PCIe version of the board).

Those machines will have ZONE_DMA32, which refers to the first 4GB of memory, so that should work fine. Alternatively you can use the iommu to get all memory mapped into the space that is accessible by the device.

Also, on 64-bit x86 or ARM machines, all memory tends to be coherent, so dma_sync_* will turn into a nop.

Quoted 5 lines

If you have that low-memory restriction, you also need to ensure that there is a ZONE_DMA that is large enough. ZONE_DMA should be sized to match the common subset that all devices can access, so a GFP_DMA request returns memory that is guaranteed to be accessible by all devices.

GFP_DMA still carries that scary remark: "on x86, GFP_DMA guarantees to be within the first 16MB of available bus addresses". Which is really scary if you want to allocate multimegabyte buffers.

Quoted 22 lines · Russell King - ARM Linux

Quoted 5 lines · Mike Looijmans

I read the rest of the thread, apparently it was never integrated.

The patch for "non-consistent" is a BUG FIX, not some feature request or so. I was already wondering why my driver had to kalloc pages to get proper caching on it.

I disagree.

Quoted 7 lines

From https://www.kernel.org/doc/Documentation/DMA-attributes.txt:
"""
DMA_ATTR_NON_CONSISTENT ... lets the platform to choose to return either consistent or non-consistent memory as it sees fit. By using this API, you are guaranteeing to the platform that you have all the correct and necessary sync points for this memory in the driver.
"""

DMA attributes are something that came in _after_ the DMA API had been around for many years. It's a "new feature" that was added to an existing subsystem, and because there have been no need for it to be implemented on ARM, the new feature was never implemented.

Quoted 2 lines

More than that, the vast majority of ARM hardware can't provide this kind of memory, and there are _no_ kernel APIs to ensure that if

By "non-coherent" memory I thought it meant the same kind of memory that kalloc would return. But from your answer it seems I am mistaken and this is something different?

From the description of DMA_ATTR_NON_CONSISTENT talks about returning

Well, I do find the language used rather loose. The term "coherent" is well understood, but when the documentation mixes it with "consistent" it becomes less clear whether the two terms mean the same thing or something different.

Quoted 12 lines · On Wednesday 29 April 2015 11:47:37 Mike Looijmans wrote:

Quoted 10 lines · Russell King - ARM Linux

Quoted 6 lines · Arnd Bergmann

You still need to synchronize MMIO register accesses with write buffers, as the readl() and writel() functions do in the kernel.

In particular, after you have written a buffer to memory from the CPU, you will need to do an outer_sync() before the MMIO write that triggers the DMA. This is still much cheaper than doing the cache flush though.

Note that outer_sync() is already done by readl/writel and/or the write memory barriers (mb()/wmb()).

Quoted 6 lines

I initiate the DMA transfers using iowrite32() so if I understand correctly, I'm already doing the right thing here.

Just to be completely clear, there is no direct register access from user space, the driver does all MMIO. Userspace only gets an mmap for DMA buffers, and uses ioctl to initiate transfers.

Ok, that seems all fine then.

Quoted 5 lines

Quoted 5 lines

Quoted 4 lines

Another possible problem would be if the driver mmaps the buffer in uncached mode to user space. This is something your kernel driver has to get right, it won't be handled automatically by setting the "dma-coherent" property in DT.

Quoted 8 lines

Quoted 4 lines

The buffer should also be mapped into userspace with the same memory type and cache attributes as the kernel side mapping. If using ACP, then you probably want "normal memory, cacheable, writeback, read allocate" or in the case of SMP, the same but "read/write allocate".

I currently use dma_alloc_coherent() to allocate buffers and dma_mmap_coherent() to map them to user space. I was under the assumption that these would do the right thing. Is that correct? If not, then what should I use?

dma_mmap_coherent() is the right interface, but I've just looked at the implementation of arm_dma_mmap() and I'm not sure that it actually uses the correct vma->vm_page_prot value here, because I don't see where it takes into account whether the device is coherent or not. Most ARM machines have only noncoherent devices, and dma_mmap_coherent() is used rarely by drivers, so it's quite possible that this interface got broken without anybody noticing.

Quoted 5 lines · Arnd Bergmann

Quoted 3 lines · On Wednesday 29 April 2015 11:47:37 Mike Looijmans wrote:

I currently use dma_alloc_coherent() to allocate buffers and dma_mmap_coherent() to map them to user space. I was under the assumption that these would do the right thing. Is that correct? If not, then what should I use?

Quoted 7 lines

dma_mmap_coherent() is the right interface, but I've just looked at the implementation of arm_dma_mmap() and I'm not sure that it actually uses the correct vma->vm_page_prot value here, because I don't see where it takes into account whether the device is coherent or not. Most ARM machines have only noncoherent devices, and dma_mmap_coherent() is used rarely by drivers, so it's quite possible that this interface got broken without anybody noticing.

My driver also supports 'classic' read/write by copying data to/from userspace using a DMA ringbuffer, which is also allocated using dma_alloc_coherent. That would suggest that the kernel mapping for this memory is also incorrect, I also get corrupted data in these transfers. These buffers are not being mapped to userspace at all. These tests all use page aligned transfer sizes.

Quoted 14 lines · On Wednesday 29 April 2015 13:09:05 Mike Looijmans wrote:

Quoted 5 lines · Arnd Bergmann

Quoted 3 lines · On Wednesday 29 April 2015 11:47:37 Mike Looijmans wrote:

I currently use dma_alloc_coherent() to allocate buffers and dma_mmap_coherent() to map them to user space. I was under the assumption that these would do the right thing. Is that correct? If not, then what should I use?

Quoted 7 lines

dma_mmap_coherent() is the right interface, but I've just looked at the implementation of arm_dma_mmap() and I'm not sure that it actually uses the correct vma->vm_page_prot value here, because I don't see where it takes into account whether the device is coherent or not. Most ARM machines have only noncoherent devices, and dma_mmap_coherent() is used rarely by drivers, so it's quite possible that this interface got broken without anybody noticing.

Quoted 5 lines

My driver also supports 'classic' read/write by copying data to/from userspace using a DMA ringbuffer, which is also allocated using dma_alloc_coherent. That would suggest that the kernel mapping for this memory is also incorrect, I also get corrupted data in these transfers. These buffers are not being mapped to userspace at all. These tests all use page aligned transfer sizes.

Ok, if it's broken when you only use a kernel mapping and no user space mapping, arm_dma_mmap() is not your (only) problem, but we should probably still fix that if it's broken.

Quoted 21 lines · Arnd Bergmann

Quoted 19 lines · On Wednesday 29 April 2015 13:09:05 Mike Looijmans wrote:

Quoted 5 lines · Arnd Bergmann

Quoted 3 lines · On Wednesday 29 April 2015 11:47:37 Mike Looijmans wrote:

I currently use dma_alloc_coherent() to allocate buffers and dma_mmap_coherent() to map them to user space. I was under the assumption that these would do the right thing. Is that correct? If not, then what should I use?

Quoted 7 lines

dma_mmap_coherent() is the right interface, but I've just looked at the implementation of arm_dma_mmap() and I'm not sure that it actually uses the correct vma->vm_page_prot value here, because I don't see where it takes into account whether the device is coherent or not. Most ARM machines have only noncoherent devices, and dma_mmap_coherent() is used rarely by drivers, so it's quite possible that this interface got broken without anybody noticing.

My driver also supports 'classic' read/write by copying data to/from userspace using a DMA ringbuffer, which is also allocated using dma_alloc_coherent. That would suggest that the kernel mapping for this memory is also incorrect, I also get corrupted data in these transfers. These buffers are not being mapped to userspace at all. These tests all use page aligned transfer sizes.

Quoted 3 lines

Ok, if it's broken when you only use a kernel mapping and no user space mapping, arm_dma_mmap() is not your (only) problem, but we should probably still fix that if it's broken.

The only difference in arm_coherent_dma_alloc() and arm_dma_alloc() is that the former passes a "true" to __dma_alloc(). That looks prettry suspicious, I'd at least expect different "prot" values.