DMA descriptor alignment
For those variants of BCM43xx cards that use 64-bit DMA, there is a requirement that all descriptor rings must be aligned on an 8K boundary and must fit within an 8K page. On the x86_64 architecture where the page size is 4K, I was getting addresses like 0x67AF000 when using dma_alloc_coherent calls. From the description of the dma_pool_create and dma_pool_allocate routines, I thought they would fix my problems; however, even with a dma_pool_create(name, dev, 8192, 8192, 8192) call, I'm still getting 4K rather than 8K alignment, which results in DMA errors.
Is there a bug in these routines, am I using them incorrectly, or do I have a misunderstanding of what it takes to get this kind of alignment?
Thanks,
Larry
Quoted 4 lines · Larry Finger
For those variants of BCM43xx cards that use 64-bit DMA, there is a requirement that all descriptor rings must be aligned on an 8K boundary and must fit within an 8K page. On the
x86_64architecture where the page size is 4K, I was getting addresses like0x67AF000when usingdma_alloc_coherentcalls.
Normally x86-64 dma_alloc_coherent calls the buddy allocator which gives you always naturally aligned blocks. But there is a fallback calling into swiotlb and swiotlb uses best fit allocation which only guarantees single page alignment. That is probably what you're seeing.
My dma zone rework would remove that fallback case and should make it work.
Quoted 3 lines
From the description of the
dma_pool_createanddma_pool_allocateroutines, I thought they would fix my problems; however, even with adma_pool_create(name, dev, 8192, 8192, 8192)call, I'm still getting 4K rather than 8K alignment, which results in DMA errors.
They cannot give you more alignment than the underlying allocator.
Quoted 2 lines
Is there a bug in these routines, am I using them incorrectly, or do I have a misunderstanding of what it takes to get this kind of alignment?
My suggestion as a short term workaround would be to first allocate 8K using dma_alloc_coherent and if that has the wrong alignment get 16K and align yourself. When the driver is loaded later that might be unreliable, but near boot or with enough free memory using order 2 should usually work.
With the dma zone rework that could be removed later.
-Andi
Quoted 23 lines · Andi Kleen
Quoted 4 lines · Larry Finger
For those variants of BCM43xx cards that use 64-bit DMA, there is a requirement that all descriptor rings must be aligned on an 8K boundary and must fit within an 8K page. On the
x86_64architecture where the page size is 4K, I was getting addresses like0x67AF000when usingdma_alloc_coherentcalls.Normally x86-64
dma_alloc_coherentcalls the buddy allocator which gives you always naturally aligned blocks. But there is a fallback calling into swiotlb and swiotlb uses best fit allocation which only guarantees single page alignment. That is probably what you're seeing.My dma zone rework would remove that fallback case and should make it work.
Quoted 3 lines
From the description of the
dma_pool_createanddma_pool_allocateroutines, I thought they would fix my problems; however, even with adma_pool_create(name, dev, 8192, 8192, 8192)call, I'm still getting 4K rather than 8K alignment, which results in DMA errors.They cannot give you more alignment than the underlying allocator.
Quoted 2 lines
Is there a bug in these routines, am I using them incorrectly, or do I have a misunderstanding of what it takes to get this kind of alignment?
Quoted 6 lines
My suggestion as a short term workaround would be to first allocate 8K using
dma_alloc_coherentand if that has the wrong alignment get 16K and align yourself. When the driver is loaded later that might be unreliable, but near boot or with enough free memory using order 2 should usually work.With the dma zone rework that could be removed later.
I will be most interested in the dma zone rework; however, I now have the descriptors properly allocated merely by asking for 8K, just as I had thought it should work. I'm not quite sure what was wrong before, but I added a test to make certain that the alignment is OK just in case the problem comes back.
Thanks for your response,
Larry