MAX_DMA_ADDRESS in include/asm/asm-i386/dma.h (2.6.x and 2.4.x)
While writing some DMA page gathering code, I realize that __get_free_page or kmalloc does not return memory more than 16 MB (typically around 11-12 MB even if it done right after a reboot.)
Since this is for a PCI device (A Fibre channel HBA), I remembered that the book Linux Device Driver, edition 2 mentions that the 16 MB limit is for DMA with ISA devices, while PCI DMA can access upto 950 MB or so, using 32 bit addresses.
Digging further into the code, I found that this being a Intel PC (a P4 2.8 GHz with 1GB RAM) with High Mem for 4 GB enabled kernel, the define in linux/include/asm/asm-i386/dma.h for MAX_DMA_ADDRESS is probably what is limiting this. The #define is the same in 2.6.8-rc1 as well as 2.4.x (2.4.20 and 2.4.25 is what I checked.)
/* The maximum address that we can perform a DMA transfer to on this platform */
#define MAX_DMA_ADDRESS (PAGE_OFFSET+0x1000000)
From what I can understand so far, this implies 0xC000 0000 + +0x1000000 = A total of 16 MB
I changed this to 256 MB which was less than the limit in the book and enough for the test code purpose
#define MAX_DMA_ADDRESS (PAGE_OFFSET+0x10000000)
After running some FC traffic and delibrately not free'ing memory, the memory above 16 MB was DMA'abled and the data was read and written correctly.
So, the question is, can someone DMA and x86 architecture expert comment on this? Am I missing something?
And if the above code is ok, is there a way to have the above change in the main kernel so that it works with ISA and PCI devices?
Thanks
Amit
On Jul 16 2004, at 14:37, Amit D. Chaudhary was caught saying:
Quoted 8 lines
While writing some DMA page gathering code, I realize that
__get_free_pageor kmalloc does not return memory more than 16 MB (typically around 11-12 MB even if it done right after a reboot.)Since this is for a PCI device (A Fibre channel HBA), I remembered that the book Linux Device Driver, edition 2 mentions that the 16 MB limit is for DMA with ISA devices, while PCI DMA can access upto 950 MB or so, using 32 bit addresses.
Using __get_free_page() or kmalloc() for device DMA'ble descriptors (I am guessing that's what you are doing) is wrong. See Documentation/DMA-API.txt and Documentation/DMA-mapping.txt for the proper way to do this.
~Deepak
Signature · 4 lines
Deepak Saxena - dsaxena at plexity dot net - http://www.plexity.net/ "Unlike me, many of you have accepted the situation of your imprisonment and will die here like rotten cabbages." - Number 6
Deepak,
I am missing what you are directing me to.
If it is,pci_alloc_consistent(), linux-2.4.25/arch/i386/kernel/pci-dma.c dma_alloc_coherent(), linux-2.6.8-rc1/arch/i386/kernel/pci-dma.c
They internally seem to __get_free_pages()
If you meant,
pci_pool_create()\pci_pool_alloc
the code in question does not need fixed size buffers like kmem_cache_alloc.
Here is the actual usage planned,
The DMA buffer is a variable size buffer (typically 1k to 128K) represent a read or write from an FC initiator. This memory is passed as data segments (sg list) with various CTIOs (Type of IOCB on Qlogic FC HBA chips). For a write command, the memory is used to read the data into the memory and then pass it to a user application without any copy (think mmap.) This is part of implementing a virtual targets in a Target mode device driver.
The memory need not be page size, as a matter of fact, using a large consecutive block, for example using alloc_bootmem_low() during kernel bootup, will simplify the data transfer and result in no internal fragmentation, it does introduce inflexibility in changing the size and other issues.
Hopefully that sheds more light on the topic.
Amit
Quoted 13 lines · Deepak Saxena
On Jul 16 2004, at 14:37, Amit D. Chaudhary was caught saying:
Quoted 3 lines
While writing some DMA page gathering code, I realize that
__get_free_pageor kmalloc does not return memory more than 16 MB (typically around 11-12 MB even if it done right after a reboot.)Since this is for a PCI device (A Fibre channel HBA), I remembered that the book Linux Device Driver, edition 2 mentions that the 16 MB limit is for DMA with ISA devices, while PCI DMA can access upto 950 MB or so, using 32 bit addresses.
Using
__get_free_page()orkmalloc()for device DMA'ble descriptors (I am guessing that's what you are doing) is wrong. SeeDocumentation/DMA-API.txtandDocumentation/DMA-mapping.txtfor the proper way to do this.~Deepak
On Jul 16 2004, at 15:11, Amit D. Chaudhary was caught saying:
Quoted 9 lines
Deepak,
I am missing what you are directing me to.
If it is,
pci_alloc_consistent(),linux-2.4.25/arch/i386/kernel/pci-dma.cdma_alloc_coherent(),linux-2.6.8-rc1/arch/i386/kernel/pci-dma.cThey internally seem to
__get_free_pages()
Correct, but take a second look at the code (2.6):
void *ret;
/* ignore region specifiers */
gfp &= ~(__GFP_DMA | __GFP_HIGHMEM);
if (dev == NULL || (dev->coherent_dma_mask < 0xffffffff))
gfp |= GFP_DMA;
ret = (void *)__get_free_pages(gfp, get_order(size));It uses GFP_DMA iff your coherent_dma_mask is != 0xffffffff. Assuming your device can address a the full 32-bit PCI address space, you need to set the coherent_dma_mask appropriately and you will get buffers from all addressable lowmem. I don't do much x86, so not sure how you go about allocating highmem DMA buffers.
Quoted 5 lines
The memory need not be page size, as a matter of fact, using a large consecutive block, for example using
alloc_bootmem_low()during kernel bootup, will simplify the data transfer and result in no internal fragmentation, it does introduce inflexibility in changing the size and other issues.
If you are using alloc_bootmem_low(), all you should have to do after allocating the memory is call pci_dma_map_single()/map_sg() to get PCI-DMA addresses. You still should have no reason to touch MAX_DMA_ADDRESS.
~Deepak
Signature · 4 lines
Deepak Saxena - dsaxena at plexity dot net - http://www.plexity.net/ "Unlike me, many of you have accepted the situation of your imprisonment and will die here like rotten cabbages." - Number 6
Quoted 24 lines · Deepak Saxena
On Jul 16 2004, at 15:11, Amit D. Chaudhary was caught saying:
Quoted 9 lines
Deepak,
I am missing what you are directing me to.
If it is,
pci_alloc_consistent(),linux-2.4.25/arch/i386/kernel/pci-dma.cdma_alloc_coherent(),linux-2.6.8-rc1/arch/i386/kernel/pci-dma.cThey internally seem to
__get_free_pages()Correct, but take a second look at the code (2.6):
void *ret; /* ignore region specifiers */ gfp &= ~(__GFP_DMA | __GFP_HIGHMEM); if (dev == NULL || (dev->coherent_dma_mask < 0xffffffff)) gfp |= GFP_DMA; ret = (void *)__get_free_pages(gfp, get_order(size));
Quoted 5 lines
It uses
GFP_DMAiff yourcoherent_dma_maskis !=0xffffffff. Assuming your device can address a the full 32-bit PCI address space, you need to set thecoherent_dma_maskappropriately and you will get buffers from all addressable lowmem. I don't do much x86, so not sure how you go about allocating highmem DMA buffers.
Thanks, noted and verified.
This chip cannot DMA with a memory buffer returned by kmalloc without a GFP_DMA flag, that is memory addresses like 0xf67e0000, it works with 0xcxxx xxxx.
I verified it by modifying the code and trying it out.
Quoted 5 lines
Quoted 1 line
The memory need not be page size, as a matter of fact, using a large consecutive block, for example using
alloc_bootmem_low()during kernel bootup, will simplify the data transfer and result in no internal fragmentation, it does introduce inflexibility in changing the size and other issues.If you are using
alloc_bootmem_low(), all you should have to do after
allocating the memory is call pci_dma_map_single()/map_sg() to get PCI-DMA addresses. You still should have no reason to touchMAX_DMA_ADDRESS.
This was a backup approach, I mentioned to provide details about the memory being allocated. I would like to avoid this approach. See reasons above.
Amit
Deepak,
Thanks a lot for the inputs. kmalloc with GFP_DMA is not to be used for pci DMA.
I continued trying it and it does work, there was some alignment for the header which was incorrect and that I limited the size to 128k.
Amit
Quoted 52 lines · Amit D. Chaudhary
Quoted 25 lines · Deepak Saxena
On Jul 16 2004, at 15:11, Amit D. Chaudhary was caught saying:
Quoted 9 lines
Deepak,
I am missing what you are directing me to.
If it is,
pci_alloc_consistent(),linux-2.4.25/arch/i386/kernel/pci-dma.cdma_alloc_coherent(),linux-2.6.8-rc1/arch/i386/kernel/pci-dma.cThey internally seem to
__get_free_pages()Correct, but take a second look at the code (2.6):
void *ret; /* ignore region specifiers */ gfp &= ~(__GFP_DMA | __GFP_HIGHMEM); if (dev == NULL || (dev->coherent_dma_mask < 0xffffffff)) gfp |= GFP_DMA; ret = (void *)__get_free_pages(gfp, get_order(size));Quoted 5 lines
It uses
GFP_DMAiff yourcoherent_dma_maskis !=0xffffffff. Assuming your device can address a the full 32-bit PCI address space, you need to set thecoherent_dma_maskappropriately and you will get buffers from all addressable lowmem. I don't do much x86, so not sure how you go about allocating highmem DMA buffers.Thanks, noted and verified.
This chip cannot DMA with a memory buffer returned by kmalloc without a
GFP_DMAflag, that is memory addresses like0xf67e0000, it works with 0xcxxx xxxx.I verified it by modifying the code and trying it out.
Quoted 6 lines
Quoted 1 line
The memory need not be page size, as a matter of fact, using a large consecutive block, for example using
alloc_bootmem_low()during kernel bootup, will simplify the data transfer and result in no internal fragmentation, it does introduce inflexibility in changing the size and other issues.If you are using
alloc_bootmem_low(), all you should have to do after
allocating the memory is call pci_dma_map_single()/map_sg() to get PCI-DMA addresses. You still should have no reason to touchMAX_DMA_ADDRESS.This was a backup approach, I mentioned to provide details about the memory being allocated. I would like to avoid this approach. See reasons above.
Amit