DMA memory limitation?
Hi,
Is there any limitation on DMA memory we can allocate using kmalloc(size, GFP_DMA)? I am not able to acquire more than 14MB of the mem using this on my PCI SMP box with 256MB ram. I think there is restriction on ISA boards of 16MB. Can we increase it ?
thx,
Vasu
Quoted 8 lines · Vasu Varma P V
Hi,
Is there any limitation on DMA memory we can allocate using
kmalloc(size, GFP_DMA)? I am not able to acquire more than 14MB of the mem using this on my PCI SMP box with 256MB ram. I think there is restriction on ISA boards of 16MB. Can we increase it ?
Why?
YOu don't have to allocate GFP_DMA memory for PCI cards! GFP_DMA is for ISA cards only
Quoted 4 lines
Is there any limitation on DMA memory we can allocate using
kmalloc(size, GFP_DMA)? I am not able to acquire more than 14MB of the mem using this on my PCI SMP box with 256MB ram. I think there is restriction on ISA boards of 16MB.
Yes. GFP_DMA allocates memory below 16Mbytes for ISA devices and that is a physical wiring issue. For PCI devices you can allocate a lot more using the pci_alloc_* and pci_map_* interface to allocate memory for PCI in a CPU independant manner. It'll also then look after cache coherency issues for you
but we have a macro in include/asm-i386/dma.h,MAX_DMA_ADDRESS (PAGE_OFFSET+0x1000000).
if i change it to a higher value, i am able to get more dma memory. Is there any way i can change this without compiling the kernel?
Quoted 13 lines · Arjan van de Ven
Quoted 8 lines · Vasu Varma P V
Hi,
Is there any limitation on DMA memory we can allocate using
kmalloc(size, GFP_DMA)? I am not able to acquire more than 14MB of the mem using this on my PCI SMP box with 256MB ram. I think there is restriction on ISA boards of 16MB. Can we increase it ?Why?
YOu don't have to allocateGFP_DMAmemory for PCI cards!GFP_DMAis for ISA cards only
Quoted 10 lines · Vasu Varma P V
Hi,
Is there any limitation on DMA memory we can allocate using
kmalloc(size, GFP_DMA)? I am not able to acquire more than 14MB of the mem using this on my PCI SMP box with 256MB ram. I think there is restriction on ISA boards of 16MB. Can we increase it ?thx,
Vasu
14MB of DMA(able) memory? Err. I think you are trying to do something you would never need to do.
If your board has a PCI interface, it's got address-space allocated where memory does not exist. That's for communicating with the board. If this address-space contains memory on that board, you treat it just like RAM. That's what the PCI bus does for you.
Given this, your board may also be a "bus mastering" board. This means that, given the correct programming sequence, it can transfer data to or from any location accessible to the PCI bus. This is the so-called DMA that PCI bus masters can perform.
When you program such a board, you simply translate the virtual address of the transfer area to the address that the board understands. Look at /linux/Documentation/IO-mapping.txt
The GFP_DMA memory type to which you refer, is for boards that must use the lower 16 MB of address space. Such boards are usually on the ISA bus. Since the first 1 MB is pretty much taken by ROM BIOS, screen-card BIOS, other controllers (like SCSI) BIOS, etc., you are not going to increase it.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
I was going to compile a list of innovations that could be attributed to Microsoft. Once I realized that Ctrl-Alt-Del was handled in the BIOS, I found that there aren't any.
Quoted 12 lines · Vasu Varma P V
Quoted 9 lines
Hi,
Is there any limitation on DMA memory we can allocate using
kmalloc(size, GFP_DMA)? I am not able to acquire more than 14MB of the mem using this on my PCI SMP box with 256MB ram. I think there is restriction on ISA boards of 16MB. Can we increase it ?
Quoted 6 lines
Quoted 2 lines
thx,
Vasu14MB of DMA(able) memory? Err. I think you are trying to do something you would never need to do.
And what is that supposed to be????
I have a piece of pretty well designed hardware here... and my application requires me to have the PCI board to random access in master mode a whole lot of memory (anywhere from 128MEGS to 1GIG... and possibly more...) so I really do need BIG DMA buffers (I don't say huge anymore as one can get 1/2 Gig of RAM for just over 120 bucks???)...
There is no way I can have the piece of hardware behave in another fashion... and NO it is NOT broken (when you do BOTH hardware & software, you know about BOTH limitations... there are just some cases where you have to face some unique issues). the mem=1024M works pretty fine once you have figured out how to handle in a fairly simple way this "reserved" memory...
So please software people... do not blame it all on the hardware... =)
Chris.
Quoted 12 lines · Christophe Beaumont
Quoted 12 lines · Vasu Varma P V
Quoted 9 lines
Hi,
Is there any limitation on DMA memory we can allocate using
kmalloc(size, GFP_DMA)? I am not able to acquire more than 14MB of the mem using this on my PCI SMP box with 256MB ram. I think there is restriction on ISA boards of 16MB. Can we increase it ?
Quoted 8 lines
Quoted 6 lines
Quoted 2 lines
thx,
Vasu14MB of DMA(able) memory? Err. I think you are trying to do something you would never need to do.
And what is that supposed to be????
Did you READ the rest of the carefully-worded paragraphs, carefully, explaining that it ** ISN'T DMA ** memory that you need?
Quoted 3 lines
I have a piece of pretty well designed hardware here... and my application requires me to have the PCI board to random access in master mode a whole lot of memory (anywhere from 128MEGS to
^^^^^^^^^^^
You JUST give it the address of any RAM you want to read/write... as explained before. As explained, in a documentation file I specified, you convert the virtual address using a macro that the kernel headers provide.
You can even give it user-mode buffers as long as they are non-paged during the access.
Quoted 11 lines
1GIG... and possibly more...) so I really do need BIG DMA buffers (I don't say huge anymore as one can get 1/2 Gig of RAM for just over 120 bucks???)...
There is no way I can have the piece of hardware behave in another fashion... and NO it is NOT broken (when you do BOTH hardware & software, you know about BOTH limitations... there are just some cases where you have to face some unique issues). the mem=1024M works pretty fine once you have figured out how to handle in a fairly simple way this "reserved" memory...
This is wrong! Good luck sucker.
Quoted 1 line
So please software people...
^^^^^^^
I am a Software Engineer, with a long history of hardware design.
Cheers,
Dick Johnson
Penguin : Linux version 2.4.1 on an i686 machine (799.53 BogoMips).
I was going to compile a list of innovations that could be attributed to Microsoft. Once I realized that Ctrl-Alt-Del was handled in the BIOS, I found that there aren't any.
Quoted 8 lines · Vasu Varma P V
Hi,
Is there any limitation on DMA memory we can allocate using
kmalloc(size, GFP_DMA)? I am not able to acquire more than 14MB of the mem using this on my PCI SMP box with 256MB ram. I think there is restriction on ISA boards of 16MB. Can we increase it ?
You can allocate a lot more memory for your pci activities. No problem there. Just drop the "GFP_DMA" and you'll get up to 1G or so.
You shouldn't use GFP_DMA because PCI cards don't need that. Only ISA cards needs GFP_DMA because they can't use more than 16M. So obviously GFP_DMA is limited to
16M because it is really ISA_DMA.
PCI don't need such special tricks, so don't use GFP_DMA! Your PCI cards is able to DMA into any memory, including the non-GFP_DMA memory.
Quoted 7 lines
but we have a macro in
include/asm-i386/dma.h,MAX_DMA_ADDRESS(PAGE_OFFSET+0x1000000).if i change it to a higher value, i am able to get more dma memory. Is there any way i can change this without compiling the kernel?
No matter what you do, DON'T change that. Yeah, you'll get a bigger GFP_DMA pool, but that'll break each and every ISA card that tries to allocate GFP_DMA memory. You achieve exactly the same effect for your PCI card by ditching the GFP_DMA parameter, but then you achieve it without breaking ISA cards.
Helge Hafting
Quoted 31 lines · Helge Hafting
Quoted 8 lines · Vasu Varma P V
Hi,
Is there any limitation on DMA memory we can allocate using
kmalloc(size, GFP_DMA)? I am not able to acquire more than 14MB of the mem using this on my PCI SMP box with 256MB ram. I think there is restriction on ISA boards of 16MB. Can we increase it ?You can allocate a lot more memory for your pci activities. No problem there. Just drop the "
GFP_DMA" and you'll get up to 1G or so.You shouldn't use
GFP_DMAbecause PCI cards don't need that. Only ISA cards needsGFP_DMAbecause they can't use more than 16M. So obviouslyGFP_DMAis limited to
16M because it is reallyISA_DMA.PCI don't need such special tricks, so don't use
GFP_DMA! Your PCI cards is able to DMA into any memory, including the non-GFP_DMA memory.Quoted 7 lines
but we have a macro in
include/asm-i386/dma.h,MAX_DMA_ADDRESS(PAGE_OFFSET+0x1000000).if i change it to a higher value, i am able to get more dma memory. Is there any way i can change this without compiling the kernel?
Quoted 7 lines
No matter what you do, DON'T change that. Yeah, you'll get a bigger
GFP_DMApool, but that'll break each and every ISA card that tries to allocateGFP_DMAmemory. You achieve exactly the same effect for your PCI card by ditching theGFP_DMAparameter, but then you achieve it without breaking ISA cards.
A problem arises on 64 bit platforms (such as IA64) if your PCI card is only 32bit (can address the first 4G) and you don't wan't to use bounce buffers. If you use GFP_DMA on IA64 you are ensured that the memory you get is below 4G and not 16M as on i386, hence no bounce buffers are needed. On Alpha GFP_DMA is not limited at all (I think). Correct me if I'm wrong, but I really think there should be a general way of allocating memory that is 32bit addressable (something like GFP_32BIT?) so you don't need a lot of #ifdef's in your code.
Regards,
Signature · 4 lines
Steffen Persvold Systems Engineer Email : mailto:[email protected] Scali AS (http://www.scali.com) Tlf : (+47) 22 62 89 50 Olaf Helsets vei 6 Fax : (+47) 22 62 89 51 N-0621 Oslo, Norway
Quoted 2 lines
A problem arises on 64 bit platforms (such as IA64) if your PCI card is only 32bit (can address the first 4G) and you don't wan't to use bounce buffers. If you use
GFP_DMAon
GFP_DMA is ISA dma reachable, Forget the IA64, their setup is weird and should best be ignored until 2.5 as and when they sort it out.
Quoted 1 line
bounce buffers are needed. On Alpha
GFP_DMAis not limited at all (I think). Correct me if
Alpha has various IOMMU facilities
Quoted 3 lines
I'm wrong, but I really think there should be a general way of allocating memory that is 32bit addressable (something like
GFP_32BIT?) so you don't need a lot of #ifdef's in your code.
No ifdefs are needed
GFP_DMA - ISA dma reachable pci_alloc_* and friends - PCI usable memory
Quoted 4 lines · Alan Cox
No ifdefs are needed
GFP_DMA- ISA dma reachable pci_alloc_* and friends - PCI usable memory
pci_alloc_* is designed to support ISA.
Pass pci_dev==NULL to pci_alloc_* for ISA devices, and it allocs GFP_DMA for you.
Signature · 3 lines
Jeff Garzik | A recent study has shown that too much soup Building 1024 | can cause malaise in laboratory mice. MandrakeSoft |
Quoted 2 lines
Quoted 2 lines
GFP_DMAis ISA dma reachable, Forget the IA64, their setup is weird and should best be ignored until 2.5 as and when they sort it out.
Really ? I don't think I can ignore IA64, there are people who ask for it....
Quoted 9 lines
Quoted 9 lines
Quoted 1 line
bounce buffers are needed. On Alpha
GFP_DMAis not limited at all (I think). Correct me ifAlpha has various IOMMU facilities
Quoted 3 lines
I'm wrong, but I really think there should be a general way of allocating memory that is 32bit addressable (something like
GFP_32BIT?) so you don't need a lot of #ifdef's in your code.No ifdefs are needed
Quoted 8 lines
Quoted 2 lines
GFP_DMA- ISA dma reachable pci_alloc_* and friends - PCI usable memorypci_alloc_* is designed to support ISA.
Pass pci_dev==NULL to pci_alloc_* for ISA devices, and it allocs
GFP_DMAfor you.
Sure, but the IA64 platforms that are out now doesn't have an IOMMU, so bounce buffers are used if you don't specify GFP_DMA in your get_free_page.
Now lets say you have a driver with a page allocator. Eventually you want to make some if the allocated pages available to a 32bit PCI device. These pages has to be consistent (i.e the driver doesn't have to wait for a PCI flush for the data to be valid, sort of like a ethernet ring buffer). I could use the pci_alloc_consistent() function (pci_alloc_consistent() allocates a buffer with GFP_DMA on IA64), but since I already have the pages, I have to use pci_map_single (or pci_map_sg). Inside pci_map_single on IA64 something called swiotlb buffers (bounce buffers) are used if the device can't support 64bit addressing and the address of the memory to map is above the 4G limit. The swiotlb buffers are below the 4G limit and therefore reachable by any PCI device. The problem about these buffers are that the content are not copied to the original location before you do a pci_sync_* or a pci_unmap_* (they are not consistent) and they are a limited resource (allocated at boot time). My solution for now was to use :
#if defined(__ia64__)
int flag = GFP_DMA;
#else
int flag = 0;
#endif
Maybe IA64 could implement GFP_HIGHMEM (as on i386) so that if no flags were used you were guaranteed to get 32bit memory ???
Regards
Signature · 4 lines
Steffen Persvold Systems Engineer Email : mailto:[email protected] Scali AS (http://www.scali.com) Tlf : (+47) 22 62 89 50 Olaf Helsets vei 6 Fax : (+47) 22 62 89 51 N-0621 Oslo, Norway
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines
GFP_DMAis ISA dma reachable, Forget the IA64, their setup is weird and should best be ignored until 2.5 as and when they sort it out.Really ? I don't think I can ignore IA64, there are people who ask for it....
Well the current IA64 tree isnt related to the standard Linux behaviour so you need to take that up with the IA64 people. Obviously their non standard behaviour causes problems.
Quoted 1 line
Quoted 7 lines · Steffen Persvold
Quoted 4 lines
pci_alloc_* is designed to support ISA.
Pass pci_dev==NULL to pci_alloc_* for ISA devices, and it allocs
GFP_DMAfor you.Sure, but the IA64 platforms that are out now doesn't have an IOMMU, so bounce buffers are used if you don't specify
GFP_DMAin yourget_free_page.
[...]
Quoted 1 line
(
pci_alloc_consistent()allocates a buffer withGFP_DMAon IA64),
The important thing is that pci_alloc_consistent and the other PCI DMA functions work as advertised on IA64. If you pass NULL to pci_alloc_consistent, IA64 should give you an ISA DMA-able address. If you don't, you get a 32-bit PCI DMA address. Use of GFP_DMA is a arch-specific detail, so don't let me confuse you there.
Signature · 3 lines
Jeff Garzik | A recent study has shown that too much soup Building 1024 | can cause malaise in laboratory mice. MandrakeSoft |