Can Linux live without DMA zone?
I am trying to reserve a block of memory (>16MB) starting from 0 and hide it from kernel. A consequence is that DMA zone now has size 0. That causes many drivers to grief (OOMs).
I see two ways out:
1. Modify individual drivers and convince them not to alloc with GFP_DMA. I have been trying to do this but do not seem to see an end of it. :)
2. Simply lie and increase MAX_DMA_ADDRESS to really big (like 1GB) so that the whole memory region belongs to DMA zone.
#2 sounds pretty hackish. I am sure something bad will happen sooner or later (like what?). But so far it appears to be working fine.
The fundamental question is: Has anybody tried to run Linux without 0 sized DMA zone before? Am I doing something that nobody has done before (which is something really hard to believe these days with Linux :P)?
Cheers.
Jun
Quoted 12 lines · Jun Sun
I am trying to reserve a block of memory (>16MB) starting from 0 and hide it from kernel. A consequence is that DMA zone now has size 0. That causes many drivers to grief (OOMs).
I see two ways out:
1. Modify individual drivers and convince them not to alloc with
GFP_DMA. I have been trying to do this but do not seem to see an end of it. :)2. Simply lie and increase
MAX_DMA_ADDRESSto really big (like 1GB) so that the whole memory region belongs to DMA zone.
Quoted 6 lines
#2 sounds pretty hackish. I am sure something bad will happen sooner or later (like what?). But so far it appears to be working fine.
The fundamental question is: Has anybody tried to run Linux without 0 sized DMA zone before? Am I doing something that nobody has done before (which is something really hard to believe these days with Linux :P)?
on a PC there are still devices that need memory in the lower 16Mb..... (like floppy)
Maybe you should reserve another area of memory instead!
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Quoted 4 lines · Jun Sun
The fundamental question is: Has anybody tried to run Linux without 0 sized DMA zone before? Am I doing something that nobody has done before (which is something really hard to believe these days with Linux :P)?
There's patches that rip out ZONE_DMA for platforms that don't have a DMA limitation (and were pretty much putting all of ZONE_NORMAL in ZONE_DMA), all of these are already in -mm.
Shouldn't only ancient ISA drivers be using GFP_DMA? You know, ones that actually require it? PCI drivers should not have this limit.
Quoted 21 lines · Jun Sun
I am trying to reserve a block of memory (>16MB) starting from 0 and hide it from kernel. A consequence is that DMA zone now has size 0. That causes many drivers to grief (OOMs).
I see two ways out:
1. Modify individual drivers and convince them not to alloc with
GFP_DMA. I have been trying to do this but do not seem to see an end of it. :)2. Simply lie and increase
MAX_DMA_ADDRESSto really big (like 1GB) so that the whole memory region belongs to DMA zone.#2 sounds pretty hackish. I am sure something bad will happen sooner or later (like what?). But so far it appears to be working fine.
The fundamental question is: Has anybody tried to run Linux without 0 sized DMA zone before? Am I doing something that nobody has done before (which is something really hard to believe these days with Linux :P)?
Cheers.
Jun
Quoted 2 lines · Phillip Susi
Shouldn't only ancient ISA drivers be using
GFP_DMA? You know, ones that actually require it? PCI drivers should not have this limit.
that is a nice theory, but unfortunately there is just a lot of "PCI" hardware out there for which the designers decided to save a bit of copper and only wire up the lower X address lines (for various values of
X)
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Quoted 4 lines · Arjan van de Ven
that is a nice theory, but unfortunately there is just a lot of "PCI" hardware out there for which the designers decided to save a bit of copper and only wire up the lower X address lines (for various values of
X)
Yea, but shouldn't PCI drivers be using another means than allocating from GFP_DMA? Wasn't there some sort of bounce buffers call I can't quite remember the details of? That performs any required translations to bus hardware addresses, and copies the buffer to a more appropriate location if required, based on the specific requirements of that device?
I know that most 32 bit PCI devices can't handle addresses above the 4 GB mark on 64 bit machines, but those drivers should NOT be limiting DMA to the first 16 MB. Especially since most machines don't have over 4 GB of ram anyhow, but quite often original buffers will be above 16 MB.
Quoted 11 lines · Phillip Susi
Quoted 4 lines · Arjan van de Ven
that is a nice theory, but unfortunately there is just a lot of "PCI" hardware out there for which the designers decided to save a bit of copper and only wire up the lower X address lines (for various values of
X)Yea, but shouldn't PCI drivers be using another means than allocating from
GFP_DMA? Wasn't there some sort of bounce buffers call I can't quite remember the details of? That performs any required translations to bus hardware addresses, and copies the buffer to a more appropriate location if required, based on the specific requirements of that device?
that's for the 32 bit boundary. THe problem is that there are 31, 30, 28 and 26 bit devices as well, and those are in more trouble, and will eventually fall back to GFP_DMA (inside the x86 PCI code; the driver just uses the pci dma allocation routines) if they can't get suitable memory otherwise....
It's all nice in theory. But then there is the reality that not all devices are nice pci device that implement the entire spec;)
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Quoted 9 lines · Arjan van de Ven
that's for the 32 bit boundary. THe problem is that there are 31, 30, 28 and 26 bit devices as well, and those are in more trouble, and will eventually fall back to
GFP_DMA(inside the x86 PCI code; the driver just uses the pci dma allocation routines) if they can't get suitable memory otherwise....It's all nice in theory. But then there is the reality that not all devices are nice pci device that implement the entire spec;)
Right, but doesn't the bounce/allocation routine take as a parameter the limit that the device can handle? If the device can handle 28 bit addresses, then the kernel should not limit it to only 24 bits.
Quoted 7 lines · Phillip Susi
Quoted 6 lines · Arjan van de Ven
that's for the 32 bit boundary. THe problem is that there are 31, 30, 28 and 26 bit devices as well, and those are in more trouble, and will eventually fall back to
GFP_DMA(inside the x86 PCI code; the driver just uses the pci dma allocation routines) if they can't get suitable memory otherwise....
Quoted 7 lines
Quoted 3 lines
It's all nice in theory. But then there is the reality that not all devices are nice pci device that implement the entire spec;)
Right, but doesn't the bounce/allocation routine take as a parameter the limit that the device can handle? If the device can handle 28 bit addresses, then the kernel should not limit it to only 24 bits.
you're right in theory, but the kernel only has a few pools of memory available, but not at every bit boundary. there is a 32 bit pool (GFP_DMA32) on some, a 30-ish bit pool (GFP_KERNEL) on others, and a 24 bit pool (GFP_DMA) with basically nothing inbetween.
Signature · 2 lines
if you want to mail me at work (you don't), use arjan (at) linux.intel.com Test the interaction between Linux and your BIOS via http://www.linuxfirmwarekit.org
Quoted 12 lines · Arjan van de Ven
Quoted 11 lines · Phillip Susi
Quoted 9 lines · Arjan van de Ven
that's for the 32 bit boundary. THe problem is that there are 31, 30, 28 and 26 bit devices as well, and those are in more trouble, and will eventually fall back to
GFP_DMA(inside the x86 PCI code; the driver just uses the pci dma allocation routines) if they can't get suitable memory otherwise....It's all nice in theory. But then there is the reality that not all devices are nice pci device that implement the entire spec;)
Quoted 9 lines
Quoted 3 lines
Right, but doesn't the bounce/allocation routine take as a parameter the limit that the device can handle? If the device can handle 28 bit addresses, then the kernel should not limit it to only 24 bits.
you're right in theory, but the kernel only has a few pools of memory available, but not at every bit boundary. there is a 32 bit pool (
GFP_DMA32) on some, a 30-ish bit pool (GFP_KERNEL) on others, and a 24 bit pool (GFP_DMA) with basically nothing inbetween.
Perhaps a better solution is to
1. get rid of DMA zone
2. have another alloc funciton (e.g., kmalloc_range()) which takes an extra pair of parameters to indicate the desired range for the allocated memory. Most DMA buffers are allocated during start-up. So the alloc operations should generally be successful.
3. convert drivers over to use the new function.
Cheers.
Jun
are allocated at start-up time.
Quoted 23 lines · On Thu, 2 Nov 2006 15:17:15 -0800 Jun Sun <[email protected]> wrote:
Quoted 12 lines · Arjan van de Ven
Quoted 11 lines · Phillip Susi
Quoted 9 lines · Arjan van de Ven
that's for the 32 bit boundary. THe problem is that there are 31, 30, 28 and 26 bit devices as well, and those are in more trouble, and will eventually fall back to
GFP_DMA(inside the x86 PCI code; the driver just uses the pci dma allocation routines) if they can't get suitable memory otherwise....It's all nice in theory. But then there is the reality that not all devices are nice pci device that implement the entire spec;)
Quoted 9 lines
Quoted 3 lines
Right, but doesn't the bounce/allocation routine take as a parameter the limit that the device can handle? If the device can handle 28 bit addresses, then the kernel should not limit it to only 24 bits.
you're right in theory, but the kernel only has a few pools of memory available, but not at every bit boundary. there is a 32 bit pool (
GFP_DMA32) on some, a 30-ish bit pool (GFP_KERNEL) on others, and a 24 bit pool (GFP_DMA) with basically nothing inbetween.
Quoted 9 lines
Perhaps a better solution is to
1. get rid of DMA zone
2. have another alloc funciton (e.g.,
kmalloc_range()) which takes an extra pair of parameters to indicate the desired range for the allocated memory. Most DMA buffers are allocated during start-up. So the alloc operations should generally be successful.
Network devices don't allocate buffer until they are brought up. By then a lot of memory allocation has happened. You could add an interface that allows a device to say:
kmalloc_range_intent(unsigned long mask, unsigned count, unsigned size)
to cause reservation before use.
Signature · 1 line
Stephen Hemminger <[email protected]>
Quoted 9 lines · Jun Sun
Perhaps a better solution is to
1. get rid of DMA zone
2. have another alloc funciton (e.g.,
kmalloc_range()) which takes an extra pair of parameters to indicate the desired range for the allocated memory. Most DMA buffers are allocated during start-up. So the alloc operations should generally be successful.
Quoted 6 lines
3. convert drivers over to use the new function.
Cheers.
Jun
are allocated at start-up time.
That is what I was thinking. You don't need lots of separate pools, you just need the standard allocator to prefer higher addresses, and then the bounce routines need to simply check if the existing user buffer happens to already be within the area the hardware can address ( which it often will be ), and if not, copy the data to pages allocated in lower memory.
Quoted 4 lines · Arjan van de Ven
you're right in theory, but the kernel only has a few pools of memory available, but not at every bit boundary. there is a 32 bit pool (
GFP_DMA32) on some, a 30-ish bit pool (GFP_KERNEL) on others, and a 24 bit pool (GFP_DMA) with basically nothing inbetween.
Perhaps naive question, but... what's wrong with allocating memory from the top (within given address mask[1], size-wise)? I think we don't allocate more than 1 page with kmalloc anymore, do we?
[1] - some devices (a specific StrongARM only?) need non-continuous masks due to a hardware bug(s).
Signature · 1 line
Krzysztof Halasa
Ar Iau, 2006-11-02 am 11:32 -0500, ysgrifennodd Phillip Susi:
Quoted 2 lines
Shouldn't only ancient ISA drivers be using
GFP_DMA? You know, ones that actually require it? PCI drivers should not have this limit.
"Should" and "Do not" are different things. Many PCI drivers have interesting little limits.