__GFP flags and kmalloc failures
I'm trying to allocate a buffer to be used for ISA DMA and I'm experiencing some difficulties.
I'm allocating a 64kB buffer (max size for low ISA DMA) using:
kmalloc(65536, GFP_KERNEL | GFP_DMA);
The choice of flags are from another driver that does ISA DMA so I didn't put too much thought into them at first.
The problem is now that this allocation doesn't always succeed. When it fails I get:
insmod: page allocation failure. order:4, mode:0x11
and a nice little stack dump.
Digging around in gfp.h to see if I have the proper flags I find that I currently have the following:
* __GFP_WAIT : This seems to indicate that the process should be put to sleep until the allocation can succeed. Doesn't seem to work that way though.
* __GFP_IO : What is meant with physical IO? PCI DMA? This buffer needs only be read by the ISA DMA controller and the driver in kernel space. Any useful data is copied to other buffers.
* __GFP_FS : Since the data is copied before use this probably isn't needed.
* __GFP_DMA : From what I've been told, this flags causes the allocator to do the magic required for the buffer to end up i memory accessible from the ISA DMA controller. So this seems to be the only flag that actually does anything useful.
My question is now, why does the allocation fail (sometimes) and what should I do about it?
Memory fragmentation and overusage seems like reasons to why but why doesn't the kernel throw out cache pages and reorganise user pages so that the allocation can succeed?
As for solutions I've tried using __GFP_REPEAT which seems to do the trick. But the double underscore indicates (at least to me) that these are internal defines that shouldn't be used except for very special cases. What is the policy about these?
Rgds
Pierre
Quoted 7 lines · Pierre Ossman
I'm trying to allocate a buffer to be used for ISA DMA and I'm experiencing some difficulties.
I'm allocating a 64kB buffer (max size for low ISA DMA) using:
kmalloc(65536, GFP_KERNEL | GFP_DMA);
Quoted 7 lines
The choice of flags are from another driver that does ISA DMA so I didn't put too much thought into them at first.
The problem is now that this allocation doesn't always succeed. When it fails I get:
insmod: page allocation failure. order:4, mode:0x11
This is a big allocation and the kernel is having problem finding such a big page, due to memory fragmentation (as you mention below).
What kernel version are you using?
-mm contains a series of patches from Nick which should make the situation better, have you tried it? Currently kswapd doenst honour high order page shortage.
Quoted 20 lines
and a nice little stack dump.
Digging around in
gfp.hto see if I have the proper flags I find that I currently have the following:*
__GFP_WAIT: This seems to indicate that the process should be put to sleep until the allocation can succeed. Doesn't seem to work that way though.*
__GFP_IO: What is meant with physical IO? PCI DMA? This buffer needs only be read by the ISA DMA controller and the driver in kernel space. Any useful data is copied to other buffers.*
__GFP_FS: Since the data is copied before use this probably isn't needed.*
__GFP_DMA: From what I've been told, this flags causes the allocator to do the magic required for the buffer to end up i memory accessible from the ISA DMA controller. So this seems to be the only flag that actually does anything useful.
Quoted 6 lines
My question is now, why does the allocation fail (sometimes) and what should I do about it?
Memory fragmentation and overusage seems like reasons to why but why doesn't the kernel throw out cache pages and reorganise user pages so that the allocation can succeed?
We're working on that.
Quoted 4 lines
As for solutions I've tried using
__GFP_REPEATwhich seems to do the trick. But the double underscore indicates (at least to me) that these are internal defines that shouldn't be used except for very special cases. What is the policy about these?
Its OK to use these flags externally. They might change in future major kernel versions though, or even future v2.6 release. ie its not a stable API.
Quoted 6 lines · Marcelo Tosatti
Quoted 3 lines · Pierre Ossman
The problem is now that this allocation doesn't always succeed. When it fails I get:
Quoted 9 lines
Quoted 3 lines
insmod: page allocation failure. order:4, mode:0x11
This is a big allocation and the kernel is having problem finding such a big page, due to memory fragmentation (as you mention below).
What kernel version are you using?
I'm currently running 2.6.9. No external patches (except for my own stuff related to this driver).
Quoted 5 lines
-mm contains a series of patches from Nick which should make the situation better, have you tried it? Currently kswapd doenst honour high order page shortage.
No I haven't. Only saw it today and I usually don't use the -mm tree. I've gotten the impression it's a bit too bleeding edge for me ;)
What do these patches add to the mix?
I'm also not familiar what the order means. I guess it's some kind of priority system? Is there a way I can raise my priority to get access to the memory that kswapd actually keeps available?
Quoted 8 lines
Quoted 3 lines
As for solutions I've tried using
__GFP_REPEATwhich seems to do the trick. But the double underscore indicates (at least to me) that these are internal defines that shouldn't be used except for very special cases. What is the policy about these?Its OK to use these flags externally. They might change in future major kernel versions though, or even future v2.6 release. ie its not a stable API.
Is there any other way of increasing the chances of actually getting the pages I need? Since it is DMA it needs to be one big block.
Rgds
Pierre
Quoted 28 lines · Pierre Ossman
Quoted 7 lines · Marcelo Tosatti
Quoted 4 lines · Pierre Ossman
The problem is now that this allocation doesn't always succeed. When it fails I get:
Quoted 10 lines
Quoted 3 lines
insmod: page allocation failure. order:4, mode:0x11
This is a big allocation and the kernel is having problem finding such a big page, due to memory fragmentation (as you mention below).
What kernel version are you using?
I'm currently running 2.6.9. No external patches (except for my own stuff related to this driver).
Quoted 6 lines
-mm contains a series of patches from Nick which should make the situation better, have you tried it? Currently kswapd doenst honour high order page shortage.
Quoted 3 lines
No I haven't. Only saw it today and I usually don't use the -mm tree. I've gotten the impression it's a bit too bleeding edge for me ;) What do these patches add to the mix?
They make kswapd free pages when high order allocations fail.
Quoted 3 lines
I'm also not familiar what the order means. I guess it's some kind of priority system? Is there a way I can raise my priority to get access to the memory that kswapd actually keeps available?
order indicates the order which 2 will be elevated, to indicate how many physical pages this "high order page" is, in size.
"2 ^ 0" = 1 page, order 0 , "2 ^ 1" = 2 pages, order 1,
"2 ^ 2" = 4 pages, order 2 and so on.
Quoted 8 lines
Quoted 8 lines
Quoted 7 lines
As for solutions I've tried using
__GFP_REPEATwhich seems to do the trick. But the double underscore indicates (at least to me) that these are internal defines that shouldn't be used except for very special cases. What is the policy about these?
Quoted 7 lines
Quoted 5 lines
Its OK to use these flags externally. They might change in future major kernel
versions though, or even future v2.6 release. ie its not a stable API.Is there any other way of increasing the chances of actually getting the pages I need? Since it is DMA it needs to be one big block.
__GFP_NOFAIL, from gfp.h:
* Action modifiers - doesn't change the zoning
*
* __GFP_REPEAT: Try hard to allocate the memory, but the allocation attempt
* _might_ fail. This depends upon the particular VM implementation. *
* __GFP_NOFAIL: The VM implementation _must_ retry infinitely: the caller
* cannot handle allocation failures.
*
* __GFP_NORETRY: The VM implementation must not retry indefinitely. */
Quoted 9 lines · Marcelo Tosatti
Quoted 3 lines · Pierre Ossman
Is there any other way of increasing the chances of actually getting the pages I need? Since it is DMA it needs to be one big block.
__GFP_NOFAIL, fromgfp.h:
Quoted 13 lines
* Action modifiers - doesn't change the zoning
*
*__GFP_REPEAT: Try hard to allocate the memory, but the allocation attempt
* _might_ fail. This depends upon the particular VM implementation. *
*__GFP_NOFAIL: The VM implementation _must_ retry infinitely: the caller
* cannot handle allocation failures.
*
*__GFP_NORETRY: The VM implementation must not retry indefinitely. */
Yes, I've browsed through these. __GFP_NOFAIL seems like it can hang for a very long time (I don't know if there is an upper bound on how long it will have to wait for a free page). __GFP_REPEAT seems to work good enough in this case. My question was meant to be more along the lines of "Is there anything I can do without resorting to unstable/interal API:s?".
Rgds
Pierre
Quoted 25 lines · Pierre Ossman
Quoted 10 lines · Marcelo Tosatti
Quoted 4 lines · Pierre Ossman
Is there any other way of increasing the chances of actually getting the pages I need? Since it is DMA it needs to be one big block.
__GFP_NOFAIL, fromgfp.h:Quoted 13 lines
* Action modifiers - doesn't change the zoning
*
*__GFP_REPEAT: Try hard to allocate the memory, but the allocation attempt
* _might_ fail. This depends upon the particular VM implementation. *
*__GFP_NOFAIL: The VM implementation _must_ retry infinitely: the caller
* cannot handle allocation failures.
*
*__GFP_NORETRY: The VM implementation must not retry indefinitely. */
Quoted 6 lines
Yes, I've browsed through these.
__GFP_NOFAILseems like it can hang for a very long time (I don't know if there is an upper bound on how long it will have to wait for a free page).__GFP_REPEATseems to work good enough in this case.
My question was meant to be more along the lines of "Is there anything I can do without resorting to unstable/interal API:s?".
Not really.
They are not that unstable, I shouldnt mean that.
These defines are not as stable as system calls - VM internals might change in v2.7 and the flags also - but for v2.6 they are very likely to remain untouched.
Its just like any driver API in Linux - they change.
Just keep an eye.
Quoted 11 lines · Marcelo Tosatti
Quoted 5 lines · Pierre Ossman
Yes, I've browsed through these.
__GFP_NOFAILseems like it can hang for a very long time (I don't know if there is an upper bound on how long it will have to wait for a free page).__GFP_REPEATseems to work good enough in this case. My question was meant to be more along the lines of "Is there anything I can do without resorting to unstable/interal API:s?".Not really.
Quoted 9 lines
They are not that unstable, I shouldnt mean that.
These defines are not as stable as system calls - VM internals might change in v2.7 and the flags also - but for v2.6 they are very likely to remain untouched.
Its just like any driver API in Linux - they change.
Just keep an eye.
Ok. I'll stick with __GFP_REPEAT for now then. If things break in the future it should be fairly obvious that this is the problem. Any users of the driver will probably point out that it's broken rather quickly :)
Thanks for the help!
Rgds
Pierre