Vanilla-Kernel 3 - page allocation failure
After updating kernel (x86_64) to stable version 3 there are a few messages appearing in the kernel log such as
kworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20
Searching the net showed that these messages are known to occur since 2004.
Some people were able to get rid of them by setting /proc/sys/vm/min_free_kbytes to a high enough value. This does not help in our case.
Is there a kernel comand line argument to avoid these messages?
As of mm/page_alloc.c these messages are marked to be only warning messages and would not appear if 'gpf_mask' was set to __GFP_NOWARN in function warn_alloc_failed.
How does this mask get set? Is it set by the "external" process knocking at the memory manager?
What is the magic behind the 'order' and 'mode'?
I'm not a subscriber, so please CC me a copy of messages related to the subject. I'm not sure if I can help much by looking at the inside of the kernel, but I will try my best to answer any questions concerning this issue.
Best regards, Philipp
Quoted 14 lines · Philipp Herz - Profihost AG
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20Searching the net showed that these messages are known to occur since 2004.
Some people were able to get rid of them by setting
/proc/sys/vm/min_free_kbytesto a high enough value. This does not help in our case.
Quoted 9 lines
Is there a kernel comand line argument to avoid these messages?
As of
mm/page_alloc.cthese messages are marked to be only warning messages and would not appear if 'gpf_mask' was set to__GFP_NOWARNin functionwarn_alloc_failed.How does this mask get set? Is it set by the "external" process knocking at the memory manager?
Hello, Philipp.
This happens when kernel tries to allocate memory, sometimes in response to some request by the user space, but also in other contexts. For example, an interrupt by a network driver may try to allocate memory. In this context, it will use GFP_ATOMIC as a mask, for example. The most usual flags in the kernel are GFP_KERNEL and GFP_ATOMIC.
Quoted 2 lines
What is the magic behind the 'order' and 'mode'?
The order is the binary log of the number of pages requested. So, order 1 allocations are 2 pages, order 4 would be 16 pages, for example.
The mode is, in fact, gfp_flags. 0x20 is GFP_ATOMIC. This kind of allocation cannot do IO or access the filesystem. Also, it cannot wait for reclaim memory from cache.
This warning is usually followed by some statistics about memory use in your system. Please post it to give more information about this situation.
I have watched some of this happen when lots of cache is used by some filesystems. Perhaps, some tweaking of the vm sysctl options may help, but I can point any magic tweaking right now.
Regards,
Cascardo.
Quoted 11 lines
I'm not a subscriber, so please CC me a copy of messages related to the subject. I'm not sure if I can help much by looking at the inside of the kernel, but I will try my best to answer any questions concerning this issue.
Best regards, Philipp
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Hello Cascardo,
thanks for your detailed answer!
I have uploaded two call traces to pastebin for further investigation.
Maybe this can help you.
* http://pastebin.com/Psg2dGYC (kworker)
* http://pastebin.com/pPFjZqxL (php5)
Regards,
Philipp
Am 18.10.2011 13:32, schrieb Thadeu Lima de Souza Cascardo:
Quoted 66 lines · Philipp Herz - Profihost AG
Quoted 14 lines
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20Searching the net showed that these messages are known to occur since 2004.
Some people were able to get rid of them by setting
/proc/sys/vm/min_free_kbytesto a high enough value. This does not help in our case.Quoted 9 lines
Is there a kernel comand line argument to avoid these messages?
As of
mm/page_alloc.cthese messages are marked to be only warning messages and would not appear if 'gpf_mask' was set to__GFP_NOWARNin functionwarn_alloc_failed.How does this mask get set? Is it set by the "external" process knocking at the memory manager?
Hello, Philipp.
This happens when kernel tries to allocate memory, sometimes in response to some request by the user space, but also in other contexts. For example, an interrupt by a network driver may try to allocate memory. In this context, it will use
GFP_ATOMICas a mask, for example. The most usual flags in the kernel areGFP_KERNELandGFP_ATOMIC.Quoted 2 lines
What is the magic behind the 'order' and 'mode'?
The order is the binary log of the number of pages requested. So, order 1 allocations are 2 pages, order 4 would be 16 pages, for example.
The mode is, in fact,
gfp_flags.0x20isGFP_ATOMIC. This kind of allocation cannot do IO or access the filesystem. Also, it cannot wait for reclaim memory from cache.This warning is usually followed by some statistics about memory use in your system. Please post it to give more information about this situation.
I have watched some of this happen when lots of cache is used by some filesystems. Perhaps, some tweaking of the vm sysctl options may help, but I can point any magic tweaking right now.
Regards,
Cascardo.Quoted 11 lines
I'm not a subscriber, so please CC me a copy of messages related to the subject. I'm not sure if I can help much by looking at the inside of the kernel, but I will try my best to answer any questions concerning this issue.
Best regards, Philipp
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines · Philipp Herz - Profihost AG
Hello Cascardo,
thanks for your detailed answer!
I have uploaded two call traces to pastebin for further investigation.
Quoted 8 lines
Maybe this can help you.
* http://pastebin.com/Psg2dGYC (kworker)
* http://pastebin.com/pPFjZqxL (php5)Regards,
Philipp
Hello, Philipp.
That only tells us that you have a TCP workload in your system. This is the subsystem that is trying to allocate memory. However, we do not know why there is failure. Usually, after the stack dump, there is some statistics about memory. I have seen that these may be suppressed if you have a NUMA system with lots of nodes. Check for NODE_SHIFT in your config. If it's greater than 8, that output may have been suppressed. But you may have just ignored the statistics because of the stack dump.
Regards,
Cascardo.
Quoted 69 lines
Am 18.10.2011 13:32, schrieb Thadeu Lima de Souza Cascardo:
Quoted 66 lines · Philipp Herz - Profihost AG
Quoted 14 lines
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20Searching the net showed that these messages are known to occur since 2004.
Some people were able to get rid of them by setting
/proc/sys/vm/min_free_kbytesto a high enough value. This does not help in our case.Quoted 9 lines
Is there a kernel comand line argument to avoid these messages?
As of
mm/page_alloc.cthese messages are marked to be only warning messages and would not appear if 'gpf_mask' was set to__GFP_NOWARNin functionwarn_alloc_failed.How does this mask get set? Is it set by the "external" process knocking at the memory manager?
Hello, Philipp.
This happens when kernel tries to allocate memory, sometimes in response to some request by the user space, but also in other contexts. For example, an interrupt by a network driver may try to allocate memory. In this context, it will use
GFP_ATOMICas a mask, for example. The most usual flags in the kernel areGFP_KERNELandGFP_ATOMIC.Quoted 2 lines
What is the magic behind the 'order' and 'mode'?
The order is the binary log of the number of pages requested. So, order 1 allocations are 2 pages, order 4 would be 16 pages, for example.
The mode is, in fact,
gfp_flags.0x20isGFP_ATOMIC. This kind of allocation cannot do IO or access the filesystem. Also, it cannot wait for reclaim memory from cache.This warning is usually followed by some statistics about memory use in your system. Please post it to give more information about this situation.
I have watched some of this happen when lots of cache is used by some filesystems. Perhaps, some tweaking of the vm sysctl options may help, but I can point any magic tweaking right now.
Regards,
Cascardo.Quoted 11 lines
I'm not a subscriber, so please CC me a copy of messages related to the subject. I'm not sure if I can help much by looking at the inside of the kernel, but I will try my best to answer any questions concerning this issue.
Best regards, Philipp
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Hello Cascardo
Quoted 2 lines
Usually, after the stack dump, there is some
statistics about memory.
Yes, i have seen this in other posts as well.
Quoted 2 lines
I have seen that these may be suppressed
if you have a NUMA system with lots of nodes.
Yes, in our case it seems to be suppressed.
Quoted 2 lines
Check for
NODE_SHIFTin your
config. If it's greater than 8, that output may have been suppressed.
CONFIG_NODES_SHIFT=10 will be the answer.
Is there any way to get those stats without recompiling the kernel?
Quoted 2 lines
But you may have just ignored the statistics because of the stack dump.
No, i was also wondering why other do have these ;-)
Regards,
Philipp
Am 18.10.2011 14:38, schrieb Thadeu Lima de Souza Cascardo:
Quoted 99 lines · Philipp Herz - Profihost AG
Quoted 6 lines
Hello Cascardo,
thanks for your detailed answer!
I have uploaded two call traces to pastebin for further investigation.
Quoted 8 lines
Maybe this can help you.
* http://pastebin.com/Psg2dGYC (kworker)
* http://pastebin.com/pPFjZqxL (php5)Regards,
PhilippHello, Philipp.
That only tells us that you have a TCP workload in your system. This is the subsystem that is trying to allocate memory. However, we do not know why there is failure. Usually, after the stack dump, there is some statistics about memory. I have seen that these may be suppressed if you have a NUMA system with lots of nodes. Check for
NODE_SHIFTin your config. If it's greater than 8, that output may have been suppressed. But you may have just ignored the statistics because of the stack dump.Regards,
Cascardo.Quoted 69 lines
Am 18.10.2011 13:32, schrieb Thadeu Lima de Souza Cascardo:
Quoted 66 lines · Philipp Herz - Profihost AG
Quoted 14 lines
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20Searching the net showed that these messages are known to occur since 2004.
Some people were able to get rid of them by setting
/proc/sys/vm/min_free_kbytesto a high enough value. This does not help in our case.Quoted 9 lines
Is there a kernel comand line argument to avoid these messages?
As of
mm/page_alloc.cthese messages are marked to be only warning messages and would not appear if 'gpf_mask' was set to__GFP_NOWARNin functionwarn_alloc_failed.How does this mask get set? Is it set by the "external" process knocking at the memory manager?
Hello, Philipp.
This happens when kernel tries to allocate memory, sometimes in response to some request by the user space, but also in other contexts. For example, an interrupt by a network driver may try to allocate memory. In this context, it will use
GFP_ATOMICas a mask, for example. The most usual flags in the kernel areGFP_KERNELandGFP_ATOMIC.Quoted 2 lines
What is the magic behind the 'order' and 'mode'?
The order is the binary log of the number of pages requested. So, order 1 allocations are 2 pages, order 4 would be 16 pages, for example.
The mode is, in fact,
gfp_flags.0x20isGFP_ATOMIC. This kind of allocation cannot do IO or access the filesystem. Also, it cannot wait for reclaim memory from cache.This warning is usually followed by some statistics about memory use in your system. Please post it to give more information about this situation.
I have watched some of this happen when lots of cache is used by some filesystems. Perhaps, some tweaking of the vm sysctl options may help, but I can point any magic tweaking right now.
Regards,
Cascardo.Quoted 11 lines
I'm not a subscriber, so please CC me a copy of messages related to the subject. I'm not sure if I can help much by looking at the inside of the kernel, but I will try my best to answer any questions concerning this issue.
Best regards, Philipp
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 14 lines · Philipp Herz - Profihost AG
Hello Cascardo
Quoted 2 lines
Usually, after the stack dump, there is some
statistics about memory.Yes, i have seen this in other posts as well.
Quoted 2 lines
I have seen that these may be suppressed
if you have a NUMA system with lots of nodes.Yes, in our case it seems to be suppressed.
Quoted 2 lines
Check for
NODE_SHIFTin your
config. If it's greater than 8, that output may have been suppressed.
CONFIG_NODES_SHIFT=10will be the answer.
Quoted 9 lines
Is there any way to get those stats without recompiling the kernel?
Quoted 2 lines
But you may have just ignored the statistics because of the stack dump.
No, i was also wondering why other do have these ;-)
Regards,
Philipp
echo m > /proc/sysrq-trigger
will show you that same output, but not at the time the memory failure happens. It may still show you what is the condition of memory on your nodes.
I am not that much versed in the VM. It just happens that I had very similar issues lately and was trying to undertand it a little more. I still have to solve these issues myself.
In my case, the workload is IO bound on extX filesystems and I see that other systems have these failures due to this memory pressure. Usually, after stopping the workload and unmounting the filesystems, I get most of the memory in the system freed.
Most of the failures are from GFP_ATOMIC allocations, because those won't reclaim memory, but they won't allocate if there is only freed memory below the threshold. Setting this threshold to a lower value like it was suggested (min_free_kbytes) would have helped, but, then, this allows whatever is putting pressure on your memory to also allocate below the threshold and you end up in the same situation (or a worse one).
Do your workload works better on a previous version? I had problems using something like 2.6.32.
Regards,
Cascardo.
Quoted 101 lines
Am 18.10.2011 14:38, schrieb Thadeu Lima de Souza Cascardo:
Quoted 99 lines · Philipp Herz - Profihost AG
Quoted 6 lines
Hello Cascardo,
thanks for your detailed answer!
I have uploaded two call traces to pastebin for further investigation.
Quoted 8 lines
Maybe this can help you.
* http://pastebin.com/Psg2dGYC (kworker)
* http://pastebin.com/pPFjZqxL (php5)Regards,
PhilippHello, Philipp.
That only tells us that you have a TCP workload in your system. This is the subsystem that is trying to allocate memory. However, we do not know why there is failure. Usually, after the stack dump, there is some statistics about memory. I have seen that these may be suppressed if you have a NUMA system with lots of nodes. Check for
NODE_SHIFTin your config. If it's greater than 8, that output may have been suppressed. But you may have just ignored the statistics because of the stack dump.Regards,
Cascardo.Quoted 69 lines
Am 18.10.2011 13:32, schrieb Thadeu Lima de Souza Cascardo:
Quoted 66 lines · Philipp Herz - Profihost AG
Quoted 14 lines
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20Searching the net showed that these messages are known to occur since 2004.
Some people were able to get rid of them by setting
/proc/sys/vm/min_free_kbytesto a high enough value. This does not help in our case.Quoted 9 lines
Is there a kernel comand line argument to avoid these messages?
As of
mm/page_alloc.cthese messages are marked to be only warning messages and would not appear if 'gpf_mask' was set to__GFP_NOWARNin functionwarn_alloc_failed.How does this mask get set? Is it set by the "external" process knocking at the memory manager?
Hello, Philipp.
This happens when kernel tries to allocate memory, sometimes in response to some request by the user space, but also in other contexts. For example, an interrupt by a network driver may try to allocate memory. In this context, it will use
GFP_ATOMICas a mask, for example. The most usual flags in the kernel areGFP_KERNELandGFP_ATOMIC.Quoted 2 lines
What is the magic behind the 'order' and 'mode'?
The order is the binary log of the number of pages requested. So, order 1 allocations are 2 pages, order 4 would be 16 pages, for example.
The mode is, in fact,
gfp_flags.0x20isGFP_ATOMIC. This kind of allocation cannot do IO or access the filesystem. Also, it cannot wait for reclaim memory from cache.This warning is usually followed by some statistics about memory use in your system. Please post it to give more information about this situation.
I have watched some of this happen when lots of cache is used by some filesystems. Perhaps, some tweaking of the vm sysctl options may help, but I can point any magic tweaking right now.
Regards,
Cascardo.Quoted 11 lines
I'm not a subscriber, so please CC me a copy of messages related to the subject. I'm not sure if I can help much by looking at the inside of the kernel, but I will try my best to answer any questions concerning this issue.
Best regards, Philipp
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 6 lines · Philipp Herz - Profihost AG
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20
You just ran out of memory.
The problem here seems to be that the kernel is unable to communicate in a language you can understand.
How do you think the message should have been phrased to make the issue more clear?
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only
Quoted 9 lines · Andi Kleen
Quoted 6 lines · Philipp Herz - Profihost AG
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20
Quoted 7 lines
You just ran out of memory.
The problem here seems to be that the kernel is unable to communicate in a language you can understand.
How do you think the message should have been phrased to make the issue more clear?
We get reports like this fairly regularly, usually accompanied by "But I had lots of free memory and/or swap!"
The order/mode stuff is completely opaque to end-users, who have no clue that there are different types of memory, and exhausting one type can happen even when plenty of other memory is free.
I've been toying with the idea of hacking up a patch to turn those mode flags into printing things like "mode:GFP_ATOMIC|GFP_NOIO" instead though, as I can never remember those flags off the top of my head.
Still won't help end-users, but it would at least speed up diagnosing reports.
Dave
Quoted 2 lines
We get reports like this fairly regularly, usually accompanied by "But I had lots of free memory and/or swap!"
I think the backtrace is also really bad. It just makes it look like a crash. It just cries "please report me", even though there's usually no good reason for it.
I understand it can be useful sometimes for debugging, but most of the time it is unnecessary and just confusing. One good thing probably would be some heuristic to see when to print the backtrace, and don't print it in common situations.
Quoted 4 lines
The order/mode stuff is completely opaque to end-users, who have no clue that there are different types of memory, and exhausting one type can happen even when plenty of other memory is free.
order should be probably replaced with a user readable size, agreed.
order:2 = "16 KB"
[note if anybody wants to reply now it should be "16 KiB", don't bother; i'll ignore you]
Quoted 5 lines
I've been toying with the idea of hacking up a patch to turn those mode flags into printing things like "mode:GFP_ATOMIC|GFP_NOIO" instead though, as I can never remember those flags off the top of my head.
Still won't help end-users, but it would at least speed up diagnosing reports.
Better decode it: "from interrupt handler", "inside a file system"
Unfortunately there's no flag for GFP_ATOMIC but not in a interrupt handler, but some code with broken locking abusing it. Perhaps there should be.
-Andi
Signature · 1 line
[email protected] -- Speaking for myself only.
Quoted 11 lines · Andi Kleen
Quoted 6 lines · Philipp Herz - Profihost AG
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20You just ran out of memory.
He ran out of order-1 physically contiguous memory and was unable to compact or reclaim because of the atomic context.
Philipp, based on your pastes from another post, it's evident you're using CONFIG_SLAB and, unfortunately, it's not possible to change to single page allocations (which would only result in a page allocation failure if you were completely out of memory) without recompiling.
You have a couple options:
- recompile with BREAK_GFP_ORDER_HI redefined to 0 in mm/slab.c, or
- recompile with CONFIG_SLUB instead of CONFIG_SLAB.
It's very possible that neither of these will help, but it will tell you whether you need to go out and buy more RAM or not. If you try to recompile with BREAK_GFP_ORDER_HI, these may turn into order-0 allocations. If you can't reboot, send the output of /proc/<pid>/net/protocols where <pid> is the pid of one of the above tasks (kworker, mysql, php5) when they are running and we'll know.
[ Changing slab_break_gfp_order should really be a CONFIG_SLAB command-
line option. It can't be runtime because slab depends on the order for caches remaining constant, but we can certainly change it on boot. ]
If you try CONFIG_SLUB instead of CONFIG_SLAB, you can pass slub_max_order=0 on the command line and see if it helps.
Hello Cascardo,
Quoted 1 line
echo m>
/proc/sysrq-trigger
Thanks,
I have pasted another Call Trace including memory stats at
* http://pastebin.com/vjLHuqtk
Not sure if memory stats are close enough to the call trace event.
If not, do we have to recompile the kernel to get call traces and memory stats at the same time?
Quoted 2 lines
Do your workload works better on a previous version? I had problems using something like 2.6.32.
Yes,
kernel version was 2.6.32.40 before and we never had those messages appearing.
Regards,
Philipp
Am 18.10.2011 16:35, schrieb Thadeu Lima de Souza Cascardo:
Quoted 156 lines · Philipp Herz - Profihost AG
Quoted 14 lines
Hello Cascardo
Quoted 2 lines
Usually, after the stack dump, there is some
statistics about memory.Yes, i have seen this in other posts as well.
Quoted 2 lines
I have seen that these may be suppressed
if you have a NUMA system with lots of nodes.Yes, in our case it seems to be suppressed.
Quoted 2 lines
Check for
NODE_SHIFTin your
config. If it's greater than 8, that output may have been suppressed.
CONFIG_NODES_SHIFT=10will be the answer.Quoted 9 lines
Is there any way to get those stats without recompiling the kernel?
Quoted 2 lines
But you may have just ignored the statistics because of the stack dump.
No, i was also wondering why other do have these ;-)
Regards,
Philippecho m>
/proc/sysrq-triggerwill show you that same output, but not at the time the memory failure happens. It may still show you what is the condition of memory on your nodes.
I am not that much versed in the VM. It just happens that I had very similar issues lately and was trying to undertand it a little more. I still have to solve these issues myself.
In my case, the workload is IO bound on extX filesystems and I see that other systems have these failures due to this memory pressure. Usually, after stopping the workload and unmounting the filesystems, I get most of the memory in the system freed.
Most of the failures are from
GFP_ATOMICallocations, because those won't reclaim memory, but they won't allocate if there is only freed memory below the threshold. Setting this threshold to a lower value like it was suggested (min_free_kbytes) would have helped, but, then, this allows whatever is putting pressure on your memory to also allocate below the threshold and you end up in the same situation (or a worse one).Do your workload works better on a previous version? I had problems using something like 2.6.32.
Regards,
Cascardo.Quoted 101 lines
Am 18.10.2011 14:38, schrieb Thadeu Lima de Souza Cascardo:
Quoted 99 lines · Philipp Herz - Profihost AG
Quoted 6 lines
Hello Cascardo,
thanks for your detailed answer!
I have uploaded two call traces to pastebin for further investigation.
Quoted 8 lines
Maybe this can help you.
* http://pastebin.com/Psg2dGYC (kworker)
* http://pastebin.com/pPFjZqxL (php5)Regards,
PhilippHello, Philipp.
That only tells us that you have a TCP workload in your system. This is the subsystem that is trying to allocate memory. However, we do not know why there is failure. Usually, after the stack dump, there is some statistics about memory. I have seen that these may be suppressed if you have a NUMA system with lots of nodes. Check for
NODE_SHIFTin your config. If it's greater than 8, that output may have been suppressed. But you may have just ignored the statistics because of the stack dump.Regards,
Cascardo.Quoted 69 lines
Am 18.10.2011 13:32, schrieb Thadeu Lima de Souza Cascardo:
Quoted 66 lines · Philipp Herz - Profihost AG
Quoted 14 lines
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20Searching the net showed that these messages are known to occur since 2004.
Some people were able to get rid of them by setting
/proc/sys/vm/min_free_kbytesto a high enough value. This does not help in our case.Quoted 9 lines
Is there a kernel comand line argument to avoid these messages?
As of
mm/page_alloc.cthese messages are marked to be only warning messages and would not appear if 'gpf_mask' was set to__GFP_NOWARNin functionwarn_alloc_failed.How does this mask get set? Is it set by the "external" process knocking at the memory manager?
Hello, Philipp.
This happens when kernel tries to allocate memory, sometimes in response to some request by the user space, but also in other contexts. For example, an interrupt by a network driver may try to allocate memory. In this context, it will use
GFP_ATOMICas a mask, for example. The most usual flags in the kernel areGFP_KERNELandGFP_ATOMIC.Quoted 2 lines
What is the magic behind the 'order' and 'mode'?
The order is the binary log of the number of pages requested. So, order 1 allocations are 2 pages, order 4 would be 16 pages, for example.
The mode is, in fact,
gfp_flags.0x20isGFP_ATOMIC. This kind of allocation cannot do IO or access the filesystem. Also, it cannot wait for reclaim memory from cache.This warning is usually followed by some statistics about memory use in your system. Please post it to give more information about this situation.
I have watched some of this happen when lots of cache is used by some filesystems. Perhaps, some tweaking of the vm sysctl options may help, but I can point any magic tweaking right now.
Regards,
Cascardo.Quoted 11 lines
I'm not a subscriber, so please CC me a copy of messages related to the subject. I'm not sure if I can help much by looking at the inside of the kernel, but I will try my best to answer any questions concerning this issue.
Best regards, Philipp
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 10 lines · Andi Kleen
Quoted 6 lines · Philipp Herz - Profihost AG
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20You just ran out of memory.
I read it as "Why is this happening when the previous kernel didn't have this issue?", which is a *much* more complicated question...
Quoted 6 lines · Andi Kleen
Quoted 5 lines
Quoted 3 lines · Philipp Herz - Profihost AG
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such as
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
kworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20You just ran out of memory.
I read it as "Why is this happening when the previous kernel didn't have this issue?", which is a *much* more complicated question...
Exactly, that was the intention of my post.
It would me nice to know, if these messages do require to focus on memory statitics?
Looking at our monitoring data, the system does have enough memory available.
How do processes get effected, when they are running into situations where kernel memory manager tells them about "page allocation failure"?
Is it just meant to be a warning/debugging message new to current kernel version 3?
How can we solve the situation? Getting rid of these message, by
- just suppressing
- increasing system's memory
- change kernel and/or kernel runtime config
?
Kind regards,
Philipp
Am 19.10.2011 03:58, schrieb David Rientjes:
Quoted 35 lines · Andi Kleen
Quoted 12 lines
Quoted 6 lines · Philipp Herz - Profihost AG
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20You just ran out of memory.
He ran out of order-1 physically contiguous memory and was unable to compact or reclaim because of the atomic context.
Philipp, based on your pastes from another post, it's evident you're using
CONFIG_SLABand, unfortunately, it's not possible to change to single page allocations (which would only result in a page allocation failure if you were completely out of memory) without recompiling.You have a couple options:
- recompile with
BREAK_GFP_ORDER_HIredefined to 0 inmm/slab.c, or- recompile with
CONFIG_SLUBinstead ofCONFIG_SLAB.It's very possible that neither of these will help, but it will tell you whether you need to go out and buy more RAM or not. If you try to recompile with
BREAK_GFP_ORDER_HI, these may turn into order-0 allocations. If you can't reboot, send the output of /proc/<pid>/net/protocols where<pid> is the pid of one of the above tasks (kworker, mysql, php5) when they are running and we'll know.
Quoted 6 lines
[ Changing
slab_break_gfp_ordershould really be aCONFIG_SLABcommand-
line option. It can't be runtime because slab depends on the order for caches remaining constant, but we can certainly change it on boot. ]If you try
CONFIG_SLUBinstead ofCONFIG_SLAB, you can pass slub_max_order=0 on the command line and see if it helps.
Hi David,
we have recompiled the kernel of one machine with CONFIG_SLUB instead of CONFIG_SLAB, but it is showing similar message.
Now it's showing failure at "order:5, mode:0x4020".
Call trace can be found at:
* http://pastebin.com/uGJiwvG1
Comparing kernel 2.6.32 (mm/page_alloc.c) there seams to be the same way of dealing with page allocation.
Do you have an idea why these (warning) messages do never appear running 2.6.32?
Regards,
Philipp
Le lundi 24 octobre 2011 à 08:33 +0200, Philipp Herz - Profihost AG a écrit :
Quoted 50 lines
Am 19.10.2011 03:58, schrieb David Rientjes:
Quoted 35 lines · Andi Kleen
Quoted 12 lines
Quoted 6 lines · Philipp Herz - Profihost AG
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20You just ran out of memory.
He ran out of order-1 physically contiguous memory and was unable to compact or reclaim because of the atomic context.
Philipp, based on your pastes from another post, it's evident you're using
CONFIG_SLABand, unfortunately, it's not possible to change to single page allocations (which would only result in a page allocation failure if you were completely out of memory) without recompiling.You have a couple options:
- recompile with
BREAK_GFP_ORDER_HIredefined to 0 inmm/slab.c, or- recompile with
CONFIG_SLUBinstead ofCONFIG_SLAB.It's very possible that neither of these will help, but it will tell you whether you need to go out and buy more RAM or not. If you try to recompile with
BREAK_GFP_ORDER_HI, these may turn into order-0 allocations. If you can't reboot, send the output of /proc/<pid>/net/protocols where<pid> is the pid of one of the above tasks (kworker, mysql, php5) when they are running and we'll know.Quoted 6 lines
[ Changing
slab_break_gfp_ordershould really be aCONFIG_SLABcommand-
line option. It can't be runtime because slab depends on the order for caches remaining constant, but we can certainly change it on boot. ]If you try
CONFIG_SLUBinstead ofCONFIG_SLAB, you can pass slub_max_order=0 on the command line and see if it helps.Hi David,
we have recompiled the kernel of one machine with
CONFIG_SLUBinstead ofCONFIG_SLAB, but it is showing similar message.Now it's showing failure at "order:5, mode:0x4020".
Quoted 8 lines
Call trace can be found at:
* http://pastebin.com/uGJiwvG1Comparing kernel 2.6.32 (
mm/page_alloc.c) there seams to be the same way of dealing with page allocation.Do you have an idea why these (warning) messages do never appear running 2.6.32?
Your tg3 has a firmware limitation, and some skbs using fragments have to be reallocated using a single and contiguous area of memory.
Initial skb delivered by tcp stack only uses order-0 pages, but the reallocated one, being 64K, can be order-5
You can avoid this by following tuning :
ethtool -K eth0 sg off
Am 24.10.2011 09:03, schrieb Eric Dumazet:
Quoted 64 lines
Le lundi 24 octobre 2011 à 08:33 +0200, Philipp Herz - Profihost AG a écrit :
Quoted 50 lines
Am 19.10.2011 03:58, schrieb David Rientjes:
Quoted 35 lines · Andi Kleen
Quoted 12 lines
Quoted 6 lines · Philipp Herz - Profihost AG
After updating kernel (
x86_64) to stable version 3 there are a few messages appearing in the kernel log such askworker/0:1: page allocation failure: order:1, mode:0x20
mysql: page allocation failure: order:1, mode:0x20
php5: page allocation failure: order:1, mode:0x20You just ran out of memory.
He ran out of order-1 physically contiguous memory and was unable to compact or reclaim because of the atomic context.
Philipp, based on your pastes from another post, it's evident you're using
CONFIG_SLABand, unfortunately, it's not possible to change to single page allocations (which would only result in a page allocation failure if you were completely out of memory) without recompiling.You have a couple options:
- recompile with
BREAK_GFP_ORDER_HIredefined to 0 inmm/slab.c, or- recompile with
CONFIG_SLUBinstead ofCONFIG_SLAB.It's very possible that neither of these will help, but it will tell you whether you need to go out and buy more RAM or not. If you try to recompile with
BREAK_GFP_ORDER_HI, these may turn into order-0 allocations. If you can't reboot, send the output of
/proc/<pid>/net/protocols where<pid> is the pid of one of the above tasks
(kworker, mysql, php5) when they are running and we'll know.Quoted 6 lines
[ Changing
slab_break_gfp_ordershould really be aCONFIG_SLABcommand-
line option. It can't be runtime because slab depends on the order for caches remaining constant, but we can certainly change it on boot. ]If you try
CONFIG_SLUBinstead ofCONFIG_SLAB, you can pass slub_max_order=0 on the command line and see if it helps.Hi David,
we have recompiled the kernel of one machine with
CONFIG_SLUBinstead ofCONFIG_SLAB, but it is showing similar message.Now it's showing failure at "order:5, mode:0x4020".
Quoted 8 lines
Call trace can be found at:
* http://pastebin.com/uGJiwvG1Comparing kernel 2.6.32 (
mm/page_alloc.c) there seams to be the same way of dealing with page allocation.Do you have an idea why these (warning) messages do never appear running 2.6.32?
Your tg3 has a firmware limitation, and some skbs using fragments have to be reallocated using a single and contiguous area of memory.
Quoted 7 lines
Initial skb delivered by tcp stack only uses order-0 pages, but the reallocated one, being 64K, can be order-5
You can avoid this by following tuning :
ethtool -K eth0 sg off
ok,
does that mean that there was no firmware limitation with kernel 2.6.32 or that the tg3 module has any "disable warnings" flag matching __GFP_NOWARN?
Le lundi 24 octobre 2011 à 09:21 +0200, Philipp Herz - Profihost AG a écrit :
Quoted 4 lines
does that mean that there was no firmware limitation with kernel 2.6.32 or that the tg3 module has any "disable warnings" flag matching
__GFP_NOWARN?
There is no __GFP_NOWARN trick on tg3.
We tend to prefer to be notified of a memory problem, instead of hide ...
By the way, apparently this driver drops the frame and doesnt increase tx_dropped device counter. A patch will follow.
Could you post your full dmesg ?
Am 24.10.2011 10:01, schrieb Eric Dumazet:
Quoted 8 lines
Le lundi 24 octobre 2011 à 09:21 +0200, Philipp Herz - Profihost AG a écrit :
Quoted 4 lines
does that mean that there was no firmware limitation with kernel 2.6.32 or that the tg3 module has any "disable warnings" flag matching
__GFP_NOWARN?
Quoted 4 lines
There is no
__GFP_NOWARNtrick on tg3.We tend to prefer to be notified of a memory problem, instead of hide ...
yes,
that's exactly what would like to understand in comparison to the behavior of kernel 2.6.32.
why does this notification show up now and never did before...
Quoted 3 lines
By the way, apparently this driver drops the frame and doesnt increase
tx_droppeddevice counter. A patch will follow.
fine
Quoted 4 lines
Could you post your full dmesg ?
Currently i can not provide any further information, 'cause server has been restarted.
What exactly are you looking for?
Le lundi 24 octobre 2011 à 10:19 +0200, Philipp Herz - Profihost AG a écrit :
Quoted 4 lines
Currently i can not provide any further information, 'cause server has been restarted.
What exactly are you looking for?
A dmesg for 2.6.32 is OK, I look for :
- memory layout (tg3 has workarounds for 4G crossing),
- and exact tg3 chip, tg3 messages...
Am 24.10.2011 10:29, schrieb Eric Dumazet:
Quoted 11 lines
Le lundi 24 octobre 2011 à 10:19 +0200, Philipp Herz - Profihost AG a écrit :
Quoted 4 lines
Currently i can not provide any further information, 'cause server has been restarted.
What exactly are you looking for?
A dmesg for 2.6.32 is OK, I look for :
- memory layout (tg3 has workarounds for 4G crossing),
- and exact tg3 chip, tg3 messages...
Here is a dmesg from 2.6.38 where this message also does not occur:
http://pastebin.com/raw.php?i=HjDEKVcp
Stefan
Quoted 5 lines · Philipp Herz - Profihost AG
we have recompiled the kernel of one machine with
CONFIG_SLUBinstead ofCONFIG_SLAB, but it is showing similar message.Now it's showing failure at "order:5, mode:0x4020".
Quoted 9 lines
Call trace can be found at:
* http://pastebin.com/uGJiwvG1Comparing kernel 2.6.32 (
mm/page_alloc.c) there seams to be the same way of dealing with page allocation.Do you have an idea why these (warning) messages do never appear running 2.6.32?
Do you have CONFIG_COMPACTION enabled? Perhaps this is a difference in the deprecation of lumpy reclaim between 2.6.35 and 2.6.38 and defragmentation being done by memory compaction instead.
It won't be triggered synchronously in this context since it's a GFP_ATOMIC allocation, which is why it emits a page allocation failure in the first place, but it will show whether defragmentation is the issue or you're just simply low on memory.
Am 26.10.2011 22:26, schrieb David Rientjes:
Quoted 13 lines · Philipp Herz - Profihost AG
Quoted 12 lines
we have recompiled the kernel of one machine with
CONFIG_SLUBinstead ofCONFIG_SLAB, but it is showing similar message.Now it's showing failure at "order:5, mode:0x4020".
Call trace can be found at:
* http://pastebin.com/uGJiwvG1Comparing kernel 2.6.32 (
mm/page_alloc.c) there seams to be the same way of dealing with page allocation.
Quoted 7 lines
Quoted 3 lines
Do you have an idea why these (warning) messages do never appear running 2.6.32?
Do you have
CONFIG_COMPACTIONenabled? Perhaps this is a difference in the deprecation of lumpy reclaim between 2.6.35 and 2.6.38 and defragmentation being done by memory compaction instead.
Yes CONFIG_COMPACTION is enabled as a dependency for TRANSPARENT_HUGEPAGE which is different to the configs before.
Quoted 5 lines
It won't be triggered synchronously in this context since it's a
GFP_ATOMICallocation, which is why it emits a page allocation failure in the first place, but it will show whether defragmentation is the issue or you're just simply low on memory.
Do you mean that "memory compaction" should be turned off again?
How can I see the difference between "deframentation issue" and "low memory"? I did not get this point.
Quoted 6 lines · Philipp Herz - Profihost AG
Quoted 4 lines
It won't be triggered synchronously in this context since it's a
GFP_ATOMICallocation, which is why it emits a page allocation failure in the first place, but it will show whether defragmentation is the issue or you're just simply low on memory.Do you mean that "memory compaction" should be turned off again?
No, I mean the difference since 2.6.32 might be that lumpy reclaim was deprecated so it is no longer causing high-order page allocations to be free in atomic context where no balancing or migration can synchronously be done.
Quoted 3 lines
How can I see the difference between "deframentation issue" and "low memory"? I did not get this point.
unusable_index and extfrag_index in debugfs if you've enabled and mounted that. You could also take a look at tweaking /proc/sys/vm/extfrag_threshold, see Documentation/sysctl/vm.txt.