[2.4] 0-order allocation failed
Subsystems (6)
Hi all,
I'm running 2.4.28 bk snapshot of 2004.09.03 The machine has an uptime of 7 days, 23:46 now.
I was running several bittorrent clients inside of a screen session. Suddenly they all died (including the screen session).
dmesg sayed this:
__alloc_pages: 0-order allocation failed (gfp=0x1f0/0) __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process python__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process screen
I already got this with kernel 2.4.27 vanilla after a higher amount of uptime (I think it was over 10 days). This was exactly the reason I updated to bk snapshot.
What can be the reason for this? Is it OOM? (I can't really believe it is).
Is it a kernel memory leak?
With 2.4.26 I never got these errors. And I ran uptimes up to 50 days.
Signature · 1 line
Regards Michael Buesch [ http://www.tuxsoft.de.vu ]
Quoted 17 lines · Michael Buesch
Hi all,
I'm running 2.4.28 bk snapshot of 2004.09.03 The machine has an uptime of 7 days, 23:46 now.
I was running several bittorrent clients inside of a screen session. Suddenly they all died (including the screen session).
dmesg sayed this:
__alloc_pages: 0-order allocation failed (gfp=0x1f0/0)__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process python__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process screen
Quoted 6 lines
I already got this with kernel 2.4.27 vanilla after a higher amount of uptime (I think it was over 10 days). This was exactly the reason I updated to bk snapshot.
What can be the reason for this? Is it OOM? (I can't really believe it is).
Can you check how much swap space is there available when the OOM killer trigger? I bet this is the case.
If its not, we have a problem.
Quoted 4 lines
Is it a kernel memory leak?
With 2.4.26 I never got these errors. And I ran uptimes up to 50 days.
Quoting Marcelo Tosatti <[email protected]>:
Quoted 22 lines · Michael Buesch
Quoted 21 lines
Hi all,
I'm running 2.4.28 bk snapshot of 2004.09.03 The machine has an uptime of 7 days, 23:46 now.
I was running several bittorrent clients inside of a screen session. Suddenly they all died (including the screen session).
dmesg sayed this:
__alloc_pages: 0-order allocation failed (gfp=0x1f0/0)__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process python__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process screenI already got this with kernel 2.4.27 vanilla after a higher amount of uptime (I think it was over 10 days). This was exactly the reason I updated to bk snapshot.
Quoted 5 lines
Quoted 2 lines
What can be the reason for this? Is it OOM? (I can't really believe it is).
Can you check how much swap space is there available when the OOM killer trigger? I bet this is the case.
The machine doesn't have swap.
Quoted 8 lines
If its not, we have a problem.
Quoted 4 lines
Is it a kernel memory leak?
With 2.4.26 I never got these errors. And I ran uptimes up to 50 days.
Signature · 1 line
Regards Michael Buesch [ http://www.tuxsoft.de.vu ]
Quoted 23 lines · Michael Buesch
Quoting Marcelo Tosatti <[email protected]>:
Quoted 22 lines · Michael Buesch
Quoted 21 lines
Hi all,
I'm running 2.4.28 bk snapshot of 2004.09.03 The machine has an uptime of 7 days, 23:46 now.
I was running several bittorrent clients inside of a screen session. Suddenly they all died (including the screen session).
dmesg sayed this:
__alloc_pages: 0-order allocation failed (gfp=0x1f0/0)__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process python__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)__alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process screenI already got this with kernel 2.4.27 vanilla after a higher amount of uptime (I think it was over 10 days). This was exactly the reason I updated to bk snapshot.
Quoted 7 lines
Quoted 5 lines
Quoted 2 lines
What can be the reason for this? Is it OOM? (I can't really believe it is).
Can you check how much swap space is there available when the OOM killer trigger? I bet this is the case.
The machine doesn't have swap.
Well then you're probably facing true OOM.
Add some swap.
Quoted 6 lines
Quoted 6 lines
If its not, we have a problem.
Quoted 4 lines
Is it a kernel memory leak?
With 2.4.26 I never got these errors. And I ran uptimes up to 50 days.
* Marcelo Tosatti <[email protected]>:
| > > Can you check how much swap space is there available when | > > the OOM killer trigger? I bet this is the case. | > | > The machine doesn't have swap. | | Well then you're probably facing true OOM. | | Add some swap.
There is really no way to run 2.4 without swap?
I have the same problem with nfsroot and ramdisk based setups after 1-2 weeks uptime.
Quoted 11 lines · Gabor Z. Papp
* Marcelo Tosatti <[email protected]>:
| > > Can you check how much swap space is there available when | > > the OOM killer trigger? I bet this is the case. | > | > The machine doesn't have swap. | | Well then you're probably facing true OOM. | | Add some swap.
Quoted 1 line
There is really no way to run 2.4 without swap?
Nope. Any kernel can't. The thing is the system overcommits memory (it allows applications to allocate more memory than the system is able to handle).
If there is no place to "save" that memory once you run out of it, you're dead. Its a physical problem. :)
Quoted 2 lines
I have the same problem with nfsroot and ramdisk based setups after 1-2 weeks uptime.
You can try to remove the following lines from mm/vmscan.c::try_to_free_pages_zone()
if (likely(current->pid != 1))
break;
if (!check_classzone_need_balance(classzone))
break;And disable CONFIG_OOM_KILLER. See if that makes a difference.
What will happen is that the kernel will try to free memory and never go into the OOM killer. If it can't free memory, the system will hang forever at certain point.
There's not much to do about it really.
In 2.6 you can decrease swappiness so for it to free pagecache harder, but its the same game.
* Marcelo Tosatti <[email protected]>:
| > There is really no way to run 2.4 without swap? | | Nope. Any kernel can't. The thing is the system overcommits | memory (it allows applications to allocate more memory than the system | is able to handle).
Okay, then whats the required minimum swap size that needed to avoid such crashes?
In the case when the system is in the ram, quite funny to allocate a swap file on the ramdisk anyway...
Quoted 8 lines · Gabor Z. Papp
* Marcelo Tosatti <[email protected]>:
| > There is really no way to run 2.4 without swap? | | Nope. Any kernel can't. The thing is the system overcommits | memory (it allows applications to allocate more memory than the system | is able to handle).
Quoted 5 lines
Okay, then whats the required minimum swap size that needed to avoid such crashes?
In the case when the system is in the ram, quite funny to allocate a swap file on the ramdisk anyway...
It depends on how much memory you have an idea of the workload you have is going to use.
How much memory do you have? Try 128MB for a test.
Also investigate swap-over-nfs/swap-over-nbd.
* Marcelo Tosatti <[email protected]>:
| Also investigate swap-over-nfs/swap-over-nbd.
I would like something *stable* solution, but I can't find swap-over-nfs/swap-over-nbd in kernel. Thanks anyway.
Quoted 6 lines · Gabor Z. Papp
* Marcelo Tosatti <[email protected]>:
| Also investigate swap-over-nfs/swap-over-nbd.
I would like something *stable* solution, but I can't find swap-over-nfs/swap-over-nbd in kernel. Thanks anyway.
google.
Quoted 8 lines · Marcelo Tosatti
Quoted 2 lines · Gabor Z. Papp
There is really no way to run 2.4 without swap?
Nope. Any kernel can't. The thing is the system overcommits memory (it allows applications to allocate more memory than the system is able to handle).
If there is no place to "save" that memory once you run out of it, you're dead. Its a physical problem. :)
Hm, shouldn't there be command line option to instruct kernel not to overcommit memory? For servers that don't really need disk, and wouldn't have much use for swap anyhow (like firewalls for example). I do vaugly remember of such option in OSF/1 (Compaq Unix or whatever it is called nowdays) or Solaris. Not sure which one. Or maybe both. If I remember correctly, there the default was not to overcommit, and (not recommended) option would allow kernel to overcommit. Or I might be wrong about the default setting, but the option was there. I don't remember if application that tried to allocate more memory that there was free on the system would be killed or if system call would simply fail.
Signature · 3 lines
Aleksandar Milivojevic <[email protected]> Pollard Banknote Limited Systems Administrator 1499 Buffalo Place Tel: (204) 474-2323 ext 276 Winnipeg, MB R3T 1L7
Quoted 10 lines · Gabor Z. Papp
* Marcelo Tosatti <[email protected]>:
| > > Can you check how much swap space is there available when | > > the OOM killer trigger? I bet this is the case. | > | > The machine doesn't have swap. | | Well then you're probably facing true OOM. | | Add some swap.There is really no way to run 2.4 without swap?
Quoted 7 lines
I have the same problem with nfsroot and ramdisk based setups after 1-2 weeks uptime.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
sure, you can run a system without swap, you just run the risk of not being able to start new processes, or OOM kills if your running processees try to allocate too much memory. You can turn the OOM killer off if you like, but then you run the risk of wedging the machine.
Neil
Signature · 8 lines
/*************************************************** *Neil Horman *Software Engineer *Red Hat, Inc. *[email protected] *gpg keyid: 1024D / 0x92A74FA1 *http://pgp.mit.edu ***************************************************/
On Thursday 07 October 2004 20:28, Gabor Z. Papp wrote:
Hi all,
Quoted 9 lines
| > > Can you check how much swap space is there available when | > > the OOM killer trigger? I bet this is the case. | > The machine doesn't have swap. | Well then you're probably facing true OOM. | Add some swap.There is really no way to run 2.4 without swap?
I have the same problem with nfsroot and ramdisk based setups after 1-2 weeks uptime.
stop whining about braindead 2.4 mainline vm. Apply the attached patch and be happy :p
Marcelo: Is there something wrong with my VM documentation update patches for 2.4? Or do you not care and think: "Hello my friend, let's stick with 2.2 VM documentation even if almost all of the documentation is not longer valid"
;-)
ciao, Marc
include/linux/swap.h +1 -1
@@ -115,7 +115,7 @@ extern void swap_setup(void);extern wait_queue_head_t kswapd_wait;extern int FASTCALL (try_to_free_pages_zone (zone_t *,unsigned int ));extern int FASTCALL (try_to_free_pages (unsigned int )); -extern int vm_vfs_scan_ratio, vm_cache_scan_ratio, vm_lru_balance_ratio, vm_passes, vm_gfp_debug, vm_mapped_ratio; +extern int vm_vfs_scan_ratio, vm_cache_scan_ratio, vm_lru_balance_ratio, vm_passes, vm_gfp_debug, vm_mapped_ratio, vm_anon_lru;/* linux/mm/page_io.c */ extern void rw_swap_page (int ,struct page *);
include/linux/sysctl.h +2 -1
@@ -154,9 +154,10 @@ enumVM_PAGEBUF =17 ,/* struct: Control pagebuf parameters */ VM_GFP_DEBUG =18 ,/* debug GFP failures */ VM_CACHE_SCAN_RATIO =19 ,/* part of the inactive cache list to scan */ -VM_MAPPED_RATIO =20 ,/* amount of unfreeable pages that triggers swapout */ +VM_MAPPED_RATIO =20 ,/* amount of unfreeable pages that triggers swapout */ VM_LAPTOP_MODE =21 ,/* kernel in laptop flush mode */ VM_BLOCK_DUMP =22 ,/* dump fs activity to log */ + VM_ANON_LRU=23, /* immediatly insert anon pages in the vm page lru */ };
kernel/sysctl.c +2 -0
@@ -287,6 +287,8 @@ static ctl_table vm_table[] = { &vm_cache_scan_ratio, sizeof(int), 0644, NULL, &proc_dointvec}, {VM_MAPPED_RATIO ,"vm_mapped_ratio" , &vm_mapped_ratio, sizeof(int), 0644, NULL, &proc_dointvec}, + {VM_ANON_LRU, "vm_anon_lru", + &vm_anon_lru, sizeof(int), 0644, NULL, &proc_dointvec}, {VM_LRU_BALANCE_RATIO, "vm_lru_balance_ratio", &vm_lru_balance_ratio, sizeof(int), 0644, NULL, &proc_dointvec}, {VM_PASSES, "vm_passes",
mm/memory.c +6 -3
@@ -997,7 +997,8 @@ static int do_wp_page(struct mm_struct *if (PageReserved (old_page)) ++mm->rss;break_cow (vma, new_page, address, page_table); -lru_cache_add (new_page); +if (vm_anon_lru) +lru_cache_add (new_page);/* Free the old page.. */ new_page = old_page;
@@ -1228,7 +1229,8 @@ static int do_anonymous_page(struct mm_s mm->rss++;flush_page_to_ram (page); entry =pte_mkwrite (pte_mkdirty (mk_pte (page, vma->vm_page_prot))); -lru_cache_add (page); +if (vm_anon_lru) +lru_cache_add (page);mark_page_accessed (page); }
@@ -1283,7 +1285,8 @@ static int do_no_page(struct mm_struct * }copy_user_highpage (page, new_page, address);page_cache_release (new_page); -lru_cache_add (page); +if (vm_anon_lru) +lru_cache_add (page); new_page = page; }
mm/vmscan.c +21 -0
@@ -65,6 +65,27 @@ int vm_lru_balance_ratio = 2;int vm_vfs_scan_ratio =6 ;/* +* "vm_anon_lru" select if to immdiatly insert anon pages in the +* lru. Immediatly means as soon as they're allocated during the +* page faults. +* +* If this is set to 0, they're inserted only after the first +* swapout. +* +* Having anon pages immediatly inserted in the lru allows the +* VM to know better when it's worthwhile to start swapping +* anonymous ram, it will start to swap earlier and it should +* swap smoother and faster, but it will decrease scalability +* on the >16-ways of an order of magnitude. Big SMP/NUMA +* definitely can't take an hit on a global spinlock at +* every anon page allocation. So this is off by default. +* +* Low ram machines that swaps all the time want to turn +* this on (i.e. set to 1). +*/ +int vm_anon_lru =0 ; + +/* * The swap-out function returns 1 if it successfully * scanned all the pages it was asked to (`count'). * It returns zero if it couldn't do anything,
Quoted 10 lines · Marc-Christian Petersen
On Thursday 07 October 2004 20:28, Gabor Z. Papp wrote:
Hi all,
Quoted 5 lines
| > > Can you check how much swap space is there available when | > > the OOM killer trigger? I bet this is the case. | > The machine doesn't have swap. | Well then you're probably facing true OOM. | Add some swap.
Quoted 6 lines
Quoted 3 lines
There is really no way to run 2.4 without swap?
I have the same problem with nfsroot and ramdisk based setups after 1-2 weeks uptime.stop whining about braindead 2.4 mainline vm. Apply the attached patch and be happy :p
As I told you in private, I can't see how badly this patch could affect performance. But then, as you answered, with all anonymous pages added to LRU you see much better behavior (tons less swapping) on several workloads. That must be due to refill_inactive()/shrink_cache() balancing.
The same patch also fixes kswapd excessive CPU consumption on huge memory box.
Its easy enough to be applied because behaviour is unchanged by default (you need to change a sysctl value for that).
I would like to understand why does it cause so much improved behaviour though.
Quoted 3 lines
Marcelo: Is there something wrong with my VM documentation update patches for 2.4? Or do you not care and think: "Hello my friend, let's stick with 2.2 VM documentation even if almost all of the documentation is not longer valid"
As I said to you in private, please resend.
Thanks!
Quoted 15 lines · Marcelo Tosatti
Quoted 14 lines · Marc-Christian Petersen
On Thursday 07 October 2004 20:28, Gabor Z. Papp wrote:
Hi all,
Quoted 9 lines
| > > Can you check how much swap space is there available when | > > the OOM killer trigger? I bet this is the case. | > The machine doesn't have swap. | Well then you're probably facing true OOM. | Add some swap.There is really no way to run 2.4 without swap?
I have the same problem with nfsroot and ramdisk based setups after 1-2 weeks uptime.
Quoted 7 lines
Quoted 2 lines
stop whining about braindead 2.4 mainline vm. Apply the attached patch and be happy :p
As I told you in private, I can't see how badly this patch could affect performance. But then, as you answered, with all anonymous pages added to LRU you see much better behavior (tons less swapping) on several workloads. That must be due to refill_inactive()/shrink_cache() balancing.
Ah, I dont think this will fix the OOM killer cases with no swap. They look like plain OOM condition to me.
Wish I'm wrong.
Quoted 17 lines
The same patch also fixes kswapd excessive CPU consumption on huge memory box.
Its easy enough to be applied because behaviour is unchanged by default (you need to change a sysctl value for that).
I would like to understand why does it cause so much improved behaviour though.
Quoted 3 lines
Marcelo: Is there something wrong with my VM documentation update patches for 2.4? Or do you not care and think: "Hello my friend, let's stick with 2.2 VM documentation even if almost all of the documentation is not longer valid"
As I said to you in private, please resend.
Thanks!