Skip to contentSkip to conversation

[2.4] 0-order allocation failed

DISCUSSION Michael Buesch 7 participants 15 messages Oct 7, 2004 – Oct 8, 2004

Subsystems (6)
lore
Share & tools
lore
#0 lore raw

Hi all,

I'm running 2.4.28 bk snapshot of 2004.09.03 The machine has an uptime of 7 days, 23:46 now.

I was running several bittorrent clients inside of a screen session. Suddenly they all died (including the screen session).
dmesg sayed this:

__alloc_pages: 0-order allocation failed (gfp=0x1f0/0) __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process python
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process screen

I already got this with kernel 2.4.27 vanilla after a higher amount of uptime (I think it was over 10 days). This was exactly the reason I updated to bk snapshot.

What can be the reason for this? Is it OOM? (I can't really believe it is).
Is it a kernel memory leak?

With 2.4.26 I never got these errors. And I ran uptimes up to 50 days.

Signature · 1 line
Regards Michael Buesch  [ http://www.tuxsoft.de.vu ]
#2 lore raw
In reply to Marcelo Tosatti

Quoting Marcelo Tosatti <[email protected]>:

Quoted 22 lines · Michael Buesch
Quoted 21 lines

Hi all,

I'm running 2.4.28 bk snapshot of 2004.09.03 The machine has an uptime of 7 days, 23:46 now.

I was running several bittorrent clients inside of a screen session. Suddenly they all died (including the screen session).
dmesg sayed this:

__alloc_pages: 0-order allocation failed (gfp=0x1f0/0) __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process python
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process screen

I already got this with kernel 2.4.27 vanilla after a higher amount of uptime (I think it was over 10 days). This was exactly the reason I updated to bk snapshot.

Quoted 5 lines
Quoted 2 lines

What can be the reason for this? Is it OOM? (I can't really believe it is).

Can you check how much swap space is there available when the OOM killer trigger? I bet this is the case.

The machine doesn't have swap.

Quoted 8 lines

If its not, we have a problem.

Quoted 4 lines

Is it a kernel memory leak?

With 2.4.26 I never got these errors. And I ran uptimes up to 50 days.

Signature · 1 line
Regards Michael Buesch  [ http://www.tuxsoft.de.vu ]
#1 lore raw
In reply to Michael Buesch
Quoted 17 lines · Michael Buesch

Hi all,

I'm running 2.4.28 bk snapshot of 2004.09.03 The machine has an uptime of 7 days, 23:46 now.

I was running several bittorrent clients inside of a screen session. Suddenly they all died (including the screen session).
dmesg sayed this:

__alloc_pages: 0-order allocation failed (gfp=0x1f0/0) __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process python
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process screen

Quoted 6 lines

I already got this with kernel 2.4.27 vanilla after a higher amount of uptime (I think it was over 10 days). This was exactly the reason I updated to bk snapshot.

What can be the reason for this? Is it OOM? (I can't really believe it is).

Can you check how much swap space is there available when the OOM killer trigger? I bet this is the case.

If its not, we have a problem.

Quoted 4 lines

Is it a kernel memory leak?

With 2.4.26 I never got these errors. And I ran uptimes up to 50 days.

#3 lore raw
In reply to Michael Buesch
Quoted 23 lines · Michael Buesch

Quoting Marcelo Tosatti <[email protected]>:

Quoted 22 lines · Michael Buesch
Quoted 21 lines

Hi all,

I'm running 2.4.28 bk snapshot of 2004.09.03 The machine has an uptime of 7 days, 23:46 now.

I was running several bittorrent clients inside of a screen session. Suddenly they all died (including the screen session).
dmesg sayed this:

__alloc_pages: 0-order allocation failed (gfp=0x1f0/0) __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process python
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) __alloc_pages: 0-order allocation failed (gfp=0x1d2/0)
VM: killing process screen

I already got this with kernel 2.4.27 vanilla after a higher amount of uptime (I think it was over 10 days). This was exactly the reason I updated to bk snapshot.

Quoted 7 lines
Quoted 5 lines
Quoted 2 lines

What can be the reason for this? Is it OOM? (I can't really believe it is).

Can you check how much swap space is there available when the OOM killer trigger? I bet this is the case.

The machine doesn't have swap.

Well then you're probably facing true OOM.

Add some swap.

Quoted 6 lines
Quoted 6 lines

If its not, we have a problem.

Quoted 4 lines

Is it a kernel memory leak?

With 2.4.26 I never got these errors. And I ran uptimes up to 50 days.

#4 lore raw

* Marcelo Tosatti <[email protected]>:

| > > Can you check how much swap space is there available when
| > > the OOM killer trigger? I bet this is the case.
| > 
| > The machine doesn't have swap.
| 
| Well then you're probably facing true OOM.
| 
| Add some swap.

There is really no way to run 2.4 without swap?

I have the same problem with nfsroot and ramdisk based setups after 1-2 weeks uptime.

#6 lore raw
In reply to Marcelo Tosatti

* Marcelo Tosatti <[email protected]>:

| > There is really no way to run 2.4 without swap?
| 
| Nope. Any kernel can't. The thing is the system overcommits 
| memory (it allows applications to allocate more memory than the system 
| is able to handle).

Okay, then whats the required minimum swap size that needed to avoid such crashes?

In the case when the system is in the ram, quite funny to allocate a swap file on the ramdisk anyway...

#5 lore raw
In reply to Gabor Z. Papp
Quoted 11 lines · Gabor Z. Papp

* Marcelo Tosatti <[email protected]>:

| > > Can you check how much swap space is there available when
| > > the OOM killer trigger? I bet this is the case.
| > 
| > The machine doesn't have swap.
| 
| Well then you're probably facing true OOM.
| 
| Add some swap.
Quoted 1 line

There is really no way to run 2.4 without swap?

Nope. Any kernel can't. The thing is the system overcommits memory (it allows applications to allocate more memory than the system is able to handle).

If there is no place to "save" that memory once you run out of it, you're dead. Its a physical problem. :)

Quoted 2 lines

I have the same problem with nfsroot and ramdisk based setups after 1-2 weeks uptime.

You can try to remove the following lines from mm/vmscan.c::try_to_free_pages_zone()

        if (likely(current->pid != 1))
                break;
        if (!check_classzone_need_balance(classzone))
                break;

And disable CONFIG_OOM_KILLER. See if that makes a difference.

What will happen is that the kernel will try to free memory and never go into the OOM killer. If it can't free memory, the system will hang forever at certain point.

There's not much to do about it really.

In 2.6 you can decrease swappiness so for it to free pagecache harder, but its the same game.

#7 lore raw
In reply to Gabor Z. Papp
Quoted 8 lines · Gabor Z. Papp

* Marcelo Tosatti <[email protected]>:

| > There is really no way to run 2.4 without swap?
| 
| Nope. Any kernel can't. The thing is the system overcommits 
| memory (it allows applications to allocate more memory than the system 
| is able to handle).
Quoted 5 lines

Okay, then whats the required minimum swap size that needed to avoid such crashes?

In the case when the system is in the ram, quite funny to allocate a swap file on the ramdisk anyway...

It depends on how much memory you have an idea of the workload you have is going to use.

How much memory do you have? Try 128MB for a test.

Also investigate swap-over-nfs/swap-over-nbd.

#11 lore raw
In reply to Gabor Z. Papp
Quoted 10 lines · Gabor Z. Papp

* Marcelo Tosatti <[email protected]>:

| > > Can you check how much swap space is there available when
| > > the OOM killer trigger? I bet this is the case.
| > | > The machine doesn't have swap.
| | Well then you're probably facing true OOM.
| | Add some swap.

There is really no way to run 2.4 without swap?

Quoted 7 lines

I have the same problem with nfsroot and ramdisk based setups after 1-2 weeks uptime.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

sure, you can run a system without swap, you just run the risk of not being able to start new processes, or OOM kills if your running processees try to allocate too much memory. You can turn the OOM killer off if you like, but then you run the risk of wedging the machine.

Neil

Signature · 8 lines
/***************************************************
 *Neil Horman
 *Software Engineer
 *Red Hat, Inc.
 *[email protected]
 *gpg keyid: 1024D / 0x92A74FA1
 *http://pgp.mit.edu
 ***************************************************/
#12 lore raw
In reply to Gabor Z. Papp

On Thursday 07 October 2004 20:28, Gabor Z. Papp wrote:

Hi all,

Quoted 9 lines
| > > Can you check how much swap space is there available when
| > > the OOM killer trigger? I bet this is the case.
| > The machine doesn't have swap.
| Well then you're probably facing true OOM.
| Add some swap.

There is really no way to run 2.4 without swap?
I have the same problem with nfsroot and ramdisk based setups after 1-2 weeks uptime.

stop whining about braindead 2.4 mainline vm. Apply the attached patch and be happy :p

Marcelo: Is there something wrong with my VM documentation update patches for 2.4? Or do you not care and think: "Hello my friend, let's stick with 2.2 VM documentation even if almost all of the documentation is not longer valid"

;-)

ciao, Marc

include/linux/swap.h +1 -1
diff -urNp --exclude CVS --exclude BitKeeper --exclude {arch} --exclude .arch-ids x-ref/include/linux/swap.h x/include/linux/swap.h
@@ -115,7 +115,7 @@ extern void swap_setup(void);
 extern wait_queue_head_t kswapd_wait;
 extern int FASTCALL(try_to_free_pages_zone(zone_t *, unsigned int));
 extern int FASTCALL(try_to_free_pages(unsigned int));
-extern int vm_vfs_scan_ratio, vm_cache_scan_ratio, vm_lru_balance_ratio, vm_passes, vm_gfp_debug, vm_mapped_ratio;
+extern int vm_vfs_scan_ratio, vm_cache_scan_ratio, vm_lru_balance_ratio, vm_passes, vm_gfp_debug, vm_mapped_ratio, vm_anon_lru;
 
 /* linux/mm/page_io.c */
 extern void rw_swap_page(int, struct page *);
include/linux/sysctl.h +2 -1
diff -urNp --exclude CVS --exclude BitKeeper --exclude {arch} --exclude .arch-ids x-ref/include/linux/sysctl.h x/include/linux/sysctl.h
@@ -154,9 +154,10 @@ enum
 	VM_PAGEBUF=17,		/* struct: Control pagebuf parameters */
 	VM_GFP_DEBUG=18,        /* debug GFP failures */
 	VM_CACHE_SCAN_RATIO=19, /* part of the inactive cache list to scan */
-	VM_MAPPED_RATIO=20,     /* amount of unfreeable pages that triggers swapout */
+	VM_MAPPED_RATIO=20,	/* amount of unfreeable pages that triggers swapout */
 	VM_LAPTOP_MODE=21,	/* kernel in laptop flush mode */
 	VM_BLOCK_DUMP=22,	/* dump fs activity to log */
+	VM_ANON_LRU=23,		/* immediatly insert anon pages in the vm page lru */
 };
 
 
kernel/sysctl.c +2 -0
diff -urNp --exclude CVS --exclude BitKeeper --exclude {arch} --exclude .arch-ids x-ref/kernel/sysctl.c x/kernel/sysctl.c
@@ -287,6 +287,8 @@ static ctl_table vm_table[] = {
 	 &vm_cache_scan_ratio, sizeof(int), 0644, NULL, &proc_dointvec},
 	{VM_MAPPED_RATIO, "vm_mapped_ratio", 
 	 &vm_mapped_ratio, sizeof(int), 0644, NULL, &proc_dointvec},
+	{VM_ANON_LRU, "vm_anon_lru", 
+	 &vm_anon_lru, sizeof(int), 0644, NULL, &proc_dointvec},
 	{VM_LRU_BALANCE_RATIO, "vm_lru_balance_ratio", 
 	 &vm_lru_balance_ratio, sizeof(int), 0644, NULL, &proc_dointvec},
 	{VM_PASSES, "vm_passes", 
mm/memory.c +6 -3
diff -urNp --exclude CVS --exclude BitKeeper --exclude {arch} --exclude .arch-ids x-ref/mm/memory.c x/mm/memory.c
@@ -997,7 +997,8 @@ static int do_wp_page(struct mm_struct *
 		if (PageReserved(old_page))
 			++mm->rss;
 		break_cow(vma, new_page, address, page_table);
-		lru_cache_add(new_page);
+		if (vm_anon_lru)
+			lru_cache_add(new_page);
 
 		/* Free the old page.. */
 		new_page = old_page;
@@ -1228,7 +1229,8 @@ static int do_anonymous_page(struct mm_s
 		mm->rss++;
 		flush_page_to_ram(page);
 		entry = pte_mkwrite(pte_mkdirty(mk_pte(page, vma->vm_page_prot)));
-		lru_cache_add(page);
+		if (vm_anon_lru)
+			lru_cache_add(page);
 		mark_page_accessed(page);
 	}
 
@@ -1283,7 +1285,8 @@ static int do_no_page(struct mm_struct *
 		}
 		copy_user_highpage(page, new_page, address);
 		page_cache_release(new_page);
-		lru_cache_add(page);
+		if (vm_anon_lru)
+			lru_cache_add(page);
 		new_page = page;
 	}
 
mm/vmscan.c +21 -0
diff -urNp --exclude CVS --exclude BitKeeper --exclude {arch} --exclude .arch-ids x-ref/mm/vmscan.c x/mm/vmscan.c
@@ -65,6 +65,27 @@ int vm_lru_balance_ratio = 2;
 int vm_vfs_scan_ratio = 6;
 
 /*
+ * "vm_anon_lru" select if to immdiatly insert anon pages in the
+ * lru. Immediatly means as soon as they're allocated during the
+ * page faults.
+ *
+ * If this is set to 0, they're inserted only after the first
+ * swapout.
+ *
+ * Having anon pages immediatly inserted in the lru allows the
+ * VM to know better when it's worthwhile to start swapping
+ * anonymous ram, it will start to swap earlier and it should
+ * swap smoother and faster, but it will decrease scalability
+ * on the >16-ways of an order of magnitude. Big SMP/NUMA
+ * definitely can't take an hit on a global spinlock at
+ * every anon page allocation. So this is off by default.
+ *
+ * Low ram machines that swaps all the time want to turn
+ * this on (i.e. set to 1).
+ */
+int vm_anon_lru = 0;
+
+/*
  * The swap-out function returns 1 if it successfully
  * scanned all the pages it was asked to (`count').
  * It returns zero if it couldn't do anything,
#10 lore raw
In reply to Marcelo Tosatti
Quoted 8 lines · Marcelo Tosatti
Quoted 2 lines · Gabor Z. Papp

There is really no way to run 2.4 without swap?

Nope. Any kernel can't. The thing is the system overcommits memory (it allows applications to allocate more memory than the system is able to handle).

If there is no place to "save" that memory once you run out of it, you're dead. Its a physical problem. :)

Hm, shouldn't there be command line option to instruct kernel not to overcommit memory? For servers that don't really need disk, and wouldn't have much use for swap anyhow (like firewalls for example). I do vaugly remember of such option in OSF/1 (Compaq Unix or whatever it is called nowdays) or Solaris. Not sure which one. Or maybe both. If I remember correctly, there the default was not to overcommit, and (not recommended) option would allow kernel to overcommit. Or I might be wrong about the default setting, but the option was there. I don't remember if application that tried to allocate more memory that there was free on the system would be killed or if system call would simply fail.

Signature · 3 lines
Aleksandar Milivojevic <[email protected]>    Pollard Banknote Limited
Systems Administrator                           1499 Buffalo Place
Tel: (204) 474-2323 ext 276                     Winnipeg, MB  R3T 1L7
#13 lore raw
In reply to Marc-Christian Petersen
Quoted 10 lines · Marc-Christian Petersen

On Thursday 07 October 2004 20:28, Gabor Z. Papp wrote:

Hi all,

Quoted 5 lines
| > > Can you check how much swap space is there available when
| > > the OOM killer trigger? I bet this is the case.
| > The machine doesn't have swap.
| Well then you're probably facing true OOM.
| Add some swap.
Quoted 6 lines
Quoted 3 lines

There is really no way to run 2.4 without swap?
I have the same problem with nfsroot and ramdisk based setups after 1-2 weeks uptime.

stop whining about braindead 2.4 mainline vm. Apply the attached patch and be happy :p

As I told you in private, I can't see how badly this patch could affect performance. But then, as you answered, with all anonymous pages added to LRU you see much better behavior (tons less swapping) on several workloads. That must be due to refill_inactive()/shrink_cache() balancing.

The same patch also fixes kswapd excessive CPU consumption on huge memory box.

Its easy enough to be applied because behaviour is unchanged by default (you need to change a sysctl value for that).

I would like to understand why does it cause so much improved behaviour though.

Quoted 3 lines

Marcelo: Is there something wrong with my VM documentation update patches for 2.4? Or do you not care and think: "Hello my friend, let's stick with 2.2 VM documentation even if almost all of the documentation is not longer valid"

As I said to you in private, please resend.

Thanks!

#14 lore raw
Quoted 15 lines · Marcelo Tosatti
Quoted 14 lines · Marc-Christian Petersen

On Thursday 07 October 2004 20:28, Gabor Z. Papp wrote:

Hi all,

Quoted 9 lines
| > > Can you check how much swap space is there available when
| > > the OOM killer trigger? I bet this is the case.
| > The machine doesn't have swap.
| Well then you're probably facing true OOM.
| Add some swap.

There is really no way to run 2.4 without swap?
I have the same problem with nfsroot and ramdisk based setups after 1-2 weeks uptime.

Quoted 7 lines
Quoted 2 lines

stop whining about braindead 2.4 mainline vm. Apply the attached patch and be happy :p

As I told you in private, I can't see how badly this patch could affect performance. But then, as you answered, with all anonymous pages added to LRU you see much better behavior (tons less swapping) on several workloads. That must be due to refill_inactive()/shrink_cache() balancing.

Ah, I dont think this will fix the OOM killer cases with no swap. They look like plain OOM condition to me.

Wish I'm wrong.

Quoted 17 lines

The same patch also fixes kswapd excessive CPU consumption on huge memory box.

Its easy enough to be applied because behaviour is unchanged by default (you need to change a sysctl value for that).

I would like to understand why does it cause so much improved behaviour though.

Quoted 3 lines

Marcelo: Is there something wrong with my VM documentation update patches for 2.4? Or do you not care and think: "Hello my friend, let's stick with 2.2 VM documentation even if almost all of the documentation is not longer valid"

As I said to you in private, please resend.

Thanks!