Skip to contentSkip to conversation

__alloc_pages: 0-order allocation failed

DISCUSSION Jacek Popławski 8 participants 28 messages Sep 24, 2001 – Sep 26, 2001

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

I just installed 2.4.10, and...

__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process donkey_s
__alloc_pages: 0-order allocation failed (gfp=0x1f0/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process screen
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process bash
(...)

I am changing kernels often, but never seen that kind of message. Last kernel I had before 2.4.10 was 2.4.10-pre4.

PS. donkey_s is application which eats a lot of memory, but I have 384MB RAM and 100MB swap.

#1 lore raw
Quoted 13 lines · On Mon, 24 Sep 2001, Jacek [iso-8859-2] Pop³awski wrote:

I just installed 2.4.10, and...

__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process donkey_s
__alloc_pages: 0-order allocation failed (gfp=0x1f0/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process screen
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process bash
(...)

Quoted 5 lines

I am changing kernels often, but never seen that kind of message. Last kernel I had before 2.4.10 was 2.4.10-pre4.

PS. donkey_s is application which eats a lot of memory, but I have 384MB RAM and 100MB swap.

Jacek,

You had available swap when the VM started to kill processes ?

#2 lore raw
Quoted 10 lines · Marcelo Tosatti
Quoted 6 lines · On Mon, 24 Sep 2001, Jacek [iso-8859-2] Pop³awski wrote:

I just installed 2.4.10, and...

__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process donkey_s

...

I'm getting a lot of this with 2.4.10 also. At the time, I had KDM running, but I was coming into the box over telnet and running the latest released version of LTP. The test it appeared to be on at the time was a filesystem test called growfiles. Nothing else was running other than these things and standard system services. The machine has 256 MB of ram, and 512 MB swap. The order that things got killed in were sadc, sar, kdm, X, in.telnetd, xinetd (ouch).

__alloc_pages: 0-order allocation failed (gfp=0xf0/0) from c012b9b2 __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c012b9b2
VM: killing process xinetd
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c012b9b2 __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c012b9b2 __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c012b9b2

Thanks,
Paul Larson

#3 lore raw

I just remembered that my sar output file is still around from before sar got killed, so I thought you might like to see the tail end of -r on it. Memory utilization got pretty high before sar got killed, but swap was still at almost nothing:

07:34:18 kbmemfree kbmemused %memused kbmemshrd kbbuffers kbcached kbswpfree kbswpused %swpused

07:34:19        59620    194728     76.56         0      6052    146688    530136         0      0.00
07:34:20        50572    203776     80.12         0      6052    155736    530136         0      0.00
07:34:21        41488    212860     83.69         0      6052    164820    530136         0      0.00
07:34:22        32484    221864     87.23         0      6056    173820    530136         0      0.00
07:34:23        23372    230976     90.81         0      6064    182924    530136         0      0.00
07:34:24        14012    240336     94.49         0      6072    192276    530136         0      0.00
07:34:25         4980    249368     98.04         0      6080    201300    530136         0      0.00
07:34:26         3876    250472     98.48         0      6088    202468    530136         0      0.00
07:34:27         3664    250684     98.56         0      3324    205444    530136         0      0.00
07:34:28         3560    250788     98.60         0      1248    207624    530136         0      0.00
07:34:29         3764    250584     98.52         0       244    208424    530136         0      0.00
07:34:30         4180    250168     98.36         0       148    209272    529532       604      0.11
07:34:31         4072    250276     98.40         0       100    213304    526196      3940      0.74
07:34:32         3488    250860     98.63         0        96    218204    519968     10168      1.92
Average:       122043    132305     52.02         0      5848     84739    530055        81      0.02

-Paul Larson

Quoted 24 lines · Paul Larson

I'm getting a lot of this with 2.4.10 also. At the time, I had KDM running, but I was coming into the box over telnet and running the latest released version of LTP. The test it appeared to be on at the time was a filesystem test called growfiles. Nothing else was running other than these things and standard system services. The machine has 256 MB of ram, and 512 MB swap. The order that things got killed in were sadc, sar, kdm, X, in.telnetd, xinetd (ouch).

__alloc_pages: 0-order allocation failed (gfp=0xf0/0) from c012b9b2 __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c012b9b2
VM: killing process xinetd
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c012b9b2 __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c012b9b2 __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c012b9b2

Thanks,
Paul Larson

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#4 lore raw
In reply to Paul Larson
Quoted 22 lines · Paul Larson
Quoted 10 lines · Marcelo Tosatti
Quoted 6 lines · On Mon, 24 Sep 2001, Jacek [iso-8859-2] Pop³awski wrote:

I just installed 2.4.10, and...

__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process donkey_s

...

I'm getting a lot of this with 2.4.10 also. At the time, I had KDM running, but I was coming into the box over telnet and running the latest released version of LTP. The test it appeared to be on at the time was a filesystem test called growfiles. Nothing else was running other than these things and standard system services. The machine has 256 MB of ram, and 512 MB swap. The order that things got killed in were sadc, sar, kdm, X, in.telnetd, xinetd (ouch).

Quoted 6 lines

__alloc_pages: 0-order allocation failed (gfp=0xf0/0) from c012b9b2 __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c012b9b2
VM: killing process xinetd
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c012b9b2 __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c012b9b2 __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c012b9b2

For the people having the allocation failure problems, please try the following patch.

Currently the page freeing success accounting is completly broken (it does not report a task has made progress while it did), and the page allocation code uses that information to know if it should or not try to keep calling the freeing code.

Please test this.

mm/vmscan.c +3 -0
@@ -567,6 +567,9 @@
 		if (nr_pages <= 0)
 			return 1;
 
+		if (nr_pages < SWAP_CLUSTER_MAX)
+			ret |= 1;
+
 		ret |= swap_out(priority, classzone, gfp_mask, SWAP_CLUSTER_MAX << 2);
 	} while (--priority);
 
mm/page_alloc.c +1 -1
@@ -400,7 +400,7 @@
 			if (!z)
 				break;
 
-			if (zone_free_pages(z, order) > z->pages_high) {
+			if (zone_free_pages(z, order) > z->pages_min) {
 				page = rmqueue(z, order);
 				if (page)
 					return page;
#5 lore raw
Quoted 2 lines

For the people having the allocation failure problems, please try the following patch.

I tried it. No success.
dmesg: after the "bad" program I wrote some mail ago.

__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0123cc1
VM: killing process a.out

I forgot to mention:

c0123c60 t page_cache_read
c0123d10 t read_cluster_nonblocking

If you need a backtrace, we should insert a panic into the code to get a full back trace for the debugging.

greetings

Christian Bornträger

#6 lore raw
In reply to Marcelo Tosatti

The patch helped for me, but there are still problems. I was able to run all the way through LTP without it shutting anything down. When I used one of the memory tests to chew up all the ram though, I noticed that VM was killing things it shouldn't have. First thing to get killed was cron, then top, then it finally killed mtest01 (the memory test mentioned before).

#7 lore raw
Quoted 7 lines · Paul Larson

The patch helped for me, but there are still problems. I was able to run all the way through LTP without it shutting anything down. When I used one of the memory tests to chew up all the ram though, I noticed that VM was killing things it shouldn't have. First thing to get killed was cron, then top, then it finally killed mtest01 (the memory test mentioned before).

Ok, its good to know that the patch at least helped.

Watch out for another patch in hours.

#8 lore raw
In reply to Paul Larson

Just to confirm I'm seeing this also. I've a machine with 512mb ram and a gig of swap. Running a filesystem i/o stress test app causes the machine to pretty much run out of memory. The swap is hardly touched. Then the VM starts killing things...klogd, the file i/o app, the shell it was in...

I didn't see any significant change here with the patch.

Here's meminfo prior to and towards the end of things for what it's worth:

[root@foobox /root]# cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 526299136 66060288 460238848 0 864256 21032960
Swap: 1074765824 4374528 1070391296
MemTotal: 513964 kB
MemFree: 449452 kB
MemShared: 0 kB
Buffers: 844 kB
Cached: 16268 kB
SwapCached: 4272 kB
Active: 5804 kB
Inactive: 15580 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 513964 kB
LowFree: 449452 kB
SwapTotal: 1049576 kB
SwapFree: 1045304 kB

[root@foobox /root]# cat /proc/meminfo
total: used: free: shared: buffers: cached:
Mem: 526299136 522604544 3694592 0 1212416 435134464
Swap: 1074765824 3366912 1071398912
MemTotal: 513964 kB
MemFree: 3608 kB
MemShared: 0 kB
Buffers: 1184 kB
Cached: 424784 kB
SwapCached: 152 kB
Active: 356640 kB
Inactive: 69480 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 513964 kB
LowFree: 3608 kB
SwapTotal: 1049576 kB
SwapFree: 1046288 kB

#9 lore raw
In reply to Marcelo Tosatti
Quoted 7 lines · Paul Larson

The patch helped for me, but there are still problems. I was able to run all the way through LTP without it shutting anything down. When I used one of the memory tests to chew up all the ram though, I noticed that VM was killing things it shouldn't have. First thing to get killed was cron, then top, then it finally killed mtest01 (the memory test mentioned before).

can you reproduce anything wrong with vm-tweaks-2 applied to plain 2.4.10? I posted it to l-k a few minutes ago.

Andrea

#10 lore raw
In reply to Paul Larson

mm/vmscan.c line 572

Quoted 9 lines · Marcelo Tosatti
mm/vmscan.c +3 -0
@@ -567,6 +567,9 @@
 		if (nr_pages <= 0)
 			return 1;
 
+		if (nr_pages < SWAP_CLUSTER_MAX)
+			ret |= 1;
+

too much permissive (vm-tweaks-1 does something similar but not that permissive)

mm/page_alloc.c line 403

Quoted 11 lines
mm/vmscan.c +0 -0
 		ret |= swap_out(priority, classzone, gfp_mask, SWAP_CLUSTER_MAX << 2);
 	} while (--priority);
 
mm/page_alloc.c +1 -1
@@ -400,7 +400,7 @@
 			if (!z)
 				break;
 
-			if (zone_free_pages(z, order) > z->pages_high) {
+			if (zone_free_pages(z, order) > z->pages_min) {

that breaks oom detection.

Andrea

#11 lore raw
Quoted 7 lines · Andrea Arcangeli
Quoted 6 lines · Marcelo Tosatti
mm/vmscan.c +0 -0
@@ -567,6 +567,9 @@
 		if (nr_pages <= 0)
 			return 1;
 
Quoted 6 lines

mm/vmscan.c line 572

Quoted 3 lines
mm/vmscan.c +3 -0
+		if (nr_pages < SWAP_CLUSTER_MAX)
+			ret |= 1;
+

too much permissive (vm-tweaks-1 does something similar but not that permissive)

Andrea,

Does vm-tweaks-1 fixes the current problem we're seeing?

Also, we have to make sure _all_ progress accounting is being done correctly (i/dcache, etc). I'll make sure that happens as soon as the OOM problem is gone.

Quoted 9 lines
Quoted 9 lines
mm/vmscan.c +0 -0
 		ret |= swap_out(priority, classzone, gfp_mask, SWAP_CLUSTER_MAX << 2);
 	} while (--priority);
 
mm/page_alloc.c +0 -0
@@ -400,7 +400,7 @@
 			if (!z)
 				break;
 
Quoted 4 lines

mm/page_alloc.c line 403

Quoted 2 lines
mm/page_alloc.c +1 -1
-			if (zone_free_pages(z, order) > z->pages_high) {
+			if (zone_free_pages(z, order) > z->pages_min) {

that breaks oom detection.

Why?

#12 lore raw
In reply to Andrea Arcangeli
Quoted 18 lines · Marcelo Tosatti
Quoted 7 lines · Andrea Arcangeli
Quoted 6 lines · Marcelo Tosatti
mm/vmscan.c +0 -0
@@ -567,6 +567,9 @@
 		if (nr_pages <= 0)
 			return 1;
 
Quoted 6 lines

mm/vmscan.c line 572

Quoted 3 lines
mm/vmscan.c +3 -0
+		if (nr_pages < SWAP_CLUSTER_MAX)
+			ret |= 1;
+

too much permissive (vm-tweaks-1 does something similar but not that permissive)

Quoted 3 lines

Andrea,

Does vm-tweaks-1 fixes the current problem we're seeing?

it seems no by reading the last email, however I'm not seeing any problem, the DEBUG_GFP will tell us where the problem cames from, pssobly it's a highmem thing since I never reproduced anything bad here. But the point is that the above isn't going to be a right fix anyways.

Quoted 13 lines

Also, we have to make sure _all_ progress accounting is being done correctly (i/dcache, etc). I'll make sure that happens as soon as the OOM problem is gone.

Quoted 9 lines
Quoted 9 lines
mm/vmscan.c +0 -0
 		ret |= swap_out(priority, classzone, gfp_mask, SWAP_CLUSTER_MAX << 2);
 	} while (--priority);
 
mm/page_alloc.c +0 -0
@@ -400,7 +400,7 @@
 			if (!z)
 				break;
 
Quoted 6 lines
Quoted 4 lines

mm/page_alloc.c line 403

Quoted 2 lines
mm/page_alloc.c +1 -1
-			if (zone_free_pages(z, order) > z->pages_high) {
+			if (zone_free_pages(z, order) > z->pages_min) {

that breaks oom detection.

Why?

the only point such code exists is to try to kill only one task, if the killed task was the right one, such code can be dropped and still the machine must not fail allocation unless truly oom, so any change there cannot obviously fix anything related to early-oom-by-mistake.

Andrea

#13 lore raw
In reply to Andrea Arcangeli
Quoted 8 lines · On Wed 26 Sep at 01:05:16 +0200 [email protected] done said:
Quoted 2 lines · Marcelo Tosatti

Does vm-tweaks-1 fixes the current problem we're seeing?

it seems no by reading the last email, however I'm not seeing any problem, the DEBUG_GFP will tell us where the problem cames from, pssobly it's a highmem thing since I never reproduced anything bad here. But the point is that the above isn't going to be a right fix anyways.

vm-tweaks-1 fixes things for me. I've got 512MB ram (kernel not configured for highmem) and 1 gig of swap. The workload is heavy file i/o and has now been running almost 24 hours (about 2 billion I/Os or a few TB of data I think so far). Previously all the memory was being consumed by cache, nothing swapped (as expected if the memory is cached buffer i/o right?) and I'd get the:
__alloc_pages: 0-order allocation failed
Now I continue to see the memory consumption / no swap, and no more error...iow the expected behaviour.

On an unrelated note if I want to backport the async I/O changes in 2.4.10, are there patches from you I should apply other than:

	2.4.10pre10aa1/40_blkdev-pagecache-17
	2.4.7pre8aa1/41_blkdev-pagecache-5_drop_get_bh_async-1

Tim

#14 lore raw
In reply to Andrea Arcangeli
Quoted 10 lines · On Wed, Sep 26, 2001 at 11:15:09AM -0700, [email protected] wrote:
Quoted 8 lines · On Wed 26 Sep at 01:05:16 +0200 [email protected] done said:
Quoted 2 lines · Marcelo Tosatti

Does vm-tweaks-1 fixes the current problem we're seeing?

it seems no by reading the last email, however I'm not seeing any problem, the DEBUG_GFP will tell us where the problem cames from, pssobly it's a highmem thing since I never reproduced anything bad here. But the point is that the above isn't going to be a right fix anyways.

Quoted 6 lines

vm-tweaks-1 fixes things for me. I've got 512MB ram (kernel not configured for highmem) and 1 gig of swap. The workload is heavy file i/o and has now been running almost 24 hours (about 2 billion I/Os or a few TB of data I think so far). Previously all the memory was being consumed by cache, nothing swapped (as expected if the memory is cached buffer i/o right?) and I'd get the:

yes, unless the buffered I/O was identified as your very working set but even in such case the 2.4.10 vm shouldn't swapout too early.

Quoted 3 lines

__alloc_pages: 0-order allocation failed
Now I continue to see the memory consumption / no swap, and no more error...iow the expected behaviour.

good. As far I can tell it is the check in swap_out that is making the difference and fixing the oom problem, it was very intentional indeed.

Quoted 4 lines

On an unrelated note if I want to backport the async I/O changes in 2.4.10, are there patches from you I should apply other than:

	2.4.10pre10aa1/40_blkdev-pagecache-17
	2.4.7pre8aa1/41_blkdev-pagecache-5_drop_get_bh_async-1

both patches are now included in mainline 2.4.10.

thanks,
Andrea

#15 lore raw
In reply to Paul Larson
Quoted 2 lines

too much permissive (vm-tweaks-1 does something similar but not that permissive)

But it doesnt help neither. I installed vm-tweaks-1 on a vanilla 2.4.10 and still got an __alloc_pages: 0-order allocation failure
I have no swap and 512 MB of RAM.

#16 lore raw
In reply to Paul Larson
Quoted 6 lines · Christian Bornträger
Quoted 2 lines

too much permissive (vm-tweaks-1 does something similar but not that permissive)

But it doesnt help neither. I installed vm-tweaks-1 on a vanilla 2.4.10 and still got an __alloc_pages: 0-order allocation failure
I have no swap and 512 MB of RAM.

Could you enable CONFIG_DEBUG_GFP (kernel debugging menu) in 2.4.10aa1 and send me full traces of the faliures so I can better see where the problem cames from? Thanks!

Andrea

#17 lore raw
In reply to Paul Larson
Quoted 5 lines

Could you enable CONFIG_DEBUG_GFP (kernel debugging menu) in 2.4.10aa1 and send me full traces of the faliures so I can better see where the problem cames from? Thanks!

Andrea

Is it enough, to take a vanilla 2.4.10 and apply 00_debug-gfp-1 and 00_vm-tweaks-1 or should I patch to a complete aa1. If yes, where can I find the complete aa1-Patch in one part?

greetings

Christian

#18 lore raw
In reply to Paul Larson
Quoted 8 lines · Christian Bornträger
Quoted 5 lines

Could you enable CONFIG_DEBUG_GFP (kernel debugging menu) in 2.4.10aa1 and send me full traces of the faliures so I can better see where the problem cames from? Thanks!

Andrea

Is it enough, to take a vanilla 2.4.10 and apply 00_debug-gfp-1 and 00_vm-tweaks-1 or should I patch to a complete aa1. If yes, where can I find

yes, that's enough, you don't need the complete aa1 just for that.

Quoted 1 line

the complete aa1-Patch in one part?

The whole aa1 patch can be found in the v2.4 directory called as 2.4.10aa1.bz2.

Andrea

#19 lore raw
In reply to Paul Larson
Quoted 3 lines

Could you enable CONFIG_DEBUG_GFP (kernel debugging menu) in 2.4.10aa1 and send me full traces of the faliures so I can better see where the problem cames from? Thanks!

OK, with the vm-tweaks and the gfp-patch I got the following output:

__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) dcb81e30 c01fd640 00000000 000001d2 00000000 00000000 dcb81e5c 00000001

       c0223848 c02239b8 000001d2 00000000 00000001 c1978d40 de9f30c0 de9855a0
       c01219d7 00000000 00000000 c012b35f c0223848 de9f30c0 c1978d40 00000001
Call Trace: [<c01219d7>] [<c012b35f>] [<c0121a60>] [<c0121b40>] [<c011152e>]
   [<c010adc2>] [<c0121e27>] [<c01113b0>] [<c0106dec>]

VM: killing process a.out

feeding it to ksymoops....:

dcb81e30 c01fd640 00000000 000001d2 00000000 00000000 dcb81e5c 00000001

       c0223848 c02239b8 000001d2 00000000 00000001 c1978d40 de9f30c0 de9855a0
       c01219d7 00000000 00000000 c012b35f c0223848 de9f30c0 c1978d40 00000001
Call Trace: [<c01219d7>] [<c012b35f>] [<c0121a60>] [<c0121b40>] [<c011152e>]
   [<c010adc2>] [<c0121e27>] [<c01113b0>] [<c0106dec>]

Warning (Oops_read): Code line not seen, dumping what data is available

Trace; c01219d7 <do_anonymous_page+37/90>
Trace; c012b35f <__alloc_pages+4f/240>
Trace; c0121a60 <do_no_page+30/b0>
Trace; c0121b40 <handle_mm_fault+60/d0>
Trace; c011152e <do_page_fault+17e/4b0>
Trace; c010adc2 <timer_interrupt+62/110>
Trace; c0121e27 <sys_brk+b7/f0>
Trace; c01113b0 <do_page_fault+0/4b0>
Trace; c0106dec <error_code+34/3c>

If you need a run on a complete aa1-patch, let me know.

greetings

Christian

#20 lore raw
In reply to Paul Larson
Quoted 36 lines · Christian Bornträger
Quoted 3 lines

Could you enable CONFIG_DEBUG_GFP (kernel debugging menu) in 2.4.10aa1 and send me full traces of the faliures so I can better see where the problem cames from? Thanks!

OK, with the vm-tweaks and the gfp-patch I got the following output:

__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) dcb81e30 c01fd640 00000000 000001d2 00000000 00000000 dcb81e5c 00000001

       c0223848 c02239b8 000001d2 00000000 00000001 c1978d40 de9f30c0 de9855a0
       c01219d7 00000000 00000000 c012b35f c0223848 de9f30c0 c1978d40 00000001
Call Trace: [<c01219d7>] [<c012b35f>] [<c0121a60>] [<c0121b40>] [<c011152e>]
   [<c010adc2>] [<c0121e27>] [<c01113b0>] [<c0106dec>]

VM: killing process a.out

feeding it to ksymoops....:

dcb81e30 c01fd640 00000000 000001d2 00000000 00000000 dcb81e5c 00000001

       c0223848 c02239b8 000001d2 00000000 00000001 c1978d40 de9f30c0 de9855a0
       c01219d7 00000000 00000000 c012b35f c0223848 de9f30c0 c1978d40 00000001
Call Trace: [<c01219d7>] [<c012b35f>] [<c0121a60>] [<c0121b40>] [<c011152e>]
   [<c010adc2>] [<c0121e27>] [<c01113b0>] [<c0106dec>]

Warning (Oops_read): Code line not seen, dumping what data is available

Trace; c01219d7 <do_anonymous_page+37/90>
Trace; c012b35f <__alloc_pages+4f/240>
Trace; c0121a60 <do_no_page+30/b0>
Trace; c0121b40 <handle_mm_fault+60/d0>
Trace; c011152e <do_page_fault+17e/4b0>
Trace; c010adc2 <timer_interrupt+62/110>
Trace; c0121e27 <sys_brk+b7/f0>
Trace; c01113b0 <do_page_fault+0/4b0>
Trace; c0106dec <error_code+34/3c>

Quoted 1 line

If you need a run on a complete aa1-patch, let me know.

ok, this sounds like a normal oom condition but of course I assume it isn't. Can you show a vmstat 1 during the oom kill + some /proc/meminfo? thanks for the great feedback!

Andrea

#21 lore raw
In reply to Paul Larson
Quoted 3 lines

ok, this sounds like a normal oom condition but of course I assume it isn't. Can you show a vmstat 1 during the oom kill + some /proc/meminfo? thanks for the great feedback!

As the VM sometimes killed me some long lasting root processes, like klogd, I think it is not the normal oom behaviour........ :-)

vmstat:

procs memory swap io system
cpu

 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us  sy id
 0  0  0      0 457824   8476  28856   0   0   225   140  156   163   4  51 45
 0  0  0      0 457440   8476  28860   0   0     0   280  143   103   1  52 47
 0  0  0      0 457440   8476  28860   0   0     0     0  101    70   1  60 39
 0  0  0      0 457440   8476  28860   0   0     0     0  109    86   1  56 43
 0  0  0      0 457436   8476  28864   0   0     0     0  102    73   1  59 40
 0  0  1      0 457436   8476  28864   0   0     0     0  101    70   1  58 41
 0  0  0      0 457436   8476  28864   0   0     0   236  130    88   0  59 41
 0  0  0      0 457436   8476  28864   0   0     0     0  101    69   1  60 39
 0  0  0      0 457436   8476  28864   0   0     0     0  101    71   1  60 39
 0  0  0      0 457436   8476  28864   0   0     0     0  101    71   0  60 40
 1  0  0      0 168236   8476  28928   0   0    64     0  117    79  11  72 17
 1  2  0      0 496532    236   1660   0   0  2948   252  415   303   5  75 20
 0  0  0      0 495568    408   2364   0   0   708     0  184   206   1  26 73
 0  0  1      0 495532    412   2396   0   0    36     0  105    78   1  59 40
 0  0  0      0 495528    416   2396   0   0     0     0  106    81   1  60 39
 0  0  0      0 495376    560   2396   0   0     0   272  139   103   1  56 43
 0  0  0      0 495376    560   2396   0   0     0     0  104    78   1  60 39

meminfo: (a made a looping bash script with sleep 1 after a cat /proc/meminfo. But During the kill there is a 3 second gap) Mit Sep 26 09:14:58 CEST 2001

total: used: free: shared: buffers: cached:

Mem:  526491648 58064896 468426752        0  8679424 29556736
Swap:        0        0        0
MemTotal:       514152 kB
MemFree:        457448 kB
MemShared:           0 kB
Buffers:          8476 kB
Cached:          28864 kB
SwapCached:          0 kB
Active:          19084 kB
Inactive:        18256 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       514152 kB
LowFree:        457448 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Mit Sep 26 09:14:59 CEST 2001
        total:    used:    free:  shared: buffers:  cached:
Mem:  526491648 162996224 363495424        0  8679424 29622272
Swap:        0        0        0
MemTotal:       514152 kB
MemFree:        354976 kB
MemShared:           0 kB
Buffers:          8476 kB
Cached:          28928 kB
SwapCached:          0 kB
Active:          19112 kB
Inactive:        18292 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       514152 kB
LowFree:        354976 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Mit Sep 26 09:15:02 CEST 2001
        total:    used:    free:  shared: buffers:  cached:
Mem:  526491648 18993152 507498496        0   409600  2408448
Swap:        0        0        0
MemTotal:       514152 kB
MemFree:        495604 kB
MemShared:           0 kB
Buffers:           400 kB
Cached:           2352 kB
SwapCached:          0 kB
Active:           1388 kB
Inactive:         1364 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       514152 kB
LowFree:        495604 kB
SwapTotal:           0 kB
SwapFree:            0 kB
Mit Sep 26 09:15:03 CEST 2001
        total:    used:    free:  shared: buffers:  cached:
Mem:  526491648 19054592 507437056        0   421888  2453504
Swap:        0        0        0
MemTotal:       514152 kB
MemFree:        495544 kB
MemShared:           0 kB
Buffers:           412 kB
Cached:           2396 kB
SwapCached:          0 kB
Active:           1496 kB
Inactive:         1312 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       514152 kB
LowFree:        495544 kB
SwapTotal:           0 kB
SwapFree:            0 kB
#22 lore raw
In reply to Jacek Popławski
Quoted 2 lines

I just installed 2.4.10, and...
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e

I saw the same message when running this c++ programm.

int main (int argc, char * argv[]) {
char * test;
while (1)
test=new char[1024];
}

My dmesg:

__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c01219e7
VM: killing process a.out

I have 512 MB RAM and no swap.
Actually the system slowed down a lot but worked fine again after the kill. And a flood ping from another PC had no lost packages.

#23 lore raw
In reply to Jacek Popławski
Quoted 12 lines · Christian Bornträger
Quoted 3 lines

I just installed 2.4.10, and...
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e

I saw the same message when running this c++ programm.

int main (int argc, char * argv[]) {
char * test;
while (1)
test=new char[1024];
}

Quoted 4 lines

My dmesg:

__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c01219e7
VM: killing process a.out

While this program is obviously "bad", it does show that something is not right. It should print "OOM: killing process a.out" as the kernel will have to deliberatly kill this "out of hand" program. the "VM: killing" message means it could just as easily have killed another program due to this DoS program...

#24 lore raw
In reply to Jacek Popławski
Quoted 20 lines · Marcelo Tosatti
Quoted 16 lines · On Mon, 24 Sep 2001, Jacek [iso-8859-2] Pop³awski wrote:

I just installed 2.4.10, and...

__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process donkey_s
__alloc_pages: 0-order allocation failed (gfp=0x1f0/0) from c0126c2e __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process screen
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process bash
(...)

I am changing kernels often, but never seen that kind of message. Last kernel I had before 2.4.10 was 2.4.10-pre4.

Quoted 6 lines
Quoted 2 lines

PS. donkey_s is application which eats a lot of memory, but I have 384MB RAM and 100MB swap.

Jacek,

You had available swap when the VM started to kill processes ?

I see that too with 2.4.10aa1 on a 4way 2gig ppc power3 box without swap:

mandarine:~ # w
bash: fork: Cannot allocate memory
mandarine:~ # w
bash: /usr/bin/w: Cannot allocate memory
mandarine:~ # w
5:50pm up 13 min, 3 users, load average: 6.27, 3.30, 1.67

USER     TTY      FROM              LOGIN@   IDLE   JCPU   PCPU  WHAT
root     ttyS0    -                 5:39pm  3.00s  3.19s  1.51s  w 
olaf     pts/0    nectarine.suse.d  5:39pm  9:45  43.90s  0.06s  sh

do_all
olh pts/1 nectarine.suse.d 5:48pm 7.00s 0.79s 0.79s -bash
mandarine:~ # free
bash: fork: Cannot allocate memory mandarine:~ # dmesg | tail
__alloc_pages: 0-order allocation failed (gfp=0x1f0/0) __alloc_pages: 0-order allocation failed (gfp=0x1f0/0) __alloc_pages: 0-order allocation failed (gfp=0x70/0) __alloc_pages: 0-order allocation failed (gfp=0x70/0) __alloc_pages: 0-order allocation failed (gfp=0x70/0) __alloc_pages: 0-order allocation failed (gfp=0x70/0) __alloc_pages: 0-order allocation failed (gfp=0x70/0) __alloc_pages: 0-order allocation failed (gfp=0x70/0) __alloc_pages: 0-order allocation failed (gfp=0x1f0/0)
VM: killing process cc1
mandarine:~ # free
total used free shared buffers
cached
Mem: 2057304 2052932 4372 0 53480
1792468
-/+ buffers/cache: 206984 1850320
Swap: 0 0 0
mandarine:~ # free
bash: fork: Cannot allocate memory mandarine:~ # vmstat
bash: fork: Cannot allocate memory mandarine:~ # vmstat
bash: fork: Cannot allocate memory mandarine:~ # vmstat
bash: fork: Cannot allocate memory mandarine:~ # vmstat
bash: fork: Cannot allocate memory mandarine:~ # vmstat
bash: fork: Cannot allocate memory mandarine:~ # vmstat
procs memory swap io system
cpu

 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 3  0  1      0   2744  53944 1794968   0   0   440   343   75   300  14
28  58

mandarine:~ # free
Killed

That did not happen with pre10aa1, at least the OOM kills. I happend with a bk pull, a build in the background. I seems that it doesnt release some memory...

Gruss Olaf

Signature · 4 lines
 $ man clone

BUGS
       Main feature not yet implemented...
#25 lore raw
In reply to Jacek Popławski
Quoted 7 lines · Olaf Hering

mandarine:~ # vmstat
procs memory swap io system
cpu

 r  b  w   swpd   free   buff  cache  si  so    bi    bo   in    cs  us
sy  id
 3  0  1      0   2744  53944 1794968   0   0   440   343   75   300  14
28  58
Quoted 7 lines

mandarine:~ # free
Killed

That did not happen with pre10aa1, at least the OOM kills. I happend with a bk pull, a build in the background. I seems that it doesnt release some memory...

it seems that the cache grows and grows, one bk process was still active. No idea who to blame, but it should not kill the box :)

27429 pts/0 D 3:59 bk idcache -q

Gruss Olaf

Signature · 4 lines
 $ man clone

BUGS
       Main feature not yet implemented...
#26 lore raw
In reply to Jacek Popławski
Quoted 3 lines · Marcelo Tosatti

Jacek,

You had available swap when the VM started to kill processes ?

Application eats whole memory, then started using swap, when swap used is 10MB kernel starting to cry:

[root@localhost /root]# free

             total       used       free     shared    buffers     cached
Mem:        320616     317348       3268          0        120     304096
-/+ buffers/cache:      13132     307484
Swap:       104380      10208      94172
[root@localhost /root]# free
             total       used       free     shared    buffers     cached
Mem:        320616     318932       1684          0        136     305372

-/+ buffers/cache: 13424 307192
Swap: 104380 10072 94308
[root@localhost /root]# __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e free

             total       used       free     shared    buffers     cached
Mem:        320616     318884       1732          0        128     305636

-/+ buffers/cache: 13120 307496
Swap: 104380 10204 94176
[root@localhost /root]# __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process donkey_s
free

             total       used       free     shared    buffers     cached
Mem:        320616     318732       1884          0        116     305480

-/+ buffers/cache: 13136 307480
Swap: 104380 9312 95068

#27 lore raw
In reply to Jacek Popławski
Quoted 6 lines · Jacek Popławski

VM: killing process donkey_s
free

             total       used       free     shared    buffers     cached
Mem:        320616     318732       1884          0        116     305480

-/+ buffers/cache: 13136 307480
Swap: 104380 9312 95068

and it's getting worse (donkey_s is already killed!):

             total       used       free     shared    buffers     cached
Mem:        320616     318828       1788          0        112     305596

-/+ buffers/cache: 13120 307496
Swap: 104380 10472 93908
[root@localhost /root]# __alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process sendmail
__alloc_pages: 0-order allocation failed (gfp=0x1d2/0) from c0126c2e
VM: killing process sendmail
(...)

PS. uptime=5h, no problems before I started "donkey_s"