Memory Problem in 2.4.9 ?

Hello all,

can anybody enlighten me about the following kernel-message:

Aug 21 14:37:39 admin kernel: __alloc_pages: 3-order allocation failed.
Aug 21 14:37:39 admin kernel: __alloc_pages: 2-order allocation failed. 
Aug 21 14:37:39 admin kernel: __alloc_pages: 3-order allocation failed.
Aug 21 14:37:39 admin kernel: __alloc_pages: 3-order allocation failed.
Aug 21 14:37:39 admin kernel: __alloc_pages: 2-order allocation failed.
Aug 21 14:37:39 admin kernel: __alloc_pages: 3-order allocation failed.
Aug 21 14:37:39 admin last message repeated 69 times

I get tons of them during verifying burned CDs with xcdroast (which takes a really long time, and must have some problem therefore). I would not worry you if it didn't work with earlier kernel-versions, but in fact it did.

Hardware: (Asus CUV4X-D, yes it works :-)
00:00.0 Host bridge: VIA Technologies, Inc. VT82C693A/694x [Apollo PRO133x] (rev c4) 00:01.0 PCI bridge: VIA Technologies, Inc. VT82C598/694x [Apollo MVP3/Pro133x AGP] 00:04.0 ISA bridge: VIA Technologies, Inc. VT82C686 [Apollo Super South] (rev 40) 00:04.1 IDE interface: VIA Technologies, Inc. Bus Master IDE (rev 06) 00:04.2 USB Controller: VIA Technologies, Inc. UHCI USB (rev 16) 00:04.3 USB Controller: VIA Technologies, Inc. UHCI USB (rev 16) 00:04.4 Host bridge: VIA Technologies, Inc. VT82C686 [Apollo Super ACPI] (rev 40) 00:09.0 PCI bridge: Digital Equipment Corporation DECchip 21152 (rev 03) 00:0a.0 Network controller: Elsa AG QuickStep 1000 (rev 01)
00:0b.0 SCSI storage controller: Adaptec 7892A (rev 02)
00:0d.0 Multimedia audio controller: Creative Labs SB Live! EMU10000 (rev 07) 00:0d.1 Input device controller: Creative Labs SB Live! (rev 07) 01:00.0 VGA compatible controller: nVidia Corporation NV11 (rev b2) 02:04.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41) 02:05.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41) 02:06.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41) 02:07.0 Ethernet controller: Digital Equipment Corporation DECchip 21142/43 (rev 41)

cpu:
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 8
model name : Pentium III (Coppermine)
stepping : 10
cpu MHz : 1004.525
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 mmx fxsr sse
bogomips : 2005.40

processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 8
model name : Pentium III (Coppermine)
stepping : 10
cpu MHz : 1004.525
cache size : 256 KB
fdiv_bug : no
hlt_bug : no
f00f_bug : no
coma_bug : no
fpu : yes
fpu_exception : yes
cpuid level : 2
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov
pat pse36 mmx fxsr sse
bogomips : 2005.40

MEM:
total: used: free: shared: buffers: cached:

Mem:  1053675520 1047502848  6172672        0 20930560 939307008
Swap: 271392768 15880192 255512576
MemTotal:      1028980 kB
MemFree:          6028 kB
MemShared:           0 kB
Buffers:         20440 kB
Cached:         911860 kB
SwapCached:       5432 kB
Active:         571980 kB
Inact_dirty:    362480 kB
Inact_clean:      3272 kB
Inact_target:     2336 kB
HighTotal:      131056 kB
HighFree:         2036 kB
LowTotal:       897924 kB
LowFree:          3992 kB
SwapTotal:      265032 kB
SwapFree:       249524 kB

The following patch will tell us more about the failure, could you please apply (patch -p0):

@@ -502,7 +502,7 @@
 	}
 
 	/* No luck.. */
-	printk(KERN_ERR "__alloc_pages: %lu-order allocation failed.\n", order);
+	printk(KERN_ERR "__alloc_pages: %lu-order allocation failed (gfp=0x%x/%i).\n", order, gfp_mask, !!(current->flags & PF_MEMALLOC));
 	return NULL;
 }
 

(gfp=0x20/0).

It's an atomic allocation, the driver is supposed to be able to handle this, and it does since you report that the burn just runs slowly, it does not stop. There is way too much in cache:

This is causing the high order allocation failures - with only a small fraction of memory free the chances of none of it being in contiguous, aligned 8 page units rises dramatically. It's likely the kprint that is slowing you down, you could check this by commenting it out (page_alloc.c, near the end of __alloc_pages).

Do you have the same problem on 2.4.8, but not in 2.4.7?

--
Daniel

I basically thought the same. In fact I do not understand why. Are there any parameters tunable to balance the whole picture a bit more towards the free pages?

I guess you mean the formerly patched debug-output, do you? I commented it out and saw a way better result than before. In fact I did not manage to break the NFS-copy at all, and although I managed to get the cpu load up to about 5 everything worked smoother. Only now and then were some moments where the display freezes "a bit", but mouse movement continues to work. Anyway I am not sure, if it is intended that my browser gets swapped out only by copying files via NFS which are alltogether smaller than my physical 1 GB of RAM. I do think that there is still a little too much caching going on.

I am going to check that tomorrow. Downgrading is a bit tricky on this system.

Thanks Daniel,
I'll be back
:-)

I tested the situation with 2.4.7 (straight, no patches) and it looks like this:

meminfo before:

total: used: free: shared: buffers: cached:

Mem:  921735168 88141824 833593344        0  6643712 36012032
Swap: 271392768        0 271392768
MemTotal:       900132 kB
MemFree:        814056 kB
MemShared:           0 kB
Buffers:          6488 kB
Cached:          35168 kB
SwapCached:          0 kB
Active:          34920 kB
Inact_dirty:      6736 kB
Inact_clean:         0 kB
Inact_target:      864 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       900132 kB
LowFree:        814056 kB
SwapTotal:      265032 kB
SwapFree:       265032 kB

meminfo after test:

total: used: free: shared: buffers: cached:

Mem:  921735168 917393408  4341760        0 221192192 567046144
Swap: 271392768        0 271392768
MemTotal:       900132 kB
MemFree:          4240 kB
MemShared:           0 kB
Buffers:        216008 kB
Cached:         553756 kB
SwapCached:          0 kB
Active:         107912 kB
Inact_dirty:    658432 kB
Inact_clean:      3420 kB
Inact_target:    11360 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       900132 kB
LowFree:          4240 kB
SwapTotal:      265032 kB
SwapFree:       265032 kB

I can see these:

Aug 22 13:34:53 admin kernel: __alloc_pages: 2-order allocation failed.
Aug 22 13:34:53 admin kernel: __alloc_pages: 3-order allocation failed.
Aug 22 13:34:53 admin last message repeated 21 times

_BUT_ I cannot see any errors during NFS-filecopy. I tried hard, but no errors. Another thing is the CPU load. It is definitely lower than with 2.4.9 around 3 - 3.5, but never 4 or above.
Swap is not used, although turned on.
Besides the above kernel-messages I would say that 2.4.7 performs a lot better (and more stable) than 2.4.9 in this test case.
I think a deep look should be taken into this topic, because it makes 2.4.9 pretty unusable for server-environment. I wonder if anybody can produce (low-memory) errors during normal file-operation on localhost (not NFS like me). I would expect that, for it doesn't look NFS specific. I can make the kernel shoot loads of error messages only by reading CDs while copying in the background from the net. The effect can be seen vice versa, too. So you could say it is clearly a memory management problem, and not related to the allocating process.

Should I try with a plain 2.4.8 ?

Regards, Stephan

I'd like to try to isolate the cause a little more. Can you please try the following patch and see if it improves the cache balance. (This would not be a good solution, it will just help show what is happening.)

@@ -980,7 +980,7 @@
 static inline void check_used_once (struct page *page)
 {
 	if (!PageActive(page)) {
-		if (page->age)
+		if (page->age > 8)
 			activate_page(page);
 		else {
 			page->age = PAGE_AGE_START;

Oops, yes, I forgot for the moment that we no longer age up in __find_page_nolock. Lets try this instead, which should capture the intended effect of requiring 4 hits to activate a page (n.b., it's just a test):

@@ -980,10 +980,9 @@
 static inline void check_used_once (struct page *page)
 {
 	if (!PageActive(page)) {
-		if (page->age)
+		if (++page->age >= 4)
 			activate_page(page);
 		else {
-			page->age = PAGE_AGE_START;
 			ClearPageReferenced(page);
 		}
 	}