Skip to contentSkip to conversation

out of memory

DISCUSSION Ingo Freund 2 participants 4 messages Apr 3, 2006 – Apr 4, 2006

lore
Share & tools
lore
#0 lore raw

Hi,

on our database machine with 4 GB RAM and 2 XEON CPUs I got the following kernel messages.
There are 2 GB RAM declared as shared memory for database usage. Can anybody explain to me what happened and -may be- why?

Apr  2 10:55:09 widbrz01 kernel: oom-killer: gfp_mask=0xd0, order=0
Apr  2 10:55:09 widbrz01 kernel: Mem-info:
Apr  2 10:55:09 widbrz01 kernel: DMA per-cpu:
Apr  2 10:55:09 widbrz01 kernel: cpu 0 hot: low 2, high 6, batch 1 used:2
Apr  2 10:55:09 widbrz01 kernel: cpu 0 cold: low 0, high 2, batch 1 used:1
Apr  2 10:55:09 widbrz01 kernel: cpu 1 hot: low 2, high 6, batch 1 used:2
Apr  2 10:55:09 widbrz01 kernel: cpu 1 cold: low 0, high 2, batch 1 used:1
Apr  2 10:55:09 widbrz01 kernel: cpu 2 hot: low 2, high 6, batch 1 used:2
Apr  2 10:55:09 widbrz01 kernel: cpu 2 cold: low 0, high 2, batch 1 used:1
Apr  2 10:55:09 widbrz01 kernel: cpu 3 hot: low 2, high 6, batch 1 used:2
Apr  2 10:55:09 widbrz01 kernel: cpu 3 cold: low 0, high 2, batch 1 used:1
Apr  2 10:55:09 widbrz01 kernel: Normal per-cpu:
Apr  2 10:55:09 widbrz01 kernel: cpu 0 hot: low 62, high 186, batch 31 used:174
Apr  2 10:56:10 widbrz01 kernel: cpu 0 cold: low 0, high 62, batch 31 used:52
Apr  2 10:56:10 widbrz01 kernel: cpu 1 hot: low 62, high 186, batch 31 used:86
Apr  2 10:56:10 dbm kernel: cpu 1 cold: low 0, high 62, batch 31 used:47
Apr  2 10:56:10 dbm kernel: cpu 2 hot: low 62, high 186, batch 31 used:107
Apr  2 10:56:10 dbm kernel: cpu 2 cold: low 0, high 62, batch 31 used:31
Apr  2 10:56:10 dbm kernel: cpu 3 hot: low 62, high 186, batch 31 used:71
Apr  2 10:56:10 dbm kernel: cpu 3 cold: low 0, high 62, batch 31 used:39
Apr  2 10:56:10 dbm kernel: HighMem per-cpu:
Apr  2 10:56:10 dbm kernel: cpu 0 hot: low 62, high 186, batch 31 used:124
Apr  2 10:56:10 dbm kernel: cpu 0 cold: low 0, high 62, batch 31 used:46
Apr  2 10:56:10 dbm kernel: cpu 1 hot: low 62, high 186, batch 31 used:93
Apr  2 10:56:10 dbm kernel: cpu 1 cold: low 0, high 62, batch 31 used:47
Apr  2 10:56:10 dbm kernel: cpu 2 hot: low 62, high 186, batch 31 used:160
Apr  2 10:56:10 dbm kernel: cpu 2 cold: low 0, high 62, batch 31 used:50
Apr  2 10:56:10 dbm kernel: cpu 3 hot: low 62, high 186, batch 31 used:87
Apr  2 10:56:10 dbm kernel: cpu 3 cold: low 0, high 62, batch 31 used:31
Apr  2 10:56:10 dbm kernel: Free pages:        7608kB (868kB HighMem)
Apr  2 10:56:10 dbm kernel: Active:401296 inactive:416067 dirty:76506 writeback:14 unstable:0 free:1902 slab:8550 mapped:399275
pagetables:20137
Apr  2 10:56:10 dbm kernel: DMA free:3572kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:4
all_unreclaimable? yes
Apr  2 10:56:10 dbm kernel: lowmem_reserve[]: 0 880 6128
Apr  2 10:56:10 dbm kernel: Normal free:3168kB min:3756kB low:4692kB high:5632kB active:700kB inactive:548kB present:901120kB
pages_scanned:1488 all_unreclaimable? yes
Apr  2 10:56:10 dbm kernel: lowmem_reserve[]: 0 0 41984
Apr  2 10:56:10 dbm kernel: HighMem free:868kB min:512kB low:640kB high:768kB active:1604484kB inactive:1663720kB present:5373952kB
pages_scanned:0 all_unreclaimable? no
Apr  2 10:56:10 dbm kernel: lowmem_reserve[]: 0 0 0
Apr  2 10:56:10 dbm kernel: DMA: 1*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3572kB
Apr  2 10:56:10 dbm kernel: Normal: 88*4kB 4*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3168kB
Apr  2 10:56:10 dbm kernel: HighMem: 5*4kB 4*8kB 3*16kB 2*32kB 3*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 868kB
Apr  2 10:56:10 dbm kernel: Swap cache: add 1, delete 1, find 0/0, race 0+0
Apr  2 10:56:10 dbm kernel: Free swap  = 4209004kB
Apr  2 10:56:10 dbm kernel: Total swap = 4209008kB
Apr  2 10:56:10 dbm kernel: Free swap:       4209004kB
Apr  2 10:56:10 dbm kernel: klogd 1.4.1, ---------- state change ----------
Apr  2 10:56:11 dbm kernel: Inspecting /boot/System.map-2.6.13.2
Apr  2 10:56:11 dbm kernel: Loaded 28714 symbols from /boot/System.map-2.6.13.2.
Apr  2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.
Apr  2 10:56:11 dbm kernel: No module symbols loaded - kernel modules not enabled.
Apr  2 10:56:11 dbm kernel: 1572864 pages of RAM
Apr  2 10:56:11 dbm kernel: 1343488 pages of HIGHMEM
Apr  2 10:56:11 dbm kernel: 538120 reserved pages
Apr  2 10:56:11 dbm kernel: 10551624 pages shared
Apr  2 10:56:11 dbm kernel: 0 pages swap cached
Apr  2 10:56:11 dbm kernel: 63038 pages dirty
Apr  2 10:56:11 dbm kernel: 14 pages writeback
Apr  2 10:56:11 dbm kernel: 399275 pages mapped
Apr  2 10:56:11 dbm kernel: 8538 pages slab
Apr  2 10:56:11 dbm kernel: 20137 pages pagetables
Apr  2 10:56:11 dbm kernel: Out of Memory: Killed process 18250 (db:SBWI).

Thanks,
Ingo.

#1 lore raw
Quoted 8 lines · Ingo Freund

Hi,

on our database machine with 4 GB RAM and 2 XEON CPUs I got the following kernel messages.
There are 2 GB RAM declared as shared memory for database usage. Can anybody explain to me what happened and -may be- why?

You have a kernel memory leak.

Quoted 2 lines
Apr  2 10:55:09 widbrz01 kernel: oom-killer: gfp_mask=0xd0, order=0

It was a GFP_KERNEL allocation.

Quoted 1 line
Apr  2 10:56:10 dbm kernel: Normal free:3168kB min:3756kB low:4692kB high:5632kB active:700kB inactive:548kB present:901120kB

There's a grand total of 1.2MB of ZONE_NORMAL memory on the LRU. The rest (900MB-odd) is lost.

Quoted 1 line
Apr  2 10:56:11 dbm kernel: 8538 pages slab

and it's not in slab.

Quoted 1 line
Apr  2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.

Boy, 2.6.13 was a long time ago - I'm sure we fixed many leaks since then, but I do not recall any particular patch which might fix this, sorry.

Your best option would be to seek a kernel upgrade.

#2 lore raw
Quoted 11 lines · Andrew Morton
Quoted 1 line · Ingo Freund
Apr  2 10:56:11 dbm kernel: 8538 pages slab

and it's not in slab.

Quoted 1 line
Apr  2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.

Boy, 2.6.13 was a long time ago - I'm sure we fixed many leaks since then, but I do not recall any particular patch which might fix this, sorry.

Your best option would be to seek a kernel upgrade.

I'll give a try to the last kernel version

Is there a way to get those kernel/memory information from time to time from the running system which I found in the syslog file?

Thank you
Ingo.

#3 lore raw
Quoted 1 line · Ingo Freund

Please don't edit Cc:'s when working with kernel developers - just do reply-to-all.

Quoted 6 lines
Quoted 5 lines · Andrew Morton
Quoted 1 line · Ingo Freund
Apr  2 10:56:11 dbm kernel: 8538 pages slab

and it's not in slab.

Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Apr  2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.

Boy, 2.6.13 was a long time ago - I'm sure we fixed many leaks since then, but I do not recall any particular patch which might fix this, sorry.

Your best option would be to seek a kernel upgrade.

I'll give a try to the last kernel version

OK, 2.6.16.1 would suit.

Quoted 2 lines

Is there a way to get those kernel/memory information from time to time from the running system which I found in the syslog file?

/proc/meminfo gives a decent summary.

To watch your ZONE_NORMAL disappearing it would be better to monitor /proc/zoneinfo. There you'll see free', active', inactive' and present'. In /proc/meminfo you'll find the slab utilisation.

It should be approximately true that free+active+inactive+slab = present.