out of memory
Hi,
on our database machine with 4 GB RAM and 2 XEON CPUs I got the following kernel messages.
There are 2 GB RAM declared as shared memory for database usage. Can anybody explain to me what happened and -may be- why?
Apr 2 10:55:09 widbrz01 kernel: oom-killer: gfp_mask=0xd0, order=0 Apr 2 10:55:09 widbrz01 kernel: Mem-info: Apr 2 10:55:09 widbrz01 kernel: DMA per-cpu: Apr 2 10:55:09 widbrz01 kernel: cpu 0 hot: low 2, high 6, batch 1 used:2 Apr 2 10:55:09 widbrz01 kernel: cpu 0 cold: low 0, high 2, batch 1 used:1 Apr 2 10:55:09 widbrz01 kernel: cpu 1 hot: low 2, high 6, batch 1 used:2 Apr 2 10:55:09 widbrz01 kernel: cpu 1 cold: low 0, high 2, batch 1 used:1 Apr 2 10:55:09 widbrz01 kernel: cpu 2 hot: low 2, high 6, batch 1 used:2 Apr 2 10:55:09 widbrz01 kernel: cpu 2 cold: low 0, high 2, batch 1 used:1 Apr 2 10:55:09 widbrz01 kernel: cpu 3 hot: low 2, high 6, batch 1 used:2 Apr 2 10:55:09 widbrz01 kernel: cpu 3 cold: low 0, high 2, batch 1 used:1 Apr 2 10:55:09 widbrz01 kernel: Normal per-cpu: Apr 2 10:55:09 widbrz01 kernel: cpu 0 hot: low 62, high 186, batch 31 used:174 Apr 2 10:56:10 widbrz01 kernel: cpu 0 cold: low 0, high 62, batch 31 used:52 Apr 2 10:56:10 widbrz01 kernel: cpu 1 hot: low 62, high 186, batch 31 used:86 Apr 2 10:56:10 dbm kernel: cpu 1 cold: low 0, high 62, batch 31 used:47 Apr 2 10:56:10 dbm kernel: cpu 2 hot: low 62, high 186, batch 31 used:107 Apr 2 10:56:10 dbm kernel: cpu 2 cold: low 0, high 62, batch 31 used:31 Apr 2 10:56:10 dbm kernel: cpu 3 hot: low 62, high 186, batch 31 used:71 Apr 2 10:56:10 dbm kernel: cpu 3 cold: low 0, high 62, batch 31 used:39 Apr 2 10:56:10 dbm kernel: HighMem per-cpu: Apr 2 10:56:10 dbm kernel: cpu 0 hot: low 62, high 186, batch 31 used:124 Apr 2 10:56:10 dbm kernel: cpu 0 cold: low 0, high 62, batch 31 used:46 Apr 2 10:56:10 dbm kernel: cpu 1 hot: low 62, high 186, batch 31 used:93 Apr 2 10:56:10 dbm kernel: cpu 1 cold: low 0, high 62, batch 31 used:47 Apr 2 10:56:10 dbm kernel: cpu 2 hot: low 62, high 186, batch 31 used:160 Apr 2 10:56:10 dbm kernel: cpu 2 cold: low 0, high 62, batch 31 used:50 Apr 2 10:56:10 dbm kernel: cpu 3 hot: low 62, high 186, batch 31 used:87 Apr 2 10:56:10 dbm kernel: cpu 3 cold: low 0, high 62, batch 31 used:31 Apr 2 10:56:10 dbm kernel: Free pages: 7608kB (868kB HighMem) Apr 2 10:56:10 dbm kernel: Active:401296 inactive:416067 dirty:76506 writeback:14 unstable:0 free:1902 slab:8550 mapped:399275 pagetables:20137 Apr 2 10:56:10 dbm kernel: DMA free:3572kB min:68kB low:84kB high:100kB active:0kB inactive:0kB present:16384kB pages_scanned:4 all_unreclaimable? yes Apr 2 10:56:10 dbm kernel: lowmem_reserve[]: 0 880 6128 Apr 2 10:56:10 dbm kernel: Normal free:3168kB min:3756kB low:4692kB high:5632kB active:700kB inactive:548kB present:901120kB pages_scanned:1488 all_unreclaimable? yes Apr 2 10:56:10 dbm kernel: lowmem_reserve[]: 0 0 41984 Apr 2 10:56:10 dbm kernel: HighMem free:868kB min:512kB low:640kB high:768kB active:1604484kB inactive:1663720kB present:5373952kB pages_scanned:0 all_unreclaimable? no Apr 2 10:56:10 dbm kernel: lowmem_reserve[]: 0 0 0 Apr 2 10:56:10 dbm kernel: DMA: 1*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 0*512kB 1*1024kB 1*2048kB 0*4096kB = 3572kB Apr 2 10:56:10 dbm kernel: Normal: 88*4kB 4*8kB 0*16kB 1*32kB 1*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 3168kB Apr 2 10:56:10 dbm kernel: HighMem: 5*4kB 4*8kB 3*16kB 2*32kB 3*64kB 2*128kB 1*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 868kB Apr 2 10:56:10 dbm kernel: Swap cache: add 1, delete 1, find 0/0, race 0+0 Apr 2 10:56:10 dbm kernel: Free swap = 4209004kB Apr 2 10:56:10 dbm kernel: Total swap = 4209008kB Apr 2 10:56:10 dbm kernel: Free swap: 4209004kB Apr 2 10:56:10 dbm kernel: klogd 1.4.1, ---------- state change ---------- Apr 2 10:56:11 dbm kernel: Inspecting /boot/System.map-2.6.13.2 Apr 2 10:56:11 dbm kernel: Loaded 28714 symbols from /boot/System.map-2.6.13.2. Apr 2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13. Apr 2 10:56:11 dbm kernel: No module symbols loaded - kernel modules not enabled. Apr 2 10:56:11 dbm kernel: 1572864 pages of RAM Apr 2 10:56:11 dbm kernel: 1343488 pages of HIGHMEM Apr 2 10:56:11 dbm kernel: 538120 reserved pages Apr 2 10:56:11 dbm kernel: 10551624 pages shared Apr 2 10:56:11 dbm kernel: 0 pages swap cached Apr 2 10:56:11 dbm kernel: 63038 pages dirty Apr 2 10:56:11 dbm kernel: 14 pages writeback Apr 2 10:56:11 dbm kernel: 399275 pages mapped Apr 2 10:56:11 dbm kernel: 8538 pages slab Apr 2 10:56:11 dbm kernel: 20137 pages pagetables Apr 2 10:56:11 dbm kernel: Out of Memory: Killed process 18250 (db:SBWI).
Thanks,
Ingo.
Quoted 8 lines · Ingo Freund
Hi,
on our database machine with 4 GB RAM and 2 XEON CPUs I got the following kernel messages.
There are 2 GB RAM declared as shared memory for database usage. Can anybody explain to me what happened and -may be- why?
You have a kernel memory leak.
Quoted 2 lines
Apr 2 10:55:09 widbrz01 kernel: oom-killer: gfp_mask=0xd0, order=0
It was a GFP_KERNEL allocation.
Quoted 1 line
Apr 2 10:56:10 dbm kernel: Normal free:3168kB min:3756kB low:4692kB high:5632kB active:700kB inactive:548kB present:901120kB
There's a grand total of 1.2MB of ZONE_NORMAL memory on the LRU. The rest (900MB-odd) is lost.
Quoted 1 line
Apr 2 10:56:11 dbm kernel: 8538 pages slab
and it's not in slab.
Quoted 1 line
Apr 2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.
Boy, 2.6.13 was a long time ago - I'm sure we fixed many leaks since then, but I do not recall any particular patch which might fix this, sorry.
Your best option would be to seek a kernel upgrade.
Quoted 11 lines · Andrew Morton
Quoted 1 line · Ingo Freund
Apr 2 10:56:11 dbm kernel: 8538 pages slaband it's not in slab.
Quoted 1 line
Apr 2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.Boy, 2.6.13 was a long time ago - I'm sure we fixed many leaks since then, but I do not recall any particular patch which might fix this, sorry.
Your best option would be to seek a kernel upgrade.
I'll give a try to the last kernel version
Is there a way to get those kernel/memory information from time to time from the running system which I found in the syslog file?
Thank you
Ingo.
Quoted 1 line · Ingo Freund
Please don't edit Cc:'s when working with kernel developers - just do reply-to-all.
Quoted 6 lines
Quoted 5 lines · Andrew Morton
Quoted 1 line · Ingo Freund
Apr 2 10:56:11 dbm kernel: 8538 pages slaband it's not in slab.
Quoted 8 lines
Quoted 6 lines
Quoted 1 line
Apr 2 10:56:11 dbm kernel: Symbols match kernel version 2.6.13.Boy, 2.6.13 was a long time ago - I'm sure we fixed many leaks since then, but I do not recall any particular patch which might fix this, sorry.
Your best option would be to seek a kernel upgrade.
I'll give a try to the last kernel version
OK, 2.6.16.1 would suit.
Quoted 2 lines
Is there a way to get those kernel/memory information from time to time from the running system which I found in the syslog file?
/proc/meminfo gives a decent summary.
To watch your ZONE_NORMAL disappearing it would be better to monitor /proc/zoneinfo. There you'll see free', active', inactive' and present'. In /proc/meminfo you'll find the slab utilisation.
It should be approximately true that free+active+inactive+slab = present.