4.2rc1 odd looking page allocator failure stats

DISCUSSION Dave Jones 2 participants 2 messages Jul 8, 2015 – Jul 9, 2015 Memory management

Share & tools

Dave Jones

author

2015-07-08 20:43 UTC #0 lore raw

I've got a box with 4GB of RAM that I've driven into oom (so much so that e1000 can't alloc a single page, so I can't even ping it). But over serial console I noticed this..

[158831.710001] DMA32 free:1624kB min:6880kB low:8600kB high:10320kB active_anon:407004kB inactive_anon:799300kB active_file:516kB inactive_file:6644kB unevictable:0kB
 isolated(anon):0kB isolated(file):0kB present:3127220kB managed:3043108kB mlocked:0kB dirty:6680kB writeback:64kB mapped:31544kB shmem:1146792kB
 slab_reclaimable:46812kB slab_unreclaimable:388364kB kernel_stack:2288kB pagetables:2076kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB
 free_cma:0kB writeback_tmp:0kB pages_scanned:70152496980 all_unreclaimable? yes

How come that 'pages_scanned' number is greater than the number of pages in the system ? Does kswapd iterate over the same pages a number of times each time the page allocator fails ?

I've managed to hit this a couple times this week, where the oom killer kicks in, kills some processes, but then the machine goes into a death spiral of looping in the page allocator. Once that begins, it never tries to oom kill again, just hours of page allocation failure messages.

Dave

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

David Rientjes

2015-07-09 00:42 UTC #1 lore raw

Quoted 8 lines · Dave Jones

I've got a box with 4GB of RAM that I've driven into oom (so much so that e1000 can't alloc a single page, so I can't even ping it). But over serial console I noticed this..

[158831.710001] DMA32 free:1624kB min:6880kB low:8600kB high:10320kB active_anon:407004kB inactive_anon:799300kB active_file:516kB inactive_file:6644kB unevictable:0kB
 isolated(anon):0kB isolated(file):0kB present:3127220kB managed:3043108kB mlocked:0kB dirty:6680kB writeback:64kB mapped:31544kB shmem:1146792kB
 slab_reclaimable:46812kB slab_unreclaimable:388364kB kernel_stack:2288kB pagetables:2076kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB
 free_cma:0kB writeback_tmp:0kB pages_scanned:70152496980 all_unreclaimable? yes

Quoted 8 lines

How come that 'pages_scanned' number is greater than the number of pages in the system ? Does kswapd iterate over the same pages a number of times each time the page allocator fails ?
I've managed to hit this a couple times this week, where the oom killer kicks in, kills some processes, but then the machine goes into a death spiral of looping in the page allocator. Once that begins, it never tries to oom kill again, just hours of page allocation failure messages.

We don't have the full oom log to see if there's any indication of a problem, but pages_scanned should be able to grow very large since it's never reset as a result of either memory freeing or periodic pcp flush (I notice free_pcp is 0kB above) so pages_scanned never gets cleared.