Memory exhaustion testing?

DISCUSSION Jesper Dangaard Brouer 4 participants 10 messages Nov 12, 2015 – Nov 20, 2015 Memory management

lore

Share & tools

lore

Jesper Dangaard Brouer

author

2015-11-12 20:55 UTC #0 lore raw

Hi MM-people,

How do you/we test the error paths when the system runs out of memory?

What kind of tools do you use?
or Any tricks to provoke this?

For testing my recent change to the SLUB allocator, I've implemented a crude kernel module that tries to allocate all memory, so I can test the error code-path in kmem_cache_alloc_bulk.

see:
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/slab_bulk_test04_exhaust_mem.c

Signature · 11 lines

Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

David Rientjes

2015-11-13 22:54 UTC #1 lore raw

Quoted 7 lines · Jesper Dangaard Brouer

Hi MM-people,
How do you/we test the error paths when the system runs out of memory?
What kind of tools do you use?
or Any tricks to provoke this?

Depends on the paths that you want to exercise when the system is out of memory :) If it's just to trigger the oom killer, then no kernel module should be necessary if you're not limited by any cgroup and just disable swap and start off an anonymous memory hog that consumes all memory.

Quoted 4 lines

For testing my recent change to the SLUB allocator, I've implemented a crude kernel module that tries to allocate all memory, so I can test the error code-path in kmem_cache_alloc_bulk.

Trying to exercise certain paths under oom is difficult because the oom killer will usually quickly kill a process or you'll get hung up somewhere else that needs memory before the function you want to test. This is why failslab had been used in the past, and does a good job at runtime testing. My suggestion would just be to instrument the kernel to randomly fail as though the system is oom and ensure that it works.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Jesper Dangaard Brouer

author

2015-11-16 14:24 UTC #2 lore raw

Quoted 3 lines · On Fri, 13 Nov 2015 14:54:37 -0800 (PST) David Rientjes <[email protected]>…

[...] This is why
failslab had been used in the past, and does a good job at runtime testing.

Thanks for mentioning CONFIG_FAILSLAB. First I disregarded "failslab" (I did notice it in the slub code) because it didn't exercised the code path I wanted in kmem_cache_alloc_bulk().

But went to looking up the config setting I notice that we do have a hole section for "Fault-injection". Which is great, and what I was looking for.

Menu config Location:
-> Kernel hacking
-> Fault-injection framework (FAULT_INJECTION [=y])

I think what I need can be covered by FAIL_PAGE_ALLOC, or should_fail_alloc_page(). I'll try and play a bit with it...

- -
Best regards,
Jesper Dangaard Brouer
MSc.CS, Principal Kernel Engineer at Red Hat
Author of http://www.iptv-analyzer.org
LinkedIn: http://www.linkedin.com/in/brouer

[*] Fault-injection framework

[*]   Fault-injection capability for kmalloc
[*]   Fault-injection capabilitiy for alloc_pages()
[ ]   Fault-injection capability for disk IO
[ ]   Fault-injection capability for faking disk interrupts
[ ]   Fault-injection capability for futexes
[*]   Debugfs entries for fault-injection capabilities

Jesper Dangaard Brouer

author

2015-11-17 13:21 UTC #3 lore raw

Quoted 14 lines · On Mon, 16 Nov 2015 15:24:40 +0100 Jesper Dangaard Brouer <[email protected]>…

Quoted 3 lines · On Fri, 13 Nov 2015 14:54:37 -0800 (PST) David Rientjes <[email protected]>…
[...] This is why
failslab had been used in the past, and does a good job at runtime testing.
Thanks for mentioning CONFIG_FAILSLAB. First I disregarded "failslab" (I did notice it in the slub code) because it didn't exercised the code path I wanted in kmem_cache_alloc_bulk().
But went to looking up the config setting I notice that we do have a hole section for "Fault-injection". Which is great, and what I was looking for.

Quoted 6 lines

Menu config Location:
-> Kernel hacking
-> Fault-injection framework (FAULT_INJECTION [=y])
I think what I need can be covered by FAIL_PAGE_ALLOC, or should_fail_alloc_page(). I'll try and play a bit with it...

I did manage to provoke/test the error path in kmem_cache_alloc_bulk(), by using fault-injection framework "fail_page_alloc".

But was a little hard to trigger SLUB errors with this, because SLUB retries after a failure, and second call to alloc_pages() is done with lower order.

If order is lowered to zero, then should_fail_alloc_page() will skip it. And just lowering /sys/kernel/debug/fail_page_alloc/min-order=0 is not feasible as even fork starts to fail. I managed to work-around this by using "space" setting.

Created a script to ease this tricky invocation:
https://github.com/netoptimizer/prototype-kernel/blob/master/tests/fault-inject/fail01_kmem_cache_alloc_bulk.sh

Signature · 11 lines

Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

David Rientjes

2015-11-19 20:40 UTC #4 lore raw

Quoted 7 lines · Jesper Dangaard Brouer

I did manage to provoke/test the error path in kmem_cache_alloc_bulk(), by using fault-injection framework "fail_page_alloc".
But was a little hard to trigger SLUB errors with this, because SLUB retries after a failure, and second call to alloc_pages() is done with lower order.

Quoted 8 lines

If order is lowered to zero, then should_fail_alloc_page() will skip it. And just lowering /sys/kernel/debug/fail_page_alloc/min-order=0 is not feasible as even fork starts to fail. I managed to work-around this by using "space" setting.
Created a script to ease this tricky invocation:
https://github.com/netoptimizer/prototype-kernel/blob/master/tests/fault-inject/fail01_kmem_cache_alloc_bulk.sh

Any chance you could proffer some of your scripts in the form of patches to the tools/testing directory? Anything that can reliably trigger rarely executed code is always useful.

Jesper Dangaard Brouer

author

2015-11-20 13:09 UTC #5 lore raw

Quoted 14 lines · On Thu, 19 Nov 2015 12:40:50 -0800 (PST) David Rientjes <[email protected]>…

Quoted 12 lines · Jesper Dangaard Brouer
I did manage to provoke/test the error path in kmem_cache_alloc_bulk(), by using fault-injection framework "fail_page_alloc".
But was a little hard to trigger SLUB errors with this, because SLUB retries after a failure, and second call to alloc_pages() is done with lower order.
If order is lowered to zero, then should_fail_alloc_page() will skip it. And just lowering /sys/kernel/debug/fail_page_alloc/min-order=0 is not feasible as even fork starts to fail. I managed to work-around this by using "space" setting.

Quoted 7 lines

Quoted 3 lines
Created a script to ease this tricky invocation:
https://github.com/netoptimizer/prototype-kernel/blob/master/tests/fault-inject/fail01_kmem_cache_alloc_bulk.sh
Any chance you could proffer some of your scripts in the form of patches to the tools/testing directory? Anything that can reliably trigger rarely executed code is always useful.

Perhaps that is a good idea.

I think should move the directory location in my git-repo prototype-kernel[1] to reflect this directory layout, like I do with real kernel stuff. And when we are happy with the quality of the scripts we can "move" it to the kernel. (Like I did with my pktgen tests[4], now located in samples/pktgen/).

A question; where should/could we place the kernel module slab_bulk_test04_exhaust_mem[1] that my fail01 script depends on?

BTW, I've also added a script for testing NULL handling in normal kmem_cache_alloc() call see[3].

Signature · 19 lines

Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

[1] https://github.com/netoptimizer/prototype-kernel/

[2] https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/slab_bulk_test04_exhaust_mem.c

[3] https://github.com/netoptimizer/prototype-kernel/blob/master/tests/fault-inject/fail02_kmem_cache_alloc.sh

[4] https://github.com/netoptimizer/network-testing/tree/master/pktgen

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

David Rientjes

2015-11-20 23:23 UTC #6 lore raw

Quoted 6 lines · Jesper Dangaard Brouer

Quoted 3 lines
Any chance you could proffer some of your scripts in the form of patches to the tools/testing directory? Anything that can reliably trigger rarely executed code is always useful.
Perhaps that is a good idea.

Quoted 9 lines

I think should move the directory location in my git-repo prototype-kernel[1] to reflect this directory layout, like I do with real kernel stuff. And when we are happy with the quality of the scripts we can "move" it to the kernel. (Like I did with my pktgen tests[4], now located in samples/pktgen/).
A question; where should/could we place the kernel module slab_bulk_test04_exhaust_mem[1] that my fail01 script depends on?

I've had the same question because I'd like to add slab and page allocator benchmark modules originally developed by Christoph Lameter to the tree. Let's add Andrew.

Andrew Morton

maintainer

2015-11-20 23:28 UTC #7 lore raw

Quoted 14 lines · On Fri, 20 Nov 2015 15:23:09 -0800 (PST) David Rientjes <[email protected]>…

Quoted 12 lines · Jesper Dangaard Brouer
Quoted 3 lines
Any chance you could proffer some of your scripts in the form of patches to the tools/testing directory? Anything that can reliably trigger rarely executed code is always useful.
Perhaps that is a good idea.
I think should move the directory location in my git-repo prototype-kernel[1] to reflect this directory layout, like I do with real kernel stuff. And when we are happy with the quality of the scripts we can "move" it to the kernel. (Like I did with my pktgen tests[4], now located in samples/pktgen/).

Quoted 7 lines

Quoted 3 lines
A question; where should/could we place the kernel module slab_bulk_test04_exhaust_mem[1] that my fail01 script depends on?
I've had the same question because I'd like to add slab and page allocator benchmark modules originally developed by Christoph Lameter to the tree. Let's add Andrew.

Well, fwiw the current approach is to build the testing module in lib/ (ls -l lib/*test*.c) and to modprobe it from selftests (grep -r modprobe tools/testing/selftests).

Does that suit? I guess we could build and install a module from mm/ if that's needed for some reason.

Tetsuo Handa

2015-11-14 08:23 UTC #8 lore raw

In reply to Jesper Dangaard Brouer

Quoted 6 lines · Jesper Dangaard Brouer

Hi MM-people,
How do you/we test the error paths when the system runs out of memory?
What kind of tools do you use?
or Any tricks to provoke this?

I use SystemTap for injecting memory allocation failure.

http://lkml.kernel.org/r/[email protected]

Quoted 8 lines

For testing my recent change to the SLUB allocator, I've implemented a crude kernel module that tries to allocate all memory, so I can test the error code-path in kmem_cache_alloc_bulk.
see:
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/slab_bulk_test04_exhaust_mem.c

I think you can test the error code-path in kmem_cache_alloc_bulk as well.

Jesper Dangaard Brouer

author

2015-11-16 14:43 UTC #9 lore raw

Quoted 17 lines · On Sat, 14 Nov 2015 17:23:15 +0900 Tetsuo Handa…

Quoted 6 lines · Jesper Dangaard Brouer
Hi MM-people,
How do you/we test the error paths when the system runs out of memory?
What kind of tools do you use?
or Any tricks to provoke this?
I use SystemTap for injecting memory allocation failure.
http://lkml.kernel.org/r/[email protected]
Quoted 5 lines
For testing my recent change to the SLUB allocator, I've implemented a crude kernel module that tries to allocate all memory, so I can test the error code-path in kmem_cache_alloc_bulk.

Quoted 6 lines

Quoted 3 lines
see:
https://github.com/netoptimizer/prototype-kernel/blob/master/kernel/mm/slab_bulk_test04_exhaust_mem.c
I think you can test the error code-path in kmem_cache_alloc_bulk as well.

Yes, making __alloc_pages_nodemask() fail should propagate all the way back into kmem_cache_alloc_bulk().

I do like your approach, but I think my use-case can be covered by CONFIG_FAIL_PAGE_ALLOC (which like you, also hook into __alloc_pages_nodemask). Although it seems I have more control with your approach, to filter in which situations it should happen in.

Thanks for your input! :-)

Signature · 11 lines

Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>