mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic
Subsystems (2)
PATCH 1/1 mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic
When I've used slub_debug kernel option (e.g. "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've seen a panic like:
Highbank #setenv bootargs console=ttyAMA0 root=/dev/sda2 kgdboc.kgdboc=ttyAMA0,115200 slub_debug=,kmalloc-4096 earlyprintk=ttyAMA0
...
Unable to handle kernel NULL pointer dereference at virtual address 00000000 pgd = c0004000 [00000000] *pgd=00000000 Internal error: Oops: 5 [#1] SMP ARM Modules linked in: CPU: 0 PID: 0 Comm: swapper Tainted: G W 3.12.0-00048-gbe408cd #314 task: c0898360 ti: c088a000 task.ti: c088a000 PC is at strncmp+0x1c/0x84 LR is at kmem_cache_flags.isra.46.part.47+0x44/0x60 pc : [<c02c6da0>] lr : [<c0110a3c>] psr: 200001d3 sp : c088bea8 ip : c088beb8 fp : c088beb4 r10: 00000000 r9 : 413fc090 r8 : 00000001 r7 : 00000000 r6 : c2984a08 r5 : c0966e78 r4 : 00000000 r3 : 0000006b r2 : 0000000c r1 : 00000000 r0 : c2984a08 Flags: nzCv IRQs off FIQs off Mode SVC_32 ISA ARM Segment kernel Control: 10c5387d Table: 0000404a DAC: 00000015 Process swapper (pid: 0, stack limit = 0xc088a248) Stack: (0xc088bea8 to 0xc088c000) bea0: c088bed4 c088beb8 c0110a3c c02c6d90 c0966e78 00000040 bec0: ef001f00 00000040 c088bf14 c088bed8 c0112070 c0110a04 00000005 c010fac8 bee0: c088bf5c c088bef0 c010fac8 ef001f00 00000040 00000000 00000040 00000001 bf00: 413fc090 00000000 c088bf34 c088bf18 c0839190 c0112040 00000000 ef001f00 bf20: 00000000 00000000 c088bf54 c088bf38 c0839200 c083914c 00000006 c0961c4c bf40: c0961c28 00000000 c088bf7c c088bf58 c08392ac c08391c0 c08a2ed8 c0966e78 bf60: c086b874 c08a3f50 c0961c28 00000001 c088bfb4 c088bf80 c083b258 c0839248 bf80: 2f800000 0f000000 c08935b4 ffffffff c08cd400 ffffffff c08cd400 c0868408 bfa0: c29849c0 00000000 c088bff4 c088bfb8 c0824974 c083b1e4 ffffffff ffffffff bfc0: c08245c0 00000000 00000000 c0868408 00000000 10c5387d c0892bcc c0868404 bfe0: c0899440 0000406a 00000000 c088bff8 00008074 c0824824 00000000 00000000 [<c02c6da0>] (strncmp+0x1c/0x84) from [<c0110a3c>] (kmem_cache_flags.isra.46.part.47+0x44/0x60) [<c0110a3c>] (kmem_cache_flags.isra.46.part.47+0x44/0x60) from [<c0112070>] (__kmem_cache_create+0x3c/0x410) [<c0112070>] (__kmem_cache_create+0x3c/0x410) from [<c0839190>] (create_boot_cache+0x50/0x74) [<c0839190>] (create_boot_cache+0x50/0x74) from [<c0839200>] (create_kmalloc_cache+0x4c/0x88) [<c0839200>] (create_kmalloc_cache+0x4c/0x88) from [<c08392ac>] (create_kmalloc_caches+0x70/0x114) [<c08392ac>] (create_kmalloc_caches+0x70/0x114) from [<c083b258>] (kmem_cache_init+0x80/0xe0) [<c083b258>] (kmem_cache_init+0x80/0xe0) from [<c0824974>] (start_kernel+0x15c/0x318) [<c0824974>] (start_kernel+0x15c/0x318) from [<00008074>] (0x8074) Code: e3520000 01a00002 089da800 e5d03000 (e5d1c000) ---[ end trace 1b75b31a2719ed1d ]--- Kernel panic - not syncing: Fatal exception
Problem is that slub_debug option is not parsed before create_boot_cache is called. Solve this by changing slub_debug to early_param.
Kernels 3.11, 3.10 are also affected.
I am not sure about older kernels.
--- mm/slub.c | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-)
Patch is against v3.12-48-gbe408cd.
Regards,
Andreas
mm/slub.c +2 -3
@@ -1205,10 +1205,9 @@ check_slabs:if (*str ==',' ) slub_debug_slabs = str +1 ; out: -return 1 ; +return 0 ; } - -__setup ("slub_debug" , setup_slub_debug); +early_param ("slub_debug" , setup_slub_debug); static unsigned long kmem_cache_flags(unsigned long object_size, unsigned long flags, const char *name,
Signature · 7 lines
1.7.9.5 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 3 lines · Andreas Herrmann
When I've used
slub_debugkernel option (e.g. "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've seen a panic like:
Hmmm.. That looks like its due to some slabs not having names during early boot. kmem_cache_flags is called with NULL as a parameter.
Are you sure that this fixes the issue? Looks like the kmem_cache_flag function should fail regardless of how early you set it.
AFAICT the right fix would be:
Subject: slub: Handle NULL parameter in kmem_cache_flags
kmem_cache_flags may be called with NULL parameter during early boot. Skip the test in that case.
mm/slub.c +1 -1
@@ -1217,7 +1217,7 @@ static unsigned long kmem_cache_flags(un/* * Enable debugging if selected on the kernel commandline. */ -if (slub_debug && (!slub_debug_slabs || +if (slub_debug && name && (!slub_debug_slabs || !strncmp (slub_debug_slabs, name,strlen (slub_debug_slabs)))) flags |= slub_debug;
Quoted 10 lines · Christoph Lameter
Quoted 3 lines · Andreas Herrmann
When I've used
slub_debugkernel option (e.g. "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've seen a panic like:Hmmm.. That looks like its due to some slabs not having names during early boot.
kmem_cache_flagsis called with NULL as a parameter.Are you sure that this fixes the issue?
?
Sure, of course I've tested my patch as I required slub debugging. Panic was gone and debugging of the selected slab enabled.
Andreas
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Andreas Herrmann
When I've used
slub_debugkernel option (e.g. "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've seen a panic like:Hmmm.. That looks like its due to some slabs not having names during early boot.
kmem_cache_flagsis called with NULL as a parameter.
That's because the slub_debug parameter is not evaluated before kmem_cache_flags is called.
Older kernels didn't show this problem. I think the sequence of those calls has changed. Not sure what patch set has made that change.
Quoted 4 lines
Are you sure that this fixes the issue? Looks like the
kmem_cache_flagfunction should fail regardless of how early you set it.AFAICT the right fix would be:
That would avoid the panic but I guess it won't enable slub debugging.
However I'll test this.
Andreas
Quoted 22 lines
Subject: slub: Handle NULL parameter in
kmem_cache_flags
kmem_cache_flagsmay be called with NULL parameter during early boot. Skip the test in that case.Signed-off-by Christoph Lameter <[email protected]>Index:
linux/mm/slub.c
===================================================================mm/slub.c +1
-1@@ -1217,7 +1217,7 @@ static unsigned long kmem_cache_flags(un/* * Enable debugging if selected on the kernel commandline. */ -if (slub_debug && (!slub_debug_slabs || +if (slub_debug && name && (!slub_debug_slabs || !strncmp (slub_debug_slabs, name,strlen (slub_debug_slabs)))) flags |= slub_debug;
Quoted 7 lines · Andreas Herrmann
Quoted 6 lines · Christoph Lameter
Quoted 3 lines · Andreas Herrmann
When I've used
slub_debugkernel option (e.g. "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've seen a panic like:
Quoted 8 lines
Quoted 2 lines
Hmmm.. That looks like its due to some slabs not having names during early boot.
kmem_cache_flagsis called with NULL as a parameter.That's because the
slub_debugparameter is not evaluated beforekmem_cache_flagsis called.Older kernels didn't show this problem. I think the sequence of those calls has changed. Not sure what patch set has made that change.
Please ignore this comment.
I revisisted the code and of course you are right. Hmm, now wondering why my patch covered the panic.
Quoted 6 lines
Quoted 4 lines
Are you sure that this fixes the issue? Looks like the
kmem_cache_flagfunction should fail regardless of how early you set it.AFAICT the right fix would be:
That would avoid the panic but I guess it won't enable slub debugging.
Ok, the latter is bogus. Of course debugging is switched on.
Quoted 1 line
However I'll test this.
Your patch fixed it.
Andreas
Quoted 14 lines · Andreas Herrmann
Quoted 13 lines · Andreas Herrmann
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Andreas Herrmann
When I've used
slub_debugkernel option (e.g. "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've seen a panic like:Hmmm.. That looks like its due to some slabs not having names during early boot.
kmem_cache_flagsis called with NULL as a parameter.That's because the
slub_debugparameter is not evaluated beforekmem_cache_flagsis called.
Quoted 6 lines
Quoted 2 lines
Older kernels didn't show this problem. I think the sequence of those calls has changed. Not sure what patch set has made that change.
Please ignore this comment.
I revisisted the code and of course you are right. Hmm, now wondering why my patch covered the panic.
Arrgh, my patch changed slub_debug_slabs to be NULL. That is why the panic didn't happen.
Would be nice, if your patch is pushed upstream asap.
Thanks,
Andreas
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Christoph Lameter
Tested-by Andreas HerrmannQuoted 1 line · Andreas Herrmann
Would be nice, if your patch is pushed upstream asap.
Ok so this is a
I think?
BTW Calxeda is a great product. Hope you get 64 bit running soon.
Quoted 9 lines · Christoph Lameter
Quoted 1 line · Andreas Herrmann
Would be nice, if your patch is pushed upstream asap.
Ok so this is a
Tested-by Andreas Herrmann <[email protected]>I think?
Yes.
Quoted 1 line
BTW Calxeda is a great product. Hope you get 64 bit running soon.
Thanks,
Andreas
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 5 lines · Andreas Herrmann
Quoted 4 lines · Christoph Lameter
Quoted 1 line · Andreas Herrmann
Would be nice, if your patch is pushed upstream asap.
Quoted 7 lines
Quoted 5 lines
Ok so this is a
Tested-by Andreas Herrmann <[email protected]>I think?
Yes.
And for sake of completeness. Here is some debug output with a kernel that had your "slub: Handle NULL parameter in kmem_cache_flags" patch applied. And of course there were a couple of unnamed slabs:
...
.bss : 0xc089fd80 - 0xc094cc4c ( 692 kB) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (c06fc90c): kmem_cache_node slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (c06fc91c): kmem_cache a?? slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null)
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
...
The third one is wheree the panic happened w/o the fix.
Andreas
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
Quoted 17 lines · Andreas Herrmann
And for sake of completeness. Here is some debug output with a kernel that had your "slub: Handle NULL parameter in
kmem_cache_flags" patch applied. And of course there were a couple of unnamed slabs:...
.bss : 0xc089fd80 - 0xc094cc4c ( 692 kB) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (c06fc90c): kmem_cache_node slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (c06fc91c): kmem_cache a?? slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null) slub_debug_slabs (c2956a08): skbuff_fclone_cache, name ( (null)): (null)
Quoted 2 lines
SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
...
Well yes on bootstrap the slabs are initially created without a name. Later the name is provided. Before the introduction of the common kmalloc code the kmalloc caches had a generic name instead of NULL. I was not sure that this was a valid thing to do so I put NULL in there so that we can catch the uses.
A message that is not in the archive (sent off-list or not yet received): [email protected]
There is still something not optimal with the patch since it would disable debuggin on the kmalloc stack. Ccheck name for NULL only if slub_debug_slabs.
Subject: slub: Handle NULL parameter in kmem_cache_flags V2
V1->V2
- flags need to be applied regardless if !slub_debug_slabs
kmem_cache_flags may be called with NULL parameter during early boot. Skip the test in that case.
mm/slub.c +2 -2
@@ -1217,8 +1217,8 @@ static unsigned long kmem_cache_flags(un/* * Enable debugging if selected on the kernel commandline. */ -if (slub_debug && (!slub_debug_slabs || - !strncmp (slub_debug_slabs, name,strlen (slub_debug_slabs)))) +if (slub_debug && (!slub_debug_slabs || (name && + !strncmp (slub_debug_slabs, name,strlen (slub_debug_slabs))))) flags |= slub_debug;return flags;
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>