Skip to contentSkip to conversation

mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

PATCH In review Andreas Herrmann 2 participants 11 messages Nov 6, 2013 – Nov 7, 2013 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 mm/slub: Switch slub_debug kernel option to early_param to avoid boot panic

When I've used slub_debug kernel option (e.g. "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've seen a panic like:

Highbank #setenv bootargs console=ttyAMA0 root=/dev/sda2 kgdboc.kgdboc=ttyAMA0,115200 slub_debug=,kmalloc-4096 earlyprintk=ttyAMA0
...

  Unable to handle kernel NULL pointer dereference at virtual address 00000000
  pgd = c0004000
  [00000000] *pgd=00000000
  Internal error: Oops: 5 [#1] SMP ARM
  Modules linked in:
  CPU: 0 PID: 0 Comm: swapper Tainted: G        W    3.12.0-00048-gbe408cd #314
  task: c0898360 ti: c088a000 task.ti: c088a000
  PC is at strncmp+0x1c/0x84
  LR is at kmem_cache_flags.isra.46.part.47+0x44/0x60
  pc : [<c02c6da0>]    lr : [<c0110a3c>]    psr: 200001d3
  sp : c088bea8  ip : c088beb8  fp : c088beb4
  r10: 00000000  r9 : 413fc090  r8 : 00000001
  r7 : 00000000  r6 : c2984a08  r5 : c0966e78  r4 : 00000000
  r3 : 0000006b  r2 : 0000000c  r1 : 00000000  r0 : c2984a08
  Flags: nzCv  IRQs off  FIQs off  Mode SVC_32  ISA ARM  Segment kernel
  Control: 10c5387d  Table: 0000404a  DAC: 00000015
  Process swapper (pid: 0, stack limit = 0xc088a248)
  Stack: (0xc088bea8 to 0xc088c000)
  bea0:                   c088bed4 c088beb8 c0110a3c c02c6d90 c0966e78 00000040
  bec0: ef001f00 00000040 c088bf14 c088bed8 c0112070 c0110a04 00000005 c010fac8
  bee0: c088bf5c c088bef0 c010fac8 ef001f00 00000040 00000000 00000040 00000001
  bf00: 413fc090 00000000 c088bf34 c088bf18 c0839190 c0112040 00000000 ef001f00
  bf20: 00000000 00000000 c088bf54 c088bf38 c0839200 c083914c 00000006 c0961c4c
  bf40: c0961c28 00000000 c088bf7c c088bf58 c08392ac c08391c0 c08a2ed8 c0966e78
  bf60: c086b874 c08a3f50 c0961c28 00000001 c088bfb4 c088bf80 c083b258 c0839248
  bf80: 2f800000 0f000000 c08935b4 ffffffff c08cd400 ffffffff c08cd400 c0868408
  bfa0: c29849c0 00000000 c088bff4 c088bfb8 c0824974 c083b1e4 ffffffff ffffffff
  bfc0: c08245c0 00000000 00000000 c0868408 00000000 10c5387d c0892bcc c0868404
  bfe0: c0899440 0000406a 00000000 c088bff8 00008074 c0824824 00000000 00000000
  [<c02c6da0>] (strncmp+0x1c/0x84) from [<c0110a3c>] (kmem_cache_flags.isra.46.part.47+0x44/0x60)
  [<c0110a3c>] (kmem_cache_flags.isra.46.part.47+0x44/0x60) from [<c0112070>] (__kmem_cache_create+0x3c/0x410)
  [<c0112070>] (__kmem_cache_create+0x3c/0x410) from [<c0839190>] (create_boot_cache+0x50/0x74)
  [<c0839190>] (create_boot_cache+0x50/0x74) from [<c0839200>] (create_kmalloc_cache+0x4c/0x88)
  [<c0839200>] (create_kmalloc_cache+0x4c/0x88) from [<c08392ac>] (create_kmalloc_caches+0x70/0x114)
  [<c08392ac>] (create_kmalloc_caches+0x70/0x114) from [<c083b258>] (kmem_cache_init+0x80/0xe0)
  [<c083b258>] (kmem_cache_init+0x80/0xe0) from [<c0824974>] (start_kernel+0x15c/0x318)
  [<c0824974>] (start_kernel+0x15c/0x318) from [<00008074>] (0x8074)
  Code: e3520000 01a00002 089da800 e5d03000 (e5d1c000)
  ---[ end trace 1b75b31a2719ed1d ]---
  Kernel panic - not syncing: Fatal exception

Problem is that slub_debug option is not parsed before create_boot_cache is called. Solve this by changing slub_debug to early_param.

Kernels 3.11, 3.10 are also affected.
I am not sure about older kernels.

---
 mm/slub.c |    5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

Patch is against v3.12-48-gbe408cd.

Regards,
Andreas

mm/slub.c +2 -3
index c3eb3d3..596c58a 100644
@@ -1205,10 +1205,9 @@ check_slabs:
 	if (*str == ',')
 		slub_debug_slabs = str + 1;
 out:
-	return 1;
+	return 0;
 }
-
-__setup("slub_debug", setup_slub_debug);
+early_param("slub_debug", setup_slub_debug);
 
 static unsigned long kmem_cache_flags(unsigned long object_size,
 	unsigned long flags, const char *name,
Signature · 7 lines
1.7.9.5

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#1 lore raw
Quoted 3 lines · Andreas Herrmann

When I've used slub_debug kernel option (e.g. "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've seen a panic like:

Hmmm.. That looks like its due to some slabs not having names during early boot. kmem_cache_flags is called with NULL as a parameter.

Are you sure that this fixes the issue? Looks like the kmem_cache_flag function should fail regardless of how early you set it.

AFAICT the right fix would be:

Subject: slub: Handle NULL parameter in kmem_cache_flags

kmem_cache_flags may be called with NULL parameter during early boot. Skip the test in that case.

mm/slub.c +1 -1
Index: linux/mm/slub.c
===================================================================
@@ -1217,7 +1217,7 @@ static unsigned long kmem_cache_flags(un
 	/*
 	 * Enable debugging if selected on the kernel commandline.
 	 */
-	if (slub_debug && (!slub_debug_slabs ||
+	if (slub_debug && name && (!slub_debug_slabs ||
 		!strncmp(slub_debug_slabs, name, strlen(slub_debug_slabs))))
 		flags |= slub_debug;
 
#2 lore raw
Quoted 10 lines · Christoph Lameter
Quoted 3 lines · Andreas Herrmann

When I've used slub_debug kernel option (e.g. "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've seen a panic like:

Hmmm.. That looks like its due to some slabs not having names during early boot. kmem_cache_flags is called with NULL as a parameter.

Are you sure that this fixes the issue?

?

Sure, of course I've tested my patch as I required slub debugging. Panic was gone and debugging of the selected slab enabled.

Andreas

#3 lore raw
In reply to Christoph Lameter
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Andreas Herrmann

When I've used slub_debug kernel option (e.g. "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've seen a panic like:

Hmmm.. That looks like its due to some slabs not having names during early boot. kmem_cache_flags is called with NULL as a parameter.

That's because the slub_debug parameter is not evaluated before kmem_cache_flags is called.

Older kernels didn't show this problem. I think the sequence of those calls has changed. Not sure what patch set has made that change.

Quoted 4 lines

Are you sure that this fixes the issue? Looks like the kmem_cache_flag function should fail regardless of how early you set it.

AFAICT the right fix would be:

That would avoid the panic but I guess it won't enable slub debugging.

However I'll test this.

Andreas

Quoted 22 lines

Subject: slub: Handle NULL parameter in kmem_cache_flags

kmem_cache_flags may be called with NULL parameter during early boot. Skip the test in that case.

Index: linux/mm/slub.c
===================================================================

mm/slub.c +1 -1
@@ -1217,7 +1217,7 @@ static unsigned long kmem_cache_flags(un
 	/*
 	 * Enable debugging if selected on the kernel commandline.
 	 */
-	if (slub_debug && (!slub_debug_slabs ||
+	if (slub_debug && name && (!slub_debug_slabs ||
 		!strncmp(slub_debug_slabs, name, strlen(slub_debug_slabs))))
 		flags |= slub_debug;
 
#4 lore raw
Quoted 7 lines · Andreas Herrmann
Quoted 6 lines · Christoph Lameter
Quoted 3 lines · Andreas Herrmann

When I've used slub_debug kernel option (e.g. "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've seen a panic like:

Quoted 8 lines
Quoted 2 lines

Hmmm.. That looks like its due to some slabs not having names during early boot. kmem_cache_flags is called with NULL as a parameter.

That's because the slub_debug parameter is not evaluated before kmem_cache_flags is called.

Older kernels didn't show this problem. I think the sequence of those calls has changed. Not sure what patch set has made that change.

Please ignore this comment.
I revisisted the code and of course you are right. Hmm, now wondering why my patch covered the panic.

Quoted 6 lines
Quoted 4 lines

Are you sure that this fixes the issue? Looks like the kmem_cache_flag function should fail regardless of how early you set it.

AFAICT the right fix would be:

That would avoid the panic but I guess it won't enable slub debugging.

Ok, the latter is bogus. Of course debugging is switched on.

Quoted 1 line

However I'll test this.

Your patch fixed it.

Andreas

#5 lore raw
Quoted 14 lines · Andreas Herrmann
Quoted 13 lines · Andreas Herrmann
Quoted 8 lines · Christoph Lameter
Quoted 3 lines · Andreas Herrmann

When I've used slub_debug kernel option (e.g. "slub_debug=,skbuff_fclone_cache" or similar) on a debug session I've seen a panic like:

Hmmm.. That looks like its due to some slabs not having names during early boot. kmem_cache_flags is called with NULL as a parameter.

That's because the slub_debug parameter is not evaluated before kmem_cache_flags is called.

Quoted 6 lines
Quoted 2 lines

Older kernels didn't show this problem. I think the sequence of those calls has changed. Not sure what patch set has made that change.

Please ignore this comment.
I revisisted the code and of course you are right. Hmm, now wondering why my patch covered the panic.

Arrgh, my patch changed slub_debug_slabs to be NULL. That is why the panic didn't happen.

Would be nice, if your patch is pushed upstream asap.

Thanks,

Andreas

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw
Quoted 9 lines · Christoph Lameter
Quoted 1 line · Andreas Herrmann

Would be nice, if your patch is pushed upstream asap.

Ok so this is a

I think?

Yes.

Quoted 1 line

BTW Calxeda is a great product. Hope you get 64 bit running soon.

Thanks,

Andreas

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#8 lore raw
Quoted 5 lines · Andreas Herrmann
Quoted 4 lines · Christoph Lameter
Quoted 1 line · Andreas Herrmann

Would be nice, if your patch is pushed upstream asap.

Quoted 7 lines
Quoted 5 lines

Ok so this is a

I think?

Yes.

And for sake of completeness. Here is some debug output with a kernel that had your "slub: Handle NULL parameter in kmem_cache_flags" patch applied. And of course there were a couple of unnamed slabs:

...

         .bss : 0xc089fd80 - 0xc094cc4c   ( 692 kB)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (c06fc90c): kmem_cache_node
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (c06fc91c): kmem_cache
a?? slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)

SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
...

The third one is wheree the panic happened w/o the fix.

Andreas

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw
Quoted 17 lines · Andreas Herrmann

And for sake of completeness. Here is some debug output with a kernel that had your "slub: Handle NULL parameter in kmem_cache_flags" patch applied. And of course there were a couple of unnamed slabs:

...

         .bss : 0xc089fd80 - 0xc094cc4c   ( 692 kB)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (c06fc90c): kmem_cache_node
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (c06fc91c): kmem_cache
a?? slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
  slub_debug_slabs (c2956a08): skbuff_fclone_cache, name (  (null)): (null)
Quoted 2 lines

SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
...

Well yes on bootstrap the slabs are initially created without a name. Later the name is provided. Before the introduction of the common kmalloc code the kmalloc caches had a generic name instead of NULL. I was not sure that this was a valid thing to do so I put NULL in there so that we can catch the uses.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#11 lore raw

There is still something not optimal with the patch since it would disable debuggin on the kmalloc stack. Ccheck name for NULL only if slub_debug_slabs.

Subject: slub: Handle NULL parameter in kmem_cache_flags V2

V1->V2
- flags need to be applied regardless if !slub_debug_slabs

kmem_cache_flags may be called with NULL parameter during early boot. Skip the test in that case.

mm/slub.c +2 -2
Index: linux/mm/slub.c
===================================================================
@@ -1217,8 +1217,8 @@ static unsigned long kmem_cache_flags(un
 	/*
 	 * Enable debugging if selected on the kernel commandline.
 	 */
-	if (slub_debug && (!slub_debug_slabs ||
-		!strncmp(slub_debug_slabs, name, strlen(slub_debug_slabs))))
+	if (slub_debug && (!slub_debug_slabs || (name &&
+		!strncmp(slub_debug_slabs, name, strlen(slub_debug_slabs)))))
 		flags |= slub_debug;
 
 	return flags;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>