Skip to contentSkip to conversation

NULL pointer in kmem_cache_alloc with 2.6.25-rc1

DISCUSSION Zhang, Yanmin 5 participants 8 messages Feb 15, 2008 – Feb 27, 2008

lore
Share & tools
lore
#0 lore raw

On my 16-core tigerton, kernel panic when I ran hackbench process testing. See below log.

Commandline to start hackbench:
#./hackbench 150 process 2000

Kernel config options:
CONFIG_PREEMPT_NONE=y
# CONFIG_PREEMPT_VOLUNTARY is not set
# CONFIG_PREEMPT is not set

Kernel panic at line 1637 in file mm/slub.c because object=c->freelist=NULL.

-yanmin

********************syslog***************

lkp-tt01-x8664 login: BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
IP: [<ffffffff80279948>] kmem_cache_alloc+0x3c/0x6c
PGD 45c5a1067 PUD 45c5a2067 PMD 0

Oops: 0000 [1] SMP
CPU 0
Modules linked in: video output
Pid: 8095, comm: hackbench Not tainted 2.6.25-rc1 #1
RIP: 0010:[<ffffffff80279948>]  [<ffffffff80279948>] kmem_cache_alloc+0x3c/0x6c
RSP: 0018:ffff81045c5b3bf8  EFLAGS: 00010246
RAX: 0000000000000000 RBX: ffff810001007d40 RCX: ffffffff805e0024
RDX: 0000000000000000 RSI: 00000000000004d0 RDI: ffffffff80928890
RBP: 00000000000004d0 R08: ffff81045cfa2a80 R09: ffff81045cfa2a80
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000064
R13: ffffffff80928890 R14: 0000000000000000 R15: 0000000000000000
FS:  00007fb63b0ab6f0(0000) GS:ffffffff808ff000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000045c5a0000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process hackbench (pid: 8095, threadinfo ffff81045c5b2000, task ffff81045c5ae7b0)
Stack:  ffff810437c399c0 0000000000000000 00000000000004d0 ffffffff805e0024
 ffff81045cfdbb80 00000000000004d0 7fffffffffffffff 0000000000000064
 ffff81045c5b3d04 ffffffff805dc389 0000000000000000 ffff81045c5ae7b0
Call Trace:
 [<ffffffff805e0024>] ? __alloc_skb+0x31/0x121
 [<ffffffff805dc389>] ? sock_alloc_send_skb+0x77/0x1d2
 [<ffffffff80243897>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff805e12d5>] ? memcpy_fromiovec+0x36/0x66
 [<ffffffff80636ec0>] ? unix_stream_sendmsg+0x165/0x333
 [<ffffffff805d9863>] ? sock_aio_write+0xd1/0xe0
 [<ffffffff80227834>] ? __wake_up_common+0x41/0x74
 [<ffffffff8027d267>] ? do_sync_write+0xc9/0x10c
 [<ffffffff80265fbb>] ? __do_fault+0x382/0x3cd
 [<ffffffff80243897>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff80268840>] ? handle_mm_fault+0x38a/0x70d
 [<ffffffff806afd39>] ? error_exit+0x0/0x51
 [<ffffffff80228008>] ? __dequeue_entity+0x1c/0x32
 [<ffffffff8027d9bb>] ? vfs_write+0xc0/0x136
 [<ffffffff8027dee5>] ? sys_write+0x45/0x6e
 [<ffffffff8020bdcb>] ? system_call_after_swapgs+0x7b/0x80
#1 lore raw
Quoted 5 lines · Yanmin

On my 16-core tigerton, kernel panic when I ran hackbench process testing. See below log.

Kernel panic at line 1637 in file mm/slub.c because object=c->freelist=NULL.

Hmmmm. freelist should never be NULL. Could you rerun the test and boot with slub_debug to make sure that there is no memory corruption?

#2 lore raw
Quoted 10 lines · Christoph Lameter
Quoted 5 lines · Yanmin

On my 16-core tigerton, kernel panic when I ran hackbench process testing. See below log.

Kernel panic at line 1637 in file mm/slub.c because object=c->freelist=NULL.

Hmmmm. freelist should never be NULL. Could you rerun the test and boot with slub_debug to make sure that there is no memory corruption?

1) Without slub_debug option, sometime I could trigger it, sometimes not.
2) With slub_debug option, or just enable debug for slab skbuff_fclone_cache and skbuff_head_cache, I couldn't trigger it.

I will do more testing and investigation, as the bug also exists in 2.6.25-rc2.

-yanmin

#3 lore raw

Hi Yanmin,

Quoted 5 lines · Christoph Lameter
Quoted 4 lines
Quoted 1 line
Kernel panic at line 1637 in file mm/slub.c because object=c->freelist=NULL.

Hmmmm. freelist should never be NULL. Could you rerun the test and boot with slub_debug to make sure that there is no memory corruption?

Quoted 5 lines · Yanmin

1) Without slub_debug option, sometime I could trigger it, sometimes not.
2) With slub_debug option, or just enable debug for slab skbuff_fclone_cache and skbuff_head_cache, I couldn't trigger it.

I will do more testing and investigation, as the bug also exists in 2.6.25-rc2.

Could you please try Ingo's patch: http://lkml.org/lkml/2008/2/19/13

Looks like there are some problems with SLUB_FASTPATH.

Pekka

#4 lore raw
Quoted 5 lines · Pekka Enberg
Quoted 1 line

I will do more testing and investigation, as the bug also exists in 2.6.25-rc2.

Could you please try Ingo's patch: http://lkml.org/lkml/2008/2/19/13

Looks like there are some problems with SLUB_FASTPATH.

Yeah that would determine if its the fastpath code. Note that the new fastpath code may expose bugs that so far have been tolerated by the slab allocators. If a store of a NULL to the first word of an object occurs then this will be tolerated in 2.6.24 because slub will take it as the end of the object list and just forget about the remaining objects. The fastpath code will terminate with a NULL pointer deref.

Enabling full debugging may get rid of the race due to the slowdown. It could help to just enable minimal debugging for the slab cache in which we see the issue.

So do

slub_debug=F,<affected cache>

#5 lore raw
In reply to Zhang, Yanmin
Quoted 9 lines · Yanmin
Call Trace:
 [<ffffffff805e0024>] ? __alloc_skb+0x31/0x121
 [<ffffffff805dc389>] ? sock_alloc_send_skb+0x77/0x1d2
 [<ffffffff80243897>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff805e12d5>] ? memcpy_fromiovec+0x36/0x66
 [<ffffffff80636ec0>] ? unix_stream_sendmsg+0x165/0x333
 [<ffffffff805d9863>] ? sock_aio_write+0xd1/0xe0
 [<ffffffff80227834>] ? __wake_up_common+0x41/0x74
 [<ffffffff8027d267>] ? do_sync_write+0xc9/0x10c
Quoted 8 lines
 [<ffffffff80265fbb>] ? __do_fault+0x382/0x3cd
 [<ffffffff80243897>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff80268840>] ? handle_mm_fault+0x38a/0x70d
 [<ffffffff806afd39>] ? error_exit+0x0/0x51
 [<ffffffff80228008>] ? __dequeue_entity+0x1c/0x32
 [<ffffffff8027d9bb>] ? vfs_write+0xc0/0x136
 [<ffffffff8027dee5>] ? sys_write+0x45/0x6e
 [<ffffffff8020bdcb>] ? system_call_after_swapgs+0x7b/0x80

off-topic, but... Why are all the backtrace decodes here marked as being unreliable?

#6 lore raw
Quoted 21 lines · On Mon, 18 Feb 2008 04:59:18 -0800 Andrew Morton <[email protected]>…
Quoted 9 lines · Yanmin
Call Trace:
 [<ffffffff805e0024>] ? __alloc_skb+0x31/0x121
 [<ffffffff805dc389>] ? sock_alloc_send_skb+0x77/0x1d2
 [<ffffffff80243897>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff805e12d5>] ? memcpy_fromiovec+0x36/0x66
 [<ffffffff80636ec0>] ? unix_stream_sendmsg+0x165/0x333
 [<ffffffff805d9863>] ? sock_aio_write+0xd1/0xe0
 [<ffffffff80227834>] ? __wake_up_common+0x41/0x74
 [<ffffffff8027d267>] ? do_sync_write+0xc9/0x10c
Quoted 8 lines
 [<ffffffff80265fbb>] ? __do_fault+0x382/0x3cd
 [<ffffffff80243897>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff80268840>] ? handle_mm_fault+0x38a/0x70d
 [<ffffffff806afd39>] ? error_exit+0x0/0x51
 [<ffffffff80228008>] ? __dequeue_entity+0x1c/0x32
 [<ffffffff8027d9bb>] ? vfs_write+0xc0/0x136
 [<ffffffff8027dee5>] ? sys_write+0x45/0x6e
 [<ffffffff8020bdcb>] ? system_call_after_swapgs+0x7b/0x80
Quoted 2 lines

off-topic, but... Why are all the backtrace decodes here marked as being unreliable?

probably because the stack is a tad confused, so the back tracer doesn't see even a single good stack frame.

Is CONFIG_FRAME_POINTER on?

Signature · 3 lines
If you want to reach me at my work email, use [email protected]
For development, discussion and tips for power savings, 
visit http://www.lesswatts.org
#7 lore raw
Quoted 24 lines · Arjan van de Ven
Quoted 21 lines · On Mon, 18 Feb 2008 04:59:18 -0800 Andrew Morton <[email protected]>…
Quoted 17 lines · Yanmin
Call Trace:
 [<ffffffff805e0024>] ? __alloc_skb+0x31/0x121
 [<ffffffff805dc389>] ? sock_alloc_send_skb+0x77/0x1d2
 [<ffffffff80243897>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff805e12d5>] ? memcpy_fromiovec+0x36/0x66
 [<ffffffff80636ec0>] ? unix_stream_sendmsg+0x165/0x333
 [<ffffffff805d9863>] ? sock_aio_write+0xd1/0xe0
 [<ffffffff80227834>] ? __wake_up_common+0x41/0x74
 [<ffffffff8027d267>] ? do_sync_write+0xc9/0x10c
 [<ffffffff80265fbb>] ? __do_fault+0x382/0x3cd
 [<ffffffff80243897>] ? autoremove_wake_function+0x0/0x2e
 [<ffffffff80268840>] ? handle_mm_fault+0x38a/0x70d
 [<ffffffff806afd39>] ? error_exit+0x0/0x51
 [<ffffffff80228008>] ? __dequeue_entity+0x1c/0x32
 [<ffffffff8027d9bb>] ? vfs_write+0xc0/0x136
 [<ffffffff8027dee5>] ? sys_write+0x45/0x6e
 [<ffffffff8020bdcb>] ? system_call_after_swapgs+0x7b/0x80
Quoted 2 lines
Quoted 2 lines

off-topic, but... Why are all the backtrace decodes here marked as being unreliable?

At least ffffffff80279948 is correct. The register values and ip address looks like matching the disassembled codes.

Quoted 5 lines

probably because the stack is a tad confused, so the back tracer doesn't see even a single good stack frame.

Is CONFIG_FRAME_POINTER on?

No.

With 2.6.25-rc2, 3 x86-64 machines hit the same issue.

-yanmin