cpu hotplug oops on 2.6.15-rc5
(apologies if this is a dup)
Hi, I'm crashing 2.6.15-rc5 when I try and offline the last and only CPU in a node on a ppc64 Power5, SMT was disabled.
Here's the backtrace:
0:mon> t
[c0000001ad033820] c000000000096a7c .kfree+0x250/0x280 [c0000001ad0338d0] c00000000009a544 .cpuup_callback+0x238/0x5fc [c0000001ad0339c0] c000000000068114 .notifier_call_chain+0x68/0x9c [c0000001ad033a50] c0000000000789fc .cpu_down+0x1fc/0x368 [c0000001ad033b40] c0000000002ac658 .store_online+0x88/0xe8 [c0000001ad033bd0] c0000000002a6f14 .sysdev_store+0x4c/0x68 [c0000001ad033c50] c000000000110368 .sysfs_write_file+0x100/0x1a0 [c0000001ad033cf0] c0000000000be854 .vfs_write+0x100/0x200 [c0000001ad033d90] c0000000000bea64 .sys_write+0x54/0x9c [c0000001ad033e30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ffc4c4f0) is in userspace
0:mon> e
cpu 0x0: Vector: 300 (Data Access) at [c0000001ad033520]
pc: c00000000048bd30: ._spin_lock+0x18/0x80
lr: c000000000096a7c: .kfree+0x250/0x280
sp: c0000001ad0337a0
msr: 8000000000001032
dar: 48
dsisr: 40000000
current = 0xc0000001aff12040
paca = 0xc0000000005c1000
pid = 17376, comm = bashShould I try this with CONFIG_DEBUG_SLAB ?
Sonny
Quoted 3 lines · Sonny Rao
(apologies if this is a dup)
Hi, I'm crashing 2.6.15-rc5 when I try and offline the last and only CPU in a node on a ppc64 Power5, SMT was disabled.
First try on -rc6 just in case it's related to the SCSI fix (the bug was corrupting the SLAB) that got merged just after rc5 iirc.
Ben.
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 3 lines · Sonny Rao
(apologies if this is a dup)
Hi, I'm crashing 2.6.15-rc5 when I try and offline the last and only CPU in a node on a ppc64 Power5, SMT was disabled.
First try on -rc6 just in case it's related to the SCSI fix (the bug was corrupting the SLAB) that got merged just after rc5 iirc.
Ok, tried it: same crash on -rc6
2:mon> t
[c000000d9f33b820] c000000000097cd0 .kfree+0x29c/0x2cc [c000000d9f33b8d0] c00000000009c3a8 .cpuup_callback+0x4f8/0x5fc [c000000d9f33b9c0] c00000000048ff4c .notifier_call_chain+0x68/0x9c [c000000d9f33ba50] c000000000078da8 .cpu_down+0x1fc/0x368 [c000000d9f33bb40] c0000000002ae514 .store_online+0x88/0xe8 [c000000d9f33bbd0] c0000000002a8dd0 .sysdev_store+0x4c/0x68 [c000000d9f33bc50] c000000000111e70 .sysfs_write_file+0x100/0x1a0 [c000000d9f33bcf0] c0000000000c0360 .vfs_write+0x100/0x200 [c000000d9f33bd90] c0000000000c0570 .sys_write+0x54/0x9c [c000000d9f33be30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ffa204f0) is in userspace
2:mon>
Sonny
Quoted 15 lines · Sonny Rao
Ok, tried it: same crash on -rc6
2:mon> t
[c000000d9f33b820]c000000000097cd0.kfree+0x29c/0x2cc [c000000d9f33b8d0]c00000000009c3a8.cpuup_callback+0x4f8/0x5fc [c000000d9f33b9c0]c00000000048ff4c.notifier_call_chain+0x68/0x9c [c000000d9f33ba50]c000000000078da8.cpu_down+0x1fc/0x368 [c000000d9f33bb40]c0000000002ae514.store_online+0x88/0xe8 [c000000d9f33bbd0]c0000000002a8dd0.sysdev_store+0x4c/0x68 [c000000d9f33bc50]c000000000111e70.sysfs_write_file+0x100/0x1a0 [c000000d9f33bcf0]c0000000000c0360.vfs_write+0x100/0x200 [c000000d9f33bd90]c0000000000c0570.sys_write+0x54/0x9c [c000000d9f33be30]c000000000008600syscall_exit+0x0/0x18
Very odd call chain.
Could you enable slab debugging?
--
Manfred
Re: SPAMHAUS-Re: cpu hotplug oops on 2.6.15-rc5
Quoted 17 lines · Manfred Spraul
Quoted 15 lines · Sonny Rao
Ok, tried it: same crash on -rc6
2:mon> t
[c000000d9f33b820]c000000000097cd0.kfree+0x29c/0x2cc [c000000d9f33b8d0]c00000000009c3a8.cpuup_callback+0x4f8/0x5fc [c000000d9f33b9c0]c00000000048ff4c.notifier_call_chain+0x68/0x9c [c000000d9f33ba50]c000000000078da8.cpu_down+0x1fc/0x368 [c000000d9f33bb40]c0000000002ae514.store_online+0x88/0xe8 [c000000d9f33bbd0]c0000000002a8dd0.sysdev_store+0x4c/0x68 [c000000d9f33bc50]c000000000111e70.sysfs_write_file+0x100/0x1a0 [c000000d9f33bcf0]c0000000000c0360.vfs_write+0x100/0x200 [c000000d9f33bd90]c0000000000c0570.sys_write+0x54/0x9c [c000000d9f33be30]c000000000008600syscall_exit+0x0/0x18
Quoted 2 lines
Very odd call chain.
Could you enable slab debugging?
Actually, I did turn on slab debugging on -rc6, but it did not seem to make any difference.
Sonny
Hi Manfred,
Quoted 2 lines
Very odd call chain.
Could you enable slab debugging?
Sonny and I had a look around, it seems to be in the cpuup_callback() / CPU_DEAD case:
if (!cpus_empty(mask)) {
spin_unlock(&l3->list_lock);
goto unlock_cache;
}
if (l3->shared) {
free_block(cachep, l3->shared->entry,
l3->shared->avail, node);
kfree(l3->shared); <-------- HERE
l3->shared = NULL;
}So we are removing the last cpu in a node, and tearing down the node related structures. We looked at kfree() -> __cache_free() and we couldnt convince ourselves that all the CONFIG_NUMA stuff in there wouldnt trip over itself (since we would be doing the free on an alien node).
Anton
Quoted 22 lines · Sonny Rao
(apologies if this is a dup)
Hi, I'm crashing 2.6.15-rc5 when I try and offline the last and only CPU in a node on a ppc64 Power5, SMT was disabled.
Here's the backtrace:
0:mon> t
[c0000001ad033820]c000000000096a7c.kfree+0x250/0x280 [c0000001ad0338d0]c00000000009a544.cpuup_callback+0x238/0x5fc [c0000001ad0339c0]c000000000068114.notifier_call_chain+0x68/0x9c [c0000001ad033a50]c0000000000789fc.cpu_down+0x1fc/0x368 [c0000001ad033b40]c0000000002ac658.store_online+0x88/0xe8 [c0000001ad033bd0]c0000000002a6f14.sysdev_store+0x4c/0x68 [c0000001ad033c50]c000000000110368.sysfs_write_file+0x100/0x1a0 [c0000001ad033cf0]c0000000000be854.vfs_write+0x100/0x200 [c0000001ad033d90]c0000000000bea64.sys_write+0x54/0x9c [c0000001ad033e30]c000000000008600syscall_exit+0x0/0x18--- Exception: c01 (System Call) at000000000fe5ec10
SP (ffc4c4f0) is in userspace0:mon> e
cpu0x0: Vector: 300 (Data Access) at [c0000001ad033520]
Quoted 11 lines
pc: c00000000048bd30: ._spin_lock+0x18/0x80
lr: c000000000096a7c: .kfree+0x250/0x280
sp: c0000001ad0337a0
msr: 8000000000001032
dar: 48
dsisr: 40000000
current = 0xc0000001aff12040
paca = 0xc0000000005c1000
pid = 17376, comm = bashSonny,
Does this patch fix the issue? This one applies cleanly on 2.6.15-rc6
unlike the one that was sent to you earlier.
Thanks,
Kiran
From: Alok N Kataria <[email protected]>
Fixes a bug in the CPU_DOWN call path, we shouldn't call kfree while holding kmem_list3's list lock, nor should drain_alien_cache be called with l3's list lock.
mm/slab.c +24 -16
@@ -824,14 +824,14 @@ static inline void __drain_alien_cache(k } } -static void drain_alien_cache (kmem_cache_t *cachep,struct kmem_list3 *l3) +static void drain_alien_cache (kmem_cache_t *cachep,struct array_cache **alien) {int i=0 ;struct array_cache *ac;unsigned long flags;for_each_online_node (i) { - ac = l3->alien[i]; + ac = alien[i];if (ac) {spin_lock_irqsave (&ac->lock, flags);__drain_alien_cache (cachep, ac, i);
@@ -842,7 +842,7 @@ static void drain_alien_cache(kmem_cache#else #define alloc_alien_cache (node, limit) do { } while (0)#define free_alien_cache (ac_ptr) do { } while (0) -#define drain_alien_cache (cachep, l3) do { } while (0) +#define drain_alien_cache (cachep, alien) do { } while (0)#endif static int __devinit cpuup_callback(struct notifier_block *nfb,
@@ -921,7 +921,7 @@ static int __devinit cpuup_callback(strudown (&cache_chain_sem);list_for_each_entry (cachep, &cache_chain, next) { -struct array_cache *nc; +struct array_cache *nc, *shared, **alien;cpumask_t mask; mask =node_to_cpumask (node);
@@ -932,7 +932,7 @@ static int __devinit cpuup_callback(stru l3 = cachep->nodelists[node];if (!l3) - goto unlock_cache; + goto free_array_cache;spin_lock (&l3->list_lock);
@@ -943,32 +943,40 @@ static int __devinit cpuup_callback(struif (!cpus_empty (mask)) {spin_unlock (&l3->list_lock); - goto unlock_cache; + goto free_array_cache; } -if (l3->shared) { +if ((shared = l3->shared)) {free_block (cachep, l3->shared->entry, l3->shared->avail, node);kfree (l3->shared); l3->shared =NULL ; } -if (l3->alien) { -drain_alien_cache (cachep, l3); -free_alien_cache (l3->alien); - l3->alien =NULL ; + + alien = l3->alien; + l3->alien =NULL ; + +spin_unlock (&l3->list_lock); + +kfree (nc); +kfree (shared); +if (alien) { +drain_alien_cache (cachep, alien); +free_alien_cache (alien); }/* free slabs belonging to this node */ if (__node_shrink (cachep, node)) { +spin_lock (&l3->list_lock); cachep->nodelists[node] =NULL ;spin_unlock (&l3->list_lock);kfree (l3); - }else { -spin_unlock (&l3->list_lock); } + goto unlock_cache; +free_array_cache: +kfree (nc); unlock_cache:spin_unlock_irq (&cachep->spinlock); -kfree (nc); }up (&cache_chain_sem);break ;
@@ -1918,7 +1926,7 @@ static void drain_cpu_caches(kmem_cache_drain_array_locked (cachep, l3->shared,1 , node);spin_unlock (&l3->list_lock);if (l3->alien) -drain_alien_cache (cachep, l3); +drain_alien_cache (cachep, l3->alien); } }spin_unlock_irq (&cachep->spinlock);
@@ -3310,7 +3318,7 @@ static void cache_reap(void *unused) l3 = searchp->nodelists[numa_node_id ()];if (l3->alien) -drain_alien_cache (searchp, l3); +drain_alien_cache (searchp, l3->alien);spin_lock_irq (&l3->list_lock); drain_array_locked(searchp, ac_data(searchp), 0,
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 56 lines · Sonny Rao
Quoted 35 lines · Ravikiran G Thirumalai
Quoted 22 lines · Sonny Rao
(apologies if this is a dup)
Hi, I'm crashing 2.6.15-rc5 when I try and offline the last and only CPU in a node on a ppc64 Power5, SMT was disabled.
Here's the backtrace:
0:mon> t
[c0000001ad033820]c000000000096a7c.kfree+0x250/0x280 [c0000001ad0338d0]c00000000009a544.cpuup_callback+0x238/0x5fc [c0000001ad0339c0]c000000000068114.notifier_call_chain+0x68/0x9c [c0000001ad033a50]c0000000000789fc.cpu_down+0x1fc/0x368 [c0000001ad033b40]c0000000002ac658.store_online+0x88/0xe8 [c0000001ad033bd0]c0000000002a6f14.sysdev_store+0x4c/0x68 [c0000001ad033c50]c000000000110368.sysfs_write_file+0x100/0x1a0 [c0000001ad033cf0]c0000000000be854.vfs_write+0x100/0x200 [c0000001ad033d90]c0000000000bea64.sys_write+0x54/0x9c [c0000001ad033e30]c000000000008600syscall_exit+0x0/0x18--- Exception: c01 (System Call) at000000000fe5ec10
SP (ffc4c4f0) is in userspace0:mon> e
cpu0x0: Vector: 300 (Data Access) at [c0000001ad033520]Quoted 11 lines
pc: c00000000048bd30: ._spin_lock+0x18/0x80 lr: c000000000096a7c: .kfree+0x250/0x280 sp: c0000001ad0337a0 msr: 8000000000001032 dar: 48 dsisr: 40000000 current = 0xc0000001aff12040 paca = 0xc0000000005c1000 pid = 17376, comm = bashQuoted 3 lines
Sonny,
Does this patch fix the issue? This one applies cleanly on 2.6.15-rc6
unlike the one that was sent to you earlier.Hi, thanks, now I'm getting a slightly different error, hitting a BUG in the slab debug code:
ihplus:~ # echo 0 >
/sys/devices/system/cpu/cpu14/onlinecpu0x4: Vector: 700 (Program Check) at [c0000003a8c233f0]pc: c00000000009bb2c: .check_slabp+0x130/0x188 lr: c00000000009bb28: .check_slabp+0x12c/0x188 sp: c0000003a8c23670 msr: 8000000000021032 current = 0xc0000001b95297f0 paca = 0xc0000000005d7000 pid = 11116, comm = bashkernel BUG in
check_slabpatmm/slab.c:2368!
enter ? for help
Quoted 14 lines
4:mon> t
[c0000003a8c23700]c00000000009d918.free_block+0x168/0x294 [c0000003a8c237e0]c00000000009d1dc.kfree+0x2b8/0x2d4 [c0000003a8c238a0]c0000000000a1644.cpuup_callback+0x144/0x618 [c0000003a8c239b0]c0000000004a0780.notifier_call_chain+0x68/0x9c [c0000003a8c23a40]c00000000007d608.cpu_down+0x1fc/0x358 [c0000003a8c23b30]c0000000002bb4ec.store_online+0x88/0xe8 [c0000003a8c23bc0]c0000000002b5c14.sysdev_store+0x4c/0x68 [c0000003a8c23c40]c000000000119c6c.sysfs_write_file+0x118/0x1bc [c0000003a8c23cf0]c0000000000c6078.vfs_write+0x100/0x200 [c0000003a8c23d90]c0000000000c6288.sys_write+0x54/0x9c [c0000003a8c23e30]c000000000008600syscall_exit+0x0/0x18--- Exception: c01 (System Call) at000000000fe5ec10
SP (ff865560) is in userspace
More details:
The above crash was with SMT on, and I had already off-lined the SMT sibling thread.
When I boot with SMT off, I get a slightly different crash:
ihplus:~ # echo 0 > /sys/devices/system/cpu/cpu14/online cpu 0x0: Vector: 700 (Program Check) at [c0000003afa13480]
pc: c00000000009d960: .free_block+0x1b0/0x294
lr: c00000000009d95c: .free_block+0x1ac/0x294
sp: c0000003afa13700
msr: 8000000000021032
current = 0xc0000003afe04000
paca = 0xc0000000005d5000
pid = 10998, comm = bashkernel BUG in free_block at mm/slab.c:2664!
enter ? for help
0:mon> t
[c0000003afa137e0] c00000000009d1dc .kfree+0x2b8/0x2d4 [c0000003afa138a0] c0000000000a1644 .cpuup_callback+0x144/0x618 [c0000003afa139b0] c0000000004a0780 .notifier_call_chain+0x68/0x9c [c0000003afa13a40] c00000000007d608 .cpu_down+0x1fc/0x358 [c0000003afa13b30] c0000000002bb4ec .store_online+0x88/0xe8 [c0000003afa13bc0] c0000000002b5c14 .sysdev_store+0x4c/0x68 [c0000003afa13c40] c000000000119c6c .sysfs_write_file+0x118/0x1bc [c0000003afa13cf0] c0000000000c6078 .vfs_write+0x100/0x200 [c0000003afa13d90] c0000000000c6288 .sys_write+0x54/0x9c [c0000003afa13e30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ff8b4560) is in userspace
This one points to a double free somewhere
Sonny
Quoted 40 lines · Sonny Rao
Quoted 24 lines · Sonny Rao
Quoted 6 lines · Ravikiran G Thirumalai
Quoted 1 line · Sonny Rao
(apologies if this is a dup)
...
Sonny,
Does this patch fix the issue? This one applies cleanly on 2.6.15-rc6
unlike the one that was sent to you earlier.Hi, thanks, now I'm getting a slightly different error, hitting a BUG in the slab debug code:
ihplus:~ # echo 0 >
/sys/devices/system/cpu/cpu14/onlinecpu0x4: Vector: 700 (Program Check) at [c0000003a8c233f0]pc: c00000000009bb2c: .check_slabp+0x130/0x188 lr: c00000000009bb28: .check_slabp+0x12c/0x188 sp: c0000003a8c23670 msr: 8000000000021032 current = 0xc0000001b95297f0 paca = 0xc0000000005d7000 pid = 11116, comm = bashkernel BUG in
check_slabpatmm/slab.c:2368!
enter ? for helpQuoted 14 lines
4:mon> t
[c0000003a8c23700]c00000000009d918.free_block+0x168/0x294 [c0000003a8c237e0]c00000000009d1dc.kfree+0x2b8/0x2d4 [c0000003a8c238a0]c0000000000a1644.cpuup_callback+0x144/0x618 [c0000003a8c239b0]c0000000004a0780.notifier_call_chain+0x68/0x9c [c0000003a8c23a40]c00000000007d608.cpu_down+0x1fc/0x358 [c0000003a8c23b30]c0000000002bb4ec.store_online+0x88/0xe8 [c0000003a8c23bc0]c0000000002b5c14.sysdev_store+0x4c/0x68 [c0000003a8c23c40]c000000000119c6c.sysfs_write_file+0x118/0x1bc [c0000003a8c23cf0]c0000000000c6078.vfs_write+0x100/0x200 [c0000003a8c23d90]c0000000000c6288.sys_write+0x54/0x9c [c0000003a8c23e30]c000000000008600syscall_exit+0x0/0x18--- Exception: c01 (System Call) at000000000fe5ec10
SP (ff865560) is in userspace
Quoted 6 lines
More details:
The above crash was with SMT on, and I had already off-lined the SMT sibling thread.
When I boot with SMT off, I get a slightly different crash:
I think i missed the first reply above. (I can't seem to find it on lkml either). So just to confirm, both these crashes are with the new patch on top of rc6?
Thanks,
Kiran
Quoted 30 lines
ihplus:~ # echo 0 >
/sys/devices/system/cpu/cpu14/onlinecpu0x0: Vector: 700 (Program Check) at [c0000003afa13480]pc: c00000000009d960: .free_block+0x1b0/0x294 lr: c00000000009d95c: .free_block+0x1ac/0x294 sp: c0000003afa13700 msr: 8000000000021032 current = 0xc0000003afe04000 paca = 0xc0000000005d5000 pid = 10998, comm = bashkernel BUG in
free_blockatmm/slab.c:2664!
enter ? for help0:mon> t
[c0000003afa137e0]c00000000009d1dc.kfree+0x2b8/0x2d4 [c0000003afa138a0]c0000000000a1644.cpuup_callback+0x144/0x618 [c0000003afa139b0]c0000000004a0780.notifier_call_chain+0x68/0x9c [c0000003afa13a40]c00000000007d608.cpu_down+0x1fc/0x358 [c0000003afa13b30]c0000000002bb4ec.store_online+0x88/0xe8 [c0000003afa13bc0]c0000000002b5c14.sysdev_store+0x4c/0x68 [c0000003afa13c40]c000000000119c6c.sysfs_write_file+0x118/0x1bc [c0000003afa13cf0]c0000000000c6078.vfs_write+0x100/0x200 [c0000003afa13d90]c0000000000c6288.sys_write+0x54/0x9c [c0000003afa13e30]c000000000008600syscall_exit+0x0/0x18--- Exception: c01 (System Call) at000000000fe5ec10
SP (ff8b4560) is in userspaceThis one points to a double free somewhere
Quoted 43 lines · Ravikiran G Thirumalai
Quoted 42 lines · Sonny Rao
Quoted 24 lines · Sonny Rao
Quoted 6 lines · Ravikiran G Thirumalai
Quoted 1 line · Sonny Rao
(apologies if this is a dup)
...
Sonny,
Does this patch fix the issue? This one applies cleanly on 2.6.15-rc6
unlike the one that was sent to you earlier.Hi, thanks, now I'm getting a slightly different error, hitting a BUG in the slab debug code:
ihplus:~ # echo 0 >
/sys/devices/system/cpu/cpu14/onlinecpu0x4: Vector: 700 (Program Check) at [c0000003a8c233f0]pc: c00000000009bb2c: .check_slabp+0x130/0x188 lr: c00000000009bb28: .check_slabp+0x12c/0x188 sp: c0000003a8c23670 msr: 8000000000021032 current = 0xc0000001b95297f0 paca = 0xc0000000005d7000 pid = 11116, comm = bashkernel BUG in
check_slabpatmm/slab.c:2368!
enter ? for helpQuoted 14 lines
4:mon> t
[c0000003a8c23700]c00000000009d918.free_block+0x168/0x294 [c0000003a8c237e0]c00000000009d1dc.kfree+0x2b8/0x2d4 [c0000003a8c238a0]c0000000000a1644.cpuup_callback+0x144/0x618 [c0000003a8c239b0]c0000000004a0780.notifier_call_chain+0x68/0x9c [c0000003a8c23a40]c00000000007d608.cpu_down+0x1fc/0x358 [c0000003a8c23b30]c0000000002bb4ec.store_online+0x88/0xe8 [c0000003a8c23bc0]c0000000002b5c14.sysdev_store+0x4c/0x68 [c0000003a8c23c40]c000000000119c6c.sysfs_write_file+0x118/0x1bc [c0000003a8c23cf0]c0000000000c6078.vfs_write+0x100/0x200 [c0000003a8c23d90]c0000000000c6288.sys_write+0x54/0x9c [c0000003a8c23e30]c000000000008600syscall_exit+0x0/0x18--- Exception: c01 (System Call) at000000000fe5ec10
SP (ff865560) is in userspaceMore details:
Quoted 8 lines
Quoted 4 lines
The above crash was with SMT on, and I had already off-lined the SMT sibling thread.
When I boot with SMT off, I get a slightly different crash:
I think i missed the first reply above. (I can't seem to find it on lkml either). So just to confirm, both these crashes are with the new patch on top of rc6?
Yes, rc6 + the patch you provided.
The stupid mail relay server I'm using for my ibm account seems to be very lethargic, sorry about that.
Sonny
Quoted 1 line · Sonny Rao
Yes, rc6 + the patch you provided.
We may be going down the wrong path here. Has someone else than Sonny reproduced the problem?
Quoted 6 lines · Christoph Lameter
Quoted 1 line · Sonny Rao
Yes, rc6 + the patch you provided.
We may be going down the wrong path here. Has someone else than Sonny reproduced the problem?
Hi, I've also just reproduced the problem on another machine which does have multiple cpus/node rather than just one cpu/node. The crash occurs at the same place when I attempt to offline the last cpu in a node.
But, I agree that somemone else should repro this. I only have ppc64 machines available to me right now.
Sonny
Quoted 46 lines · Ravikiran G Thirumalai
Quoted 45 lines · Sonny Rao
Quoted 24 lines · Sonny Rao
Quoted 6 lines · Ravikiran G Thirumalai
Quoted 1 line · Sonny Rao
(apologies if this is a dup)
...
Sonny,
Does this patch fix the issue? This one applies cleanly on 2.6.15-rc6
unlike the one that was sent to you earlier.Hi, thanks, now I'm getting a slightly different error, hitting a BUG in the slab debug code:
ihplus:~ # echo 0 >
/sys/devices/system/cpu/cpu14/onlinecpu0x4: Vector: 700 (Program Check) at [c0000003a8c233f0]pc: c00000000009bb2c: .check_slabp+0x130/0x188 lr: c00000000009bb28: .check_slabp+0x12c/0x188 sp: c0000003a8c23670 msr: 8000000000021032 current = 0xc0000001b95297f0 paca = 0xc0000000005d7000 pid = 11116, comm = bashkernel BUG in
check_slabpatmm/slab.c:2368!
enter ? for helpQuoted 14 lines
4:mon> t
[c0000003a8c23700]c00000000009d918.free_block+0x168/0x294 [c0000003a8c237e0]c00000000009d1dc.kfree+0x2b8/0x2d4 [c0000003a8c238a0]c0000000000a1644.cpuup_callback+0x144/0x618 [c0000003a8c239b0]c0000000004a0780.notifier_call_chain+0x68/0x9c [c0000003a8c23a40]c00000000007d608.cpu_down+0x1fc/0x358 [c0000003a8c23b30]c0000000002bb4ec.store_online+0x88/0xe8 [c0000003a8c23bc0]c0000000002b5c14.sysdev_store+0x4c/0x68 [c0000003a8c23c40]c000000000119c6c.sysfs_write_file+0x118/0x1bc [c0000003a8c23cf0]c0000000000c6078.vfs_write+0x100/0x200 [c0000003a8c23d90]c0000000000c6288.sys_write+0x54/0x9c [c0000003a8c23e30]c000000000008600syscall_exit+0x0/0x18--- Exception: c01 (System Call) at000000000fe5ec10
SP (ff865560) is in userspaceMore details:
The above crash was with SMT on, and I had already off-lined the SMT sibling thread.
Quoted 37 lines
Quoted 1 line
When I boot with SMT off, I get a slightly different crash:
I think i missed the first reply above. (I can't seem to find it on lkml either). So just to confirm, both these crashes are with the new patch on top of rc6?
Thanks,
KiranQuoted 28 lines
ihplus:~ # echo 0 >
/sys/devices/system/cpu/cpu14/onlinecpu0x0: Vector: 700 (Program Check) at [c0000003afa13480]pc: c00000000009d960: .free_block+0x1b0/0x294 lr: c00000000009d95c: .free_block+0x1ac/0x294 sp: c0000003afa13700 msr: 8000000000021032 current = 0xc0000003afe04000 paca = 0xc0000000005d5000 pid = 10998, comm = bashkernel BUG in
free_blockatmm/slab.c:2664!
enter ? for help0:mon> t
[c0000003afa137e0]c00000000009d1dc.kfree+0x2b8/0x2d4 [c0000003afa138a0]c0000000000a1644.cpuup_callback+0x144/0x618 [c0000003afa139b0]c0000000004a0780.notifier_call_chain+0x68/0x9c [c0000003afa13a40]c00000000007d608.cpu_down+0x1fc/0x358 [c0000003afa13b30]c0000000002bb4ec.store_online+0x88/0xe8 [c0000003afa13bc0]c0000000002b5c14.sysdev_store+0x4c/0x68 [c0000003afa13c40]c000000000119c6c.sysfs_write_file+0x118/0x1bc [c0000003afa13cf0]c0000000000c6078.vfs_write+0x100/0x200 [c0000003afa13d90]c0000000000c6288.sys_write+0x54/0x9c [c0000003afa13e30]c000000000008600syscall_exit+0x0/0x18--- Exception: c01 (System Call) at000000000fe5ec10
SP (ff8b4560) is in userspaceThis one points to a double free somewhere
Hi, I think I've found the double free in the rc6 kernel + your patch :
starting on line 949 of the patched slab.c
if ((shared = l3->shared)) {
free_block(cachep, l3->shared->entry,
l3->shared->avail, node);
kfree(l3->shared);
l3->shared = NULL;
}
alien = l3->alien;
l3->alien = NULL;
spin_unlock(&l3->list_lock);
kfree(nc);
kfree(shared);You conditionally free l3->shared after assigning it to the auto var "shared" then below that you call kfree on "shared" again == double free.
So, I got rid of the extra free. I don't know if this was correct but I tried it anyway. Unfortunately this still does not work correctly. The system hangs for a period of time and then drops into the debugger again:
0:mon> t
[c00000000f71f890] c00000000049e5ec ._spin_lock+0x10/0x24 [c00000000f71f910] c00000000009d550 .kmem_cache_free+0x270/0x2a4 [c00000000f71f9d0] c0000000003f35e8 .kfree_skbmem+0xa0/0xfc [c00000000f71fa50] c00000000044d01c .udp_rcv+0x7ac/0x818 [c00000000f71fb60] c000000000420b14 .ip_local_deliver+0xf8/0x3f0 [c00000000f71fbf0] c000000000420328 .ip_rcv+0x3a8/0x724 [c00000000f71fc90] c0000000003fa054 .netif_receive_skb+0x378/0x3d0 [c00000000f71fd30] c0000000003fa1c4 .process_backlog+0x118/0x254 [c00000000f71fe10] c0000000003f7d3c .net_rx_action+0x188/0x2b8 [c00000000f71fed0] c000000000060f18 .__do_softirq+0xd4/0x1b8 [c00000000f71ff90] c00000000002c78c .call_do_softirq+0x14/0x24 [c0000000005ab870] c00000000000bd30 .do_softirq+0x8c/0x9c [c0000000005ab900] c00000000006143c .irq_exit+0x6c/0x84 [c0000000005ab980] c00000000000c060 .do_IRQ+0xe8/0x194 [c0000000005aba10] c000000000004134 hardware_interrupt_entry+0x8/0x54 --- Exception: 501 (Hardware Interrupt) at c000000000040670
.pseries_dedicated_idle+0x114/0x268 [c0000000005abde0] c000000000021048 .cpu_idle+0x4c/0x60 [c0000000005abe50] c0000000000091f4 .rest_init+0x44/0x5c [c0000000005abed0] c00000000054e7f4 .start_kernel+0x29c/0x318 [c0000000005abf90] c000000000008494 .hmt_init+0x0/0x6c
0:mon>
0:mon> e
cpu 0x0: Vector: 300 (Data Access) at [c00000000f71f580]
pc: c000000000238db4: ._raw_spin_lock+0x2c/0x1d0
lr: c00000000049e5ec: ._spin_lock+0x10/0x24
sp: c00000000f71f800 msr: 8000000000001032
dar: 4c
dsisr: 40000000
current = 0xc00000000061b2f0
paca = 0xc0000000005d5000
pid = 0, comm = swapper
0:mon>
I wonder if this is related to the problem Sonny is seeing -- powerpc's definitions of cpu_to_node et al. are not being used. The culprit is some too-clever preprocessor usage in asm-generic/topology.h, for example:
#ifndef cpu_to_node
#define cpu_to_node(cpu) (0)
#endif
But asm-powerpc/topology.h has cpu_to_node defined as a static inline (which does not make it a preprocessor symbol), so we get the generic
- and incorrect - definition.
Does removing the #include of asm-generic/topology.h from the bottom of asm-powerpc/topology.h have any effect?
Quoted 10 lines · Nathan Lynch
I wonder if this is related to the problem Sonny is seeing -- powerpc's definitions of
cpu_to_nodeet al. are not being used. The culprit is some too-clever preprocessor usage inasm-generic/topology.h, for example:#ifndef
cpu_to_node
#definecpu_to_node(cpu)(0)
#endif
Quoted 6 lines
But
asm-powerpc/topology.hhascpu_to_nodedefined as a static inline (which does not make it a preprocessor symbol), so we get the generic
- and incorrect - definition.Does removing the #include of
asm-generic/topology.hfrom the bottom ofasm-powerpc/topology.hhave any effect?
Hi, no it doesn't make a difference. That include is protected by CONFIG_NUMA as well, so it never gets hit. At Anton's suggestion I even put in an #error into asm-generic/topology.h to make sure it wasn't an issue -- it didn't hit.
Sonny
Quoted 9 lines · Sonny Rao
Quoted 3 lines · Nathan Lynch
Does removing the #include of
asm-generic/topology.hfrom the bottom ofasm-powerpc/topology.hhave any effect?Hi, no it doesn't make a difference. That include is protected by
CONFIG_NUMAas well, so it never gets hit. At Anton's suggestion I even put in an #error intoasm-generic/topology.hto make sure it wasn't an issue -- it didn't hit.
Gah, sorry, forgot Anton fixed this a while back.