Skip to contentSkip to conversation

cpu hotplug oops on 2.6.15-rc5

DISCUSSION Sonny Rao 7 participants 16 messages Dec 19, 2005 – Dec 29, 2005 1 message not in the archive Memory management

lore
Share & tools
lore

Sonny Rao

author
#0 lore raw

(apologies if this is a dup)

Hi, I'm crashing 2.6.15-rc5 when I try and offline the last and only CPU in a node on a ppc64 Power5, SMT was disabled.

Here's the backtrace:

0:mon> t
[c0000001ad033820] c000000000096a7c .kfree+0x250/0x280 [c0000001ad0338d0] c00000000009a544 .cpuup_callback+0x238/0x5fc [c0000001ad0339c0] c000000000068114 .notifier_call_chain+0x68/0x9c [c0000001ad033a50] c0000000000789fc .cpu_down+0x1fc/0x368 [c0000001ad033b40] c0000000002ac658 .store_online+0x88/0xe8 [c0000001ad033bd0] c0000000002a6f14 .sysdev_store+0x4c/0x68 [c0000001ad033c50] c000000000110368 .sysfs_write_file+0x100/0x1a0 [c0000001ad033cf0] c0000000000be854 .vfs_write+0x100/0x200 [c0000001ad033d90] c0000000000bea64 .sys_write+0x54/0x9c [c0000001ad033e30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ffc4c4f0) is in userspace

0:mon> e
cpu 0x0: Vector: 300 (Data Access) at [c0000001ad033520]

    pc: c00000000048bd30: ._spin_lock+0x18/0x80
    lr: c000000000096a7c: .kfree+0x250/0x280
    sp: c0000001ad0337a0
   msr: 8000000000001032
   dar: 48
 dsisr: 40000000
  current = 0xc0000001aff12040
  paca    = 0xc0000000005c1000
    pid   = 17376, comm = bash

Should I try this with CONFIG_DEBUG_SLAB ?

Sonny

#1 lore raw
Quoted 3 lines · Sonny Rao

(apologies if this is a dup)

Hi, I'm crashing 2.6.15-rc5 when I try and offline the last and only CPU in a node on a ppc64 Power5, SMT was disabled.

First try on -rc6 just in case it's related to the SCSI fix (the bug was corrupting the SLAB) that got merged just after rc5 iirc.

Ben.

author
#2 lore raw
Quoted 7 lines · Benjamin Herrenschmidt
Quoted 3 lines · Sonny Rao

(apologies if this is a dup)

Hi, I'm crashing 2.6.15-rc5 when I try and offline the last and only CPU in a node on a ppc64 Power5, SMT was disabled.

First try on -rc6 just in case it's related to the SCSI fix (the bug was corrupting the SLAB) that got merged just after rc5 iirc.

Ok, tried it: same crash on -rc6

2:mon> t
[c000000d9f33b820] c000000000097cd0 .kfree+0x29c/0x2cc [c000000d9f33b8d0] c00000000009c3a8 .cpuup_callback+0x4f8/0x5fc [c000000d9f33b9c0] c00000000048ff4c .notifier_call_chain+0x68/0x9c [c000000d9f33ba50] c000000000078da8 .cpu_down+0x1fc/0x368 [c000000d9f33bb40] c0000000002ae514 .store_online+0x88/0xe8 [c000000d9f33bbd0] c0000000002a8dd0 .sysdev_store+0x4c/0x68 [c000000d9f33bc50] c000000000111e70 .sysfs_write_file+0x100/0x1a0 [c000000d9f33bcf0] c0000000000c0360 .vfs_write+0x100/0x200 [c000000d9f33bd90] c0000000000c0570 .sys_write+0x54/0x9c [c000000d9f33be30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ffa204f0) is in userspace
2:mon>

Sonny

#3 lore raw
Quoted 15 lines · Sonny Rao

Ok, tried it: same crash on -rc6

2:mon> t
[c000000d9f33b820] c000000000097cd0 .kfree+0x29c/0x2cc [c000000d9f33b8d0] c00000000009c3a8 .cpuup_callback+0x4f8/0x5fc [c000000d9f33b9c0] c00000000048ff4c .notifier_call_chain+0x68/0x9c [c000000d9f33ba50] c000000000078da8 .cpu_down+0x1fc/0x368 [c000000d9f33bb40] c0000000002ae514 .store_online+0x88/0xe8 [c000000d9f33bbd0] c0000000002a8dd0 .sysdev_store+0x4c/0x68 [c000000d9f33bc50] c000000000111e70 .sysfs_write_file+0x100/0x1a0 [c000000d9f33bcf0] c0000000000c0360 .vfs_write+0x100/0x200 [c000000d9f33bd90] c0000000000c0570 .sys_write+0x54/0x9c [c000000d9f33be30] c000000000008600 syscall_exit+0x0/0x18

Very odd call chain.
Could you enable slab debugging?

--
Manfred

author
#4 lore raw

Re: SPAMHAUS-Re: cpu hotplug oops on 2.6.15-rc5

Quoted 17 lines · Manfred Spraul
Quoted 15 lines · Sonny Rao

Ok, tried it: same crash on -rc6

2:mon> t
[c000000d9f33b820] c000000000097cd0 .kfree+0x29c/0x2cc [c000000d9f33b8d0] c00000000009c3a8 .cpuup_callback+0x4f8/0x5fc [c000000d9f33b9c0] c00000000048ff4c .notifier_call_chain+0x68/0x9c [c000000d9f33ba50] c000000000078da8 .cpu_down+0x1fc/0x368 [c000000d9f33bb40] c0000000002ae514 .store_online+0x88/0xe8 [c000000d9f33bbd0] c0000000002a8dd0 .sysdev_store+0x4c/0x68 [c000000d9f33bc50] c000000000111e70 .sysfs_write_file+0x100/0x1a0 [c000000d9f33bcf0] c0000000000c0360 .vfs_write+0x100/0x200 [c000000d9f33bd90] c0000000000c0570 .sys_write+0x54/0x9c [c000000d9f33be30] c000000000008600 syscall_exit+0x0/0x18

Quoted 2 lines

Very odd call chain.
Could you enable slab debugging?

Actually, I did turn on slab debugging on -rc6, but it did not seem to make any difference.

Sonny

#5 lore raw
In reply to Manfred Spraul

Hi Manfred,

Quoted 2 lines

Very odd call chain.
Could you enable slab debugging?

Sonny and I had a look around, it seems to be in the cpuup_callback() / CPU_DEAD case:

      if (!cpus_empty(mask)) {
              spin_unlock(&l3->list_lock);
              goto unlock_cache;
      }

      if (l3->shared) {
              free_block(cachep, l3->shared->entry,
                              l3->shared->avail, node);
              kfree(l3->shared);                <-------- HERE
              l3->shared = NULL;
      }

So we are removing the last cpu in a node, and tearing down the node related structures. We looked at kfree() -> __cache_free() and we couldnt convince ourselves that all the CONFIG_NUMA stuff in there wouldnt trip over itself (since we would be doing the free on an alien node).

Anton

#6 lore raw
In reply to Sonny Rao
Quoted 22 lines · Sonny Rao

(apologies if this is a dup)

Hi, I'm crashing 2.6.15-rc5 when I try and offline the last and only CPU in a node on a ppc64 Power5, SMT was disabled.

Here's the backtrace:

0:mon> t
[c0000001ad033820] c000000000096a7c .kfree+0x250/0x280 [c0000001ad0338d0] c00000000009a544 .cpuup_callback+0x238/0x5fc [c0000001ad0339c0] c000000000068114 .notifier_call_chain+0x68/0x9c [c0000001ad033a50] c0000000000789fc .cpu_down+0x1fc/0x368 [c0000001ad033b40] c0000000002ac658 .store_online+0x88/0xe8 [c0000001ad033bd0] c0000000002a6f14 .sysdev_store+0x4c/0x68 [c0000001ad033c50] c000000000110368 .sysfs_write_file+0x100/0x1a0 [c0000001ad033cf0] c0000000000be854 .vfs_write+0x100/0x200 [c0000001ad033d90] c0000000000bea64 .sys_write+0x54/0x9c [c0000001ad033e30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ffc4c4f0) is in userspace

0:mon> e
cpu 0x0: Vector: 300 (Data Access) at [c0000001ad033520]

Quoted 11 lines
    pc: c00000000048bd30: ._spin_lock+0x18/0x80
    lr: c000000000096a7c: .kfree+0x250/0x280
    sp: c0000001ad0337a0
   msr: 8000000000001032
   dar: 48
 dsisr: 40000000
  current = 0xc0000001aff12040
  paca    = 0xc0000000005c1000
    pid   = 17376, comm = bash

Sonny,
Does this patch fix the issue? This one applies cleanly on 2.6.15-rc6
unlike the one that was sent to you earlier.

Thanks,
Kiran

From: Alok N Kataria <[email protected]>

Fixes a bug in the CPU_DOWN call path, we shouldn't call kfree while holding kmem_list3's list lock, nor should drain_alien_cache be called with l3's list lock.

mm/slab.c +24 -16
Index: linux-2.6.15-rc6/mm/slab.c
===================================================================
@@ -824,14 +824,14 @@ static inline void __drain_alien_cache(k
 	}
 }
 
-static void drain_alien_cache(kmem_cache_t *cachep, struct kmem_list3 *l3)
+static void drain_alien_cache(kmem_cache_t *cachep, struct array_cache **alien)
 {
 	int i=0;
 	struct array_cache *ac;
 	unsigned long flags;
 
 	for_each_online_node(i) {
-		ac = l3->alien[i];
+		ac = alien[i];
 		if (ac) {
 			spin_lock_irqsave(&ac->lock, flags);
 			__drain_alien_cache(cachep, ac, i);
@@ -842,7 +842,7 @@ static void drain_alien_cache(kmem_cache
 #else
 #define alloc_alien_cache(node, limit) do { } while (0)
 #define free_alien_cache(ac_ptr) do { } while (0)
-#define drain_alien_cache(cachep, l3) do { } while (0)
+#define drain_alien_cache(cachep, alien) do { } while (0)
 #endif
 
 static int __devinit cpuup_callback(struct notifier_block *nfb,
@@ -921,7 +921,7 @@ static int __devinit cpuup_callback(stru
 		down(&cache_chain_sem);
 
 		list_for_each_entry(cachep, &cache_chain, next) {
-			struct array_cache *nc;
+			struct array_cache *nc, *shared, **alien;
 			cpumask_t mask;
 
 			mask = node_to_cpumask(node);
@@ -932,7 +932,7 @@ static int __devinit cpuup_callback(stru
 			l3 = cachep->nodelists[node];
 
 			if (!l3)
-				goto unlock_cache;
+				goto free_array_cache;
 
 			spin_lock(&l3->list_lock);
 
@@ -943,32 +943,40 @@ static int __devinit cpuup_callback(stru
 
 			if (!cpus_empty(mask)) {
                                 spin_unlock(&l3->list_lock);
-                                goto unlock_cache;
+                                goto free_array_cache;
                         }
 
-			if (l3->shared) {
+			if ((shared = l3->shared)) {
 				free_block(cachep, l3->shared->entry,
 						l3->shared->avail, node);
 				kfree(l3->shared);
 				l3->shared = NULL;
 			}
-			if (l3->alien) {
-				drain_alien_cache(cachep, l3);
-				free_alien_cache(l3->alien);
-				l3->alien = NULL;
+
+			alien = l3->alien;
+			l3->alien = NULL;
+
+			spin_unlock(&l3->list_lock);
+
+			kfree(nc);
+			kfree(shared);
+			if (alien) {
+				drain_alien_cache(cachep, alien);
+				free_alien_cache(alien);
 			}
 
 			/* free slabs belonging to this node */
 			if (__node_shrink(cachep, node)) {
+				spin_lock(&l3->list_lock);
 				cachep->nodelists[node] = NULL;
 				spin_unlock(&l3->list_lock);
 				kfree(l3);
-			} else {
-				spin_unlock(&l3->list_lock);
 			}
+			goto unlock_cache;
+free_array_cache:
+			kfree(nc);
 unlock_cache:
 			spin_unlock_irq(&cachep->spinlock);
-			kfree(nc);
 		}
 		up(&cache_chain_sem);
 		break;
@@ -1918,7 +1926,7 @@ static void drain_cpu_caches(kmem_cache_
 			drain_array_locked(cachep, l3->shared, 1, node);
 			spin_unlock(&l3->list_lock);
 			if (l3->alien)
-				drain_alien_cache(cachep, l3);
+				drain_alien_cache(cachep, l3->alien);
 		}
 	}
 	spin_unlock_irq(&cachep->spinlock);
@@ -3310,7 +3318,7 @@ static void cache_reap(void *unused)
 
 		l3 = searchp->nodelists[numa_node_id()];
 		if (l3->alien)
-			drain_alien_cache(searchp, l3);
+			drain_alien_cache(searchp, l3->alien);
 		spin_lock_irq(&l3->list_lock);
 
 		drain_array_locked(searchp, ac_data(searchp), 0,

A message that is not in the archive (sent off-list or not yet received): [email protected]

author
#8 lore raw
Quoted 56 lines · Sonny Rao
Quoted 35 lines · Ravikiran G Thirumalai
Quoted 22 lines · Sonny Rao

(apologies if this is a dup)

Hi, I'm crashing 2.6.15-rc5 when I try and offline the last and only CPU in a node on a ppc64 Power5, SMT was disabled.

Here's the backtrace:

0:mon> t
[c0000001ad033820] c000000000096a7c .kfree+0x250/0x280 [c0000001ad0338d0] c00000000009a544 .cpuup_callback+0x238/0x5fc [c0000001ad0339c0] c000000000068114 .notifier_call_chain+0x68/0x9c [c0000001ad033a50] c0000000000789fc .cpu_down+0x1fc/0x368 [c0000001ad033b40] c0000000002ac658 .store_online+0x88/0xe8 [c0000001ad033bd0] c0000000002a6f14 .sysdev_store+0x4c/0x68 [c0000001ad033c50] c000000000110368 .sysfs_write_file+0x100/0x1a0 [c0000001ad033cf0] c0000000000be854 .vfs_write+0x100/0x200 [c0000001ad033d90] c0000000000bea64 .sys_write+0x54/0x9c [c0000001ad033e30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ffc4c4f0) is in userspace

0:mon> e
cpu 0x0: Vector: 300 (Data Access) at [c0000001ad033520]

Quoted 11 lines
    pc: c00000000048bd30: ._spin_lock+0x18/0x80
    lr: c000000000096a7c: .kfree+0x250/0x280
    sp: c0000001ad0337a0
   msr: 8000000000001032
   dar: 48
 dsisr: 40000000
  current = 0xc0000001aff12040
  paca    = 0xc0000000005c1000
    pid   = 17376, comm = bash
Quoted 3 lines

Sonny,
Does this patch fix the issue? This one applies cleanly on 2.6.15-rc6
unlike the one that was sent to you earlier.

Hi, thanks, now I'm getting a slightly different error, hitting a BUG in the slab debug code:

ihplus:~ # echo 0 > /sys/devices/system/cpu/cpu14/online cpu 0x4: Vector: 700 (Program Check) at [c0000003a8c233f0]

    pc: c00000000009bb2c: .check_slabp+0x130/0x188
    lr: c00000000009bb28: .check_slabp+0x12c/0x188
    sp: c0000003a8c23670
   msr: 8000000000021032
  current = 0xc0000001b95297f0
  paca    = 0xc0000000005d7000
    pid   = 11116, comm = bash

kernel BUG in check_slabp at mm/slab.c:2368!
enter ? for help

Quoted 14 lines

4:mon> t
[c0000003a8c23700] c00000000009d918 .free_block+0x168/0x294 [c0000003a8c237e0] c00000000009d1dc .kfree+0x2b8/0x2d4 [c0000003a8c238a0] c0000000000a1644 .cpuup_callback+0x144/0x618 [c0000003a8c239b0] c0000000004a0780 .notifier_call_chain+0x68/0x9c [c0000003a8c23a40] c00000000007d608 .cpu_down+0x1fc/0x358 [c0000003a8c23b30] c0000000002bb4ec .store_online+0x88/0xe8 [c0000003a8c23bc0] c0000000002b5c14 .sysdev_store+0x4c/0x68 [c0000003a8c23c40] c000000000119c6c .sysfs_write_file+0x118/0x1bc [c0000003a8c23cf0] c0000000000c6078 .vfs_write+0x100/0x200 [c0000003a8c23d90] c0000000000c6288 .sys_write+0x54/0x9c [c0000003a8c23e30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ff865560) is in userspace

More details:

The above crash was with SMT on, and I had already off-lined the SMT sibling thread.

When I boot with SMT off, I get a slightly different crash:

ihplus:~ # echo 0 > /sys/devices/system/cpu/cpu14/online cpu 0x0: Vector: 700 (Program Check) at [c0000003afa13480]

    pc: c00000000009d960: .free_block+0x1b0/0x294
    lr: c00000000009d95c: .free_block+0x1ac/0x294
    sp: c0000003afa13700
   msr: 8000000000021032
  current = 0xc0000003afe04000
  paca    = 0xc0000000005d5000
    pid   = 10998, comm = bash

kernel BUG in free_block at mm/slab.c:2664!
enter ? for help

0:mon> t
[c0000003afa137e0] c00000000009d1dc .kfree+0x2b8/0x2d4 [c0000003afa138a0] c0000000000a1644 .cpuup_callback+0x144/0x618 [c0000003afa139b0] c0000000004a0780 .notifier_call_chain+0x68/0x9c [c0000003afa13a40] c00000000007d608 .cpu_down+0x1fc/0x358 [c0000003afa13b30] c0000000002bb4ec .store_online+0x88/0xe8 [c0000003afa13bc0] c0000000002b5c14 .sysdev_store+0x4c/0x68 [c0000003afa13c40] c000000000119c6c .sysfs_write_file+0x118/0x1bc [c0000003afa13cf0] c0000000000c6078 .vfs_write+0x100/0x200 [c0000003afa13d90] c0000000000c6288 .sys_write+0x54/0x9c [c0000003afa13e30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ff8b4560) is in userspace

This one points to a double free somewhere

Sonny

#9 lore raw
Quoted 40 lines · Sonny Rao
Quoted 24 lines · Sonny Rao
Quoted 6 lines · Ravikiran G Thirumalai
Quoted 1 line · Sonny Rao

(apologies if this is a dup)

...
Sonny,
Does this patch fix the issue? This one applies cleanly on 2.6.15-rc6
unlike the one that was sent to you earlier.

Hi, thanks, now I'm getting a slightly different error, hitting a BUG in the slab debug code:

ihplus:~ # echo 0 > /sys/devices/system/cpu/cpu14/online cpu 0x4: Vector: 700 (Program Check) at [c0000003a8c233f0]

    pc: c00000000009bb2c: .check_slabp+0x130/0x188
    lr: c00000000009bb28: .check_slabp+0x12c/0x188
    sp: c0000003a8c23670
   msr: 8000000000021032
  current = 0xc0000001b95297f0
  paca    = 0xc0000000005d7000
    pid   = 11116, comm = bash

kernel BUG in check_slabp at mm/slab.c:2368!
enter ? for help

Quoted 14 lines

4:mon> t
[c0000003a8c23700] c00000000009d918 .free_block+0x168/0x294 [c0000003a8c237e0] c00000000009d1dc .kfree+0x2b8/0x2d4 [c0000003a8c238a0] c0000000000a1644 .cpuup_callback+0x144/0x618 [c0000003a8c239b0] c0000000004a0780 .notifier_call_chain+0x68/0x9c [c0000003a8c23a40] c00000000007d608 .cpu_down+0x1fc/0x358 [c0000003a8c23b30] c0000000002bb4ec .store_online+0x88/0xe8 [c0000003a8c23bc0] c0000000002b5c14 .sysdev_store+0x4c/0x68 [c0000003a8c23c40] c000000000119c6c .sysfs_write_file+0x118/0x1bc [c0000003a8c23cf0] c0000000000c6078 .vfs_write+0x100/0x200 [c0000003a8c23d90] c0000000000c6288 .sys_write+0x54/0x9c [c0000003a8c23e30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ff865560) is in userspace

Quoted 6 lines

More details:

The above crash was with SMT on, and I had already off-lined the SMT sibling thread.

When I boot with SMT off, I get a slightly different crash:

I think i missed the first reply above. (I can't seem to find it on lkml either). So just to confirm, both these crashes are with the new patch on top of rc6?

Thanks,
Kiran

Quoted 30 lines

ihplus:~ # echo 0 > /sys/devices/system/cpu/cpu14/online cpu 0x0: Vector: 700 (Program Check) at [c0000003afa13480]

    pc: c00000000009d960: .free_block+0x1b0/0x294
    lr: c00000000009d95c: .free_block+0x1ac/0x294
    sp: c0000003afa13700
   msr: 8000000000021032
  current = 0xc0000003afe04000
  paca    = 0xc0000000005d5000
    pid   = 10998, comm = bash

kernel BUG in free_block at mm/slab.c:2664!
enter ? for help

0:mon> t
[c0000003afa137e0] c00000000009d1dc .kfree+0x2b8/0x2d4 [c0000003afa138a0] c0000000000a1644 .cpuup_callback+0x144/0x618 [c0000003afa139b0] c0000000004a0780 .notifier_call_chain+0x68/0x9c [c0000003afa13a40] c00000000007d608 .cpu_down+0x1fc/0x358 [c0000003afa13b30] c0000000002bb4ec .store_online+0x88/0xe8 [c0000003afa13bc0] c0000000002b5c14 .sysdev_store+0x4c/0x68 [c0000003afa13c40] c000000000119c6c .sysfs_write_file+0x118/0x1bc [c0000003afa13cf0] c0000000000c6078 .vfs_write+0x100/0x200 [c0000003afa13d90] c0000000000c6288 .sys_write+0x54/0x9c [c0000003afa13e30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ff8b4560) is in userspace

This one points to a double free somewhere

author
#10 lore raw
Quoted 43 lines · Ravikiran G Thirumalai
Quoted 42 lines · Sonny Rao
Quoted 24 lines · Sonny Rao
Quoted 6 lines · Ravikiran G Thirumalai
Quoted 1 line · Sonny Rao

(apologies if this is a dup)

...
Sonny,
Does this patch fix the issue? This one applies cleanly on 2.6.15-rc6
unlike the one that was sent to you earlier.

Hi, thanks, now I'm getting a slightly different error, hitting a BUG in the slab debug code:

ihplus:~ # echo 0 > /sys/devices/system/cpu/cpu14/online cpu 0x4: Vector: 700 (Program Check) at [c0000003a8c233f0]

    pc: c00000000009bb2c: .check_slabp+0x130/0x188
    lr: c00000000009bb28: .check_slabp+0x12c/0x188
    sp: c0000003a8c23670
   msr: 8000000000021032
  current = 0xc0000001b95297f0
  paca    = 0xc0000000005d7000
    pid   = 11116, comm = bash

kernel BUG in check_slabp at mm/slab.c:2368!
enter ? for help

Quoted 14 lines

4:mon> t
[c0000003a8c23700] c00000000009d918 .free_block+0x168/0x294 [c0000003a8c237e0] c00000000009d1dc .kfree+0x2b8/0x2d4 [c0000003a8c238a0] c0000000000a1644 .cpuup_callback+0x144/0x618 [c0000003a8c239b0] c0000000004a0780 .notifier_call_chain+0x68/0x9c [c0000003a8c23a40] c00000000007d608 .cpu_down+0x1fc/0x358 [c0000003a8c23b30] c0000000002bb4ec .store_online+0x88/0xe8 [c0000003a8c23bc0] c0000000002b5c14 .sysdev_store+0x4c/0x68 [c0000003a8c23c40] c000000000119c6c .sysfs_write_file+0x118/0x1bc [c0000003a8c23cf0] c0000000000c6078 .vfs_write+0x100/0x200 [c0000003a8c23d90] c0000000000c6288 .sys_write+0x54/0x9c [c0000003a8c23e30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ff865560) is in userspace

More details:

Quoted 8 lines
Quoted 4 lines

The above crash was with SMT on, and I had already off-lined the SMT sibling thread.

When I boot with SMT off, I get a slightly different crash:

I think i missed the first reply above. (I can't seem to find it on lkml either). So just to confirm, both these crashes are with the new patch on top of rc6?

Yes, rc6 + the patch you provided.

The stupid mail relay server I'm using for my ibm account seems to be very lethargic, sorry about that.

Sonny

#11 lore raw
Quoted 1 line · Sonny Rao

Yes, rc6 + the patch you provided.

We may be going down the wrong path here. Has someone else than Sonny reproduced the problem?

author
#12 lore raw
Quoted 6 lines · Christoph Lameter
Quoted 1 line · Sonny Rao

Yes, rc6 + the patch you provided.

We may be going down the wrong path here. Has someone else than Sonny reproduced the problem?

Hi, I've also just reproduced the problem on another machine which does have multiple cpus/node rather than just one cpu/node. The crash occurs at the same place when I attempt to offline the last cpu in a node.

But, I agree that somemone else should repro this. I only have ppc64 machines available to me right now.

Sonny

author
#13 lore raw
In reply to Ravikiran G Thirumalai
Quoted 46 lines · Ravikiran G Thirumalai
Quoted 45 lines · Sonny Rao
Quoted 24 lines · Sonny Rao
Quoted 6 lines · Ravikiran G Thirumalai
Quoted 1 line · Sonny Rao

(apologies if this is a dup)

...
Sonny,
Does this patch fix the issue? This one applies cleanly on 2.6.15-rc6
unlike the one that was sent to you earlier.

Hi, thanks, now I'm getting a slightly different error, hitting a BUG in the slab debug code:

ihplus:~ # echo 0 > /sys/devices/system/cpu/cpu14/online cpu 0x4: Vector: 700 (Program Check) at [c0000003a8c233f0]

    pc: c00000000009bb2c: .check_slabp+0x130/0x188
    lr: c00000000009bb28: .check_slabp+0x12c/0x188
    sp: c0000003a8c23670
   msr: 8000000000021032
  current = 0xc0000001b95297f0
  paca    = 0xc0000000005d7000
    pid   = 11116, comm = bash

kernel BUG in check_slabp at mm/slab.c:2368!
enter ? for help

Quoted 14 lines

4:mon> t
[c0000003a8c23700] c00000000009d918 .free_block+0x168/0x294 [c0000003a8c237e0] c00000000009d1dc .kfree+0x2b8/0x2d4 [c0000003a8c238a0] c0000000000a1644 .cpuup_callback+0x144/0x618 [c0000003a8c239b0] c0000000004a0780 .notifier_call_chain+0x68/0x9c [c0000003a8c23a40] c00000000007d608 .cpu_down+0x1fc/0x358 [c0000003a8c23b30] c0000000002bb4ec .store_online+0x88/0xe8 [c0000003a8c23bc0] c0000000002b5c14 .sysdev_store+0x4c/0x68 [c0000003a8c23c40] c000000000119c6c .sysfs_write_file+0x118/0x1bc [c0000003a8c23cf0] c0000000000c6078 .vfs_write+0x100/0x200 [c0000003a8c23d90] c0000000000c6288 .sys_write+0x54/0x9c [c0000003a8c23e30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ff865560) is in userspace

More details:

The above crash was with SMT on, and I had already off-lined the SMT sibling thread.

Quoted 37 lines
Quoted 1 line

When I boot with SMT off, I get a slightly different crash:

I think i missed the first reply above. (I can't seem to find it on lkml either). So just to confirm, both these crashes are with the new patch on top of rc6?

Thanks,
Kiran

Quoted 28 lines

ihplus:~ # echo 0 > /sys/devices/system/cpu/cpu14/online cpu 0x0: Vector: 700 (Program Check) at [c0000003afa13480]

    pc: c00000000009d960: .free_block+0x1b0/0x294
    lr: c00000000009d95c: .free_block+0x1ac/0x294
    sp: c0000003afa13700
   msr: 8000000000021032
  current = 0xc0000003afe04000
  paca    = 0xc0000000005d5000
    pid   = 10998, comm = bash

kernel BUG in free_block at mm/slab.c:2664!
enter ? for help

0:mon> t
[c0000003afa137e0] c00000000009d1dc .kfree+0x2b8/0x2d4 [c0000003afa138a0] c0000000000a1644 .cpuup_callback+0x144/0x618 [c0000003afa139b0] c0000000004a0780 .notifier_call_chain+0x68/0x9c [c0000003afa13a40] c00000000007d608 .cpu_down+0x1fc/0x358 [c0000003afa13b30] c0000000002bb4ec .store_online+0x88/0xe8 [c0000003afa13bc0] c0000000002b5c14 .sysdev_store+0x4c/0x68 [c0000003afa13c40] c000000000119c6c .sysfs_write_file+0x118/0x1bc [c0000003afa13cf0] c0000000000c6078 .vfs_write+0x100/0x200 [c0000003afa13d90] c0000000000c6288 .sys_write+0x54/0x9c [c0000003afa13e30] c000000000008600 syscall_exit+0x0/0x18 --- Exception: c01 (System Call) at 000000000fe5ec10
SP (ff8b4560) is in userspace

This one points to a double free somewhere

Hi, I think I've found the double free in the rc6 kernel + your patch :

starting on line 949 of the patched slab.c

                        if ((shared = l3->shared)) {
                                free_block(cachep, l3->shared->entry,
                                                l3->shared->avail, node);
                                kfree(l3->shared);
                                l3->shared = NULL;
                        }

                        alien = l3->alien;
                        l3->alien = NULL;

                        spin_unlock(&l3->list_lock);

                        kfree(nc);
                        kfree(shared);

You conditionally free l3->shared after assigning it to the auto var "shared" then below that you call kfree on "shared" again == double free.

So, I got rid of the extra free. I don't know if this was correct but I tried it anyway. Unfortunately this still does not work correctly. The system hangs for a period of time and then drops into the debugger again:

0:mon> t
[c00000000f71f890] c00000000049e5ec ._spin_lock+0x10/0x24 [c00000000f71f910] c00000000009d550 .kmem_cache_free+0x270/0x2a4 [c00000000f71f9d0] c0000000003f35e8 .kfree_skbmem+0xa0/0xfc [c00000000f71fa50] c00000000044d01c .udp_rcv+0x7ac/0x818 [c00000000f71fb60] c000000000420b14 .ip_local_deliver+0xf8/0x3f0 [c00000000f71fbf0] c000000000420328 .ip_rcv+0x3a8/0x724 [c00000000f71fc90] c0000000003fa054 .netif_receive_skb+0x378/0x3d0 [c00000000f71fd30] c0000000003fa1c4 .process_backlog+0x118/0x254 [c00000000f71fe10] c0000000003f7d3c .net_rx_action+0x188/0x2b8 [c00000000f71fed0] c000000000060f18 .__do_softirq+0xd4/0x1b8 [c00000000f71ff90] c00000000002c78c .call_do_softirq+0x14/0x24 [c0000000005ab870] c00000000000bd30 .do_softirq+0x8c/0x9c [c0000000005ab900] c00000000006143c .irq_exit+0x6c/0x84 [c0000000005ab980] c00000000000c060 .do_IRQ+0xe8/0x194 [c0000000005aba10] c000000000004134 hardware_interrupt_entry+0x8/0x54 --- Exception: 501 (Hardware Interrupt) at c000000000040670

.pseries_dedicated_idle+0x114/0x268
[c0000000005abde0] c000000000021048 .cpu_idle+0x4c/0x60
[c0000000005abe50] c0000000000091f4 .rest_init+0x44/0x5c
[c0000000005abed0] c00000000054e7f4 .start_kernel+0x29c/0x318
[c0000000005abf90] c000000000008494 .hmt_init+0x0/0x6c

0:mon>

0:mon> e
cpu 0x0: Vector: 300 (Data Access) at [c00000000f71f580]

    pc: c000000000238db4: ._raw_spin_lock+0x2c/0x1d0
    lr: c00000000049e5ec: ._spin_lock+0x10/0x24
    sp: c00000000f71f800

msr: 8000000000001032
dar: 4c
dsisr: 40000000
current = 0xc00000000061b2f0
paca = 0xc0000000005d5000
pid = 0, comm = swapper
0:mon>

#14 lore raw

I wonder if this is related to the problem Sonny is seeing -- powerpc's definitions of cpu_to_node et al. are not being used. The culprit is some too-clever preprocessor usage in asm-generic/topology.h, for example:

#ifndef cpu_to_node
#define cpu_to_node(cpu) (0)
#endif

But asm-powerpc/topology.h has cpu_to_node defined as a static inline (which does not make it a preprocessor symbol), so we get the generic
- and incorrect - definition.

Does removing the #include of asm-generic/topology.h from the bottom of asm-powerpc/topology.h have any effect?

author
#15 lore raw
Quoted 10 lines · Nathan Lynch

I wonder if this is related to the problem Sonny is seeing -- powerpc's definitions of cpu_to_node et al. are not being used. The culprit is some too-clever preprocessor usage in asm-generic/topology.h, for example:

#ifndef cpu_to_node
#define cpu_to_node(cpu) (0)
#endif

Quoted 6 lines

But asm-powerpc/topology.h has cpu_to_node defined as a static inline (which does not make it a preprocessor symbol), so we get the generic
- and incorrect - definition.

Does removing the #include of asm-generic/topology.h from the bottom of asm-powerpc/topology.h have any effect?

Hi, no it doesn't make a difference. That include is protected by CONFIG_NUMA as well, so it never gets hit. At Anton's suggestion I even put in an #error into asm-generic/topology.h to make sure it wasn't an issue -- it didn't hit.

Sonny

#16 lore raw
Quoted 9 lines · Sonny Rao
Quoted 3 lines · Nathan Lynch

Does removing the #include of asm-generic/topology.h from the bottom of asm-powerpc/topology.h have any effect?

Hi, no it doesn't make a difference. That include is protected by CONFIG_NUMA as well, so it never gets hit. At Anton's suggestion I even put in an #error into asm-generic/topology.h to make sure it wasn't an issue -- it didn't hit.

Gah, sorry, forgot Anton fixed this a while back.