Skip to contentSkip to conversation

2.6.17-rc5-mm3

PATCH In review Martin Bligh 6 participants 14 messages Jun 5, 2006 – Jun 7, 2006

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

panic on NUMA-Q during LTP. Was fine in -mm2.

BUG: unable to handle kernel paging request at virtual address 22222232
 printing eip:
c012cf84
*pde = 25b5a001
*pte = 00000000
Oops: 0000 [#1]
SMP
last sysfs file: /devices/pci0000:00/0000:00:0a.0/resource
Modules linked in:
CPU:    12
EIP:    0060:[<c012cf84>]    Not tainted VLI
EFLAGS: 00010002   (2.6.17-rc5-mm3-autokern1 #1)

EIP is at check_deadlock+0x19/0xe1
eax: 00000001 ebx: e4453030 ecx: 00000000 edx: e4008000
esi: 22222222 edi: 00000001 ebp: 22222222 esp: e47ebec0
ds: 007b es: 007b ss: 0068
Process mkdir09 (pid: 18319, threadinfo=e47ea000 task=e5f91ab0)

Stack: e4453030 22222222 00000000 e459231c c012d015 22222222 00000001 e4008000
       e459231c e47ea000 e47ebf1c e5f91ab0 c012d1ce e459231c 00000000 
e47ea000
       e47ebf1c e459231c 00000246 c02f1d74 e459231c e47ebf1c e47ea000 
e47ebf1c
Call Trace:
 [<c012d015>] check_deadlock+0xaa/0xe1
 [<c012d1ce>] debug_mutex_add_waiter+0x4a/0x5c
 [<c02f1d74>] __mutex_lock_slowpath+0x9e/0x1cb
 [<c01648a9>] do_rmdir+0x67/0xc2
 [<c02001da>] __put_user_4+0x12/0x18
 [<c016490f>] sys_rmdir+0xb/0xe
 [<c02f2f1f>] syscall_call+0x7/0xb
Code: 0c 68 60 07 31 c0 e8 22 c0 fe ff 58 fa 5b 5e 5f 5d c3 55 83 3d cc 11 36 c0 00 57 56 53 8b 6c 24 14 8b 7c 24 18 0f 84 c1 00 00 00 <8b> 55 10 31 c0 85 d2 0f 84 b6 00 00 00 8b 1a 31 f6 8b 83 c4 04
EIP: [<c012cf84>] check_deadlock+0x19/0xe1 SS:ESP 0068:e47ebec0
#1 lore raw
Quoted 7 lines · Martin Bligh

panic on NUMA-Q during LTP. Was fine in -mm2.

BUG: unable to handle kernel paging request at virtual address 22222232

EIP is at check_deadlock+0x19/0xe1
eax: 00000001 ebx: e4453030 ecx: 00000000 edx: e4008000
esi: 22222222 edi: 00000001 ebp: 22222222 esp: e47ebec0

again these 0x22222222 entries on the stack. What on earth does this? Andy got a similar crash on x86_64, with a 0x2222222222222222 entry ...

nothing of our magic values are 0x22 or 0x222222222.

Ingo

#2 lore raw
Quoted 7 lines · On Mon, 5 Jun 2006 21:44:22 +0200 Ingo Molnar wrote:
Quoted 3 lines · Martin Bligh

panic on NUMA-Q during LTP. Was fine in -mm2.

BUG: unable to handle kernel paging request at virtual address 22222232
Quoted 8 lines
Quoted 3 lines

EIP is at check_deadlock+0x19/0xe1
eax: 00000001 ebx: e4453030 ecx: 00000000 edx: e4008000
esi: 22222222 edi: 00000001 ebp: 22222222 esp: e47ebec0

again these 0x22222222 entries on the stack. What on earth does this? Andy got a similar crash on x86_64, with a 0x2222222222222222 entry ...

nothing of our magic values are 0x22 or 0x222222222.

kernel/mutex-debug.c:
void debug_mutex_free_waiter(struct mutex_waiter *waiter)

{
	DEBUG_WARN_ON(!list_empty(&waiter->list));
	memset(waiter, 0x22, sizeof(*waiter));
}

---
~Randy

#3 lore raw
Quoted 16 lines · Randy.Dunlap
Quoted 14 lines · On Mon, 5 Jun 2006 21:44:22 +0200 Ingo Molnar wrote:
Quoted 7 lines · Martin Bligh

panic on NUMA-Q during LTP. Was fine in -mm2.

BUG: unable to handle kernel paging request at virtual address 22222232

EIP is at check_deadlock+0x19/0xe1
eax: 00000001 ebx: e4453030 ecx: 00000000 edx: e4008000
esi: 22222222 edi: 00000001 ebp: 22222222 esp: e47ebec0

again these 0x22222222 entries on the stack. What on earth does this? Andy got a similar crash on x86_64, with a 0x2222222222222222 entry ...

Quoted 8 lines
Quoted 1 line

nothing of our magic values are 0x22 or 0x222222222.

kernel/mutex-debug.c:
void debug_mutex_free_waiter(struct mutex_waiter *waiter)

{
	DEBUG_WARN_ON(!list_empty(&waiter->list));
	memset(waiter, 0x22, sizeof(*waiter));
}

Documentation/magic-number.txt sounds so promising, but we scatter definitions of numbers all over the place. (No mention of the slab poison values, or similar numbers there for eg, and various pointers to _other_ lists of magic numbers).

Dave

Signature · 1 line
http://www.codemonkey.org.uk
#8 lore raw
In reply to Randy.Dunlap
Quoted 11 lines · Randy.Dunlap
Quoted 11 lines
Quoted 7 lines

panic on NUMA-Q during LTP. Was fine in -mm2.

BUG: unable to handle kernel paging request at virtual address 22222232

EIP is at check_deadlock+0x19/0xe1
eax: 00000001 ebx: e4453030 ecx: 00000000 edx: e4008000
esi: 22222222 edi: 00000001 ebp: 22222222 esp: e47ebec0

again these 0x22222222 entries on the stack. What on earth does this? Andy got a similar crash on x86_64, with a 0x2222222222222222 entry ...

Quoted 8 lines
Quoted 1 line

nothing of our magic values are 0x22 or 0x222222222.

kernel/mutex-debug.c:
void debug_mutex_free_waiter(struct mutex_waiter *waiter)

{
	DEBUG_WARN_ON(!list_empty(&waiter->list));
	memset(waiter, 0x22, sizeof(*waiter));
}

ah!!! that's indeed a hint. Will take a look tomorrow.

Ingo

#4 lore raw
In reply to Dave Jones
Quoted 11 lines · Dave Jones
Quoted 6 lines

kernel/mutex-debug.c:
void debug_mutex_free_waiter(struct mutex_waiter *waiter)

{
	DEBUG_WARN_ON(!list_empty(&waiter->list));
	memset(waiter, 0x22, sizeof(*waiter));
}

Documentation/magic-number.txt sounds so promising, but we scatter definitions of numbers all over the place. (No mention of the slab poison values, or similar numbers there for eg, and various pointers to _other_ lists of magic numbers).

we've also got include/linux/poison.h - i'll move this value there.

Ingo

#5 lore raw
In reply to Dave Jones
Quoted 26 lines · On Mon, 5 Jun 2006 16:05:54 -0400 Dave Jones wrote:
Quoted 16 lines · Randy.Dunlap
Quoted 14 lines · On Mon, 5 Jun 2006 21:44:22 +0200 Ingo Molnar wrote:
Quoted 7 lines · Martin Bligh

panic on NUMA-Q during LTP. Was fine in -mm2.

BUG: unable to handle kernel paging request at virtual address 22222232

EIP is at check_deadlock+0x19/0xe1
eax: 00000001 ebx: e4453030 ecx: 00000000 edx: e4008000
esi: 22222222 edi: 00000001 ebp: 22222222 esp: e47ebec0

again these 0x22222222 entries on the stack. What on earth does this? Andy got a similar crash on x86_64, with a 0x2222222222222222 entry ...

Quoted 8 lines
Quoted 1 line

nothing of our magic values are 0x22 or 0x222222222.

kernel/mutex-debug.c:
void debug_mutex_free_waiter(struct mutex_waiter *waiter)

{
	DEBUG_WARN_ON(!list_empty(&waiter->list));
	memset(waiter, 0x22, sizeof(*waiter));
}
Quoted 4 lines

Documentation/magic-number.txt sounds so promising, but we scatter definitions of numbers all over the place. (No mention of the slab poison values, or similar numbers there for eg, and various pointers to _other_ lists of magic numbers).

I have a few more that I can add to include/linux/poison.h, like this one above (only in -mm at present).

./include/linux/libata.h:#define ATA_TAG_POISON 0xfafbfcfdU

./arch/ppc/8260_io/fcc_enet.c:1918: memset((char *)(&(immap->im_dprambase[(mem_addr+64)])), 0x88, 32);
./drivers/usb/mon/mon_text.c:429: memset(mem, 0xe5, sizeof(struct mon_event_text));

./kernel/mutex-debug.c:384: memset(waiter, 0x11, sizeof(*waiter));
./kernel/mutex-debug.c:400: memset(waiter, 0x22, sizeof(*waiter));

./security/keys/key.c:985: memset(&key->payload, 0xbd, sizeof(key->payload));

./drivers/char/ftape/lowlevel/ftape-ctl.c:738: memset(ft_buffer[i]->address, 0xAA, FT_BUFF_SIZE);

./drivers/block/sx8.c:/* 0xf is just arbitrary, non-zero noise; this is sorta like poisoning */

---
~Randy

#6 lore raw

PATCH 1/1 poison: add & use more constants

From: Randy Dunlap <[email protected]>

Add more poison values to include/linux/poison.h.
It's not clear to me whether some others should be added or not, so I haven't added any of these:

./include/linux/libata.h:#define ATA_TAG_POISON 0xfafbfcfdU
./arch/ppc/8260_io/fcc_enet.c:1918: memset((char *)(&(immap->im_dprambase[(mem_addr+64)])), 0x88, 32);
./drivers/usb/mon/mon_text.c:429: memset(mem, 0xe5, sizeof(struct mon_event_text));
./drivers/char/ftape/lowlevel/ftape-ctl.c:738: memset(ft_buffer[i]->address, 0xAA, FT_BUFF_SIZE);
./drivers/block/sx8.c:/* 0xf is just arbitrary, non-zero noise; this is sorta like poisoning */

---
 include/linux/poison.h |    7 +++++++
 kernel/mutex-debug.c   |    5 +++--
 security/keys/key.c    |    3 ++-
 3 files changed, 12 insertions(+), 3 deletions(-)
include/linux/poison.h +7 -0
@@ -45,6 +45,13 @@
 /********** drivers/atm/ **********/
 #define ATM_POISON_FREE		0x12
 
+/********** kernel/mutexes **********/
+#define MUTEX_DEBUG_INIT	0x11
+#define MUTEX_DEBUG_FREE	0x22
+
+/********** security/ **********/
+#define KEY_DESTROY		0xbd
+
 /********** sound/oss/ **********/
 #define OSS_POISON_FREE		0xAB
 
kernel/mutex-debug.c +3 -2
@@ -16,6 +16,7 @@
 #include <linux/sched.h>
 #include <linux/delay.h>
 #include <linux/module.h>
+#include <linux/poison.h>
 #include <linux/spinlock.h>
 #include <linux/kallsyms.h>
 #include <linux/interrupt.h>
@@ -155,7 +156,7 @@ void debug_mutex_set_owner(struct mutex 
 
 void debug_mutex_lock_common(struct mutex *lock, struct mutex_waiter *waiter)
 {
-	memset(waiter, 0x11, sizeof(*waiter));
+	memset(waiter, MUTEX_DEBUG_INIT, sizeof(*waiter));
 	waiter->magic = waiter;
 	INIT_LIST_HEAD(&waiter->list);
 }
@@ -171,7 +172,7 @@ void debug_mutex_wake_waiter(struct mute
 void debug_mutex_free_waiter(struct mutex_waiter *waiter)
 {
 	DEBUG_WARN_ON(!list_empty(&waiter->list));
-	memset(waiter, 0x22, sizeof(*waiter));
+	memset(waiter, MUTEX_DEBUG_FREE, sizeof(*waiter));
 }
 
 void debug_mutex_add_waiter(struct mutex *lock, struct mutex_waiter *waiter,
security/keys/key.c +2 -1
@@ -11,6 +11,7 @@
 
 #include <linux/module.h>
 #include <linux/init.h>
+#include <linux/poison.h>
 #include <linux/sched.h>
 #include <linux/slab.h>
 #include <linux/security.h>
@@ -986,7 +987,7 @@ void unregister_key_type(struct key_type
 		if (key->type == ktype) {
 			if (ktype->destroy)
 				ktype->destroy(key);
-			memset(&key->payload, 0xbd, sizeof(key->payload));
+			memset(&key->payload, KEY_DESTROY, sizeof(key->payload));
 		}
 	}
 

---

#9 lore raw

PATCH 1/1 better lock debugging: remove mutex deadlock checking code

In reply to Randy.Dunlap
Quoted 1 line · Randy.Dunlap
BUG: unable to handle kernel paging request at virtual address 22222232

ok, this was a big thinko on my part, and it was right before our eyes. Mutex deadlock checking relied on the 'big mutex debugging lock', but that one is gone now - so mutex deadlock checking became racy (as your crashes nicely pinpointed that). The races are more likely with an increasing number of CPUs.

so the patch below finishes the cleanup i started: it removes deadlock checking from the mutex code and lets the lock validator do that. This should also be (much) faster on SMP, because the lock validator is lockless in the fastpath. (if CONFIG_DEBUG_LOCKDEP is disabled)

Ingo

----------------
Subject: better lock debugging: remove mutex deadlock checking code
From: Ingo Molnar <[email protected]>

with the lock validator we detect mutex deadlocks (and more), the mutex deadlock checking code is both redundant and slower. So remove it.

---
 kernel/mutex-debug.c |  126 ---------------------------------------------------
 lib/Kconfig.debug    |    8 ---
 2 files changed, 1 insertion(+), 133 deletions(-)
kernel/mutex-debug.c +1 -125
Index: linux/kernel/mutex-debug.c
===================================================================
@@ -23,128 +23,6 @@
 
 #include "mutex-debug.h"
 
-static void printk_task(struct task_struct *p)
-{
-	if (p)
-		printk("%16s:%5d [%p, %3d]", p->comm, p->pid, p, p->prio);
-	else
-		printk("<none>");
-}
-
-static void printk_ti(struct thread_info *ti)
-{
-	if (ti)
-		printk_task(ti->task);
-	else
-		printk("<none>");
-}
-
-static void printk_lock(struct mutex *lock, int print_owner)
-{
-#ifdef CONFIG_PROVE_MUTEX_LOCKING
-	printk(" [%p] {%s}\n", lock, lock->dep_map.name);
-#else
-	printk(" [%p]\n", lock);
-#endif
-
-	if (print_owner && lock->owner) {
-		printk(".. held by:  ");
-		printk_ti(lock->owner);
-		printk("\n");
-	}
-}
-
-static void report_deadlock(struct task_struct *task, struct mutex *lock,
-			    struct mutex *lockblk)
-{
-	printk("\n%s/%d is trying to acquire this lock:\n",
-		current->comm, current->pid);
-	printk_lock(lock, 1);
-	debug_show_held_locks(current);
-
-	if (lockblk) {
-		printk("but %s/%d is deadlocking current task %s/%d!\n\n",
-			task->comm, task->pid, current->comm, current->pid);
-		printk("\n%s/%d is blocked on this lock:\n",
-			task->comm, task->pid);
-		printk_lock(lockblk, 1);
-
-		debug_show_held_locks(task);
-
-		printk("\n%s/%d's [blocked] stackdump:\n\n",
-			task->comm, task->pid);
-		show_stack(task, NULL);
-	}
-
-	printk("\n%s/%d's [current] stackdump:\n\n",
-		current->comm, current->pid);
-	dump_stack();
-	debug_show_all_locks();
-	printk("[ turning off deadlock detection. Please report this. ]\n\n");
-	local_irq_disable();
-}
-
-/*
- * Recursively check for mutex deadlocks:
- */
-static int check_deadlock(struct mutex *lock, int depth, struct thread_info *ti)
-{
-	struct mutex *lockblk;
-	struct task_struct *task;
-
-	if (!debug_locks)
-		return 0;
-
-	ti = lock->owner;
-	if (!ti)
-		return 0;
-
-	task = ti->task;
-	/*
-	 * In the PROVE_MUTEX_LOCKING we are tracking all held
-	 * locks already, which allows us to optimize this:
-	 */
-#ifdef CONFIG_PROVE_MUTEX_LOCKING
-	if (!task->lockdep_depth)
-		return 0;
-#endif
-	lockblk = NULL;
-	if (task->blocked_on)
-		lockblk = task->blocked_on->lock;
-
-	/* Self-deadlock: */
-	if (current == task) {
-		debug_locks_off();
-		if (depth)
-			return 1;
-		printk("\n==========================================\n");
-		printk(  "[ BUG: lock recursion deadlock detected! |\n");
-		printk(  "------------------------------------------\n");
-		report_deadlock(task, lock, NULL);
-		return 0;
-	}
-
-	/* Ugh, something corrupted the lock data structure? */
-	if (depth > 20) {
-		debug_locks_off();
-		printk("\n===========================================\n");
-		printk(  "[ BUG: infinite lock dependency detected!? |\n");
-		printk(  "-------------------------------------------\n");
-		report_deadlock(task, lock, lockblk);
-		return 0;
-	}
-
-	/* Recursively check for dependencies: */
-	if (lockblk && check_deadlock(lockblk, depth+1, ti)) {
-		printk("\n============================================\n");
-		printk(  "[ BUG: circular locking deadlock detected! ]\n");
-		printk(  "--------------------------------------------\n");
-		report_deadlock(task, lock, lockblk);
-		return 0;
-	}
-	return 0;
-}
-
 /*
  * Must be called with lock->wait_lock held.
  */
@@ -178,9 +56,7 @@ void debug_mutex_add_waiter(struct mutex
 			    struct thread_info *ti)
 {
 	SMP_DEBUG_WARN_ON(!spin_is_locked(&lock->wait_lock));
-#ifdef CONFIG_DEBUG_MUTEX_DEADLOCKS
-	check_deadlock(lock, 0, ti);
-#endif
+
 	/* Mark the current thread as blocked on the lock: */
 	ti->task->blocked_on = waiter;
 	waiter->lock = lock;
lib/Kconfig.debug +0 -8
Index: linux/lib/Kconfig.debug
===================================================================
@@ -164,14 +164,6 @@ config DEBUG_MUTEX_ALLOC
 	 (kfree(), kmem_cache_free(), free_pages(), vfree(), etc.),
 	 or whether there is any lock held during task exit.
 
-config DEBUG_MUTEX_DEADLOCKS
-	bool "Detect mutex related deadlocks"
-	default y
-	depends on DEBUG_MUTEXES
-	help
-	 This feature will automatically detect and report mutex related
-	 deadlocks, as they happen.
-
 config DEBUG_RT_MUTEXES
 	bool "RT Mutex debugging, deadlock detection"
 	default y
#10 lore raw

Re: [patch, -rc5-mm3] better lock debugging: remove mutex deadlock checking code

Quoted 185 lines · Ingo Molnar
Quoted 1 line · Randy.Dunlap
BUG: unable to handle kernel paging request at virtual address 22222232

ok, this was a big thinko on my part, and it was right before our eyes. Mutex deadlock checking relied on the 'big mutex debugging lock', but that one is gone now - so mutex deadlock checking became racy (as your crashes nicely pinpointed that). The races are more likely with an increasing number of CPUs.

so the patch below finishes the cleanup i started: it removes deadlock checking from the mutex code and lets the lock validator do that. This should also be (much) faster on SMP, because the lock validator is lockless in the fastpath. (if CONFIG_DEBUG_LOCKDEP is disabled)

Ingo

----------------
Subject: better lock debugging: remove mutex deadlock checking code
From: Ingo Molnar <[email protected]>

with the lock validator we detect mutex deadlocks (and more), the mutex deadlock checking code is both redundant and slower. So remove it.

---
 kernel/mutex-debug.c |  126 ---------------------------------------------------
 lib/Kconfig.debug    |    8 ---
 2 files changed, 1 insertion(+), 133 deletions(-)

Index: linux/kernel/mutex-debug.c
===================================================================

kernel/mutex-debug.c +1 -125
@@ -23,128 +23,6 @@
 
 #include "mutex-debug.h"
 
-static void printk_task(struct task_struct *p)
-{
-	if (p)
-		printk("%16s:%5d [%p, %3d]", p->comm, p->pid, p, p->prio);
-	else
-		printk("<none>");
-}
-
-static void printk_ti(struct thread_info *ti)
-{
-	if (ti)
-		printk_task(ti->task);
-	else
-		printk("<none>");
-}
-
-static void printk_lock(struct mutex *lock, int print_owner)
-{
-#ifdef CONFIG_PROVE_MUTEX_LOCKING
-	printk(" [%p] {%s}\n", lock, lock->dep_map.name);
-#else
-	printk(" [%p]\n", lock);
-#endif
-
-	if (print_owner && lock->owner) {
-		printk(".. held by:  ");
-		printk_ti(lock->owner);
-		printk("\n");
-	}
-}
-
-static void report_deadlock(struct task_struct *task, struct mutex *lock,
-			    struct mutex *lockblk)
-{
-	printk("\n%s/%d is trying to acquire this lock:\n",
-		current->comm, current->pid);
-	printk_lock(lock, 1);
-	debug_show_held_locks(current);
-
-	if (lockblk) {
-		printk("but %s/%d is deadlocking current task %s/%d!\n\n",
-			task->comm, task->pid, current->comm, current->pid);
-		printk("\n%s/%d is blocked on this lock:\n",
-			task->comm, task->pid);
-		printk_lock(lockblk, 1);
-
-		debug_show_held_locks(task);
-
-		printk("\n%s/%d's [blocked] stackdump:\n\n",
-			task->comm, task->pid);
-		show_stack(task, NULL);
-	}
-
-	printk("\n%s/%d's [current] stackdump:\n\n",
-		current->comm, current->pid);
-	dump_stack();
-	debug_show_all_locks();
-	printk("[ turning off deadlock detection. Please report this. ]\n\n");
-	local_irq_disable();
-}
-
-/*
- * Recursively check for mutex deadlocks:
- */
-static int check_deadlock(struct mutex *lock, int depth, struct thread_info *ti)
-{
-	struct mutex *lockblk;
-	struct task_struct *task;
-
-	if (!debug_locks)
-		return 0;
-
-	ti = lock->owner;
-	if (!ti)
-		return 0;
-
-	task = ti->task;
-	/*
-	 * In the PROVE_MUTEX_LOCKING we are tracking all held
-	 * locks already, which allows us to optimize this:
-	 */
-#ifdef CONFIG_PROVE_MUTEX_LOCKING
-	if (!task->lockdep_depth)
-		return 0;
-#endif
-	lockblk = NULL;
-	if (task->blocked_on)
-		lockblk = task->blocked_on->lock;
-
-	/* Self-deadlock: */
-	if (current == task) {
-		debug_locks_off();
-		if (depth)
-			return 1;
-		printk("\n==========================================\n");
-		printk(  "[ BUG: lock recursion deadlock detected! |\n");
-		printk(  "------------------------------------------\n");
-		report_deadlock(task, lock, NULL);
-		return 0;
-	}
-
-	/* Ugh, something corrupted the lock data structure? */
-	if (depth > 20) {
-		debug_locks_off();
-		printk("\n===========================================\n");
-		printk(  "[ BUG: infinite lock dependency detected!? |\n");
-		printk(  "-------------------------------------------\n");
-		report_deadlock(task, lock, lockblk);
-		return 0;
-	}
-
-	/* Recursively check for dependencies: */
-	if (lockblk && check_deadlock(lockblk, depth+1, ti)) {
-		printk("\n============================================\n");
-		printk(  "[ BUG: circular locking deadlock detected! ]\n");
-		printk(  "--------------------------------------------\n");
-		report_deadlock(task, lock, lockblk);
-		return 0;
-	}
-	return 0;
-}
-
 /*
  * Must be called with lock->wait_lock held.
  */
@@ -178,9 +56,7 @@ void debug_mutex_add_waiter(struct mutex
 			    struct thread_info *ti)
 {
 	SMP_DEBUG_WARN_ON(!spin_is_locked(&lock->wait_lock));
-#ifdef CONFIG_DEBUG_MUTEX_DEADLOCKS
-	check_deadlock(lock, 0, ti);
-#endif
+
 	/* Mark the current thread as blocked on the lock: */
 	ti->task->blocked_on = waiter;
 	waiter->lock = lock;

Index: linux/lib/Kconfig.debug
===================================================================

lib/Kconfig.debug +0 -1
@@ -164,14 +164,6 @@ config DEBUG_MUTEX_ALLOC
 	 (kfree(), kmem_cache_free(), free_pages(), vfree(), etc.),
 	 or whether there is any lock held during task exit.
 
-config DEBUG_MUTEX_DEADLOCKS

lib/Kconfig.debug config DEBUG_MUTEX_ALLOC · line 169

Quoted 10 lines
lib/Kconfig.debug +0 -7
-	bool "Detect mutex related deadlocks"
-	default y
-	depends on DEBUG_MUTEXES
-	help
-	 This feature will automatically detect and report mutex related
-	 deadlocks, as they happen.
-
 config DEBUG_RT_MUTEXES
 	bool "RT Mutex debugging, deadlock detection"
 	default y

I'll shove this one in for testing too. Results on TKO as I have them.

-apw

#7 lore raw

Re: [PATCH] poison: add & use more constants

In reply to Randy.Dunlap
Quoted 11 lines · Randy.Dunlap

From: Randy Dunlap <[email protected]>

Add more poison values to include/linux/poison.h.
It's not clear to me whether some others should be added or not, so I haven't added any of these:

./include/linux/libata.h:#define ATA_TAG_POISON 0xfafbfcfdU
./arch/ppc/8260_io/fcc_enet.c:1918: memset((char *)(&(immap->im_dprambase[(mem_addr+64)])), 0x88, 32);
./drivers/usb/mon/mon_text.c:429: memset(mem, 0xe5, sizeof(struct mon_event_text));
./drivers/char/ftape/lowlevel/ftape-ctl.c:738: memset(ft_buffer[i]->address, 0xAA, FT_BUFF_SIZE);
./drivers/block/sx8.c:/* 0xf is just arbitrary, non-zero noise; this is sorta like poisoning */

You don't have my personal favorite? From AIX that would poison pages with 0xdeadbeef :)

-- Steve

#11 lore raw

Re: [patch, -rc5-mm3] better lock debugging: remove mutex deadlock checking code

In reply to Andy Whitcroft
Quoted 202 lines · Andy Whitcroft
Quoted 193 lines · Ingo Molnar
Quoted 1 line · Randy.Dunlap
BUG: unable to handle kernel paging request at virtual address 22222232

ok, this was a big thinko on my part, and it was right before our eyes. Mutex deadlock checking relied on the 'big mutex debugging lock', but that one is gone now - so mutex deadlock checking became racy (as your crashes nicely pinpointed that). The races are more likely with an increasing number of CPUs.

so the patch below finishes the cleanup i started: it removes deadlock checking from the mutex code and lets the lock validator do that. This should also be (much) faster on SMP, because the lock validator is lockless in the fastpath. (if CONFIG_DEBUG_LOCKDEP is disabled)

Ingo

----------------
Subject: better lock debugging: remove mutex deadlock checking code
From: Ingo Molnar <[email protected]>

with the lock validator we detect mutex deadlocks (and more), the mutex deadlock checking code is both redundant and slower. So remove it.

---
kernel/mutex-debug.c | 126 ---------------------------------------------------
lib/Kconfig.debug | 8 ---

2 files changed, 1 insertion(+), 133 deletions(-)

Index: linux/kernel/mutex-debug.c
===================================================================

kernel/mutex-debug.c +1 -125
@@ -23,128 +23,6 @@
 
 #include "mutex-debug.h"
 
-static void printk_task(struct task_struct *p)
-{
-	if (p)
-		printk("%16s:%5d [%p, %3d]", p->comm, p->pid, p, p->prio);
-	else
-		printk("<none>");
-}
-
-static void printk_ti(struct thread_info *ti)
-{
-	if (ti)
-		printk_task(ti->task);
-	else
-		printk("<none>");
-}
-
-static void printk_lock(struct mutex *lock, int print_owner)
-{
-#ifdef CONFIG_PROVE_MUTEX_LOCKING
-	printk(" [%p] {%s}\n", lock, lock->dep_map.name);
-#else
-	printk(" [%p]\n", lock);
-#endif
-
-	if (print_owner && lock->owner) {
-		printk(".. held by:  ");
-		printk_ti(lock->owner);
-		printk("\n");
-	}
-}
-
-static void report_deadlock(struct task_struct *task, struct mutex *lock,
-			    struct mutex *lockblk)
-{
-	printk("\n%s/%d is trying to acquire this lock:\n",
-		current->comm, current->pid);
-	printk_lock(lock, 1);
-	debug_show_held_locks(current);
-
-	if (lockblk) {
-		printk("but %s/%d is deadlocking current task %s/%d!\n\n",
-			task->comm, task->pid, current->comm, current->pid);
-		printk("\n%s/%d is blocked on this lock:\n",
-			task->comm, task->pid);
-		printk_lock(lockblk, 1);
-
-		debug_show_held_locks(task);
-
-		printk("\n%s/%d's [blocked] stackdump:\n\n",
-			task->comm, task->pid);
-		show_stack(task, NULL);
-	}
-
-	printk("\n%s/%d's [current] stackdump:\n\n",
-		current->comm, current->pid);
-	dump_stack();
-	debug_show_all_locks();
-	printk("[ turning off deadlock detection. Please report this. ]\n\n");
-	local_irq_disable();
-}
-
-/*
- * Recursively check for mutex deadlocks:
- */
-static int check_deadlock(struct mutex *lock, int depth, struct thread_info *ti)
-{
-	struct mutex *lockblk;
-	struct task_struct *task;
-
-	if (!debug_locks)
-		return 0;
-
-	ti = lock->owner;
-	if (!ti)
-		return 0;
-
-	task = ti->task;
-	/*
-	 * In the PROVE_MUTEX_LOCKING we are tracking all held
-	 * locks already, which allows us to optimize this:
-	 */
-#ifdef CONFIG_PROVE_MUTEX_LOCKING
-	if (!task->lockdep_depth)
-		return 0;
-#endif
-	lockblk = NULL;
-	if (task->blocked_on)
-		lockblk = task->blocked_on->lock;
-
-	/* Self-deadlock: */
-	if (current == task) {
-		debug_locks_off();
-		if (depth)
-			return 1;
-		printk("\n==========================================\n");
-		printk(  "[ BUG: lock recursion deadlock detected! |\n");
-		printk(  "------------------------------------------\n");
-		report_deadlock(task, lock, NULL);
-		return 0;
-	}
-
-	/* Ugh, something corrupted the lock data structure? */
-	if (depth > 20) {
-		debug_locks_off();
-		printk("\n===========================================\n");
-		printk(  "[ BUG: infinite lock dependency detected!? |\n");
-		printk(  "-------------------------------------------\n");
-		report_deadlock(task, lock, lockblk);
-		return 0;
-	}
-
-	/* Recursively check for dependencies: */
-	if (lockblk && check_deadlock(lockblk, depth+1, ti)) {
-		printk("\n============================================\n");
-		printk(  "[ BUG: circular locking deadlock detected! ]\n");
-		printk(  "--------------------------------------------\n");
-		report_deadlock(task, lock, lockblk);
-		return 0;
-	}
-	return 0;
-}
-
 /*
 * Must be called with lock->wait_lock held.
 */
@@ -178,9 +56,7 @@ void debug_mutex_add_waiter(struct mutex
 			    struct thread_info *ti)
 {
 	SMP_DEBUG_WARN_ON(!spin_is_locked(&lock->wait_lock));
-#ifdef CONFIG_DEBUG_MUTEX_DEADLOCKS
-	check_deadlock(lock, 0, ti);
-#endif
+
 	/* Mark the current thread as blocked on the lock: */
 	ti->task->blocked_on = waiter;
 	waiter->lock = lock;

Index: linux/lib/Kconfig.debug
===================================================================

lib/Kconfig.debug +0 -8
@@ -164,14 +164,6 @@ config DEBUG_MUTEX_ALLOC
 	 (kfree(), kmem_cache_free(), free_pages(), vfree(), etc.),
 	 or whether there is any lock held during task exit.
 
-config DEBUG_MUTEX_DEADLOCKS
-	bool "Detect mutex related deadlocks"
-	default y
-	depends on DEBUG_MUTEXES
-	help
-	 This feature will automatically detect and report mutex related
-	 deadlocks, as they happen.
-
Quoted 3 lines

config DEBUG_RT_MUTEXES
bool "RT Mutex debugging, deadlock detection" default y

I'll shove this one in for testing too. Results on TKO as I have them.

Quoted 2 lines

-apw

This is definatly clearing up a bunch of problems with the current -mm.

-apw

#12 lore raw

Re: [patch, -rc5-mm3] better lock debugging: remove mutex deadlock checking code

Quoted 4 lines · Andy Whitcroft
Quoted 1 line

I'll shove this one in for testing too. Results on TKO as I have them.

This is definatly clearing up a bunch of problems with the current -mm.

great! Thanks for testing this out, this bug was the scariest pending one.

Ingo

Acked-by on v1 1/1
#13 lore raw

Re: [patch, -rc5-mm3] better lock debugging: remove mutex deadlock checking code

In reply to Ingo Molnar
Quoted 23 lines · Ingo Molnar
Quoted 1 line · Randy.Dunlap
BUG: unable to handle kernel paging request at virtual address 22222232

ok, this was a big thinko on my part, and it was right before our eyes. Mutex deadlock checking relied on the 'big mutex debugging lock', but that one is gone now - so mutex deadlock checking became racy (as your crashes nicely pinpointed that). The races are more likely with an increasing number of CPUs.

so the patch below finishes the cleanup i started: it removes deadlock checking from the mutex code and lets the lock validator do that. This should also be (much) faster on SMP, because the lock validator is lockless in the fastpath. (if CONFIG_DEBUG_LOCKDEP is disabled)

Ingo

----------------
Subject: better lock debugging: remove mutex deadlock checking code
From: Ingo Molnar <[email protected]>

Quoted 5 lines

with the lock validator we detect mutex deadlocks (and more), the mutex deadlock checking code is both redundant and slower. So remove it.

---

Ok, this patch in combination with either fix for the swap max bug are showing passes across the board.

-apw