use canary at end of stack to indicate overruns at oops time

PATCH In review Eric Sandeen 6 participants 12 messages Apr 22, 2008 – Apr 28, 2008

Subsystems (5)

x86 MM core Scheduler exec and binfmt +1 more subsystems

lore

Share & tools

lore

Eric Sandeen

author

2008-04-22 03:45 UTC #0 lore raw

PATCH 1/1 use canary at end of stack to indicate overruns at oops time

Use a canary at the end of the stack to clearly indicate at oops time whether the stack has ever overflowed.

This is a very simple implementation with a couple of drawbacks:

1) a thread may legitimately use exactly up to the last word on the stack

-- but the chances of doing this and then oopsing later seem slim

2) it's possible that the stack usage isn't dense enough that the canary location could get skipped over

-- but the worst that happens is that we don't flag the overrun

With the code in place, an intentionally-bloated stack oops does:

BUG: unable to handle kernel paging request at ffff8103f84cc680
IP: [<ffffffff810253df>] update_curr+0x9a/0xa8
PGD 8063 PUD 0 
Thread overran stack or stack corrupted
Oops: 0000 [1] SMP

CPU 0
...

Signed-off-by Eric Sandeen <[email protected]>

---

arch/x86/mm/fault.c +7 -0

Index: linux-2.6.25/arch/x86/mm/fault.c
===================================================================
@@ -25,6 +25,7 @@
 #include <linux/kprobes.h>
 #include <linux/uaccess.h>
 #include <linux/kdebug.h>
+#include <linux/magic.h>
 
 #include <asm/system.h>
 #include <asm/desc.h>

@@ -581,6 +582,8 @@ void __kprobes do_page_fault(struct pt_r
 	unsigned long address;
 	int write, si_code;
 	int fault;
+	unsigned long *stackend;
+
 #ifdef CONFIG_X86_64
 	unsigned long flags;
 #endif

@@ -850,6 +853,10 @@ no_context:
 
 	show_fault_oops(regs, error_code, address);
 
+	stackend = end_of_stack(tsk);
+	if (*stackend != STACK_END_MAGIC)
+		printk(KERN_ALERT "Thread overran stack or stack corrupted\n");
+
 	tsk->thread.cr2 = address;
 	tsk->thread.trap_no = 14;
 	tsk->thread.error_code = error_code;

include/linux/magic.h +1 -0

Index: linux-2.6.25/include/linux/magic.h
===================================================================
@@ -42,4 +42,5 @@
 #define FUTEXFS_SUPER_MAGIC	0xBAD1DEA
 #define INOTIFYFS_SUPER_MAGIC	0x2BAD1DEA
 
+#define STACK_END_MAGIC		0x57AC6E9D
 #endif /* __LINUX_MAGIC_H__ */

kernel/fork.c +5 -0

Index: linux-2.6.25/kernel/fork.c
===================================================================
@@ -53,6 +53,7 @@
 #include <linux/tty.h>
 #include <linux/proc_fs.h>
 #include <linux/blkdev.h>
+#include <linux/magic.h>
 
 #include <asm/pgtable.h>
 #include <asm/pgalloc.h>

@@ -167,6 +168,8 @@ static struct task_struct *dup_task_stru
 {
 	struct task_struct *tsk;
 	struct thread_info *ti;
+	unsigned long *stackend;
+
 	int err;
 
 	prepare_to_copy(orig);

@@ -192,6 +195,8 @@ static struct task_struct *dup_task_stru
 	}
 
 	setup_thread_stack(tsk, orig);
+	stackend = end_of_stack(tsk);
+	*stackend = STACK_END_MAGIC;	/* for overflow detection */
 
 #ifdef CONFIG_CC_STACKPROTECTOR
 	tsk->stack_canary = get_random_int();

Arjan van de Ven

Acked-by on v1 1/1

2008-04-22 04:21 UTC #1 lore raw

Quoted 16 lines · On Mon, 21 Apr 2008 22:44:39 -0500 Eric Sandeen <[email protected]> wrote:

Use a canary at the end of the stack to clearly indicate at oops time whether the stack has ever overflowed.
This is a very simple implementation with a couple of drawbacks:
1) a thread may legitimately use exactly up to the last word on the stack
-- but the chances of doing this and then oopsing later seem slim
2) it's possible that the stack usage isn't dense enough that the canary location could get skipped over
-- but the worst that happens is that we don't flag the overrun

Quoted 8 lines

With the code in place, an intentionally-bloated stack oops does:

BUG: unable to handle kernel paging request at ffff8103f84cc680
IP: [<ffffffff810253df>] update_curr+0x9a/0xa8
PGD 8063 PUD 0 
Thread overran stack or stack corrupted
Oops: 0000 [1] SMP

CPU 0

I like this but I wonder if it's useful to do this for the irq stacks too;

either way

Acked-by Arjan van de Ven <[email protected]>

(I'll also add detection of your printk to the kerneloops.org website)

Ingo Molnar

2008-04-22 08:44 UTC #2 lore raw

In reply to Eric Sandeen

Quoted 9 lines · Eric Sandeen

Use a canary at the end of the stack to clearly indicate at oops time whether the stack has ever overflowed.
This is a very simple implementation with a couple of drawbacks:
1) a thread may legitimately use exactly up to the last word on the stack
-- but the chances of doing this and then oopsing later seem slim

even that narrow case is a bug - an NMI might arrive at exactly that point and overflow the stack for real.

Quoted 4 lines

2) it's possible that the stack usage isn't dense enough that the canary location could get skipped over
-- but the worst that happens is that we don't flag the overrun

yeah.

Quoted 9 lines

With the code in place, an intentionally-bloated stack oops does:

BUG: unable to handle kernel paging request at ffff8103f84cc680
IP: [<ffffffff810253df>] update_curr+0x9a/0xa8
PGD 8063 PUD 0 
Thread overran stack or stack corrupted
Oops: 0000 [1] SMP

CPU 0
...

excellent. I've queued this up, it's definitely an improvement in debuggability.

we used to have something comparable in ancient kernels (it was called the stack red zone IIRC) - but it was not printed in oopses and we lost the feature somewhere anyway.

Ingo

Eric Sandeen

2008-04-22 16:44 UTC #3 lore raw

Quoted 12 lines · Ingo Molnar

Quoted 9 lines · Eric Sandeen
With the code in place, an intentionally-bloated stack oops does:
BUG: unable to handle kernel paging request at ffff8103f84cc680
IP: [<ffffffff810253df>] update_curr+0x9a/0xa8
PGD 8063 PUD 0 
Thread overran stack or stack corrupted
Oops: 0000 [1] SMP 
CPU 0
...

Quoted 2 lines

excellent. I've queued this up, it's definitely an improvement in debuggability.

Crud, just realized this probably doesn't play well with CONFIG_DEBUG_STACK_USAGE. I think it will need something like:

kernel/exit.c +2 -0

Index: linux-2.6.25-rc7/kernel/exit.c
===================================================================
@@ -826,6 +826,8 @@ static void check_stack_usage(void)
 	unsigned long *n = end_of_stack(current);
 	unsigned long free;
 
+	n++;	/* skip over canary at end */
+
 	while (*n == 0)
 		n++;
 	free = (unsigned long)n - (unsigned long)end_of_stack(current);

Testing now... want me to resend the whole patch, Ingo, or you want to just fix it up? (I'll follow up with the testing results)

Thanks,
-Eric

Eric Sandeen

2008-04-22 17:18 UTC #4 lore raw

PATCH 1/1 Fix max-stack calculators to skip canary

With the canary on the end of the stack, anything which looks for 0 to mean unused when calculating max stack excursions must skip over this non-zero magic number.

Signed-off-by Eric Sandeen <[email protected]>

---

kernel/exit.c +2 -0

Index: linux-2.6.25/kernel/exit.c
===================================================================
@@ -826,6 +826,8 @@ static void check_stack_usage(void)
 	unsigned long *n = end_of_stack(current);
 	unsigned long free;
 
+	n++;	/* skip over canary at end */
+
 	while (*n == 0)
 		n++;
 	free = (unsigned long)n - (unsigned long)end_of_stack(current);

kernel/sched.c +2 -0

Index: linux-2.6.25/kernel/sched.c
===================================================================
@@ -5190,6 +5190,8 @@ void sched_show_task(struct task_struct 
 #ifdef CONFIG_DEBUG_STACK_USAGE
 	{
 		unsigned long *n = end_of_stack(p);
+
+		n++;	/* skip over canary at end */
 		while (!*n)
 			n++;
 		free = (unsigned long)n - (unsigned long)end_of_stack(p);

Joe Perches

2008-04-22 17:34 UTC #5 lore raw

Re: [PATCH] Fix max-stack calculators to skip canary

Quoted 19 lines · Eric Sandeen

Index: linux-2.6.25/kernel/exit.c
===================================================================
kernel/exit.c +2 -0
@@ -826,6 +826,8 @@ static void check_stack_usage(void)
 	unsigned long *n = end_of_stack(current);
 	unsigned long free;
 
+	n++;	/* skip over canary at end */
+
 	while (*n == 0)
 		n++;
 	free = (unsigned long)n - (unsigned long)end_of_stack(current);
Index: linux-2.6.25/kernel/sched.c
===================================================================

kernel/sched.c void sched_show_task(struct task_struct · line 5197

Quoted 8 lines

kernel/sched.c +2 -0

@@ -5190,6 +5190,8 @@ void sched_show_task(struct task_struct 
 #ifdef CONFIG_DEBUG_STACK_USAGE
 	{
 		unsigned long *n = end_of_stack(p);
+
+		n++;	/* skip over canary at end */
 		while (!*n)
 			n++;
 		free = (unsigned long)n - (unsigned long)end_of_stack(p);

Wouldn't it be better to have exactly the same code? How about using a statement expression macro?

#define DEBUG_STACK_FREE(process) \

({ \
	unsigned long *n = end_of_stack(process); \
	do { 	/* Skip over canary */\
		n++; \
	} while (!*n); \
	(unsigned long)n - (unsigned long)end_of_stack(process); \
})

Eric Sandeen

2008-04-22 18:09 UTC #6 lore raw

Re: [PATCH] Fix max-stack calculators to skip canary

Quoted 12 lines · Joe Perches

Wouldn't it be better to have exactly the same code? How about using a statement expression macro?
#define DEBUG_STACK_FREE(process) \
({ \
	unsigned long *n = end_of_stack(process); \
	do { 	/* Skip over canary */\
		n++; \
	} while (!*n); \
	(unsigned long)n - (unsigned long)end_of_stack(process); \
})

That'd probably be good although I don't think we like macros anymore :)

-Eric

Eric Sandeen

2008-04-22 21:38 UTC #8 lore raw

PATCH v2 1/1 use canary at end of stack to indicate overruns at oops time

In reply to Eric Sandeen

(Updated with a common max-stack-used checker that knows about the canary, as suggested by Joe Perches)

Use a canary at the end of the stack to clearly indicate at oops time whether the stack has ever overflowed.

This is a very simple implementation with a couple of drawbacks:

1) a thread may legitimately use exactly up to the last word on the stack

-- but the chances of doing this and then oopsing later seem slim

2) it's possible that the stack usage isn't dense enough that the canary location could get skipped over

-- but the worst that happens is that we don't flag the overrun -- though this happens fairly often in my testing :(

With the code in place, an intentionally-bloated stack oops might do:

BUG: unable to handle kernel paging request at ffff8103f84cc680
IP: [<ffffffff810253df>] update_curr+0x9a/0xa8
PGD 8063 PUD 0 
Thread overran stack or stack corrupted
Oops: 0000 [1] SMP

CPU 0
...

... unless the stack overrun is so bad that it corrupts some other thread.

Signed-off-by Eric Sandeen <[email protected]>

---

arch/x86/mm/fault.c +7 -0

Index: linux-2.6.25/arch/x86/mm/fault.c
===================================================================
@@ -25,6 +25,7 @@
 #include <linux/kprobes.h>
 #include <linux/uaccess.h>
 #include <linux/kdebug.h>
+#include <linux/magic.h>
 
 #include <asm/system.h>
 #include <asm/desc.h>

@@ -581,6 +582,8 @@ void __kprobes do_page_fault(struct pt_r
 	unsigned long address;
 	int write, si_code;
 	int fault;
+	unsigned long *stackend;
+
 #ifdef CONFIG_X86_64
 	unsigned long flags;
 #endif

@@ -850,6 +853,10 @@ no_context:
 
 	show_fault_oops(regs, error_code, address);
 
+ 	stackend = end_of_stack(tsk);
+	if (*stackend != STACK_END_MAGIC)
+		printk(KERN_ALERT "Thread overran stack, or stack corrupted\n");
+
 	tsk->thread.cr2 = address;
 	tsk->thread.trap_no = 14;
 	tsk->thread.error_code = error_code;

include/linux/magic.h +1 -0

Index: linux-2.6.25/include/linux/magic.h
===================================================================
@@ -42,4 +42,5 @@
 #define FUTEXFS_SUPER_MAGIC	0xBAD1DEA
 #define INOTIFYFS_SUPER_MAGIC	0x2BAD1DEA
 
+#define STACK_END_MAGIC		0x57AC6E9D
 #endif /* __LINUX_MAGIC_H__ */

kernel/fork.c +5 -0

Index: linux-2.6.25/kernel/fork.c
===================================================================
@@ -53,6 +53,7 @@
 #include <linux/tty.h>
 #include <linux/proc_fs.h>
 #include <linux/blkdev.h>
+#include <linux/magic.h>
 
 #include <asm/pgtable.h>
 #include <asm/pgalloc.h>

@@ -167,6 +168,8 @@ static struct task_struct *dup_task_stru
 {
 	struct task_struct *tsk;
 	struct thread_info *ti;
+	unsigned long *stackend;
+
 	int err;
 
 	prepare_to_copy(orig);

@@ -192,6 +195,8 @@ static struct task_struct *dup_task_stru
 	}
 
 	setup_thread_stack(tsk, orig);
+	stackend = end_of_stack(tsk);
+	*stackend = STACK_END_MAGIC;	/* for overflow detection */
 
 #ifdef CONFIG_CC_STACKPROTECTOR
 	tsk->stack_canary = get_random_int();

kernel/exit.c +1 -4

Index: linux-2.6.25/kernel/exit.c
===================================================================
@@ -823,12 +823,9 @@ static void check_stack_usage(void)
 {
 	static DEFINE_SPINLOCK(low_water_lock);
 	static int lowest_to_date = THREAD_SIZE;
-	unsigned long *n = end_of_stack(current);
 	unsigned long free;
 
-	while (*n == 0)
-		n++;
-	free = (unsigned long)n - (unsigned long)end_of_stack(current);
+	free = stack_not_used(current);
 
 	if (free >= lowest_to_date)
 		return;

kernel/sched.c +1 -6

Index: linux-2.6.25/kernel/sched.c
===================================================================
@@ -5188,12 +5188,7 @@ void sched_show_task(struct task_struct 
 		printk(KERN_CONT " %016lx ", thread_saved_pc(p));
 #endif
 #ifdef CONFIG_DEBUG_STACK_USAGE
-	{
-		unsigned long *n = end_of_stack(p);
-		while (!*n)
-			n++;
-		free = (unsigned long)n - (unsigned long)end_of_stack(p);
-	}
+	free = stack_not_used(p);
 #endif
 	printk(KERN_CONT "%5lu %5d %6d\n", free,
 		task_pid_nr(p), task_pid_nr(p->real_parent));

include/linux/sched.h +13 -0

Index: linux-2.6.25/include/linux/sched.h
===================================================================
@@ -1893,6 +1893,19 @@ static inline unsigned long *end_of_stac
 
 #endif
 
+#ifdef CONFIG_DEBUG_STACK_USAGE
+static inline unsigned long stack_not_used(struct task_struct *p)
+{
+	unsigned long *n = end_of_stack(p);
+
+	do { 	/* Skip over canary */
+		n++;
+	} while (!*n);
+
+	return (unsigned long)n - (unsigned long)end_of_stack(p);
+}
+#endif
+
 /* set thread flags in other task's structures
  * - see asm/thread_info.h for TIF_xxxx flags available
  */

Harvey Harrison

2008-04-22 22:14 UTC #9 lore raw

Quoted 102 lines · Eric Sandeen

... unless the stack overrun is so bad that it corrupts some other thread.

Signed-off-by Eric Sandeen <[email protected]>

---

Index: linux-2.6.25/arch/x86/mm/fault.c
===================================================================

arch/x86/mm/fault.c +7 -0

@@ -25,6 +25,7 @@
 #include <linux/kprobes.h>
 #include <linux/uaccess.h>
 #include <linux/kdebug.h>
+#include <linux/magic.h>
 
 #include <asm/system.h>
 #include <asm/desc.h>

@@ -581,6 +582,8 @@ void __kprobes do_page_fault(struct pt_r
 	unsigned long address;
 	int write, si_code;
 	int fault;
+	unsigned long *stackend;
+
 #ifdef CONFIG_X86_64
 	unsigned long flags;
 #endif

@@ -850,6 +853,10 @@ no_context:
 
 	show_fault_oops(regs, error_code, address);
 
+ 	stackend = end_of_stack(tsk);
+	if (*stackend != STACK_END_MAGIC)
+		printk(KERN_ALERT "Thread overran stack, or stack corrupted\n");
+
 	tsk->thread.cr2 = address;
 	tsk->thread.trap_no = 14;
 	tsk->thread.error_code = error_code;

Index: linux-2.6.25/include/linux/magic.h
===================================================================

include/linux/magic.h +1 -0

@@ -42,4 +42,5 @@
 #define FUTEXFS_SUPER_MAGIC	0xBAD1DEA
 #define INOTIFYFS_SUPER_MAGIC	0x2BAD1DEA
 
+#define STACK_END_MAGIC		0x57AC6E9D
 #endif /* __LINUX_MAGIC_H__ */

Index: linux-2.6.25/kernel/fork.c
===================================================================

kernel/fork.c +5 -0

@@ -53,6 +53,7 @@
 #include <linux/tty.h>
 #include <linux/proc_fs.h>
 #include <linux/blkdev.h>
+#include <linux/magic.h>
 
 #include <asm/pgtable.h>
 #include <asm/pgalloc.h>

@@ -167,6 +168,8 @@ static struct task_struct *dup_task_stru
 {
 	struct task_struct *tsk;
 	struct thread_info *ti;
+	unsigned long *stackend;
+
 	int err;
 
 	prepare_to_copy(orig);

@@ -192,6 +195,8 @@ static struct task_struct *dup_task_stru
 	}
 
 	setup_thread_stack(tsk, orig);
+	stackend = end_of_stack(tsk);
+	*stackend = STACK_END_MAGIC;	/* for overflow detection */
 
 #ifdef CONFIG_CC_STACKPROTECTOR
 	tsk->stack_canary = get_random_int();

Index: linux-2.6.25/kernel/exit.c
===================================================================

kernel/exit.c +1 -4

@@ -823,12 +823,9 @@ static void check_stack_usage(void)
 {
 	static DEFINE_SPINLOCK(low_water_lock);
 	static int lowest_to_date = THREAD_SIZE;
-	unsigned long *n = end_of_stack(current);
 	unsigned long free;
 
-	while (*n == 0)
-		n++;
-	free = (unsigned long)n - (unsigned long)end_of_stack(current);
+	free = stack_not_used(current);
 
 	if (free >= lowest_to_date)
 		return;

Index: linux-2.6.25/kernel/sched.c
===================================================================

kernel/sched.c +0 -0

@@ -5188,12 +5188,7 @@ void sched_show_task(struct task_struct 
 		printk(KERN_CONT " %016lx ", thread_saved_pc(p));

kernel/sched.c void sched_show_task(struct task_struct · line 5192

Quoted 10 lines

kernel/sched.c +1 -6

 #endif
 #ifdef CONFIG_DEBUG_STACK_USAGE
-	{
-		unsigned long *n = end_of_stack(p);
-		while (!*n)
-			n++;
-		free = (unsigned long)n - (unsigned long)end_of_stack(p);
-	}
+	free = stack_not_used(p);
 #endif

Maybe remove the #ifdef CONFIG_DEBUG_STACK_USAGE block and move it into stack_not_used...call it debug_stack_not_used.

Quoted 13 lines

kernel/sched.c +0 -0
 	printk(KERN_CONT "%5lu %5d %6d\n", free,
 		task_pid_nr(p), task_pid_nr(p->real_parent));
Index: linux-2.6.25/include/linux/sched.h
===================================================================
include/linux/sched.h +3 -0
@@ -1893,6 +1893,19 @@ static inline unsigned long *end_of_stac
 
 #endif
 
+#ifdef CONFIG_DEBUG_STACK_USAGE
+static inline unsigned long stack_not_used(struct task_struct *p)
+{

include/linux/sched.h static inline unsigned long *end_of_stac · line 1908

Quoted 10 lines

include/linux/sched.h +10 -0

+	unsigned long *n = end_of_stack(p);
+
+	do { 	/* Skip over canary */
+		n++;
+	} while (!*n);
+
+	return (unsigned long)n - (unsigned long)end_of_stack(p);
+}
+#endif
+

static inline unsigned long debug_stack_not_used(struct task_struct *p) {

#ifdef CONFIG_DEBUG_STACK_USAGE
	unsigned long *n = end_of_stack(p);

	do { 	/* Skip over canary */
		n++;
	} while (!*n);

	return (unsigned long)n - (unsigned long)end_of_stack(p);
#else
	return $(large_value)....maybe or just some known value.

#endif
}

Also, do you expect this to ever be used outside of sched.c? Maybe just leave it as a static function there rather than an inline in the header.

Harvey

Eric Sandeen

author

2008-04-22 22:29 UTC #10 lore raw

Quoted 2 lines · Harvey Harrison

Also, do you expect this to ever be used outside of sched.c? Maybe just leave it as a static function there rather than an inline in the header.

the patch uses it in exit.c....

-Eric

Ingo Molnar

2008-04-28 17:29 UTC #7 lore raw

Re: [PATCH] Fix max-stack calculators to skip canary

In reply to Eric Sandeen

Quoted 5 lines · Eric Sandeen

With the canary on the end of the stack, anything which looks for 0 to mean unused when calculating max stack excursions must skip over this non-zero magic number.
Signed-off-by Eric Sandeen <[email protected]>

thanks Eric, applied.

Ingo

Ingo Molnar

2008-04-28 17:32 UTC #11 lore raw

In reply to Eric Sandeen

Quoted 2 lines · Eric Sandeen

(Updated with a common max-stack-used checker that knows about the canary, as suggested by Joe Perches)

thanks - picked up this updated version.

Ingo