SLUB memory debugging improvements.

PATCH Superseded [email protected] 2 participants 9 messages Jun 27, 2011 – Jun 28, 2011

Subsystems (2)

Memory management Slab allocator

lore

Share & tools

lore

[email protected]

author

2011-06-27 23:20 UTC #0 lore raw

cover SLUB memory debugging improvements.

From: Ben Greear <[email protected]>

The idea is to grab a useful sized stack trace for creation/deletion points instead of just a single method.

A second patch adds a command to check if memory should be in use. This is in an attempt to track down some use-after-free bugs I'm seeing in nfs/rpc.

However, I did not exactly hit the bug I was expecting, but got this output instead. I'm not sure if my debug patches are not working right, or if this warning is just another symptom of the problem:

=============================================================================
BUG kmalloc-64: Wrong object count. Counter is 2 but counted were 1
-----------------------------------------------------------------------------

INFO: Slab 0xffffea0001e19fd0 objects=20 used=20 fp=0x (null) flags=0x20000000004080
Pid: 2349, comm: kworker/1:2 Not tainted 3.0.0-rc4+ #4

Call Trace:
 [<ffffffff81105484>] slab_err+0x8a/0x98
 [<ffffffff81077c0b>] ? register_lock_class+0x1e/0x336
 [<ffffffff81078c91>] ? mark_lock+0x2d/0x22d
 [<ffffffff81105783>] on_freelist+0x179/0x1ad
 [<ffffffffa0290601>] ? __rpc_execute+0x1ce/0x24b [sunrpc]
 [<ffffffff81106849>] verify_mem_not_deleted+0x5f/0xb3
 [<ffffffffa02906c0>] ? rpc_execute+0x42/0x42 [sunrpc]
 [<ffffffffa02981b5>] rpcb_getport_done+0x23/0x126 [sunrpc]
 [<ffffffffa02900ba>] rpc_exit_task+0x3f/0x6d [sunrpc]
 [<ffffffffa02904ab>] __rpc_execute+0x78/0x24b [sunrpc]
 [<ffffffffa02906c0>] ? rpc_execute+0x42/0x42 [sunrpc]
 [<ffffffffa02906d0>] rpc_async_schedule+0x10/0x12 [sunrpc]
 [<ffffffff810611b7>] process_one_work+0x230/0x41d
 [<ffffffff81061102>] ? process_one_work+0x17b/0x41d
 [<ffffffff81063613>] worker_thread+0x133/0x217
 [<ffffffff810634e0>] ? manage_workers+0x191/0x191
 [<ffffffff81066e10>] kthread+0x7d/0x85
 [<ffffffff81485924>] kernel_thread_helper+0x4/0x10
 [<ffffffff8147eb18>] ? retint_restore_args+0x13/0x13
 [<ffffffff81066d93>] ? __init_kthread_worker+0x56/0x56
 [<ffffffff81485920>] ? gs_change+0x13/0x13

FIX kmalloc-64: Object count adjusted.
=============================================================================
BUG kmalloc-64: Wrong object count. Counter is 1 but counted were 20
-----------------------------------------------------------------------------

INFO: Slab 0xffffea0001e19fd0 objects=20 used=1 fp=0x (null) flags=0x20000000004081
Pid: 2349, comm: kworker/1:2 Not tainted 3.0.0-rc4+ #4

Call Trace:
 [<ffffffff81105484>] slab_err+0x8a/0x98
 [<ffffffff81078c91>] ? mark_lock+0x2d/0x22d
 [<ffffffff8147e4c2>] ? _raw_spin_unlock_irqrestore+0x42/0x79
 [<ffffffff81041b91>] ? get_parent_ip+0x11/0x41
 [<ffffffff81105783>] on_freelist+0x179/0x1ad
 [<ffffffffa029889c>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
 [<ffffffffa02906c0>] ? rpc_execute+0x42/0x42 [sunrpc]
 [<ffffffff81106d4b>] free_debug_processing+0x5c/0x18e
 [<ffffffffa02906c0>] ? rpc_execute+0x42/0x42 [sunrpc]
 [<ffffffff811073b0>] __slab_free+0x57/0x150
 [<ffffffffa029889c>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
 [<ffffffffa029889c>] ? rpcb_map_release+0x3f/0x44 [sunrpc]
 [<ffffffff81107e9c>] kfree+0x107/0x13a
 [<ffffffffa029889c>] rpcb_map_release+0x3f/0x44 [sunrpc]
 [<ffffffffa028fa5a>] rpc_release_calldata+0x12/0x14 [sunrpc]
 [<ffffffffa028fc41>] rpc_free_task+0x59/0x61 [sunrpc]
 [<ffffffffa028fccb>] rpc_final_put_task+0x82/0x8a [sunrpc]
 [<ffffffffa029066f>] __rpc_execute+0x23c/0x24b [sunrpc]
 [<ffffffffa02906c0>] ? rpc_execute+0x42/0x42 [sunrpc]
 [<ffffffffa02906d0>] rpc_async_schedule+0x10/0x12 [sunrpc]
 [<ffffffff810611b7>] process_one_work+0x230/0x41d
 [<ffffffff81061102>] ? process_one_work+0x17b/0x41d
 [<ffffffff81063613>] worker_thread+0x133/0x217
 [<ffffffff810634e0>] ? manage_workers+0x191/0x191
 [<ffffffff81066e10>] kthread+0x7d/0x85
 [<ffffffff81485924>] kernel_thread_helper+0x4/0x10
 [<ffffffff8147eb18>] ? retint_restore_args+0x13/0x13
 [<ffffffff81066d93>] ? __init_kthread_worker+0x56/0x56
 [<ffffffff81485920>] ? gs_change+0x13/0x13

FIX kmalloc-64: Object count adjusted.

Ben Greear (2):
slub: Enable backtrace for create/delete points.
slub: Add method to verify memory is not deleted.

 include/linux/slab.h |    9 ++++++
 mm/slub.c            |   73 +++++++++++++++++++++++++++++++++++++++++++++-----
 2 files changed, 75 insertions(+), 7 deletions(-)

Signature · 1 line

1.7.3.4

[email protected]

author

2011-06-27 23:20 UTC #1 lore raw

PATCH 1/2 slub: Enable backtrace for create/delete points.

From: Ben Greear <[email protected]>

This patch attempts to grab a backtrace for the creation and deletion points of the slub object. When a fault is detected, we can then get a better idea of where the item was deleted.

Signed-off-by Ben Greear <[email protected]>

---
:100644 100644 35f351f... 14d0135... M mm/slub.c

 mm/slub.c |   39 ++++++++++++++++++++++++++++++++-------
 1 files changed, 32 insertions(+), 7 deletions(-)

mm/slub.c +32 -7

index 35f351f..14d0135 100644
@@ -191,8 +191,10 @@ static LIST_HEAD(slab_caches);
 /*
  * Tracking user of a slab.
  */
+#define TRACK_ADDRS_COUNT 16
 struct track {
-	unsigned long addr;	/* Called from address */
+	unsigned long caddr;
+	unsigned long addrs[TRACK_ADDRS_COUNT];	/* Called from address */
 	int cpu;		/* Was running on cpu */
 	int pid;		/* Pid context */
 	unsigned long when;	/* When did the operation occur */

@@ -420,7 +422,24 @@ static void set_track(struct kmem_cache *s, void *object,
 	struct track *p = get_track(s, object, alloc);
 
 	if (addr) {
-		p->addr = addr;
+		struct stack_trace trace;
+		int i;
+
+		trace.nr_entries = 0;
+		trace.max_entries = TRACK_ADDRS_COUNT;
+		trace.entries = p->addrs;
+		trace.skip = 3;
+		save_stack_trace(&trace);
+
+		/* See rant in lockdep.c */
+		if (trace.nr_entries != 0 &&
+		    trace.entries[trace.nr_entries - 1] == ULONG_MAX)
+			trace.nr_entries--;
+
+		for (i = trace.nr_entries; i<TRACK_ADDRS_COUNT; i++)
+			p->addrs[i] = 0;
+
+		p->caddr = addr;
 		p->cpu = smp_processor_id();
 		p->pid = current->pid;
 		p->when = jiffies;

@@ -439,11 +458,17 @@ static void init_tracking(struct kmem_cache *s, void *object)
 
 static void print_track(const char *s, struct track *t)
 {
-	if (!t->addr)
+	int i;
+	if (!t->caddr)
 		return;
 
 	printk(KERN_ERR "INFO: %s in %pS age=%lu cpu=%u pid=%d\n",
-		s, (void *)t->addr, jiffies - t->when, t->cpu, t->pid);
+		s, (void *)t->caddr, jiffies - t->when, t->cpu, t->pid);
+	for (i = 0; i < TRACK_ADDRS_COUNT; i++)
+		if (t->addrs[i])
+			printk(KERN_ERR "\t%pS\n", (void *)t->addrs[i]);
+		else
+			break;
 }
 
 static void print_tracking(struct kmem_cache *s, void *object)

@@ -3721,7 +3746,7 @@ static int add_location(struct loc_track *t, struct kmem_cache *s,
 			break;
 
 		caddr = t->loc[pos].addr;
-		if (track->addr == caddr) {
+		if (track->caddr == caddr) {
 
 			l = &t->loc[pos];
 			l->count++;

@@ -3744,7 +3769,7 @@ static int add_location(struct loc_track *t, struct kmem_cache *s,
 			return 1;
 		}
 
-		if (track->addr < caddr)
+		if (track->caddr < caddr)
 			end = pos;
 		else
 			start = pos;

@@ -3762,7 +3787,7 @@ static int add_location(struct loc_track *t, struct kmem_cache *s,
 			(t->count - pos) * sizeof(struct location));
 	t->count++;
 	l->count = 1;
-	l->addr = track->addr;
+	l->addr = track->caddr;
 	l->sum_time = age;
 	l->min_time = age;
 	l->max_time = age;

Signature · 1 line

1.7.3.4

[email protected]

author

2011-06-27 23:17 UTC #2 lore raw

PATCH 2/2 slub: Add method to verify memory is not deleted.

In reply to [email protected]

From: Ben Greear <[email protected]>

This is for tracking down suspect memory usage.

Signed-off-by Ben Greear <[email protected]>

---
:100644 100644 ad4dd1c... 35a13d2... M include/linux/slab.h
:100644 100644 14d0135... b3d7680... M mm/slub.c

 include/linux/slab.h |    9 +++++++++
 mm/slub.c            |   34 ++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 0 deletions(-)

include/linux/slab.h +9 -0

index ad4dd1c..35a13d2 100644
@@ -302,6 +302,15 @@ extern void *__kmalloc_node_track_caller(size_t, gfp_t, int, unsigned long);
 
 #endif /* CONFIG_NUMA */
 
+/** Calling this on deleted objects will print some
+ * SLUB debugging information.
+ */
+#if defined(CONFIG_SLUB) && defined(CONFIG_SLUB_DEBUG)
+extern bool verify_mem_not_deleted(const void *x);
+#else
+#define verify_mem_not_deleted(x)
+#endif
+
 /*
  * Shortcuts
  */

mm/slub.c +34 -0

index 14d0135..b3d7680 100644
@@ -2953,6 +2953,40 @@ size_t ksize(const void *object)
 }
 EXPORT_SYMBOL(ksize);
 
+#if defined(CONFIG_SLUB_DEBUG)
+bool verify_mem_not_deleted(const void *x)
+{
+	struct page *page;
+	void *object = (void *)x;
+	struct kmem_cache *s;
+	unsigned long flags;
+	bool rv = false;
+
+	if (unlikely(ZERO_OR_NULL_PTR(x)))
+		false;
+
+	local_irq_save(flags);
+
+	page = virt_to_head_page(x);
+	if (unlikely(!PageSlab(page))) {
+		BUG_ON(!PageCompound(page));
+		goto out_unlock;
+	}
+
+	s = page->slab;
+
+	if (on_freelist(s, page, object)) {
+		object_err(s, page, object, "Object is on fee-list");
+		goto out_unlock;
+	}
+	rv = true;
+out_unlock:
+	local_irq_restore(flags);
+	return rv;
+}
+EXPORT_SYMBOL(verify_mem_not_deleted);
+#endif
+
 void kfree(const void *x)
 {
 	struct page *page;

Signature · 1 line

1.7.3.4

David Rientjes

reviewer

2011-06-27 23:28 UTC #3 lore raw

Re: [PATCH 2/2] slub: Add method to verify memory is not deleted.

Quoted 4 lines · On Mon, 27 Jun 2011, [email protected] wrote:

From: Ben Greear <[email protected]>
This is for tracking down suspect memory usage.

Several things wrong with this:

- I have no idea where patch 1/2 is.

- the subject line is ambiguous, when you say memory is "deleted," I thought at first you were talking about hot-removed, but it seems like you're talking about "freed."

- what "suspect memory usage" are you adding functionality to catch and how are you doing it?

- you didn't cc the slab maintainers, Pekka Enberg and Christoph Lameter (I added them).

Quoted 13 lines

Signed-off-by Ben Greear <[email protected]>
---
:100644 100644 ad4dd1c... 35a13d2... M include/linux/slab.h
:100644 100644 14d0135... b3d7680... M mm/slub.c
 include/linux/slab.h |    9 +++++++++
 mm/slub.c            |   34 ++++++++++++++++++++++++++++++++++
 2 files changed, 43 insertions(+), 0 deletions(-)
include/linux/slab.h +0 -0
index ad4dd1c..35a13d2 100644

include/linux/slab.h extern void *__kmalloc_node_track_caller(size_t, gfp_t, int, unsigned long); · line 307

Quoted 6 lines

include/linux/slab.h +3 -0

@@ -302,6 +302,15 @@ extern void *__kmalloc_node_track_caller(size_t, gfp_t, int, unsigned long);
 
 #endif /* CONFIG_NUMA */
 
+/** Calling this on deleted objects will print some
+ * SLUB debugging information.
+ */

Ambiguous as to what it will be printing and violation of the comment style used in the kernel (see Documentation/CodingStyle).

include/linux/slab.h extern void *__kmalloc_node_track_caller(size_t, gfp_t, int, unsigned long); · line 311

Quoted 4 lines

include/linux/slab.h +4 -0

+#if defined(CONFIG_SLUB) && defined(CONFIG_SLUB_DEBUG)
+extern bool verify_mem_not_deleted(const void *x);
+#else
+#define verify_mem_not_deleted(x)

This will surely break if anybody isn't using slub or CONFIG_SLUB_DEBUG and it testing verify_mem_not_deleted(). You probably want

	static inline bool verify_mem_not_deleted(const void *x)
	{
		return false;
	}

Quoted 10 lines

include/linux/slab.h +2 -0

+#endif
+
 /*
  * Shortcuts
  */

mm/slub.c +0 -0

index 14d0135..b3d7680 100644

mm/slub.c size_t ksize(const void *object) · line 2956

Quoted 4 lines

mm/slub.c +1 -0

@@ -2953,6 +2953,40 @@ size_t ksize(const void *object)
 }
 EXPORT_SYMBOL(ksize);
 
+#if defined(CONFIG_SLUB_DEBUG)

#ifdef CONFIG_SLUB_DEBUG

mm/slub.c size_t ksize(const void *object) · line 2967

Quoted 11 lines

mm/slub.c +11 -0

+bool verify_mem_not_deleted(const void *x)
+{
+	struct page *page;
+	void *object = (void *)x;
+	struct kmem_cache *s;
+	unsigned long flags;
+	bool rv = false;
+
+	if (unlikely(ZERO_OR_NULL_PTR(x)))
+		false;
+

Did you even compile-test this?

mm/slub.c size_t ksize(const void *object) · line 2972

Quoted 5 lines

mm/slub.c +5 -0

+	local_irq_save(flags);
+
+	page = virt_to_head_page(x);
+	if (unlikely(!PageSlab(page))) {
+		BUG_ON(!PageCompound(page));

Why is there a BUG_ON() here if we didn't pass a pointer to a slab object?

mm/slub.c size_t ksize(const void *object) · line 2979

Quoted 7 lines

mm/slub.c +7 -0

+		goto out_unlock;
+	}
+
+	s = page->slab;
+
+	if (on_freelist(s, page, object)) {
+		object_err(s, page, object, "Object is on fee-list");

s/fee/free/

Quoted 10 lines

mm/slub.c +10 -0

+		goto out_unlock;
+	}
+	rv = true;
+out_unlock:
+	local_irq_restore(flags);
+	return rv;
+}
+EXPORT_SYMBOL(verify_mem_not_deleted);
+#endif
+

Ben Greear

author

2011-06-27 23:53 UTC #4 lore raw

Re: [PATCH 2/2] slub: Add method to verify memory is not deleted.

Quoted 10 lines · David Rientjes

Quoted 4 lines · On Mon, 27 Jun 2011, [email protected] wrote:
From: Ben Greear<[email protected]>
This is for tracking down suspect memory usage.
Several things wrong with this:
- I have no idea where patch 1/2 is.

It was sent to lkml...should show up soon.

Quoted 3 lines

- the subject line is ambiguous, when you say memory is "deleted," I thought at first you were talking about hot-removed, but it seems like you're talking about "freed."

Ok, I can fix that. I'm talking about memory that was freed.

Quoted 3 lines

- what "suspect memory usage" are you adding functionality to catch and how are you doing it?

I saw a case where xprt was 0x6b6b6b6b. I'm trying to figure out what freed it. I am not certain if this is a bug I introduced in some nfs changes I am testing, or if it is in a standard kernel. I am unable to reproduce it in standard kernel, but that is partly because my test case depends on my nfs patches.

The code below contains my debugging hacks, definitely not for kernel inclusion as is.

/*
* Rpcbind child task calls this callback via tk_exit. */
static void rpcb_getport_done(struct rpc_task *child, void *data)

{
	struct rpcbind_args *map = data;
	struct rpc_xprt *xprt = map->r_xprt;
	int status = child->tk_status;

	verify_mem_not_deleted(map);
	verify_mem_not_deleted(xprt);

	BUG_ON((unsigned int)(map) == (unsigned int)(0x6b6b6b6b));
	if ((unsigned int)(xprt) == (unsigned int)(0x6b6b6b6b)) {
		printk("xprt: %p is invalid, which means map: %p is likely deleted already.\n",
		       xprt, map);
		printk("map: r_owner: %p  r_prog: %u r_status: %u\n",
		       map->r_owner, map->r_prog, map->r_status);
		printk("child: %p  status: %i\n", child, status);
		BUG_ON(1);
	}
	BUG_ON((unsigned int)(status) == (unsigned int)(0x6b6b6b6b));

Quoted 2 lines

- you didn't cc the slab maintainers, Pekka Enberg and Christoph Lameter (I added them).

Thanks for that.

Quoted 15 lines

Quoted 3 lines
+/** Calling this on deleted objects will print some
+ * SLUB debugging information.
+ */
Ambiguous as to what it will be printing and violation of the comment style used in the kernel (see Documentation/CodingStyle).
include/linux/slab.h extern void *__kmalloc_node_track_caller(size_t, gfp_t, int, unsigned long); · line 311
Quoted 4 lines
include/linux/slab.h +4 -0
+#if defined(CONFIG_SLUB)&&  defined(CONFIG_SLUB_DEBUG)
+extern bool verify_mem_not_deleted(const void *x);
+#else
+#define verify_mem_not_deleted(x)
This will surely break if anybody isn't using slub or CONFIG_SLUB_DEBUG and it testing verify_mem_not_deleted(). You probably want

Quoted 4 lines

	static inline bool verify_mem_not_deleted(const void *x)
	{
		return false;
	}

Ok.

Quoted 18 lines

Quoted 11 lines
include/linux/slab.h +2 -0
+#endif
+
  /*
   * Shortcuts
   */
mm/slub.c +0 -0
index 14d0135..b3d7680 100644
mm/slub.c size_t ksize(const void *object) · line 2956
Quoted 4 lines
mm/slub.c +1 -0
@@ -2953,6 +2953,40 @@ size_t ksize(const void *object)
  }
  EXPORT_SYMBOL(ksize);
 
+#if defined(CONFIG_SLUB_DEBUG)
#ifdef CONFIG_SLUB_DEBUG

Quoted 13 lines

mm/slub.c size_t ksize(const void *object) · line 2967

Quoted 11 lines

mm/slub.c +11 -0

+bool verify_mem_not_deleted(const void *x)
+{
+	struct page *page;
+	void *object = (void *)x;
+	struct kmem_cache *s;
+	unsigned long flags;
+	bool rv = false;
+
+	if (unlikely(ZERO_OR_NULL_PTR(x)))
+		false;
+

Did you even compile-test this?

Well yes, but obviously I'm having a bad day!

Aside from the lame mistakes in my patch..is the general approach usable?

Quoted 7 lines

mm/slub.c size_t ksize(const void *object) · line 2972
Quoted 5 lines
mm/slub.c +5 -0
+	local_irq_save(flags);
+
+	page = virt_to_head_page(x);
+	if (unlikely(!PageSlab(page))) {
+		BUG_ON(!PageCompound(page));
Why is there a BUG_ON() here if we didn't pass a pointer to a slab object?

Just copying the kfree code. Just return 'true' in this case?

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Ben Greear

author

2011-06-28 00:19 UTC #5 lore raw

Re: [PATCH 2/2] slub: Add method to verify memory is not deleted.

Quoted 6 lines · Ben Greear

Quoted 1 line
Did you even compile-test this?
Well yes, but obviously I'm having a bad day!
Aside from the lame mistakes in my patch..is the general approach usable?

I was missing slab_lock() around the on_freelist() call, that is probably why I got the printouts about object count being off.

Fixing that and will let it cook overnight.

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

David Rientjes

reviewer

2011-06-28 00:20 UTC #6 lore raw

Re: [PATCH 2/2] slub: Add method to verify memory is not deleted.

In reply to Ben Greear

Quoted 2 lines · Ben Greear

I saw a case where xprt was 0x6b6b6b6b. I'm trying to figure out what freed it.

And echo 1 > /sys/kernel/slab/<cache>/store_user doesn't help?

Ben Greear

author

2011-06-28 00:25 UTC #7 lore raw

Re: [PATCH 2/2] slub: Add method to verify memory is not deleted.

Quoted 6 lines · David Rientjes

Quoted 2 lines · Ben Greear
I saw a case where xprt was 0x6b6b6b6b. I'm trying to figure out what freed it.
And echo 1> /sys/kernel/slab/<cache>/store_user doesn't help?

That gives one method, right? The rpc stuff is freed at the bottom of a complicated callback chain, and the interesting stuff is what caused the memory to be freed, not the actually free method.

In previous network (ath9k) hacking I had the same trouble..I think for most cases you need a full or mostly full stack to make use of the slub debug logic.

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com

Ben Greear

author

2011-06-28 03:47 UTC #8 lore raw

Re: [PATCH 2/2] slub: Add method to verify memory is not deleted.

Quoted 8 lines · Ben Greear

Quoted 6 lines · David Rientjes
Quoted 2 lines · Ben Greear
I saw a case where xprt was 0x6b6b6b6b. I'm trying to figure out what freed it.
And echo 1> /sys/kernel/slab/<cache>/store_user doesn't help?

Quoted 7 lines

That gives one method, right? The rpc stuff is freed at the bottom of a complicated callback chain, and the interesting stuff is what caused the memory to be freed, not the actually free method.
In previous network (ath9k) hacking I had the same trouble..I think for most cases you need a full or mostly full stack to make use of the slub debug logic.

Hit my bug after a while...here's some example output with the slub patches applied:

=============================================================================
BUG kmalloc-64: Object is on free-list
-----------------------------------------------------------------------------

INFO: Allocated in rpcb_getport_async+0x39c/0x5a5 [sunrpc] age=381 cpu=3 pid=3750

	__slab_alloc+0x348/0x3ba
	kmem_cache_alloc_trace+0x67/0xe7
	rpcb_getport_async+0x39c/0x5a5 [sunrpc]
	call_bind+0x70/0x75 [sunrpc]
	__rpc_execute+0x78/0x24b [sunrpc]
	rpc_execute+0x3d/0x42 [sunrpc]
	rpc_run_task+0x79/0x81 [sunrpc]
	rpc_call_sync+0x3f/0x60 [sunrpc]
	rpc_ping+0x42/0x58 [sunrpc]
	rpc_create+0x4aa/0x527 [sunrpc]
	nfs_create_rpc_client+0xb1/0xf6 [nfs]
	nfs_init_client+0x3b/0x7d [nfs]
	nfs_get_client+0x453/0x5ab [nfs]
	nfs_create_server+0x10b/0x437 [nfs]
	nfs_fs_mount+0x4ca/0x708 [nfs]
	mount_fs+0x6b/0x152
INFO: Freed in rpcb_map_release+0x3f/0x44 [sunrpc] age=30 cpu=2 pid=29049
	__slab_free+0x57/0x150
	kfree+0x107/0x13a
	rpcb_map_release+0x3f/0x44 [sunrpc]
	rpc_release_calldata+0x12/0x14 [sunrpc]
	rpc_free_task+0x59/0x61 [sunrpc]
	rpc_final_put_task+0x82/0x8a [sunrpc]
	__rpc_execute+0x23c/0x24b [sunrpc]
	rpc_async_schedule+0x10/0x12 [sunrpc]
	process_one_work+0x230/0x41d
	worker_thread+0x133/0x217
	kthread+0x7d/0x85
	kernel_thread_helper+0x4/0x10

INFO: Slab 0xffffea00029aa470 objects=20 used=9 fp=0xffff8800be7830d8 flags=0x20000000004081
INFO: Object 0xffff8800be7830d8 @offset=4312 fp=0xffff8800be7827a8

Bytes b4 0xffff8800be7830c8:  87 a8 96 00 01 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a .�......ZZZZZZZZ
  Object 0xffff8800be7830d8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
  Object 0xffff8800be7830e8:  6b 6b 6b 6b 01 08 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkk..kkkkkkkkkk
  Object 0xffff8800be7830f8:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
  Object 0xffff8800be783108:  6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b a5 kkkkkkkkkkkkkkk�
 Redzone 0xffff8800be783118:  bb bb bb bb bb bb bb bb                         ��������
 Padding 0xffff8800be783258:  5a 5a 5a 5a 5a 5a 5a 5a                         ZZZZZZZZ
Pid: 29049, comm: kworker/2:2 Not tainted 3.0.0-rc4+ #8
Call Trace:
 [<ffffffff811055c3>] print_trailer+0x131/0x13a
 [<ffffffff81105601>] object_err+0x35/0x3e
 [<ffffffff8110746f>] verify_mem_not_deleted+0x7a/0xb7
 [<ffffffffa02851b5>] rpcb_getport_done+0x23/0x126 [sunrpc]
 [<ffffffffa027d0ba>] rpc_exit_task+0x3f/0x6d [sunrpc]
 [<ffffffffa027d4ab>] __rpc_execute+0x78/0x24b [sunrpc]
 [<ffffffffa027d6c0>] ? rpc_execute+0x42/0x42 [sunrpc]
 [<ffffffffa027d6d0>] rpc_async_schedule+0x10/0x12 [sunrpc]
 [<ffffffff810611b7>] process_one_work+0x230/0x41d
 [<ffffffff81061102>] ? process_one_work+0x17b/0x41d
 [<ffffffff81063613>] worker_thread+0x133/0x217
 [<ffffffff810634e0>] ? manage_workers+0x191/0x191
 [<ffffffff81066e10>] kthread+0x7d/0x85
 [<ffffffff81485924>] kernel_thread_helper+0x4/0x10
 [<ffffffff8147eb18>] ? retint_restore_args+0x13/0x13
 [<ffffffff81066d93>] ? __init_kthread_worker+0x56/0x56
 [<ffffffff81485920>] ? gs_change+0x13/0x13
general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
CPU 2
Modules linked in: xt_addrtype xt_TPROXY nf_tproxy_core xt_socket nf_defrag_ipv6 xt_set ip_set nfnetlink xt_connlimit 8021q garp ip6table_filter ip6_tables macvlan ebtable_nat ebtables fuse pktgen iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi stp llc nfs lockd fscache auth_rpcgss nfs_acl sunrpc ipv6 kvm_intel kvm uinput i5k_amb i5000_edac edac_core iTCO_wdt e1000e iTCO_vendor_support ioatdma microcode pcspkr i2c_i801 shpchp dca floppy radeon ttm drm_kms_helper drm hwmon i2c_algo_bit i2c_core [last unloaded: xt_connmark]

Pid: 29049, comm: kworker/2:2 Not tainted 3.0.0-rc4+ #8 Supermicro X7DBU/X7DBU

RIP: 0010:[<ffffffff81105eae>]  [<ffffffff81105eae>] virt_to_head_page+0x1e/0x2c
RSP: 0000:ffff8801021d3ce0  EFLAGS: 00010003
RAX: 0177e39bf7f7f7d0 RBX: ffff8800be7830d8 RCX: ffff8800be7830d8
RDX: ffffea0000000000 RSI: ffff8801021d3f58 RDI: 6b6b6b6b6b6b6b6b
RBP: ffff8801021d3ce0 R08: ffff8800be7830d8 R09: ffff8801021d3990
R10: 0000000000000001 R11: 0000000000000078 R12: 6b6b6b6b6b6b6b6b
R13: 0000000000000202 R14: 0000000000000001 R15: ffffffffa027d6c0
FS:  0000000000000000(0000) GS:ffff88012fc80000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000311421b080 CR3: 0000000001a03000 CR4: 00000000000006e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
Process kworker/2:2 (pid: 29049, threadinfo ffff8801021d2000, task ffff8801288ab3f0)
Stack:
 ffff8801021d3d10 ffffffff81107431 ffff8800be7830d8 0000000000000000
 ffff8800c7635d40 6b6b6b6b6b6b6b6b ffff8801021d3d40 ffffffffa02851bd
 ffff8800c7635d40 ffff8800c7635db0 0000000000000001 0000000000000000
Call Trace:
 [<ffffffff81107431>] verify_mem_not_deleted+0x3c/0xb7
 [<ffffffffa02851bd>] rpcb_getport_done+0x2b/0x126 [sunrpc]
 [<ffffffffa027d0ba>] rpc_exit_task+0x3f/0x6d [sunrpc]
 [<ffffffffa027d4ab>] __rpc_execute+0x78/0x24b [sunrpc]
 [<ffffffffa027d6c0>] ? rpc_execute+0x42/0x42 [sunrpc]
 [<ffffffffa027d6d0>] rpc_async_schedule+0x10/0x12 [sunrpc]
 [<ffffffff810611b7>] process_one_work+0x230/0x41d
 [<ffffffff81061102>] ? process_one_work+0x17b/0x41d
 [<ffffffff81063613>] worker_thread+0x133/0x217
 [<ffffffff810634e0>] ? manage_workers+0x191/0x191
 [<ffffffff81066e10>] kthread+0x7d/0x85
 [<ffffffff81485924>] kernel_thread_helper+0x4/0x10
 [<ffffffff8147eb18>] ? retint_restore_args+0x13/0x13
 [<ffffffff81066d93>] ? __init_kthread_worker+0x56/0x56
 [<ffffffff81485920>] ? gs_change+0x13/0x13
Code: 02 00 00 3d 00 02 00 00 0f 4f c2 c9 c3 55 48 89 e5 e8 eb e9 f2 ff 48 c1 e8 0c 48 ba 00 00 00 00 00 ea ff ff 48 6b c0 38 48 01 d0
 8b 10 66 85 d2 79 04 48 8b 40 10 c9 c3 55 48 89 e5 41 57 41
RIP  [<ffffffff81105eae>] virt_to_head_page+0x1e/0x2c
 RSP <ffff8801021d3ce0>
---[ end trace 4cb7eac5b28823fd ]---

Thanks,
Ben

Signature · 2 lines

Ben Greear <[email protected]>
Candela Technologies Inc  http://www.candelatech.com