Skip to contentSkip to conversation

git-slub crashes on the t16p

DISCUSSION Andrew Morton 3 participants 14 messages Apr 10, 2008

Subsystems (2)
Memory management Slab allocator
lore
Share & tools
lore

Andrew Morton

author maintainer
#0 lore raw

It's the tree I pulled about 12 hours ago. Quite early in boot.

crash: http://userweb.kernel.org/~akpm/p4105087.jpg
config: http://userweb.kernel.org/~akpm/config-t61p.txt
git-slub.patch: http://userweb.kernel.org/~akpm/mmotm/broken-out/git-slub.patch

A t61p is a dual-core x86_64.

I was testing with all of the -mm series up to and including git-slub.patch applied.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#1 lore raw

Hi,

Quoted 10 lines · Andrew Morton

It's the tree I pulled about 12 hours ago. Quite early in boot.

crash: http://userweb.kernel.org/~akpm/p4105087.jpg
config: http://userweb.kernel.org/~akpm/config-t61p.txt
git-slub.patch: http://userweb.kernel.org/~akpm/mmotm/broken-out/git-slub.patch

A t61p is a dual-core x86_64.

I was testing with all of the -mm series up to and including git-slub.patch applied.

You have CONFIG_NUMA enabled, so we check for NULL in inc_slabs_node():

        if (!NUMA_BUILD || n) {
                atomic_long_inc(&n->nr_slabs);
                atomic_long_add(objects, &n->total_objects);

I think I hit the same problem and it went away after make clean. Hmm...

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw
In reply to Andrew Morton
Quoted 10 lines · Andrew Morton

It's the tree I pulled about 12 hours ago. Quite early in boot.

crash: http://userweb.kernel.org/~akpm/p4105087.jpg
config: http://userweb.kernel.org/~akpm/config-t61p.txt
git-slub.patch: http://userweb.kernel.org/~akpm/mmotm/broken-out/git-slub.patch

A t61p is a dual-core x86_64.

I was testing with all of the -mm series up to and including git-slub.patch applied.

Does the following patch fix it?

mm/slub.c +1 -1
index 4b694a7..3916b4d 100644
@@ -895,7 +895,7 @@ static inline void inc_slabs_node(struct kmem_cache *s, int node, int objects)
 	 * dilemma by deferring the increment of the count during
 	 * bootstrap (see early_kmem_cache_node_alloc).
 	 */
-	if (!NUMA_BUILD || n) {
+	if (n) {
 		atomic_long_inc(&n->nr_slabs);
 		atomic_long_add(objects, &n->total_objects);
 	}

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#3 lore raw

Hi Andrew,

Quoted 10 lines · Pekka J Enberg
Quoted 10 lines

It's the tree I pulled about 12 hours ago. Quite early in boot.

crash: http://userweb.kernel.org/~akpm/p4105087.jpg
config: http://userweb.kernel.org/~akpm/config-t61p.txt
git-slub.patch: http://userweb.kernel.org/~akpm/mmotm/broken-out/git-slub.patch

A t61p is a dual-core x86_64.

I was testing with all of the -mm series up to and including git-slub.patch applied.

Quoted 1 line · Andrew Morton

Does the following patch fix it?

Okay, forget the patch. Looking at disassembly of the oops:

0000000000000000 <.text>:

   0:   eb ce                   jmp    0xffffffffffffffd0
   2:   48 89 de                mov    %rbx,%rsi
   5:   4c 89 e7                mov    %r12,%rdi
   8:   e8 38 fe ff ff          callq  0xfffffffffffffe45
   d:   b8 01 00 00 00          mov    $0x1,%eax
  12:   5b                      pop    %rbx
  13:   41 5c                   pop    %r12
  15:   c9                      leaveq
  16:   c3                      retq
  17:   c3                      retq
  18:   48 63 f6                movslq %esi,%rsi
  1b:   55                      push   %rbp
  1c:   48 8b 8c f7 20 01 00    mov    0x120(%rdi,%rsi,8),%rcx
  23:   00
  24:   48 89 e5                mov    %rsp,%rbp
  27:   48 85 c9                test   %rcx,%rcx
  2a:   74 0d                   je     0x39
  2c:   f0 48 ff 41 50          lock incq 0x50(%rcx) # %rcx == 0x64
  31:   48 63 c2                movslq %edx,%rax
  34:   f0 48 01 41 58          lock add %rax,0x58(%rcx)
  39:   c9                      leaveq
  3a:   c3                      retq
  3b:   48 8b 07                mov    (%rdi),%rax
  3e:   55                      push   %rbp
  3f:   48                      rex.W
  40:   89                      .byte 0x89

Somehow s->node[node] gets to be 0x64 which makes no sense. I checked my logs and I hit the exact same problem but it went away with "make clean". Andrew, can you please try that as well?

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw
Quoted 44 lines · Pekka Enberg
Quoted 10 lines
Quoted 10 lines

It's the tree I pulled about 12 hours ago. Quite early in boot.

crash: http://userweb.kernel.org/~akpm/p4105087.jpg
config: http://userweb.kernel.org/~akpm/config-t61p.txt
git-slub.patch: http://userweb.kernel.org/~akpm/mmotm/broken-out/git-slub.patch

A t61p is a dual-core x86_64.

I was testing with all of the -mm series up to and including git-slub.patch applied.

Quoted 1 line · Andrew Morton

Does the following patch fix it?

Okay, forget the patch. Looking at disassembly of the oops:

0000000000000000 <.text>:

   0:   eb ce                   jmp    0xffffffffffffffd0
   2:   48 89 de                mov    %rbx,%rsi
   5:   4c 89 e7                mov    %r12,%rdi
   8:   e8 38 fe ff ff          callq  0xfffffffffffffe45
   d:   b8 01 00 00 00          mov    $0x1,%eax
  12:   5b                      pop    %rbx
  13:   41 5c                   pop    %r12
  15:   c9                      leaveq
  16:   c3                      retq
  17:   c3                      retq
  18:   48 63 f6                movslq %esi,%rsi
  1b:   55                      push   %rbp
  1c:   48 8b 8c f7 20 01 00    mov    0x120(%rdi,%rsi,8),%rcx
  23:   00
  24:   48 89 e5                mov    %rsp,%rbp
  27:   48 85 c9                test   %rcx,%rcx
  2a:   74 0d                   je     0x39
  2c:   f0 48 ff 41 50          lock incq 0x50(%rcx) # %rcx == 0x64
  31:   48 63 c2                movslq %edx,%rax
  34:   f0 48 01 41 58          lock add %rax,0x58(%rcx)
  39:   c9                      leaveq
  3a:   c3                      retq
  3b:   48 8b 07                mov    (%rdi),%rax
  3e:   55                      push   %rbp
  3f:   48                      rex.W
  40:   89                      .byte 0x89
Quoted 1 line

Somehow s->node[node] gets to be 0x64 which makes no sense. I checked my logs and I hit the exact same problem but it went away with "make clean". Andrew, can you please try that as well?

I can't reproduce this on my 2-way with CONFIG_NUMA with or without numa=fake enabled and Mel Gorman tells me my tree boots on their monster numa boxen so I'm a wee bit at of a loss here.... Hmm

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#13 lore raw
In reply to Pekka Enberg
Quoted 3 lines · Pekka Enberg

Somehow s->node[node] gets to be 0x64 which makes no sense. I checked my logs and I hit the exact same problem but it went away with "make clean". Andrew, can you please try that as well?

Memory corruption? 0x64 is 100 btw.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author maintainer
#5 lore raw
In reply to Pekka Enberg
Quoted 47 lines · On Thu, 10 Apr 2008 19:35:58 +0300 Pekka Enberg <[email protected]> wrote:
Quoted 46 lines · Pekka Enberg
Quoted 12 lines
Quoted 12 lines

It's the tree I pulled about 12 hours ago. Quite early in boot.

crash: http://userweb.kernel.org/~akpm/p4105087.jpg
config: http://userweb.kernel.org/~akpm/config-t61p.txt
git-slub.patch:
http://userweb.kernel.org/~akpm/mmotm/broken-out/git-slub.patch

A t61p is a dual-core x86_64.

I was testing with all of the -mm series up to and including git-slub.patch
applied.

Quoted 1 line · Andrew Morton

Does the following patch fix it?

Okay, forget the patch. Looking at disassembly of the oops:

0000000000000000 <.text>:

   0:   eb ce                   jmp    0xffffffffffffffd0
   2:   48 89 de                mov    %rbx,%rsi
   5:   4c 89 e7                mov    %r12,%rdi
   8:   e8 38 fe ff ff          callq  0xfffffffffffffe45
   d:   b8 01 00 00 00          mov    $0x1,%eax
  12:   5b                      pop    %rbx
  13:   41 5c                   pop    %r12
  15:   c9                      leaveq
  16:   c3                      retq
  17:   c3                      retq
  18:   48 63 f6                movslq %esi,%rsi
  1b:   55                      push   %rbp
  1c:   48 8b 8c f7 20 01 00    mov    0x120(%rdi,%rsi,8),%rcx
  23:   00
  24:   48 89 e5                mov    %rsp,%rbp
  27:   48 85 c9                test   %rcx,%rcx
  2a:   74 0d                   je     0x39
  2c:   f0 48 ff 41 50          lock incq 0x50(%rcx) # %rcx == 0x64
  31:   48 63 c2                movslq %edx,%rax
  34:   f0 48 01 41 58          lock add %rax,0x58(%rcx)
  39:   c9                      leaveq
  3a:   c3                      retq
  3b:   48 8b 07                mov    (%rdi),%rax
  3e:   55                      push   %rbp
  3f:   48                      rex.W
  40:   89                      .byte 0x89
Quoted 7 lines
Quoted 3 lines

Somehow s->node[node] gets to be 0x64 which makes no sense. I checked my logs and I hit the exact same problem but it went away with "make clean". Andrew, can you please try that as well?

I can't reproduce this on my 2-way with CONFIG_NUMA with or without numa=fake enabled and Mel Gorman tells me my tree boots on their monster numa boxen so I'm a wee bit at of a loss here.... Hmm

I retested with just origin.patch and git-slub.patch. Did a `make mproper' first. Similar crash happens: http://userweb.kernel.org/~akpm/p4105088.jpg

That's within the call to atomic64_inc(), from the inc_slabs_node() here:

0xffffffff8109c348 is in new_slab (mm/slub.c:1159).
1154 flags & (GFP_RECLAIM_MASK | GFP_CONSTRAINT_MASK), node);
1155 if (!page)
1156 goto out;
1157

1158            inc_slabs_node(s, node, page->objects);
1159            page->slab = s;
1160            page->flags |= 1 << PG_slab;
1161            if (s->flags & (SLAB_DEBUG_FREE | SLAB_RED_ZONE | SLAB_POISON |

1162 SLAB_STORE_USER | SLAB_TRACE))
1163 SetSlabDebug(page);

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw
Quoted 1 line · Andrew Morton

That's within the call to atomic64_inc(), from the inc_slabs_node() here:

Right. The slab counter cleanup patch did a non equivalent transformation here.

mm/slub.c +2 -0
Index: linux-2.6/mm/slub.c
===================================================================
@@ -1174,6 +1174,8 @@
 	if (!page)
 		goto out;
 
+	/* Must use the node that the page allocator determined for us. */
+	node = page_to_nid(page);
 	inc_slabs_node(s, node, page->objects);
 	page->slab = s;
 	page->flags |= 1 << PG_slab;

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw
Quoted 11 lines · Christoph Lameter
Quoted 1 line · Andrew Morton

That's within the call to atomic64_inc(), from the inc_slabs_node() here:

Right. The slab counter cleanup patch did a non equivalent transformation here.

Index: linux-2.6/mm/slub.c
===================================================================

mm/slub.c line 1181

Quoted 8 lines
mm/slub.c +2 -0
@@ -1174,6 +1174,8 @@
 	if (!page)
 		goto out;
 
+	/* Must use the node that the page allocator determined for us. */
+	node = page_to_nid(page);
 	inc_slabs_node(s, node, page->objects);
 	page->slab = s;
 	page->flags |= 1 << PG_slab;

Actually, that's fixed in my tree since Saturday. So unfortunately I don't think this is the problem...

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#8 lore raw
Quoted 1 line · Pekka Enberg

Actually, that's fixed in my tree since Saturday. So unfortunately I don't think this is the problem...

Aah, it is, Andrew has this:

+ inc_slabs_node(s, node, page->objects);

Did I mess up my git tree or something? At least git clone gives me the correct results...

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw
Quoted 10 lines · Pekka Enberg
Quoted 2 lines · Pekka Enberg

Actually, that's fixed in my tree since Saturday. So unfortunately I don't think this is the problem...

Aah, it is, Andrew has this:

(quoted diff) +1 -0
+	inc_slabs_node(s, node, page->objects);

Did I mess up my git tree or something? At least git clone gives me the correct results...

I guess he was pulling my tree and not yours.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#11 lore raw

Hi Christoph,

Quoted 1 line · Christoph Lameter

One thing that does not make sense is that there was 0x64 in there. All unused node pointers should be NULL (they are zapped in kmem_cache_open()). So there may still be something else at play.

One thing that looks fishy to me is this the compilation of:

  return s->node[node];

to this for Andrew:

1b: 48 8b 8c f7 20 01 00 mov 0x120(%rdi,%rsi,8),%rcx

For me, the offset of ->node is 0x140 and _not_ 0x120 even with Andrew's config.

Pekka

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author maintainer
#12 lore raw
In reply to Christoph Lameter
Quoted 12 lines · On Thu, 10 Apr 2008 10:30:05 -0700 (PDT) Christoph Lameter <[email protected]>…
Quoted 1 line · Andrew Morton

That's within the call to atomic64_inc(), from the inc_slabs_node() here:

Right. The slab counter cleanup patch did a non equivalent transformation here.

Index: linux-2.6/mm/slub.c
===================================================================

mm/slub.c line 1181

Quoted 8 lines
mm/slub.c +2 -0
@@ -1174,6 +1174,8 @@
 	if (!page)
 		goto out;
 
+	/* Must use the node that the page allocator determined for us. */
+	node = page_to_nid(page);
 	inc_slabs_node(s, node, page->objects);
 	page->slab = s;
 	page->flags |= 1 << PG_slab;

That fixed it.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>