mm: kernel BUG at mm/huge_memory.c:2785!

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel I've stumbled on the following spew:

[ 1428.146261] kernel BUG at mm/huge_memory.c:2785!

[ 1428.147100] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 1428.148512] Dumping ftrace buffer:
[ 1428.149497]    (ftrace buffer empty)
[ 1428.150248] Modules linked in:
[ 1428.151164] CPU: 106 PID: 29430 Comm: trinity-c106 Tainted: G        W 
3.14.0-rc4-next-20140226-sasha-00013-g082bdac-dirty #4
[ 1428.153515] task: ffff8808906c8000 ti: ffff880890aa6000 task.ti: ffff880890aa6000
[ 1428.154274] RIP: 0010:[<mm/huge_memory.c:2785>]  [<mm/huge_memory.c:2785>] 
__split_huge_page_pmd+0x3f/0x2e0
[ 1428.154274] RSP: 0018:ffff880890aa7c98  EFLAGS: 00010287
[ 1428.154274] RAX: ffff880890f34000 RBX: ffff880b3c6791a0 RCX: 00003ffffffff000
[ 1428.154274] RDX: ffff880b3c6791a0 RSI: 00007f29869b4000 RDI: ffff8803dcbd6400
[ 1428.154274] RBP: ffff880890aa7ce8 R08: 00007f29869b5000 R09: 0000000000000000
[ 1428.154274] R10: 0000000000000000 R11: 0000000000000000 R12: ffff880b3c6791a0
[ 1428.154274] R13: 00007f2986800000 R14: ffff8803dcbd6400 R15: 00007f29869b4fff
[ 1428.154274] FS:  00007f298bb67700(0000) GS:ffff8803de800000(0000) knlGS:0000000000000000
[ 1428.154274] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 1428.154274] CR2: 00007f298b978608 CR3: 00000008907c3000 CR4: 00000000000006e0
[ 1428.154274] DR0: 8200000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 1428.154274] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
[ 1428.154274] Stack:
[ 1428.154274]  00000000001d8400 0000000000000001 ffff880890aa7cc8 ffff880890f34000
[ 1428.154274]  ffffffff876bcfe0 ffff880b3c6791a0 00007f29869b4000 00007f29869b5000
[ 1428.154274]  ffff880890aa7e18 00007f29869b4fff ffff880890aa7d78 ffffffff8127ff06
[ 1428.154274] Call Trace:
[ 1428.154274]  [<mm/memory.c:1230 mm/memory.c:1265 mm/memory.c:1290>] unmap_page_range+0x2a6/0x410
[ 1428.154274]  [<mm/memory.c:1338>] unmap_single_vma+0xf1/0x110
[ 1428.154274]  [<mm/memory.c:1365>] unmap_vmas+0x61/0xa0
[ 1428.154274]  [<mm/mmap.c:2361>] unmap_region+0xbc/0x120
[ 1428.154274]  [<include/linux/mm.h:1315 mm/mmap.c:2332 mm/mmap.c:2557>] do_munmap+0x27a/0x350
[ 1428.154274]  [<mm/mmap.c:2568>] ? vm_munmap+0x41/0x80
[ 1428.154274]  [<mm/mmap.c:2569>] vm_munmap+0x4f/0x80
[ 1428.154274]  [<mm/mmap.c:2574>] SyS_munmap+0x27/0x40
[ 1428.154274]  [<arch/x86/kernel/entry_64.S:749>] tracesys+0xdd/0xe2
[ 1428.154274] Code: e5 00 00 e0 ff 53 49 89 d4 48 83 ec 28 48 8b 47 40 48 89 45 c8 4c 3b 2f 72 11 
49 8d 95 00 00 20 00 48 89 55 c0 48 39 57 08 73 11 <0f> 0b 0f 1f 80 00 00 00 00 eb fe 66 0f 1f 44 00 
00 4c 89 e3 49
[ 1428.154274] RIP  [<mm/huge_memory.c:2785>] __split_huge_page_pmd+0x3f/0x2e0
[ 1428.154274]  RSP <ffff880890aa7c98>

Thanks,
Sasha

Quoted 6 lines · Sasha Levin

Quoted 6 lines

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel I've stumbled on the following spew:

[ 1428.146261] kernel BUG at mm/huge_memory.c:2785!

Hm, interesting.

It seems we either failed to split huge page on vma split or it materialized from under us. I don't see how it can happen:

- it seems we do the right thing with vma_adjust_trans_huge() in

     __split_vma();
   - we hold ->mmap_sem all the way from vm_munmap(). At least I don't see
     a place where we could drop it;

Andrea, any ideas?

Quoted 12 lines · Kirill A. Shutemov

Quoted 6 lines · Sasha Levin

Quoted 6 lines

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel I've stumbled on the following spew:

[ 1428.146261] kernel BUG at mm/huge_memory.c:2785!

Hm, interesting.

It seems we either failed to split huge page on vma split or it materialized from under us. I don't see how it can happen:

Quoted 6 lines

- it seems we do the right thing with vma_adjust_trans_huge() in

     __split_vma();
   - we hold ->mmap_sem all the way from vm_munmap(). At least I don't see
     a place where we could drop it;

Andrea, any ideas?

And a somewhat related issue (please correct me if I'm wrong):

[ 2208.713223] kernel BUG at mm/mlock.c:528!
[ 2208.713692] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 2208.714488] Dumping ftrace buffer:
[ 2208.715209]    (ftrace buffer empty)
[ 2208.715759] Modules linked in:
[ 2208.716206] CPU: 34 PID: 3736 Comm: trinity-c209 Tainted: G        W    3.14.0-rc5-next-20140304-sasha-00009-geaa4df0 #77
[ 2208.717637] task: ffff880ff90c8000 ti: ffff880ff90c6000 task.ti: ffff880ff90c6000
[ 2208.718742] RIP: 0010:[<ffffffff812a53d6>]  [<ffffffff812a53d6>] munlock_vma_pages_range+0x176/0x1d0
[ 2208.720107] RSP: 0018:ffff880ff90c7e08  EFLAGS: 00010206
[ 2208.720711] RAX: 00000000000001ff RBX: 000000000003f000 RCX: 0000000000000000
[ 2208.721456] RDX: 000000000000003f RSI: ffffffff8129d92d RDI: ffffffff84476115
[ 2208.721456] RBP: ffff880ff90c7ec8 R08: 0000000000000000 R09: 0000000000000000
[ 2208.721456] R10: 0000000000000001 R11: 0000000000000000 R12: fffffffffffffff2
[ 2208.721456] R13: ffff880313b41600 R14: 0000000000040000 R15: ffff880ff90c7e94
[ 2208.721456] FS:  00007f2bd5330700(0000) GS:ffff88032bc00000(0000) knlGS:0000000000000000
[ 2208.721456] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2208.721456] CR2: 0000000002767c90 CR3: 0000000ffa1d4000 CR4: 00000000000006e0
[ 2208.721456] DR0: 00007f15fe555000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2208.721456] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
[ 2208.721456] Stack:
[ 2208.721456]  0000000000000000 0000000000000000 0001880ff90c7e38 0000000000000000
[ 2208.721456]  0000000000000000 ffff880313b41600 00000000f90c7e88 0000000000000000
[ 2208.721456]  00ff880ff90c7e58 ffff880815e8cba0 ffff880ff90c7eb8 ffff880313b41600
[ 2208.721456] Call Trace:
[ 2208.721456]  [<ffffffff812a8a52>] do_munmap+0x1d2/0x350
[ 2208.721456]  [<ffffffff84473eb6>] ? down_write+0xa6/0xc0
[ 2208.721456]  [<ffffffff812a8c16>] ? vm_munmap+0x46/0x80
[ 2208.721456]  [<ffffffff812a8c24>] vm_munmap+0x54/0x80
[ 2208.721456]  [<ffffffff812a8c7c>] SyS_munmap+0x2c/0x40
[ 2208.721456]  [<ffffffff84480110>] tracesys+0xdd/0xe2
[ 2208.721456] Code: fd ff ff 4c 89 e6 48 89 c3 48 8d bd 40 ff ff ff e8 80 fa ff ff eb 2f 66 0f 1f 44 00 00 8b 45 cc 48 89 da 48 c1 ea 0c 85 d0 74 12 <0f> 0b 0f 1f 84 00 00 00 00 00 eb fe 66 0f 1f 44 00 00 ff c0 48
[ 2208.721456] RIP  [<ffffffff812a53d6>] munlock_vma_pages_range+0x176/0x1d0
[ 2208.721456]  RSP <ffff880ff90c7e08>

Thanks,
Sasha

Quoted 18 lines · Sasha Levin

Quoted 17 lines · Kirill A. Shutemov

Quoted 6 lines · Sasha Levin

Quoted 6 lines

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel I've stumbled on the following spew:

[ 1428.146261] kernel BUG at mm/huge_memory.c:2785!

Hm, interesting.

It seems we either failed to split huge page on vma split or it materialized from under us. I don't see how it can happen:

- it seems we do the right thing with vma_adjust_trans_huge() in __split_vma();
- we hold ->mmap_sem all the way from vm_munmap(). At least I don't see a place where we could drop it;

Quoted 3 lines

Quoted 1 line

Andrea, any ideas?

And a somewhat related issue (please correct me if I'm wrong):

Yeah. Looks similar. And I still have no idea how it could happened.

Do you trinity logs for the crash?

Quoted 6 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel I've stumbled on the following spew:

[ 1428.146261] kernel BUG at mm/huge_memory.c:2785!

Hm, interesting.

It seems we either failed to split huge page on vma split or it materialized from under us. I don't see how it can happen:

- it seems we do the right thing with vma_adjust_trans_huge() in

    __split_vma();
  - we hold ->mmap_sem all the way from vm_munmap(). At least I don't see
    a place where we could drop it;

Quoted 10 lines · Kirill A. Shutemov

Quoted 6 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel I've stumbled on the following spew:

[ 1428.146261] kernel BUG at mm/huge_memory.c:2785!

Hm, interesting.

Quoted 8 lines

It seems we either failed to split huge page on vma split or it materialized from under us. I don't see how it can happen:

- it seems we do the right thing with vma_adjust_trans_huge() in

     __split_vma();
   - we hold ->mmap_sem all the way from vm_munmap(). At least I don't see
     a place where we could drop it;

Enable CONFIG_DEBUG_VM may show some useful information, at least we can confirm weather rwsem_is_locked(&tlb->mm->mmap_sem) before split_huge_page_pmd().

Quoted 21 lines · Bob Liu

Quoted 18 lines · Kirill A. Shutemov

Quoted 6 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel I've stumbled on the following spew:

[ 1428.146261] kernel BUG at mm/huge_memory.c:2785!

Hm, interesting.

It seems we either failed to split huge page on vma split or it materialized from under us. I don't see how it can happen:

- it seems we do the right thing with vma_adjust_trans_huge() in

     __split_vma();
   - we hold ->mmap_sem all the way from vm_munmap(). At least I don't see
     a place where we could drop it;

Quoted 3 lines

Enable CONFIG_DEBUG_VM may show some useful information, at least we can confirm weather rwsem_is_locked(&tlb->mm->mmap_sem) before split_huge_page_pmd().

I have CONFIG_DEBUG_VM enabled and that code you're talking is not triggering, so mmap_sem is locked.

Quoted 22 lines · Sasha Levin

Quoted 21 lines · Bob Liu

Quoted 18 lines · Kirill A. Shutemov

Quoted 6 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel I've stumbled on the following spew:

[ 1428.146261] kernel BUG at mm/huge_memory.c:2785!

Hm, interesting.

It seems we either failed to split huge page on vma split or it materialized from under us. I don't see how it can happen:

- it seems we do the right thing with vma_adjust_trans_huge() in

    __split_vma();
  - we hold ->mmap_sem all the way from vm_munmap(). At least I don't see
    a place where we could drop it;

Quoted 6 lines

Quoted 3 lines

Enable CONFIG_DEBUG_VM may show some useful information, at least we can confirm weather rwsem_is_locked(&tlb->mm->mmap_sem) before split_huge_page_pmd().

I have CONFIG_DEBUG_VM enabled and that code you're talking is not triggering, so mmap_sem is locked.

Guess what. I've just hit it.

It's worth keeping in mind that this is the first time I see it.

Quoted 23 lines · Sasha Levin

Quoted 22 lines · Sasha Levin

Quoted 21 lines · Bob Liu

Quoted 18 lines · Kirill A. Shutemov

Quoted 6 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel I've stumbled on the following spew:

[ 1428.146261] kernel BUG at mm/huge_memory.c:2785!

Hm, interesting.

It seems we either failed to split huge page on vma split or it materialized from under us. I don't see how it can happen:

- it seems we do the right thing with vma_adjust_trans_huge() in

     __split_vma();
   - we hold ->mmap_sem all the way from vm_munmap(). At least I don't see
     a place where we could drop it;

Quoted 8 lines

Quoted 6 lines

Quoted 3 lines

Enable CONFIG_DEBUG_VM may show some useful information, at least we can confirm weather rwsem_is_locked(&tlb->mm->mmap_sem) before split_huge_page_pmd().

I have CONFIG_DEBUG_VM enabled and that code you're talking is not triggering, so mmap_sem is locked.

Guess what. I've just hit it.

I think this particular traceback is not a real problem: by time of exit_mm() we shouldn't race with anybody for the mm_struct.

We probably could drop ->mmap_sem later in mmput() rather then in exit_mm() to fix this false positive.

Quoted 1 line

It's worth keeping in mind that this is the first time I see it.

Hm. That's strange exit_mmap() is called without holding ->mmap_sem.