Skip to contentSkip to conversation

mm: kernel BUG at include/linux/swapops.h:131!

DISCUSSION Sasha Levin 6 participants 16 messages Dec 18, 2013 – Jan 4, 2014

Subsystems (2)
Memory management MM swap
lore
Share & tools
lore

Sasha Levin

author
#0 lore raw

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel, I've stumbled on the following spew.

The code is in zap_pte_range():

                if (!non_swap_entry(entry))
                        rss[MM_SWAPENTS]--;
                else if (is_migration_entry(entry)) {
                        struct page *page;

                        page = migration_entry_to_page(entry);	<==== HERE

                        if (PageAnon(page))
                                rss[MM_ANONPAGES]--;
                        else
                                rss[MM_FILEPAGES]--;
[ 2622.589064] kernel BUG at include/linux/swapops.h:131!
[ 2622.589064] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 2622.589064] Dumping ftrace buffer:
[ 2622.589064]    (ftrace buffer empty)
[ 2622.589064] Modules linked in:
[ 2622.589064] CPU: 9 PID: 15984 Comm: trinity-child16 Tainted: G        W    3.13.0-rc
4-next-20131217-sasha-00013-ga878504-dirty #4150
[ 2622.589064] task: ffff88168346b000 ti: ffff8816561d8000 task.ti: ffff8816561d8000
[ 2622.589064] RIP: 0010:[<ffffffff8127c730>]  [<ffffffff8127c730>] zap_pte_range+0x360
/0x4a0
[ 2622.589064] RSP: 0018:ffff8816561d9c18  EFLAGS: 00010246
[ 2622.589064] RAX: ffffea00736a6600 RBX: ffff88200299d068 RCX: 0000000000000009
[ 2622.589064] RDX: 022fffff80380000 RSI: ffffea0000000000 RDI: 3c00000001cda998
[ 2622.589064] RBP: ffff8816561d9cb8 R08: 0000000000000000 R09: 0000000000000000
[ 2622.589064] R10: 0000000000000001 R11: 0000000000000000 R12: 00007fc7ee20d000
[ 2622.589064] R13: ffff8816561d9de8 R14: 000000039b53303c R15: 00007fc7ee29b000
[ 2622.589064] FS:  00007fc7eeceb700(0000) GS:ffff882011a00000(0000) knlGS:000000000000
0000
[ 2622.589064] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2622.589064] CR2: 000000000068c000 CR3: 0000000005c26000 CR4: 00000000000006e0
[ 2622.589064] Stack:
[ 2622.589064]  ffff8816561d9c58 0000000000000286 ffff88168327b060 ffff88168327b060
[ 2622.589064]  00007fc700000160 ffff881667067b88 00000000c8f4b120 00ff88168327b060
[ 2622.589064]  ffff88168327b060 ffff8820051a8600 0000000000000000 ffff88168327b000
[ 2622.589064] Call Trace:
[ 2622.589064]  [<ffffffff8127cc5e>] unmap_page_range+0x3ee/0x400
[ 2622.589064]  [<ffffffff8127cd71>] unmap_single_vma+0x101/0x120
[ 2622.589064]  [<ffffffff8127cdf1>] unmap_vmas+0x61/0xa0
[ 2622.589064]  [<ffffffff81283980>] exit_mmap+0xd0/0x170
[ 2622.589064]  [<ffffffff8112d430>] mmput+0x70/0xe0
[ 2622.589064]  [<ffffffff8113144d>] exit_mm+0x18d/0x1a0
[ 2622.589064]  [<ffffffff811defb5>] ? acct_collect+0x175/0x1b0
[ 2622.589064]  [<ffffffff8113389f>] do_exit+0x24f/0x500
[ 2622.589064]  [<ffffffff81133bf9>] do_group_exit+0xa9/0xe0
[ 2622.589064]  [<ffffffff81133c47>] SyS_exit_group+0x17/0x20
[ 2622.589064]  [<ffffffff843a6150>] tracesys+0xdd/0xe2
[ 2622.589064] Code: 83 f8 1f 75 46 48 b8 ff ff ff ff ff ff ff 01 48 be 00 00 00 00 00 ea ff ff 48 21 f8 48 c1 e0 06 48 01 f0 48 8b 10 80 e2 01 75 0a <0f> 0b 66 0f 1f 44 00 00 eb fe f6 40 08 01 74 05 ff 4d c4 eb 0b
[ 2622.589064] RIP  [<ffffffff8127c730>] zap_pte_range+0x360/0x4a0
[ 2622.589064]  RSP <ffff8816561d9c18>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#1 lore raw
Quoted 7 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel, I've stumbled on the following spew.

The code is in zap_pte_range():

Quoted 11 lines
                if (!non_swap_entry(entry))
                        rss[MM_SWAPENTS]--;
                else if (is_migration_entry(entry)) {
                        struct page *page;

                        page = migration_entry_to_page(entry);	<==== HERE

                        if (PageAnon(page))
                                rss[MM_ANONPAGES]--;
                        else
                                rss[MM_FILEPAGES]--;

This I think is my issue, I'll take a look, thanks Sasha!

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#2 lore raw
Quoted 13 lines · Cyrill Gorcunov
Quoted 12 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel, I've stumbled on the following spew.

The code is in zap_pte_range():

                if (!non_swap_entry(entry))
                        rss[MM_SWAPENTS]--;
                else if (is_migration_entry(entry)) {
                        struct page *page;
Quoted 8 lines
Quoted 6 lines
                        page = migration_entry_to_page(entry);	<==== HERE

                        if (PageAnon(page))
                                rss[MM_ANONPAGES]--;
                        else
                                rss[MM_FILEPAGES]--;

This I think is my issue, I'll take a look, thanks Sasha!

Ah, no. I thought it somehow related to dirty tracking, but it's not, different issue.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#3 lore raw
In reply to Sasha Levin

Ping?

I've also Cc'ed the "this page shouldn't be locked at all" team.

Quoted 53 lines · Sasha Levin

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel, I've stumbled on the following spew.

The code is in zap_pte_range():

                 if (!non_swap_entry(entry))
                         rss[MM_SWAPENTS]--;
                 else if (is_migration_entry(entry)) {
                         struct page *page;

                         page = migration_entry_to_page(entry);    <==== HERE

                         if (PageAnon(page))
                                 rss[MM_ANONPAGES]--;
                         else
                                 rss[MM_FILEPAGES]--;
[ 2622.589064] kernel BUG at include/linux/swapops.h:131!
[ 2622.589064] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 2622.589064] Dumping ftrace buffer:
[ 2622.589064]    (ftrace buffer empty)
[ 2622.589064] Modules linked in:
[ 2622.589064] CPU: 9 PID: 15984 Comm: trinity-child16 Tainted: G        W    3.13.0-rc
4-next-20131217-sasha-00013-ga878504-dirty #4150
[ 2622.589064] task: ffff88168346b000 ti: ffff8816561d8000 task.ti: ffff8816561d8000
[ 2622.589064] RIP: 0010:[<ffffffff8127c730>]  [<ffffffff8127c730>] zap_pte_range+0x360
/0x4a0
[ 2622.589064] RSP: 0018:ffff8816561d9c18  EFLAGS: 00010246
[ 2622.589064] RAX: ffffea00736a6600 RBX: ffff88200299d068 RCX: 0000000000000009
[ 2622.589064] RDX: 022fffff80380000 RSI: ffffea0000000000 RDI: 3c00000001cda998
[ 2622.589064] RBP: ffff8816561d9cb8 R08: 0000000000000000 R09: 0000000000000000
[ 2622.589064] R10: 0000000000000001 R11: 0000000000000000 R12: 00007fc7ee20d000
[ 2622.589064] R13: ffff8816561d9de8 R14: 000000039b53303c R15: 00007fc7ee29b000
[ 2622.589064] FS:  00007fc7eeceb700(0000) GS:ffff882011a00000(0000) knlGS:000000000000
0000
[ 2622.589064] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2622.589064] CR2: 000000000068c000 CR3: 0000000005c26000 CR4: 00000000000006e0
[ 2622.589064] Stack:
[ 2622.589064]  ffff8816561d9c58 0000000000000286 ffff88168327b060 ffff88168327b060
[ 2622.589064]  00007fc700000160 ffff881667067b88 00000000c8f4b120 00ff88168327b060
[ 2622.589064]  ffff88168327b060 ffff8820051a8600 0000000000000000 ffff88168327b000
[ 2622.589064] Call Trace:
[ 2622.589064]  [<ffffffff8127cc5e>] unmap_page_range+0x3ee/0x400
[ 2622.589064]  [<ffffffff8127cd71>] unmap_single_vma+0x101/0x120
[ 2622.589064]  [<ffffffff8127cdf1>] unmap_vmas+0x61/0xa0
[ 2622.589064]  [<ffffffff81283980>] exit_mmap+0xd0/0x170
[ 2622.589064]  [<ffffffff8112d430>] mmput+0x70/0xe0
[ 2622.589064]  [<ffffffff8113144d>] exit_mm+0x18d/0x1a0
[ 2622.589064]  [<ffffffff811defb5>] ? acct_collect+0x175/0x1b0
[ 2622.589064]  [<ffffffff8113389f>] do_exit+0x24f/0x500
Quoted 8 lines
[ 2622.589064]  [<ffffffff81133bf9>] do_group_exit+0xa9/0xe0
[ 2622.589064]  [<ffffffff81133c47>] SyS_exit_group+0x17/0x20
[ 2622.589064]  [<ffffffff843a6150>] tracesys+0xdd/0xe2
[ 2622.589064] Code: 83 f8 1f 75 46 48 b8 ff ff ff ff ff ff ff 01 48 be 00 00 00 00 00 ea ff ff 48
21 f8 48 c1 e0 06 48 01 f0 48 8b 10 80 e2 01 75 0a <0f> 0b 66 0f 1f 44 00 00 eb fe f6 40 08 01 74 05
ff 4d c4 eb 0b
[ 2622.589064] RIP  [<ffffffff8127c730>] zap_pte_range+0x360/0x4a0
[ 2622.589064]  RSP <ffff8816561d9c18>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#4 lore raw
Quoted 3 lines · Sasha Levin

Ping?

I've also Cc'ed the "this page shouldn't be locked at all" team.

Hello,

I can't find the reason of this problem.
If it is reproducible, how about bisecting?

Thanks.

Quoted 64 lines · Sasha Levin
Quoted 53 lines

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel, I've stumbled on the following spew.

The code is in zap_pte_range():

                if (!non_swap_entry(entry))
                        rss[MM_SWAPENTS]--;
                else if (is_migration_entry(entry)) {
                        struct page *page;

                        page = migration_entry_to_page(entry);    <==== HERE

                        if (PageAnon(page))
                                rss[MM_ANONPAGES]--;
                        else
                                rss[MM_FILEPAGES]--;
[ 2622.589064] kernel BUG at include/linux/swapops.h:131!
[ 2622.589064] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 2622.589064] Dumping ftrace buffer:
[ 2622.589064]    (ftrace buffer empty)
[ 2622.589064] Modules linked in:
[ 2622.589064] CPU: 9 PID: 15984 Comm: trinity-child16 Tainted: G        W    3.13.0-rc
4-next-20131217-sasha-00013-ga878504-dirty #4150
[ 2622.589064] task: ffff88168346b000 ti: ffff8816561d8000 task.ti: ffff8816561d8000
[ 2622.589064] RIP: 0010:[<ffffffff8127c730>]  [<ffffffff8127c730>] zap_pte_range+0x360
/0x4a0
[ 2622.589064] RSP: 0018:ffff8816561d9c18  EFLAGS: 00010246
[ 2622.589064] RAX: ffffea00736a6600 RBX: ffff88200299d068 RCX: 0000000000000009
[ 2622.589064] RDX: 022fffff80380000 RSI: ffffea0000000000 RDI: 3c00000001cda998
[ 2622.589064] RBP: ffff8816561d9cb8 R08: 0000000000000000 R09: 0000000000000000
[ 2622.589064] R10: 0000000000000001 R11: 0000000000000000 R12: 00007fc7ee20d000
[ 2622.589064] R13: ffff8816561d9de8 R14: 000000039b53303c R15: 00007fc7ee29b000
[ 2622.589064] FS:  00007fc7eeceb700(0000) GS:ffff882011a00000(0000) knlGS:000000000000
0000
[ 2622.589064] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2622.589064] CR2: 000000000068c000 CR3: 0000000005c26000 CR4: 00000000000006e0
[ 2622.589064] Stack:
[ 2622.589064]  ffff8816561d9c58 0000000000000286 ffff88168327b060 ffff88168327b060
[ 2622.589064]  00007fc700000160 ffff881667067b88 00000000c8f4b120 00ff88168327b060
[ 2622.589064]  ffff88168327b060 ffff8820051a8600 0000000000000000 ffff88168327b000
[ 2622.589064] Call Trace:
[ 2622.589064]  [<ffffffff8127cc5e>] unmap_page_range+0x3ee/0x400
[ 2622.589064]  [<ffffffff8127cd71>] unmap_single_vma+0x101/0x120
[ 2622.589064]  [<ffffffff8127cdf1>] unmap_vmas+0x61/0xa0
[ 2622.589064]  [<ffffffff81283980>] exit_mmap+0xd0/0x170
[ 2622.589064]  [<ffffffff8112d430>] mmput+0x70/0xe0
[ 2622.589064]  [<ffffffff8113144d>] exit_mm+0x18d/0x1a0
[ 2622.589064]  [<ffffffff811defb5>] ? acct_collect+0x175/0x1b0
[ 2622.589064]  [<ffffffff8113389f>] do_exit+0x24f/0x500
Quoted 8 lines
[ 2622.589064]  [<ffffffff81133bf9>] do_group_exit+0xa9/0xe0
[ 2622.589064]  [<ffffffff81133c47>] SyS_exit_group+0x17/0x20
[ 2622.589064]  [<ffffffff843a6150>] tracesys+0xdd/0xe2
[ 2622.589064] Code: 83 f8 1f 75 46 48 b8 ff ff ff ff ff ff ff 01 48 be 00 00 00 00 00 ea ff ff 48
21 f8 48 c1 e0 06 48 01 f0 48 8b 10 80 e2 01 75 0a <0f> 0b 66 0f 1f 44 00 00 eb fe f6 40 08 01 74 05
ff 4d c4 eb 0b
[ 2622.589064] RIP  [<ffffffff8127c730>] zap_pte_range+0x360/0x4a0
[ 2622.589064]  RSP <ffff8816561d9c18>
Quoted 5 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#5 lore raw
Quoted 8 lines · Joonsoo Kim
Quoted 3 lines · Sasha Levin
Quoted 3 lines

Ping?

I've also Cc'ed the "this page shouldn't be locked at all" team.

Hello,

I can't find the reason of this problem.
If it is reproducible, how about bisecting?

While it reproduces under fuzzing it's pretty hard to bisect it with the amount of issues uncovered by trinity recently.

I can add any debug code to the site of the BUG if that helps.

Thanks,
Sasha

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#6 lore raw
Quoted 7 lines · Sasha Levin
Quoted 6 lines · Joonsoo Kim
Quoted 3 lines · Sasha Levin
Quoted 3 lines

Ping?

I've also Cc'ed the "this page shouldn't be locked at all" team.

Hello,

Quoted 7 lines
Quoted 2 lines

I can't find the reason of this problem.
If it is reproducible, how about bisecting?

While it reproduces under fuzzing it's pretty hard to bisect it with the amount of issues uncovered by trinity recently.

I can add any debug code to the site of the BUG if that helps.

Good!
It will be helpful to add dump_page() in migration_entry_to_page().

Thanks.

--------8<------

include/linux/swapops.h +3 -0
index c0f7526..f695abc 100644
@@ -3,6 +3,7 @@
 
 #include <linux/radix-tree.h>
 #include <linux/bug.h>
+#include <linux/mm.h>
 
 /*
  * swapcache pages are stored in the swapper_space radix tree.  We want to

@@ -128,6 +129,8 @@ static inline struct page *migration_entry_to_page(swp_entry_t entry)
         * Any use of migration entries may only occur while the
         * corresponding page is locked
         */
+       if (!PageLocked(p))
+               dump_page(p);
        BUG_ON(!PageLocked(p));
        return p;
 }

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#7 lore raw
Quoted 14 lines · Joonsoo Kim
Quoted 13 lines · Sasha Levin
Quoted 8 lines · Joonsoo Kim
Quoted 3 lines · Sasha Levin
Quoted 3 lines

Ping?

I've also Cc'ed the "this page shouldn't be locked at all" team.

Hello,

I can't find the reason of this problem.
If it is reproducible, how about bisecting?

While it reproduces under fuzzing it's pretty hard to bisect it with the amount of issues uncovered by trinity recently.

Quoted 7 lines
Quoted 1 line

I can add any debug code to the site of the BUG if that helps.

Good!
It will be helpful to add dump_page() in migration_entry_to_page().

Thanks.

Minchan teaches me that there is possible race condition between fork and migration.

Please consider following situation.

Process A (do migration) Process B (parents) Process C (child)

try_to_unmap() for migration <begin> fork
setup migration entry to B's vma
...
try_to_unmap() for migration <end>
move_to_new_page()

link new vma
into interval tree
remove_migration_ptes() <begin>
check and clear migration entry on C's vma
... copy_one_pte:
... now, B and C have migration entry
...
...
check and clear migration entry on B's vma
...
...
remove_migration_ptes() <end>

Eventually, migration entry on C's vma is left. And then, when C exits, above BUG_ON() can be triggered.

I'm not sure the I am right, so please think of it together. :) And I'm not sure again that above assumption is related to this trigger report, since this may exist for a long time.

So my question to mm folks is is above assumption possible and do we have any protection mechanism on this race?

Thanks.

#8 lore raw

Hi Joonsoo,

Quoted 45 lines · Joonsoo Kim
Quoted 14 lines · Joonsoo Kim
Quoted 13 lines · Sasha Levin
Quoted 8 lines · Joonsoo Kim
Quoted 3 lines · Sasha Levin
Quoted 3 lines

Ping?

I've also Cc'ed the "this page shouldn't be locked at all" team.

Hello,

I can't find the reason of this problem.
If it is reproducible, how about bisecting?

While it reproduces under fuzzing it's pretty hard to bisect it with the amount of issues uncovered by trinity recently.

Quoted 7 lines
Quoted 1 line

I can add any debug code to the site of the BUG if that helps.

Good!
It will be helpful to add dump_page() in migration_entry_to_page().

Thanks.

Minchan teaches me that there is possible race condition between fork and migration.

Please consider following situation.

Process A (do migration) Process B (parents) Process C (child)

try_to_unmap() for migration <begin> fork
setup migration entry to B's vma
...
try_to_unmap() for migration <end>
move_to_new_page()

link new vma
into interval tree
remove_migration_ptes() <begin>
check and clear migration entry on C's vma
... copy_one_pte:
... now, B and C have migration entry
...

Quoted 1 line

From Sasha's report:

| [ 3800.520039] page:ffffea0000245800 count:12 mapcount:4 mapping:ffff88001d0c3668 index:0x7de
| [ 3800.521404] page flags: 0x1fffff8038003c(referenced|uptodate|dirty|lru|swapbacked|unevictable|mlocked)
| [ 3800.522585] pc:ffff88001ed91600 pc->flags:2 pc->mem_cgroup:ffffc90000c0a000

IIUC, C's mapcount should be 0 as B's in the race condition you mentioned.

Regards,
Wanpeng Li

Quoted 17 lines

...
check and clear migration entry on B's vma
...
...
remove_migration_ptes() <end>

Eventually, migration entry on C's vma is left. And then, when C exits, above BUG_ON() can be triggered.

I'm not sure the I am right, so please think of it together. :) And I'm not sure again that above assumption is related to this trigger report, since this may exist for a long time.

So my question to mm folks is is above assumption possible and do we have any protection mechanism on this race?

Quoted 7 lines

Thanks.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#9 lore raw
In reply to Joonsoo Kim
Quoted 51 lines · Joonsoo Kim
Quoted 14 lines · Joonsoo Kim
Quoted 13 lines · Sasha Levin
Quoted 8 lines · Joonsoo Kim
Quoted 3 lines · Sasha Levin
Quoted 3 lines

Ping?

I've also Cc'ed the "this page shouldn't be locked at all" team.

Hello,

I can't find the reason of this problem.
If it is reproducible, how about bisecting?

While it reproduces under fuzzing it's pretty hard to bisect it with the amount of issues uncovered by trinity recently.

Quoted 7 lines
Quoted 1 line

I can add any debug code to the site of the BUG if that helps.

Good!
It will be helpful to add dump_page() in migration_entry_to_page().

Thanks.

Minchan teaches me that there is possible race condition between fork and migration.

Please consider following situation.

Process A (do migration) Process B (parents) Process C (child)

try_to_unmap() for migration <begin> fork
setup migration entry to B's vma
...
try_to_unmap() for migration <end>
move_to_new_page()

link new vma
into interval tree
remove_migration_ptes() <begin>
check and clear migration entry on C's vma
... copy_one_pte:
... now, B and C have migration entry
...
...
check and clear migration entry on B's vma
...
...
remove_migration_ptes() <end>

Quoted 3 lines

Eventually, migration entry on C's vma is left. And then, when C exits, above BUG_ON() can be triggered.

Yes, Looks like this is a potential race condition.

Quoted 7 lines

I'm not sure the I am right, so please think of it together. :) And I'm not sure again that above assumption is related to this trigger report, since this may exist for a long time.

So my question to mm folks is is above assumption possible and do we have any protection mechanism on this race?

I think we can down_read(&mm->mmap_sem) before remove_migration_ptes() to fix this issue, but I don't have time to verify it currently.

Signature · 8 lines
Regards,
-Bob

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>
#10 lore raw

Hmm. This kind of race looks impossible: dup_mmap() always places child's vma in into rmap tree after parent's one. For file-vma it's done explicitly (vma_interval_tree_insert_after), for anon vma it's true because rb-tree insert function goes to right branch if elements are equal.

Thus remove_migration_ptes() sees parent's pte first:
If child has the copy this function will check it after that. And they are already synchronized with parent's and child's pte locks.

Quoted 25 lines · Bob Liu
Quoted 23 lines · Joonsoo Kim
Quoted 21 lines · Joonsoo Kim
Quoted 7 lines · Sasha Levin
Quoted 6 lines · Joonsoo Kim
Quoted 3 lines · Sasha Levin
Quoted 3 lines

Ping?

I've also Cc'ed the "this page shouldn't be locked at all" team.

Hello,

Quoted 7 lines
Quoted 2 lines

I can't find the reason of this problem.
If it is reproducible, how about bisecting?

While it reproduces under fuzzing it's pretty hard to bisect it with the amount of issues uncovered by trinity recently.

I can add any debug code to the site of the BUG if that helps.

Good!
It will be helpful to add dump_page() in migration_entry_to_page().

Thanks.

Quoted 7 lines
Quoted 7 lines

Minchan teaches me that there is possible race condition between fork and migration.

Please consider following situation.

Process A (do migration) Process B (parents)

Process C (child)

Quoted 7 lines
Quoted 7 lines

try_to_unmap() for migration <begin> fork
setup migration entry to B's vma
...
try_to_unmap() for migration <end>
move_to_new_page()

Quoted 6 lines
Quoted 6 lines

link new vma
into interval tree
remove_migration_ptes() <begin>
check and clear migration entry on C's vma
... copy_one_pte:
... now, B and C have

migration entry

Quoted 8 lines
Quoted 8 lines

...
...
check and clear migration entry on B's vma
...
...
remove_migration_ptes() <end>

Quoted 8 lines
Quoted 3 lines

Eventually, migration entry on C's vma is left. And then, when C exits, above BUG_ON() can be triggered.

Yes, Looks like this is a potential race condition.

Quoted 2 lines

I'm not sure the I am right, so please think of it together. :) And I'm not sure again that above assumption is related to this trigger

report,

Quoted 3 lines
Quoted 3 lines

since this may exist for a long time.

So my question to mm folks is is above assumption possible and do we

have

Quoted 15 lines
Quoted 2 lines

any protection mechanism on this race?

I think we can down_read(&mm->mmap_sem) before remove_migration_ptes() to fix this issue, but I don't have time to verify it currently.

--
Regards,
-Bob

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#11 lore raw
In reply to Bob Liu
Quoted 64 lines · Bob Liu
Quoted 6 lines · Joonsoo Kim
Quoted 5 lines · Joonsoo Kim
Quoted 4 lines · Sasha Levin
Quoted 3 lines · Joonsoo Kim
Quoted 2 lines · Sasha Levin
Quoted 2 lines

Ping?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line

I've also Cc'ed the "this page shouldn't be locked at all" team.

Hello,

I can't find the reason of this problem.
If it is reproducible, how about bisecting?

While it reproduces under fuzzing it's pretty hard to bisect it

with

Quoted 38 lines
Quoted 9 lines
Quoted 3 lines

the amount of issues uncovered by trinity recently.

I can add any debug code to the site of the BUG if that helps.

Good!
It will be helpful to add dump_page() in migration_entry_to_page().

Thanks.

Minchan teaches me that there is possible race condition between fork and migration.

Please consider following situation.

Process A (do migration) Process B (parents) Process C (child)

try_to_unmap() for migration <begin> fork
setup migration entry to B's vma
...
try_to_unmap() for migration <end>
move_to_new_page()

link new vma
into interval tree
remove_migration_ptes() <begin>
check and clear migration entry on C's vma
... copy_one_pte:
... now, B and C have migration entry
...
...
check and clear migration entry on B's vma
...
...
remove_migration_ptes() <end>

Quoted 3 lines

Eventually, migration entry on C's vma is left. And then, when C exits, above BUG_ON() can be triggered.

Yes, Looks like this is a potential race condition.

Quoted 2 lines

I'm not sure the I am right, so please think of it together. :) And I'm not sure again that above assumption is related to this

trigger report,

Quoted 2 lines

since this may exist for a long time.

Quoted 7 lines
Quoted 1 line

So my question to mm folks is is above assumption possible and do we

have

Quoted 2 lines

any protection mechanism on this race?

I think we can down_read(&mm->mmap_sem) before remove_migration_ptes() to fix this issue, but I don't have time to verify it currently.

Hmm. This kind of race looks impossible: dup_mmap() always places child's vma in into rmap tree after parent's one. For file-vma it's done explicitly (vma_interval_tree_insert_after), for anon vma it's true because rb-tree insert function goes to right branch if elements are equal.

Thus remove_migration_ptes() sees parent's pte first:
If child has the copy this function will check it after that. And they are already synchronized with parent's and child's pte locks.

Sorry for double posting, gmail cannot into plain text =)

Signature · 1 line
Sent from my Android device with K-9 Mail. Please excuse my brevity.
#12 lore raw
Quoted 73 lines · Konstantin Khlebnikov
Quoted 64 lines · Bob Liu
Quoted 6 lines · Joonsoo Kim
Quoted 5 lines · Joonsoo Kim
Quoted 4 lines · Sasha Levin
Quoted 3 lines · Joonsoo Kim
Quoted 2 lines · Sasha Levin
Quoted 2 lines

Ping?

Quoted 7 lines
Quoted 7 lines
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line

I've also Cc'ed the "this page shouldn't be locked at all" team.

Hello,

I can't find the reason of this problem.
If it is reproducible, how about bisecting?

While it reproduces under fuzzing it's pretty hard to bisect it

with

Quoted 38 lines
Quoted 9 lines
Quoted 3 lines

the amount of issues uncovered by trinity recently.

I can add any debug code to the site of the BUG if that helps.

Good!
It will be helpful to add dump_page() in migration_entry_to_page().

Thanks.

Minchan teaches me that there is possible race condition between fork and migration.

Please consider following situation.

Process A (do migration) Process B (parents) Process C (child)

try_to_unmap() for migration <begin> fork
setup migration entry to B's vma
...
try_to_unmap() for migration <end>
move_to_new_page()

link new vma
into interval tree
remove_migration_ptes() <begin>
check and clear migration entry on C's vma
... copy_one_pte:
... now, B and C have migration entry
...
...
check and clear migration entry on B's vma
...
...
remove_migration_ptes() <end>

Quoted 3 lines

Eventually, migration entry on C's vma is left. And then, when C exits, above BUG_ON() can be triggered.

Yes, Looks like this is a potential race condition.

Quoted 2 lines

I'm not sure the I am right, so please think of it together. :) And I'm not sure again that above assumption is related to this

trigger report,

Quoted 2 lines

since this may exist for a long time.

Quoted 7 lines
Quoted 1 line

So my question to mm folks is is above assumption possible and do we

have

Quoted 2 lines

any protection mechanism on this race?

I think we can down_read(&mm->mmap_sem) before remove_migration_ptes() to fix this issue, but I don't have time to verify it currently.

Quoted 9 lines

Hmm. This kind of race looks impossible: dup_mmap() always places child's vma in into rmap tree after parent's one. For file-vma it's done explicitly (vma_interval_tree_insert_after), for anon vma it's true because rb-tree insert function goes to right branch if elements are equal.

Thus remove_migration_ptes() sees parent's pte first:
If child has the copy this function will check it after that. And they are already synchronized with parent's and child's pte locks.i>>?

Agreed.

Quoted 10 lines

Sorry for double posting, gmail cannot into plain text =)

Signature · 7 lines
Sent from my Android device with K-9 Mail. Please excuse my brevity.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#13 lore raw
In reply to Joonsoo Kim
Quoted 11 lines · Joonsoo Kim
Quoted 10 lines · Sasha Levin
Quoted 8 lines · Joonsoo Kim
Quoted 3 lines · Sasha Levin
Quoted 3 lines

Ping?

I've also Cc'ed the "this page shouldn't be locked at all" team.

Hello,

I can't find the reason of this problem.
If it is reproducible, how about bisecting?

Quoted 7 lines
Quoted 4 lines

While it reproduces under fuzzing it's pretty hard to bisect it with the amount of issues uncovered by trinity recently.

I can add any debug code to the site of the BUG if that helps.

Good!
It will be helpful to add dump_page() in migration_entry_to_page().

[ 3800.520039] page:ffffea0000245800 count:12 mapcount:4 mapping:ffff88001d0c3668 index:0x7de
[ 3800.521404] page flags: 0x1fffff8038003c(referenced|uptodate|dirty|lru|swapbacked|unevictable|mlocked)
[ 3800.522585] pc:ffff88001ed91600 pc->flags:2 pc->mem_cgroup:ffffc90000c0a000

Thanks,
Sasha

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#14 lore raw
In reply to Sasha Levin
Quoted 4 lines · Sasha Levin

Ping?

I've also Cc'ed the "this page shouldn't be locked at all" team.

I have no idea why this BUG_ON was triggered.
And it looks like 'mm: kernel BUG at mm/huge_memory.c:1440!' have the same call trace with this one. Perhaps they were introduced by the same reason.
Could you confirm whether those issues exist in v3.13-rc6?

Thanks,
-Bob

Quoted 71 lines · Sasha Levin
Quoted 70 lines

Hi all,

While fuzzing with trinity inside a KVM tools guest running latest -next kernel, I've stumbled on
the following spew.

The code is in zap_pte_range():

                 if (!non_swap_entry(entry))
                         rss[MM_SWAPENTS]--;
                 else if (is_migration_entry(entry)) {
                         struct page *page;

                         page = migration_entry_to_page(entry);

<==== HERE

                         if (PageAnon(page))
                                 rss[MM_ANONPAGES]--;
                         else
                                 rss[MM_FILEPAGES]--;
[ 2622.589064] kernel BUG at include/linux/swapops.h:131!
[ 2622.589064] invalid opcode: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 2622.589064] Dumping ftrace buffer:
[ 2622.589064]    (ftrace buffer empty)
[ 2622.589064] Modules linked in:
[ 2622.589064] CPU: 9 PID: 15984 Comm: trinity-child16 Tainted:
G        W    3.13.0-rc
4-next-20131217-sasha-00013-ga878504-dirty #4150
[ 2622.589064] task: ffff88168346b000 ti: ffff8816561d8000 task.ti:
ffff8816561d8000
[ 2622.589064] RIP: 0010:[<ffffffff8127c730>]  [<ffffffff8127c730>]
zap_pte_range+0x360
/0x4a0
[ 2622.589064] RSP: 0018:ffff8816561d9c18  EFLAGS: 00010246
[ 2622.589064] RAX: ffffea00736a6600 RBX: ffff88200299d068 RCX:
0000000000000009
[ 2622.589064] RDX: 022fffff80380000 RSI: ffffea0000000000 RDI:
3c00000001cda998
[ 2622.589064] RBP: ffff8816561d9cb8 R08: 0000000000000000 R09:
0000000000000000
[ 2622.589064] R10: 0000000000000001 R11: 0000000000000000 R12:
00007fc7ee20d000
[ 2622.589064] R13: ffff8816561d9de8 R14: 000000039b53303c R15:
00007fc7ee29b000
[ 2622.589064] FS:  00007fc7eeceb700(0000) GS:ffff882011a00000(0000)
knlGS:000000000000
0000
[ 2622.589064] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 2622.589064] CR2: 000000000068c000 CR3: 0000000005c26000 CR4:
00000000000006e0
[ 2622.589064] Stack:
[ 2622.589064]  ffff8816561d9c58 0000000000000286 ffff88168327b060
ffff88168327b060
[ 2622.589064]  00007fc700000160 ffff881667067b88 00000000c8f4b120
00ff88168327b060
[ 2622.589064]  ffff88168327b060 ffff8820051a8600 0000000000000000
ffff88168327b000
[ 2622.589064] Call Trace:
[ 2622.589064]  [<ffffffff8127cc5e>] unmap_page_range+0x3ee/0x400
[ 2622.589064]  [<ffffffff8127cd71>] unmap_single_vma+0x101/0x120
[ 2622.589064]  [<ffffffff8127cdf1>] unmap_vmas+0x61/0xa0
[ 2622.589064]  [<ffffffff81283980>] exit_mmap+0xd0/0x170
[ 2622.589064]  [<ffffffff8112d430>] mmput+0x70/0xe0
[ 2622.589064]  [<ffffffff8113144d>] exit_mm+0x18d/0x1a0
[ 2622.589064]  [<ffffffff811defb5>] ? acct_collect+0x175/0x1b0
[ 2622.589064]  [<ffffffff8113389f>] do_exit+0x24f/0x500
[ 2622.589064]  [<ffffffff81133bf9>] do_group_exit+0xa9/0xe0
[ 2622.589064]  [<ffffffff81133c47>] SyS_exit_group+0x17/0x20
Quoted 8 lines
Quoted 8 lines
[ 2622.589064]  [<ffffffff843a6150>] tracesys+0xdd/0xe2
[ 2622.589064] Code: 83 f8 1f 75 46 48 b8 ff ff ff ff ff ff ff 01 48

be 00 00 00 00 00 ea ff ff 48
21 f8 48 c1 e0 06 48 01 f0 48 8b 10 80 e2 01 75 0a <0f> 0b 66 0f 1f 44 00 00 eb fe f6 40 08 01 74 05
ff 4d c4 eb 0b

[ 2622.589064] RIP  [<ffffffff8127c730>] zap_pte_range+0x360/0x4a0
[ 2622.589064]  RSP <ffff8816561d9c18>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#15 lore raw
Quoted 5 lines · Bob Liu

I have no idea why this BUG_ON was triggered.
And it looks like 'mm: kernel BUG at mm/huge_memory.c:1440!' have the same call trace with this one. Perhaps they were introduced by the same reason.
Could you confirm whether those issues exist in v3.13-rc6?

Yes, this is reproducible in 3.13-rc6.

Thanks,
Sasha

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>