oops in pgtable_trans_huge_withdraw

DISCUSSION Dave Jones 5 participants 9 messages Dec 6, 2013 – Dec 12, 2013 Memory management

Share & tools

Dave Jones

author

2013-12-06 21:03 UTC #0 lore raw

I've spent a few days enhancing trinity's use of mmap's, trying to make it reproduce https://lkml.org/lkml/2013/12/4/499
Instead, I hit this.. related ?

Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
Modules linked in: tun snd_seq_dummy rfcomm fuse hidp bnep can_raw caif_socket caif phonet af_rxrpc llc2 af_key rose netrom pppoe pppox ppp_generic slhc scsi_transport_iscsi bluetooth nfnetlink can_bcm can af_802154 ipt_ULOG nfc irda crc_ccitt rds x25 atm appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill xfs snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc libcrc32c snd_timer e1000e snd coretemp hwmon ptp x86_pkg_temp_thermal shpchp serio_raw pps_core kvm_intel pcspkr kvm crct10dif_pclmul crc32c_intel usb_debug ghash_clmulni_intel soundcore microcode
CPU: 3 PID: 11758 Comm: trinity-child3 Not tainted 3.13.0-rc2+ #23
task: ffff8800719415d0 ti: ffff8801cfa74000 task.ti: ffff8801cfa74000
RIP: 0010:[<ffffffff8118e415>]  [<ffffffff8118e415>] pgtable_trans_huge_withdraw+0x55/0xc0
RSP: 0000:ffff8801cfa75ac8  EFLAGS: 00010206
RAX: 00000000060e4800 RBX: 0000000000000000 RCX: 0000000000000027
RDX: 0000000000000000 RSI: ffff880183920000 RDI: ffff880183920000
RBP: ffff8801cfa75ae8 R08: ffff880071941cd8 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: ffff880183920000
R13: ffffea0000000000 R14: 0000000000000020 R15: 0000000040000000
FS:  00007f6dd67aa740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000020 CR3: 00000001ce03a000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 00000000018efae0 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000f5060a
Stack:
 ffffea0000000000 ffff8801cfa75c40 80000001a82008e7 ffff8801cfa75c40
 ffff8801cfa75b20 ffffffff811b13e2 ffff8801e0a8e9f0 ffff8801aa3ab000
 ffffffffffffffff 0000000040200000 ffff8801cfa75c40 ffff8801cfa75bf0
Call Trace:
 [<ffffffff811b13e2>] zap_huge_pmd+0x62/0x140
 [<ffffffff8117ac58>] unmap_single_vma+0x678/0x830
 [<ffffffff8117bea9>] unmap_vmas+0x49/0x90
 [<ffffffff81184da5>] exit_mmap+0xc5/0x170
 [<ffffffff8105104b>] mmput+0x6b/0x100
 [<ffffffff81055a18>] do_exit+0x298/0xce0
 [<ffffffff8105782c>] do_group_exit+0x4c/0xc0
 [<ffffffff8106a671>] get_signal_to_deliver+0x2d1/0x930
 [<ffffffff810024a8>] do_signal+0x48/0x610
 [<ffffffff810a9af9>] ? get_lock_stats+0x19/0x60
 [<ffffffff810aa27e>] ? put_lock_stats.isra.28+0xe/0x30
 [<ffffffff810aa7de>] ? lock_release_holdtime.part.29+0xee/0x170
 [<ffffffff8114f18e>] ? context_tracking_user_exit+0x4e/0x190
 [<ffffffff810ad1f5>] ? trace_hardirqs_on_caller+0x115/0x1e0
 [<ffffffff81002acc>] do_notify_resume+0x5c/0xa0
 [<ffffffff817587c6>] retint_signal+0x46/0x90
Code: c1 e0 06 4a 8b 44 28 30 0f b7 00 38 c4 74 79 4c 89 e7 e8 af 03 eb ff 4c 89 e7 48 c1 e8 0c 48 c1 e0 06 49 8b 5c 05 20 4c 8d 73 20 <4c> 3b 73 20 74 35 e8 90 03 eb ff 4c 89 f7 48 89 c2 48 8b 43 20 
RIP  [<ffffffff8118e415>] pgtable_trans_huge_withdraw+0x55/0xc0
 RSP <ffff8801cfa75ac8>

        pgtable = pmd_huge_pte(mm, pmdp);
        if (list_empty(&pgtable->lru))
 231:   4c 8d 73 20             lea    0x20(%rbx),%r14
 235:   4c 3b 73 20             cmp    0x20(%rbx),%r14		<--- faulting instruction.
 239:   74 35                   je     270 <pgtable_trans_huge_withdraw+0x90>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Dave Jones

author

2013-12-06 21:21 UTC #1 lore raw

Quoted 9 lines · Dave Jones

Call Trace:
 [<ffffffff811b13e2>] zap_huge_pmd+0x62/0x140
 [<ffffffff8117ac58>] unmap_single_vma+0x678/0x830
 [<ffffffff8117bea9>] unmap_vmas+0x49/0x90
 [<ffffffff81184da5>] exit_mmap+0xc5/0x170
 [<ffffffff8105104b>] mmput+0x6b/0x100
 [<ffffffff81055a18>] do_exit+0x298/0xce0
 [<ffffffff8105782c>] do_group_exit+0x4c/0xc0
 [<ffffffff8106a671>] get_signal_to_deliver+0x2d1/0x930

Quoted 8 lines

 [<ffffffff810024a8>] do_signal+0x48/0x610
 [<ffffffff810a9af9>] ? get_lock_stats+0x19/0x60
 [<ffffffff810aa27e>] ? put_lock_stats.isra.28+0xe/0x30
 [<ffffffff810aa7de>] ? lock_release_holdtime.part.29+0xee/0x170
 [<ffffffff8114f18e>] ? context_tracking_user_exit+0x4e/0x190
 [<ffffffff810ad1f5>] ? trace_hardirqs_on_caller+0x115/0x1e0
 [<ffffffff81002acc>] do_notify_resume+0x5c/0xa0
 [<ffffffff817587c6>] retint_signal+0x46/0x90

Same end result, but different trace..

Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
Modules linked in: ipt_ULOG nfnetlink scsi_transport_iscsi af_802154 nfc irda crc_ccitt rds x25 atm appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill xfs libcrc32c snd_hda_codec_realtek snd_hda_codec_hdmi coretemp hwmon x86_pkg_temp_thermal kvm_intel kvm crct10dif_pclmul crc32c_intel ghash_clmulni_intel microcode serio_raw pcspkr snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc e1000e usb_debug snd_timer shpchp ptp snd pps_core soundcore
CPU: 3 PID: 24319 Comm: trinity-child3 Not tainted 3.13.0-rc2+ #23 
task: ffff88023e60d740 ti: ffff88021e64a000 task.ti: ffff88021e64a000
RIP: 0010:[<ffffffff8118e415>]  [<ffffffff8118e415>] pgtable_trans_huge_withdraw+0x55/0xc0
RSP: 0018:ffff88021e64bd28  EFLAGS: 00010206
RAX: 00000000026fffc0 RBX: 0000000000000000 RCX: 0000000000000027
RDX: 0000000000000000 RSI: ffff88009bfff000 RDI: ffff88009bfff000
RBP: ffff88021e64bd48 R08: ffff88023e60deb8 R09: ffff880224d73c00
R10: 0000000000000000 R11: 0000000000000000 R12: ffff88009bfff000
R13: ffffea0000000000 R14: 0000000000000020 R15: ffff880221bf2200
FS:  00007f70aa5cc740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000020 CR3: 000000021ea97000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Stack:
 ffff88021e4e3b18 ffffea0008738000 0000000080000000 ffffea0000000000
 ffff88021e64bdc0 ffffffff811b1cec ffff88021ddc9550 0000000008738000
 ffff880200000000 ffff88009bfff000 0000000100000246 0000000000000000
Call Trace:
 [<ffffffff811b1cec>] split_huge_page_to_list+0x4ac/0x740
 [<ffffffff811b32ac>] __split_huge_page_pmd+0x11c/0x290
 [<ffffffff8118648d>] move_page_tables+0x21d/0x620
 [<ffffffff8118697b>] move_vma+0xeb/0x270
 [<ffffffff81186e76>] SyS_mremap+0x376/0x520
 [<ffffffff81760b64>] tracesys+0xdd/0xe2
Code: c1 e0 06 4a 8b 44 28 30 0f b7 00 38 c4 74 79 4c 89 e7 e8 af 03 eb ff 4c 89 e7 48 c1 e8 0c 48 c1 e0 06 49 8b 5c 05 20 4c 8d 73 20 <4c> 3b 73 20 74 35 e8 90 03 eb ff 4c 89 f7 48 89 c2 48 8b 43 20

Kirill A. Shutemov

2013-12-07 09:16 UTC #2 lore raw

In reply to Dave Jones

Quoted 3 lines · Dave Jones

I've spent a few days enhancing trinity's use of mmap's, trying to make it reproduce https://lkml.org/lkml/2013/12/4/499
Instead, I hit this.. related ?

Could you try this:

https://lkml.org/lkml/2013/12/4/499

Signature · 1 line

 Kirill A. Shutemov

Dave Jones

author

2013-12-09 15:42 UTC #3 lore raw

Quoted 8 lines · Kirill A. Shutemov

Quoted 3 lines · Dave Jones
I've spent a few days enhancing trinity's use of mmap's, trying to make it reproduce https://lkml.org/lkml/2013/12/4/499
Instead, I hit this.. related ?
Could you try this:
https://lkml.org/lkml/2013/12/4/499

I thought I had tried that on Friday, but apparently I had booted the wrong kernel. This does seem to be the same problem. I can't reproduce the failure any more with this patch applied.

thanks,

Dave

Vlastimil Babka

2013-12-11 15:40 UTC #4 lore raw

In reply to Dave Jones

Quoted 2 lines · Dave Jones

I've spent a few days enhancing trinity's use of mmap's, trying to make it reproduce https://lkml.org/lkml/2013/12/4/499

FYI, I managed to reproduce that using trinity today,
trinity was from git at commit e8912cc which is from Dec 09 so I guess your enhancements were already there? kernel was linux-next-20131209
I was running trinity -c mmap -c munmap -c mremap -c remap_file_pages -c mlock -c munlock

Now I'm running with Kirill's patch, will post results later.

My goal was to reproduce Sasha Levin's BUG in munlock_vma_pages_range
https://lkml.org/lkml/2013/12/7/130

Perhaps it could be related as well.
Sasha, do you know at which commit your trinity clone was at?

Thanks,
Vlastimil

Quoted 51 lines

Instead, I hit this.. related ?

Oops: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC
Modules linked in: tun snd_seq_dummy rfcomm fuse hidp bnep can_raw caif_socket caif phonet af_rxrpc llc2 af_key rose netrom pppoe pppox ppp_generic slhc scsi_transport_iscsi bluetooth nfnetlink can_bcm can af_802154 ipt_ULOG nfc irda crc_ccitt rds x25 atm appletalk ipx p8023 psnap p8022 llc ax25 cfg80211 rfkill xfs snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_intel snd_hda_codec snd_hwdep snd_seq snd_seq_device snd_pcm snd_page_alloc libcrc32c snd_timer e1000e snd coretemp hwmon ptp x86_pkg_temp_thermal shpchp serio_raw pps_core kvm_intel pcspkr kvm crct10dif_pclmul crc32c_intel usb_debug ghash_clmulni_intel soundcore microcode
CPU: 3 PID: 11758 Comm: trinity-child3 Not tainted 3.13.0-rc2+ #23
task: ffff8800719415d0 ti: ffff8801cfa74000 task.ti: ffff8801cfa74000
RIP: 0010:[<ffffffff8118e415>]  [<ffffffff8118e415>] pgtable_trans_huge_withdraw+0x55/0xc0
RSP: 0000:ffff8801cfa75ac8  EFLAGS: 00010206
RAX: 00000000060e4800 RBX: 0000000000000000 RCX: 0000000000000027
RDX: 0000000000000000 RSI: ffff880183920000 RDI: ffff880183920000
RBP: ffff8801cfa75ae8 R08: ffff880071941cd8 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000000 R12: ffff880183920000
R13: ffffea0000000000 R14: 0000000000000020 R15: 0000000040000000
FS:  00007f6dd67aa740(0000) GS:ffff880244e00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000020 CR3: 00000001ce03a000 CR4: 00000000001407e0
DR0: 0000000000000000 DR1: 00000000018efae0 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000f5060a
Stack:
  ffffea0000000000 ffff8801cfa75c40 80000001a82008e7 ffff8801cfa75c40
  ffff8801cfa75b20 ffffffff811b13e2 ffff8801e0a8e9f0 ffff8801aa3ab000
  ffffffffffffffff 0000000040200000 ffff8801cfa75c40 ffff8801cfa75bf0
Call Trace:
  [<ffffffff811b13e2>] zap_huge_pmd+0x62/0x140
  [<ffffffff8117ac58>] unmap_single_vma+0x678/0x830
  [<ffffffff8117bea9>] unmap_vmas+0x49/0x90
  [<ffffffff81184da5>] exit_mmap+0xc5/0x170
  [<ffffffff8105104b>] mmput+0x6b/0x100
  [<ffffffff81055a18>] do_exit+0x298/0xce0
  [<ffffffff8105782c>] do_group_exit+0x4c/0xc0
  [<ffffffff8106a671>] get_signal_to_deliver+0x2d1/0x930
  [<ffffffff810024a8>] do_signal+0x48/0x610
  [<ffffffff810a9af9>] ? get_lock_stats+0x19/0x60
  [<ffffffff810aa27e>] ? put_lock_stats.isra.28+0xe/0x30
  [<ffffffff810aa7de>] ? lock_release_holdtime.part.29+0xee/0x170
  [<ffffffff8114f18e>] ? context_tracking_user_exit+0x4e/0x190
  [<ffffffff810ad1f5>] ? trace_hardirqs_on_caller+0x115/0x1e0
  [<ffffffff81002acc>] do_notify_resume+0x5c/0xa0
  [<ffffffff817587c6>] retint_signal+0x46/0x90
Code: c1 e0 06 4a 8b 44 28 30 0f b7 00 38 c4 74 79 4c 89 e7 e8 af 03 eb ff 4c 89 e7 48 c1 e8 0c 48 c1 e0 06 49 8b 5c 05 20 4c 8d 73 20 <4c> 3b 73 20 74 35 e8 90 03 eb ff 4c 89 f7 48 89 c2 48 8b 43 20
RIP  [<ffffffff8118e415>] pgtable_trans_huge_withdraw+0x55/0xc0
  RSP <ffff8801cfa75ac8>

         pgtable = pmd_huge_pte(mm, pmdp);
         if (list_empty(&pgtable->lru))
  231:   4c 8d 73 20             lea    0x20(%rbx),%r14
  235:   4c 3b 73 20             cmp    0x20(%rbx),%r14		<--- faulting instruction.
  239:   74 35                   je     270 <pgtable_trans_huge_withdraw+0x90>

Quoted 6 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Sasha Levin

2013-12-11 15:54 UTC #5 lore raw

Quoted 10 lines · Vlastimil Babka

Quoted 2 lines · Dave Jones
I've spent a few days enhancing trinity's use of mmap's, trying to make it reproduce https://lkml.org/lkml/2013/12/4/499
FYI, I managed to reproduce that using trinity today,
trinity was from git at commit e8912cc which is from Dec 09 so I guess your enhancements were already there?
kernel was linux-next-20131209
I was running trinity -c mmap -c munmap -c mremap -c remap_file_pages -c mlock -c munlock

Quoted 7 lines

Now I'm running with Kirill's patch, will post results later.
My goal was to reproduce Sasha Levin's BUG in munlock_vma_pages_range
https://lkml.org/lkml/2013/12/7/130
Perhaps it could be related as well.
Sasha, do you know at which commit your trinity clone was at?

Didn't think those two were related. I've hit this one when I've started fuzzing too, but Kirill's patch solved it - so I've mostly ignored it.

Trinity is usually pulled and updated before testing, so it's at whatever the latest Dave has pushed.

Thanks,
Sasha

Sasha Levin

2013-12-11 19:59 UTC #6 lore raw

Quoted 16 lines · Sasha Levin

Quoted 15 lines · Vlastimil Babka
Quoted 2 lines · Dave Jones
I've spent a few days enhancing trinity's use of mmap's, trying to make it reproduce https://lkml.org/lkml/2013/12/4/499
FYI, I managed to reproduce that using trinity today,
trinity was from git at commit e8912cc which is from Dec 09 so I guess your enhancements were already there?
kernel was linux-next-20131209
I was running trinity -c mmap -c munmap -c mremap -c remap_file_pages -c mlock -c munlock
Now I'm running with Kirill's patch, will post results later.
My goal was to reproduce Sasha Levin's BUG in munlock_vma_pages_range
https://lkml.org/lkml/2013/12/7/130

Quoted 8 lines

Quoted 2 lines
Perhaps it could be related as well.
Sasha, do you know at which commit your trinity clone was at?
Didn't think those two were related. I've hit this one when I've started fuzzing too, but Kirill's patch solved it - so I've mostly ignored it.
Trinity is usually pulled and updated before testing, so it's at whatever the latest Dave has pushed.

I can't seem to get kexec tools to properly work with the weird layout KVM tools creates for the guest kernel.

I'd be happy to try out any debug patches sent my way, this reproduces rather easily.

Thanks,
Sasha

Dave Jones

author

2013-12-11 16:38 UTC #7 lore raw

In reply to Vlastimil Babka

Quoted 7 lines · Vlastimil Babka

Quoted 2 lines · Dave Jones
I've spent a few days enhancing trinity's use of mmap's, trying to make it reproduce https://lkml.org/lkml/2013/12/4/499
FYI, I managed to reproduce that using trinity today, trinity was from git at commit e8912cc which is from Dec 09 so I guess your enhancements were already there?

yep, everything I had pending is in HEAD now.

Quoted 3 lines

kernel was linux-next-20131209
I was running trinity -c mmap -c munmap -c mremap -c remap_file_pages -c mlock -c munlock

A shorthand for all those -c's is '-G vm'. (Though there might be a couple extra syscalls in the list too, madvise, mprotect etc).

Quoted 1 line

Now I'm running with Kirill's patch, will post results later.

Seemed to fix it for me, I've not reproduced that particular bug since applying it. Kirill, is that on its way to Linus soon ?

thanks,

Dave

Kirill A. Shutemov

2013-12-12 00:24 UTC #8 lore raw

Quoted 15 lines · Dave Jones

Quoted 7 lines · Vlastimil Babka
Quoted 2 lines · Dave Jones
I've spent a few days enhancing trinity's use of mmap's, trying to make it reproduce https://lkml.org/lkml/2013/12/4/499
FYI, I managed to reproduce that using trinity today, trinity was from git at commit e8912cc which is from Dec 09 so I guess your enhancements were already there?
yep, everything I had pending is in HEAD now.
Quoted 3 lines
kernel was linux-next-20131209
I was running trinity -c mmap -c munmap -c mremap -c remap_file_pages -c mlock -c munlock

Quoted 7 lines

A shorthand for all those -c's is '-G vm'. (Though there might be a couple extra syscalls in the list too, madvise, mprotect etc).
Quoted 1 line
Now I'm running with Kirill's patch, will post results later.
Seemed to fix it for me, I've not reproduced that particular bug since applying it. Kirill, is that on its way to Linus soon ?

It's in -mm tree for a week already.

Andrew, could you push it to Linus? More people steps on the bug.

Signature · 7 lines

 Kirill A. Shutemov

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>