next-20170515: WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0
For a few kernel releases now I have managed to trigger the warning added via commit e1a58320a38dfa ("x86/mm: Warn on W^X mappings", merged upstream since v4.4) on my KVM qemu x86_64 system. Since I just booted into the shiny new linux-next tag next-20170515 (based on v4.12-rc1) and this is still triggering I figured its time to tackle this.
Let me know if this is already known or what can be done to try to fix this.
Using QEMU emulator version 2.7.94 (v2.8.0-rc4-dirty)
I will try updating my distro package for qemu and see if perhaps its this and for the other odd fork issue I reported [0].
[0] https://lkml.kernel.org/r/CAB=NE6VZXq3y-3pfouYTBUco2Cq2xqoLZrgDFdVx+_=_=SwG_Q@mail.gmail.com
My config:
http://drvbp1.linux-foundation.org/~mcgrof/2017/05/15/configs/piggy-x86_64_qemu_fork_kmemleak.config
The splat:
[ 0.911209] x86/mm: Found insecure W+X mapping at address ffffffffc0288000/0xffffffffc0288000 [ 0.912066] ------------[ cut here ]------------ [ 0.912544] WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 0.913381] Modules linked in: [ 0.913672] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170515+ #144 [ 0.914434] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.1-0-g8891697-prebuilt.qemu-project.org 04/01/2014 [ 0.915595] task: ffff98d43a5eac80 task.stack: ffffad22c0630000 [ 0.916174] RIP: 0010:note_page+0x630/0x7e0 [ 0.916595] RSP: 0018:ffffad22c0633df0 EFLAGS: 00010286 [ 0.917101] RAX: 0000000000000051 RBX: ffffad22c0633e88 RCX: ffffffff91256708 [ 0.917805] RDX: 0000000000000000 RSI: 0000000000000096 RDI: 0000000000000246 [ 0.918511] RBP: ffffad22c0633e28 R08: 6666666666666678 R09: 0000000000000160 [ 0.919214] R10: ffffad22c0633dd8 R11: 3030303838323063 R12: 0000000000000000 [ 0.919917] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000 [ 0.920615] FS: 0000000000000000(0000) GS:ffff98d43fc00000(0000) knlGS:0000000000000000 [ 0.921384] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.921943] CR2: 0000000000000000 CR3: 00000000a3a09000 CR4: 00000000000006f0 [ 0.922657] Call Trace: [ 0.922901] ptdump_walk_pgd_level_core+0x3e7/0x490 [ 0.923354] ? 0xffffffff90600000 [ 0.923662] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.924145] mark_rodata_ro+0xf4/0x100 [ 0.924536] ? rest_init+0x80/0x80 [ 0.924862] kernel_init+0x2f/0x100 [ 0.925197] ret_from_fork+0x2c/0x40 [ 0.925552] Code: 48 c7 43 28 00 00 00 00 48 89 43 20 e9 05 fd ff ff 48 8b 73 10 48 c7 c7 c8 34 fe 90 c6 05 c8 eb bc 00 01 48 89 f2 e8 8d fc 11 00 <0f> ff e9 1f fa ff ff 48 8b 70 20 48 c7 c7 05 b1 fe 90 e8 76 fc [ 0.927368] ---[ end trace 97137ae213b9cb25 ]--- [ 0.927830] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.
Luis
Quoted 6 lines · Luis R. Rodriguez
Using QEMU emulator version 2.7.94 (v2.8.0-rc4-dirty)
I will try updating my distro package for qemu and see if perhaps its this and for the other odd fork issue I reported [0].
[0] https://lkml.kernel.org/r/CAB=NE6VZXq3y-3pfouYTBUco2Cq2xqoLZrgDFdVx+_=_=SwG_Q@mail.gmail.com
Yeah nope, using my distribution latest:
QEMU emulator version 2.8.0(openSUSE Tumbleweed)
And still both issues are present.
Luis
Quoted 8 lines · Luis R. Rodriguez
Quoted 6 lines · Luis R. Rodriguez
Using QEMU emulator version 2.7.94 (v2.8.0-rc4-dirty)
I will try updating my distro package for qemu and see if perhaps its this and for the other odd fork issue I reported [0].
[0] https://lkml.kernel.org/r/CAB=NE6VZXq3y-3pfouYTBUco2Cq2xqoLZrgDFdVx+_=_=SwG_Q@mail.gmail.com
Quoted 7 lines
Yeah nope, using my distribution latest:
QEMU emulator version 2.8.0(openSUSE Tumbleweed)
And still both issues are present.
Luis
Can you enable CONFIG_X86_PTDUMP=y and then find out what is located at ffffffffc0288000 via /sys/kernel/debug/kernel_page_tables ?
-Kees
Signature · 2 lines
Kees Cook Pixel Security
Quoted 11 lines · Kees Cook
Quoted 10 lines · Luis R. Rodriguez
Quoted 6 lines · Luis R. Rodriguez
Using QEMU emulator version 2.7.94 (v2.8.0-rc4-dirty)
I will try updating my distro package for qemu and see if perhaps its this and for the other odd fork issue I reported [0].
[0] https://lkml.kernel.org/r/CAB=NE6VZXq3y-3pfouYTBUco2Cq2xqoLZrgDFdVx+_=_=SwG_Q@mail.gmail.com
Yeah nope, using my distribution latest:
Quoted 8 lines
Quoted 5 lines
QEMU emulator version 2.8.0(openSUSE Tumbleweed)
And still both issues are present.
Luis
Can you enable
CONFIG_X86_PTDUMP=yand then find out what is located atffffffffc0288000via/sys/kernel/debug/kernel_page_tables?
Sure thing.
Recompiled with this enabled, new warning:
[ 0.891559] x86/mm: Found insecure W+X mapping at address ffffffffc00e4000/0xffffffffc00e4000 [ 0.892394] ------------[ cut here ]------------ [ 0.892834] WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 0.893674] Modules linked in: [ 0.893972] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170515+ #145 [ 0.894687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.895828] task: ffff8ed7fa5ccc80 task.stack: ffffae3900630000 [ 0.896403] RIP: 0010:note_page+0x630/0x7e0 [ 0.896780] RSP: 0018:ffffae3900633df0 EFLAGS: 00010286 [ 0.897271] RAX: 0000000000000051 RBX: ffffae3900633e88 RCX: ffffffff9b456708 [ 0.897940] RDX: 0000000000000000 RSI: 0000000000000096 RDI: 0000000000000246 [ 0.898624] RBP: ffffae3900633e28 R08: 203a6d6d2f363878 R09: 0000000000000165 [ 0.899314] R10: ffffae3900633dd8 R11: 736e6920646e756f R12: 0000000000000000 [ 0.899987] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000 [ 0.900629] FS: 0000000000000000(0000) GS:ffff8ed7ffc00000(0000) knlGS:0000000000000000 [ 0.901398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.901908] CR2: 0000000000000000 CR3: 0000000118009000 CR4: 00000000000006f0 [ 0.902590] Call Trace: [ 0.902827] ptdump_walk_pgd_level_core+0x3e7/0x490 [ 0.903274] ? 0xffffffff9a800000 [ 0.903595] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.904064] mark_rodata_ro+0xf4/0x100 [ 0.904423] ? rest_init+0x80/0x80 [ 0.904744] kernel_init+0x2f/0x100 [ 0.905068] ret_from_fork+0x2c/0x40 [ 0.905393] Code: 48 c7 43 28 00 00 00 00 48 89 43 20 e9 05 fd ff ff 48 8b 73 10 48 c7 c7 28 36 1e 9b c6 05 c8 eb bc 00 01 48 89 f2 e8 cd fc 11 00 <0f> ff e9 1f fa ff ff 48 8b 70 20 48 c7 c7 65 b2 1e 9b e8 b6 fc [ 0.907173] ---[ end trace 878b39cb0c248e66 ]--- [ 0.907655] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.
And ffffffffc00e4000 is:
---[ Modules ]---
0xffffffffc0000000-0xffffffffc00e4000 912K
pte
0xffffffffc00e4000-0xffffffffc00e5000 4K RW
GLB x pte
In case someone needs the full /sys/kernel/debug/kernel_page_tables file:
Luis
Quoted 70 lines · Luis R. Rodriguez
Quoted 11 lines · Kees Cook
Quoted 10 lines · Luis R. Rodriguez
Quoted 6 lines · Luis R. Rodriguez
Using QEMU emulator version 2.7.94 (v2.8.0-rc4-dirty)
I will try updating my distro package for qemu and see if perhaps its this and for the other odd fork issue I reported [0].
[0] https://lkml.kernel.org/r/CAB=NE6VZXq3y-3pfouYTBUco2Cq2xqoLZrgDFdVx+_=_=SwG_Q@mail.gmail.com
Yeah nope, using my distribution latest:
Quoted 8 lines
Quoted 5 lines
QEMU emulator version 2.8.0(openSUSE Tumbleweed)
And still both issues are present.
Luis
Can you enable
CONFIG_X86_PTDUMP=yand then find out what is located atffffffffc0288000via/sys/kernel/debug/kernel_page_tables?Sure thing.
Recompiled with this enabled, new warning:
[ 0.891559] x86/mm: Found insecure W+X mapping at address ffffffffc00e4000/0xffffffffc00e4000 [ 0.892394] ------------[ cut here ]------------ [ 0.892834] WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 0.893674] Modules linked in: [ 0.893972] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170515+ #145 [ 0.894687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.895828] task: ffff8ed7fa5ccc80 task.stack: ffffae3900630000 [ 0.896403] RIP: 0010:note_page+0x630/0x7e0 [ 0.896780] RSP: 0018:ffffae3900633df0 EFLAGS: 00010286 [ 0.897271] RAX: 0000000000000051 RBX: ffffae3900633e88 RCX: ffffffff9b456708 [ 0.897940] RDX: 0000000000000000 RSI: 0000000000000096 RDI: 0000000000000246 [ 0.898624] RBP: ffffae3900633e28 R08: 203a6d6d2f363878 R09: 0000000000000165 [ 0.899314] R10: ffffae3900633dd8 R11: 736e6920646e756f R12: 0000000000000000 [ 0.899987] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000 [ 0.900629] FS: 0000000000000000(0000) GS:ffff8ed7ffc00000(0000) knlGS:0000000000000000 [ 0.901398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.901908] CR2: 0000000000000000 CR3: 0000000118009000 CR4: 00000000000006f0 [ 0.902590] Call Trace: [ 0.902827] ptdump_walk_pgd_level_core+0x3e7/0x490 [ 0.903274] ? 0xffffffff9a800000 [ 0.903595] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.904064] mark_rodata_ro+0xf4/0x100 [ 0.904423] ? rest_init+0x80/0x80 [ 0.904744] kernel_init+0x2f/0x100 [ 0.905068] ret_from_fork+0x2c/0x40 [ 0.905393] Code: 48 c7 43 28 00 00 00 00 48 89 43 20 e9 05 fd ff ff 48 8b 73 10 48 c7 c7 28 36 1e 9b c6 05 c8 eb bc 00 01 48 89 f2 e8 cd fc 11 00 <0f> ff e9 1f fa ff ff 48 8b 70 20 48 c7 c7 65 b2 1e 9b e8 b6 fc [ 0.907173] ---[ end trace 878b39cb0c248e66 ]--- [ 0.907655] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.And
ffffffffc00e4000is:---[ Modules ]---
0xffffffffc0000000-0xffffffffc00e4000 912K
pte
0xffffffffc00e4000-0xffffffffc00e5000 4K RW
GLB x pte
Quoted 3 lines
In case someone needs the full
/sys/kernel/debug/kernel_page_tablesfile:
---[ Modules ]---
0xffffffffc0000000-0xffffffffc00e4000 912K
pte
This should be the modules ASLR gap
0xffffffffc00e4000-0xffffffffc00e5000 4K RW
GLB x pte
This is part of the same gap, but it's RW+x strangely?
0xffffffffc00e5000-0xffffffffc00e6000 4K
pte
This is more of the gap?
0xffffffffc00e6000-0xffffffffc00fa000 80K ro
GLB x pte
0xffffffffc00fa000-0xffffffffc010c000 72K ro
GLB NX pte
0xffffffffc010c000-0xffffffffc011b000 60K RW
GLB NX pte
This should be the first loaded module. Can you check that 0xffffffffc00e6000 matches the first module in /proc/modules?
Something touched the module gap and left is RW+x...
Are you able to bisect this?
-Kees
Signature · 2 lines
Kees Cook Pixel Security
Quoted 98 lines · Kees Cook
Quoted 70 lines · Luis R. Rodriguez
Quoted 11 lines · Kees Cook
Quoted 10 lines · Luis R. Rodriguez
Quoted 6 lines · Luis R. Rodriguez
Using QEMU emulator version 2.7.94 (v2.8.0-rc4-dirty)
I will try updating my distro package for qemu and see if perhaps its this and for the other odd fork issue I reported [0].
[0] https://lkml.kernel.org/r/CAB=NE6VZXq3y-3pfouYTBUco2Cq2xqoLZrgDFdVx+_=_=SwG_Q@mail.gmail.com
Yeah nope, using my distribution latest:
Quoted 8 lines
Quoted 5 lines
QEMU emulator version 2.8.0(openSUSE Tumbleweed)
And still both issues are present.
Luis
Can you enable
CONFIG_X86_PTDUMP=yand then find out what is located atffffffffc0288000via/sys/kernel/debug/kernel_page_tables?Sure thing.
Recompiled with this enabled, new warning:
[ 0.891559] x86/mm: Found insecure W+X mapping at address ffffffffc00e4000/0xffffffffc00e4000 [ 0.892394] ------------[ cut here ]------------ [ 0.892834] WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 0.893674] Modules linked in: [ 0.893972] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170515+ #145 [ 0.894687] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.895828] task: ffff8ed7fa5ccc80 task.stack: ffffae3900630000 [ 0.896403] RIP: 0010:note_page+0x630/0x7e0 [ 0.896780] RSP: 0018:ffffae3900633df0 EFLAGS: 00010286 [ 0.897271] RAX: 0000000000000051 RBX: ffffae3900633e88 RCX: ffffffff9b456708 [ 0.897940] RDX: 0000000000000000 RSI: 0000000000000096 RDI: 0000000000000246 [ 0.898624] RBP: ffffae3900633e28 R08: 203a6d6d2f363878 R09: 0000000000000165 [ 0.899314] R10: ffffae3900633dd8 R11: 736e6920646e756f R12: 0000000000000000 [ 0.899987] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000 [ 0.900629] FS: 0000000000000000(0000) GS:ffff8ed7ffc00000(0000) knlGS:0000000000000000 [ 0.901398] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 0.901908] CR2: 0000000000000000 CR3: 0000000118009000 CR4: 00000000000006f0 [ 0.902590] Call Trace: [ 0.902827] ptdump_walk_pgd_level_core+0x3e7/0x490 [ 0.903274] ? 0xffffffff9a800000 [ 0.903595] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.904064] mark_rodata_ro+0xf4/0x100 [ 0.904423] ? rest_init+0x80/0x80 [ 0.904744] kernel_init+0x2f/0x100 [ 0.905068] ret_from_fork+0x2c/0x40 [ 0.905393] Code: 48 c7 43 28 00 00 00 00 48 89 43 20 e9 05 fd ff ff 48 8b 73 10 48 c7 c7 28 36 1e 9b c6 05 c8 eb bc 00 01 48 89 f2 e8 cd fc 11 00 <0f> ff e9 1f fa ff ff 48 8b 70 20 48 c7 c7 65 b2 1e 9b e8 b6 fc [ 0.907173] ---[ end trace 878b39cb0c248e66 ]--- [ 0.907655] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.And
ffffffffc00e4000is:---[ Modules ]---
0xffffffffc0000000-0xffffffffc00e4000 912K
pte
0xffffffffc00e4000-0xffffffffc00e5000 4K RW
GLB x pteQuoted 3 lines
In case someone needs the full
/sys/kernel/debug/kernel_page_tablesfile:---[ Modules ]---
0xffffffffc0000000-0xffffffffc00e4000 912K
pteThis should be the modules ASLR gap
0xffffffffc00e4000-0xffffffffc00e5000 4K RW
GLB x pteThis is part of the same gap, but it's RW+x strangely?
0xffffffffc00e5000-0xffffffffc00e6000 4K
pteThis is more of the gap?
0xffffffffc00e6000-0xffffffffc00fa000 80K ro
GLB x pte
0xffffffffc00fa000-0xffffffffc010c000 72K ro
GLB NX pte
0xffffffffc010c000-0xffffffffc011b000 60K RW
GLB NX pte
Quoted 2 lines
This should be the first loaded module. Can you check that
0xffffffffc00e6000matches the first module in/proc/modules?
Yes, but I had killed that boot session again, so upon my next boot I had a different layout, the ASLR gap was much larger:
---[ Modules ]---
0xffffffffc0000000-0xffffffffc01b0000 1728K pte 0xffffffffc01b0000-0xffffffffc01b1000 4K RW GLB x pte 0xffffffffc01b1000-0xffffffffc01b2000 4K pte 0xffffffffc01b2000-0xffffffffc01c6000 80K ro GLB x pte 0xffffffffc01c6000-0xffffffffc01cc000 24K ro GLB NX pte 0xffffffffc01cc000-0xffffffffc01d5000 36K RW GLB NX pte
As you can guess if we follow similar pattern the RW hole is the one this boot warned about:
[ 1.450483] x86/mm: Found insecure W+X mapping at address ffffffffc01b0000/0xffffffffc01b0000 [ 1.451280] ------------[ cut here ]------------ [ 1.451721] WARNING: CPU: 1 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 1.452499] Modules linked in: [ 1.452791] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170515+ #145
I checked and indeed 0xffffffffc01b2000 is part of a module, it was not the first one on the /proc/modules list but then again /proc/modules does not seem to have a specific order other than perhaps being pegged into a linked list of modules once they go live, and it seems its typically output backwards from when that happened, sorting that by address we get:
root@piggy:~# cat /proc/modules | sort -k 6 | head -3 e1000 143360 0 - Live 0xffffffffc01b2000 (E) mbcache 16384 1 ext4, Live 0xffffffffc01d6000 (E) scsi_mod 217088 4 sg,sr_mod,sd_mod,libata, Live 0xffffffffc01df000 (E)
And this then seems to be the first module loaded:
e1000 143360 0 - Live 0xffffffffc01b2000 (E)
The output of dmesg seems to confirm this as per the list of modules sorted as per above.
Quoted 1 line
Something touched the module gap and left is RW+x...
Lemme try booting with e1000 renamed to e1000.ko.ignore and see how that goes.
Quoted 1 line
Are you able to bisect this?
This issue has been present for a while so since I recall this I might be able to reduce the number of needed target kernels to bisect. Lemme tinker a bit and if no clear culprit comes up then will try bisect.
Luis
Quoted 20 lines · Luis R. Rodriguez
Yes, but I had killed that boot session again, so upon my next boot I had a different layout, the ASLR gap was much larger:
---[ Modules ]---
0xffffffffc0000000-0xffffffffc01b0000 1728K pte 0xffffffffc01b0000-0xffffffffc01b1000 4K RW GLB x pte 0xffffffffc01b1000-0xffffffffc01b2000 4K pte 0xffffffffc01b2000-0xffffffffc01c6000 80K ro GLB x pte 0xffffffffc01c6000-0xffffffffc01cc000 24K ro GLB NX pte 0xffffffffc01cc000-0xffffffffc01d5000 36K RW GLB NX pteAs you can guess if we follow similar pattern the RW hole is the one this boot warned about:
[ 1.450483] x86/mm: Found insecure W+X mapping at address ffffffffc01b0000/0xffffffffc01b0000 [ 1.451280] ------------[ cut here ]------------ [ 1.451721] WARNING: CPU: 1 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 1.452499] Modules linked in: [ 1.452791] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170515+ #145
Quoted 5 lines
I checked and indeed
0xffffffffc01b2000is part of a module, it was not the first one on the/proc/moduleslist but then again/proc/modulesdoes not seem to have a specific order other than perhaps being pegged into a linked list of modules once they go live, and it seems its typically output backwards from when that happened, sorting that by address we get:
Right, sorry, I'd expect it at the bottom of the list in /proc/modules, but that's fine, it's there.
Quoted 8 lines
root@piggy:~# cat /proc/modules | sort -k 6 | head -3 e1000 143360 0 - Live 0xffffffffc01b2000 (E) mbcache 16384 1 ext4, Live 0xffffffffc01d6000 (E) scsi_mod 217088 4 sg,sr_mod,sd_mod,libata, Live 0xffffffffc01df000 (E)And this then seems to be the first module loaded:
Quoted 8 lines
e1000 143360 0 - Live
0xffffffffc01b2000(E)The output of dmesg seems to confirm this as per the list of modules sorted as per above.
Quoted 1 line
Something touched the module gap and left is RW+x...
Lemme try booting with e1000 renamed to e1000.ko.ignore and see how that goes.
Is it possible a module got loaded before e1000 and then unloaded? That seems odd, but maybe unload isn't cleaning up?
Quoted 5 lines
Quoted 1 line
Are you able to bisect this?
This issue has been present for a while so since I recall this I might be able to reduce the number of needed target kernels to bisect. Lemme tinker a bit and if no clear culprit comes up then will try bisect.
Okay, thanks!
-Kees
Signature · 2 lines
Kees Cook Pixel Security
Quoted 50 lines · Kees Cook
Quoted 20 lines · Luis R. Rodriguez
Yes, but I had killed that boot session again, so upon my next boot I had a different layout, the ASLR gap was much larger:
---[ Modules ]---
0xffffffffc0000000-0xffffffffc01b0000 1728K pte 0xffffffffc01b0000-0xffffffffc01b1000 4K RW GLB x pte 0xffffffffc01b1000-0xffffffffc01b2000 4K pte 0xffffffffc01b2000-0xffffffffc01c6000 80K ro GLB x pte 0xffffffffc01c6000-0xffffffffc01cc000 24K ro GLB NX pte 0xffffffffc01cc000-0xffffffffc01d5000 36K RW GLB NX pteAs you can guess if we follow similar pattern the RW hole is the one this boot warned about:
[ 1.450483] x86/mm: Found insecure W+X mapping at address ffffffffc01b0000/0xffffffffc01b0000 [ 1.451280] ------------[ cut here ]------------ [ 1.451721] WARNING: CPU: 1 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 1.452499] Modules linked in: [ 1.452791] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170515+ #145Quoted 5 lines
I checked and indeed
0xffffffffc01b2000is part of a module, it was not the first one on the/proc/moduleslist but then again/proc/modulesdoes not seem to have a specific order other than perhaps being pegged into a linked list of modules once they go live, and it seems its typically output backwards from when that happened, sorting that by address we get:Right, sorry, I'd expect it at the bottom of the list in
/proc/modules, but that's fine, it's there.Quoted 8 lines
root@piggy:~# cat /proc/modules | sort -k 6 | head -3 e1000 143360 0 - Live 0xffffffffc01b2000 (E) mbcache 16384 1 ext4, Live 0xffffffffc01d6000 (E) scsi_mod 217088 4 sg,sr_mod,sd_mod,libata, Live 0xffffffffc01df000 (E)And this then seems to be the first module loaded:
Quoted 8 lines
e1000 143360 0 - Live
0xffffffffc01b2000(E)The output of dmesg seems to confirm this as per the list of modules sorted as per above.
Quoted 1 line
Something touched the module gap and left is RW+x...
Lemme try booting with e1000 renamed to e1000.ko.ignore and see how that goes.
Is it possible a module got loaded before e1000 and then unloaded? That seems odd, but maybe unload isn't cleaning up?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Are you able to bisect this?
This issue has been present for a while so since I recall this I might be able to reduce the number of needed target kernels to bisect. Lemme tinker a bit and if no clear culprit comes up then will try bisect.
Okay, thanks!
Sorry to report that this issue is present since the feature's addition. So the issue is there since its addition and is still present today. *But* it may also be a configuration issue, given I have booted this guest *without* this issue ...
So:
git checkout -b WX e1a58320a38dfa72be48a0f1a3a92273663ba6db
That boots with the warning. To help debug further I've minimized my modules to only a few: scsi_mod, e1000, libata.
I suspect at this point this is not the fault of a particular module but instead just an accounting semantic (>= or <= on an edge) but let's see.
I now boot on 4.3.0-rc3 on commit (e1a58320a38df ("x86/mm: Warn on W^X mappings") and I with:
[ 0.949435] ------------[ cut here ]------------ [ 0.949992] WARNING: CPU: 2 PID: 1 at arch/x86/mm/dump_pagetables.c:225 note_page+0x635/0x7e0() [ 0.950996] x86/mm: Found insecure W+X mapping at address ffffffffc0000000/0xffffffffc0000000 [ 0.951814] Modules linked in: [ 0.952123] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.3.0-rc3-FINAL-TEST-WITH-WX-NOFLOPPY+ #365 [ 0.952929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.954033] 0000000000000000 000000001f722925 ffff88013a5d7d40 ffffffff812ff335 [ 0.954742] ffff88013a5d7d88 ffff88013a5d7d78 ffffffff81079be2 ffff88013a5d7e90 [ 0.955522] 0000000000000000 0000000000000004 0000000000000000 0000000000000000 [ 0.956256] Call Trace: [ 0.956496] [<ffffffff812ff335>] dump_stack+0x44/0x5f [ 0.956953] [<ffffffff81079be2>] warn_slowpath_common+0x82/0xc0 [ 0.957519] [<ffffffff81079c7c>] warn_slowpath_fmt+0x5c/0x80 [ 0.958066] [<ffffffff8106c155>] note_page+0x635/0x7e0 [ 0.958595] [<ffffffff8106c5eb>] ptdump_walk_pgd_level_core+0x2eb/0x410 [ 0.959219] [<ffffffff8106c7b7>] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.959856] [<ffffffff8106260d>] mark_rodata_ro+0xed/0x100 [ 0.960372] [<ffffffff815aa7d0>] ? rest_init+0x80/0x80 [ 0.960869] [<ffffffff815aa7ed>] kernel_init+0x1d/0xe0 [ 0.961358] [<ffffffff815b798f>] ret_from_fork+0x3f/0x70 [ 0.961900] [<ffffffff815aa7d0>] ? rest_init+0x80/0x80 [ 0.962389] ---[ end trace 6125ebcb24c9e3d0 ]--- [ 0.962822] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.
---[ High Kernel Mapping ]---
0xffffffff80000000-0xffffffff81000000 16M pmd
0xffffffff81000000-0xffffffff81600000 6M ro PSE GLB x pmd 0xffffffff81600000-0xffffffff81a00000 4M ro PSE GLB NX pmd 0xffffffff81a00000-0xffffffff81c00000 2M RW GLB NX pte 0xffffffff81c00000-0xffffffff82200000 6M RW PSE GLB NX pmd 0xffffffff82200000-0xffffffff82400000 2M RW GLB NX pte 0xffffffff82400000-0xffffffffc0000000 988M pmd ---[ Modules ]--- 0xffffffffc0000000-0xffffffffc0001000 4K RW GLB x pte 0xffffffffc0001000-0xffffffffc0002000 4K pte 0xffffffffc0002000-0xffffffffc0039000 220K RW GLB x pte
root@piggy:~# cat /proc/modules | sort -k 6 | head -3 scsi_mod 221979 4 sg,sd_mod,sr_mod,libata, Live 0xffffffffc0002000 (E) e1000 127757 0 - Live 0xffffffffc004d000 (E) libata 229931 2 ata_generic,ata_piix, Live 0xffffffffc0076000 (E)
So that 4K RW seems suspect of getting used for allocation purpose on edge for a particular reason and it also happens to be on the edge of the high kernel mapping. Could it be the boundary semantic issue ?
For instance can it be that since 0xffffffffc0002000 is given to the first module by the allocator, scsi_mod, and since that address is *technically* part of two boundaries we get a splat ?
0xffffffffc0001000-0xffffffffc0002000 4K pte
0xffffffffc0002000-0xffffffffc0039000 220K RW GLB x pte
Luis
Quoted 124 lines · Luis R. Rodriguez
Quoted 50 lines · Kees Cook
Quoted 20 lines · Luis R. Rodriguez
Yes, but I had killed that boot session again, so upon my next boot I had a different layout, the ASLR gap was much larger:
---[ Modules ]---
0xffffffffc0000000-0xffffffffc01b0000 1728K pte 0xffffffffc01b0000-0xffffffffc01b1000 4K RW GLB x pte 0xffffffffc01b1000-0xffffffffc01b2000 4K pte 0xffffffffc01b2000-0xffffffffc01c6000 80K ro GLB x pte 0xffffffffc01c6000-0xffffffffc01cc000 24K ro GLB NX pte 0xffffffffc01cc000-0xffffffffc01d5000 36K RW GLB NX pteAs you can guess if we follow similar pattern the RW hole is the one this boot warned about:
[ 1.450483] x86/mm: Found insecure W+X mapping at address ffffffffc01b0000/0xffffffffc01b0000 [ 1.451280] ------------[ cut here ]------------ [ 1.451721] WARNING: CPU: 1 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 1.452499] Modules linked in: [ 1.452791] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170515+ #145Quoted 5 lines
I checked and indeed
0xffffffffc01b2000is part of a module, it was not the first one on the/proc/moduleslist but then again/proc/modulesdoes not seem to have a specific order other than perhaps being pegged into a linked list of modules once they go live, and it seems its typically output backwards from when that happened, sorting that by address we get:Right, sorry, I'd expect it at the bottom of the list in
/proc/modules, but that's fine, it's there.Quoted 8 lines
root@piggy:~# cat /proc/modules | sort -k 6 | head -3 e1000 143360 0 - Live 0xffffffffc01b2000 (E) mbcache 16384 1 ext4, Live 0xffffffffc01d6000 (E) scsi_mod 217088 4 sg,sr_mod,sd_mod,libata, Live 0xffffffffc01df000 (E)And this then seems to be the first module loaded:
Quoted 8 lines
e1000 143360 0 - Live
0xffffffffc01b2000(E)The output of dmesg seems to confirm this as per the list of modules sorted as per above.
Quoted 1 line
Something touched the module gap and left is RW+x...
Lemme try booting with e1000 renamed to e1000.ko.ignore and see how that goes.
Is it possible a module got loaded before e1000 and then unloaded? That seems odd, but maybe unload isn't cleaning up?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Are you able to bisect this?
This issue has been present for a while so since I recall this I might be able to reduce the number of needed target kernels to bisect. Lemme tinker a bit and if no clear culprit comes up then will try bisect.
Okay, thanks!
Sorry to report that this issue is present since the feature's addition. So the issue is there since its addition and is still present today. *But* it may also be a configuration issue, given I have booted this guest *without* this issue ...
So:
git checkout -b WX
e1a58320a38dfa72be48a0f1a3a92273663ba6dbThat boots with the warning. To help debug further I've minimized my modules to only a few:
scsi_mod, e1000, libata.I suspect at this point this is not the fault of a particular module but instead just an accounting semantic (>= or <= on an edge) but let's see.
I now boot on 4.3.0-rc3 on commit (
e1a58320a38df("x86/mm: Warn on W^X mappings") and I with:[ 0.949435] ------------[ cut here ]------------ [ 0.949992] WARNING: CPU: 2 PID: 1 at arch/x86/mm/dump_pagetables.c:225 note_page+0x635/0x7e0() [ 0.950996] x86/mm: Found insecure W+X mapping at address ffffffffc0000000/0xffffffffc0000000 [ 0.951814] Modules linked in: [ 0.952123] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.3.0-rc3-FINAL-TEST-WITH-WX-NOFLOPPY+ #365 [ 0.952929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.954033] 0000000000000000 000000001f722925 ffff88013a5d7d40 ffffffff812ff335 [ 0.954742] ffff88013a5d7d88 ffff88013a5d7d78 ffffffff81079be2 ffff88013a5d7e90 [ 0.955522] 0000000000000000 0000000000000004 0000000000000000 0000000000000000 [ 0.956256] Call Trace: [ 0.956496] [<ffffffff812ff335>] dump_stack+0x44/0x5f [ 0.956953] [<ffffffff81079be2>] warn_slowpath_common+0x82/0xc0 [ 0.957519] [<ffffffff81079c7c>] warn_slowpath_fmt+0x5c/0x80 [ 0.958066] [<ffffffff8106c155>] note_page+0x635/0x7e0 [ 0.958595] [<ffffffff8106c5eb>] ptdump_walk_pgd_level_core+0x2eb/0x410 [ 0.959219] [<ffffffff8106c7b7>] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.959856] [<ffffffff8106260d>] mark_rodata_ro+0xed/0x100 [ 0.960372] [<ffffffff815aa7d0>] ? rest_init+0x80/0x80 [ 0.960869] [<ffffffff815aa7ed>] kernel_init+0x1d/0xe0 [ 0.961358] [<ffffffff815b798f>] ret_from_fork+0x3f/0x70 [ 0.961900] [<ffffffff815aa7d0>] ? rest_init+0x80/0x80 [ 0.962389] ---[ end trace 6125ebcb24c9e3d0 ]--- [ 0.962822] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.---[ High Kernel Mapping ]---
0xffffffff80000000-0xffffffff81000000 16M pmd0xffffffff81000000-0xffffffff81600000 6M ro PSE GLB x pmd 0xffffffff81600000-0xffffffff81a00000 4M ro PSE GLB NX pmd 0xffffffff81a00000-0xffffffff81c00000 2M RW GLB NX pte 0xffffffff81c00000-0xffffffff82200000 6M RW PSE GLB NX pmd 0xffffffff82200000-0xffffffff82400000 2M RW GLB NX pte 0xffffffff82400000-0xffffffffc0000000 988M pmd ---[ Modules ]--- 0xffffffffc0000000-0xffffffffc0001000 4K RW GLB x pte 0xffffffffc0001000-0xffffffffc0002000 4K pte 0xffffffffc0002000-0xffffffffc0039000 220K RW GLB x pteroot@piggy:~# cat /proc/modules | sort -k 6 | head -3 scsi_mod 221979 4 sg,sd_mod,sr_mod,libata, Live 0xffffffffc0002000 (E) e1000 127757 0 - Live 0xffffffffc004d000 (E) libata 229931 2 ata_generic,ata_piix, Live 0xffffffffc0076000 (E)So that 4K RW seems suspect of getting used for allocation purpose on edge for a particular reason and it also happens to be on the edge of the high kernel mapping. Could it be the boundary semantic issue ?
Quoted 6 lines
For instance can it be that since
0xffffffffc0002000is given to the first module by the allocator,scsi_mod, and since that address is *technically* part of two boundaries we get a splat ?0xffffffffc0001000-0xffffffffc0002000 4K pte
0xffffffffc0002000-0xffffffffc0039000 220K RW GLB x pte
Note on the latest linux-next and on the commit that introduced this the config and kernel yields only *one* page:
x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.
I believe this is more indications my suspicion might be right.
Luis
Quoted 132 lines · Luis R. Rodriguez
Quoted 124 lines · Luis R. Rodriguez
Quoted 50 lines · Kees Cook
Quoted 20 lines · Luis R. Rodriguez
Yes, but I had killed that boot session again, so upon my next boot I had a different layout, the ASLR gap was much larger:
---[ Modules ]---
0xffffffffc0000000-0xffffffffc01b0000 1728K pte 0xffffffffc01b0000-0xffffffffc01b1000 4K RW GLB x pte 0xffffffffc01b1000-0xffffffffc01b2000 4K pte 0xffffffffc01b2000-0xffffffffc01c6000 80K ro GLB x pte 0xffffffffc01c6000-0xffffffffc01cc000 24K ro GLB NX pte 0xffffffffc01cc000-0xffffffffc01d5000 36K RW GLB NX pteAs you can guess if we follow similar pattern the RW hole is the one this boot warned about:
[ 1.450483] x86/mm: Found insecure W+X mapping at address ffffffffc01b0000/0xffffffffc01b0000 [ 1.451280] ------------[ cut here ]------------ [ 1.451721] WARNING: CPU: 1 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 1.452499] Modules linked in: [ 1.452791] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170515+ #145Quoted 5 lines
I checked and indeed
0xffffffffc01b2000is part of a module, it was not the first one on the/proc/moduleslist but then again/proc/modulesdoes not seem to have a specific order other than perhaps being pegged into a linked list of modules once they go live, and it seems its typically output backwards from when that happened, sorting that by address we get:Right, sorry, I'd expect it at the bottom of the list in
/proc/modules, but that's fine, it's there.Quoted 8 lines
root@piggy:~# cat /proc/modules | sort -k 6 | head -3 e1000 143360 0 - Live 0xffffffffc01b2000 (E) mbcache 16384 1 ext4, Live 0xffffffffc01d6000 (E) scsi_mod 217088 4 sg,sr_mod,sd_mod,libata, Live 0xffffffffc01df000 (E)And this then seems to be the first module loaded:
Quoted 8 lines
e1000 143360 0 - Live
0xffffffffc01b2000(E)The output of dmesg seems to confirm this as per the list of modules sorted as per above.
Quoted 1 line
Something touched the module gap and left is RW+x...
Lemme try booting with e1000 renamed to e1000.ko.ignore and see how that goes.
Is it possible a module got loaded before e1000 and then unloaded? That seems odd, but maybe unload isn't cleaning up?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Are you able to bisect this?
This issue has been present for a while so since I recall this I might be able to reduce the number of needed target kernels to bisect. Lemme tinker a bit and if no clear culprit comes up then will try bisect.
Okay, thanks!
Sorry to report that this issue is present since the feature's addition. So the issue is there since its addition and is still present today. *But* it may also be a configuration issue, given I have booted this guest *without* this issue ...
So:
git checkout -b WX
e1a58320a38dfa72be48a0f1a3a92273663ba6dbThat boots with the warning. To help debug further I've minimized my modules to only a few:
scsi_mod, e1000, libata.I suspect at this point this is not the fault of a particular module but instead just an accounting semantic (>= or <= on an edge) but let's see.
I now boot on 4.3.0-rc3 on commit (
e1a58320a38df("x86/mm: Warn on W^X mappings") and I with:[ 0.949435] ------------[ cut here ]------------ [ 0.949992] WARNING: CPU: 2 PID: 1 at arch/x86/mm/dump_pagetables.c:225 note_page+0x635/0x7e0() [ 0.950996] x86/mm: Found insecure W+X mapping at address ffffffffc0000000/0xffffffffc0000000 [ 0.951814] Modules linked in: [ 0.952123] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.3.0-rc3-FINAL-TEST-WITH-WX-NOFLOPPY+ #365 [ 0.952929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.954033] 0000000000000000 000000001f722925 ffff88013a5d7d40 ffffffff812ff335 [ 0.954742] ffff88013a5d7d88 ffff88013a5d7d78 ffffffff81079be2 ffff88013a5d7e90 [ 0.955522] 0000000000000000 0000000000000004 0000000000000000 0000000000000000 [ 0.956256] Call Trace: [ 0.956496] [<ffffffff812ff335>] dump_stack+0x44/0x5f [ 0.956953] [<ffffffff81079be2>] warn_slowpath_common+0x82/0xc0 [ 0.957519] [<ffffffff81079c7c>] warn_slowpath_fmt+0x5c/0x80 [ 0.958066] [<ffffffff8106c155>] note_page+0x635/0x7e0 [ 0.958595] [<ffffffff8106c5eb>] ptdump_walk_pgd_level_core+0x2eb/0x410 [ 0.959219] [<ffffffff8106c7b7>] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.959856] [<ffffffff8106260d>] mark_rodata_ro+0xed/0x100 [ 0.960372] [<ffffffff815aa7d0>] ? rest_init+0x80/0x80 [ 0.960869] [<ffffffff815aa7ed>] kernel_init+0x1d/0xe0 [ 0.961358] [<ffffffff815b798f>] ret_from_fork+0x3f/0x70 [ 0.961900] [<ffffffff815aa7d0>] ? rest_init+0x80/0x80 [ 0.962389] ---[ end trace 6125ebcb24c9e3d0 ]--- [ 0.962822] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.---[ High Kernel Mapping ]---
0xffffffff80000000-0xffffffff81000000 16M pmd0xffffffff81000000-0xffffffff81600000 6M ro PSE GLB x pmd 0xffffffff81600000-0xffffffff81a00000 4M ro PSE GLB NX pmd 0xffffffff81a00000-0xffffffff81c00000 2M RW GLB NX pte 0xffffffff81c00000-0xffffffff82200000 6M RW PSE GLB NX pmd 0xffffffff82200000-0xffffffff82400000 2M RW GLB NX pte 0xffffffff82400000-0xffffffffc0000000 988M pmd ---[ Modules ]--- 0xffffffffc0000000-0xffffffffc0001000 4K RW GLB x pte 0xffffffffc0001000-0xffffffffc0002000 4K pte 0xffffffffc0002000-0xffffffffc0039000 220K RW GLB x pteroot@piggy:~# cat /proc/modules | sort -k 6 | head -3 scsi_mod 221979 4 sg,sd_mod,sr_mod,libata, Live 0xffffffffc0002000 (E) e1000 127757 0 - Live 0xffffffffc004d000 (E) libata 229931 2 ata_generic,ata_piix, Live 0xffffffffc0076000 (E)So that 4K RW seems suspect of getting used for allocation purpose on edge for a particular reason and it also happens to be on the edge of the high kernel mapping. Could it be the boundary semantic issue ?
Quoted 6 lines
For instance can it be that since
0xffffffffc0002000is given to the first module by the allocator,scsi_mod, and since that address is *technically* part of two boundaries we get a splat ?0xffffffffc0001000-0xffffffffc0002000 4K pte
0xffffffffc0002000-0xffffffffc0039000 220K RW GLB x pte
Quoted 6 lines
Note on the latest linux-next and on the commit that introduced this the config and kernel yields only *one* page:
x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.
I believe this is more indications my suspicion might be right.
If the following is a legit forced way to get query the kernel to ask it who owns a page then perhaps this technique can be used in the future to figure out who the hell caused this. Catalin, can you confirm? In this case this is perhaps not a leaked page but I am trying to abuse the kmemleak debugfs API to query who allocated the page. Is that fine?
[ 0.916771] WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:235 note_page+0x63c/0x7e0 [ 0.917636] x86/mm: Found insecure W+X mapping at address ffffffffc03d5000/0xffffffffc03d5000 [ 0.918502] Modules linked in: [ 0.918819] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.11.0-mcgrof-force-config #340 [ 0.919631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.920011] Call Trace: [ 0.920011] dump_stack+0x63/0x81 [ 0.920011] __warn+0xcb/0xf0 [ 0.920011] warn_slowpath_fmt+0x5a/0x80 [ 0.920011] note_page+0x63c/0x7e0 [ 0.920011] ptdump_walk_pgd_level_core+0x3b1/0x460 [ 0.920011] ? 0xffffffff86c00000 [ 0.920011] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.920011] mark_rodata_ro+0xf4/0x100 [ 0.920011] ? rest_init+0x80/0x80 [ 0.920011] kernel_init+0x2a/0x100 [ 0.920011] ret_from_fork+0x2c/0x40 [ 0.925474] ---[ end trace dca00cd779490a2b ]--- [ 0.925959] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.
echo dump=0xffffffffc03d5000 > /sys/kernel/debug/kmemleak dmesg | tail
[ 49.209565] kmemleak: Object 0xffffffffc03d5000 (size 335): [ 49.210814] kmemleak: comm "swapper/0", pid 1, jiffies 4294892440 [ 49.212148] kmemleak: min_count = 2 [ 49.212852] kmemleak: count = 0 [ 49.213363] kmemleak: flags = 0x1 [ 49.213363] kmemleak: checksum = 0 [ 49.213363] kmemleak: backtrace: [ 49.213363] kmemleak_alloc+0x4a/0xa0 [ 49.213363] __vmalloc_node_range+0x20a/0x2b0 [ 49.213363] module_alloc+0x67/0xc0 [ 49.213363] arch_ftrace_update_trampoline+0xba/0x260 [ 49.213363] ftrace_startup+0x90/0x210 [ 49.213363] register_ftrace_function+0x4b/0x60 [ 49.213363] arm_kprobe+0x84/0xe0 [ 49.213363] register_kprobe+0x56e/0x5b0 [ 49.213363] init_test_probes+0x61/0x560 [ 49.213363] init_kprobes+0x1e3/0x206 [ 49.213363] do_one_initcall+0x52/0x1a0 [ 49.213363] kernel_init_freeable+0x178/0x200 [ 49.213363] kernel_init+0xe/0x100 [ 49.213363] ret_from_fork+0x2c/0x40 [ 49.213363] 0xffffffffffffffff
Luis
Quoted 183 lines · Luis R. Rodriguez
Quoted 132 lines · Luis R. Rodriguez
Quoted 124 lines · Luis R. Rodriguez
Quoted 50 lines · Kees Cook
Quoted 20 lines · Luis R. Rodriguez
Yes, but I had killed that boot session again, so upon my next boot I had a different layout, the ASLR gap was much larger:
---[ Modules ]---
0xffffffffc0000000-0xffffffffc01b0000 1728K pte 0xffffffffc01b0000-0xffffffffc01b1000 4K RW GLB x pte 0xffffffffc01b1000-0xffffffffc01b2000 4K pte 0xffffffffc01b2000-0xffffffffc01c6000 80K ro GLB x pte 0xffffffffc01c6000-0xffffffffc01cc000 24K ro GLB NX pte 0xffffffffc01cc000-0xffffffffc01d5000 36K RW GLB NX pteAs you can guess if we follow similar pattern the RW hole is the one this boot warned about:
[ 1.450483] x86/mm: Found insecure W+X mapping at address ffffffffc01b0000/0xffffffffc01b0000 [ 1.451280] ------------[ cut here ]------------ [ 1.451721] WARNING: CPU: 1 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 1.452499] Modules linked in: [ 1.452791] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170515+ #145Quoted 5 lines
I checked and indeed
0xffffffffc01b2000is part of a module, it was not the first one on the/proc/moduleslist but then again/proc/modulesdoes not seem to have a specific order other than perhaps being pegged into a linked list of modules once they go live, and it seems its typically output backwards from when that happened, sorting that by address we get:Right, sorry, I'd expect it at the bottom of the list in
/proc/modules, but that's fine, it's there.Quoted 8 lines
root@piggy:~# cat /proc/modules | sort -k 6 | head -3 e1000 143360 0 - Live 0xffffffffc01b2000 (E) mbcache 16384 1 ext4, Live 0xffffffffc01d6000 (E) scsi_mod 217088 4 sg,sr_mod,sd_mod,libata, Live 0xffffffffc01df000 (E)And this then seems to be the first module loaded:
Quoted 8 lines
e1000 143360 0 - Live
0xffffffffc01b2000(E)The output of dmesg seems to confirm this as per the list of modules sorted as per above.
Quoted 1 line
Something touched the module gap and left is RW+x...
Lemme try booting with e1000 renamed to e1000.ko.ignore and see how that goes.
Is it possible a module got loaded before e1000 and then unloaded? That seems odd, but maybe unload isn't cleaning up?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Are you able to bisect this?
This issue has been present for a while so since I recall this I might be able to reduce the number of needed target kernels to bisect. Lemme tinker a bit and if no clear culprit comes up then will try bisect.
Okay, thanks!
Sorry to report that this issue is present since the feature's addition. So the issue is there since its addition and is still present today. *But* it may also be a configuration issue, given I have booted this guest *without* this issue ...
So:
git checkout -b WX
e1a58320a38dfa72be48a0f1a3a92273663ba6dbThat boots with the warning. To help debug further I've minimized my modules to only a few:
scsi_mod, e1000, libata.I suspect at this point this is not the fault of a particular module but instead just an accounting semantic (>= or <= on an edge) but let's see.
I now boot on 4.3.0-rc3 on commit (
e1a58320a38df("x86/mm: Warn on W^X mappings") and I with:[ 0.949435] ------------[ cut here ]------------ [ 0.949992] WARNING: CPU: 2 PID: 1 at arch/x86/mm/dump_pagetables.c:225 note_page+0x635/0x7e0() [ 0.950996] x86/mm: Found insecure W+X mapping at address ffffffffc0000000/0xffffffffc0000000 [ 0.951814] Modules linked in: [ 0.952123] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.3.0-rc3-FINAL-TEST-WITH-WX-NOFLOPPY+ #365 [ 0.952929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.954033] 0000000000000000 000000001f722925 ffff88013a5d7d40 ffffffff812ff335 [ 0.954742] ffff88013a5d7d88 ffff88013a5d7d78 ffffffff81079be2 ffff88013a5d7e90 [ 0.955522] 0000000000000000 0000000000000004 0000000000000000 0000000000000000 [ 0.956256] Call Trace: [ 0.956496] [<ffffffff812ff335>] dump_stack+0x44/0x5f [ 0.956953] [<ffffffff81079be2>] warn_slowpath_common+0x82/0xc0 [ 0.957519] [<ffffffff81079c7c>] warn_slowpath_fmt+0x5c/0x80 [ 0.958066] [<ffffffff8106c155>] note_page+0x635/0x7e0 [ 0.958595] [<ffffffff8106c5eb>] ptdump_walk_pgd_level_core+0x2eb/0x410 [ 0.959219] [<ffffffff8106c7b7>] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.959856] [<ffffffff8106260d>] mark_rodata_ro+0xed/0x100 [ 0.960372] [<ffffffff815aa7d0>] ? rest_init+0x80/0x80 [ 0.960869] [<ffffffff815aa7ed>] kernel_init+0x1d/0xe0 [ 0.961358] [<ffffffff815b798f>] ret_from_fork+0x3f/0x70 [ 0.961900] [<ffffffff815aa7d0>] ? rest_init+0x80/0x80 [ 0.962389] ---[ end trace 6125ebcb24c9e3d0 ]--- [ 0.962822] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.---[ High Kernel Mapping ]---
0xffffffff80000000-0xffffffff81000000 16M pmd0xffffffff81000000-0xffffffff81600000 6M ro PSE GLB x pmd 0xffffffff81600000-0xffffffff81a00000 4M ro PSE GLB NX pmd 0xffffffff81a00000-0xffffffff81c00000 2M RW GLB NX pte 0xffffffff81c00000-0xffffffff82200000 6M RW PSE GLB NX pmd 0xffffffff82200000-0xffffffff82400000 2M RW GLB NX pte 0xffffffff82400000-0xffffffffc0000000 988M pmd ---[ Modules ]--- 0xffffffffc0000000-0xffffffffc0001000 4K RW GLB x pte 0xffffffffc0001000-0xffffffffc0002000 4K pte 0xffffffffc0002000-0xffffffffc0039000 220K RW GLB x pteroot@piggy:~# cat /proc/modules | sort -k 6 | head -3 scsi_mod 221979 4 sg,sd_mod,sr_mod,libata, Live 0xffffffffc0002000 (E) e1000 127757 0 - Live 0xffffffffc004d000 (E) libata 229931 2 ata_generic,ata_piix, Live 0xffffffffc0076000 (E)So that 4K RW seems suspect of getting used for allocation purpose on edge for a particular reason and it also happens to be on the edge of the high kernel mapping. Could it be the boundary semantic issue ?
Quoted 6 lines
For instance can it be that since
0xffffffffc0002000is given to the first module by the allocator,scsi_mod, and since that address is *technically* part of two boundaries we get a splat ?0xffffffffc0001000-0xffffffffc0002000 4K pte
0xffffffffc0002000-0xffffffffc0039000 220K RW GLB x pteQuoted 6 lines
Note on the latest linux-next and on the commit that introduced this the config and kernel yields only *one* page:
x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.
I believe this is more indications my suspicion might be right.
If the following is a legit forced way to get query the kernel to ask it who owns a page then perhaps this technique can be used in the future to figure out who the hell caused this. Catalin, can you confirm? In this case this is perhaps not a leaked page but I am trying to abuse the kmemleak debugfs API to query who allocated the page. Is that fine?
[ 0.916771] WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:235 note_page+0x63c/0x7e0 [ 0.917636] x86/mm: Found insecure W+X mapping at address ffffffffc03d5000/0xffffffffc03d5000 [ 0.918502] Modules linked in: [ 0.918819] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.11.0-mcgrof-force-config #340 [ 0.919631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.920011] Call Trace: [ 0.920011] dump_stack+0x63/0x81 [ 0.920011] __warn+0xcb/0xf0 [ 0.920011] warn_slowpath_fmt+0x5a/0x80 [ 0.920011] note_page+0x63c/0x7e0 [ 0.920011] ptdump_walk_pgd_level_core+0x3b1/0x460 [ 0.920011] ? 0xffffffff86c00000 [ 0.920011] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.920011] mark_rodata_ro+0xf4/0x100 [ 0.920011] ? rest_init+0x80/0x80 [ 0.920011] kernel_init+0x2a/0x100 [ 0.920011] ret_from_fork+0x2c/0x40 [ 0.925474] ---[ end trace dca00cd779490a2b ]--- [ 0.925959] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.echo dump=0xffffffffc03d5000 >
/sys/kernel/debug/kmemleakdmesg | tail[ 49.209565] kmemleak: Object 0xffffffffc03d5000 (size 335): [ 49.210814] kmemleak: comm "swapper/0", pid 1, jiffies 4294892440 [ 49.212148] kmemleak: min_count = 2 [ 49.212852] kmemleak: count = 0 [ 49.213363] kmemleak: flags = 0x1 [ 49.213363] kmemleak: checksum = 0 [ 49.213363] kmemleak: backtrace: [ 49.213363] kmemleak_alloc+0x4a/0xa0 [ 49.213363] __vmalloc_node_range+0x20a/0x2b0 [ 49.213363] module_alloc+0x67/0xc0 [ 49.213363] arch_ftrace_update_trampoline+0xba/0x260 [ 49.213363] ftrace_startup+0x90/0x210 [ 49.213363] register_ftrace_function+0x4b/0x60 [ 49.213363] arm_kprobe+0x84/0xe0
Quoted 8 lines
[ 49.213363] register_kprobe+0x56e/0x5b0 [ 49.213363] init_test_probes+0x61/0x560 [ 49.213363] init_kprobes+0x1e3/0x206 [ 49.213363] do_one_initcall+0x52/0x1a0 [ 49.213363] kernel_init_freeable+0x178/0x200 [ 49.213363] kernel_init+0xe/0x100 [ 49.213363] ret_from_fork+0x2c/0x40 [ 49.213363] 0xffffffffffffffff
Aha! And the winner is:
CONFIG_KPROBES_SANITY_TEST
I confirm disabling it on 4.3.0-rc3 and on linux-next next-20170519 avoids the WARN. I also can confirm using the 'echo dump=mem-area > /sys/kernel/debug/kmemleak' yields the same trace for both of these kernels.
So -- the above kmemleak hack seems to actually work to seek who owns that page.
Now to figure out how the hell kernel/test_kprobes.c screws around with things.
Luis
Hi Luis,
Quoted 197 lines · On Fri, 19 May 2017 19:28:54 +0200 "Luis R. Rodriguez" <[email protected]>…
Quoted 183 lines · Luis R. Rodriguez
Quoted 132 lines · Luis R. Rodriguez
Quoted 124 lines · Luis R. Rodriguez
Quoted 50 lines · Kees Cook
Quoted 20 lines · Luis R. Rodriguez
Yes, but I had killed that boot session again, so upon my next boot I had a different layout, the ASLR gap was much larger:
---[ Modules ]---
0xffffffffc0000000-0xffffffffc01b0000 1728K pte 0xffffffffc01b0000-0xffffffffc01b1000 4K RW GLB x pte 0xffffffffc01b1000-0xffffffffc01b2000 4K pte 0xffffffffc01b2000-0xffffffffc01c6000 80K ro GLB x pte 0xffffffffc01c6000-0xffffffffc01cc000 24K ro GLB NX pte 0xffffffffc01cc000-0xffffffffc01d5000 36K RW GLB NX pteAs you can guess if we follow similar pattern the RW hole is the one this boot warned about:
[ 1.450483] x86/mm: Found insecure W+X mapping at address ffffffffc01b0000/0xffffffffc01b0000 [ 1.451280] ------------[ cut here ]------------ [ 1.451721] WARNING: CPU: 1 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 1.452499] Modules linked in: [ 1.452791] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170515+ #145Quoted 5 lines
I checked and indeed
0xffffffffc01b2000is part of a module, it was not the first one on the/proc/moduleslist but then again/proc/modulesdoes not seem to have a specific order other than perhaps being pegged into a linked list of modules once they go live, and it seems its typically output backwards from when that happened, sorting that by address we get:Right, sorry, I'd expect it at the bottom of the list in
/proc/modules, but that's fine, it's there.Quoted 8 lines
root@piggy:~# cat /proc/modules | sort -k 6 | head -3 e1000 143360 0 - Live 0xffffffffc01b2000 (E) mbcache 16384 1 ext4, Live 0xffffffffc01d6000 (E) scsi_mod 217088 4 sg,sr_mod,sd_mod,libata, Live 0xffffffffc01df000 (E)And this then seems to be the first module loaded:
Quoted 8 lines
e1000 143360 0 - Live
0xffffffffc01b2000(E)The output of dmesg seems to confirm this as per the list of modules sorted as per above.
Quoted 1 line
Something touched the module gap and left is RW+x...
Lemme try booting with e1000 renamed to e1000.ko.ignore and see how that goes.
Is it possible a module got loaded before e1000 and then unloaded? That seems odd, but maybe unload isn't cleaning up?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Are you able to bisect this?
This issue has been present for a while so since I recall this I might be able to reduce the number of needed target kernels to bisect. Lemme tinker a bit and if no clear culprit comes up then will try bisect.
Okay, thanks!
Sorry to report that this issue is present since the feature's addition. So the issue is there since its addition and is still present today. *But* it may also be a configuration issue, given I have booted this guest *without* this issue ...
So:
git checkout -b WX
e1a58320a38dfa72be48a0f1a3a92273663ba6dbThat boots with the warning. To help debug further I've minimized my modules to only a few:
scsi_mod, e1000, libata.I suspect at this point this is not the fault of a particular module but instead just an accounting semantic (>= or <= on an edge) but let's see.
I now boot on 4.3.0-rc3 on commit (
e1a58320a38df("x86/mm: Warn on W^X mappings") and I with:[ 0.949435] ------------[ cut here ]------------ [ 0.949992] WARNING: CPU: 2 PID: 1 at arch/x86/mm/dump_pagetables.c:225 note_page+0x635/0x7e0() [ 0.950996] x86/mm: Found insecure W+X mapping at address ffffffffc0000000/0xffffffffc0000000 [ 0.951814] Modules linked in: [ 0.952123] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.3.0-rc3-FINAL-TEST-WITH-WX-NOFLOPPY+ #365 [ 0.952929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.954033] 0000000000000000 000000001f722925 ffff88013a5d7d40 ffffffff812ff335 [ 0.954742] ffff88013a5d7d88 ffff88013a5d7d78 ffffffff81079be2 ffff88013a5d7e90 [ 0.955522] 0000000000000000 0000000000000004 0000000000000000 0000000000000000 [ 0.956256] Call Trace: [ 0.956496] [<ffffffff812ff335>] dump_stack+0x44/0x5f [ 0.956953] [<ffffffff81079be2>] warn_slowpath_common+0x82/0xc0 [ 0.957519] [<ffffffff81079c7c>] warn_slowpath_fmt+0x5c/0x80 [ 0.958066] [<ffffffff8106c155>] note_page+0x635/0x7e0 [ 0.958595] [<ffffffff8106c5eb>] ptdump_walk_pgd_level_core+0x2eb/0x410 [ 0.959219] [<ffffffff8106c7b7>] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.959856] [<ffffffff8106260d>] mark_rodata_ro+0xed/0x100 [ 0.960372] [<ffffffff815aa7d0>] ? rest_init+0x80/0x80 [ 0.960869] [<ffffffff815aa7ed>] kernel_init+0x1d/0xe0 [ 0.961358] [<ffffffff815b798f>] ret_from_fork+0x3f/0x70 [ 0.961900] [<ffffffff815aa7d0>] ? rest_init+0x80/0x80 [ 0.962389] ---[ end trace 6125ebcb24c9e3d0 ]--- [ 0.962822] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.---[ High Kernel Mapping ]---
0xffffffff80000000-0xffffffff81000000 16M pmd0xffffffff81000000-0xffffffff81600000 6M ro PSE GLB x pmd 0xffffffff81600000-0xffffffff81a00000 4M ro PSE GLB NX pmd 0xffffffff81a00000-0xffffffff81c00000 2M RW GLB NX pte 0xffffffff81c00000-0xffffffff82200000 6M RW PSE GLB NX pmd 0xffffffff82200000-0xffffffff82400000 2M RW GLB NX pte 0xffffffff82400000-0xffffffffc0000000 988M pmd ---[ Modules ]--- 0xffffffffc0000000-0xffffffffc0001000 4K RW GLB x pte 0xffffffffc0001000-0xffffffffc0002000 4K pte 0xffffffffc0002000-0xffffffffc0039000 220K RW GLB x pteroot@piggy:~# cat /proc/modules | sort -k 6 | head -3 scsi_mod 221979 4 sg,sd_mod,sr_mod,libata, Live 0xffffffffc0002000 (E) e1000 127757 0 - Live 0xffffffffc004d000 (E) libata 229931 2 ata_generic,ata_piix, Live 0xffffffffc0076000 (E)So that 4K RW seems suspect of getting used for allocation purpose on edge for a particular reason and it also happens to be on the edge of the high kernel mapping. Could it be the boundary semantic issue ?
Quoted 6 lines
For instance can it be that since
0xffffffffc0002000is given to the first module by the allocator,scsi_mod, and since that address is *technically* part of two boundaries we get a splat ?0xffffffffc0001000-0xffffffffc0002000 4K pte
0xffffffffc0002000-0xffffffffc0039000 220K RW GLB x pteQuoted 6 lines
Note on the latest linux-next and on the commit that introduced this the config and kernel yields only *one* page:
x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.
I believe this is more indications my suspicion might be right.
If the following is a legit forced way to get query the kernel to ask it who owns a page then perhaps this technique can be used in the future to figure out who the hell caused this. Catalin, can you confirm? In this case this is perhaps not a leaked page but I am trying to abuse the kmemleak debugfs API to query who allocated the page. Is that fine?
[ 0.916771] WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:235 note_page+0x63c/0x7e0 [ 0.917636] x86/mm: Found insecure W+X mapping at address ffffffffc03d5000/0xffffffffc03d5000 [ 0.918502] Modules linked in: [ 0.918819] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.11.0-mcgrof-force-config #340 [ 0.919631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.920011] Call Trace: [ 0.920011] dump_stack+0x63/0x81 [ 0.920011] __warn+0xcb/0xf0 [ 0.920011] warn_slowpath_fmt+0x5a/0x80 [ 0.920011] note_page+0x63c/0x7e0 [ 0.920011] ptdump_walk_pgd_level_core+0x3b1/0x460 [ 0.920011] ? 0xffffffff86c00000 [ 0.920011] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.920011] mark_rodata_ro+0xf4/0x100 [ 0.920011] ? rest_init+0x80/0x80 [ 0.920011] kernel_init+0x2a/0x100 [ 0.920011] ret_from_fork+0x2c/0x40 [ 0.925474] ---[ end trace dca00cd779490a2b ]--- [ 0.925959] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.echo dump=0xffffffffc03d5000 >
/sys/kernel/debug/kmemleakdmesg | tail[ 49.209565] kmemleak: Object 0xffffffffc03d5000 (size 335): [ 49.210814] kmemleak: comm "swapper/0", pid 1, jiffies 4294892440 [ 49.212148] kmemleak: min_count = 2 [ 49.212852] kmemleak: count = 0 [ 49.213363] kmemleak: flags = 0x1 [ 49.213363] kmemleak: checksum = 0 [ 49.213363] kmemleak: backtrace: [ 49.213363] kmemleak_alloc+0x4a/0xa0 [ 49.213363] __vmalloc_node_range+0x20a/0x2b0 [ 49.213363] module_alloc+0x67/0xc0 [ 49.213363] arch_ftrace_update_trampoline+0xba/0x260 [ 49.213363] ftrace_startup+0x90/0x210 [ 49.213363] register_ftrace_function+0x4b/0x60 [ 49.213363] arm_kprobe+0x84/0xe0Quoted 8 lines
[ 49.213363] register_kprobe+0x56e/0x5b0 [ 49.213363] init_test_probes+0x61/0x560 [ 49.213363] init_kprobes+0x1e3/0x206 [ 49.213363] do_one_initcall+0x52/0x1a0 [ 49.213363] kernel_init_freeable+0x178/0x200 [ 49.213363] kernel_init+0xe/0x100 [ 49.213363] ret_from_fork+0x2c/0x40 [ 49.213363] 0xffffffffffffffffAha! And the winner is:
CONFIG_KPROBES_SANITY_TEST
Quoted 7 lines
I confirm disabling it on 4.3.0-rc3 and on linux-next next-20170519 avoids the WARN. I also can confirm using the 'echo dump=mem-area >
/sys/kernel/debug/kmemleak' yields the same trace for both of these kernels.So -- the above kmemleak hack seems to actually work to seek who owns that page.
Now to figure out how the hell
kernel/test_kprobes.cscrews around with things.
Ah, that was fixed recently;
https://marc.info/?l=linux-kernel&m=149076389011850
Note that this patch depends another patch in the series;
https://marc.info/?l=linux-kernel&m=149076370111812&w=2
Thank you,
Signature · 1 line
Masami Hiramatsu <[email protected]>
Quoted 210 lines · Masami Hiramatsu
Hi Luis,
Quoted 197 lines · On Fri, 19 May 2017 19:28:54 +0200 "Luis R. Rodriguez" <[email protected]>…
Quoted 183 lines · Luis R. Rodriguez
Quoted 132 lines · Luis R. Rodriguez
Quoted 124 lines · Luis R. Rodriguez
Quoted 50 lines · Kees Cook
Quoted 20 lines · Luis R. Rodriguez
Yes, but I had killed that boot session again, so upon my next boot I had a different layout, the ASLR gap was much larger:
---[ Modules ]---
0xffffffffc0000000-0xffffffffc01b0000 1728K pte 0xffffffffc01b0000-0xffffffffc01b1000 4K RW GLB x pte 0xffffffffc01b1000-0xffffffffc01b2000 4K pte 0xffffffffc01b2000-0xffffffffc01c6000 80K ro GLB x pte 0xffffffffc01c6000-0xffffffffc01cc000 24K ro GLB NX pte 0xffffffffc01cc000-0xffffffffc01d5000 36K RW GLB NX pteAs you can guess if we follow similar pattern the RW hole is the one this boot warned about:
[ 1.450483] x86/mm: Found insecure W+X mapping at address ffffffffc01b0000/0xffffffffc01b0000 [ 1.451280] ------------[ cut here ]------------ [ 1.451721] WARNING: CPU: 1 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 1.452499] Modules linked in: [ 1.452791] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170515+ #145Quoted 5 lines
I checked and indeed
0xffffffffc01b2000is part of a module, it was not the first one on the/proc/moduleslist but then again/proc/modulesdoes not seem to have a specific order other than perhaps being pegged into a linked list of modules once they go live, and it seems its typically output backwards from when that happened, sorting that by address we get:Right, sorry, I'd expect it at the bottom of the list in
/proc/modules, but that's fine, it's there.Quoted 8 lines
root@piggy:~# cat /proc/modules | sort -k 6 | head -3 e1000 143360 0 - Live 0xffffffffc01b2000 (E) mbcache 16384 1 ext4, Live 0xffffffffc01d6000 (E) scsi_mod 217088 4 sg,sr_mod,sd_mod,libata, Live 0xffffffffc01df000 (E)And this then seems to be the first module loaded:
Quoted 8 lines
e1000 143360 0 - Live
0xffffffffc01b2000(E)The output of dmesg seems to confirm this as per the list of modules sorted as per above.
Quoted 1 line
Something touched the module gap and left is RW+x...
Lemme try booting with e1000 renamed to e1000.ko.ignore and see how that goes.
Is it possible a module got loaded before e1000 and then unloaded? That seems odd, but maybe unload isn't cleaning up?
Quoted 7 lines
Quoted 5 lines
Quoted 1 line
Are you able to bisect this?
This issue has been present for a while so since I recall this I might be able to reduce the number of needed target kernels to bisect. Lemme tinker a bit and if no clear culprit comes up then will try bisect.
Okay, thanks!
Sorry to report that this issue is present since the feature's addition. So the issue is there since its addition and is still present today. *But* it may also be a configuration issue, given I have booted this guest *without* this issue ...
So:
git checkout -b WX
e1a58320a38dfa72be48a0f1a3a92273663ba6dbThat boots with the warning. To help debug further I've minimized my modules to only a few:
scsi_mod, e1000, libata.I suspect at this point this is not the fault of a particular module but instead just an accounting semantic (>= or <= on an edge) but let's see.
I now boot on 4.3.0-rc3 on commit (
e1a58320a38df("x86/mm: Warn on W^X mappings") and I with:[ 0.949435] ------------[ cut here ]------------ [ 0.949992] WARNING: CPU: 2 PID: 1 at arch/x86/mm/dump_pagetables.c:225 note_page+0x635/0x7e0() [ 0.950996] x86/mm: Found insecure W+X mapping at address ffffffffc0000000/0xffffffffc0000000 [ 0.951814] Modules linked in: [ 0.952123] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.3.0-rc3-FINAL-TEST-WITH-WX-NOFLOPPY+ #365 [ 0.952929] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.954033] 0000000000000000 000000001f722925 ffff88013a5d7d40 ffffffff812ff335 [ 0.954742] ffff88013a5d7d88 ffff88013a5d7d78 ffffffff81079be2 ffff88013a5d7e90 [ 0.955522] 0000000000000000 0000000000000004 0000000000000000 0000000000000000 [ 0.956256] Call Trace: [ 0.956496] [<ffffffff812ff335>] dump_stack+0x44/0x5f [ 0.956953] [<ffffffff81079be2>] warn_slowpath_common+0x82/0xc0 [ 0.957519] [<ffffffff81079c7c>] warn_slowpath_fmt+0x5c/0x80 [ 0.958066] [<ffffffff8106c155>] note_page+0x635/0x7e0 [ 0.958595] [<ffffffff8106c5eb>] ptdump_walk_pgd_level_core+0x2eb/0x410 [ 0.959219] [<ffffffff8106c7b7>] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.959856] [<ffffffff8106260d>] mark_rodata_ro+0xed/0x100 [ 0.960372] [<ffffffff815aa7d0>] ? rest_init+0x80/0x80 [ 0.960869] [<ffffffff815aa7ed>] kernel_init+0x1d/0xe0 [ 0.961358] [<ffffffff815b798f>] ret_from_fork+0x3f/0x70 [ 0.961900] [<ffffffff815aa7d0>] ? rest_init+0x80/0x80 [ 0.962389] ---[ end trace 6125ebcb24c9e3d0 ]--- [ 0.962822] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.---[ High Kernel Mapping ]---
0xffffffff80000000-0xffffffff81000000 16M pmd0xffffffff81000000-0xffffffff81600000 6M ro PSE GLB x pmd 0xffffffff81600000-0xffffffff81a00000 4M ro PSE GLB NX pmd 0xffffffff81a00000-0xffffffff81c00000 2M RW GLB NX pte 0xffffffff81c00000-0xffffffff82200000 6M RW PSE GLB NX pmd 0xffffffff82200000-0xffffffff82400000 2M RW GLB NX pte 0xffffffff82400000-0xffffffffc0000000 988M pmd ---[ Modules ]--- 0xffffffffc0000000-0xffffffffc0001000 4K RW GLB x pte 0xffffffffc0001000-0xffffffffc0002000 4K pte 0xffffffffc0002000-0xffffffffc0039000 220K RW GLB x pteroot@piggy:~# cat /proc/modules | sort -k 6 | head -3 scsi_mod 221979 4 sg,sd_mod,sr_mod,libata, Live 0xffffffffc0002000 (E) e1000 127757 0 - Live 0xffffffffc004d000 (E) libata 229931 2 ata_generic,ata_piix, Live 0xffffffffc0076000 (E)So that 4K RW seems suspect of getting used for allocation purpose on edge for a particular reason and it also happens to be on the edge of the high kernel mapping. Could it be the boundary semantic issue ?
Quoted 6 lines
For instance can it be that since
0xffffffffc0002000is given to the first module by the allocator,scsi_mod, and since that address is *technically* part of two boundaries we get a splat ?0xffffffffc0001000-0xffffffffc0002000 4K pte
0xffffffffc0002000-0xffffffffc0039000 220K RW GLB x pteQuoted 6 lines
Note on the latest linux-next and on the commit that introduced this the config and kernel yields only *one* page:
x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.
I believe this is more indications my suspicion might be right.
If the following is a legit forced way to get query the kernel to ask it who owns a page then perhaps this technique can be used in the future to figure out who the hell caused this. Catalin, can you confirm? In this case this is perhaps not a leaked page but I am trying to abuse the kmemleak debugfs API to query who allocated the page. Is that fine?
[ 0.916771] WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:235 note_page+0x63c/0x7e0 [ 0.917636] x86/mm: Found insecure W+X mapping at address ffffffffc03d5000/0xffffffffc03d5000 [ 0.918502] Modules linked in: [ 0.918819] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.11.0-mcgrof-force-config #340 [ 0.919631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.920011] Call Trace: [ 0.920011] dump_stack+0x63/0x81 [ 0.920011] __warn+0xcb/0xf0 [ 0.920011] warn_slowpath_fmt+0x5a/0x80 [ 0.920011] note_page+0x63c/0x7e0 [ 0.920011] ptdump_walk_pgd_level_core+0x3b1/0x460 [ 0.920011] ? 0xffffffff86c00000 [ 0.920011] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.920011] mark_rodata_ro+0xf4/0x100 [ 0.920011] ? rest_init+0x80/0x80 [ 0.920011] kernel_init+0x2a/0x100 [ 0.920011] ret_from_fork+0x2c/0x40 [ 0.925474] ---[ end trace dca00cd779490a2b ]--- [ 0.925959] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.echo dump=0xffffffffc03d5000 >
/sys/kernel/debug/kmemleakdmesg | tail[ 49.209565] kmemleak: Object 0xffffffffc03d5000 (size 335): [ 49.210814] kmemleak: comm "swapper/0", pid 1, jiffies 4294892440 [ 49.212148] kmemleak: min_count = 2 [ 49.212852] kmemleak: count = 0 [ 49.213363] kmemleak: flags = 0x1 [ 49.213363] kmemleak: checksum = 0 [ 49.213363] kmemleak: backtrace: [ 49.213363] kmemleak_alloc+0x4a/0xa0 [ 49.213363] __vmalloc_node_range+0x20a/0x2b0 [ 49.213363] module_alloc+0x67/0xc0 [ 49.213363] arch_ftrace_update_trampoline+0xba/0x260 [ 49.213363] ftrace_startup+0x90/0x210 [ 49.213363] register_ftrace_function+0x4b/0x60 [ 49.213363] arm_kprobe+0x84/0xe0Quoted 8 lines
[ 49.213363] register_kprobe+0x56e/0x5b0 [ 49.213363] init_test_probes+0x61/0x560 [ 49.213363] init_kprobes+0x1e3/0x206 [ 49.213363] do_one_initcall+0x52/0x1a0 [ 49.213363] kernel_init_freeable+0x178/0x200 [ 49.213363] kernel_init+0xe/0x100 [ 49.213363] ret_from_fork+0x2c/0x40 [ 49.213363] 0xffffffffffffffffAha! And the winner is:
CONFIG_KPROBES_SANITY_TESTQuoted 7 lines
I confirm disabling it on 4.3.0-rc3 and on linux-next next-20170519 avoids the WARN. I also can confirm using the 'echo dump=mem-area >
/sys/kernel/debug/kmemleak' yields the same trace for both of these kernels.So -- the above kmemleak hack seems to actually work to seek who owns that page.
Now to figure out how the hell
kernel/test_kprobes.cscrews around with things.
Quoted 7 lines
Ah, that was fixed recently;
https://marc.info/?l=linux-kernel&m=149076389011850
Note that this patch depends another patch in the series;
I actually boot tested linux-next tag next-20170519 which carries these patches and the WARNING still is there. Please note the issue is with CONFIG_KPROBES_SANITY_TEST enabled.
[ 1.025601] x86/mm: Found insecure W+X mapping at address ffffffffc01e7000/0xffffffffc01e7000 [ 1.026429] ------------[ cut here ]------------ [ 1.026885] WARNING: CPU: 1 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 1.027711] Modules linked in: [ 1.028032] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170519 #151 [ 1.028788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 1.029928] task: ffff9fd47a5ccc80 task.stack: ffffb6bcc0630000 [ 1.030509] RIP: 0010:note_page+0x630/0x7e0 [ 1.030917] RSP: 0000:ffffb6bcc0633df0 EFLAGS: 00010286 [ 1.031425] RAX: 0000000000000051 RBX: ffffb6bcc0633e88 RCX: ffffffffbb656708 [ 1.032132] RDX: 0000000000000000 RSI: 0000000000000096 RDI: 0000000000000246 [ 1.032834] RBP: ffffb6bcc0633e28 R08: 203a6d6d2f363878 R09: 0000000000000161 [ 1.033539] R10: ffffb6bcc0633dd8 R11: 736e6920646e756f R12: 0000000000000000 [ 1.034235] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000 [ 1.034927] FS: 0000000000000000(0000) GS:ffff9fd47fc80000(0000) knlGS:0000000000000000 [ 1.035722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.036290] CR2: ffffb6bcc073c000 CR3: 0000000053209000 CR4: 00000000000006e0 [ 1.036839] Call Trace: [ 1.037034] ptdump_walk_pgd_level_core+0x3e7/0x490 [ 1.037367] ? 0xffffffffbaa00000 [ 1.037705] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 1.038187] mark_rodata_ro+0xf4/0x100 [ 1.038559] ? rest_init+0x80/0x80 [ 1.038890] kernel_init+0x2f/0x100 [ 1.039235] ret_from_fork+0x2c/0x40 [ 1.039582] Code: 48 c7 43 28 00 00 00 00 48 89 43 20 e9 05 fd ff ff 48 8b 73 10 48 c7 c7 f0 3d 3e bb c6 05 f8 eb bc 00 01 48 89 f2 e8 1d 02 12 00 <0f> ff e9 1f fa ff ff 48 8b 70 20 48 c7 c7 3c ba 3e bb e8 06 02 [ 1.041416] ---[ end trace e726c1b63e5a81a9 ]--- [ 1.041872] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.
root@piggy:~# echo dump=0xffffffffc01e7000 > /sys/kernel/debug/kmemleak
On dmesg:
May 23 07:44:51 piggy kernel: kmemleak: Object 0xffffffffc01e7000 (size 335): May 23 07:44:51 piggy kernel: kmemleak: comm "swapper/0", pid 1, jiffies 4294892451 May 23 07:44:51 piggy kernel: kmemleak: min_count = 2 May 23 07:44:51 piggy kernel: kmemleak: count = 2 May 23 07:44:51 piggy kernel: kmemleak: flags = 0x1 May 23 07:44:51 piggy kernel: kmemleak: checksum = 0 May 23 07:44:51 piggy kernel: kmemleak: backtrace: May 23 07:44:51 piggy kernel: kmemleak_alloc+0x4a/0xa0 May 23 07:44:51 piggy kernel: __vmalloc_node_range+0x20c/0x2b0 May 23 07:44:51 piggy kernel: module_alloc+0x67/0xc0 May 23 07:44:51 piggy kernel: arch_ftrace_update_trampoline+0xc1/0x240 May 23 07:44:51 piggy kernel: ftrace_startup+0x92/0x210 May 23 07:44:51 piggy kernel: register_ftrace_function+0x4b/0x60 May 23 07:44:51 piggy kernel: arm_kprobe+0x84/0xc0 May 23 07:44:51 piggy kernel: register_kprobe+0x59c/0x5e0 May 23 07:44:51 piggy kernel: init_test_probes+0x61/0x560 May 23 07:44:51 piggy kernel: init_kprobes+0x1ea/0x20d May 23 07:44:51 piggy kernel: do_one_initcall+0x52/0x1a0 May 23 07:44:51 piggy kernel: kernel_init_freeable+0x17d/0x205 May 23 07:44:51 piggy kernel: kernel_init+0xe/0x100 May 23 07:44:51 piggy kernel: ret_from_fork+0x2c/0x40 May 23 07:44:51 piggy kernel: 0xffffffffffffffff
Luis
Quoted 77 lines · Luis R. Rodriguez
Quoted 17 lines · Masami Hiramatsu
Hi Luis,
Quoted 5 lines · On Fri, 19 May 2017 19:28:54 +0200 "Luis R. Rodriguez" <[email protected]>…
Aha! And the winner is:
CONFIG_KPROBES_SANITY_TESTQuoted 7 lines
I confirm disabling it on 4.3.0-rc3 and on linux-next next-20170519 avoids the WARN. I also can confirm using the 'echo dump=mem-area >
/sys/kernel/debug/kmemleak' yields the same trace for both of these kernels.So -- the above kmemleak hack seems to actually work to seek who owns that page.
Now to figure out how the hell
kernel/test_kprobes.cscrews around with things.Quoted 7 lines
Ah, that was fixed recently;
https://marc.info/?l=linux-kernel&m=149076389011850
Note that this patch depends another patch in the series;
I actually boot tested linux-next tag next-20170519 which carries these patches and the WARNING still is there. Please note the issue is with
CONFIG_KPROBES_SANITY_TESTenabled.[ 1.025601] x86/mm: Found insecure W+X mapping at address ffffffffc01e7000/0xffffffffc01e7000 [ 1.026429] ------------[ cut here ]------------ [ 1.026885] WARNING: CPU: 1 PID: 1 at arch/x86/mm/dump_pagetables.c:236 note_page+0x630/0x7e0 [ 1.027711] Modules linked in: [ 1.028032] CPU: 1 PID: 1 Comm: swapper/0 Not tainted 4.12.0-rc1-next-20170519 #151 [ 1.028788] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 1.029928] task: ffff9fd47a5ccc80 task.stack: ffffb6bcc0630000 [ 1.030509] RIP: 0010:note_page+0x630/0x7e0 [ 1.030917] RSP: 0000:ffffb6bcc0633df0 EFLAGS: 00010286 [ 1.031425] RAX: 0000000000000051 RBX: ffffb6bcc0633e88 RCX: ffffffffbb656708 [ 1.032132] RDX: 0000000000000000 RSI: 0000000000000096 RDI: 0000000000000246 [ 1.032834] RBP: ffffb6bcc0633e28 R08: 203a6d6d2f363878 R09: 0000000000000161 [ 1.033539] R10: ffffb6bcc0633dd8 R11: 736e6920646e756f R12: 0000000000000000 [ 1.034235] R13: 0000000000000004 R14: 0000000000000000 R15: 0000000000000000 [ 1.034927] FS: 0000000000000000(0000) GS:ffff9fd47fc80000(0000) knlGS:0000000000000000 [ 1.035722] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 1.036290] CR2: ffffb6bcc073c000 CR3: 0000000053209000 CR4: 00000000000006e0 [ 1.036839] Call Trace: [ 1.037034] ptdump_walk_pgd_level_core+0x3e7/0x490 [ 1.037367] ? 0xffffffffbaa00000 [ 1.037705] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 1.038187] mark_rodata_ro+0xf4/0x100 [ 1.038559] ? rest_init+0x80/0x80 [ 1.038890] kernel_init+0x2f/0x100 [ 1.039235] ret_from_fork+0x2c/0x40 [ 1.039582] Code: 48 c7 43 28 00 00 00 00 48 89 43 20 e9 05 fd ff ff 48 8b 73 10 48 c7 c7 f0 3d 3e bb c6 05 f8 eb bc 00 01 48 89 f2 e8 1d 02 12 00 <0f> ff e9 1f fa ff ff 48 8b 70 20 48 c7 c7 3c ba 3e bb e8 06 02 [ 1.041416] ---[ end trace e726c1b63e5a81a9 ]--- [ 1.041872] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.root@piggy:~# echo dump=0xffffffffc01e7000 > /sys/kernel/debug/kmemleakOn dmesg:
May 23 07:44:51 piggy kernel: kmemleak: Object 0xffffffffc01e7000 (size 335): May 23 07:44:51 piggy kernel: kmemleak: comm "swapper/0", pid 1, jiffies 4294892451 May 23 07:44:51 piggy kernel: kmemleak: min_count = 2 May 23 07:44:51 piggy kernel: kmemleak: count = 2 May 23 07:44:51 piggy kernel: kmemleak: flags = 0x1 May 23 07:44:51 piggy kernel: kmemleak: checksum = 0 May 23 07:44:51 piggy kernel: kmemleak: backtrace: May 23 07:44:51 piggy kernel: kmemleak_alloc+0x4a/0xa0 May 23 07:44:51 piggy kernel: __vmalloc_node_range+0x20c/0x2b0 May 23 07:44:51 piggy kernel: module_alloc+0x67/0xc0 May 23 07:44:51 piggy kernel: arch_ftrace_update_trampoline+0xc1/0x240 May 23 07:44:51 piggy kernel: ftrace_startup+0x92/0x210 May 23 07:44:51 piggy kernel: register_ftrace_function+0x4b/0x60 May 23 07:44:51 piggy kernel: arm_kprobe+0x84/0xc0
Quoted 8 lines
May 23 07:44:51 piggy kernel: register_kprobe+0x59c/0x5e0 May 23 07:44:51 piggy kernel: init_test_probes+0x61/0x560 May 23 07:44:51 piggy kernel: init_kprobes+0x1ea/0x20d May 23 07:44:51 piggy kernel: do_one_initcall+0x52/0x1a0 May 23 07:44:51 piggy kernel: kernel_init_freeable+0x17d/0x205 May 23 07:44:51 piggy kernel: kernel_init+0xe/0x100 May 23 07:44:51 piggy kernel: ret_from_fork+0x2c/0x40 May 23 07:44:51 piggy kernel: 0xffffffffffffffff
Turns out that Thomas Gleixner's patch from today [0] fixes this as the same module_alloc() path was the culprit of the issue. Steven Rostedt however just reported that this patch crashes on his ftracetests, so it would seem we just need to address that last kink to fix this properly.
We can take this further on, in that other thread.
[0] https://lkml.kernel.org/r/alpine.DEB.2.20.1705241459480.2201@nanos
[1] https://lkml.kernel.org/r/[email protected]
Luis
Quoted 43 lines · Luis R. Rodriguez
If the following is a legit forced way to get query the kernel to ask it who owns a page then perhaps this technique can be used in the future to figure out who the hell caused this. Catalin, can you confirm? In this case this is perhaps not a leaked page but I am trying to abuse the kmemleak debugfs API to query who allocated the page. Is that fine?
[ 0.916771] WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:235 note_page+0x63c/0x7e0 [ 0.917636] x86/mm: Found insecure W+X mapping at address ffffffffc03d5000/0xffffffffc03d5000 [ 0.918502] Modules linked in: [ 0.918819] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.11.0-mcgrof-force-config #340 [ 0.919631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.920011] Call Trace: [ 0.920011] dump_stack+0x63/0x81 [ 0.920011] __warn+0xcb/0xf0 [ 0.920011] warn_slowpath_fmt+0x5a/0x80 [ 0.920011] note_page+0x63c/0x7e0 [ 0.920011] ptdump_walk_pgd_level_core+0x3b1/0x460 [ 0.920011] ? 0xffffffff86c00000 [ 0.920011] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.920011] mark_rodata_ro+0xf4/0x100 [ 0.920011] ? rest_init+0x80/0x80 [ 0.920011] kernel_init+0x2a/0x100 [ 0.920011] ret_from_fork+0x2c/0x40 [ 0.925474] ---[ end trace dca00cd779490a2b ]--- [ 0.925959] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.echo dump=0xffffffffc03d5000 >
/sys/kernel/debug/kmemleakdmesg | tail[ 49.209565] kmemleak: Object 0xffffffffc03d5000 (size 335): [ 49.210814] kmemleak: comm "swapper/0", pid 1, jiffies 4294892440 [ 49.212148] kmemleak: min_count = 2 [ 49.212852] kmemleak: count = 0 [ 49.213363] kmemleak: flags = 0x1 [ 49.213363] kmemleak: checksum = 0 [ 49.213363] kmemleak: backtrace: [ 49.213363] kmemleak_alloc+0x4a/0xa0 [ 49.213363] __vmalloc_node_range+0x20a/0x2b0 [ 49.213363] module_alloc+0x67/0xc0 [ 49.213363] arch_ftrace_update_trampoline+0xba/0x260 [ 49.213363] ftrace_startup+0x90/0x210 [ 49.213363] register_ftrace_function+0x4b/0x60 [ 49.213363] arm_kprobe+0x84/0xe0
Quoted 8 lines
[ 49.213363] register_kprobe+0x56e/0x5b0 [ 49.213363] init_test_probes+0x61/0x560 [ 49.213363] init_kprobes+0x1e3/0x206 [ 49.213363] do_one_initcall+0x52/0x1a0 [ 49.213363] kernel_init_freeable+0x178/0x200 [ 49.213363] kernel_init+0xe/0x100 [ 49.213363] ret_from_fork+0x2c/0x40 [ 49.213363] 0xffffffffffffffff
You could as well use kmemleak this way since it tracks the memory allocations. However, it doesn't track alloc_pages and also doesn't track mapping existing pages (vmap etc.)
Signature · 1 line
Catalin
Quoted 53 lines · Catalin Marinas
Quoted 43 lines · Luis R. Rodriguez
If the following is a legit forced way to get query the kernel to ask it who owns a page then perhaps this technique can be used in the future to figure out who the hell caused this. Catalin, can you confirm? In this case this is perhaps not a leaked page but I am trying to abuse the kmemleak debugfs API to query who allocated the page. Is that fine?
[ 0.916771] WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:235 note_page+0x63c/0x7e0 [ 0.917636] x86/mm: Found insecure W+X mapping at address ffffffffc03d5000/0xffffffffc03d5000 [ 0.918502] Modules linked in: [ 0.918819] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.11.0-mcgrof-force-config #340 [ 0.919631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.920011] Call Trace: [ 0.920011] dump_stack+0x63/0x81 [ 0.920011] __warn+0xcb/0xf0 [ 0.920011] warn_slowpath_fmt+0x5a/0x80 [ 0.920011] note_page+0x63c/0x7e0 [ 0.920011] ptdump_walk_pgd_level_core+0x3b1/0x460 [ 0.920011] ? 0xffffffff86c00000 [ 0.920011] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.920011] mark_rodata_ro+0xf4/0x100 [ 0.920011] ? rest_init+0x80/0x80 [ 0.920011] kernel_init+0x2a/0x100 [ 0.920011] ret_from_fork+0x2c/0x40 [ 0.925474] ---[ end trace dca00cd779490a2b ]--- [ 0.925959] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.echo dump=0xffffffffc03d5000 >
/sys/kernel/debug/kmemleakdmesg | tail[ 49.209565] kmemleak: Object 0xffffffffc03d5000 (size 335): [ 49.210814] kmemleak: comm "swapper/0", pid 1, jiffies 4294892440 [ 49.212148] kmemleak: min_count = 2 [ 49.212852] kmemleak: count = 0 [ 49.213363] kmemleak: flags = 0x1 [ 49.213363] kmemleak: checksum = 0 [ 49.213363] kmemleak: backtrace: [ 49.213363] kmemleak_alloc+0x4a/0xa0 [ 49.213363] __vmalloc_node_range+0x20a/0x2b0 [ 49.213363] module_alloc+0x67/0xc0 [ 49.213363] arch_ftrace_update_trampoline+0xba/0x260 [ 49.213363] ftrace_startup+0x90/0x210 [ 49.213363] register_ftrace_function+0x4b/0x60 [ 49.213363] arm_kprobe+0x84/0xe0Quoted 8 lines
[ 49.213363] register_kprobe+0x56e/0x5b0 [ 49.213363] init_test_probes+0x61/0x560 [ 49.213363] init_kprobes+0x1e3/0x206 [ 49.213363] do_one_initcall+0x52/0x1a0 [ 49.213363] kernel_init_freeable+0x178/0x200 [ 49.213363] kernel_init+0xe/0x100 [ 49.213363] ret_from_fork+0x2c/0x40 [ 49.213363] 0xffffffffffffffff
Quoted 3 lines
You could as well use kmemleak this way since it tracks the memory allocations. However, it doesn't track
alloc_pagesand also doesn't track mapping existing pages (vmap etc.)
One thing I've pondered: can we make some debugging mode (kmemleak, perhaps?) check that freed memory is RW at the time it's freed? I once wrote some buggy code that freed an R page and caused an OOPS much later, and this bug here seems likely to be some code that frees RWX memory.
--Andy
Quoted 5 lines · Andy Lutomirski
One thing I've pondered: can we make some debugging mode (kmemleak, perhaps?) check that freed memory is RW at the time it's freed? I once wrote some buggy code that freed an R page and caused an OOPS much later, and this bug here seems likely to be some code that frees RWX memory.
Which begs for even more checks: nothing should ever make a page RWX. Either R, RW, or RX only... (or X too I guess, in the future).
-Kees
Signature · 2 lines
Kees Cook Pixel Security
Quoted 9 lines · Kees Cook
Quoted 5 lines · Andy Lutomirski
One thing I've pondered: can we make some debugging mode (kmemleak, perhaps?) check that freed memory is RW at the time it's freed? I once wrote some buggy code that freed an R page and caused an OOPS much later, and this bug here seems likely to be some code that frees RWX memory.
Which begs for even more checks: nothing should ever make a page RWX. Either R, RW, or RX only... (or X too I guess, in the future).
I could see pages being RWX temporarily during boot. OTOH if we ban RWX outright (after very early boot, anyway), then catching code that messes up and leaves pages RWX gets much easier.
--Andy
Quoted 8 lines · Andy Lutomirski
Quoted 7 lines · Kees Cook
Quoted 5 lines · Andy Lutomirski
One thing I've pondered: can we make some debugging mode (kmemleak, perhaps?) check that freed memory is RW at the time it's freed? I once wrote some buggy code that freed an R page and caused an OOPS much later, and this bug here seems likely to be some code that frees RWX memory.
Quoted 6 lines
Quoted 2 lines
Which begs for even more checks: nothing should ever make a page RWX. Either R, RW, or RX only... (or X too I guess, in the future).
I could see pages being RWX temporarily during boot. OTOH if we ban RWX outright (after very early boot, anyway), then catching code that messes up and leaves pages RWX gets much easier.
Right, early boot is kind of special. It'd be nice to have there, but I meant during normal runtime. We'd probably need to adjust set_memory_rw/ro/nx/x around to have the correct side-effects, instead of just controlling specific bits:
set_memory_rw() (RW_)set_memory_ro() (R__)set_memory_rx() (R_X)set_memory_x() (__X)
That kind of refactoring might be not _too_ bad:
- add set_memory_rx()
- s/\bset_memory_x\b/set_memory_rx/g
- fix what breaks from expecting writable-executable memory
- adjust set_memory_rw() to drop x
- fix what breaks from expecting writable-executable memory
- adjust set_memory_ro() to drop x
- fix what breaks from expecting executable memory
- add set_memory_x() some day...
-Kees
Signature · 2 lines
Kees Cook Pixel Security
Quoted 53 lines · Catalin Marinas
Quoted 43 lines · Luis R. Rodriguez
If the following is a legit forced way to get query the kernel to ask it who owns a page then perhaps this technique can be used in the future to figure out who the hell caused this. Catalin, can you confirm? In this case this is perhaps not a leaked page but I am trying to abuse the kmemleak debugfs API to query who allocated the page. Is that fine?
[ 0.916771] WARNING: CPU: 0 PID: 1 at arch/x86/mm/dump_pagetables.c:235 note_page+0x63c/0x7e0 [ 0.917636] x86/mm: Found insecure W+X mapping at address ffffffffc03d5000/0xffffffffc03d5000 [ 0.918502] Modules linked in: [ 0.918819] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.11.0-mcgrof-force-config #340 [ 0.919631] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.10.2-0-g5f4c7b1-prebuilt.qemu-project.org 04/01/2014 [ 0.920011] Call Trace: [ 0.920011] dump_stack+0x63/0x81 [ 0.920011] __warn+0xcb/0xf0 [ 0.920011] warn_slowpath_fmt+0x5a/0x80 [ 0.920011] note_page+0x63c/0x7e0 [ 0.920011] ptdump_walk_pgd_level_core+0x3b1/0x460 [ 0.920011] ? 0xffffffff86c00000 [ 0.920011] ptdump_walk_pgd_level_checkwx+0x17/0x20 [ 0.920011] mark_rodata_ro+0xf4/0x100 [ 0.920011] ? rest_init+0x80/0x80 [ 0.920011] kernel_init+0x2a/0x100 [ 0.920011] ret_from_fork+0x2c/0x40 [ 0.925474] ---[ end trace dca00cd779490a2b ]--- [ 0.925959] x86/mm: Checked W+X mappings: FAILED, 1 W+X pages found.echo dump=0xffffffffc03d5000 >
/sys/kernel/debug/kmemleakdmesg | tail[ 49.209565] kmemleak: Object 0xffffffffc03d5000 (size 335): [ 49.210814] kmemleak: comm "swapper/0", pid 1, jiffies 4294892440 [ 49.212148] kmemleak: min_count = 2 [ 49.212852] kmemleak: count = 0 [ 49.213363] kmemleak: flags = 0x1 [ 49.213363] kmemleak: checksum = 0 [ 49.213363] kmemleak: backtrace: [ 49.213363] kmemleak_alloc+0x4a/0xa0 [ 49.213363] __vmalloc_node_range+0x20a/0x2b0 [ 49.213363] module_alloc+0x67/0xc0 [ 49.213363] arch_ftrace_update_trampoline+0xba/0x260 [ 49.213363] ftrace_startup+0x90/0x210 [ 49.213363] register_ftrace_function+0x4b/0x60 [ 49.213363] arm_kprobe+0x84/0xe0Quoted 8 lines
[ 49.213363] register_kprobe+0x56e/0x5b0 [ 49.213363] init_test_probes+0x61/0x560 [ 49.213363] init_kprobes+0x1e3/0x206 [ 49.213363] do_one_initcall+0x52/0x1a0 [ 49.213363] kernel_init_freeable+0x178/0x200 [ 49.213363] kernel_init+0xe/0x100 [ 49.213363] ret_from_fork+0x2c/0x40 [ 49.213363] 0xffffffffffffffff
Quoted 2 lines
You could as well use kmemleak this way since it tracks the memory allocations.
Great!
Quoted 2 lines
However, it doesn't track
alloc_pagesand also doesn't track mapping existing pages (vmap etc.)
Can we verify that? If so then the splat from the above complaint could include a follow up dump of the trace, no ? That's *much* more useful.
Luis
Quoted 6 lines · Luis R. Rodriguez
Quoted 5 lines · Luis R. Rodriguez
Using QEMU emulator version 2.7.94 (v2.8.0-rc4-dirty)
I will try updating my distro package for qemu and see if perhaps its this and for the other odd fork issue I reported [0].
Quoted 7 lines
Quoted 1 line
[0] https://lkml.kernel.org/r/CAB=NE6VZXq3y-3pfouYTBUco2Cq2xqoLZrgDFdVx+_=_=SwG_Q@mail.gmail.com
Yeah nope, using my distribution latest:
QEMU emulator version 2.8.0(openSUSE Tumbleweed)
And still both issues are present.
FWIW also compiled and tried to boot with the latest qemu, v2.9.0-rc5 and it also has both issues, so I don't think this is because of the version of qemu.
Luis