KASAN: use-after-free Read in link_path_walk
Reporting the crash: KASAN: use-after-free Read in link_path_walk
This crash has been found in v4.17-rc1 using RaceFuzzer (a modified version of Syzkaller), which we describe more at the end of this report. Our analysis shows that the race occurs when invoking two syscalls concurrently, open() and chroot().
Diagnosis:
We think that it is possible that link_path_walk() dereferences a freed pointer when cleanup_mnt() is executed between path_init() and link_path_walk().
Since I'm not an expert on a file system and don't fully understand the crash, please see a executed program and a crash log below in case that my understanding is wrong.
Executed Program:
Thread0 Thread1mkdir("./file0")
|--------------------------|
| mount("./file0", "./file0", "devpts", 0x0, "")
| |
openat(AT_FDCWD, chroot("./file0")
"/dev/vcs", 0x200, 0x0) umount("./file0", 0x2)openat(), chroot(), umount() syscalls are executed after mount() syscall. We think a race occurs between openat() and chroot() because RaceFuzzer executed openat() and chroot() concurrently.
(Possible) Thread interleaving:
CPU0 (path_openat) CPU1 (cleanup_mnt)
===== =====
s = path_init(nd, flags);
if (IS_ERR(s)) {
put_filp(file);
return ERR_CAST(s);
}deactivate_super(mnt->mnt.mnt_sb);
while (!(error = link_path_walk(s, nd)) &&
// (in link_path_walk())
struct dentry *parent = nd->path.dentry;nd->flags &= ~LOOKUP_JUMPED;
if (unlikely(parent->d_flags & DCACHE_OP_HASH)) { // UAF occured
Crash log:
==================================================================
BUG: KASAN: use-after-free in link_path_walk+0x46e/0xcd0 fs/namei.c:2061 Read of size 4 at addr ffff8801cbe6cb80 by task syz-executor0/28699 CPU: 0 PID: 28699 Comm: syz-executor0 Not tainted 4.17.0-rc1 #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x166/0x21c lib/dump_stack.c:113 print_address_description+0x73/0x250 mm/kasan/report.c:256 kasan_report_error mm/kasan/report.c:354 [inline] kasan_report+0x23f/0x360 mm/kasan/report.c:412 check_memory_region_inline mm/kasan/kasan.c:260 [inline] __asan_load4+0x78/0x80 mm/kasan/kasan.c:698 link_path_walk+0x46e/0xcd0 fs/namei.c:2061 path_openat+0x23c/0x2040 fs/namei.c:3500 do_filp_open+0x175/0x230 fs/namei.c:3535 do_sys_open+0x3c7/0x4a0 fs/open.c:1093 __do_sys_open fs/open.c:1111 [inline] __se_sys_open fs/open.c:1106 [inline] __x64_sys_open+0x4c/0x60 fs/open.c:1106 do_syscall_64+0x15f/0x4a0 arch/x86/entry/common.c:287 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x410601 RSP: 002b:00007f7345489660 EFLAGS: 00000293 ORIG_RAX: 0000000000000002 RAX: ffffffffffffffda RBX: cccccccccccccccd RCX: 0000000000410601 RDX: 0000000000000000 RSI: 0000000000010180 RDI: 00007f7345489710 RBP: 00000000000006e1 R08: 236573756f6d2f74 R09: 0000000000000000 R10: 00000000200004c0 R11: 0000000000000293 R12: 00007f734548a6d4 R13: 00000000ffffffff R14: 00000000006ff5b8 R15: 0000000000000000 Allocated by task 28699: save_stack+0x43/0xd0 mm/kasan/kasan.c:448 set_track mm/kasan/kasan.c:460 [inline] kasan_kmalloc+0xae/0xe0 mm/kasan/kasan.c:553 kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:490 kmem_cache_alloc+0x12e/0x760 mm/slab.c:3554 __d_alloc+0xc0/0x6e0 fs/dcache.c:1638 d_alloc_anon fs/dcache.c:1742 [inline] d_make_root+0x2d/0x70 fs/dcache.c:1934 devpts_fill_super+0x23b/0x500 fs/devpts/inode.c:482 mount_nodev+0x59/0xd0 fs/super.c:1211 devpts_mount+0x2c/0x40 fs/devpts/inode.c:509 mount_fs+0x50/0x200 fs/super.c:1268 vfs_kern_mount.part.26+0xbc/0x2c0 fs/namespace.c:1037 vfs_kern_mount fs/namespace.c:2514 [inline] do_new_mount fs/namespace.c:2517 [inline] do_mount+0xb82/0x1bb0 fs/namespace.c:2847 ksys_mount+0xab/0x120 fs/namespace.c:3063 __do_sys_mount fs/namespace.c:3077 [inline] __se_sys_mount fs/namespace.c:3074 [inline] __x64_sys_mount+0x67/0x80 fs/namespace.c:3074 do_syscall_64+0x15f/0x4a0 arch/x86/entry/common.c:287 entry_SYSCALL_64_after_hwframe+0x49/0xbe Freed by task 28700: save_stack+0x43/0xd0 mm/kasan/kasan.c:448 set_track mm/kasan/kasan.c:460 [inline] __kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521 kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528 __cache_free mm/slab.c:3498 [inline] kmem_cache_free+0x83/0x2a0 mm/slab.c:3756 __d_free fs/dcache.c:257 [inline] dentry_free+0x8c/0xe0 fs/dcache.c:347 __dentry_kill+0x3d6/0x440 fs/dcache.c:582 dentry_kill+0x8f/0x320 fs/dcache.c:686 dput.part.22+0x430/0x4e0 fs/dcache.c:850 dput fs/dcache.c:830 [inline] do_one_tree+0x43/0x50 fs/dcache.c:1523 shrink_dcache_for_umount+0xa5/0x1c0 fs/dcache.c:1537 generic_shutdown_super+0xb0/0x330 fs/super.c:425 kill_anon_super fs/super.c:1037 [inline] kill_litter_super+0x48/0x60 fs/super.c:1047 devpts_kill_sb+0x49/0x50 fs/devpts/inode.c:519 deactivate_locked_super+0x71/0xb0 fs/super.c:313 deactivate_super+0x10f/0x150 fs/super.c:344 cleanup_mnt+0x6b/0xc0 fs/namespace.c:1173 __cleanup_mnt+0x16/0x20 fs/namespace.c:1180 task_work_run+0x152/0x1b0 kernel/task_work.c:113 tracehook_notify_resume include/linux/tracehook.h:191 [inline] exit_to_usermode_loop+0x262/0x270 arch/x86/entry/common.c:166 prepare_exit_to_usermode arch/x86/entry/common.c:196 [inline] syscall_return_slowpath arch/x86/entry/common.c:265 [inline] do_syscall_64+0x473/0x4a0 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe The buggy address belongs to the object at ffff8801cbe6cb80 which belongs to the cache dentry(17:syz0) of size 288 The buggy address is located 0 bytes inside of 288-byte region [ffff8801cbe6cb80, ffff8801cbe6cca0) The buggy address belongs to the page: page:ffffea00072f9b00 count:1 mapcount:0 mapping:ffff8801cbe6c080 index:0x0 flags: 0x2fffc0000000100(slab) raw: 02fffc0000000100 ffff8801cbe6c080 0000000000000000 000000010000000b raw: ffffea00072f8ca0 ffffea00072f8da0 ffff8801dc812c80 ffff8801de41a740 page dumped because: kasan: bad access detected
page->mem_cgroup:ffff8801de41a740
Memory state around the buggy address:
ffff8801cbe6ca80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff8801cbe6cb00: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
>ffff8801cbe6cb80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff8801cbe6cc00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff8801cbe6cc80: fb fb fb fb fc fc fc fc fc fc fc fc fb fb fb fb
=================================================================== About RaceFuzzer
RaceFuzzer is a customized version of Syzkaller, specifically tailored to find race condition bugs in the Linux kernel. While we leverage many different technique, the notable feature of RaceFuzzer is in leveraging a custom hypervisor (QEMU/KVM) to interleave the scheduling. In particular, we modified the hypervisor to intentionally stall a per-core execution, which is similar to supporting per-core breakpoint functionality. This allows RaceFuzzer to force the kernel to deterministically trigger racy condition (which may rarely happen in practice due to randomness in scheduling).
RaceFuzzer's C repro always pinpoints two racy syscalls. Since C repro's scheduling synchronization should be performed at the user space, its reproducibility is limited (reproduction may take from 1 second to 10 minutes (or even more), depending on a bug). This is because, while RaceFuzzer precisely interleaves the scheduling at the kernel's instruction level when finding this bug, C repro cannot fully utilize such a feature. Please disregard all code related to "should_hypercall" in the C repro, as this is only for our debugging purposes using our own hypervisor.
I think that below two crashes are also related to the same race issue.
KASAN: use-after-free Read in nd_jump_root, found in v4.17-rc1 KASAN: use-after-free in set_root, found in v4.18-rc3
==================================================================
BUG: KASAN: use-after-free in nd_jump_root+0x69/0x160 fs/namei.c:852 Read of size 8 at addr ffff8801eb677e58 by task syz-executor0/20521 CPU: 0 PID: 20521 Comm: syz-executor0 Not tainted 4.17.0-rc1 #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x166/0x21c lib/dump_stack.c:113 print_address_description+0x73/0x250 mm/kasan/report.c:256 kasan_report_error mm/kasan/report.c:354 [inline] kasan_report+0x23f/0x360 mm/kasan/report.c:412 check_memory_region_inline mm/kasan/kasan.c:260 [inline] __asan_load8+0x54/0x90 mm/kasan/kasan.c:699 nd_jump_root+0x69/0x160 fs/namei.c:852 path_init+0x9ca/0x1190 fs/namei.c:2165 path_openat+0x140/0x2040 fs/namei.c:3495 do_filp_open+0x175/0x230 fs/namei.c:3535 do_sys_open+0x3c7/0x4a0 fs/open.c:1093 __do_sys_openat fs/open.c:1120 [inline] __se_sys_openat fs/open.c:1114 [inline] __x64_sys_openat+0x59/0x70 fs/open.c:1114 do_syscall_64+0x15f/0x4a0 arch/x86/entry/common.c:287 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x456419 RSP: 002b:00007fb317cd2b28 EFLAGS: 00000246 ORIG_RAX: 0000000000000101 RAX: ffffffffffffffda RBX: 000000000072bee0 RCX: 0000000000456419 RDX: 0000000000000101 RSI: 00000000200001c0 RDI: ffffffffffffff9c RBP: 0000000000000497 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 00007fb317cd36d4 R13: 00000000ffffffff R14: 00000000006fbec8 R15: 0000000000000000 Allocated by task 20521: save_stack+0x43/0xd0 mm/kasan/kasan.c:448 set_track mm/kasan/kasan.c:460 [inline] kasan_kmalloc+0xae/0xe0 mm/kasan/kasan.c:553 kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:490 kmem_cache_alloc+0x12e/0x760 mm/slab.c:3554 __d_alloc+0xc0/0x6e0 fs/dcache.c:1638 d_alloc_anon fs/dcache.c:1742 [inline] d_make_root+0x2d/0x70 fs/dcache.c:1934 devpts_fill_super+0x23b/0x500 fs/devpts/inode.c:482 mount_nodev+0x59/0xd0 fs/super.c:1211 devpts_mount+0x2c/0x40 fs/devpts/inode.c:509 mount_fs+0x50/0x200 fs/super.c:1268 vfs_kern_mount.part.26+0xbc/0x2c0 fs/namespace.c:1037 vfs_kern_mount fs/namespace.c:2514 [inline] do_new_mount fs/namespace.c:2517 [inline] do_mount+0xb82/0x1bb0 fs/namespace.c:2847 ksys_mount+0xab/0x120 fs/namespace.c:3063 __do_sys_mount fs/namespace.c:3077 [inline] __se_sys_mount fs/namespace.c:3074 [inline] __x64_sys_mount+0x67/0x80 fs/namespace.c:3074 do_syscall_64+0x15f/0x4a0 arch/x86/entry/common.c:287 entry_SYSCALL_64_after_hwframe+0x49/0xbe Freed by task 20522: save_stack+0x43/0xd0 mm/kasan/kasan.c:448 set_track mm/kasan/kasan.c:460 [inline] __kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521 kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528 __cache_free mm/slab.c:3498 [inline] kmem_cache_free+0x83/0x2a0 mm/slab.c:3756 __d_free fs/dcache.c:257 [inline] dentry_free+0x8c/0xe0 fs/dcache.c:347 __dentry_kill+0x3d6/0x440 fs/dcache.c:582 dentry_kill+0x8f/0x320 fs/dcache.c:686 dput.part.22+0x430/0x4e0 fs/dcache.c:850 dput fs/dcache.c:830 [inline] do_one_tree+0x43/0x50 fs/dcache.c:1523 shrink_dcache_for_umount+0xa5/0x1c0 fs/dcache.c:1537 generic_shutdown_super+0xb0/0x330 fs/super.c:425 kill_anon_super fs/super.c:1037 [inline] kill_litter_super+0x48/0x60 fs/super.c:1047 devpts_kill_sb+0x49/0x50 fs/devpts/inode.c:519 deactivate_locked_super+0x71/0xb0 fs/super.c:313 deactivate_super+0x10f/0x150 fs/super.c:344 cleanup_mnt+0x6b/0xc0 fs/namespace.c:1173 __cleanup_mnt+0x16/0x20 fs/namespace.c:1180 task_work_run+0x152/0x1b0 kernel/task_work.c:113 tracehook_notify_resume include/linux/tracehook.h:191 [inline] exit_to_usermode_loop+0x262/0x270 arch/x86/entry/common.c:166 prepare_exit_to_usermode arch/x86/entry/common.c:196 [inline] syscall_return_slowpath arch/x86/entry/common.c:265 [inline] do_syscall_64+0x473/0x4a0 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe The buggy address belongs to the object at ffff8801eb677e00 which belongs to the cache dentry(17:syz0) of size 288 The buggy address is located 88 bytes inside of 288-byte region [ffff8801eb677e00, ffff8801eb677f20) The buggy address belongs to the page: page:ffffea0007ad9dc0 count:1 mapcount:0 mapping:ffff8801eb677040 index:0x0 flags: 0x2fffc0000000100(slab) raw: 02fffc0000000100 ffff8801eb677040 0000000000000000 000000010000000b raw: ffffea0007b9d2a0 ffffea0007b9ed20 ffff8801ef225300 ffff8801ed0249c0 page dumped because: kasan: bad access detected
page->mem_cgroup:ffff8801ed0249c0
Memory state around the buggy address:
ffff8801eb677d00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff8801eb677d80: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
>ffff8801eb677e00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff8801eb677e80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff8801eb677f00: fb fb fb fb fc fc fc fc fc fc fc fc fc fc fc fc
====================================================================================================================================
BUG: KASAN: use-after-free in __read_once_size include/linux/compiler.h:188 [inline] BUG: KASAN: use-after-free in __read_seqcount_begin include/linux/seqlock.h:113 [inline] BUG: KASAN: use-after-free in set_root+0x252/0x3f0 fs/namei.c:820 Read of size 4 at addr ffff88019e5d4b88 by task syz-executor0/30297 CPU: 1 PID: 30297 Comm: syz-executor0 Not tainted 4.18.0-rc3 #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x16e/0x22c lib/dump_stack.c:113 print_address_description+0x73/0x250 mm/kasan/report.c:256 kasan_report_error mm/kasan/report.c:354 [inline] kasan_report+0x259/0x380 mm/kasan/report.c:412 check_memory_region_inline mm/kasan/kasan.c:260 [inline] __asan_load4+0x78/0x80 mm/kasan/kasan.c:698 __read_once_size include/linux/compiler.h:188 [inline] __read_seqcount_begin include/linux/seqlock.h:113 [inline] set_root+0x252/0x3f0 fs/namei.c:820 path_init+0x9dd/0x11b0 fs/namei.c:2164 path_openat+0x147/0x1fb0 fs/namei.c:3534 do_filp_open+0x181/0x250 fs/namei.c:3574 do_sys_open+0x3da/0x4b0 fs/open.c:1101 __do_sys_openat fs/open.c:1128 [inline] __se_sys_openat fs/open.c:1122 [inline] __x64_sys_openat+0x59/0x70 fs/open.c:1122 do_syscall_64+0x167/0x4b0 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x456469 Code: 1d ba fb ff c3 66 2e 0f 1f 84 00 00 00 00 00 66 90 48 89 f8 48
89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 0f 83 eb b9 fb ff c3 66 2e 0f 1f 84 00 00 00 00
RSP: 002b:00007f5e06564b28 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 000000000072bfa0 RCX: 0000000000456469
RDX: 0000000000000080 RSI: 0000000020000140 RDI: ffffffffffffff9c
RBP: 00000000000004e4 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 00007f5e065656d4
R13: 00000000ffffffff R14: 00000000006fc600 R15: 0000000000000000
Allocated by task 30296: save_stack+0x43/0xd0 mm/kasan/kasan.c:448 set_track mm/kasan/kasan.c:460 [inline] kasan_kmalloc+0xae/0xe0 mm/kasan/kasan.c:553 kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:490 kmem_cache_alloc+0x12e/0x750 mm/slab.c:3554 __d_alloc+0xc2/0x720 fs/dcache.c:1616 d_alloc_anon fs/dcache.c:1720 [inline] d_make_root+0x2d/0x70 fs/dcache.c:1934 devpts_fill_super+0x23b/0x500 fs/devpts/inode.c:482 mount_nodev+0x59/0xd0 fs/super.c:1220 devpts_mount+0x2c/0x40 fs/devpts/inode.c:509 mount_fs+0x50/0x200 fs/super.c:1277 vfs_kern_mount.part.26+0xc4/0x2d0 fs/namespace.c:1037 vfs_kern_mount fs/namespace.c:2515 [inline] do_new_mount fs/namespace.c:2518 [inline] do_mount+0xbd7/0x1c90 fs/namespace.c:2848 ksys_mount+0xab/0x120 fs/namespace.c:3064 __do_sys_mount fs/namespace.c:3078 [inline] __se_sys_mount fs/namespace.c:3075 [inline] __x64_sys_mount+0x67/0x80 fs/namespace.c:3075 do_syscall_64+0x167/0x4b0 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe Freed by task 30296: save_stack+0x43/0xd0 mm/kasan/kasan.c:448 set_track mm/kasan/kasan.c:460 [inline] __kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521 kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528 __cache_free mm/slab.c:3498 [inline] kmem_cache_free+0x83/0x2a0 mm/slab.c:3756 __d_free fs/dcache.c:257 [inline] dentry_free+0x8c/0xe0 fs/dcache.c:347 __dentry_kill+0x3fe/0x470 fs/dcache.c:582 dentry_kill+0x8f/0x320 fs/dcache.c:687 dput+0x450/0x4e0 fs/dcache.c:848 do_one_tree+0x37/0x40 fs/dcache.c:1531 shrink_dcache_for_umount+0xad/0x1d0 fs/dcache.c:1545 generic_shutdown_super+0xb8/0x340 fs/super.c:438 kill_anon_super fs/super.c:1046 [inline] kill_litter_super+0x48/0x60 fs/super.c:1056 devpts_kill_sb+0x49/0x50 fs/devpts/inode.c:519 deactivate_locked_super+0x71/0xb0 fs/super.c:326 deactivate_super+0x11d/0x160 fs/super.c:357 cleanup_mnt+0x6b/0xc0 fs/namespace.c:1174 __cleanup_mnt+0x16/0x20 fs/namespace.c:1181 task_work_run+0x15a/0x1c0 kernel/task_work.c:113 tracehook_notify_resume include/linux/tracehook.h:192 [inline] exit_to_usermode_loop+0x2a3/0x2b0 arch/x86/entry/common.c:166 prepare_exit_to_usermode arch/x86/entry/common.c:197 [inline] syscall_return_slowpath arch/x86/entry/common.c:268 [inline] do_syscall_64+0x485/0x4b0 arch/x86/entry/common.c:293 entry_SYSCALL_64_after_hwframe+0x49/0xbe The buggy address belongs to the object at ffff88019e5d4b80 which belongs to the cache dentry(17:syz0) of size 288 The buggy address is located 8 bytes inside of 288-byte region [ffff88019e5d4b80, ffff88019e5d4ca0) The buggy address belongs to the page: page:ffffea0006797500 count:1 mapcount:0 mapping:ffff8801ec05d7c0 index:0x0 flags: 0x2fffc0000000100(slab) raw: 02fffc0000000100 ffffea0006796148 ffffea00067cd648 ffff8801ec05d7c0 raw: 0000000000000000 ffff88019e5d4080 000000010000000b ffff8801dc63e8c0 page dumped because: kasan: bad access detected
page->mem_cgroup:ffff8801dc63e8c0
Memory state around the buggy address:
ffff88019e5d4a80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88019e5d4b00: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc
>ffff88019e5d4b80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff88019e5d4c00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff88019e5d4c80: fb fb fb fb fc fc fc fc fc fc fc fc fb fb fb fb
==================================================================Quoted 180 lines · Dae R. Jeong
Reporting the crash: KASAN: use-after-free Read in
link_path_walkThis crash has been found in v4.17-rc1 using RaceFuzzer (a modified version of Syzkaller), which we describe more at the end of this report. Our analysis shows that the race occurs when invoking two syscalls concurrently,
open()andchroot().Diagnosis:
We think that it is possible thatlink_path_walk()dereferences a freed pointer whencleanup_mnt()is executed betweenpath_init()andlink_path_walk().Since I'm not an expert on a file system and don't fully understand the crash, please see a executed program and a crash log below in case that my understanding is wrong.
Executed Program:
Thread0 Thread1mkdir("./file0")|--------------------------| | mount("./file0", "./file0", "devpts", 0x0, "") | | openat(AT_FDCWD, chroot("./file0") "/dev/vcs", 0x200, 0x0) umount("./file0", 0x2)
openat(),chroot(),umount()syscalls are executed aftermount()syscall. We think a race occurs betweenopenat()andchroot()because RaceFuzzer executedopenat()andchroot()concurrently.(Possible) Thread interleaving:
CPU0 (path_openat) CPU1 (cleanup_mnt)
===== =====
s =path_init(nd, flags);if (IS_ERR(s)) { put_filp(file); return ERR_CAST(s); }deactivate_super(mnt->mnt.mnt_sb);while (!(error =
link_path_walk(s, nd)) &&// (in
link_path_walk())
struct dentry *parent =nd->path.dentry;nd->flags&= ~LOOKUP_JUMPED;
if (unlikely(parent->d_flags & DCACHE_OP_HASH)) { // UAF occuredCrash log:
==================================================================BUG: KASAN: use-after-free in link_path_walk+0x46e/0xcd0 fs/namei.c:2061 Read of size 4 at addr ffff8801cbe6cb80 by task syz-executor0/28699 CPU: 0 PID: 28699 Comm: syz-executor0 Not tainted 4.17.0-rc1 #1 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.8.2-0-g33fbe13 by qemu-project.org 04/01/2014 Call Trace: __dump_stack lib/dump_stack.c:77 [inline] dump_stack+0x166/0x21c lib/dump_stack.c:113 print_address_description+0x73/0x250 mm/kasan/report.c:256 kasan_report_error mm/kasan/report.c:354 [inline] kasan_report+0x23f/0x360 mm/kasan/report.c:412 check_memory_region_inline mm/kasan/kasan.c:260 [inline] __asan_load4+0x78/0x80 mm/kasan/kasan.c:698 link_path_walk+0x46e/0xcd0 fs/namei.c:2061 path_openat+0x23c/0x2040 fs/namei.c:3500 do_filp_open+0x175/0x230 fs/namei.c:3535 do_sys_open+0x3c7/0x4a0 fs/open.c:1093 __do_sys_open fs/open.c:1111 [inline] __se_sys_open fs/open.c:1106 [inline] __x64_sys_open+0x4c/0x60 fs/open.c:1106 do_syscall_64+0x15f/0x4a0 arch/x86/entry/common.c:287 entry_SYSCALL_64_after_hwframe+0x49/0xbe RIP: 0033:0x410601 RSP: 002b:00007f7345489660 EFLAGS: 00000293 ORIG_RAX: 0000000000000002 RAX: ffffffffffffffda RBX: cccccccccccccccd RCX: 0000000000410601 RDX: 0000000000000000 RSI: 0000000000010180 RDI: 00007f7345489710 RBP: 00000000000006e1 R08: 236573756f6d2f74 R09: 0000000000000000 R10: 00000000200004c0 R11: 0000000000000293 R12: 00007f734548a6d4 R13: 00000000ffffffff R14: 00000000006ff5b8 R15: 0000000000000000 Allocated by task 28699: save_stack+0x43/0xd0 mm/kasan/kasan.c:448 set_track mm/kasan/kasan.c:460 [inline] kasan_kmalloc+0xae/0xe0 mm/kasan/kasan.c:553 kasan_slab_alloc+0x12/0x20 mm/kasan/kasan.c:490 kmem_cache_alloc+0x12e/0x760 mm/slab.c:3554 __d_alloc+0xc0/0x6e0 fs/dcache.c:1638 d_alloc_anon fs/dcache.c:1742 [inline] d_make_root+0x2d/0x70 fs/dcache.c:1934 devpts_fill_super+0x23b/0x500 fs/devpts/inode.c:482 mount_nodev+0x59/0xd0 fs/super.c:1211 devpts_mount+0x2c/0x40 fs/devpts/inode.c:509 mount_fs+0x50/0x200 fs/super.c:1268 vfs_kern_mount.part.26+0xbc/0x2c0 fs/namespace.c:1037 vfs_kern_mount fs/namespace.c:2514 [inline] do_new_mount fs/namespace.c:2517 [inline] do_mount+0xb82/0x1bb0 fs/namespace.c:2847 ksys_mount+0xab/0x120 fs/namespace.c:3063 __do_sys_mount fs/namespace.c:3077 [inline] __se_sys_mount fs/namespace.c:3074 [inline] __x64_sys_mount+0x67/0x80 fs/namespace.c:3074 do_syscall_64+0x15f/0x4a0 arch/x86/entry/common.c:287 entry_SYSCALL_64_after_hwframe+0x49/0xbe Freed by task 28700: save_stack+0x43/0xd0 mm/kasan/kasan.c:448 set_track mm/kasan/kasan.c:460 [inline] __kasan_slab_free+0x11a/0x170 mm/kasan/kasan.c:521 kasan_slab_free+0xe/0x10 mm/kasan/kasan.c:528 __cache_free mm/slab.c:3498 [inline] kmem_cache_free+0x83/0x2a0 mm/slab.c:3756 __d_free fs/dcache.c:257 [inline] dentry_free+0x8c/0xe0 fs/dcache.c:347 __dentry_kill+0x3d6/0x440 fs/dcache.c:582 dentry_kill+0x8f/0x320 fs/dcache.c:686 dput.part.22+0x430/0x4e0 fs/dcache.c:850 dput fs/dcache.c:830 [inline] do_one_tree+0x43/0x50 fs/dcache.c:1523 shrink_dcache_for_umount+0xa5/0x1c0 fs/dcache.c:1537 generic_shutdown_super+0xb0/0x330 fs/super.c:425 kill_anon_super fs/super.c:1037 [inline] kill_litter_super+0x48/0x60 fs/super.c:1047 devpts_kill_sb+0x49/0x50 fs/devpts/inode.c:519 deactivate_locked_super+0x71/0xb0 fs/super.c:313 deactivate_super+0x10f/0x150 fs/super.c:344 cleanup_mnt+0x6b/0xc0 fs/namespace.c:1173 __cleanup_mnt+0x16/0x20 fs/namespace.c:1180 task_work_run+0x152/0x1b0 kernel/task_work.c:113 tracehook_notify_resume include/linux/tracehook.h:191 [inline] exit_to_usermode_loop+0x262/0x270 arch/x86/entry/common.c:166 prepare_exit_to_usermode arch/x86/entry/common.c:196 [inline] syscall_return_slowpath arch/x86/entry/common.c:265 [inline] do_syscall_64+0x473/0x4a0 arch/x86/entry/common.c:290 entry_SYSCALL_64_after_hwframe+0x49/0xbe The buggy address belongs to the object at ffff8801cbe6cb80 which belongs to the cache dentry(17:syz0) of size 288 The buggy address is located 0 bytes inside of 288-byte region [ffff8801cbe6cb80, ffff8801cbe6cca0) The buggy address belongs to the page: page:ffffea00072f9b00 count:1 mapcount:0 mapping:ffff8801cbe6c080 index:0x0 flags: 0x2fffc0000000100(slab) raw: 02fffc0000000100 ffff8801cbe6c080 0000000000000000 000000010000000b raw: ffffea00072f8ca0 ffffea00072f8da0 ffff8801dc812c80 ffff8801de41a740 page dumped because: kasan: bad access detectedpage->mem_cgroup:ffff8801de41a740
Memory state around the buggy address: ffff8801cbe6ca80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ffff8801cbe6cb00: fb fb fb fb fb fb fb fb fc fc fc fc fc fc fc fc >ffff8801cbe6cb80: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ^ ffff8801cbe6cc00: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb ffff8801cbe6cc80: fb fb fb fb fc fc fc fc fc fc fc fc fb fb fb fb =================================================================== About RaceFuzzer
RaceFuzzer is a customized version of Syzkaller, specifically tailored to find race condition bugs in the Linux kernel. While we leverage many different technique, the notable feature of RaceFuzzer is in leveraging a custom hypervisor (QEMU/KVM) to interleave the scheduling. In particular, we modified the hypervisor to intentionally stall a per-core execution, which is similar to supporting per-core breakpoint functionality. This allows RaceFuzzer to force the kernel to deterministically trigger racy condition (which may rarely happen in practice due to randomness in scheduling).
RaceFuzzer's C repro always pinpoints two racy syscalls. Since C repro's scheduling synchronization should be performed at the user space, its reproducibility is limited (reproduction may take from 1 second to 10 minutes (or even more), depending on a bug). This is because, while RaceFuzzer precisely interleaves the scheduling at the kernel's instruction level when finding this bug, C repro cannot fully utilize such a feature. Please disregard all code related to "
should_hypercall" in the C repro, as this is only for our debugging purposes using our own hypervisor.
Al Viro
maintainerQuoted 33 lines · Dae R. Jeong
Diagnosis:
We think that it is possible thatlink_path_walk()dereferences a freed pointer whencleanup_mnt()is executed betweenpath_init()andlink_path_walk().Since I'm not an expert on a file system and don't fully understand the crash, please see a executed program and a crash log below in case that my understanding is wrong.
Executed Program:
Thread0 Thread1mkdir("./file0")|--------------------------| | mount("./file0", "./file0", "devpts", 0x0, "") | | openat(AT_FDCWD, chroot("./file0") "/dev/vcs", 0x200, 0x0) umount("./file0", 0x2)
openat(),chroot(),umount()syscalls are executed aftermount()syscall. We think a race occurs betweenopenat()andchroot()because RaceFuzzer executedopenat()andchroot()concurrently.(Possible) Thread interleaving:
CPU0 (path_openat) CPU1 (cleanup_mnt)
===== =====
s =path_init(nd, flags);if (IS_ERR(s)) { put_filp(file); return ERR_CAST(s); }
Quoted 8 lines
deactivate_super(mnt->mnt.mnt_sb);while (!(error =
link_path_walk(s, nd)) &&// (in
link_path_walk())
struct dentry *parent =nd->path.dentry;nd->flags&= ~LOOKUP_JUMPED;
if (unlikely(parent->d_flags & DCACHE_OP_HASH)) { // UAF occured
Do we have LOOKUP_RCU in nd->flags at that point? And how in hell did we get that dentry there? In LOOKUP_RCU mode no freeing should be happening until after we call rcu_read_unlock(), unless the final dput() has happened before rcu_read_lock(). In which case we shouldn't have gotten to that dentry in the first place. And in non-LOOKUP_RCU mode we are bloody well holding references to everything (vfsmount and dentry alike), so that deactivate_super() shouldn't have been called as long as we are holding that reference.
Details, please. Ideally - how to reproduce that.
Al Viro
maintainerQuoted 20 lines · Al Viro
Quoted 19 lines · Dae R. Jeong
Diagnosis:
We think that it is possible thatlink_path_walk()dereferences a freed pointer whencleanup_mnt()is executed betweenpath_init()andlink_path_walk().Since I'm not an expert on a file system and don't fully understand the crash, please see a executed program and a crash log below in case that my understanding is wrong.
Executed Program:
Thread0 Thread1mkdir("./file0")|--------------------------| | mount("./file0", "./file0", "devpts", 0x0, "") | | openat(AT_FDCWD, chroot("./file0") "/dev/vcs", 0x200, 0x0) umount("./file0", 0x2)
Quoted 7 lines
Quoted 7 lines
openat(),chroot(),umount()syscalls are executed aftermount()syscall. We think a race occurs betweenopenat()andchroot()because RaceFuzzer executedopenat()andchroot()concurrently.(Possible) Thread interleaving:
CPU0 (path_openat) CPU1 (cleanup_mnt)
Wait a bloody minute. Where does cleanup_mnt() come from in that thing? You are doing lazy-umount of the thing you've chrooted into; if it ends up with zero refcount on that mount, we are already in deep, deep trouble, races with open() on not. Simply following that with stat / (in thread 1, without thread0 at all) would end up accessing the same vfsmount. And if it's been freed, we are well and truly fucked, race or no race.
I really want details. *Is* cleanup_mnt() called by thread 1 in your reproducer before the use-after-free hits? And what's the root of thread 0 at that point?
Because our fuzzer has a problem, I don't have a C reproducer so far.
I reported the crash becasue I saw the crash repeatedly in our fuzzer and I hoped the report is helpful. But it seems not enough. If I was wrong and I made you confused, I am really sorry for that.
Could you give me a second?
I am trying to fix our fuzzer and to make a C reproducer.
I think the C reproducer is necessary here.
Quoted 39 lines · On 24 Jul 2018, 2:29 PM +0900, Al Viro <[email protected]>, wrote:
Quoted 20 lines · Al Viro
Quoted 19 lines · Dae R. Jeong
Diagnosis:
We think that it is possible thatlink_path_walk()dereferences a freed pointer whencleanup_mnt()is executed betweenpath_init()andlink_path_walk().Since I'm not an expert on a file system and don't fully understand the crash, please see a executed program and a crash log below in case that my understanding is wrong.
Executed Program:
Thread0 Thread1mkdir("./file0")|--------------------------| | mount("./file0", "./file0", "devpts", 0x0, "") | |openat(AT_FDCWD,
chroot("./file0")
"/dev/vcs",0x200,0x0)umount("./file0", 0x2)Quoted 7 lines
Quoted 7 lines
openat(),chroot(),umount()syscalls are executed aftermount()syscall. We think a race occurs betweenopenat()andchroot()because RaceFuzzer executedopenat()andchroot()concurrently.(Possible) Thread interleaving:
CPU0 (path_openat) CPU1 (cleanup_mnt)Wait a bloody minute. Where does
cleanup_mnt()come from in that thing? You are doing lazy-umount of the thing you've chrooted into; if it ends up with zero refcount on that mount, we are already in deep, deep trouble, races withopen()on not. Simply following that with stat / (in thread 1, without thread0 at all) would end up accessing the same vfsmount. And if it's been freed, we are well and truly fucked, race or no race.I really want details. *Is*
cleanup_mnt()called by thread 1 in your reproducer before the use-after-free hits? And what's the root of thread 0 at that point?
Al Viro
maintainerQuoted 5 lines · Al Viro
Do we have
LOOKUP_RCUinnd->flagsat that point? And how in hell did we get that dentry there? InLOOKUP_RCUmode no freeing should be happening until after we callrcu_read_unlock(), unless the finaldput()has happened beforercu_read_lock(). In which case we shouldn't have gotten to that dentry in the first place.
... except that we never set DCACHE_RCUACCESS for root dentry. Which invalidates the normal "if we run into dentry in lazy mode, its memory won't be freed until we drop rcu_read_lock"... d_make_root() definitely needs to set DCACHE_RCUACCESS; whether it's all there is or you are hitting something else is a separate question, of course...
Quoted 6 lines
And in non-LOOKUP_RCU
mode we are bloody well holding references to everything (vfsmount and dentry alike), so thatdeactivate_super()shouldn't have been called as long as we are holding that reference.Details, please. Ideally - how to reproduce that.
Is there any way to tell KASAN that we want a crashdump triggered? That would've been really useful for post-mortems...