fanotify use after free.
Subsystems (2)
Jan,
since yesterdays changes, on boot I see a flood of messages from slub debug during boot..
=============================================================================
BUG fanotify_event_info (Not tainted): Poison overwritten
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: 0xffff880247e45bc8-0xffff880247e45bcb. First byte 0x0 instead of 0x6b
INFO: Allocated in fanotify_handle_event+0x136/0x390 age=0 cpu=0 pid=293
__slab_alloc+0x456/0x565 kmem_cache_alloc+0x1fe/0x260 fanotify_handle_event+0x136/0x390 send_to_group+0xd3/0x1c0 fsnotify+0x1c8/0x340 open_exec+0xe2/0x120 load_elf_binary+0x7b7/0x18e0 search_binary_handler+0x94/0x1b0 do_execve_common.isra.26+0x5d7/0x7d0 SyS_execve+0x36/0x50 stub_execve+0x69/0xa0 INFO: Freed in fanotify_free_event+0x2e/0x40 age=0 cpu=3 pid=290 __slab_free+0x4a/0x382 kmem_cache_free+0x1c9/0x210 fanotify_free_event+0x2e/0x40 fsnotify_destroy_event+0x21/0x30 fanotify_read+0x39e/0x5e0 vfs_read+0x9b/0x160 SyS_read+0x58/0xb0 tracesys+0xdd/0xe2
INFO: Slab 0xffffea00091f9100 objects=20 used=20 fp=0x (null) flags=0x20000000004080
INFO: Object 0xffff880247e45b90 @offset=7056 fp=0xffff880247e44000
Bytes b4 ffff880247e45b80: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ Object ffff880247e45b90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object ffff880247e45ba0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object ffff880247e45bb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object ffff880247e45bc0: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk.
Redzone ffff880247e45bd0: bb bb bb bb bb bb bb bb ........
Padding ffff880247e45d10: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
CPU: 0 PID: 293 Comm: mount Tainted: G B 3.13.0+ #28 ffff880247e45b90 000000008c7fe87c ffff8800874cbb28 ffffffff9c710632 ffff88024a776ac0 ffff8800874cbb68 ffffffff9c194dad 0000000000000008 ffff880200000001 ffff880247e45bcc ffff88024a776ac0 000000000000006b Call Trace: [<ffffffff9c710632>] dump_stack+0x4e/0x7a [<ffffffff9c194dad>] print_trailer+0x14d/0x200 [<ffffffff9c19505f>] check_bytes_and_report+0xcf/0x110 [<ffffffff9c196037>] check_object+0x1d7/0x250 [<ffffffff9c1f4ae6>] ? fanotify_handle_event+0x136/0x390 [<ffffffff9c70ead7>] alloc_debug_processing+0x76/0x118 [<ffffffff9c70f77d>] __slab_alloc+0x456/0x565 [<ffffffff9c1f4ae6>] ? fanotify_handle_event+0x136/0x390 [<ffffffff9c1ccea4>] ? mntput+0x24/0x40 [<ffffffff9c1b5dc9>] ? terminate_walk+0x69/0x70 [<ffffffff9c1ba6fe>] ? do_last+0x25e/0x1390 [<ffffffff9c1b6cf8>] ? inode_permission+0x18/0x50 [<ffffffff9c1f4ae6>] ? fanotify_handle_event+0x136/0x390 [<ffffffff9c1980fe>] kmem_cache_alloc+0x1fe/0x260 [<ffffffff9c1f4ae6>] fanotify_handle_event+0x136/0x390 [<ffffffff9c1bb8fd>] ? path_openat+0xcd/0x6a0 [<ffffffff9c1f0e63>] send_to_group+0xd3/0x1c0 [<ffffffff9c1f0fdf>] ? fsnotify+0x8f/0x340 [<ffffffff9c1f1118>] fsnotify+0x1c8/0x340 [<ffffffff9c1a9b4f>] do_sys_open+0x19f/0x230 [<ffffffff9c1a9bfe>] SyS_open+0x1e/0x20 [<ffffffff9c723764>] tracesys+0xdd/0xe2
FIX fanotify_event_info: Restoring 0xffff880247e45bc8-0xffff880247e45bcb=0x6b
Quoted 31 lines · Dave Jones
Jan,
since yesterdays changes, on boot I see a flood of messages from slub debug during boot..
=============================================================================
BUGfanotify_event_info(Not tainted): Poison overwritten
-----------------------------------------------------------------------------Disabling lock debugging due to kernel taint
INFO: 0xffff880247e45bc8-0xffff880247e45bcb. First byte0x0instead of0x6b
INFO: Allocated infanotify_handle_event+0x136/0x390age=0 cpu=0 pid=293__slab_alloc+0x456/0x565 kmem_cache_alloc+0x1fe/0x260 fanotify_handle_event+0x136/0x390 send_to_group+0xd3/0x1c0 fsnotify+0x1c8/0x340 open_exec+0xe2/0x120 load_elf_binary+0x7b7/0x18e0 search_binary_handler+0x94/0x1b0 do_execve_common.isra.26+0x5d7/0x7d0 SyS_execve+0x36/0x50 stub_execve+0x69/0xa0 INFO: Freed in fanotify_free_event+0x2e/0x40 age=0 cpu=3 pid=290 __slab_free+0x4a/0x382 kmem_cache_free+0x1c9/0x210 fanotify_free_event+0x2e/0x40 fsnotify_destroy_event+0x21/0x30 fanotify_read+0x39e/0x5e0 vfs_read+0x9b/0x160 SyS_read+0x58/0xb0 tracesys+0xdd/0xe2
Quoted 1 line
INFO: Slab
0xffffea00091f9100objects=20 used=20 fp=0x (null) flags=0x20000000004080
Reverting 7053aee26a3548ebaba046ae2e52396ccf56ac6c makes this go away.
Dave
Quoted 2 lines · Dave Jones
BUG
fanotify_event_info(Not tainted): Poison overwritten
Looking at the poison data, it seems that is is the
u32 response;
field that has been overwritten (with all zero).
That doesn't really help me guess where the bug is, though. That code is crazy. For example, looking at one place where it is set, we have:
- process_access_response():
re->event->response = response;
wake_up(&group->fanotify_data.access_waitq);
kmem_cache_free(fanotify_response_event_cache, re);which looks buggy in *so* many ways. In particular, we're doing a kmem_cache_free() on "re", but what happened to "re->event" that we just used? There was no release of that anywhere. Wut?
So it seems that the lifetime of these "fanotify_event_info" structures is completely buggered. I don't even see any *attempt* to maintain reference counts or other lifetime info. People free the containers that point to them without doing anything at all about the fsnotify_event that contains the fanotify_event_info that they point to.
Jan - how is the lifetime of the fanotify_event_info tied to the lifetime of the fanotify_response_event structure that contains pointers into it? Because I don't see it.
And considering that it's the response field that gets overwritten, it really sounds like *that* is the exact issue at play here - there is some fanotify_response_event structure holding a pointer to the fanotify_event that is embedded into a fanotify_event_info that has been freed.
Jan?
Linus
Jan Kara
maintainerQuoted 32 lines · Linus Torvalds
Quoted 2 lines · Dave Jones
BUG
fanotify_event_info(Not tainted): Poison overwrittenLooking at the poison data, it seems that is is the
u32 response;field that has been overwritten (with all zero).
That doesn't really help me guess where the bug is, though. That code is crazy. For example, looking at one place where it is set, we have:
-
process_access_response():re->event->response = response; wake_up(&group->fanotify_data.access_waitq); kmem_cache_free(fanotify_response_event_cache, re);which looks buggy in *so* many ways. In particular, we're doing a
kmem_cache_free()on "re", but what happened to "re->event" that we just used? There was no release of that anywhere. Wut?So it seems that the lifetime of these "
fanotify_event_info" structures is completely buggered. I don't even see any *attempt* to maintain reference counts or other lifetime info. People free the containers that point to them without doing anything at all about thefsnotify_eventthat contains thefanotify_event_infothat they point to.
Quoted 3 lines
Jan - how is the lifetime of the
fanotify_event_infotied to the lifetime of thefanotify_response_eventstructure that contains pointers into it? Because I don't see it.
Yeah, I messed that up. They aren't tied in any way - well, in fact they
end up being tied but in a wrong way. fanotify_event_info lives from the moment event happens to the moment user reads the event. At that moment the fanotify_response_event gets created (for those special permission events), pointing to fanotify_event_info which will get freed just several lines further :-|
But refcounting seems like an overkill for this - there is exactly one fanotify_response_event structure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.
Quoted 5 lines
And considering that it's the response field that gets overwritten, it really sounds like *that* is the exact issue at play here - there is some
fanotify_response_eventstructure holding a pointer to thefanotify_eventthat is embedded into afanotify_event_infothat has been freed.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
fs/notify/fanotify/fanotify.c +3 -1
@@ -201,8 +201,10 @@ static int fanotify_handle_event(struct fsnotify_group *group, }#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS -if (fsn_event->mask &FAN_ALL_PERM_EVENTS ) +if (fsn_event->mask &FAN_ALL_PERM_EVENTS ) { ret =fanotify_get_response_from_access (group, event); +fsnotify_destroy_event (group, event); + }#endif return ret; }
fs/notify/fanotify/fanotify_user.c +2 -1
@@ -319,7 +319,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,if (IS_ERR (kevent))break ; ret =copy_event_to_user (group, kevent, buf); -fsnotify_destroy_event (group, kevent); +if (!(kevent->mask &FAN_ALL_PERM_EVENTS )) +fsnotify_destroy_event (group, kevent);if (ret <0 )break ; buf += ret;
Quoted 6 lines · Jan Kara
But refcounting seems like an overkill for this - there is exactly one
fanotify_response_eventstructure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.
Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?
Linus
Quoted 10 lines · Linus Torvalds
Quoted 6 lines · Jan Kara
But refcounting seems like an overkill for this - there is exactly one
fanotify_response_eventstructure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?
It actually seems worse. I see the tail end of what looks like a slab corruption trace, and then a total lockup. And of course none of this makes it over ttyUSB0 because it happens so early. Grr.
Dave
Quoted 8 lines · Linus Torvalds
Quoted 5 lines
But refcounting seems like an overkill for this - there is exactly one
fanotify_response_eventstructure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?
I reported the same slab corruption yesterday as well here:
https://lkml.org/lkml/2014/1/22/173
With the patch applied, I am still seeing the slab corruption, preceeded by GPF (which is not there without the patch) in lockref_put_or_lock(&dentry->d_lockref) in dput():
general protection fault: 0000 [#1] SMP Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys CPU: 1 PID: 275 Comm: systemd-readahe Not tainted 3.13.0-03478-g670d0ac #1 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 task: ffff880037c09150 ti: ffff88007359e000 task.ti: ffff88007359e000 RIP: 0010:[<ffffffff810a51e7>] [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 RSP: 0018:ffff88007359fc78 EFLAGS: 00010286 RAX: ffff880037c09150 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000 tpm_tis 00:09: 1.2 TPM (device-id 0x1020, rev-id 6) tpm_tis 00:09: Intel iTPM workaround enabled RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb RBP: ffff88007359fc98 R08: 0000000000000002 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb R13: ffff880037310690 R14: 0000000000000020 R15: ffff880036dbfc10 FS: 00007fa953e4a700(0000) GS:ffff88007c280000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fff76a10258 CR3: 000000003659c000 CR4: 00000000000007e0 Stack: 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880037310690 ffff88007359fcb8 ffffffff8159c59c ffffffff812fe101 6b6b6b6b6b6b6beb ffff88007359fcd8 ffffffff812fe101 ffff88007359fd08 6b6b6b6b6b6b6b6b Call Trace: [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50 [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40 [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40 [<ffffffff811b1442>] dput+0x22/0x130 [<ffffffff811a3d45>] path_put+0x15/0x30 [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40 [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390 [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [<ffffffff81198859>] SyS_open+0x19/0x20 [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff RIP [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 RSP <ffff88007359fc78> ---[ end trace 7a918209ee213d28 ]--- BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20 in_atomic(): 1, irqs_disabled(): 0, pid: 275, name: systemd-readahe INFO: lockdep is turned off. CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G D 3.13.0-03478-g670d0ac #1 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 ffff880037c09150 ffff88007359fa78 ffffffff8159702b ffff88007359fa98 ffffffff8107f621 ffffffff81a3d000 ffff880037b98d90 ffff88007359fab8 ffffffff8159b4ff ffff880037c09150 ffff880037c09150 ffff88007359fae8 Call Trace: [<ffffffff8159702b>] dump_stack+0x72/0x87 [<ffffffff8107f621>] __might_sleep+0xe1/0x100 [<ffffffff8159b4ff>] down_read+0x1f/0x60 [<ffffffff810627ff>] exit_signals+0x1f/0x140 [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20 [<ffffffff81052844>] do_exit+0xb4/0x4b0 [<ffffffff8159e23c>] oops_end+0xdc/0xe0 [<ffffffff81005f86>] die+0x56/0x90 [<ffffffff8159dea2>] do_general_protection+0x162/0x170 [<ffffffff8159d40c>] ? restore_args+0x30/0x30 [<ffffffff8159d592>] general_protection+0x22/0x30 [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50 [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40 [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40 [<ffffffff811b1442>] dput+0x22/0x130 [<ffffffff811a3d45>] path_put+0x15/0x30 [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40 [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390 [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [<ffffffff81198859>] SyS_open+0x19/0x20 [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b note: systemd-readahe[275] exited with preempt_count 1 BUG: scheduling while atomic: systemd-readahe/275/0x00000002 INFO: lockdep is turned off. Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G D 3.13.0-03478-g670d0ac #1 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 ffff88007c293a00 ffff88007359f6f8 ffffffff8159702b ffff88007359f718 ffffffff810810f1 ffff88007c293a00 0000000000000001 ffff88007359f848 ffffffff8159789c ffff88007359f758 ffff88007359f768 ffff88007359e010 Call Trace: [<ffffffff8159702b>] dump_stack+0x72/0x87 [<ffffffff810810f1>] __schedule_bug+0x61/0x80 [<ffffffff8159789c>] __schedule+0xbc/0x7c0 [<ffffffff8105defc>] ? mod_timer+0x14c/0x1f0 [<ffffffff815980e4>] schedule+0x24/0x70 [<ffffffff81597205>] schedule_timeout+0x1c5/0x210 [<ffffffff8159914f>] ? wait_for_completion+0xcf/0x120 [<ffffffff810a0d8d>] ? trace_hardirqs_on+0xd/0x10 [<ffffffff81599157>] wait_for_completion+0xd7/0x120 [<ffffffff81083330>] ? try_to_wake_up+0x250/0x250 [<ffffffff810b93bf>] ? srcu_reschedule+0x4f/0xf0 [<ffffffff810b965c>] __synchronize_srcu+0xec/0x130 [<ffffffff810b96e0>] ? srcu_barrier+0x10/0x10 [<ffffffff810b96c8>] synchronize_srcu+0x18/0x20 [<ffffffff811ddbdd>] fsnotify_destroy_group+0x1d/0x40 [<ffffffff811dfdf1>] inotify_release+0x21/0x50 [<ffffffff8119b2dd>] __fput+0xbd/0x2b0 [<ffffffff8119b569>] ____fput+0x9/0x10 [<ffffffff81070f41>] task_work_run+0xb1/0xe0 [<ffffffff81052979>] do_exit+0x1e9/0x4b0 [<ffffffff8159e23c>] oops_end+0xdc/0xe0 [<ffffffff81005f86>] die+0x56/0x90 [<ffffffff8159dea2>] do_general_protection+0x162/0x170 [<ffffffff8159d40c>] ? restore_args+0x30/0x30 [<ffffffff8159d592>] general_protection+0x22/0x30 [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50 [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40 [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40 [<ffffffff811b1442>] dput+0x22/0x130 [<ffffffff811a3d45>] path_put+0x15/0x30 [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40 [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390 [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [<ffffffff81198859>] SyS_open+0x19/0x20 [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b ACPI: AC Adapter [AC] (on-line) Slab corruption (Tainted: G D W ): fanotify_event_info start=ffff880037310690, len=64 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40) 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk. Prev obj: start=ffff880037310638, len=64 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Next obj: start=ffff8800373106e8, len=64 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Signature · 2 lines
Jiri Kosina SUSE Labs
Jan Kara
maintainerQuoted 9 lines · Dave Jones
Quoted 8 lines · Linus Torvalds
Quoted 6 lines · Jan Kara
But refcounting seems like an overkill for this - there is exactly one
fanotify_response_eventstructure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.
Quoted 6 lines
Quoted 2 lines
Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?
It actually seems worse. I see the tail end of what looks like a slab corruption trace, and then a total lockup. And of course none of this makes it over ttyUSB0 because it happens so early. Grr.
Drat. Since this seems reasonably reproducible, I'll try to reproduce
and debug this (it seems systemd is using fanotify in a way that triggers this). Thanks for testing.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Jan Kara
maintainerQuoted 11 lines · Jiri Kosina
Quoted 8 lines · Linus Torvalds
Quoted 5 lines
But refcounting seems like an overkill for this - there is exactly one
fanotify_response_eventstructure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?
Quoted 7 lines
I reported the same slab corruption yesterday as well here:
https://lkml.org/lkml/2014/1/22/173With the patch applied, I am still seeing the slab corruption, preceeded by GPF (which is not there without the patch) in
lockref_put_or_lock(&dentry->d_lockref)indput():
Hmm, OK. Can you please send me your .config? I'll try to reproduce this
myself.
Honza
Quoted 151 lines
general protection fault: 0000 [#1] SMP Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys CPU: 1 PID: 275 Comm: systemd-readahe Not tainted 3.13.0-03478-g670d0ac #1 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 task: ffff880037c09150 ti: ffff88007359e000 task.ti: ffff88007359e000 RIP: 0010:[<ffffffff810a51e7>] [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 RSP: 0018:ffff88007359fc78 EFLAGS: 00010286 RAX: ffff880037c09150 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000 tpm_tis 00:09: 1.2 TPM (device-id 0x1020, rev-id 6) tpm_tis 00:09: Intel iTPM workaround enabled RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb RBP: ffff88007359fc98 R08: 0000000000000002 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb R13: ffff880037310690 R14: 0000000000000020 R15: ffff880036dbfc10 FS: 00007fa953e4a700(0000) GS:ffff88007c280000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fff76a10258 CR3: 000000003659c000 CR4: 00000000000007e0 Stack: 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880037310690 ffff88007359fcb8 ffffffff8159c59c ffffffff812fe101 6b6b6b6b6b6b6beb ffff88007359fcd8 ffffffff812fe101 ffff88007359fd08 6b6b6b6b6b6b6b6b Call Trace: [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50 [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40 [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40 [<ffffffff811b1442>] dput+0x22/0x130 [<ffffffff811a3d45>] path_put+0x15/0x30 [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40 [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390 [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [<ffffffff81198859>] SyS_open+0x19/0x20 [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff RIP [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 RSP <ffff88007359fc78> ---[ end trace 7a918209ee213d28 ]--- BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20 in_atomic(): 1, irqs_disabled(): 0, pid: 275, name: systemd-readahe INFO: lockdep is turned off. CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G D 3.13.0-03478-g670d0ac #1 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 ffff880037c09150 ffff88007359fa78 ffffffff8159702b ffff88007359fa98 ffffffff8107f621 ffffffff81a3d000 ffff880037b98d90 ffff88007359fab8 ffffffff8159b4ff ffff880037c09150 ffff880037c09150 ffff88007359fae8 Call Trace: [<ffffffff8159702b>] dump_stack+0x72/0x87 [<ffffffff8107f621>] __might_sleep+0xe1/0x100 [<ffffffff8159b4ff>] down_read+0x1f/0x60 [<ffffffff810627ff>] exit_signals+0x1f/0x140 [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20 [<ffffffff81052844>] do_exit+0xb4/0x4b0 [<ffffffff8159e23c>] oops_end+0xdc/0xe0 [<ffffffff81005f86>] die+0x56/0x90 [<ffffffff8159dea2>] do_general_protection+0x162/0x170 [<ffffffff8159d40c>] ? restore_args+0x30/0x30 [<ffffffff8159d592>] general_protection+0x22/0x30 [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50 [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40 [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40 [<ffffffff811b1442>] dput+0x22/0x130 [<ffffffff811a3d45>] path_put+0x15/0x30 [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40 [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390 [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [<ffffffff81198859>] SyS_open+0x19/0x20 [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b note: systemd-readahe[275] exited with preempt_count 1 BUG: scheduling while atomic: systemd-readahe/275/0x00000002 INFO: lockdep is turned off. Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G D 3.13.0-03478-g670d0ac #1 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 ffff88007c293a00 ffff88007359f6f8 ffffffff8159702b ffff88007359f718 ffffffff810810f1 ffff88007c293a00 0000000000000001 ffff88007359f848 ffffffff8159789c ffff88007359f758 ffff88007359f768 ffff88007359e010 Call Trace: [<ffffffff8159702b>] dump_stack+0x72/0x87 [<ffffffff810810f1>] __schedule_bug+0x61/0x80 [<ffffffff8159789c>] __schedule+0xbc/0x7c0 [<ffffffff8105defc>] ? mod_timer+0x14c/0x1f0 [<ffffffff815980e4>] schedule+0x24/0x70 [<ffffffff81597205>] schedule_timeout+0x1c5/0x210 [<ffffffff8159914f>] ? wait_for_completion+0xcf/0x120 [<ffffffff810a0d8d>] ? trace_hardirqs_on+0xd/0x10 [<ffffffff81599157>] wait_for_completion+0xd7/0x120 [<ffffffff81083330>] ? try_to_wake_up+0x250/0x250 [<ffffffff810b93bf>] ? srcu_reschedule+0x4f/0xf0 [<ffffffff810b965c>] __synchronize_srcu+0xec/0x130 [<ffffffff810b96e0>] ? srcu_barrier+0x10/0x10 [<ffffffff810b96c8>] synchronize_srcu+0x18/0x20 [<ffffffff811ddbdd>] fsnotify_destroy_group+0x1d/0x40 [<ffffffff811dfdf1>] inotify_release+0x21/0x50 [<ffffffff8119b2dd>] __fput+0xbd/0x2b0 [<ffffffff8119b569>] ____fput+0x9/0x10 [<ffffffff81070f41>] task_work_run+0xb1/0xe0 [<ffffffff81052979>] do_exit+0x1e9/0x4b0 [<ffffffff8159e23c>] oops_end+0xdc/0xe0 [<ffffffff81005f86>] die+0x56/0x90 [<ffffffff8159dea2>] do_general_protection+0x162/0x170 [<ffffffff8159d40c>] ? restore_args+0x30/0x30 [<ffffffff8159d592>] general_protection+0x22/0x30 [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50 [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40 [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40 [<ffffffff811b1442>] dput+0x22/0x130 [<ffffffff811a3d45>] path_put+0x15/0x30 [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40 [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390 [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [<ffffffff81198859>] SyS_open+0x19/0x20 [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b ACPI: AC Adapter [AC] (on-line) Slab corruption (Tainted: G D W ): fanotify_event_info start=ffff880037310690, len=64 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40) 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk. Prev obj: start=ffff880037310638, len=64 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Next obj: start=ffff8800373106e8, len=64 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkkSignature · 2 lines
Jiri Kosina SUSE Labs
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 11 lines · Jan Kara
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines
But refcounting seems like an overkill for this - there is exactly one
fanotify_response_eventstructure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?
I reported the same slab corruption yesterday as well here:
Quoted 7 lines
Quoted 5 lines
https://lkml.org/lkml/2014/1/22/173With the patch applied, I am still seeing the slab corruption, preceeded by GPF (which is not there without the patch) in
lockref_put_or_lock(&dentry->d_lockref)indput():Hmm, OK. Can you please send me your .config? I'll try to reproduce this
myself.
Attached.
The userspace is systemd-based.
Signature · 2 lines
Jiri Kosina SUSE Labs
Jan Kara
maintainerQuoted 18 lines · Jiri Kosina
Quoted 16 lines · Jan Kara
Quoted 16 lines
Quoted 8 lines
Quoted 5 lines
But refcounting seems like an overkill for this - there is exactly one
fanotify_response_eventstructure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?
I reported the same slab corruption yesterday as well here:
https://lkml.org/lkml/2014/1/22/173With the patch applied, I am still seeing the slab corruption, preceeded by GPF (which is not there without the patch) in
lockref_put_or_lock(&dentry->d_lockref)indput():
Quoted 6 lines
Quoted 2 lines
Hmm, OK. Can you please send me your .config? I'll try to reproduce this
myself.Attached.
The userspace is systemd-based.
Strange. I've installed systemd system (openSUSE 13.1) and it boots with
the latest Linus' kernel just fine (and I have at least FANOTIFY and SLAB debugging set the same way as you). But it was only a KVM guest. I'll try tomorrow with a physical machine I guess.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 4 lines · Jan Kara
Strange. I've installed systemd system (openSUSE 13.1) and it boots
with the latest Linus' kernel just fine (and I have at least FANOTIFY and SLAB debugging set the same way as you). But it was only a KVM guest. I'll try tomorrow with a physical machine I guess.
FWIW the system I am reliably able to reproduce this on is opensuse 12.3 with this systemd version:
Version : 195
Release : 13.18.1
Signature · 2 lines
Jiri Kosina SUSE Labs
Jan Kara
maintainerQuoted 7 lines · Jiri Kosina
Quoted 4 lines · Jan Kara
Strange. I've installed systemd system (openSUSE 13.1) and it boots
with the latest Linus' kernel just fine (and I have at least FANOTIFY and SLAB debugging set the same way as you). But it was only a KVM guest. I'll try tomorrow with a physical machine I guess.
Quoted 5 lines
FWIW the system I am reliably able to reproduce this on is opensuse 12.3 with this systemd version:
Version : 195
Release : 13.18.1
Hum, still no luck with reproduction (either on physical machine or with
KVM). Anyway, I've looked at the code again and the previous patch had a stupid bug (passing different pointer to fsnotify_destroy_event() than we should have), plus also the merging function in fanotify was too aggressive. Can you try the attached patch? It boots for me but that means nothing since I cannot reproduce the issue... Thanks!
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
fs/notify/fanotify/fanotify.c +6 -3
@@ -18,7 +18,7 @@ static bool should_merge(struct fsnotify_event *old_fsn,#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS /* dont merge two permission events */ -if ((old_fsn->mask &FAN_ALL_PERM_EVENTS ) && +if ((old_fsn->mask &FAN_ALL_PERM_EVENTS ) || (new_fsn->mask &FAN_ALL_PERM_EVENTS ))return false;#endif
@@ -201,8 +201,10 @@ static int fanotify_handle_event(struct fsnotify_group *group, }#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS -if (fsn_event->mask &FAN_ALL_PERM_EVENTS ) +if (fsn_event->mask &FAN_ALL_PERM_EVENTS ) { ret =fanotify_get_response_from_access (group, event); +fsnotify_destroy_event (group, fsn_event); + }#endif return ret; }
@@ -221,7 +223,8 @@ static void fanotify_free_event(struct fsnotify_event *fsn_event)struct fanotify_event_info *event; event =FANOTIFY_E (fsn_event); -path_put (&event->path); +if (event->path.mnt) +path_put (&event->path);put_pid (event->tgid);kmem_cache_free (fanotify_event_cachep, event); }
fs/notify/fanotify/fanotify_user.c +2 -1
@@ -319,7 +319,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,if (IS_ERR (kevent))break ; ret =copy_event_to_user (group, kevent, buf); -fsnotify_destroy_event (group, kevent); +if (!(kevent->mask &FAN_ALL_PERM_EVENTS )) +fsnotify_destroy_event (group, kevent);if (ret <0 )break ; buf += ret;
Quoted 11 lines · Jan Kara
Quoted 10 lines · Jiri Kosina
Quoted 4 lines · Jan Kara
Strange. I've installed systemd system (openSUSE 13.1) and it boots
with the latest Linus' kernel just fine (and I have at least FANOTIFY and SLAB debugging set the same way as you). But it was only a KVM guest. I'll try tomorrow with a physical machine I guess.FWIW the system I am reliably able to reproduce this on is opensuse 12.3 with this systemd version:
Quoted 8 lines
Quoted 2 lines
Version : 195
Release : 13.18.1Hum, still no luck with reproduction (either on physical machine or with
KVM). Anyway, I've looked at the code again and the previous patch had a stupid bug (passing different pointer tofsnotify_destroy_event()than we should have), plus also the merging function in fanotify was too aggressive. Can you try the attached patch? It boots for me but that means nothing since I cannot reproduce the issue... Thanks!
still not good I'm afraid. I still see corruption very early on in boot and now it panics and locks up too.
Again, this happens so early that I can't grab it over usb-serial. I stuck an mdelay(10000) in the slub corruption detector, and managed to grab a photo of the first trace.
Trace:
? preempt_schedulelock_acquire
? lockref_put_or_lock_raw_spin_lock
? lockref_put_or_lock
dputpath_putfanotify_free_eventfsnotify_destroy_eventfanotify_handle_event
? mntput
? path_openat
? handle_mm_faultsend_to_group
? fsnotify
fsnotifydo_sys_opensys_open
RIP: lock_acquire
2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction
R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free.
I also notice you mention SLAB above, but I've been using SLUB. I don't know if the choice of allocator makes a difference in reproducability.
It's also worth noting that I have lockdep enabled, which may be perturbing things to some degree.
Dave
Jan Kara
maintainerQuoted 21 lines · Dave Jones
Quoted 11 lines · Jan Kara
Quoted 10 lines · Jiri Kosina
Quoted 4 lines · Jan Kara
Strange. I've installed systemd system (openSUSE 13.1) and it boots
with the latest Linus' kernel just fine (and I have at least FANOTIFY and SLAB debugging set the same way as you). But it was only a KVM guest. I'll try tomorrow with a physical machine I guess.FWIW the system I am reliably able to reproduce this on is opensuse 12.3 with this systemd version:
Quoted 8 lines
Quoted 2 lines
Version : 195
Release : 13.18.1Hum, still no luck with reproduction (either on physical machine or with
KVM). Anyway, I've looked at the code again and the previous patch had a stupid bug (passing different pointer tofsnotify_destroy_event()than we should have), plus also the merging function in fanotify was too aggressive. Can you try the attached patch? It boots for me but that means nothing since I cannot reproduce the issue... Thanks!
Quoted 2 lines
still not good I'm afraid. I still see corruption very early on in boot and now it panics and locks up too.
Ew, thanks for testing.
Quoted 23 lines
Again, this happens so early that I can't grab it over usb-serial. I stuck an mdelay(10000) in the slub corruption detector, and managed to grab a photo of the first trace.
Trace:
?preempt_schedulelock_acquire
?lockref_put_or_lock_raw_spin_lock
?lockref_put_or_lock
dputpath_putfanotify_free_eventfsnotify_destroy_eventfanotify_handle_event
? mntput
?path_openat
?handle_mm_faultsend_to_group
? fsnotify
fsnotifydo_sys_opensys_open
Quoted 5 lines
RIP: lock_acquire2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction
R14 is
0x6b6b6b6b6b6b6c03, which looks like a use-after-free.
Yup. But I'm somewhat puzzled by the trace. We crash when callingfsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code has been called from do_sys_open() so the event was a 'FS_OPEN' which fails the fsn_event->mask & FAN_ALL_PERM_EVENTS test.
Slapping my forehead, that's a really stupid bug. The event fsnotify_add_notify_event() returns may be freed by the time we return because we already dropped the notification mutex. And then fsn_event->mask & FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches with the poison pattern 0x6b6b6b6b. So yet another hacked up version of fanotify fix is attached. And I have to seriously think about use counts for fanotify version of that struct.
Quoted 2 lines
I also notice you mention SLAB above, but I've been using SLUB. I don't know if the choice of allocator makes a difference in reproducability.
Jiri Kosina has SLAB so SLAB/SLUB apparently doesn't matter.
Quoted 2 lines
It's also worth noting that I have lockdep enabled, which may be perturbing things to some degree.
And I've compiled my kernel with lockdep as well since Jiri has it in his
config. But no luck.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
fs/notify/fanotify/fanotify.c +6 -3
@@ -18,7 +18,7 @@ static bool should_merge(struct fsnotify_event *old_fsn,#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS /* dont merge two permission events */ -if ((old_fsn->mask &FAN_ALL_PERM_EVENTS ) && +if ((old_fsn->mask &FAN_ALL_PERM_EVENTS ) || (new_fsn->mask &FAN_ALL_PERM_EVENTS ))return false;#endif
@@ -201,8 +201,10 @@ static int fanotify_handle_event(struct fsnotify_group *group, }#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS -if (fsn_event->mask &FAN_ALL_PERM_EVENTS ) +if (mask &FAN_ALL_PERM_EVENTS ) { ret =fanotify_get_response_from_access (group, event); +fsnotify_destroy_event (group, fsn_event); + }#endif return ret; }
@@ -221,7 +223,8 @@ static void fanotify_free_event(struct fsnotify_event *fsn_event)struct fanotify_event_info *event; event =FANOTIFY_E (fsn_event); -path_put (&event->path); +if (event->path.mnt) +path_put (&event->path);put_pid (event->tgid);kmem_cache_free (fanotify_event_cachep, event); }
fs/notify/fanotify/fanotify_user.c +2 -1
@@ -319,7 +319,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,if (IS_ERR (kevent))break ; ret =copy_event_to_user (group, kevent, buf); -fsnotify_destroy_event (group, kevent); +if (!(kevent->mask &FAN_ALL_PERM_EVENTS )) +fsnotify_destroy_event (group, kevent);if (ret <0 )break ; buf += ret;
Quoted 6 lines · Jan Kara
Hum, still no luck with reproduction (either on physical machine or with
KVM). Anyway, I've looked at the code again and the previous patch had a stupid bug (passing different pointer tofsnotify_destroy_event()than we should have), plus also the merging function in fanotify was too aggressive. Can you try the attached patch? It boots for me but that means nothing since I cannot reproduce the issue... Thanks!
I am attaching dmesg with the patch applied; I've removed irrelevant parts.
There is a GPF, followed by scheduling in atomic context, followed by slab corruption, followed by another scheduling while atomic and leak of preempt_count.
[ 0.000000] Initializing cgroup subsys cpuset [ 5.081301] systemd-udevd[332]: starting version 195 [ 5.083694] random: nonblocking pool is initialized [ 5.299400] systemd-journald[307]: Received SIGUSR1 [ 5.625120] general protection fault: 0000 [#1] SMP [ 5.626464] Modules linked in: acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm usbcore i2c_algo_bit usb_common button video edd fan processor ata_generic thermal thermal_sys [ 5.628008] CPU: 0 PID: 302 Comm: systemd-readahe Not tainted 3.13.0-03478-gae75a37 #1 [ 5.628008] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 [ 5.628008] task: ffff8800364b04d0 ti: ffff8800734b8000 task.ti: ffff8800734b8000 [ 5.628008] RIP: 0010:[<ffffffff810a51e7>] [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 [ 5.628008] RSP: 0018:ffff8800734b9c68 EFLAGS: 00010282 [ 5.628008] RAX: ffff8800364b04d0 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000 [ 5.628008] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb [ 5.628008] RBP: ffff8800734b9c88 R08: 0000000000000002 R09: 0000000000000000 [ 5.628008] R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb [ 5.628008] R13: ffff880035d0db28 R14: 0000000000000020 R15: ffff880037fffd50 [ 5.628008] FS: 00007fd2f4728700(0000) GS:ffff88007c200000(0000) knlGS:0000000000000000 [ 5.628008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 5.628008] CR2: 00007fa7dc98c000 CR3: 0000000037f3b000 CR4: 00000000000007f0 [ 5.628008] Stack: [ 5.628008] 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880035d0db28 [ 5.628008] ffff8800734b9ca8 ffffffff8159c5ac ffffffff812fe111 6b6b6b6b6b6b6beb [ 5.628008] ffff8800734b9cc8 ffffffff812fe111 ffff8800734b9cf8 6b6b6b6b6b6b6b6b [ 5.628008] Call Trace: [ 5.628008] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50 [ 5.628008] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40 [ 5.628008] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40 [ 5.628008] [<ffffffff811b1442>] dput+0x22/0x130 [ 5.628008] [<ffffffff811a3d45>] path_put+0x15/0x30 [ 5.628008] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40 [ 5.628008] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [ 5.628008] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390 [ 5.628008] [<ffffffff811a3d4d>] ? path_put+0x1d/0x30 [ 5.628008] [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [ 5.628008] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [ 5.628008] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [ 5.628008] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [ 5.628008] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [ 5.628008] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 5.628008] [<ffffffff81198859>] SyS_open+0x19/0x20 [ 5.628008] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b [ 5.628008] Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff [ 5.628008] RIP [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 [ 5.628008] RSP <ffff8800734b9c68> [ 5.683491] ---[ end trace 5b4e9ae52ab9b0f6 ]--- [ 5.685076] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20 [ 5.686578] in_atomic(): 1, irqs_disabled(): 0, pid: 302, name: systemd-readahe [ 5.688058] INFO: lockdep is turned off. [ 5.689503] CPU: 0 PID: 302 Comm: systemd-readahe Tainted: G D 3.13.0-03478-gae75a37 #1 [ 5.690966] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 [ 5.692464] ffff8800364b04d0 ffff8800734b9a68 ffffffff8159703b ffff8800734b9a88 [ 5.694027] ffffffff8107f621 ffffffff81a3d000 ffff88003641c750 ffff8800734b9aa8 [ 5.695555] ffffffff8159b50f ffff8800364b04d0 ffff8800364b04d0 ffff8800734b9ad8 [ 5.697111] Call Trace: [ 5.698595] [<ffffffff8159703b>] dump_stack+0x72/0x87 [ 5.700108] [<ffffffff8107f621>] __might_sleep+0xe1/0x100 [ 5.701623] [<ffffffff8159b50f>] down_read+0x1f/0x60 [ 5.703133] [<ffffffff810627ff>] exit_signals+0x1f/0x140 [ 5.704661] [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20 [ 5.706105] [<ffffffff81052844>] do_exit+0xb4/0x4b0 [ 5.707470] [<ffffffff8159e23c>] oops_end+0xdc/0xe0 [ 5.708845] [<ffffffff81005f86>] die+0x56/0x90 [ 5.710229] [<ffffffff8159dea2>] do_general_protection+0x162/0x170 [ 5.711545] [<ffffffff8159d40c>] ? restore_args+0x30/0x30 [ 5.712883] [<ffffffff8159d592>] general_protection+0x22/0x30 [ 5.714213] [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 [ 5.715491] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50 [ 5.716781] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40 [ 5.718113] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40 [ 5.719390] [<ffffffff811b1442>] dput+0x22/0x130 [ 5.720681] [<ffffffff811a3d45>] path_put+0x15/0x30 [ 5.721984] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40 [ 5.723243] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [ 5.724507] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390 [ 5.725779] [<ffffffff811a3d4d>] ? path_put+0x1d/0x30 [ 5.727014] [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [ 5.728258] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [ 5.729515] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [ 5.730734] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [ 5.731945] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [ 5.733169] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 5.734456] [<ffffffff81198859>] SyS_open+0x19/0x20 [ 5.735681] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b [ 5.736933] note: systemd-readahe[302] exited with preempt_count 1 [ 5.738378] BUG: scheduling while atomic: systemd-readahe/302/0x00000002 [ 5.739652] INFO: lockdep is turned off. [ 5.740925] Modules linked in: acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm usbcore i2c_algo_bit usb_common button video edd fan processor ata_generic thermal thermal_sys [ 5.743781] CPU: 0 PID: 302 Comm: systemd-readahe Tainted: G D 3.13.0-03478-gae75a37 #1 [ 5.745231] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 [ 5.746708] ffff88007c213a00 ffff8800734b96e8 ffffffff8159703b ffff8800734b9708 [ 5.748190] ffffffff810810f1 ffff88007c213a00 0000000000000000 ffff8800734b9838 [ 5.749690] ffffffff815978ac ffff8800734b9748 ffff8800734b9758 ffff8800734b8010 [ 5.751161] Call Trace: [ 5.752635] [<ffffffff8159703b>] dump_stack+0x72/0x87 [ 5.754126] [<ffffffff810810f1>] __schedule_bug+0x61/0x80 [ 5.755551] [<ffffffff815978ac>] __schedule+0xbc/0x7c0 [ 5.756925] [<ffffffff8105defc>] ? mod_timer+0x14c/0x1f0 [ 5.758302] [<ffffffff815980f4>] schedule+0x24/0x70 [ 5.759632] [<ffffffff81597215>] schedule_timeout+0x1c5/0x210 [ 5.760982] [<ffffffff8159915f>] ? wait_for_completion+0xcf/0x120 [ 5.762327] [<ffffffff810a0d8d>] ? trace_hardirqs_on+0xd/0x10 [ 5.763630] [<ffffffff81599167>] wait_for_completion+0xd7/0x120 [ 5.764935] [<ffffffff81083330>] ? try_to_wake_up+0x250/0x250 [ 5.766261] [<ffffffff810b93bf>] ? srcu_reschedule+0x4f/0xf0 [ 5.767521] [<ffffffff810b965c>] __synchronize_srcu+0xec/0x130 [ 5.768775] [<ffffffff810b96e0>] ? srcu_barrier+0x10/0x10 [ 5.770059] [<ffffffff810b96c8>] synchronize_srcu+0x18/0x20 [ 5.771302] [<ffffffff811ddbdd>] fsnotify_destroy_group+0x1d/0x40 [ 5.772550] [<ffffffff811dfdf1>] inotify_release+0x21/0x50 [ 5.773814] [<ffffffff8119b2dd>] __fput+0xbd/0x2b0 [ 5.775055] [<ffffffff8119b569>] ____fput+0x9/0x10 [ 5.776311] [<ffffffff81070f41>] task_work_run+0xb1/0xe0 [ 5.777577] [<ffffffff81052979>] do_exit+0x1e9/0x4b0 [ 5.778803] [<ffffffff8159e23c>] oops_end+0xdc/0xe0 [ 5.780027] [<ffffffff81005f86>] die+0x56/0x90 [ 5.781264] [<ffffffff8159dea2>] do_general_protection+0x162/0x170 [ 5.782472] [<ffffffff8159d40c>] ? restore_args+0x30/0x30 [ 5.783687] [<ffffffff8159d592>] general_protection+0x22/0x30 [ 5.784917] [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 [ 5.786160] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50 [ 5.787367] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40 [ 5.788606] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40 [ 5.789848] [<ffffffff811b1442>] dput+0x22/0x130 [ 5.791073] [<ffffffff811a3d45>] path_put+0x15/0x30 [ 5.792324] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40 [ 5.793589] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [ 5.794809] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390 [ 5.796040] [<ffffffff811a3d4d>] ? path_put+0x1d/0x30 [ 5.797283] [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [ 5.798505] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [ 5.799732] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [ 5.800970] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [ 5.802219] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [ 5.803461] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 5.804729] [<ffffffff81198859>] SyS_open+0x19/0x20 [ 5.805970] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b [ ... snip ... ] [ 5.968718] Slab corruption (Tainted: G D W ): fanotify_event_info start=ffff880035d2a798, len=64 [ 5.968923] hub 7-0:1.0: USB hub found [ 5.971406] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. [ 5.972756] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40) [ 5.974098] 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk. [ 5.974837] hub 7-0:1.0: 6 ports detected [ 5.976799] Prev obj: start=ffff880035d2a740, len=64 [ 5.978189] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. [ 5.979526] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40) [ 5.980910] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 5.982307] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 5.983691] Next obj: start=ffff880035d2a7f0, len=64 [ 5.985070] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. [ 5.986455] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40) [ 5.987835] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 5.989248] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ ... snip ... ] [ 7.044083] usb 2-1: new full-speed USB device number 4 using uhci_hcd [ 7.131735] general protection fault: 0000 [#2] SMP [ 7.131842] Slab corruption (Tainted: G D W ): fanotify_event_info start=ffff880035da5320, len=64 [ 7.131844] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. [ 7.131850] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40) [ 7.131853] 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk. [ 7.131854] Prev obj: start=ffff880035da52c8, len=64 [ 7.131855] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. [ 7.131857] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40) [ 7.131859] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 7.131861] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 7.131862] Next obj: start=ffff880035da5378, len=64 [ 7.131863] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b. [ 7.131864] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40) [ 7.131866] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 7.131868] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk [ 7.135045] Modules linked in: cpufreq_conservative cpufreq_userspace snd_hda_codec_conexant cpufreq_powersave snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hwdep snd_pcm thinkpad_acpi kvm_intel snd_seq iTCO_wdt iTCO_vendor_support kvm iwldvm mac80211 snd_timer snd_seq_device btusb bluetooth iwlwifi sg cfg80211 e1000e snd ptp pcspkr lpc_ich i2c_i801 mfd_core rfkill pps_core ehci_pci wmi soundcore battery ac tpm_tis tpm acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm usbcore i2c_algo_bit usb_common button video edd fan processor ata_generic thermal thermal_sys [ 7.135045] CPU: 1 PID: 757 Comm: grep Tainted: G D W 3.13.0-03478-gae75a37 #1 [ 7.135045] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 [ 7.135045] task: ffff8800362fddd0 ti: ffff880036d42000 task.ti: ffff880036d42000 [ 7.135045] RIP: 0010:[<ffffffff810a51e7>] [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 [ 7.135045] RSP: 0018:ffff880036d43c68 EFLAGS: 00010282 [ 7.135045] RAX: ffff8800362fddd0 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000 [ 7.135045] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb [ 7.135045] RBP: ffff880036d43c88 R08: 0000000000000002 R09: 0000000000000000 [ 7.135045] R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb [ 7.135045] R13: ffff880035d0db28 R14: 0000000000000020 R15: ffff880037900310 [ 7.135045] FS: 0000000000000000(0000) GS:ffff88007c280000(0000) knlGS:0000000000000000 [ 7.135045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 7.135045] CR2: 00007fff296c9f54 CR3: 0000000079067000 CR4: 00000000000007e0 [ 7.135045] Stack: [ 7.135045] 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880035d0db28 [ 7.135045] ffff880036d43ca8 ffffffff8159c5ac ffffffff812fe111 6b6b6b6b6b6b6beb [ 7.135045] ffff880036d43cc8 ffffffff812fe111 ffff880036d43cf8 6b6b6b6b6b6b6b6b [ 7.135045] Call Trace: [ 7.135045] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50 [ 7.135045] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40 [ 7.135045] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40 [ 7.135045] [<ffffffff811b1442>] dput+0x22/0x130 [ 7.135045] [<ffffffff811a3d45>] path_put+0x15/0x30 [ 7.135045] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40 [ 7.135045] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [ 7.135045] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390 [ 7.135045] [<ffffffff815a0c84>] ? __do_page_fault+0x2c4/0x480 [ 7.135045] [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [ 7.135045] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [ 7.135045] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [ 7.135045] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [ 7.135045] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [ 7.135045] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 7.135045] [<ffffffff81198859>] SyS_open+0x19/0x20 [ 7.135045] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b [ 7.135045] Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff [ 7.135045] RIP [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160 [ 7.135045] RSP <ffff880036d43c68> [ 7.212496] ---[ end trace 5b4e9ae52ab9b0f7 ]--- [ 7.214048] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20 [ 7.214049] in_atomic(): 1, irqs_disabled(): 0, pid: 757, name: grep [ 7.214050] INFO: lockdep is turned off. [ 7.214051] CPU: 1 PID: 757 Comm: grep Tainted: G D W 3.13.0-03478-gae75a37 #1 [ 7.214052] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008 [ 7.214055] ffff8800362fddd0 ffff880036d43a68 ffffffff8159703b ffff880036d43a88 [ 7.214057] ffffffff8107f621 ffffffff81a3d000 ffff880037946cd0 ffff880036d43aa8 [ 7.214059] ffffffff8159b50f ffff8800362fddd0 ffff8800362fddd0 ffff880036d43ad8 [ 7.214060] Call Trace: [ 7.214071] [<ffffffff8159703b>] dump_stack+0x72/0x87 [ 7.214074] [<ffffffff8107f621>] __might_sleep+0xe1/0x100 [ 7.214076] [<ffffffff8159b50f>] down_read+0x1f/0x60 [ 7.214079] [<ffffffff810627ff>] exit_signals+0x1f/0x140 [ 7.214083] [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20 [ 7.214086] [<ffffffff81052844>] do_exit+0xb4/0x4b0 [ 7.214089] [<ffffffff8159e23c>] oops_end+0xdc/0xe0 [ 7.214092] [<ffffffff81005f86>] die+0x56/0x90 [ 7.214095] [<ffffffff8159dea2>] do_general_protection+0x162/0x170 [ 7.214097] [<ffffffff8159d40c>] ? restore_args+0x30/0x30 [ 7.214099] [<ffffffff8159d592>] general_protection+0x22/0x30 [ 7.214102] [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160 [ 7.214104] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50 [ 7.214107] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40 [ 7.214109] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40 [ 7.214113] [<ffffffff811b1442>] dput+0x22/0x130 [ 7.214115] [<ffffffff811a3d45>] path_put+0x15/0x30 [ 7.214117] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40 [ 7.214119] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30 [ 7.214121] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390 [ 7.214124] [<ffffffff815a0c84>] ? __do_page_fault+0x2c4/0x480 [ 7.214127] [<ffffffff811dd18b>] send_to_group+0xfb/0x180 [ 7.214129] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0 [ 7.214131] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0 [ 7.214134] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0 [ 7.214136] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220 [ 7.214139] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f [ 7.214141] [<ffffffff81198859>] SyS_open+0x19/0x20 [ 7.214143] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b [ 7.214145] note: grep[757] exited with preempt_count 1
Signature · 2 lines
Jiri Kosina SUSE Labs
Quoted 8 lines · Jan Kara
Quoted 3 lines
2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction
R14 is
0x6b6b6b6b6b6b6c03, which looks like a use-after-free.Yup. But I'm somewhat puzzled by the trace. We crash when calling
fsnotify_destroy_event()fromfanotify_handle_event(). The fsnotify code has been called fromdo_sys_open()so the event was a 'FS_OPEN' which fails thefsn_event->mask&FAN_ALL_PERM_EVENTStest.
Quoted 7 lines
Slapping my forehead, that's a really stupid bug. The event
fsnotify_add_notify_event()returns may be freed by the time we return because we already dropped the notification mutex. And thenfsn_event->mask&FAN_ALL_PERM_EVENTStest will pass becauseFAN_ALL_PERM_EVENTSmatches with the poison pattern0x6b6b6b6b. So yet another hacked up version of fanotify fix is attached. And I have to seriously think about use counts for fanotify version of that struct.
With the fixed version of the patch, all the fanotify-related oopses are gone on my system.
Signature · 2 lines
Jiri Kosina SUSE Labs
Jan Kara
maintainerQuoted 18 lines · Jiri Kosina
Quoted 8 lines · Jan Kara
Quoted 3 lines
2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction
R14 is
0x6b6b6b6b6b6b6c03, which looks like a use-after-free.Yup. But I'm somewhat puzzled by the trace. We crash when calling
fsnotify_destroy_event()fromfanotify_handle_event(). The fsnotify code has been called fromdo_sys_open()so the event was a 'FS_OPEN' which fails thefsn_event->mask&FAN_ALL_PERM_EVENTStest.Quoted 7 lines
Slapping my forehead, that's a really stupid bug. The event
fsnotify_add_notify_event()returns may be freed by the time we return because we already dropped the notification mutex. And thenfsn_event->mask&FAN_ALL_PERM_EVENTStest will pass becauseFAN_ALL_PERM_EVENTSmatches with the poison pattern0x6b6b6b6b. So yet another hacked up version of fanotify fix is attached. And I have to seriously think about use counts for fanotify version of that struct.
Quoted 2 lines
With the fixed version of the patch, all the fanotify-related oopses are gone on my system.
Thanks for testing. So now I have to come up with something mergeable :)
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 19 lines · Jan Kara
Quoted 18 lines · Jiri Kosina
Quoted 15 lines · Jan Kara
Quoted 3 lines
2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction
R14 is
0x6b6b6b6b6b6b6c03, which looks like a use-after-free.Yup. But I'm somewhat puzzled by the trace. We crash when calling
fsnotify_destroy_event()fromfanotify_handle_event(). The fsnotify code has been called fromdo_sys_open()so the event was a 'FS_OPEN' which fails thefsn_event->mask&FAN_ALL_PERM_EVENTStest.Slapping my forehead, that's a really stupid bug. The event
fsnotify_add_notify_event()returns may be freed by the time we return because we already dropped the notification mutex. And thenfsn_event->mask&FAN_ALL_PERM_EVENTStest will pass becauseFAN_ALL_PERM_EVENTSmatches with the poison pattern0x6b6b6b6b. So yet another hacked up version of fanotify fix is attached. And I have to seriously think about use counts for fanotify version of that struct.
Quoted 3 lines
Quoted 2 lines
With the fixed version of the patch, all the fanotify-related oopses are gone on my system.
Thanks for testing. So now I have to come up with something mergeable :)
Yep, looks good to me too. Thanks.
Dave