Skip to contentSkip to conversation

fanotify use after free.

DISCUSSION Dave Jones 4 participants 19 messages Jan 22, 2014 – Jan 28, 2014

Subsystems (2)
lore
Share & tools
lore

Dave Jones

author
#0 lore raw

Jan,

since yesterdays changes, on boot I see a flood of messages from slub debug during boot..

=============================================================================
BUG fanotify_event_info (Not tainted): Poison overwritten
-----------------------------------------------------------------------------

Disabling lock debugging due to kernel taint
INFO: 0xffff880247e45bc8-0xffff880247e45bcb. First byte 0x0 instead of 0x6b
INFO: Allocated in fanotify_handle_event+0x136/0x390 age=0 cpu=0 pid=293

 __slab_alloc+0x456/0x565
 kmem_cache_alloc+0x1fe/0x260
 fanotify_handle_event+0x136/0x390
 send_to_group+0xd3/0x1c0
 fsnotify+0x1c8/0x340
 open_exec+0xe2/0x120
 load_elf_binary+0x7b7/0x18e0
 search_binary_handler+0x94/0x1b0
 do_execve_common.isra.26+0x5d7/0x7d0
 SyS_execve+0x36/0x50
 stub_execve+0x69/0xa0
INFO: Freed in fanotify_free_event+0x2e/0x40 age=0 cpu=3 pid=290
 __slab_free+0x4a/0x382
 kmem_cache_free+0x1c9/0x210
 fanotify_free_event+0x2e/0x40
 fsnotify_destroy_event+0x21/0x30
 fanotify_read+0x39e/0x5e0
 vfs_read+0x9b/0x160
 SyS_read+0x58/0xb0
 tracesys+0xdd/0xe2

INFO: Slab 0xffffea00091f9100 objects=20 used=20 fp=0x (null) flags=0x20000000004080
INFO: Object 0xffff880247e45b90 @offset=7056 fp=0xffff880247e44000

Bytes b4 ffff880247e45b80: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ Object ffff880247e45b90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object ffff880247e45ba0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object ffff880247e45bb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk Object ffff880247e45bc0: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk.
Redzone ffff880247e45bd0: bb bb bb bb bb bb bb bb ........
Padding ffff880247e45d10: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ

CPU: 0 PID: 293 Comm: mount Tainted: G    B        3.13.0+ #28 
 ffff880247e45b90 000000008c7fe87c ffff8800874cbb28 ffffffff9c710632
 ffff88024a776ac0 ffff8800874cbb68 ffffffff9c194dad 0000000000000008
 ffff880200000001 ffff880247e45bcc ffff88024a776ac0 000000000000006b
Call Trace:
 [<ffffffff9c710632>] dump_stack+0x4e/0x7a
 [<ffffffff9c194dad>] print_trailer+0x14d/0x200
 [<ffffffff9c19505f>] check_bytes_and_report+0xcf/0x110
 [<ffffffff9c196037>] check_object+0x1d7/0x250
 [<ffffffff9c1f4ae6>] ? fanotify_handle_event+0x136/0x390
 [<ffffffff9c70ead7>] alloc_debug_processing+0x76/0x118
 [<ffffffff9c70f77d>] __slab_alloc+0x456/0x565
 [<ffffffff9c1f4ae6>] ? fanotify_handle_event+0x136/0x390
 [<ffffffff9c1ccea4>] ? mntput+0x24/0x40
 [<ffffffff9c1b5dc9>] ? terminate_walk+0x69/0x70
 [<ffffffff9c1ba6fe>] ? do_last+0x25e/0x1390
 [<ffffffff9c1b6cf8>] ? inode_permission+0x18/0x50
 [<ffffffff9c1f4ae6>] ? fanotify_handle_event+0x136/0x390
 [<ffffffff9c1980fe>] kmem_cache_alloc+0x1fe/0x260
 [<ffffffff9c1f4ae6>] fanotify_handle_event+0x136/0x390
 [<ffffffff9c1bb8fd>] ? path_openat+0xcd/0x6a0
 [<ffffffff9c1f0e63>] send_to_group+0xd3/0x1c0
 [<ffffffff9c1f0fdf>] ? fsnotify+0x8f/0x340
 [<ffffffff9c1f1118>] fsnotify+0x1c8/0x340
 [<ffffffff9c1a9b4f>] do_sys_open+0x19f/0x230
 [<ffffffff9c1a9bfe>] SyS_open+0x1e/0x20
 [<ffffffff9c723764>] tracesys+0xdd/0xe2

FIX fanotify_event_info: Restoring 0xffff880247e45bc8-0xffff880247e45bcb=0x6b

author
#1 lore raw
Quoted 31 lines · Dave Jones

Jan,

since yesterdays changes, on boot I see a flood of messages from slub debug during boot..

=============================================================================
BUG fanotify_event_info (Not tainted): Poison overwritten
-----------------------------------------------------------------------------

Disabling lock debugging due to kernel taint
INFO: 0xffff880247e45bc8-0xffff880247e45bcb. First byte 0x0 instead of 0x6b
INFO: Allocated in fanotify_handle_event+0x136/0x390 age=0 cpu=0 pid=293

 __slab_alloc+0x456/0x565
 kmem_cache_alloc+0x1fe/0x260
 fanotify_handle_event+0x136/0x390
 send_to_group+0xd3/0x1c0
 fsnotify+0x1c8/0x340
 open_exec+0xe2/0x120
 load_elf_binary+0x7b7/0x18e0
 search_binary_handler+0x94/0x1b0
 do_execve_common.isra.26+0x5d7/0x7d0
 SyS_execve+0x36/0x50
 stub_execve+0x69/0xa0
INFO: Freed in fanotify_free_event+0x2e/0x40 age=0 cpu=3 pid=290
 __slab_free+0x4a/0x382
 kmem_cache_free+0x1c9/0x210
 fanotify_free_event+0x2e/0x40
 fsnotify_destroy_event+0x21/0x30
 fanotify_read+0x39e/0x5e0
 vfs_read+0x9b/0x160
 SyS_read+0x58/0xb0
 tracesys+0xdd/0xe2
Quoted 1 line

INFO: Slab 0xffffea00091f9100 objects=20 used=20 fp=0x (null) flags=0x20000000004080

Reverting 7053aee26a3548ebaba046ae2e52396ccf56ac6c makes this go away.

Dave

#2 lore raw
In reply to Dave Jones
Quoted 2 lines · Dave Jones

BUG fanotify_event_info (Not tainted): Poison overwritten

Looking at the poison data, it seems that is is the

        u32 response;

field that has been overwritten (with all zero).

That doesn't really help me guess where the bug is, though. That code is crazy. For example, looking at one place where it is set, we have:

- process_access_response():

        re->event->response = response;

        wake_up(&group->fanotify_data.access_waitq);

        kmem_cache_free(fanotify_response_event_cache, re);

which looks buggy in *so* many ways. In particular, we're doing a kmem_cache_free() on "re", but what happened to "re->event" that we just used? There was no release of that anywhere. Wut?

So it seems that the lifetime of these "fanotify_event_info" structures is completely buggered. I don't even see any *attempt* to maintain reference counts or other lifetime info. People free the containers that point to them without doing anything at all about the fsnotify_event that contains the fanotify_event_info that they point to.

Jan - how is the lifetime of the fanotify_event_info tied to the lifetime of the fanotify_response_event structure that contains pointers into it? Because I don't see it.

And considering that it's the response field that gets overwritten, it really sounds like *that* is the exact issue at play here - there is some fanotify_response_event structure holding a pointer to the fanotify_event that is embedded into a fanotify_event_info that has been freed.

Jan?

Linus

maintainer
#3 lore raw
Quoted 32 lines · Linus Torvalds
Quoted 2 lines · Dave Jones

BUG fanotify_event_info (Not tainted): Poison overwritten

Looking at the poison data, it seems that is is the

        u32 response;

field that has been overwritten (with all zero).

That doesn't really help me guess where the bug is, though. That code is crazy. For example, looking at one place where it is set, we have:

- process_access_response():

        re->event->response = response;

        wake_up(&group->fanotify_data.access_waitq);

        kmem_cache_free(fanotify_response_event_cache, re);

which looks buggy in *so* many ways. In particular, we're doing a kmem_cache_free() on "re", but what happened to "re->event" that we just used? There was no release of that anywhere. Wut?

So it seems that the lifetime of these "fanotify_event_info" structures is completely buggered. I don't even see any *attempt* to maintain reference counts or other lifetime info. People free the containers that point to them without doing anything at all about the fsnotify_event that contains the fanotify_event_info that they point to.

Quoted 3 lines

Jan - how is the lifetime of the fanotify_event_info tied to the lifetime of the fanotify_response_event structure that contains pointers into it? Because I don't see it.

Yeah, I messed that up. They aren't tied in any way - well, in fact they
end up being tied but in a wrong way. fanotify_event_info lives from the moment event happens to the moment user reads the event. At that moment the fanotify_response_event gets created (for those special permission events), pointing to fanotify_event_info which will get freed just several lines further :-|

But refcounting seems like an overkill for this - there is exactly one fanotify_response_event structure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.

Quoted 5 lines

And considering that it's the response field that gets overwritten, it really sounds like *that* is the exact issue at play here - there is some fanotify_response_event structure holding a pointer to the fanotify_event that is embedded into a fanotify_event_info that has been freed.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
fs/notify/fanotify/fanotify.c +3 -1
index 58772623f02a..756e9b047e27 100644
@@ -201,8 +201,10 @@ static int fanotify_handle_event(struct fsnotify_group *group,
 	}
 
 #ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS
-	if (fsn_event->mask & FAN_ALL_PERM_EVENTS)
+	if (fsn_event->mask & FAN_ALL_PERM_EVENTS) {
 		ret = fanotify_get_response_from_access(group, event);
+		fsnotify_destroy_event(group, event);
+	}
 #endif
 	return ret;
 }
fs/notify/fanotify/fanotify_user.c +2 -1
index 57d7c083cb4b..d493c72c71fd 100644
@@ -319,7 +319,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,
 			if (IS_ERR(kevent))
 				break;
 			ret = copy_event_to_user(group, kevent, buf);
-			fsnotify_destroy_event(group, kevent);
+			if (!(kevent->mask & FAN_ALL_PERM_EVENTS))
+				fsnotify_destroy_event(group, kevent);
 			if (ret < 0)
 				break;
 			buf += ret;
#4 lore raw
Quoted 6 lines · Jan Kara

But refcounting seems like an overkill for this - there is exactly one fanotify_response_event structure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.

Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?

Linus

author
#5 lore raw
Quoted 10 lines · Linus Torvalds
Quoted 6 lines · Jan Kara

But refcounting seems like an overkill for this - there is exactly one fanotify_response_event structure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.

Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?

It actually seems worse. I see the tail end of what looks like a slab corruption trace, and then a total lockup. And of course none of this makes it over ttyUSB0 because it happens so early. Grr.

Dave

maintainer
#6 lore raw
Quoted 9 lines · Dave Jones
Quoted 8 lines · Linus Torvalds
Quoted 6 lines · Jan Kara

But refcounting seems like an overkill for this - there is exactly one fanotify_response_event structure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.

Quoted 6 lines
Quoted 2 lines

Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?

It actually seems worse. I see the tail end of what looks like a slab corruption trace, and then a total lockup. And of course none of this makes it over ttyUSB0 because it happens so early. Grr.

Drat. Since this seems reasonably reproducible, I'll try to reproduce
and debug this (it seems systemd is using fanotify in a way that triggers this). Thanks for testing.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#7 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 5 lines

But refcounting seems like an overkill for this - there is exactly one fanotify_response_event structure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.

Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?

I reported the same slab corruption yesterday as well here:

	https://lkml.org/lkml/2014/1/22/173

With the patch applied, I am still seeing the slab corruption, preceeded by GPF (which is not there without the patch) in lockref_put_or_lock(&dentry->d_lockref) in dput():

 general protection fault: 0000 [#1] SMP 
 Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys
 CPU: 1 PID: 275 Comm: systemd-readahe Not tainted 3.13.0-03478-g670d0ac #1
 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
 task: ffff880037c09150 ti: ffff88007359e000 task.ti: ffff88007359e000
 RIP: 0010:[<ffffffff810a51e7>]  [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
 RSP: 0018:ffff88007359fc78  EFLAGS: 00010286
 RAX: ffff880037c09150 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000
 tpm_tis 00:09: 1.2 TPM (device-id 0x1020, rev-id 6)
 tpm_tis 00:09: Intel iTPM workaround enabled
 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb
 RBP: ffff88007359fc98 R08: 0000000000000002 R09: 0000000000000000
 R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb
 R13: ffff880037310690 R14: 0000000000000020 R15: ffff880036dbfc10
 FS:  00007fa953e4a700(0000) GS:ffff88007c280000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 00007fff76a10258 CR3: 000000003659c000 CR4: 00000000000007e0
 Stack:
  6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880037310690
  ffff88007359fcb8 ffffffff8159c59c ffffffff812fe101 6b6b6b6b6b6b6beb
  ffff88007359fcd8 ffffffff812fe101 ffff88007359fd08 6b6b6b6b6b6b6b6b
 Call Trace:
  [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50
  [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40
  [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40
  [<ffffffff811b1442>] dput+0x22/0x130
  [<ffffffff811a3d45>] path_put+0x15/0x30
  [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40
  [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
  [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390
  [<ffffffff811dd18b>] send_to_group+0xfb/0x180
  [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
  [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
  [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
  [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
  [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
  [<ffffffff81198859>] SyS_open+0x19/0x20
  [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
 Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff 
 RIP  [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
  RSP <ffff88007359fc78>
 ---[ end trace 7a918209ee213d28 ]---
 BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20
 in_atomic(): 1, irqs_disabled(): 0, pid: 275, name: systemd-readahe
 INFO: lockdep is turned off.
 CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G      D      3.13.0-03478-g670d0ac #1
 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
  ffff880037c09150 ffff88007359fa78 ffffffff8159702b ffff88007359fa98
  ffffffff8107f621 ffffffff81a3d000 ffff880037b98d90 ffff88007359fab8
  ffffffff8159b4ff ffff880037c09150 ffff880037c09150 ffff88007359fae8
 Call Trace:
  [<ffffffff8159702b>] dump_stack+0x72/0x87
  [<ffffffff8107f621>] __might_sleep+0xe1/0x100
  [<ffffffff8159b4ff>] down_read+0x1f/0x60
  [<ffffffff810627ff>] exit_signals+0x1f/0x140
  [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20
  [<ffffffff81052844>] do_exit+0xb4/0x4b0
  [<ffffffff8159e23c>] oops_end+0xdc/0xe0
  [<ffffffff81005f86>] die+0x56/0x90
  [<ffffffff8159dea2>] do_general_protection+0x162/0x170
  [<ffffffff8159d40c>] ? restore_args+0x30/0x30
  [<ffffffff8159d592>] general_protection+0x22/0x30
  [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
  [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50
  [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40
  [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40
  [<ffffffff811b1442>] dput+0x22/0x130
  [<ffffffff811a3d45>] path_put+0x15/0x30
  [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40
  [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
  [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390
  [<ffffffff811dd18b>] send_to_group+0xfb/0x180
  [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
  [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
  [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
  [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
  [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
  [<ffffffff81198859>] SyS_open+0x19/0x20
  [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
 note: systemd-readahe[275] exited with preempt_count 1
 BUG: scheduling while atomic: systemd-readahe/275/0x00000002
 INFO: lockdep is turned off.
 Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys
 CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G      D      3.13.0-03478-g670d0ac #1
 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
  ffff88007c293a00 ffff88007359f6f8 ffffffff8159702b ffff88007359f718
  ffffffff810810f1 ffff88007c293a00 0000000000000001 ffff88007359f848
  ffffffff8159789c ffff88007359f758 ffff88007359f768 ffff88007359e010
 Call Trace:
  [<ffffffff8159702b>] dump_stack+0x72/0x87
  [<ffffffff810810f1>] __schedule_bug+0x61/0x80
  [<ffffffff8159789c>] __schedule+0xbc/0x7c0
  [<ffffffff8105defc>] ? mod_timer+0x14c/0x1f0
  [<ffffffff815980e4>] schedule+0x24/0x70
  [<ffffffff81597205>] schedule_timeout+0x1c5/0x210
  [<ffffffff8159914f>] ? wait_for_completion+0xcf/0x120
  [<ffffffff810a0d8d>] ? trace_hardirqs_on+0xd/0x10
  [<ffffffff81599157>] wait_for_completion+0xd7/0x120
  [<ffffffff81083330>] ? try_to_wake_up+0x250/0x250
  [<ffffffff810b93bf>] ? srcu_reschedule+0x4f/0xf0
  [<ffffffff810b965c>] __synchronize_srcu+0xec/0x130
  [<ffffffff810b96e0>] ? srcu_barrier+0x10/0x10
  [<ffffffff810b96c8>] synchronize_srcu+0x18/0x20
  [<ffffffff811ddbdd>] fsnotify_destroy_group+0x1d/0x40
  [<ffffffff811dfdf1>] inotify_release+0x21/0x50
  [<ffffffff8119b2dd>] __fput+0xbd/0x2b0
  [<ffffffff8119b569>] ____fput+0x9/0x10
  [<ffffffff81070f41>] task_work_run+0xb1/0xe0
  [<ffffffff81052979>] do_exit+0x1e9/0x4b0
  [<ffffffff8159e23c>] oops_end+0xdc/0xe0
  [<ffffffff81005f86>] die+0x56/0x90
  [<ffffffff8159dea2>] do_general_protection+0x162/0x170
  [<ffffffff8159d40c>] ? restore_args+0x30/0x30
  [<ffffffff8159d592>] general_protection+0x22/0x30
  [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
  [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50
  [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40
  [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40
  [<ffffffff811b1442>] dput+0x22/0x130
  [<ffffffff811a3d45>] path_put+0x15/0x30
  [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40
  [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
  [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390
  [<ffffffff811dd18b>] send_to_group+0xfb/0x180
  [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
  [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
  [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
  [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
  [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
  [<ffffffff81198859>] SyS_open+0x19/0x20
  [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
 ACPI: AC Adapter [AC] (on-line)
 Slab corruption (Tainted: G      D W   ): fanotify_event_info start=ffff880037310690, len=64
 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
 Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40)
 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5  kkkkkkkk....kkk.
 Prev obj: start=ffff880037310638, len=64
 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
 Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40)
 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 Next obj: start=ffff8800373106e8, len=64
 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
 Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40)
 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Signature · 2 lines
Jiri Kosina
SUSE Labs
maintainer
#8 lore raw
Quoted 11 lines · Jiri Kosina
Quoted 8 lines · Linus Torvalds
Quoted 5 lines

But refcounting seems like an overkill for this - there is exactly one fanotify_response_event structure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.

Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?

Quoted 7 lines

I reported the same slab corruption yesterday as well here:

	https://lkml.org/lkml/2014/1/22/173

With the patch applied, I am still seeing the slab corruption, preceeded by GPF (which is not there without the patch) in lockref_put_or_lock(&dentry->d_lockref) in dput():

Hmm, OK. Can you please send me your .config? I'll try to reproduce this
myself.

Honza

Quoted 151 lines
 general protection fault: 0000 [#1] SMP 
 Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys
 CPU: 1 PID: 275 Comm: systemd-readahe Not tainted 3.13.0-03478-g670d0ac #1
 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
 task: ffff880037c09150 ti: ffff88007359e000 task.ti: ffff88007359e000
 RIP: 0010:[<ffffffff810a51e7>]  [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
 RSP: 0018:ffff88007359fc78  EFLAGS: 00010286
 RAX: ffff880037c09150 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000
 tpm_tis 00:09: 1.2 TPM (device-id 0x1020, rev-id 6)
 tpm_tis 00:09: Intel iTPM workaround enabled
 RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb
 RBP: ffff88007359fc98 R08: 0000000000000002 R09: 0000000000000000
 R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb
 R13: ffff880037310690 R14: 0000000000000020 R15: ffff880036dbfc10
 FS:  00007fa953e4a700(0000) GS:ffff88007c280000(0000) knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 00007fff76a10258 CR3: 000000003659c000 CR4: 00000000000007e0
 Stack:
  6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880037310690
  ffff88007359fcb8 ffffffff8159c59c ffffffff812fe101 6b6b6b6b6b6b6beb
  ffff88007359fcd8 ffffffff812fe101 ffff88007359fd08 6b6b6b6b6b6b6b6b
 Call Trace:
  [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50
  [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40
  [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40
  [<ffffffff811b1442>] dput+0x22/0x130
  [<ffffffff811a3d45>] path_put+0x15/0x30
  [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40
  [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
  [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390
  [<ffffffff811dd18b>] send_to_group+0xfb/0x180
  [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
  [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
  [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
  [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
  [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
  [<ffffffff81198859>] SyS_open+0x19/0x20
  [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
 Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff 
 RIP  [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
  RSP <ffff88007359fc78>
 ---[ end trace 7a918209ee213d28 ]---
 BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20
 in_atomic(): 1, irqs_disabled(): 0, pid: 275, name: systemd-readahe
 INFO: lockdep is turned off.
 CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G      D      3.13.0-03478-g670d0ac #1
 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
  ffff880037c09150 ffff88007359fa78 ffffffff8159702b ffff88007359fa98
  ffffffff8107f621 ffffffff81a3d000 ffff880037b98d90 ffff88007359fab8
  ffffffff8159b4ff ffff880037c09150 ffff880037c09150 ffff88007359fae8
 Call Trace:
  [<ffffffff8159702b>] dump_stack+0x72/0x87
  [<ffffffff8107f621>] __might_sleep+0xe1/0x100
  [<ffffffff8159b4ff>] down_read+0x1f/0x60
  [<ffffffff810627ff>] exit_signals+0x1f/0x140
  [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20
  [<ffffffff81052844>] do_exit+0xb4/0x4b0
  [<ffffffff8159e23c>] oops_end+0xdc/0xe0
  [<ffffffff81005f86>] die+0x56/0x90
  [<ffffffff8159dea2>] do_general_protection+0x162/0x170
  [<ffffffff8159d40c>] ? restore_args+0x30/0x30
  [<ffffffff8159d592>] general_protection+0x22/0x30
  [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
  [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50
  [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40
  [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40
  [<ffffffff811b1442>] dput+0x22/0x130
  [<ffffffff811a3d45>] path_put+0x15/0x30
  [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40
  [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
  [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390
  [<ffffffff811dd18b>] send_to_group+0xfb/0x180
  [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
  [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
  [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
  [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
  [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
  [<ffffffff81198859>] SyS_open+0x19/0x20
  [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
 note: systemd-readahe[275] exited with preempt_count 1
 BUG: scheduling while atomic: systemd-readahe/275/0x00000002
 INFO: lockdep is turned off.
 Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys
 CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G      D      3.13.0-03478-g670d0ac #1
 Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
  ffff88007c293a00 ffff88007359f6f8 ffffffff8159702b ffff88007359f718
  ffffffff810810f1 ffff88007c293a00 0000000000000001 ffff88007359f848
  ffffffff8159789c ffff88007359f758 ffff88007359f768 ffff88007359e010
 Call Trace:
  [<ffffffff8159702b>] dump_stack+0x72/0x87
  [<ffffffff810810f1>] __schedule_bug+0x61/0x80
  [<ffffffff8159789c>] __schedule+0xbc/0x7c0
  [<ffffffff8105defc>] ? mod_timer+0x14c/0x1f0
  [<ffffffff815980e4>] schedule+0x24/0x70
  [<ffffffff81597205>] schedule_timeout+0x1c5/0x210
  [<ffffffff8159914f>] ? wait_for_completion+0xcf/0x120
  [<ffffffff810a0d8d>] ? trace_hardirqs_on+0xd/0x10
  [<ffffffff81599157>] wait_for_completion+0xd7/0x120
  [<ffffffff81083330>] ? try_to_wake_up+0x250/0x250
  [<ffffffff810b93bf>] ? srcu_reschedule+0x4f/0xf0
  [<ffffffff810b965c>] __synchronize_srcu+0xec/0x130
  [<ffffffff810b96e0>] ? srcu_barrier+0x10/0x10
  [<ffffffff810b96c8>] synchronize_srcu+0x18/0x20
  [<ffffffff811ddbdd>] fsnotify_destroy_group+0x1d/0x40
  [<ffffffff811dfdf1>] inotify_release+0x21/0x50
  [<ffffffff8119b2dd>] __fput+0xbd/0x2b0
  [<ffffffff8119b569>] ____fput+0x9/0x10
  [<ffffffff81070f41>] task_work_run+0xb1/0xe0
  [<ffffffff81052979>] do_exit+0x1e9/0x4b0
  [<ffffffff8159e23c>] oops_end+0xdc/0xe0
  [<ffffffff81005f86>] die+0x56/0x90
  [<ffffffff8159dea2>] do_general_protection+0x162/0x170
  [<ffffffff8159d40c>] ? restore_args+0x30/0x30
  [<ffffffff8159d592>] general_protection+0x22/0x30
  [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
  [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50
  [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40
  [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40
  [<ffffffff811b1442>] dput+0x22/0x130
  [<ffffffff811a3d45>] path_put+0x15/0x30
  [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40
  [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
  [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390
  [<ffffffff811dd18b>] send_to_group+0xfb/0x180
  [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
  [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
  [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
  [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
  [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
  [<ffffffff81198859>] SyS_open+0x19/0x20
  [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
 ACPI: AC Adapter [AC] (on-line)
 Slab corruption (Tainted: G      D W   ): fanotify_event_info start=ffff880037310690, len=64
 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
 Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40)
 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5  kkkkkkkk....kkk.
 Prev obj: start=ffff880037310638, len=64
 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
 Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40)
 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 Next obj: start=ffff8800373106e8, len=64
 Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
 Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40)
 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
Signature · 2 lines
Jiri Kosina
SUSE Labs
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#9 lore raw
Quoted 11 lines · Jan Kara
Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

But refcounting seems like an overkill for this - there is exactly one fanotify_response_event structure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.

Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?

I reported the same slab corruption yesterday as well here:

Quoted 7 lines
Quoted 5 lines
	https://lkml.org/lkml/2014/1/22/173

With the patch applied, I am still seeing the slab corruption, preceeded by GPF (which is not there without the patch) in lockref_put_or_lock(&dentry->d_lockref) in dput():

Hmm, OK. Can you please send me your .config? I'll try to reproduce this
myself.

Attached.

The userspace is systemd-based.

Signature · 2 lines
Jiri Kosina
SUSE Labs
maintainer
#10 lore raw
Quoted 18 lines · Jiri Kosina
Quoted 16 lines · Jan Kara
Quoted 16 lines
Quoted 8 lines
Quoted 5 lines

But refcounting seems like an overkill for this - there is exactly one fanotify_response_event structure iff it is a permission event. So something like the (completely untested) attached patch should fix the problem. But I agree it's a bit ugly so we might want something different. I'll try to think about something better tomorrow.

Ok, In the meantime, Dave, can you verify whether this hacky patch fixes your problem?

I reported the same slab corruption yesterday as well here:

	https://lkml.org/lkml/2014/1/22/173

With the patch applied, I am still seeing the slab corruption, preceeded by GPF (which is not there without the patch) in lockref_put_or_lock(&dentry->d_lockref) in dput():

Quoted 6 lines
Quoted 2 lines

Hmm, OK. Can you please send me your .config? I'll try to reproduce this
myself.

Attached.

The userspace is systemd-based.

Strange. I've installed systemd system (openSUSE 13.1) and it boots with
the latest Linus' kernel just fine (and I have at least FANOTIFY and SLAB debugging set the same way as you). But it was only a KVM guest. I'll try tomorrow with a physical machine I guess.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#11 lore raw
Quoted 4 lines · Jan Kara

Strange. I've installed systemd system (openSUSE 13.1) and it boots
with the latest Linus' kernel just fine (and I have at least FANOTIFY and SLAB debugging set the same way as you). But it was only a KVM guest. I'll try tomorrow with a physical machine I guess.

FWIW the system I am reliably able to reproduce this on is opensuse 12.3 with this systemd version:

Version : 195
Release : 13.18.1

Signature · 2 lines
Jiri Kosina
SUSE Labs
maintainer
#12 lore raw
Quoted 7 lines · Jiri Kosina
Quoted 4 lines · Jan Kara

Strange. I've installed systemd system (openSUSE 13.1) and it boots
with the latest Linus' kernel just fine (and I have at least FANOTIFY and SLAB debugging set the same way as you). But it was only a KVM guest. I'll try tomorrow with a physical machine I guess.

Quoted 5 lines

FWIW the system I am reliably able to reproduce this on is opensuse 12.3 with this systemd version:

Version : 195
Release : 13.18.1

Hum, still no luck with reproduction (either on physical machine or with
KVM). Anyway, I've looked at the code again and the previous patch had a stupid bug (passing different pointer to fsnotify_destroy_event() than we should have), plus also the merging function in fanotify was too aggressive. Can you try the attached patch? It boots for me but that means nothing since I cannot reproduce the issue... Thanks!

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
fs/notify/fanotify/fanotify.c +6 -3
index 58772623f02a..1b3dd9de8518 100644
@@ -18,7 +18,7 @@ static bool should_merge(struct fsnotify_event *old_fsn,
 
 #ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS
 	/* dont merge two permission events */
-	if ((old_fsn->mask & FAN_ALL_PERM_EVENTS) &&
+	if ((old_fsn->mask & FAN_ALL_PERM_EVENTS) ||
 	    (new_fsn->mask & FAN_ALL_PERM_EVENTS))
 		return false;
 #endif
@@ -201,8 +201,10 @@ static int fanotify_handle_event(struct fsnotify_group *group,
 	}
 
 #ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS
-	if (fsn_event->mask & FAN_ALL_PERM_EVENTS)
+	if (fsn_event->mask & FAN_ALL_PERM_EVENTS) {
 		ret = fanotify_get_response_from_access(group, event);
+		fsnotify_destroy_event(group, fsn_event);
+	}
 #endif
 	return ret;
 }
@@ -221,7 +223,8 @@ static void fanotify_free_event(struct fsnotify_event *fsn_event)
 	struct fanotify_event_info *event;
 
 	event = FANOTIFY_E(fsn_event);
-	path_put(&event->path);
+	if (event->path.mnt)
+		path_put(&event->path);
 	put_pid(event->tgid);
 	kmem_cache_free(fanotify_event_cachep, event);
 }
fs/notify/fanotify/fanotify_user.c +2 -1
index 57d7c083cb4b..d493c72c71fd 100644
@@ -319,7 +319,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,
 			if (IS_ERR(kevent))
 				break;
 			ret = copy_event_to_user(group, kevent, buf);
-			fsnotify_destroy_event(group, kevent);
+			if (!(kevent->mask & FAN_ALL_PERM_EVENTS))
+				fsnotify_destroy_event(group, kevent);
 			if (ret < 0)
 				break;
 			buf += ret;
author
#13 lore raw
Quoted 11 lines · Jan Kara
Quoted 10 lines · Jiri Kosina
Quoted 4 lines · Jan Kara

Strange. I've installed systemd system (openSUSE 13.1) and it boots
with the latest Linus' kernel just fine (and I have at least FANOTIFY and SLAB debugging set the same way as you). But it was only a KVM guest. I'll try tomorrow with a physical machine I guess.

FWIW the system I am reliably able to reproduce this on is opensuse 12.3 with this systemd version:

Quoted 8 lines
Quoted 2 lines

Version : 195
Release : 13.18.1

Hum, still no luck with reproduction (either on physical machine or with
KVM). Anyway, I've looked at the code again and the previous patch had a stupid bug (passing different pointer to fsnotify_destroy_event() than we should have), plus also the merging function in fanotify was too aggressive. Can you try the attached patch? It boots for me but that means nothing since I cannot reproduce the issue... Thanks!

still not good I'm afraid. I still see corruption very early on in boot and now it panics and locks up too.

Again, this happens so early that I can't grab it over usb-serial. I stuck an mdelay(10000) in the slub corruption detector, and managed to grab a photo of the first trace.

Trace:
? preempt_schedule
lock_acquire
? lockref_put_or_lock
_raw_spin_lock
? lockref_put_or_lock
dput
path_put
fanotify_free_event
fsnotify_destroy_event
fanotify_handle_event
? mntput
? path_openat
? handle_mm_fault
send_to_group
? fsnotify
fsnotify
do_sys_open
sys_open

RIP: lock_acquire

2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction

R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free.

I also notice you mention SLAB above, but I've been using SLUB. I don't know if the choice of allocator makes a difference in reproducability.

It's also worth noting that I have lockdep enabled, which may be perturbing things to some degree.

Dave

maintainer
#14 lore raw
Quoted 21 lines · Dave Jones
Quoted 11 lines · Jan Kara
Quoted 10 lines · Jiri Kosina
Quoted 4 lines · Jan Kara

Strange. I've installed systemd system (openSUSE 13.1) and it boots
with the latest Linus' kernel just fine (and I have at least FANOTIFY and SLAB debugging set the same way as you). But it was only a KVM guest. I'll try tomorrow with a physical machine I guess.

FWIW the system I am reliably able to reproduce this on is opensuse 12.3 with this systemd version:

Quoted 8 lines
Quoted 2 lines

Version : 195
Release : 13.18.1

Hum, still no luck with reproduction (either on physical machine or with
KVM). Anyway, I've looked at the code again and the previous patch had a stupid bug (passing different pointer to fsnotify_destroy_event() than we should have), plus also the merging function in fanotify was too aggressive. Can you try the attached patch? It boots for me but that means nothing since I cannot reproduce the issue... Thanks!

Quoted 2 lines

still not good I'm afraid. I still see corruption very early on in boot and now it panics and locks up too.

Ew, thanks for testing.

Quoted 23 lines

Again, this happens so early that I can't grab it over usb-serial. I stuck an mdelay(10000) in the slub corruption detector, and managed to grab a photo of the first trace.

Trace:
? preempt_schedule
lock_acquire
? lockref_put_or_lock
_raw_spin_lock
? lockref_put_or_lock
dput
path_put
fanotify_free_event
fsnotify_destroy_event
fanotify_handle_event
? mntput
? path_openat
? handle_mm_fault
send_to_group
? fsnotify
fsnotify
do_sys_open
sys_open

Quoted 5 lines
RIP: lock_acquire

2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction

R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free.

Yup. But I'm somewhat puzzled by the trace. We crash when calling
fsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code has been called from do_sys_open() so the event was a 'FS_OPEN' which fails the fsn_event->mask & FAN_ALL_PERM_EVENTS test.

Slapping my forehead, that's a really stupid bug. The event fsnotify_add_notify_event() returns may be freed by the time we return because we already dropped the notification mutex. And then fsn_event->mask & FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches with the poison pattern 0x6b6b6b6b. So yet another hacked up version of fanotify fix is attached. And I have to seriously think about use counts for fanotify version of that struct.

Quoted 2 lines

I also notice you mention SLAB above, but I've been using SLUB. I don't know if the choice of allocator makes a difference in reproducability.

Jiri Kosina has SLAB so SLAB/SLUB apparently doesn't matter.

Quoted 2 lines

It's also worth noting that I have lockdep enabled, which may be perturbing things to some degree.

And I've compiled my kernel with lockdep as well since Jiri has it in his
config. But no luck.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
fs/notify/fanotify/fanotify.c +6 -3
index 58772623f02a..f80895fb5ca1 100644
@@ -18,7 +18,7 @@ static bool should_merge(struct fsnotify_event *old_fsn,
 
 #ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS
 	/* dont merge two permission events */
-	if ((old_fsn->mask & FAN_ALL_PERM_EVENTS) &&
+	if ((old_fsn->mask & FAN_ALL_PERM_EVENTS) ||
 	    (new_fsn->mask & FAN_ALL_PERM_EVENTS))
 		return false;
 #endif
@@ -201,8 +201,10 @@ static int fanotify_handle_event(struct fsnotify_group *group,
 	}
 
 #ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS
-	if (fsn_event->mask & FAN_ALL_PERM_EVENTS)
+	if (mask & FAN_ALL_PERM_EVENTS) {
 		ret = fanotify_get_response_from_access(group, event);
+		fsnotify_destroy_event(group, fsn_event);
+	}
 #endif
 	return ret;
 }
@@ -221,7 +223,8 @@ static void fanotify_free_event(struct fsnotify_event *fsn_event)
 	struct fanotify_event_info *event;
 
 	event = FANOTIFY_E(fsn_event);
-	path_put(&event->path);
+	if (event->path.mnt)
+		path_put(&event->path);
 	put_pid(event->tgid);
 	kmem_cache_free(fanotify_event_cachep, event);
 }
fs/notify/fanotify/fanotify_user.c +2 -1
index 57d7c083cb4b..d493c72c71fd 100644
@@ -319,7 +319,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,
 			if (IS_ERR(kevent))
 				break;
 			ret = copy_event_to_user(group, kevent, buf);
-			fsnotify_destroy_event(group, kevent);
+			if (!(kevent->mask & FAN_ALL_PERM_EVENTS))
+				fsnotify_destroy_event(group, kevent);
 			if (ret < 0)
 				break;
 			buf += ret;
#15 lore raw
Quoted 8 lines · Jan Kara
Quoted 3 lines

2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction

R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free.

Yup. But I'm somewhat puzzled by the trace. We crash when calling
fsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code has been called from do_sys_open() so the event was a 'FS_OPEN' which fails the fsn_event->mask & FAN_ALL_PERM_EVENTS test.

Quoted 7 lines

Slapping my forehead, that's a really stupid bug. The event fsnotify_add_notify_event() returns may be freed by the time we return because we already dropped the notification mutex. And then fsn_event->mask & FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches with the poison pattern 0x6b6b6b6b. So yet another hacked up version of fanotify fix is attached. And I have to seriously think about use counts for fanotify version of that struct.

With the fixed version of the patch, all the fanotify-related oopses are gone on my system.

Signature · 2 lines
Jiri Kosina
SUSE Labs
maintainer
#16 lore raw
Quoted 18 lines · Jiri Kosina
Quoted 8 lines · Jan Kara
Quoted 3 lines

2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction

R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free.

Yup. But I'm somewhat puzzled by the trace. We crash when calling
fsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code has been called from do_sys_open() so the event was a 'FS_OPEN' which fails the fsn_event->mask & FAN_ALL_PERM_EVENTS test.

Quoted 7 lines

Slapping my forehead, that's a really stupid bug. The event fsnotify_add_notify_event() returns may be freed by the time we return because we already dropped the notification mutex. And then fsn_event->mask & FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches with the poison pattern 0x6b6b6b6b. So yet another hacked up version of fanotify fix is attached. And I have to seriously think about use counts for fanotify version of that struct.

Quoted 2 lines

With the fixed version of the patch, all the fanotify-related oopses are gone on my system.

Thanks for testing. So now I have to come up with something mergeable :)

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
author
#17 lore raw
Quoted 19 lines · Jan Kara
Quoted 18 lines · Jiri Kosina
Quoted 15 lines · Jan Kara
Quoted 3 lines

2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction

R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free.

Yup. But I'm somewhat puzzled by the trace. We crash when calling
fsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code has been called from do_sys_open() so the event was a 'FS_OPEN' which fails the fsn_event->mask & FAN_ALL_PERM_EVENTS test.

Slapping my forehead, that's a really stupid bug. The event fsnotify_add_notify_event() returns may be freed by the time we return because we already dropped the notification mutex. And then fsn_event->mask & FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches with the poison pattern 0x6b6b6b6b. So yet another hacked up version of fanotify fix is attached. And I have to seriously think about use counts for fanotify version of that struct.

Quoted 3 lines
Quoted 2 lines

With the fixed version of the patch, all the fanotify-related oopses are gone on my system.

Thanks for testing. So now I have to come up with something mergeable :)

Yep, looks good to me too. Thanks.

Dave

#18 lore raw
In reply to Jan Kara
Quoted 6 lines · Jan Kara

Hum, still no luck with reproduction (either on physical machine or with
KVM). Anyway, I've looked at the code again and the previous patch had a stupid bug (passing different pointer to fsnotify_destroy_event() than we should have), plus also the merging function in fanotify was too aggressive. Can you try the attached patch? It boots for me but that means nothing since I cannot reproduce the issue... Thanks!

I am attaching dmesg with the patch applied; I've removed irrelevant parts.

There is a GPF, followed by scheduling in atomic context, followed by slab corruption, followed by another scheduling while atomic and leak of preempt_count.

[    0.000000] Initializing cgroup subsys cpuset
[    5.081301] systemd-udevd[332]: starting version 195
[    5.083694] random: nonblocking pool is initialized
[    5.299400] systemd-journald[307]: Received SIGUSR1
[    5.625120] general protection fault: 0000 [#1] SMP 
[    5.626464] Modules linked in: acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm usbcore i2c_algo_bit usb_common button video edd fan processor ata_generic thermal thermal_sys
[    5.628008] CPU: 0 PID: 302 Comm: systemd-readahe Not tainted 3.13.0-03478-gae75a37 #1
[    5.628008] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
[    5.628008] task: ffff8800364b04d0 ti: ffff8800734b8000 task.ti: ffff8800734b8000
[    5.628008] RIP: 0010:[<ffffffff810a51e7>]  [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
[    5.628008] RSP: 0018:ffff8800734b9c68  EFLAGS: 00010282
[    5.628008] RAX: ffff8800364b04d0 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000
[    5.628008] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb
[    5.628008] RBP: ffff8800734b9c88 R08: 0000000000000002 R09: 0000000000000000
[    5.628008] R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb
[    5.628008] R13: ffff880035d0db28 R14: 0000000000000020 R15: ffff880037fffd50
[    5.628008] FS:  00007fd2f4728700(0000) GS:ffff88007c200000(0000) knlGS:0000000000000000
[    5.628008] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    5.628008] CR2: 00007fa7dc98c000 CR3: 0000000037f3b000 CR4: 00000000000007f0
[    5.628008] Stack:
[    5.628008]  6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880035d0db28
[    5.628008]  ffff8800734b9ca8 ffffffff8159c5ac ffffffff812fe111 6b6b6b6b6b6b6beb
[    5.628008]  ffff8800734b9cc8 ffffffff812fe111 ffff8800734b9cf8 6b6b6b6b6b6b6b6b
[    5.628008] Call Trace:
[    5.628008]  [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50
[    5.628008]  [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40
[    5.628008]  [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40
[    5.628008]  [<ffffffff811b1442>] dput+0x22/0x130
[    5.628008]  [<ffffffff811a3d45>] path_put+0x15/0x30
[    5.628008]  [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40
[    5.628008]  [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[    5.628008]  [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390
[    5.628008]  [<ffffffff811a3d4d>] ? path_put+0x1d/0x30
[    5.628008]  [<ffffffff811dd18b>] send_to_group+0xfb/0x180
[    5.628008]  [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[    5.628008]  [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[    5.628008]  [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[    5.628008]  [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[    5.628008]  [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[    5.628008]  [<ffffffff81198859>] SyS_open+0x19/0x20
[    5.628008]  [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
[    5.628008] Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff 
[    5.628008] RIP  [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
[    5.628008]  RSP <ffff8800734b9c68>
[    5.683491] ---[ end trace 5b4e9ae52ab9b0f6 ]---
[    5.685076] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20
[    5.686578] in_atomic(): 1, irqs_disabled(): 0, pid: 302, name: systemd-readahe
[    5.688058] INFO: lockdep is turned off.
[    5.689503] CPU: 0 PID: 302 Comm: systemd-readahe Tainted: G      D      3.13.0-03478-gae75a37 #1
[    5.690966] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
[    5.692464]  ffff8800364b04d0 ffff8800734b9a68 ffffffff8159703b ffff8800734b9a88
[    5.694027]  ffffffff8107f621 ffffffff81a3d000 ffff88003641c750 ffff8800734b9aa8
[    5.695555]  ffffffff8159b50f ffff8800364b04d0 ffff8800364b04d0 ffff8800734b9ad8
[    5.697111] Call Trace:
[    5.698595]  [<ffffffff8159703b>] dump_stack+0x72/0x87
[    5.700108]  [<ffffffff8107f621>] __might_sleep+0xe1/0x100
[    5.701623]  [<ffffffff8159b50f>] down_read+0x1f/0x60
[    5.703133]  [<ffffffff810627ff>] exit_signals+0x1f/0x140
[    5.704661]  [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20
[    5.706105]  [<ffffffff81052844>] do_exit+0xb4/0x4b0
[    5.707470]  [<ffffffff8159e23c>] oops_end+0xdc/0xe0
[    5.708845]  [<ffffffff81005f86>] die+0x56/0x90
[    5.710229]  [<ffffffff8159dea2>] do_general_protection+0x162/0x170
[    5.711545]  [<ffffffff8159d40c>] ? restore_args+0x30/0x30
[    5.712883]  [<ffffffff8159d592>] general_protection+0x22/0x30
[    5.714213]  [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
[    5.715491]  [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50
[    5.716781]  [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40
[    5.718113]  [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40
[    5.719390]  [<ffffffff811b1442>] dput+0x22/0x130
[    5.720681]  [<ffffffff811a3d45>] path_put+0x15/0x30
[    5.721984]  [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40
[    5.723243]  [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[    5.724507]  [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390
[    5.725779]  [<ffffffff811a3d4d>] ? path_put+0x1d/0x30
[    5.727014]  [<ffffffff811dd18b>] send_to_group+0xfb/0x180
[    5.728258]  [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[    5.729515]  [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[    5.730734]  [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[    5.731945]  [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[    5.733169]  [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[    5.734456]  [<ffffffff81198859>] SyS_open+0x19/0x20
[    5.735681]  [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
[    5.736933] note: systemd-readahe[302] exited with preempt_count 1
[    5.738378] BUG: scheduling while atomic: systemd-readahe/302/0x00000002
[    5.739652] INFO: lockdep is turned off.
[    5.740925] Modules linked in: acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm usbcore i2c_algo_bit usb_common button video edd fan processor ata_generic thermal thermal_sys
[    5.743781] CPU: 0 PID: 302 Comm: systemd-readahe Tainted: G      D      3.13.0-03478-gae75a37 #1
[    5.745231] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
[    5.746708]  ffff88007c213a00 ffff8800734b96e8 ffffffff8159703b ffff8800734b9708
[    5.748190]  ffffffff810810f1 ffff88007c213a00 0000000000000000 ffff8800734b9838
[    5.749690]  ffffffff815978ac ffff8800734b9748 ffff8800734b9758 ffff8800734b8010
[    5.751161] Call Trace:
[    5.752635]  [<ffffffff8159703b>] dump_stack+0x72/0x87
[    5.754126]  [<ffffffff810810f1>] __schedule_bug+0x61/0x80
[    5.755551]  [<ffffffff815978ac>] __schedule+0xbc/0x7c0
[    5.756925]  [<ffffffff8105defc>] ? mod_timer+0x14c/0x1f0
[    5.758302]  [<ffffffff815980f4>] schedule+0x24/0x70
[    5.759632]  [<ffffffff81597215>] schedule_timeout+0x1c5/0x210
[    5.760982]  [<ffffffff8159915f>] ? wait_for_completion+0xcf/0x120
[    5.762327]  [<ffffffff810a0d8d>] ? trace_hardirqs_on+0xd/0x10
[    5.763630]  [<ffffffff81599167>] wait_for_completion+0xd7/0x120
[    5.764935]  [<ffffffff81083330>] ? try_to_wake_up+0x250/0x250
[    5.766261]  [<ffffffff810b93bf>] ? srcu_reschedule+0x4f/0xf0
[    5.767521]  [<ffffffff810b965c>] __synchronize_srcu+0xec/0x130
[    5.768775]  [<ffffffff810b96e0>] ? srcu_barrier+0x10/0x10
[    5.770059]  [<ffffffff810b96c8>] synchronize_srcu+0x18/0x20
[    5.771302]  [<ffffffff811ddbdd>] fsnotify_destroy_group+0x1d/0x40
[    5.772550]  [<ffffffff811dfdf1>] inotify_release+0x21/0x50
[    5.773814]  [<ffffffff8119b2dd>] __fput+0xbd/0x2b0
[    5.775055]  [<ffffffff8119b569>] ____fput+0x9/0x10
[    5.776311]  [<ffffffff81070f41>] task_work_run+0xb1/0xe0
[    5.777577]  [<ffffffff81052979>] do_exit+0x1e9/0x4b0
[    5.778803]  [<ffffffff8159e23c>] oops_end+0xdc/0xe0
[    5.780027]  [<ffffffff81005f86>] die+0x56/0x90
[    5.781264]  [<ffffffff8159dea2>] do_general_protection+0x162/0x170
[    5.782472]  [<ffffffff8159d40c>] ? restore_args+0x30/0x30
[    5.783687]  [<ffffffff8159d592>] general_protection+0x22/0x30
[    5.784917]  [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
[    5.786160]  [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50
[    5.787367]  [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40
[    5.788606]  [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40
[    5.789848]  [<ffffffff811b1442>] dput+0x22/0x130
[    5.791073]  [<ffffffff811a3d45>] path_put+0x15/0x30
[    5.792324]  [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40
[    5.793589]  [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[    5.794809]  [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390
[    5.796040]  [<ffffffff811a3d4d>] ? path_put+0x1d/0x30
[    5.797283]  [<ffffffff811dd18b>] send_to_group+0xfb/0x180
[    5.798505]  [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[    5.799732]  [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[    5.800970]  [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[    5.802219]  [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[    5.803461]  [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[    5.804729]  [<ffffffff81198859>] SyS_open+0x19/0x20
[    5.805970]  [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
[ ... snip ... ]
[    5.968718] Slab corruption (Tainted: G      D W   ): fanotify_event_info start=ffff880035d2a798, len=64
[    5.968923] hub 7-0:1.0: USB hub found
[    5.971406] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
[    5.972756] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40)
[    5.974098] 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5  kkkkkkkk....kkk.
[    5.974837] hub 7-0:1.0: 6 ports detected
[    5.976799] Prev obj: start=ffff880035d2a740, len=64
[    5.978189] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
[    5.979526] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40)
[    5.980910] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[    5.982307] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[    5.983691] Next obj: start=ffff880035d2a7f0, len=64
[    5.985070] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
[    5.986455] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40)
[    5.987835] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[    5.989248] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[ ... snip ... ]
[    7.044083] usb 2-1: new full-speed USB device number 4 using uhci_hcd
[    7.131735] general protection fault: 0000 [#2] SMP 
[    7.131842] Slab corruption (Tainted: G      D W   ): fanotify_event_info start=ffff880035da5320, len=64
[    7.131844] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
[    7.131850] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40)
[    7.131853] 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5  kkkkkkkk....kkk.
[    7.131854] Prev obj: start=ffff880035da52c8, len=64
[    7.131855] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
[    7.131857] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40)
[    7.131859] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[    7.131861] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[    7.131862] Next obj: start=ffff880035da5378, len=64
[    7.131863] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
[    7.131864] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40)
[    7.131866] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[    7.131868] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b  kkkkkkkkkkkkkkkk
[    7.135045] Modules linked in: cpufreq_conservative cpufreq_userspace snd_hda_codec_conexant cpufreq_powersave snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hwdep snd_pcm thinkpad_acpi kvm_intel snd_seq iTCO_wdt iTCO_vendor_support kvm iwldvm mac80211 snd_timer snd_seq_device btusb bluetooth iwlwifi sg cfg80211 e1000e snd ptp pcspkr lpc_ich i2c_i801 mfd_core rfkill pps_core ehci_pci wmi soundcore battery ac tpm_tis tpm acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm usbcore i2c_algo_bit usb_common button video edd fan processor ata_generic thermal thermal_sys
[    7.135045] CPU: 1 PID: 757 Comm: grep Tainted: G      D W    3.13.0-03478-gae75a37 #1
[    7.135045] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
[    7.135045] task: ffff8800362fddd0 ti: ffff880036d42000 task.ti: ffff880036d42000
[    7.135045] RIP: 0010:[<ffffffff810a51e7>]  [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
[    7.135045] RSP: 0018:ffff880036d43c68  EFLAGS: 00010282
[    7.135045] RAX: ffff8800362fddd0 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000
[    7.135045] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb
[    7.135045] RBP: ffff880036d43c88 R08: 0000000000000002 R09: 0000000000000000
[    7.135045] R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb
[    7.135045] R13: ffff880035d0db28 R14: 0000000000000020 R15: ffff880037900310
[    7.135045] FS:  0000000000000000(0000) GS:ffff88007c280000(0000) knlGS:0000000000000000
[    7.135045] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[    7.135045] CR2: 00007fff296c9f54 CR3: 0000000079067000 CR4: 00000000000007e0
[    7.135045] Stack:
[    7.135045]  6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880035d0db28
[    7.135045]  ffff880036d43ca8 ffffffff8159c5ac ffffffff812fe111 6b6b6b6b6b6b6beb
[    7.135045]  ffff880036d43cc8 ffffffff812fe111 ffff880036d43cf8 6b6b6b6b6b6b6b6b
[    7.135045] Call Trace:
[    7.135045]  [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50
[    7.135045]  [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40
[    7.135045]  [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40
[    7.135045]  [<ffffffff811b1442>] dput+0x22/0x130
[    7.135045]  [<ffffffff811a3d45>] path_put+0x15/0x30
[    7.135045]  [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40
[    7.135045]  [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[    7.135045]  [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390
[    7.135045]  [<ffffffff815a0c84>] ? __do_page_fault+0x2c4/0x480
[    7.135045]  [<ffffffff811dd18b>] send_to_group+0xfb/0x180
[    7.135045]  [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[    7.135045]  [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[    7.135045]  [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[    7.135045]  [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[    7.135045]  [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[    7.135045]  [<ffffffff81198859>] SyS_open+0x19/0x20
[    7.135045]  [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
[    7.135045] Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff 
[    7.135045] RIP  [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
[    7.135045]  RSP <ffff880036d43c68>
[    7.212496] ---[ end trace 5b4e9ae52ab9b0f7 ]---
[    7.214048] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20
[    7.214049] in_atomic(): 1, irqs_disabled(): 0, pid: 757, name: grep
[    7.214050] INFO: lockdep is turned off.
[    7.214051] CPU: 1 PID: 757 Comm: grep Tainted: G      D W    3.13.0-03478-gae75a37 #1
[    7.214052] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
[    7.214055]  ffff8800362fddd0 ffff880036d43a68 ffffffff8159703b ffff880036d43a88
[    7.214057]  ffffffff8107f621 ffffffff81a3d000 ffff880037946cd0 ffff880036d43aa8
[    7.214059]  ffffffff8159b50f ffff8800362fddd0 ffff8800362fddd0 ffff880036d43ad8
[    7.214060] Call Trace:
[    7.214071]  [<ffffffff8159703b>] dump_stack+0x72/0x87
[    7.214074]  [<ffffffff8107f621>] __might_sleep+0xe1/0x100
[    7.214076]  [<ffffffff8159b50f>] down_read+0x1f/0x60
[    7.214079]  [<ffffffff810627ff>] exit_signals+0x1f/0x140
[    7.214083]  [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20
[    7.214086]  [<ffffffff81052844>] do_exit+0xb4/0x4b0
[    7.214089]  [<ffffffff8159e23c>] oops_end+0xdc/0xe0
[    7.214092]  [<ffffffff81005f86>] die+0x56/0x90
[    7.214095]  [<ffffffff8159dea2>] do_general_protection+0x162/0x170
[    7.214097]  [<ffffffff8159d40c>] ? restore_args+0x30/0x30
[    7.214099]  [<ffffffff8159d592>] general_protection+0x22/0x30
[    7.214102]  [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
[    7.214104]  [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50
[    7.214107]  [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40
[    7.214109]  [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40
[    7.214113]  [<ffffffff811b1442>] dput+0x22/0x130
[    7.214115]  [<ffffffff811a3d45>] path_put+0x15/0x30
[    7.214117]  [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40
[    7.214119]  [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[    7.214121]  [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390
[    7.214124]  [<ffffffff815a0c84>] ? __do_page_fault+0x2c4/0x480
[    7.214127]  [<ffffffff811dd18b>] send_to_group+0xfb/0x180
[    7.214129]  [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[    7.214131]  [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[    7.214134]  [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[    7.214136]  [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[    7.214139]  [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[    7.214141]  [<ffffffff81198859>] SyS_open+0x19/0x20
[    7.214143]  [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
[    7.214145] note: grep[757] exited with preempt_count 1
Signature · 2 lines
Jiri Kosina
SUSE Labs