Process/Scanner hang with ecryptfs and fanotify
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hello,
I've continued to investigate this issue, and got back-traces from the stuck processes (checked into the git repository as dmesg.{1,2,3}.
From these it looks like the problem is an fanotify_read is itself calling into fanotify_handle_event via ecryptfs_get_lower_file:
[ 1197.922036] [<ffffffff8161366f>] schedule+0x3f/0x60 [ 1197.922042] [<ffffffff811b28b9>] fanotify_handle_event+0x119/0x160 [ 1197.922046] [<ffffffff81089180>] ? add_wait_queue+0x60/0x60 [ 1197.922050] [<ffffffff811aedda>] send_to_group.isra.1+0x13a/0x190 [ 1197.922055] [<ffffffff81183e80>] ? poll_freewait+0xe0/0xe0 [ 1197.922059] [<ffffffff811aef93>] fsnotify+0x163/0x2a0 [ 1197.922065] [<ffffffff81298229>] security_dentry_open+0x79/0x80 [ 1197.922068] [<ffffffff8116ed8c>] __dentry_open+0x10c/0x360 [ 1197.922073] [<ffffffff812d186c>] ? apparmor_file_alloc_security+0x2c/0x60 [ 1197.922076] [<ffffffff8116f58d>] vfs_open+0x3d/0x40 [ 1197.922080] [<ffffffff8116f5de>] dentry_open+0x4e/0x70 [ 1197.922084] [<ffffffff811b2900>] ? fanotify_handle_event+0x160/0x160 [ 1197.922089] [<ffffffff81271805>] ecryptfs_privileged_open+0x85/0x290 [ 1197.922093] [<ffffffff8126830d>] ecryptfs_get_lower_file+0xad/0x120 [ 1197.922097] [<ffffffff81265590>] ecryptfs_open+0xa0/0x2a0 [ 1197.922100] [<ffffffff8116ef10>] __dentry_open+0x290/0x360 [ 1197.922104] [<ffffffff812654f0>] ? ecryptfs_release+0x40/0x40 [ 1197.922108] [<ffffffff812d186c>] ? apparmor_file_alloc_security+0x2c/0x60 [ 1197.922111] [<ffffffff8116f58d>] vfs_open+0x3d/0x40 [ 1197.922115] [<ffffffff8116f5de>] dentry_open+0x4e/0x70 [ 1197.922118] [<ffffffff811b3246>] fanotify_read+0x2c6/0x510 [ 1197.922122] [<ffffffff81089180>] ? add_wait_queue+0x60/0x60 [ 1197.922126] [<ffffffff811713b0>] vfs_read+0xb0/0x180 [ 1197.922129] [<ffffffff811714ca>] sys_read+0x4a/0x90 [ 1197.922133] [<ffffffff810b3ee5>] ? compat_sys_time+0x25/0x60 [ 1197.922137] [<ffffffff8161fb70>] sysenter_dispatch+0x7/0x2e
However I have not been able to reproduce the problem once I applied the attached patch, which ensures that only one file access in included in each fanotify_read event. With this patch I was able to run firefox 5 times, through the 2 minute test cycle of the test code, without causing a hang.
What do you think?
-----Original Message-----
From: Douglas Leeder [mailto:[email protected]]
Sent: Thursday, August 09, 2012 2:33 PM
To: [email protected]; [email protected]
Cc: [email protected]; [email protected]; Eric Paris
Subject: Process/Scanner hang with ecryptfs and fanotify
Hello,
We are in the final stages of producing a version of Sophos Anti-Virus for Linux that uses fanotify, and have hit an issue with fanotify against ecryptfs mounts.
ecryptfs is used for encrypted home directories on (at least) Ubuntu.
This issue I'm seeing is on Ubuntu 12.04 with encrypted home directories. (3.2 kernel)
After using Firefox for a short while with fanotify scanning happening (for both the host fs, and the ecryptfs mount), it will go grey (or gray :-)), signalling it has hung.
Two threads in Firefox with be in the D state (IO sleep), and two of the scanner threads will be stuck reading from the fanotify fd. These remain stuck until the machine is rebooted.
I've got a test 'scanner' that I've managed to reproduce the issue with:
github:
https://github.com/paperclip/fanotify-ecryptfs-test
(If anyone wants this tarred up, I can send it directly - but I can't send it to the mailing list due to size limits)
However it doesn't happen every time, and I've not reproduced it on a VM.
Points on note:
1) Every file on ecryptfs is backed by a file on the host fs
2) Thus every access to an ecryptfs also generates a file access to the host fs.
3) fanotify can deliver multiple file accesses in a single event.
Given the unreliable nature of the reproduction steps; and the fact that when things hang it is always 2 threads, rather than 1 or 3 that hang: I believe that there is some kind of race condition when more than one file on an ecryptfs mount is accessed at the same time.
I wonder if each fanotify event contains the ecryptfs access from one client thread, and the host fs access from the other?
In the git repository there are two dmesg outputs, from the stuck threads' backtraces. Definitely stuck between fanotify and ecryptfs.
Is there any advice you can offer? I wonder if fanotify needs to deliver permission events one at a time? Or can you point me at appropriate sources to look at for further information.
In an ideal world there would be some fix to the scanner to avoid the problem, but I'm not sure what it could be, so the solution might only be applicable to future kernels.
(Sophos defect DEF83746)
Thanks.
--
Douglas Leeder, Senior Software Engineer
________________________________
Sophos Limited, The Pentagon, Abingdon Science Park, Abingdon, OX14 3YP, United Kingdom. Company Reg No 2096520. VAT Reg No GB 991 2418 08.
From 8ae7faf4560ae6bc48f575f029abe6e4a3f6a6ed Mon Sep 17 00:00:00 2001
From: Douglas Leeder <[email protected]>
Date: Mon, 13 Aug 2012 14:07:14 +0100
Subject: [PATCH] Make fanotify events only contain one file access if
permission checking is enabled.
fanotify scanning can get stuck if multiple events are included in a single read() call, with ecryptfs.
--- fs/notify/fanotify/fanotify_user.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-)
fs/notify/fanotify/fanotify_user.c +5 -1
@@ -343,7 +343,11 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,break ; buf += ret; count -= ret; -continue ; +#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS +break ;/* When doing permission events we should restrict to one event per-read */ +#else +continue ;/* When not using permissions we can combine events */ +#endif } ret = -EAGAIN ;
--
1.7.9.5