Reproducible, long-standing fanotify+autofs problem
We have run into an easily reproducible fanotify hang that affects several distros as well as the latest development kernel. (It isn't fixed by Amir Goldstein's recent fanotify patch, BTW.)
Here is a simple reproduction:
1. Compile and build the tiny test program below, which has been adapted from the example in "man fanotify":
cc -o fantest3 fantest.c
2. Install autofs.
3. Run these commands as root:
mkdir -p foo/home/test foo/user/test
echo /sshome /etc/auto.sshome >/etc/auto.master
echo "test / :/root/foo/home/test /user :/root/foo/user/test" \
>/etc/auto.sshome
systemctl start autofs
ls -l /sshome/test
./fantest3 / /sshome/test /sshome/test/user/
# NB! The final slash in the fantest3 command can be significant!
4. Observe the last command hang. While it is hung, other file system operations from other processes are blocked.
5. Press Ctrl-C to force fantest3 to exit. The file system becomes alive. Rerunning the fantest3 command no longer hangs.
6. To get back into the problem state, execute (as root):
systemctl restart autofs
ls -l /sshome/test
./fantest3 / /sshome/test /sshome/test/user/
===Begin fantest.c====================================================== #include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/fanotify.h>
int main(int argc, char *argv[])
{
int fd, i;
if (argc < 2) {
fprintf(stderr, "Usage: %s MOUNT ...\n", argv[0]);
exit(EXIT_FAILURE);
}
fd = fanotify_init(FAN_CLOEXEC | FAN_CLASS_CONTENT | FAN_NONBLOCK,
O_RDONLY);
if (fd == -1) {
perror("fanotify_init");
exit(EXIT_FAILURE);
}
for (i = 1; i < argc; i++)
if (fanotify_mark(fd, FAN_MARK_ADD | FAN_MARK_MOUNT,
FAN_OPEN_PERM | FAN_CLOSE_WRITE, AT_FDCWD,
argv[i]) == -1) {
perror("fanotify_mark");
exit(EXIT_FAILURE);
}
exit(EXIT_SUCCESS);
}===End fantest.c========================================================
Marko
Quoted 5 lines · Marko Rauhamaa
We have run into an easily reproducible fanotify hang that affects several distros as well as the latest development kernel. (It isn't fixed by Amir Goldstein's recent fanotify patch, BTW.)
Not sure what is the role that autofs is playing in this hang, but it is worth checking if Jan's latest work fixes the problem. I estimate that it should, because it isolated the user space handling of permission events from affecting processes generating fsnotify events on other fs objects.
git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git #for_testing
Quoted 19 lines
Here is a simple reproduction:
1. Compile and build the tiny test program below, which has been adapted from the example in "man fanotify":
cc -o fantest3
fantest.c2. Install autofs.
3. Run these commands as root:
mkdir -p foo/home/test foo/user/test
echo /sshome/etc/auto.sshome>/etc/auto.master
echo "test / :/root/foo/home/test /user :/root/foo/user/test" \
>/etc/auto.sshome
systemctl start autofs
ls -l /sshome/test
./fantest3 / /sshome/test /sshome/test/user/
Quoted 4 lines
# NB! The final slash in the fantest3 command can be significant!
4. Observe the last command hang. While it is hung, other file system operations from other processes are blocked.
You mean other processes not trying to access objects under the watched mounts. right?
Quoted 43 lines
5. Press Ctrl-C to force fantest3 to exit. The file system becomes alive. Rerunning the fantest3 command no longer hangs.
6. To get back into the problem state, execute (as root):
systemctl restart autofs
ls -l /sshome/test
./fantest3 / /sshome/test /sshome/test/user/===Begin fantest.c====================================================== #include <
fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/fanotify.h>int
main(int argc, char *argv[]){ int fd, i; if (argc < 2) { fprintf(stderr, "Usage: %s MOUNT ...\n", argv[0]); exit(EXIT_FAILURE); } fd = fanotify_init(FAN_CLOEXEC | FAN_CLASS_CONTENT | FAN_NONBLOCK, O_RDONLY); if (fd == -1) { perror("fanotify_init"); exit(EXIT_FAILURE); } for (i = 1; i < argc; i++) if (fanotify_mark(fd, FAN_MARK_ADD | FAN_MARK_MOUNT, FAN_OPEN_PERM | FAN_CLOSE_WRITE, AT_FDCWD, argv[i]) == -1) { perror("fanotify_mark"); exit(EXIT_FAILURE); } exit(EXIT_SUCCESS); }===End fantest.c========================================================
Marko
Amir Goldstein <[email protected]>:
Quoted 6 lines
Quoted 3 lines · Marko Rauhamaa
We have run into an easily reproducible fanotify hang that affects several distros as well as the latest development kernel. (It isn't fixed by Amir Goldstein's recent fanotify patch, BTW.)
Quoted 7 lines
Not sure what is the role that autofs is playing in this hang, but it is worth checking if Jan's latest work fixes the problem. I estimate that it should, because it isolated the user space handling of permission events from affecting processes generating fsnotify events on other fs objects.
git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git#for_testing
I could try.
Quoted 5 lines
Quoted 2 lines
4. Observe the last command hang. While it is hung, other file system operations from other processes are blocked.
You mean other processes not trying to access objects under the watched mounts. right?
No, accessing the watched mounts. Don't know if the hang is limited to them.
Marko
Quoted 18 lines · Marko Rauhamaa
Amir Goldstein <[email protected]>:
Quoted 6 lines
Quoted 3 lines · Marko Rauhamaa
We have run into an easily reproducible fanotify hang that affects several distros as well as the latest development kernel. (It isn't fixed by Amir Goldstein's recent fanotify patch, BTW.)
Quoted 7 lines
Not sure what is the role that autofs is playing in this hang, but it is worth checking if Jan's latest work fixes the problem. I estimate that it should, because it isolated the user space handling of permission events from affecting processes generating fsnotify events on other fs objects.
git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git#for_testingI could try.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
4. Observe the last command hang. While it is hung, other file system operations from other processes are blocked.
You mean other processes not trying to access objects under the watched mounts. right?
No, accessing the watched mounts. Don't know if the hang is limited to them.
So I am confused.
Your test program sets a watch on permission events on the mount and does not respond to permission events on the mount, so all file system operations on the mount SHOULD be blocked. What am I missing?
Amir Goldstein <[email protected]>:
Quoted 8 lines
Quoted 1 line · Marko Rauhamaa
Amir Goldstein <[email protected]>:
So I am confused.
Your test program sets a watch on permission events on the mount and does not respond to permission events on the mount, so all file system operations on the mount SHOULD be blocked. What am I missing?
The problem is that the fanotify_mark() system call is hanging and as a side effect blocks everything.
The side effect is probably completely understandable, but fanotify_mark() should never hang, should it?
Marko
Signature · 2 lines
+358 44 990 4795 Skype: marko.rauhamaa_f-secure
Quoted 11 lines · Marko Rauhamaa
Amir Goldstein <[email protected]>:
Quoted 8 lines
Quoted 1 line · Marko Rauhamaa
Amir Goldstein <[email protected]>:
So I am confused.
Your test program sets a watch on permission events on the mount and does not respond to permission events on the mount, so all file system operations on the mount SHOULD be blocked. What am I missing?
Quoted 7 lines
The problem is that the
fanotify_mark()system call is hanging and as a side effect blocks everything.The side effect is probably completely understandable, but
fanotify_mark()should never hang, should it?
I see. I guess it shouldn't block.
Also with this understanding, I am not sure Jan's work is relevant to your issue.
Jan Kara
reviewerQuoted 16 lines · Amir Goldstein
Quoted 14 lines · Marko Rauhamaa
Amir Goldstein <[email protected]>:
Quoted 8 lines
Quoted 1 line · Marko Rauhamaa
Amir Goldstein <[email protected]>:
So I am confused.
Your test program sets a watch on permission events on the mount and does not respond to permission events on the mount, so all file system operations on the mount SHOULD be blocked. What am I missing?The problem is that the
fanotify_mark()system call is hanging and as a side effect blocks everything.
Quoted 8 lines
Quoted 4 lines
The side effect is probably completely understandable, but
fanotify_mark()should never hang, should it?I see. I guess it shouldn't block.
Also with this understanding, I am not sure Jan's work is relevant to your issue.
Yeah, it should not, although I expect this is some weird interaction with autofs which tries to open something, which generates fanotify open event for already created mark and thus following mark creation gets hung. Can you reproduce the hang and send contents of dmesg after you do 'echo t >/proc/sysrq-trigger'? Thanks.
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Amir Goldstein <[email protected]>:
Quoted 2 lines
I see. I guess it shouldn't block. Also with this understanding, I am not sure Jan's work is relevant to your issue.
I have narrowed the problem down a bit in the kernel. The hanging function is "user_path_at":
user_path_at(filename="/sshome/test/user/", lookup_flags=0x1)
called from: fanotify_find_path()
called from: fanotify_mark()
And Ctrl-C wakes it up.
Marko
Marko Rauhamaa <[email protected]>:
Quoted 9 lines
Amir Goldstein <[email protected]>:
I have narrowed the problem down a bit in the kernel. The hanging function is "user_path_at":
user_path_at(filename="/sshome/test/user/", lookup_flags=0x1)
called from:fanotify_find_path()
called from:fanotify_mark()And Ctrl-C wakes it up.
I have dug a bit deeper. What hangs is this statement:
mnt = path->dentry->d_op->d_automount(path);
in:
follow_automount() // fs/namei.c
called from: follow_managed()
called from: walk_component()
called from: lookup_last()
called from: path_lookupat()
called from: filename_lookup()
called from: user_path_at_empty()
called from: user_path_at(filename="/sshome/test/user/", lookup_flags=0x1) called from: fanotify_find_path() // fs/notify/fanotify_user.c called from: fanotify_mark()
And Ctrl-C wakes it up.
The d_automount() method is bound to autofs4_d_automount() in fs/autofs4/root.c. The hanging statement is:
wait_event_interruptible(wq->queue, wq->name.name == NULL);
in:
autofs4_wait() // fs/autofs4/waitq.c
called from: autofs4_mount_wait() // fs/autofs4/root.c called from: autofs4_d_automount()
Marko
Signature · 2 lines
+358 44 990 4795 Skype: marko.rauhamaa_f-secure
Marko Rauhamaa <[email protected]>:
Quoted 8 lines
I have dug a bit deeper. What hangs is this statement:
mnt = path->dentry->d_op->d_automount(path);in:
follow_automount()//fs/namei.c
called from:follow_managed()
called from: lookup_fast()
Quoted 14 lines
called from:
walk_component()
called from:lookup_last()
called from:path_lookupat()
called from:filename_lookup()
called from:user_path_at_empty()
called from:user_path_at(filename="/sshome/test/user/", lookup_flags=0x1)called from:fanotify_find_path()//fs/notify/fanotify_user.ccalled from:fanotify_mark()And Ctrl-C wakes it up.
The
d_automount()method is bound toautofs4_d_automount()infs/autofs4/root.c. The hanging statement is:
Quoted 7 lines
wait_event_interruptible(wq->queue, wq->name.name == NULL);in:
autofs4_wait()//fs/autofs4/waitq.c
called from:autofs4_mount_wait()//fs/autofs4/root.ccalled from:autofs4_d_automount()
Added a missing call to the call trace.
Marko
Jan Kara <[email protected]>:
Quoted 9 lines
Yes, so from the backtraces it is pretty clear what is going on:
You have placed
OPEN_PERMfanotify mark on a / mountpoint. Once you do that, any open attempt anywhere in / must be approved by your process. Then you want to place an fanotify mark on /sshome/test. As a part of directory lookup of that path automounter triggers and wants to mount /sshome. That tries to open something in / (likely the config file in /etc) and blocks on approval from your process. Deadlock.
Quoted 6 lines
So the kernel behaves as it was asked to, you just have to be *very* careful when placing permission marks into the system as it is very easy to deadlock it. Another similar type of deadlocks users of fanotify permission events hit is that the process responding to fanotify permission events does something (perhaps indirectly through some library) that generates another fanotify event.
That means it is not possible, in the general case, to use fanotify from a single thread/process. A worthwhile lesson.
Thanks, Jan, for the clarification. Now we all know that fanotify_mark() *can* block. The fanotify_mark() man page could benefit from the addition of EINTR in the ERRORS section.
Marko