Skip to contentSkip to conversation

Reproducible, long-standing fanotify+autofs problem

DISCUSSION Marko Rauhamaa 3 participants 11 messages Jan 25, 2017 – Jan 27, 2017 2 messages not in the archive VFS

lore
Share & tools
lore
#0 lore raw

We have run into an easily reproducible fanotify hang that affects several distros as well as the latest development kernel. (It isn't fixed by Amir Goldstein's recent fanotify patch, BTW.)

Here is a simple reproduction:

1. Compile and build the tiny test program below, which has been adapted from the example in "man fanotify":

cc -o fantest3 fantest.c

2. Install autofs.

3. Run these commands as root:

mkdir -p foo/home/test foo/user/test
echo /sshome /etc/auto.sshome >/etc/auto.master
echo "test / :/root/foo/home/test /user :/root/foo/user/test" \
>/etc/auto.sshome
systemctl start autofs
ls -l /sshome/test
./fantest3 / /sshome/test /sshome/test/user/

# NB! The final slash in the fantest3 command can be significant!

4. Observe the last command hang. While it is hung, other file system operations from other processes are blocked.

5. Press Ctrl-C to force fantest3 to exit. The file system becomes alive. Rerunning the fantest3 command no longer hangs.

6. To get back into the problem state, execute (as root):

systemctl restart autofs
ls -l /sshome/test
./fantest3 / /sshome/test /sshome/test/user/

===Begin fantest.c====================================================== #include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/fanotify.h>

int main(int argc, char *argv[])

{
    int fd, i;
    if (argc < 2) {
        fprintf(stderr, "Usage: %s MOUNT ...\n", argv[0]);
        exit(EXIT_FAILURE);
    }
    fd = fanotify_init(FAN_CLOEXEC | FAN_CLASS_CONTENT | FAN_NONBLOCK,
                       O_RDONLY);
    if (fd == -1) {
        perror("fanotify_init");
        exit(EXIT_FAILURE);
    }
    for (i = 1; i < argc; i++)
        if (fanotify_mark(fd, FAN_MARK_ADD | FAN_MARK_MOUNT,
                          FAN_OPEN_PERM | FAN_CLOSE_WRITE, AT_FDCWD,
                          argv[i]) == -1) {
            perror("fanotify_mark");
            exit(EXIT_FAILURE);
        }
    exit(EXIT_SUCCESS);
}

===End fantest.c========================================================

Marko

#1 lore raw
Quoted 5 lines · Marko Rauhamaa

We have run into an easily reproducible fanotify hang that affects several distros as well as the latest development kernel. (It isn't fixed by Amir Goldstein's recent fanotify patch, BTW.)

Not sure what is the role that autofs is playing in this hang, but it is worth checking if Jan's latest work fixes the problem. I estimate that it should, because it isolated the user space handling of permission events from affecting processes generating fsnotify events on other fs objects.

git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git #for_testing

Quoted 19 lines

Here is a simple reproduction:

1. Compile and build the tiny test program below, which has been adapted from the example in "man fanotify":

cc -o fantest3 fantest.c

2. Install autofs.

3. Run these commands as root:

mkdir -p foo/home/test foo/user/test
echo /sshome /etc/auto.sshome >/etc/auto.master
echo "test / :/root/foo/home/test /user :/root/foo/user/test" \
>/etc/auto.sshome
systemctl start autofs
ls -l /sshome/test
./fantest3 / /sshome/test /sshome/test/user/

Quoted 4 lines

# NB! The final slash in the fantest3 command can be significant!

4. Observe the last command hang. While it is hung, other file system operations from other processes are blocked.

You mean other processes not trying to access objects under the watched mounts. right?

Quoted 43 lines

5. Press Ctrl-C to force fantest3 to exit. The file system becomes alive. Rerunning the fantest3 command no longer hangs.

6. To get back into the problem state, execute (as root):

systemctl restart autofs
ls -l /sshome/test
./fantest3 / /sshome/test /sshome/test/user/

===Begin fantest.c====================================================== #include <fcntl.h>
#include <stdio.h>
#include <stdlib.h>
#include <sys/fanotify.h>

int main(int argc, char *argv[])

{
    int fd, i;
    if (argc < 2) {
        fprintf(stderr, "Usage: %s MOUNT ...\n", argv[0]);
        exit(EXIT_FAILURE);
    }
    fd = fanotify_init(FAN_CLOEXEC | FAN_CLASS_CONTENT | FAN_NONBLOCK,
                       O_RDONLY);
    if (fd == -1) {
        perror("fanotify_init");
        exit(EXIT_FAILURE);
    }
    for (i = 1; i < argc; i++)
        if (fanotify_mark(fd, FAN_MARK_ADD | FAN_MARK_MOUNT,
                          FAN_OPEN_PERM | FAN_CLOSE_WRITE, AT_FDCWD,
                          argv[i]) == -1) {
            perror("fanotify_mark");
            exit(EXIT_FAILURE);
        }
    exit(EXIT_SUCCESS);
}

===End fantest.c========================================================

Marko

#2 lore raw

Amir Goldstein <[email protected]>:

Quoted 6 lines
Quoted 3 lines · Marko Rauhamaa

We have run into an easily reproducible fanotify hang that affects several distros as well as the latest development kernel. (It isn't fixed by Amir Goldstein's recent fanotify patch, BTW.)

Quoted 7 lines

Not sure what is the role that autofs is playing in this hang, but it is worth checking if Jan's latest work fixes the problem. I estimate that it should, because it isolated the user space handling of permission events from affecting processes generating fsnotify events on other fs objects.

git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git #for_testing

I could try.

Quoted 5 lines
Quoted 2 lines

4. Observe the last command hang. While it is hung, other file system operations from other processes are blocked.

You mean other processes not trying to access objects under the watched mounts. right?

No, accessing the watched mounts. Don't know if the hang is limited to them.

Marko

#3 lore raw
Quoted 18 lines · Marko Rauhamaa

Amir Goldstein <[email protected]>:

Quoted 6 lines
Quoted 3 lines · Marko Rauhamaa

We have run into an easily reproducible fanotify hang that affects several distros as well as the latest development kernel. (It isn't fixed by Amir Goldstein's recent fanotify patch, BTW.)

Quoted 7 lines

Not sure what is the role that autofs is playing in this hang, but it is worth checking if Jan's latest work fixes the problem. I estimate that it should, because it isolated the user space handling of permission events from affecting processes generating fsnotify events on other fs objects.

git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs.git #for_testing

I could try.

Quoted 8 lines
Quoted 5 lines
Quoted 2 lines

4. Observe the last command hang. While it is hung, other file system operations from other processes are blocked.

You mean other processes not trying to access objects under the watched mounts. right?

No, accessing the watched mounts. Don't know if the hang is limited to them.

So I am confused.
Your test program sets a watch on permission events on the mount and does not respond to permission events on the mount, so all file system operations on the mount SHOULD be blocked. What am I missing?

#4 lore raw

Amir Goldstein <[email protected]>:

Quoted 8 lines
Quoted 1 line · Marko Rauhamaa

Amir Goldstein <[email protected]>:

So I am confused.
Your test program sets a watch on permission events on the mount and does not respond to permission events on the mount, so all file system operations on the mount SHOULD be blocked. What am I missing?

The problem is that the fanotify_mark() system call is hanging and as a side effect blocks everything.

The side effect is probably completely understandable, but fanotify_mark() should never hang, should it?

Marko

Signature · 2 lines
+358 44 990 4795
Skype: marko.rauhamaa_f-secure
#5 lore raw
Quoted 11 lines · Marko Rauhamaa

Amir Goldstein <[email protected]>:

Quoted 8 lines
Quoted 1 line · Marko Rauhamaa

Amir Goldstein <[email protected]>:

So I am confused.
Your test program sets a watch on permission events on the mount and does not respond to permission events on the mount, so all file system operations on the mount SHOULD be blocked. What am I missing?

Quoted 7 lines

The problem is that the fanotify_mark() system call is hanging and as a side effect blocks everything.

The side effect is probably completely understandable, but fanotify_mark() should never hang, should it?

I see. I guess it shouldn't block.
Also with this understanding, I am not sure Jan's work is relevant to your issue.

reviewer
#6 lore raw
Quoted 16 lines · Amir Goldstein
Quoted 14 lines · Marko Rauhamaa

Amir Goldstein <[email protected]>:

Quoted 8 lines
Quoted 1 line · Marko Rauhamaa

Amir Goldstein <[email protected]>:

So I am confused.
Your test program sets a watch on permission events on the mount and does not respond to permission events on the mount, so all file system operations on the mount SHOULD be blocked. What am I missing?

The problem is that the fanotify_mark() system call is hanging and as a side effect blocks everything.

Quoted 8 lines
Quoted 4 lines

The side effect is probably completely understandable, but fanotify_mark() should never hang, should it?

I see. I guess it shouldn't block.
Also with this understanding, I am not sure Jan's work is relevant to your issue.

Yeah, it should not, although I expect this is some weird interaction with autofs which tries to open something, which generates fanotify open event for already created mark and thus following mark creation gets hung. Can you reproduce the hang and send contents of dmesg after you do 'echo t >/proc/sysrq-trigger'? Thanks.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#10 lore raw
In reply to Amir Goldstein

Amir Goldstein <[email protected]>:

Quoted 2 lines

I see. I guess it shouldn't block. Also with this understanding, I am not sure Jan's work is relevant to your issue.

I have narrowed the problem down a bit in the kernel. The hanging function is "user_path_at":

user_path_at(filename="/sshome/test/user/", lookup_flags=0x1)
called from: fanotify_find_path()
called from: fanotify_mark()

And Ctrl-C wakes it up.

Marko

#11 lore raw

Marko Rauhamaa <[email protected]>:

Quoted 9 lines

Amir Goldstein <[email protected]>:
I have narrowed the problem down a bit in the kernel. The hanging function is "user_path_at":

user_path_at(filename="/sshome/test/user/", lookup_flags=0x1)
called from: fanotify_find_path()
called from: fanotify_mark()

And Ctrl-C wakes it up.

I have dug a bit deeper. What hangs is this statement:

    mnt = path->dentry->d_op->d_automount(path);

in:

follow_automount() // fs/namei.c
called from: follow_managed()
called from: walk_component()
called from: lookup_last()
called from: path_lookupat()
called from: filename_lookup()
called from: user_path_at_empty()
called from: user_path_at(filename="/sshome/test/user/", lookup_flags=0x1) called from: fanotify_find_path() // fs/notify/fanotify_user.c called from: fanotify_mark()

And Ctrl-C wakes it up.

The d_automount() method is bound to autofs4_d_automount() in fs/autofs4/root.c. The hanging statement is:

     wait_event_interruptible(wq->queue, wq->name.name == NULL);

in:

autofs4_wait() // fs/autofs4/waitq.c
called from: autofs4_mount_wait() // fs/autofs4/root.c called from: autofs4_d_automount()

Marko

Signature · 2 lines
+358 44 990 4795
Skype: marko.rauhamaa_f-secure
#12 lore raw

Marko Rauhamaa <[email protected]>:

Quoted 8 lines

I have dug a bit deeper. What hangs is this statement:

    mnt = path->dentry->d_op->d_automount(path);

in:

follow_automount() // fs/namei.c
called from: follow_managed()

called from: lookup_fast()

Quoted 14 lines

called from: walk_component()
called from: lookup_last()
called from: path_lookupat()
called from: filename_lookup()
called from: user_path_at_empty()
called from: user_path_at(filename="/sshome/test/user/", lookup_flags=0x1) called from: fanotify_find_path() // fs/notify/fanotify_user.c called from: fanotify_mark()

And Ctrl-C wakes it up.

The d_automount() method is bound to autofs4_d_automount() in fs/autofs4/root.c. The hanging statement is:

Quoted 7 lines
     wait_event_interruptible(wq->queue, wq->name.name == NULL);

in:

autofs4_wait() // fs/autofs4/waitq.c
called from: autofs4_mount_wait() // fs/autofs4/root.c called from: autofs4_d_automount()

Added a missing call to the call trace.

Marko

#9 lore raw
In reply to a message not in the archive

Jan Kara <[email protected]>:

Quoted 9 lines

Yes, so from the backtraces it is pretty clear what is going on:

You have placed OPEN_PERM fanotify mark on a / mountpoint. Once you do that, any open attempt anywhere in / must be approved by your process. Then you want to place an fanotify mark on /sshome/test. As a part of directory lookup of that path automounter triggers and wants to mount /sshome. That tries to open something in / (likely the config file in /etc) and blocks on approval from your process. Deadlock.

Quoted 6 lines

So the kernel behaves as it was asked to, you just have to be *very* careful when placing permission marks into the system as it is very easy to deadlock it. Another similar type of deadlocks users of fanotify permission events hit is that the process responding to fanotify permission events does something (perhaps indirectly through some library) that generates another fanotify event.

That means it is not possible, in the general case, to use fanotify from a single thread/process. A worthwhile lesson.

Thanks, Jan, for the clarification. Now we all know that fanotify_mark() *can* block. The fanotify_mark() man page could benefit from the addition of EINTR in the ERRORS section.

Marko