Skip to contentSkip to conversation

dnotify/inotify and vfs questions

DISCUSSION Asser Femø 3 participants 5 messages Aug 23, 2005 – Aug 25, 2005 VFS

lore
Share & tools
lore
#0 lore raw

Hi,

I'm currently implementing change notification support for the linux cifs client as part of Google's Summer of Code program.

In cifs, change notification works pretty much the same as dnotify does in the kernel, and you cancel the notification by sending a NT_CANCEL request.

According to the fcntl manual you can cancel a notification by doing fcntl(fd, F_NOTIFY, 0) (ie. sending 0 as the notification mask), but looking in the kernel code fcntl_dirnotify() immediately calls dnotify_flush() with neither telling the vfs module about it. Is there a reason for this? Otherwise I'd propose calling filp->f_op->dir_notify(filp, 0) at some point in this scenario.

Regarding inotify, inotify_add_watch doesn't seem to pass on the request either, which works fine for local filesystem operations as they call fsnotify_* functions every time, but that isn't really feasible for filesystems like cifs because we'd have to request change notification on everything. Is there plans for implementing a mechanism to let vfs modules get watch requests too?

cheers,
Asser

#1 lore raw
Quoted 7 lines · Asser Femø

According to the fcntl manual you can cancel a notification by doing fcntl(fd, F_NOTIFY, 0) (ie. sending 0 as the notification mask), but looking in the kernel code fcntl_dirnotify() immediately calls dnotify_flush() with neither telling the vfs module about it. Is there a reason for this? Otherwise I'd propose calling filp->f_op->dir_notify(filp, 0) at some point in this scenario.

Quoted 6 lines

Regarding inotify, inotify_add_watch doesn't seem to pass on the request either, which works fine for local filesystem operations as they call fsnotify_* functions every time, but that isn't really feasible for filesystems like cifs because we'd have to request change notification on everything. Is there plans for implementing a mechanism to let vfs modules get watch requests too?

On a related note:

dnotify and inotify on local filesystems appear to be synchronous, in the following rather useful sense:

If you have previously registered for inotify/dnotify events that will catch a change to a file, and called stat() on the file, then the following operation:

<receive some request>...
stat_info = stat(file)

may be replaced in userspace code with:

    <receive some request>...
    if (any_dnotify_or_inotify_events_pending) {
        read_dnotify_or_inotify_events();
        if (any_events_related_to(file)) {
            store_in_userspace_stat_cache(file, stat(file));
        }
    }
    stat_info = lookup_userspace_stat_cache(file);

Now that's a silly way to save one system call in the fast path by itself.

But when the stat_info is a prerequisite for validating cached data -- such as the contents of a file parsed into a data structure -- it can save a lot of system calls and logical work.

For example, an Apache-style path walk which checks for .htaccess, or a Samba-style path walk which is checking for unsafe symbolic links, can be reduced from say 20 system calls to zero using this method.

Pre-compiled or pre-parsed programs/scripts/templates/config-files where all the source files used are prerequisites for invalidating a cached compiled form, reduces from say 40 system calls to stat() all the source files, to zero.... that's quite a saving.

It's not just reducing system calls. The logical tests in userspace are also skipped, if coded properly, facilitating very quick decisions about things that depend on files which mostly don't change. (Cascading structured cache prerequisites...mmm).

Remote dnotify/inotify doesn't _necessarily_ have this synchronous property. It may do in some cases, depending on the implementation (this is subtle...).

So, it would be nice if there was a way to query this... rather than the tedious method of testing the filesystem type and having a table of "known local filesystem types" where it's safe to depend on this property. Alternatively, a way to specify at dnotify/inotify creation type that synchronous notifications are required, and have the request rejected if those can't be provided.

-- Jamie

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#2 lore raw
Quoted 10 lines · Jamie Lokier
    <receive some request>...
    if (any_dnotify_or_inotify_events_pending) {
        read_dnotify_or_inotify_events();
        if (any_events_related_to(file)) {
            store_in_userspace_stat_cache(file, stat(file));
        }
    }
    stat_info = lookup_userspace_stat_cache(file);

Now that's a silly way to save one system call in the fast path by itself.

I'm not that familiar with inotify internals but doesn't read_dnotify_or_inotify_events() or any_dnotify_or_inotify_events_pending() involve a syscall?

Ian.

Signature · 4 lines
Ian Campbell
Current Noise: Primordial - The Coffin Ships

Today is the first day of the rest of the mess.
#3 lore raw
Quoted 10 lines · Ian Campbell
Quoted 9 lines · Jamie Lokier
    <receive some request>...
    if (any_dnotify_or_inotify_events_pending) {
        read_dnotify_or_inotify_events();
        if (any_events_related_to(file)) {
            store_in_userspace_stat_cache(file, stat(file));
        }
    }
    stat_info = lookup_userspace_stat_cache(file);
Quoted 5 lines
Quoted 1 line

Now that's a silly way to save one system call in the fast path by itself.

I'm not that familiar with inotify internals but doesn't read_dnotify_or_inotify_events() or any_dnotify_or_inotify_events_pending() involve a syscall?

The fast path is just any_dnotify_or_inotify_events_pending: there aren't any relevant events pending in the fast path.

There's a few methods of doing this for free per individual stat cache check.

1. Signal handler.

dnotify: Check a variable set by the signal handler.

[ There's a small time window between dnotify sending the signal and the receiving thread noticing on an SMP system due to the IPI, during which the sending task might have another way to signal the recieving task that it's finished some operation, so this method of using dnotify to invalidate a stat cache only has the correct ordering properties on UP systems. ]

This works because dnotify signals are thread-specific, so the checking thread will definitely have received the signal after the time another process modifies the file.

inotify: Disappointingly inotify doesn't support SIGIO readiness :(

2. If you have to mix the test with a poll/select/epoll/rtsig fd waiting for some other purpose. For example: a file/web/local server, where the constraint is only that each stat() to revalidate a cached response appears to happen any time after the beginning of receiving the network request is known.

dnotify: It's free if you were using sigtimedwait anyway for I/O events,
provided you completely read the queue, or get the signal priority right.

inotify: It's free if you were using poll/select/epoll anyway for I/O
events, provided in the case of epoll that you completely read the queue, or use a two-level queue.

3. Amortising the test over many stat cache checks.

Even if you must use a system call to check for any pending events, for revalidating an object which depends on multiple files, only one call is needed for all of the stat cache checks.

More generally (this is more flexible), you can separate the notion of "cache time checkpoint" from "cache validation". It's enough to know that a stat result was valid any time between the checkpoint time, and the current time. That's how I'm implementing the file/web/local server case described above in step 2. Then the events only need to be checked once during that time interval, no matter how many complex objects are being revalidated.

It gets slightly more efficient when you have multiple, overlapping checkpoint->validation time intervals due to multiple outstanding requests being processed concurrently.

As I explained in the previous mail, all this is absolutely pointless to save one system call. It's a lot of work for negligable gain.

The point is when it saves lots of calls and userspace logic together, for things like web page templates and compiled programs, which depend on many files which can be revalidated in a small number of operations.

-- Jamie

#4 lore raw
Quoted 19 lines · Jamie Lokier
Quoted 10 lines · Ian Campbell
Quoted 9 lines · Jamie Lokier
    <receive some request>...
    if (any_dnotify_or_inotify_events_pending) {
        read_dnotify_or_inotify_events();
        if (any_events_related_to(file)) {
            store_in_userspace_stat_cache(file, stat(file));
        }
    }
    stat_info = lookup_userspace_stat_cache(file);
Quoted 5 lines
Quoted 1 line

Now that's a silly way to save one system call in the fast path by itself.

I'm not that familiar with inotify internals but doesn't read_dnotify_or_inotify_events() or any_dnotify_or_inotify_events_pending() involve a syscall?

The fast path is just any_dnotify_or_inotify_events_pending: there aren't any relevant events pending in the fast path.

[snip]

Quoted 6 lines

As I explained in the previous mail, all this is absolutely pointless to save one system call. It's a lot of work for negligable gain.

The point is when it saves lots of calls and userspace logic together, for things like web page templates and compiled programs, which depend on many files which can be revalidated in a small number of operations.

Thanks for the explaination.

Ian.

Signature · 6 lines
Ian Campbell
Current Noise: Iron Maiden - Prodigal Son

Fay: The British police force used to be run by men of integrity.
Truscott: That is a mistake which has been rectified.
		-- Joe Orton, "Loot"