inotify limits - thousands (tens of thousands?) of watches
Hello everyone,
First, apols for using up bandwidth, but I honestly found no other place where I can ask about this (and get meaningful reply).
I'm not a kernel programmer, but I want to develop a program that would watch modifications in *all* user directories on a busy server using inotify.
This is for high-availability purposes - events would be collected and once every several minutes changed dirs would be rsync'ed to failover server or smth like that would be done.
As inotify watches particular directory and not its subdirs, I would have to watch all directories.
This means I would have to create thousands or even tens of thousands of inotify watches.
So my questions are:
1. is it safe? that is, will it not lock the kernel up, or cause excessive memory consumption?
2. is it economic in terms of CPU time and RAM? I have no idea how to even measure such a thing happening in the kernel..
Here's first test take on inotify watch (runs on Debian):
######################### cut ################################
#include "stdio.h"
#include "stdlib.h"
#include "errno.h"
#include <linux/types.h>
#include <linux/inotify.h>
#include <sys/syscall.h>
#define EVENT_SIZE ( sizeof (struct inotify_event) )
#define BUF_LEN ( 1024 * ( EVENT_SIZE + 16 ) )
static inline int inotify_init (void)
{
return syscall (__NR_inotify_init);
}static inline int inotify_add_watch (int fd, const char *name, __u32 mask)
{
return syscall (__NR_inotify_add_watch, fd, name, mask);
}static inline int inotify_rm_watch (int fd, __u32 wd)
{
return syscall (__NR_inotify_rm_watch, fd, wd);
}int main( int argc, char **argv )
{
int length, i = 0;
int fd;
int wd;
char buffer[BUF_LEN];
fd = inotify_init();
if ( fd < 0 ) {
perror( "inotify_init" );
}
wd = inotify_add_watch( fd, argv[1], IN_ALL_EVENTS );
length = read( fd, buffer, BUF_LEN );
if ( length < 0 ) {
perror( "read" );
} while ( i < length ) {
struct inotify_event *event = ( struct inotify_event * ) &buffer[ i ];
if ( event->len ) {
if ( event->mask & IN_CREATE ) {
if ( event->mask & IN_ISDIR ) {
printf( "The directory %s was created.\n", event->name );
}
else {
printf( "The file %s was created.\n", event->name );
}
}
else if ( event->mask & IN_DELETE ) {
if ( event->mask & IN_ISDIR ) {
printf( "The directory %s was deleted.\n", event->name );
}
else {
printf( "The file %s was deleted.\n", event->name );
}
}
else if ( event->mask & IN_MODIFY ) {
if ( event->mask & IN_ISDIR ) {
printf( "The directory %s was modified.\n", event->name );
}
else {
printf( "The file %s was modified.\n", event->name );
}
}
else if ( event->mask & IN_ALL_EVENTS ) {
printf("Some other event on %s, mask %x.\n", event->name,
event->mask);
}
}
i += EVENT_SIZE + event->len;}
( void ) inotify_rm_watch( fd, wd ); ( void ) close( fd );
exit( 0 );
}
######################### cut ################################
Regards,
mk
El Wed, May 20, 2009 at 11:26:57AM +0200 Marcin Krol ha dit:
Quoted 17 lines
First, apols for using up bandwidth, but I honestly found no other place where I can ask about this (and get meaningful reply).
I'm not a kernel programmer, but I want to develop a program that would watch modifications in *all* user directories on a busy server using inotify.
This is for high-availability purposes - events would be collected and once every several minutes changed dirs would be rsync'ed to failover server or smth like that would be done.
As inotify watches particular directory and not its subdirs, I would have to watch all directories.
This means I would have to create thousands or even tens of thousands of inotify watches.
Quoted 7 lines
So my questions are:
1. is it safe? that is, will it not lock the kernel up, or cause excessive memory consumption?
2. is it economic in terms of CPU time and RAM? I have no idea how to even measure such a thing happening in the kernel..
i can't answer your question whether inotify scales or not, but maybe DRBD (http://en.wikipedia.org/wiki/DRBD) could be an alternative approach for your problem
Signature · 10 lines
Matthias Kaehlcke
Embedded Linux Engineer
Barcelona
Me lo contaron y lo olvidé, lo vi y lo entendí, lo hice y lo aprendí
(Confucio)
.''`.
using free software / Debian GNU/Linux | http://debian.org : :' :
`. `'`
gpg --keyserver pgp.mit.edu --recv-keys 47D8E5D4 `-Quoted 19 lines · On Wednesday 20 May 2009 10:26:57 Marcin Krol wrote:
Hello everyone,
First, apols for using up bandwidth, but I honestly found no other place where I can ask about this (and get meaningful reply).
I'm not a kernel programmer, but I want to develop a program that would watch modifications in *all* user directories on a busy server using inotify.
This is for high-availability purposes - events would be collected and once every several minutes changed dirs would be rsync'ed to failover server or smth like that would be done.
As inotify watches particular directory and not its subdirs, I would have to watch all directories.
This means I would have to create thousands or even tens of thousands of inotify watches.
Quoted 7 lines
So my questions are:
1. is it safe? that is, will it not lock the kernel up, or cause excessive memory consumption?
2. is it economic in terms of CPU time and RAM? I have no idea how to even measure such a thing happening in the kernel..
Hi,
I don't know the specific answers apart my intuitive feeling, which is that it probably would cause excessive memory use and that it is not elegant at all.
What may be interesting to you is the notification "engine" rewrite Eric Paris is doing and new fanotify interface which will, hopefully, come soon on top of it. With fanotify you will be able to monitor all changes much more easily and efficiently.
Have a look at patch descriptions in:
http://people.redhat.com/~eparis/fsnotify/
Tvrtko
Am Mittwoch, 20. Mai 2009 schrieb Marcin Krol:
Quoted 5 lines
Hello everyone,
First, apols for using up bandwidth, but I honestly found no other place where I can ask about this (and get meaningful reply).
Quoted 7 lines
I'm not a kernel programmer, but I want to develop a program that would watch modifications in *all* user directories on a busy server using inotify.
This is for high-availability purposes - events would be collected and once every several minutes changed dirs would be rsync'ed to failover server or smth like that would be done.
Hmmm, I think you could just run a rsync periodically. It might even be faster detecting changed files. Unless you are talking really high number of directories and files.
Quoted 2 lines
As inotify watches particular directory and not its subdirs, I would have to watch all directories.
Yes. Thats cumbersome.
Quoted 2 lines
This means I would have to create thousands or even tens of thousands of inotify watches.
I wrote a ruby script using libinotify-ruby which does just that. I only syncs on demand tough. I.e. when someplace places a special sync file in a watched directory.
Quoted 7 lines
So my questions are:
1. is it safe? that is, will it not lock the kernel up, or cause excessive memory consumption?
2. is it economic in terms of CPU time and RAM? I have no idea how to even measure such a thing happening in the kernel..
That script is running productively for well over a year now.
There have been some problems with it stopping doing work occasionally. It has to be restarted. Now there is a monitor process which restarts it automatically should it end. Might be a race condition or programming error in the script. But the customer is happy with the workaround and didn't want me to put any further efforts in finding the real cause.
Regarding the inotify implementation in the kernel I darkly remember that there were some improvements. Search inotify or fsnotify in kernel-ml.
Ciao,
Signature · 2 lines
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90
Quoted 1 line · Martin Steigerwald
Hmmm, I think you could just run a rsync periodically. It might even be faster detecting changed files.
I beg to differ on this: rsync does quite intensive (in terms of disk activity and CPU activity) comparisons at the beginning of synchronization. It's pretty light later, true, but running rsync every few minutes on entire /home is IMO out of question.
Quoted 3 lines
I wrote a ruby script using libinotify-ruby which does just that. I only syncs on demand tough. I.e. when someplace places a special sync file in a watched directory.
That script is running productively for well over a year now.
Good to know the idea is not totally off the wall.. Thanks.
Anyway, I'll try using fsnotify / fanotify.
My main gripe with it, though, is that it is not in the mainline kernel, and thus in all probability it is not tested as widely as inotify.
Are there any chances for its inclusion in the near future?
Regards,
mk
Am Mittwoch, 20. Mai 2009 schrieb Marcin Krol:
Quoted 8 lines
Quoted 2 lines · Martin Steigerwald
Hmmm, I think you could just run a rsync periodically. It might even be faster detecting changed files.
I beg to differ on this: rsync does quite intensive (in terms of disk activity and CPU activity) comparisons at the beginning of synchronization. It's pretty light later, true, but running rsync every few minutes on entire /home is IMO out of question.
It depends on the amount of directories being watched and the amount directory entries in there. Rsync version 3 starts of synchronizing before building up the entire trees on both sides. It uses an incremental approach and this appears to make quite a difference.
For syncing /home I wonder whether inotify or fsnotify/fanotify is the right approach. A cluster filesystem of some sort comes to my mind instead. But question is, which one really works good enough. Didn't do any testing in that area yet.
Quoted 7 lines
Quoted 5 lines
I wrote a ruby script using libinotify-ruby which does just that. I only syncs on demand tough. I.e. when someplace places a special sync file in a watched directory.
That script is running productively for well over a year now.
Good to know the idea is not totally off the wall.. Thanks.
Main difference is that the sync is only triggered on demand. But the watching part appears to work quite nicely. Its not used for /home, but for the static html/php/image content for an apache2.2 which handles a lot of domains.
Quoted 4 lines
Anyway, I'll try using fsnotify / fanotify.
My main gripe with it, though, is that it is not in the mainline kernel, and thus in all probability it is not tested as widely as inotify.
Well then it will get more testing now I think ;-).
Quoted 1 line
Are there any chances for its inclusion in the near future?
Indeed would be nice to have.
Signature · 2 lines
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90
Am Mittwoch, 20. Mai 2009 schrieb Marcin Krol:
Quoted 8 lines
Quoted 2 lines · Martin Steigerwald
Hmmm, I think you could just run a rsync periodically. It might even be faster detecting changed files.
I beg to differ on this: rsync does quite intensive (in terms of disk activity and CPU activity) comparisons at the beginning of synchronization. It's pretty light later, true, but running rsync every few minutes on entire /home is IMO out of question.
Another idea that might be applicable:
We have a clustered setup - exactly also where that inotify ruby script runs - that used LVM and SoftRAID 1 for providing mirroring between both locations.
In each locations there is a RAID array with some hardware RAID, i.e. redundant in itself. Each RAID array is connected via FC to both cluster servers. Then we layer a SoftRAID 1 on top of it. Both cluster servers see the SoftRAID 1 device. One usually only does NFS and one usually only MySQL. Thus we made two volume groups. One of them is used by the NFS server only and the other one by the MySQL server.
In failover case the remaining server stoniths the failed server and takes over the volume group of it.
This way one of the servers could fail and the remaining server will be able to access the most recent data. And one of the externel RAID arrays could fail as well.
This worked remarkably well for more than a year, too. It won't work when you need to access the same volumes on both servers simultaneously, obviously.
Signature · 2 lines
Martin Steigerwald - team(ix) GmbH - http://www.teamix.de gpg: 19E3 8D42 896F D004 08AC A0CA 1E10 C593 0399 AE90
Marcin Krol:
Quoted 3 lines
I'm not a kernel programmer, but I want to develop a program that would watch modifications in *all* user directories on a busy server using inotify.
:::
Quoted 5 lines
1. is it safe? that is, will it not lock the kernel up, or cause excessive memory consumption?
2. is it economic in terms of CPU time and RAM? I have no idea how to even measure such a thing happening in the kernel..
The maximum number of inotify instances per user is limited to /proc/sys/fs/inotify/max_user_instances which is 128 by default. And also the number of watches per user is limited by /proc/sys/fs/inotify/max_user_watches (8192 by defaut). Theoretically you may be able to monitor 8192 directories, but it consumes memory.
If periodical rsync is out of question for you, how about kprobe? While I don't think it is a beautiful solution, to set a hook to vfs_mkdir, vfs_unlink, etc and to compare the target super_block may work for you.
See Documentation/kprobe.txt in detail.
J. R. Okajima