Skip to contentSkip to conversation

inotify_add_watch/inotify_rm_watch loops trigger oom

BUG Eryu Guan 4 participants 6 messages Feb 14, 2016 – Feb 15, 2016 VFS

lore
Share & tools
lore

Eryu Guan

author
#0 lore raw

Hi,

Starting from v4.5-rc1 running inotify_add_watch/inotify_rm_watch in loop could trigger OOM and system becomes unusuable. v4.4 kernel is fine with the same stress test.

Reverting c510eff6beba ("fsnotify: destroy marks with call_srcu instead of dedicated thread") on top of v4.5-rc3 passed the same test, seems that this patch introduced some kind of memleak?

On v4.5-rc[1-3] the test program triggers oom within 10 minutes on my test vm with 8G mem. After reverting the commit in question the same vm survived more than 1 hour stress test.

	./inotify <mnt>

I attached the test program and oom console log. If more information is needed please let me know.

Thanks,
Eryu

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <stdio.h>
#include <sys/inotify.h>

int main(int argc, char *argv[])

{
	int notify_fd;
	int wd, ret;

	notify_fd = inotify_init();

	for (;;) {
		wd = inotify_add_watch(notify_fd, argv[1], IN_MODIFY | IN_CREATE | IN_DELETE );
		if (wd < 0) {
			perror("inotify_add_watch");
		}
		ret = inotify_rm_watch(notify_fd, wd);
		if (ret < 0) {
			perror("inotify_rm_watch");
		}
	}
out:
	close(notify_fd);

	return 0;
}
#1 lore raw
Quoted 14 lines · On Sun, 14 Feb 2016 16:35:43 +0800 Eryu Guan <[email protected]> wrote:

Hi,

Starting from v4.5-rc1 running inotify_add_watch/inotify_rm_watch in loop could trigger OOM and system becomes unusuable. v4.4 kernel is fine with the same stress test.

Reverting c510eff6beba ("fsnotify: destroy marks with call_srcu instead of dedicated thread") on top of v4.5-rc3 passed the same test, seems that this patch introduced some kind of memleak?

On v4.5-rc[1-3] the test program triggers oom within 10 minutes on my test vm with 8G mem. After reverting the commit in question the same vm survived more than 1 hour stress test.

Quoted 7 lines
	./inotify <mnt>

I attached the test program and oom console log. If more information is needed please let me know.

Thanks,
Eryu

Thanks Eryu, I think I see what the problem is. This reproducer is creating and deleting marks very rapidly. But the SRCU code has this:

    #define SRCU_CALLBACK_BATCH     10                                              
    #define SRCU_INTERVAL           1                                               

So, process_srcu will only process 10 entries at a time, and only once per jiffy. The upshot there is that that reproducer can create entries _much_ faster than they can be cleaned up now that we're using call_srcu in this codepath. If you kill the program before the OOM killer kicks in, they all eventually get cleaned up but it does take a while (minutes).

I clearly didn't educate myself enough as to the limitations of call_srcu before converting this code over to use it (and I missed Paul's subtle hints in that regard). We may need to revert that patch before v4.5 ships, but I'd like to ponder it for a few days to and see whether there is some way to batch them up so that they get reaped more efficiently without requiring the dedicated thread.

Signature · 1 line
Jeff Layton <[email protected]>
#2 lore raw
Quoted 38 lines · Jeff Layton
Quoted 14 lines · On Sun, 14 Feb 2016 16:35:43 +0800 Eryu Guan <[email protected]> wrote:

Hi,

Starting from v4.5-rc1 running inotify_add_watch/inotify_rm_watch in loop could trigger OOM and system becomes unusuable. v4.4 kernel is fine with the same stress test.

Reverting c510eff6beba ("fsnotify: destroy marks with call_srcu instead of dedicated thread") on top of v4.5-rc3 passed the same test, seems that this patch introduced some kind of memleak?

On v4.5-rc[1-3] the test program triggers oom within 10 minutes on my test vm with 8G mem. After reverting the commit in question the same vm survived more than 1 hour stress test.

Quoted 7 lines
	./inotify <mnt>

I attached the test program and oom console log. If more information is needed please let me know.

Thanks,
Eryu

Thanks Eryu, I think I see what the problem is. This reproducer is creating and deleting marks very rapidly. But the SRCU code has this:

    #define SRCU_CALLBACK_BATCH     10                                              
    #define SRCU_INTERVAL           1                                               

So, process_srcu will only process 10 entries at a time, and only once per jiffy. The upshot there is that that reproducer can create entries _much_ faster than they can be cleaned up now that we're using call_srcu in this codepath. If you kill the program before the OOM killer kicks in, they all eventually get cleaned up but it does take a while (minutes).

Quoted 6 lines

I clearly didn't educate myself enough as to the limitations of call_srcu before converting this code over to use it (and I missed Paul's subtle hints in that regard). We may need to revert that patch before v4.5 ships, but I'd like to ponder it for a few days to and see whether there is some way to batch them up so that they get reaped more efficiently without requiring the dedicated thread.

One thought would be to add an "emergency mode" to SRCU similar to that already in RCU. Something to the effect that if the current list of callbacks is going to take more than a second to drain at the configured per-jiffy rate, just process them without waiting.

Would that help in this case, or am I missing something about the reproducer?

Thanx, Paul

#3 lore raw
Quoted 51 lines · On Sun, 14 Feb 2016 17:02:38 -0800 "Paul E. McKenney"…
Quoted 38 lines · Jeff Layton
Quoted 14 lines · On Sun, 14 Feb 2016 16:35:43 +0800 Eryu Guan <[email protected]> wrote:

Hi,

Starting from v4.5-rc1 running inotify_add_watch/inotify_rm_watch in loop could trigger OOM and system becomes unusuable. v4.4 kernel is fine with the same stress test.

Reverting c510eff6beba ("fsnotify: destroy marks with call_srcu instead of dedicated thread") on top of v4.5-rc3 passed the same test, seems that this patch introduced some kind of memleak?

On v4.5-rc[1-3] the test program triggers oom within 10 minutes on my test vm with 8G mem. After reverting the commit in question the same vm survived more than 1 hour stress test.

Quoted 7 lines
	./inotify <mnt>

I attached the test program and oom console log. If more information is needed please let me know.

Thanks,
Eryu

Thanks Eryu, I think I see what the problem is. This reproducer is creating and deleting marks very rapidly. But the SRCU code has this:

    #define SRCU_CALLBACK_BATCH     10                                              
    #define SRCU_INTERVAL           1                                               

So, process_srcu will only process 10 entries at a time, and only once per jiffy. The upshot there is that that reproducer can create entries _much_ faster than they can be cleaned up now that we're using call_srcu in this codepath. If you kill the program before the OOM killer kicks in, they all eventually get cleaned up but it does take a while (minutes).

Quoted 6 lines

I clearly didn't educate myself enough as to the limitations of call_srcu before converting this code over to use it (and I missed Paul's subtle hints in that regard). We may need to revert that patch before v4.5 ships, but I'd like to ponder it for a few days to and see whether there is some way to batch them up so that they get reaped more efficiently without requiring the dedicated thread.

One thought would be to add an "emergency mode" to SRCU similar to that already in RCU. Something to the effect that if the current list of callbacks is going to take more than a second to drain at the configured per-jiffy rate, just process them without waiting.

Quoted 4 lines

Would that help in this case, or am I missing something about the reproducer?

Thanx, Paul

I sent a patchset just a little while ago that should fix this in an even better way, I think, without using call_srcu. In addition to the problem that Eryu mentions, fsnotify_put_mark can cause a cascade of other "put" routines. While I don't see any that obviously can block, we probably don't want to that activity under local_bh_disable, so I think that may be the best solution for fsnotify_marks.

That said, the "emergency mode" you describe might make srcu more useful overall. What may make even more sense is to simply run all of the callbacks without waiting when there is a synchronize_srcu (or srcu_barrier) that is blocked and waiting on all of the callbacks to complete.

Signature · 1 line
Jeff Layton <[email protected]>
#4 lore raw
In reply to Paul E. McKenney
Quoted 8 lines · Paul E. McKenney

One thought would be to add an "emergency mode" to SRCU similar to that already in RCU. Something to the effect that if the current list of callbacks is going to take more than a second to drain at the configured per-jiffy rate, just process them without waiting.

Would that help in this case, or am I missing something about the reproducer?

Let's see if we can avoid srcu callbacks entirely as per Jeff, that would be the best option.

Linus

#5 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 6 lines · Paul E. McKenney

One thought would be to add an "emergency mode" to SRCU similar to that already in RCU. Something to the effect that if the current list of callbacks is going to take more than a second to drain at the configured per-jiffy rate, just process them without waiting.

Quoted 5 lines
Quoted 2 lines

Would that help in this case, or am I missing something about the reproducer?

Let's see if we can avoid srcu callbacks entirely as per Jeff, that would be the best option.

Agreed, no point in adding complication without use cases.

Thanx, Paul