Skip to contentSkip to conversation

PROBLEM: Inotify leaks file descriptors.

DISCUSSION David Turner 2 participants 6 messages Mar 4, 2014 – Mar 19, 2014

lore
Share & tools
lore
#0 lore raw

I apologize for the slightly convoluted reproduction steps here, but I was not easily able to find a simpler test case in the time that I had available.

First, you'll need Facebook's watchman:
https://github.com/facebook/watchman

Build and install it. Then run the attached Python script. After a few hundred lines, you'll start to see errors of the form inotify_init error: Too many open files. That could just indicate that watchman is leaking, but I think that's not what's going on, because killing watchman does not fix the problem.

To demonstrate, kill the python script, then kill watchman. Then run tail -f /etc/hosts. You'll get "tail: inotify cannot be used, reverting to polling: Too many open files" (you may need to run a few tails to see the error). In fact, the only way I have found to get back to normal is to reboot.

I tried increasing the ulimit to 10000 (from the default 1024). The error still happens, but it seems to take a bit longer.

I have tried on a couple of Ubuntu kernels:

Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4 21:25:43 UTC 2014

And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give the full version).

I've also tried a stock kernel built from source (in a virtualbox):

Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014

I get the error on all of these.
There is no output in dmesg.

I was running these tests on ext4 filesystems:
(for the Ubuntu kernels)
/dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro) (for the stock kernel, in the virtualbox)
/dev/sda1 on / type ext4 (rw,errors=remount-ro)

Please let me know if you need any more information.

FWIW, I did find this bug while googling, but it was on older kernels and was allegedly fixed:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666

#1 lore raw

(script attached)

Quoted 45 lines · David Turner

I apologize for the slightly convoluted reproduction steps here, but I was not easily able to find a simpler test case in the time that I had available.

First, you'll need Facebook's watchman:
https://github.com/facebook/watchman

Build and install it. Then run the attached Python script. After a few hundred lines, you'll start to see errors of the form inotify_init error: Too many open files. That could just indicate that watchman is leaking, but I think that's not what's going on, because killing watchman does not fix the problem.

To demonstrate, kill the python script, then kill watchman. Then run tail -f /etc/hosts. You'll get "tail: inotify cannot be used, reverting to polling: Too many open files" (you may need to run a few tails to see the error). In fact, the only way I have found to get back to normal is to reboot.

I tried increasing the ulimit to 10000 (from the default 1024). The error still happens, but it seems to take a bit longer.

I have tried on a couple of Ubuntu kernels:

Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4 21:25:43 UTC 2014

And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give the full version).

I've also tried a stock kernel built from source (in a virtualbox):

Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014

I get the error on all of these.
There is no output in dmesg.

I was running these tests on ext4 filesystems:
(for the Ubuntu kernels)
/dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro) (for the stock kernel, in the virtualbox)
/dev/sda1 on / type ext4 (rw,errors=remount-ro)

Quoted 8 lines

Please let me know if you need any more information.

FWIW, I did find this bug while googling, but it was on older kernels and was allegedly fixed:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666

#!/usr/bin/python

from atomicinteger import AtomicInteger from json import loads, dumps
from random import random
from subprocess import call, check_output from tempfile import mkdtemp
from time import sleep, time

import os
import socket
import stat
import threading

#from https://github.com/littlehedgehog/base/blob/master/atomicinteger.py
class AtomicInteger:
def __init__(self, integer = 0):
self.counter = integer
self.lock = threading.RLock()
return

def increase(self, inc = 1):
self.lock.acquire() self.counter = self.counter + inc self.lock.release()
return

def decrease(self, dec = 1):
self.lock.acquire() self.counter = self.counter - dec self.lock.release()
return

def get(self):
return self.counter

def get_sockname():
result = check_output(["watchman", "get-sockname"]) result = loads(result)
return result['sockname']

def connect():
sockname = get_sockname()
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) sock.connect(sockname)
sock.setblocking(False)
return sock

def watch(sock, directory):
watch = ['watch', directory] sock.sendall(dumps(watch) + "\n") result = readline(sock)
result = loads(result)
if not result.get("watch"):
print result

def readline(sock):
message = []
start = time()
while True:
elapsed = time() - start
if elapsed > 5:
print "We have been waiting a very long time for data from watchman. We have so far: %s" % "".join(message)
try:
data = sock.recv(1024, socket.MSG_DONTWAIT)
if "\n" in data:
message.append(data[:data.index("\n")])
break
except socket.error:
pass
sleep(0.001)
return "".join(message)

def since(sock, directory, since="c:1:2:3:4"):
expression = {'since' : since}
watch = ["query", directory, expression] sock.sendall(dumps(watch) + "\n")
message = readline(sock)
result = loads(message)
if "error" in result:
print "Error in since: %s (since = %s)" % (result["error"], since)
return result

def create(sock, directory=None):
directory = mkdtemp(dir=directory) watch(sock, directory)
return directory

def touch(directory):

    f = open(os.path.join(directory, "file-%s" % random()), "w")
    f.write("x")
    f.close()

def isdir(f):
result = os.lstat(f)
return stat.S_ISDIR(result.st_mode)

def recursive_rmdir(directory):
for f in os.listdir(directory):
qualified = os.path.join(directory, f) if isdir(qualified):
recursive_rmdir(qualified)
else:
os.unlink(qualified)
os.rmdir(directory)

nthreads = AtomicInteger()
runs = AtomicInteger()

def run():
nthreads.increase()
runs.increase()
directory = None
sock = None
try:
print "RUN: %d %d" % (runs.get(), nthreads.get()) sock = connect()
directory = create(sock)

result = since(sock, directory)
assert "files" not in result or len(result["files"]) == 0 clock = result.get("clock")
if not clock:
print "Failed since: %s" % result
assert clock

#this stanza is only necesary on unbuntu 3.11; on 3.15, it can be skipped

        touch(directory)
        result = since(sock, directory, clock)
        assert result["clock"] != clock
        clock = result["clock"]
        assert len(result["files"]) == 1

        sleep(0.1)

#ditto
for i in range(5):
touch(directory)
result = since(sock, directory, clock) assert result["clock"] != clock if len(result["files"]) < 5:
print result

finally:
if sock:
sock.close()
if directory:
recursive_rmdir(directory)
nthreads.decrease()

def threaded(target, *args, **kwargs):
thread = threading.Thread(target=target, args=args, kwargs=kwargs) thread.start()

while True:

    if nthreads.get() < 15:
        threaded(run)
    else:
        sleep(0.1)
#2 lore raw

I was running code related to the python script -- multiple threads doing inotify things in parallel, using watchman, and I got the following Oops:

[152513.914195] watchman[4963]: segfault at 7ff04ddb09d0 ip
00007ff05b831f60 sp 00007ff04c5acce8 error 4 in
libpthread-2.15.so[7ff05b825000+18000]
[152516.577861] watchman[6138]: segfault at 7f1962e099d0 ip
00007f1970489f60 sp 00007f1962406ce8 error 4 in
libpthread-2.15.so[7f197047d000+18000]
[153010.703990] BUG: unable to handle kernel NULL pointer dereference at  
        (null)
[153010.704036] IP: [<          (null)>]           (null)
[153010.704060] PGD 1b1b4e067 PUD 1cc1f1067 PMD 0
[153010.704084] Oops: 0010 [#1] SMP
[153010.704103] Modules linked in: btrfs raid6_pq zlib_deflate xor ufs

qnx4 hfsplus hfs minix ntfs msdos jfs xfs reiserfs usb_storage cdc_acm joydev pci_stub vboxpci(OF) vboxnetadp(OF) vboxnetflt(OF) vboxdrv(OF) bnep rfcomm bluetooth parport_pc ppdev uvcvideo videobuf2_core binfmt_misc videodev snd_hda_codec_hdmi snd_hda_codec_conexant videobuf2_vmalloc videobuf2_memops snd_hda_intel snd_hda_codec snd_hwdep snd_pcm snd_seq_midi arc4 snd_rawmidi iwldvm mac80211 snd_seq_midi_event snd_seq psmouse thinkpad_acpi snd_timer snd_seq_device iwlwifi nvram serio_raw snd tpm_tis cfg80211 soundcore snd_page_alloc mac_hid mei_me mei lpc_ich lp ext2 parport dm_crypt i915 drm_kms_helper e1000e wmi drm ptp pps_core ahci libahci sdhci_pci sdhci i2c_algo_bit video

[153010.704453] CPU: 1 PID: 3586 Comm: watchman Tainted: GF       W  O
3.11.0-17-generic #31~precise1-Ubuntu
[153010.704493] Hardware name: LENOVO 4177Q5U/4177Q5U, BIOS 83ET76WW (1.46
) 07/05/2013
[153010.704529] task: ffff8801b6ae0000 ti: ffff88009ed30000 task.ti:
ffff88009ed30000
[153010.704564] RIP: 0010:[<0000000000000000>]  [<          (null)>]      
    (null)
[153010.704600] RSP: 0018:ffff88009ed31dc0  EFLAGS: 00010246
[153010.704624] RAX: 00000000b98ab901 RBX: ffff88015a9b5228 RCX:
00000000000188d0
[153010.704655] RDX: 000000000000b98a RSI: ffff880100b95c00 RDI:
ffff88015a9b5228
[153010.704686] RBP: ffff88009ed31dd8 R08: 0000000000000001 R09:
ffffea0000d85640
[153010.704718] R10: ffffffff811f9628 R11: 0000000000000000 R12:
ffff88015a9b5228
[153010.704750] R13: ffff880100b95ca0 R14: 00000000ffffffff R15:
ffff880100b95c00
[153010.704783] FS:  00007f5b19087700(0000) GS:ffff88021e240000(0000)
knlGS:0000000000000000
[153010.704819] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[153010.704846] CR2: 0000000000000000 CR3: 00000001dd9d1000 CR4:
00000000000427e0
[153010.704877] Stack:
[153010.704888]  ffffffff811f7120 ffff88015a9b5228 ffff8801e460c7f0
ffff88009ed31e28
[153010.704926]  ffffffff811f7847 ffff88009ed31e18 ffff880100b95c70
ffff88009ed31f50
[153010.704965]  ffff880100b95c00 0000000000000010 ffff8802120433c0
ffff8802120433c0
[153010.705005] Call Trace:
[153010.705024]  [<ffffffff811f7120>] ? fsnotify_put_mark+0x30/0x40
[153010.705054]  [<ffffffff811f7847>]
fsnotify_clear_marks_by_group_flags+0x87/0xb0
[153010.705088]  [<ffffffff811f7883>] fsnotify_clear_marks_by_group+0x13/0x20
[153010.705119]  [<ffffffff811f68d6>] fsnotify_destroy_group+0x16/0x40
[153010.705150]  [<ffffffff811f8be6>] inotify_release+0x26/0x50
[153010.705177]  [<ffffffff811b640a>] __fput+0xba/0x240
[153010.705201]  [<ffffffff811b65de>] ____fput+0xe/0x10
[153010.705226]  [<ffffffff810859d8>] task_work_run+0xc8/0xf0
[153010.706464]  [<ffffffff81013dfc>] do_notify_resume+0xac/0xc0
[153010.707730]  [<ffffffff8175159a>] int_signal+0x12/0x17
[153010.708933] Code:  Bad RIP value.
[153010.710089] RIP  [<          (null)>]           (null)
[153010.711251]  RSP <ffff88009ed31dc0>
[153010.712319] CR2: 0000000000000000
[153010.718669] ---[ end trace 17ed2927fe522cd1 ]---
#3 lore raw
In reply to David Turner
Quoted 55 lines · David Turner

(script attached)

Quoted 53 lines · David Turner

I apologize for the slightly convoluted reproduction steps here, but I was not easily able to find a simpler test case in the time that I had available.

First, you'll need Facebook's watchman:
https://github.com/facebook/watchman

Build and install it. Then run the attached Python script. After a few hundred lines, you'll start to see errors of the form inotify_init error: Too many open files. That could just indicate that watchman is leaking, but I think that's not what's going on, because killing watchman does not fix the problem.

To demonstrate, kill the python script, then kill watchman. Then run tail -f /etc/hosts. You'll get "tail: inotify cannot be used, reverting to polling: Too many open files" (you may need to run a few tails to see the error). In fact, the only way I have found to get back to normal is to reboot.

I tried increasing the ulimit to 10000 (from the default 1024). The error still happens, but it seems to take a bit longer.

I have tried on a couple of Ubuntu kernels:

Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4 21:25:43 UTC 2014

And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give the full version).

I've also tried a stock kernel built from source (in a virtualbox):

Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014

I get the error on all of these.
There is no output in dmesg.

I was running these tests on ext4 filesystems:
(for the Ubuntu kernels)
/dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro) (for the stock kernel, in the virtualbox)
/dev/sda1 on / type ext4 (rw,errors=remount-ro)

Please let me know if you need any more information.

FWIW, I did find this bug while googling, but it was on older kernels and was allegedly fixed:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666

So when I run your script on 3.14-rc5, I don't see any problems. So maybe
the problem really got fixed? BTW, I'll note that /proc/sys/fs/inotify/max_user_instances is 128 on my system. Not sure what it is on your system...

Honza

Quoted 160 lines

#!/usr/bin/python

from atomicinteger import AtomicInteger from json import loads, dumps
from random import random
from subprocess import call, check_output from tempfile import mkdtemp
from time import sleep, time

import os
import socket
import stat
import threading

#from https://github.com/littlehedgehog/base/blob/master/atomicinteger.py
class AtomicInteger:
def __init__(self, integer = 0):
self.counter = integer
self.lock = threading.RLock()
return

def increase(self, inc = 1):
self.lock.acquire() self.counter = self.counter + inc self.lock.release()
return

def decrease(self, dec = 1):
self.lock.acquire() self.counter = self.counter - dec self.lock.release()
return

def get(self):
return self.counter

def get_sockname():
result = check_output(["watchman", "get-sockname"]) result = loads(result)
return result['sockname']

def connect():
sockname = get_sockname()
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) sock.connect(sockname)
sock.setblocking(False)
return sock

def watch(sock, directory):
watch = ['watch', directory] sock.sendall(dumps(watch) + "\n") result = readline(sock)
result = loads(result)
if not result.get("watch"):
print result

def readline(sock):
message = []
start = time()
while True:
elapsed = time() - start
if elapsed > 5:
print "We have been waiting a very long time for data from watchman. We have so far: %s" % "".join(message)
try:
data = sock.recv(1024, socket.MSG_DONTWAIT)
if "\n" in data:
message.append(data[:data.index("\n")])
break
except socket.error:
pass
sleep(0.001)
return "".join(message)

def since(sock, directory, since="c:1:2:3:4"):
expression = {'since' : since}
watch = ["query", directory, expression] sock.sendall(dumps(watch) + "\n")
message = readline(sock)
result = loads(message)
if "error" in result:
print "Error in since: %s (since = %s)" % (result["error"], since)
return result

def create(sock, directory=None):
directory = mkdtemp(dir=directory) watch(sock, directory)
return directory

def touch(directory):

    f = open(os.path.join(directory, "file-%s" % random()), "w")
    f.write("x")
    f.close()

def isdir(f):
result = os.lstat(f)
return stat.S_ISDIR(result.st_mode)

def recursive_rmdir(directory):
for f in os.listdir(directory):
qualified = os.path.join(directory, f) if isdir(qualified):
recursive_rmdir(qualified)
else:
os.unlink(qualified)
os.rmdir(directory)

nthreads = AtomicInteger()
runs = AtomicInteger()

def run():
nthreads.increase()
runs.increase()
directory = None
sock = None
try:
print "RUN: %d %d" % (runs.get(), nthreads.get()) sock = connect()
directory = create(sock)

result = since(sock, directory)
assert "files" not in result or len(result["files"]) == 0 clock = result.get("clock")
if not clock:
print "Failed since: %s" % result
assert clock

#this stanza is only necesary on unbuntu 3.11; on 3.15, it can be skipped

        touch(directory)
        result = since(sock, directory, clock)
        assert result["clock"] != clock
        clock = result["clock"]
        assert len(result["files"]) == 1

        sleep(0.1)

#ditto
for i in range(5):
touch(directory)
result = since(sock, directory, clock) assert result["clock"] != clock if len(result["files"]) < 5:
print result

finally:
if sock:
sock.close()
if directory:
recursive_rmdir(directory)
nthreads.decrease()

def threaded(target, *args, **kwargs):
thread = threading.Thread(target=target, args=args, kwargs=kwargs) thread.start()

while True:

    if nthreads.get() < 15:
        threaded(run)
    else:
        sleep(0.1)
Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#4 lore raw
In reply to David Turner

(script attached)

Quoted 45 lines · David Turner

I apologize for the slightly convoluted reproduction steps here, but I was not easily able to find a simpler test case in the time that I had available.

First, you'll need Facebook's watchman:
https://github.com/facebook/watchman

Build and install it. Then run the attached Python script. After a few hundred lines, you'll start to see errors of the form inotify_init error: Too many open files. That could just indicate that watchman is leaking, but I think that's not what's going on, because killing watchman does not fix the problem.

To demonstrate, kill the python script, then kill watchman. Then run tail -f /etc/hosts. You'll get "tail: inotify cannot be used, reverting to polling: Too many open files" (you may need to run a few tails to see the error). In fact, the only way I have found to get back to normal is to reboot.

I tried increasing the ulimit to 10000 (from the default 1024). The error still happens, but it seems to take a bit longer.

I have tried on a couple of Ubuntu kernels:

Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4 21:25:43 UTC 2014

And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give the full version).

I've also tried a stock kernel built from source (in a virtualbox):

Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014

I get the error on all of these.
There is no output in dmesg.

I was running these tests on ext4 filesystems:
(for the Ubuntu kernels)
/dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro) (for the stock kernel, in the virtualbox)
/dev/sda1 on / type ext4 (rw,errors=remount-ro)

Quoted 8 lines

Please let me know if you need any more information.

FWIW, I did find this bug while googling, but it was on older kernels and was allegedly fixed:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666

#!/usr/bin/python

from atomicinteger import AtomicInteger from json import loads, dumps
from random import random
from subprocess import call, check_output from tempfile import mkdtemp
from time import sleep, time

import os
import socket
import stat
import threading

#from https://github.com/littlehedgehog/base/blob/master/atomicinteger.py
class AtomicInteger:
def __init__(self, integer = 0):
self.counter = integer
self.lock = threading.RLock()
return

def increase(self, inc = 1):
self.lock.acquire() self.counter = self.counter + inc self.lock.release()
return

def decrease(self, dec = 1):
self.lock.acquire() self.counter = self.counter - dec self.lock.release()
return

def get(self):
return self.counter

def get_sockname():
result = check_output(["watchman", "get-sockname"]) result = loads(result)
return result['sockname']

def connect():
sockname = get_sockname()
sock = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) sock.connect(sockname)
sock.setblocking(False)
return sock

def watch(sock, directory):
watch = ['watch', directory] sock.sendall(dumps(watch) + "\n") result = readline(sock)
result = loads(result)
if not result.get("watch"):
print result

def readline(sock):
message = []
start = time()
while True:
elapsed = time() - start
if elapsed > 5:
print "We have been waiting a very long time for data from watchman. We have so far: %s" % "".join(message)
try:
data = sock.recv(1024, socket.MSG_DONTWAIT)
if "\n" in data:
message.append(data[:data.index("\n")])
break
except socket.error:
pass
sleep(0.001)
return "".join(message)

def since(sock, directory, since="c:1:2:3:4"):
expression = {'since' : since}
watch = ["query", directory, expression] sock.sendall(dumps(watch) + "\n")
message = readline(sock)
result = loads(message)
if "error" in result:
print "Error in since: %s (since = %s)" % (result["error"], since)
return result

def create(sock, directory=None):
directory = mkdtemp(dir=directory) watch(sock, directory)
return directory

def touch(directory):

    f = open(os.path.join(directory, "file-%s" % random()), "w")
    f.write("x")
    f.close()

def isdir(f):
result = os.lstat(f)
return stat.S_ISDIR(result.st_mode)

def recursive_rmdir(directory):
for f in os.listdir(directory):
qualified = os.path.join(directory, f) if isdir(qualified):
recursive_rmdir(qualified)
else:
os.unlink(qualified)
os.rmdir(directory)

nthreads = AtomicInteger()
runs = AtomicInteger()

def run():
nthreads.increase()
runs.increase()
directory = None
sock = None
try:
print "RUN: %d %d" % (runs.get(), nthreads.get()) sock = connect()
directory = create(sock)

result = since(sock, directory)
assert "files" not in result or len(result["files"]) == 0 clock = result.get("clock")
if not clock:
print "Failed since: %s" % result
assert clock

#this stanza is only necesary on unbuntu 3.11; on 3.15, it can be skipped

        touch(directory)
        result = since(sock, directory, clock)
        assert result["clock"] != clock
        clock = result["clock"]
        assert len(result["files"]) == 1

        sleep(0.1)

#ditto
for i in range(5):
touch(directory)
result = since(sock, directory, clock) assert result["clock"] != clock if len(result["files"]) < 5:
print result

finally:
if sock:
sock.close()
if directory:
recursive_rmdir(directory)
nthreads.decrease()

def threaded(target, *args, **kwargs):
thread = threading.Thread(target=target, args=args, kwargs=kwargs) thread.start()

while True:

    if nthreads.get() < 15:
        threaded(run)
    else:
        sleep(0.1)
#5 lore raw
In reply to David Turner
Quoted 4 lines · David Turner

I apologize for the slightly convoluted reproduction steps here, but I was not easily able to find a simpler test case in the time that I had available.

Quoted 8 lines

First, you'll need Facebook's watchman:
https://github.com/facebook/watchman

Build and install it. Then run the attached Python script. After a few hundred lines, you'll start to see errors of the form inotify_init error: Too many open files. That could just indicate that watchman is leaking, but I think that's not what's going on, because killing watchman does not fix the problem.

Since opening other files clearly works, what is likely leaking somewhere
is 'inotify_devs' counter - that's a counter of inotify instances per user. And that leak is likely happening because we leak a fsnotify group reference count somewhere. Why that happens isn't clear to me but the refcounting isn't quite simple so some bug seems possible. I guess I'll try to reproduce this and see.

Hpnza

Quoted 45 lines

To demonstrate, kill the python script, then kill watchman. Then run tail -f /etc/hosts. You'll get "tail: inotify cannot be used, reverting to polling: Too many open files" (you may need to run a few tails to see the error). In fact, the only way I have found to get back to normal is to reboot.

I tried increasing the ulimit to 10000 (from the default 1024). The error still happens, but it seems to take a bit longer.

I have tried on a couple of Ubuntu kernels:

Linux version 3.11.0-17-generic (buildd@toyol) (gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #31~precise1-Ubuntu SMP Tue Feb 4 21:25:43 UTC 2014

And Ubuntu's 3.8.0-36-generic (it's not running right now so I can't give the full version).

I've also tried a stock kernel built from source (in a virtualbox):

Linux version 3.13.5 (dturner@dturner-virtualbox) (gcc version 4.8.1 (Ubuntu/Linaro 4.8.1-10ubuntu8) ) #1 SMP Mon Mar 3 20:41:51 EST 2014

I get the error on all of these.
There is no output in dmesg.

I was running these tests on ext4 filesystems:
(for the Ubuntu kernels)
/dev/mapper/stross--vg-root on / type ext4 (rw,errors=remount-ro) (for the stock kernel, in the virtualbox)
/dev/sda1 on / type ext4 (rw,errors=remount-ro)

Please let me know if you need any more information.

FWIW, I did find this bug while googling, but it was on older kernels and was allegedly fixed:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1101666

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR