Inotify problem [was Re: 2.6.13-rc6-mm1]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Hi,
Quoted 71 lines · John McCutchan
Quoted 52 lines · Andrew Morton
Quoted 51 lines · Reuben Farrelly
Hi,
Quoted 4 lines · Andrew Morton
ftp://ftp.kernel.org/pub/linux/kernel/people/akpm/patches/2.6/2.6.13-rc6/2.6.13-rc6-mm1/
- Lots of fixes, updates and cleanups all over the place.
Quoted 8 lines
- If you have the right debugging options set, this kernel will generate a storm of sleeping-in-atomic-code warnings at boot, from the scsi code. It is being worked on.
Changes since 2.6.13-rc5-mm1:
linus.patch
Noted this in my log earlier today.
Is this inotify related?
Aug 21 08:33:04 tornado kernel: idr_remove called for id=2048 which is not allocated. Aug 21 08:33:04 tornado kernel: [<c0103a00>] dump_stack+0x17/0x19 Aug 21 08:33:04 tornado kernel: [<c01c9f9a>] idr_remove_warning+0x1b/0x1d Aug 21 08:33:04 tornado kernel: [<c01ca024>] sub_remove+0x88/0xea Aug 21 08:33:04 tornado kernel: [<c01ca0a1>] idr_remove+0x1b/0x7f Aug 21 08:33:04 tornado kernel: [<c018176a>] remove_watch_no_event+0x7a/0x12e Aug 21 08:33:04 tornado kernel: [<c0181f64>] inotify_release+0x8f/0x1af Aug 21 08:33:04 tornado kernel: [<c015ca80>] __fput+0xaf/0x199 Aug 21 08:33:04 tornado kernel: [<c015c9b8>] fput+0x22/0x3b Aug 21 08:33:04 tornado kernel: [<c015b2ed>] filp_close+0x41/0x67 Aug 21 08:33:04 tornado kernel: [<c015b383>] sys_close+0x70/0x92 Aug 21 08:33:04 tornado kernel: [<c0102a9b>] sysenter_past_esp+0x54/0x75 Aug 21 08:33:04 tornado kernel: idr_remove called for id=3072 which is not allocated. Aug 21 08:33:05 tornado kernel: [<c0103a00>] dump_stack+0x17/0x19 Aug 21 08:33:05 tornado kernel: [<c01c9f9a>] idr_remove_warning+0x1b/0x1d Aug 21 08:33:05 tornado kernel: [<c01ca024>] sub_remove+0x88/0xea Aug 21 08:33:05 tornado kernel: [<c01ca0a1>] idr_remove+0x1b/0x7f Aug 21 08:33:05 tornado kernel: [<c018176a>] remove_watch_no_event+0x7a/0x12e Aug 21 08:33:05 tornado kernel: [<c0181f64>] inotify_release+0x8f/0x1af Aug 21 08:33:05 tornado kernel: [<c015ca80>] __fput+0xaf/0x199 Aug 21 08:33:05 tornado kernel: [<c015c9b8>] fput+0x22/0x3b Aug 21 08:33:05 tornado kernel: [<c015b2ed>] filp_close+0x41/0x67 Aug 21 08:33:05 tornado kernel: [<c015b383>] sys_close+0x70/0x92 Aug 21 08:33:05 tornado kernel: [<c0102a9b>] sysenter_past_esp+0x54/0x75This would have been triggered by using dovecot IMAP which is configured to use inotify on Maildir. I'm also seeing some userspace errors logged for dovecot:
"Aug 21 04:17:22 Error: IMAP(reuben):
inotify_rm_watch()failed: Invalid argument"I'll deal with those with the guy who wrote the inotify code in dovecot.
Quoted 9 lines
Quoted 2 lines
I'm not so sure userspace should be able or need to cause the kernel to dump stack traces like that though?
Yes, the stack dumps would appear to be due to an inotify bug.
The message from dovecot is allegedly due to dovecot passing in a file descriptor which was not obtained from the
inotify_init()syscall. But until we know what caused those stack dumps we cannot definitely say whether dovecot is at fault.Inotify has a check on both add and rm watch syscalls:
/* verify that this is indeed an inotify instance */ if (unlikely(filp->f_op != &inotify_fops)) { ret = -EINVAL; goto out; }
Quoted 5 lines
This is crashing in
inotify_release, which is called on close of the inotify instance. So this fd must be from an inotify instance right?I looked at the dovecot code, it looks fine wrt inotify. Long shot, but the close-on-exec flag is set. Could this be tripping anything up?
I have also observed another problem with inotify with dovecot - so I spoke with Johannes Berg who wrote the inotify code in dovecot. He suggested I post here to LKML since his opinion is that this to be a kernel bug.
The problem I am observing is this, logged by dovecot after a period of time when a client is connected:
dovecot: Aug 22 14:31:23 Error: IMAP(gilly): inotify_rm_watch() failed: Invalid argument
dovecot: Aug 22 14:31:23 Error: IMAP(gilly): inotify_rm_watch() failed: Invalid argument
dovecot: Aug 22 14:31:23 Error: IMAP(gilly): inotify_rm_watch() failed: Invalid argument
Multiply that by about 1000 ;-)
Some debugging shows this:
dovecot: Aug 25 19:31:22 Warning: IMAP(gilly): removing wd 1019 from inotify fd 4
dovecot: Aug 25 19:31:22 Warning: IMAP(gilly): removing wd 1018 from inotify fd 4
dovecot: Aug 25 19:31:22 Warning: IMAP(gilly): inotify_add_watch returned 1019
dovecot: Aug 25 19:31:22 Warning: IMAP(gilly): inotify_add_watch returned 1020
dovecot: Aug 25 19:31:23 Warning: IMAP(gilly): removing wd 1020 from inotify fd 4
dovecot: Aug 25 19:31:23 Warning: IMAP(gilly): removing wd 1019 from inotify fd 4
dovecot: Aug 25 19:31:24 Warning: IMAP(gilly): inotify_add_watch returned 1020
dovecot: Aug 25 19:31:24 Warning: IMAP(gilly): inotify_add_watch returned 1021
dovecot: Aug 25 19:31:24 Warning: IMAP(gilly): removing wd 1021 from inotify fd 4
dovecot: Aug 25 19:31:24 Warning: IMAP(gilly): removing wd 1020 from inotify fd 4
dovecot: Aug 25 19:31:25 Warning: IMAP(gilly): inotify_add_watch returned 1021
dovecot: Aug 25 19:31:25 Warning: IMAP(gilly): inotify_add_watch returned 1022
dovecot: Aug 25 19:31:25 Warning: IMAP(gilly): removing wd 1022 from inotify fd 4
dovecot: Aug 25 19:31:25 Warning: IMAP(gilly): removing wd 1021 from inotify fd 4
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): inotify_add_watch returned 1022
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): inotify_add_watch returned 1023
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): removing wd 1023 from inotify fd 4
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): removing wd 1022 from inotify fd 4
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): inotify_add_watch returned 1023
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): inotify_add_watch returned 1024
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): removing wd 1024 from inotify fd 4
dovecot: Aug 25 19:31:27 Error: IMAP(gilly): inotify_rm_watch() failed: Invalid argument
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): removing wd 1023 from inotify fd 4
dovecot: Aug 25 19:31:28 Warning: IMAP(gilly): inotify_add_watch returned 1024
dovecot: Aug 25 19:31:28 Warning: IMAP(gilly): inotify_add_watch returned 1024
Note the incrementing wd value even though we are removing them as we go..
This is using latest CVS of dovecot code and with 2.6.12-rc6-mm(1|2) kernel.
Robert, John, what do you think? Is this possibly related to the oops seen in the log that I reported earlier? (Which is still showing up 2-3 times per day, btw)
Reuben
Hi,
Quoted 3 lines
I have also observed another problem with inotify with dovecot - so I spoke with Johannes Berg who wrote the inotify code in dovecot. He suggested I post here to LKML since his opinion is that this to be a kernel bug.
Allow me to jump in at this point. The small tool below triggers this problem for Reuben (confirmed via private mail) but works fine for me on 2.6.13-rc6.
johannes
--begin test program--
/* Author: Johannes Berg <[email protected]>
*
* This is a small inotify test program that simply
* repeatedly adds and removes watches.
*/
#include <stdio.h>
#include <linux/inotify.h>
#include <linux/inotify-syscalls.h>
int main()
{
int fd;
int wd1, wd2;
int ret;
fd = inotify_init();
if (fd < 0)
perror("inotify_init");
printf("inotify_init returned fd %d\n", fd);
while (1) {
wd1 = inotify_add_watch(fd, "/tmp", IN_ALL_EVENTS);
if (wd1 < 0) {
perror("inotify_add_watch");
break;
}
printf("inotify_add_watch returned wd1 %d\n", wd1);
wd2 = inotify_add_watch(fd, "/", IN_ALL_EVENTS);
if (wd2 < 0) {
perror("inotify_add_watch");
break;
}
printf("inotify_add_watch returned wd2 %d\n", wd2);
ret = inotify_rm_watch(fd, wd1);
if (ret < 0) {
perror("inotify_rm_watch wd1");
break;
}
ret = inotify_rm_watch(fd, wd2);
if (ret < 0) {
perror("inotify_rm_watch wd2");
break;
}
}
}Quoted 9 lines · Johannes Berg
Hi,
Quoted 3 lines
I have also observed another problem with inotify with dovecot - so I spoke with Johannes Berg who wrote the inotify code in dovecot. He suggested I post here to LKML since his opinion is that this to be a kernel bug.
Allow me to jump in at this point. The small tool below triggers this problem for Reuben (confirmed via private mail) but works fine for me on 2.6.13-rc6.
On 2.6.13-rc7 the test program fails. It always fails when a wd == 1024. If I skip inotify_rm_watch when wd == 1024, it will fail at wd == 2048. It seems the idr layer has an aversion to multiples of 1024.
When I run your test program I get this a lot:
inotify_add_watch returned wd1 5inotify_add_watch returned wd2 6inotify_add_watch returned wd1 6inotify_add_watch returned wd2 7
The pattern of
add_watch wd1 = Xadd_watch wd2 = X+1rm_watch Xrm_watch X+1add_watch wd1 = X+1add_watch wd2 = X+2
Should never happen. We tell the idr layer to always give us something bigger than the last wd we received.
Also, idr_get_new_above doesn't work all the time. Under 2.6.13-rc7, I added this to inotify.c:359:
if (ret <= dev->last_wd) {
printk(KERN_INFO "idr_get_new_above returned <= dev->last_wd\n");
}I get that message a lot. I know I have said this before (and was wrong) but I think the idr layer is busted.
Signature · 1 line
John McCutchan <[email protected]>
Quoted 2 lines · John McCutchan
I get that message a lot. I know I have said this before (and was wrong) but I think the idr layer is busted.
This time I think I agree with you. ;-)
Let's just pass zero for the "above" parameter in idr_get_new_above(), which is I believe the behavior of the other interface, and see if the 1024-multiple problem goes away. We definitely did not have that before.
If it does, and we don't have another solution, let's run with that for 2.6.13. I don't want this bug released.
Robert Love
Quoted 4 lines · Robert Love
Let's just pass zero for the "above" parameter in
idr_get_new_above(), which is I believe the behavior of the other interface, and see if the 1024-multiple problem goes away. We definitely did not have that before.
Will we then need to test if it fails for more than 1024 watches?
If I adjust the program to
1) create /tmp/test/%d
2) watch /tmp/test/%d
3) repeat
it fails on 2.6.13-rc6 as soon as the device is full and doesn't hold any more directories.
johannes
Quoted 9 lines · Johannes Berg
Quoted 4 lines · Robert Love
Let's just pass zero for the "above" parameter in
idr_get_new_above(), which is I believe the behavior of the other interface, and see if the 1024-multiple problem goes away. We definitely did not have that before.Will we then need to test if it fails for more than 1024 watches?
Quoted 8 lines
If I adjust the program to
1) create /tmp/test/%d
2) watch /tmp/test/%d
3) repeatit fails on 2.6.13-rc6 as soon as the device is full and doesn't hold any more directories.
Could you send me the new test program?
Signature · 1 line
John McCutchan <[email protected]>
Quoted 2 lines · John McCutchan
Quoted 2 lines
it fails on 2.6.13-rc6 as soon as the device is full and doesn't hold any more directories.
Obviously this wasn't true, I was hitting the 8192 watches limit and misinterpreted the error message. I just tested up to 100000 watches with this program.
Quoted 1 line
Could you send me the new test program?
Below.
johannes
/* Author: Johannes Berg <[email protected]>
*
* This is another small inotify test program that simply
* repeatedly adds watches.
*/
#include <stdio.h>
#include <linux/inotify.h>
#include <linux/inotify-syscalls.h>
int main()
{
int fd;
int wd1, wd2;
int ret, i = 0;
char buf[1024];
fd = inotify_init();
if (fd < 0)
perror("inotify_init");
printf("inotify_init returned fd %d\n", fd);
mkdir("/tmp/inotify-test-dir-rm-rf-this", 0777);
while (1) {
i++;
snprintf(buf,sizeof(buf),"/tmp/inotify-test-dir-rm-rf-this/%d",i>>10);
mkdir(buf,0777);
snprintf(buf,sizeof(buf),"/tmp/inotify-test-dir-rm-rf-this/%d/%d",i>>10,i);
mkdir(buf, 0777);
wd1 = inotify_add_watch(fd, buf, IN_ALL_EVENTS);
if (wd1 < 0) {
perror("inotify_add_watch");
break;
}
printf("inotify_add_watch returned wd1 %d\n", wd1);
}
}Quoted 7 lines · Johannes Berg
Quoted 2 lines · John McCutchan
Quoted 2 lines
it fails on 2.6.13-rc6 as soon as the device is full and doesn't hold any more directories.
Obviously this wasn't true, I was hitting the 8192 watches limit and misinterpreted the error message. I just tested up to 100000 watches with this program.
Thanks. The program runs fine, it climbs up until 8192. This confirmed a hunch I had. The problem only manifests itself when the lower idr numbers aren't actually being used. The thread
http://www.redhat.com/archives/dm-devel/2004-July/msg00003.html seems somewhat related to this problem, but it suggests that this was fixed in 2.6.7.
Signature · 1 line
John McCutchan <[email protected]>
Quoted 5 lines · Robert Love
Quoted 2 lines · John McCutchan
I get that message a lot. I know I have said this before (and was wrong) but I think the idr layer is busted.
Quoted 7 lines
This time I think I agree with you. ;-)
Let's just pass zero for the "above" parameter in
idr_get_new_above(), which is I believe the behavior of the other interface, and see if the 1024-multiple problem goes away. We definitely did not have that before.
I will test this.
Quoted 2 lines
If it does, and we don't have another solution, let's run with that for 2.6.13. I don't want this bug released.
I really don't want 2.6.13 to go out with this bug or the compromise. If we use 0, we will have a lot of wd re-use. Which will cause "strange" problems in inotify using applications that cleanup upon receipt of an IN_IGNORE event.
The problem will manifest it self when a program does this:
inotify_add_watch "/x" returns 1 inotify_rm_watch 1
[IN_IGNORE event is queued with wd == 1] inotify_add_watch "/y" returns 1 application reads events
cleans up data structures associated with wd == 1.
Signature · 1 line
John McCutchan <[email protected]>
Quoted 4 lines · John McCutchan
I really don't want 2.6.13 to go out with this bug or the compromise. If we use 0, we will have a lot of wd re-use. Which will cause "strange" problems in inotify using applications that cleanup upon receipt of an
IN_IGNOREevent.
What happens when, given bug-free code that doesn't reuse, you hit the 8192 limit with wd, even if they're not all open at the same time? Does it still add them, or will inotify give an error? And does the idr layer handle something like that gracefully without using lots of memory?
The background is that the process using this is potentially quite long-running and keeps opening/closing wds, so 8192 doesn't sound like a high barrier, after all Reuben observed hitting the 1024 limit after 15 minutes or so.
johannes
Quoted 7 lines · Johannes Berg
Quoted 4 lines · John McCutchan
I really don't want 2.6.13 to go out with this bug or the compromise. If we use 0, we will have a lot of wd re-use. Which will cause "strange" problems in inotify using applications that cleanup upon receipt of an
IN_IGNOREevent.
Quoted 5 lines
What happens when, given bug-free code that doesn't reuse, you hit the 8192 limit with wd, even if they're not all open at the same time? Does it still add them, or will inotify give an error? And does the idr layer handle something like that gracefully without using lots of memory?
The 8192 limit is the total watches in use at one time. If over time, you use more than 8192 watches but not all at one time you will just keep getting higher and higher wd's. I'm not 100% sure if idr handles it gracefully.
Quoted 4 lines
The background is that the process using this is potentially quite long-running and keeps opening/closing wds, so 8192 doesn't sound like a high barrier, after all Reuben observed hitting the 1024 limit after 15 minutes or so.
There isn't a 1024 limit, that is a bug in the idr code. He got to that number by having used 1024 wd's throughout the life of the program.'
FYI, if we pass 0 to idr_get_new_above, both your test programs work fine. But the first one re-uses wd's (back and forth between 0 & 1).
Signature · 1 line
John McCutchan <[email protected]>
Quoted 5 lines · John McCutchan
On 2.6.13-rc7 the test program fails. It always fails when a wd == 1024. If I skip
inotify_rm_watchwhen wd == 1024, it will fail at wd == 2048. It seems the idr layer has an aversion to multiples of 1024.When I run your test program I get this a lot:
I forgot to mention this -- but I just get (on -rc6):
inotify_add_watch returned wd1 0 inotify_add_watch returned wd2 1 inotify_add_watch returned wd1 0 inotify_add_watch returned wd2 1 inotify_add_watch returned wd1 0 inotify_add_watch returned wd2 1
etc.
IOW, they are directly reused, not even with the +1 offset you were seeing.
johannes
Quoted 8 lines · Johannes Berg
Quoted 5 lines · John McCutchan
On 2.6.13-rc7 the test program fails. It always fails when a wd == 1024. If I skip
inotify_rm_watchwhen wd == 1024, it will fail at wd == 2048. It seems the idr layer has an aversion to multiples of 1024.When I run your test program I get this a lot:
Quoted 8 lines
I forgot to mention this -- but I just get (on -rc6):
inotify_add_watchreturned wd1 0inotify_add_watchreturned wd2 1inotify_add_watchreturned wd1 0inotify_add_watchreturned wd2 1inotify_add_watchreturned wd1 0inotify_add_watchreturned wd2 1
Yeah, pre -rc7 we were always passing in 0 to idr_get_new_above. With rc7 we pass in the last wd returned.
Signature · 1 line
John McCutchan <[email protected]>
Quoted 49 lines · Reuben Farrelly
Hi,
I have also observed another problem with inotify with dovecot - so I spoke with Johannes Berg who wrote the inotify code in dovecot. He suggested I post here to LKML since his opinion is that this to be a kernel bug.
The problem I am observing is this, logged by dovecot after a period of time when a client is connected:
dovecot: Aug 22 14:31:23 Error: IMAP(gilly):
inotify_rm_watch()failed:
Invalid argument
dovecot: Aug 22 14:31:23 Error: IMAP(gilly):inotify_rm_watch()failed:
Invalid argument
dovecot: Aug 22 14:31:23 Error: IMAP(gilly):inotify_rm_watch()failed:
Invalid argumentMultiply that by about 1000 ;-)
Some debugging shows this:
dovecot: Aug 25 19:31:22 Warning: IMAP(gilly): removing wd 1019 from inotify fd 4
dovecot: Aug 25 19:31:22 Warning: IMAP(gilly): removing wd 1018 from inotify fd 4
dovecot: Aug 25 19:31:22 Warning: IMAP(gilly):inotify_add_watchreturned 1019
dovecot: Aug 25 19:31:22 Warning: IMAP(gilly):inotify_add_watchreturned 1020
dovecot: Aug 25 19:31:23 Warning: IMAP(gilly): removing wd 1020 from inotify fd 4
dovecot: Aug 25 19:31:23 Warning: IMAP(gilly): removing wd 1019 from inotify fd 4
dovecot: Aug 25 19:31:24 Warning: IMAP(gilly):inotify_add_watchreturned 1020dovecot: Aug 25 19:31:24 Warning: IMAP(gilly):
inotify_add_watchreturned 1021
dovecot: Aug 25 19:31:24 Warning: IMAP(gilly): removing wd 1021 from inotify fd 4
dovecot: Aug 25 19:31:24 Warning: IMAP(gilly): removing wd 1020 from inotify fd 4
dovecot: Aug 25 19:31:25 Warning: IMAP(gilly):inotify_add_watchreturned 1021
dovecot: Aug 25 19:31:25 Warning: IMAP(gilly):inotify_add_watchreturned 1022
dovecot: Aug 25 19:31:25 Warning: IMAP(gilly): removing wd 1022 from inotify fd 4
dovecot: Aug 25 19:31:25 Warning: IMAP(gilly): removing wd 1021 from inotify fd 4
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly):inotify_add_watchreturned 1022
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly):inotify_add_watchreturned 1023
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): removing wd 1023 from inotify fd 4
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): removing wd 1022 from inotify fd 4
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly):inotify_add_watchreturned 1023
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly):inotify_add_watchreturned 1024
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): removing wd 1024 from inotify fd 4
dovecot: Aug 25 19:31:27 Error: IMAP(gilly):inotify_rm_watch()failed:
Invalid argument
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): removing wd 1023 from inotify fd 4
dovecot: Aug 25 19:31:28 Warning: IMAP(gilly):inotify_add_watchreturned 1024
dovecot: Aug 25 19:31:28 Warning: IMAP(gilly):inotify_add_watchreturned 1024
Quoted 2 lines
Note the incrementing wd value even though we are removing them as we go..
What kernel are you running? The wd's should ALWAYS be incrementing, you should never get the same wd as you did before. From your log, you are getting the same wd (after you inotify_rm_watch it). I can reproduce this bug on 2.6.13-rc7.
idr_get_new_above
isn't returning something above.
Also, the idr layer seems to be breaking when we pass in 1024. I can reproduce that on my 2.6.13-rc7 system as well.
Quoted 5 lines
This is using latest CVS of dovecot code and with 2.6.12-rc6-mm(1|2) kernel.
Robert, John, what do you think? Is this possibly related to the oops seen
in the log that I reported earlier? (Which is still showing up 2-3 times per day, btw)
There is definitely something broken here.
Signature · 1 line
John McCutchan <[email protected]>
Quoted 65 lines · John McCutchan
Quoted 49 lines · Reuben Farrelly
Hi,
I have also observed another problem with inotify with dovecot - so I spoke with Johannes Berg who wrote the inotify code in dovecot. He suggested I post here to LKML since his opinion is that this to be a kernel bug.
The problem I am observing is this, logged by dovecot after a period of time when a client is connected:
dovecot: Aug 22 14:31:23 Error: IMAP(gilly):
inotify_rm_watch()failed:
Invalid argument
dovecot: Aug 22 14:31:23 Error: IMAP(gilly):inotify_rm_watch()failed:
Invalid argument
dovecot: Aug 22 14:31:23 Error: IMAP(gilly):inotify_rm_watch()failed:
Invalid argumentMultiply that by about 1000 ;-)
Some debugging shows this:
dovecot: Aug 25 19:31:22 Warning: IMAP(gilly): removing wd 1019 from inotify fd 4
dovecot: Aug 25 19:31:22 Warning: IMAP(gilly): removing wd 1018 from inotify fd 4
dovecot: Aug 25 19:31:22 Warning: IMAP(gilly):inotify_add_watchreturned 1019
dovecot: Aug 25 19:31:22 Warning: IMAP(gilly):inotify_add_watchreturned 1020
dovecot: Aug 25 19:31:23 Warning: IMAP(gilly): removing wd 1020 from inotify fd 4
dovecot: Aug 25 19:31:23 Warning: IMAP(gilly): removing wd 1019 from inotify fd 4
dovecot: Aug 25 19:31:24 Warning: IMAP(gilly):inotify_add_watchreturned 1020dovecot: Aug 25 19:31:24 Warning: IMAP(gilly):
inotify_add_watchreturned 1021
dovecot: Aug 25 19:31:24 Warning: IMAP(gilly): removing wd 1021 from inotify fd 4
dovecot: Aug 25 19:31:24 Warning: IMAP(gilly): removing wd 1020 from inotify fd 4
dovecot: Aug 25 19:31:25 Warning: IMAP(gilly):inotify_add_watchreturned 1021
dovecot: Aug 25 19:31:25 Warning: IMAP(gilly):inotify_add_watchreturned 1022
dovecot: Aug 25 19:31:25 Warning: IMAP(gilly): removing wd 1022 from inotify fd 4
dovecot: Aug 25 19:31:25 Warning: IMAP(gilly): removing wd 1021 from inotify fd 4
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly):inotify_add_watchreturned 1022
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly):inotify_add_watchreturned 1023
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): removing wd 1023 from inotify fd 4
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): removing wd 1022 from inotify fd 4
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly):inotify_add_watchreturned 1023
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly):inotify_add_watchreturned 1024
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): removing wd 1024 from inotify fd 4
dovecot: Aug 25 19:31:27 Error: IMAP(gilly):inotify_rm_watch()failed:
Invalid argument
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): removing wd 1023 from inotify fd 4
dovecot: Aug 25 19:31:28 Warning: IMAP(gilly):inotify_add_watchreturned 1024
dovecot: Aug 25 19:31:28 Warning: IMAP(gilly):inotify_add_watchreturned 1024Quoted 2 lines
Note the incrementing wd value even though we are removing them as we go..
What kernel are you running? The wd's should ALWAYS be incrementing, you should never get the same wd as you did before. From your log, you are getting the same wd (after you
inotify_rm_watchit). I can reproduce this bug on 2.6.13-rc7.
idr_get_new_aboveisn't returning something above.
Also, the idr layer seems to be breaking when we pass in 1024. I can reproduce that on my 2.6.13-rc7 system as well.
Quoted 7 lines
Quoted 5 lines
This is using latest CVS of dovecot code and with 2.6.12-rc6-mm(1|2) kernel.
Robert, John, what do you think? Is this possibly related to the oops seen
in the log that I reported earlier? (Which is still showing up 2-3 times per day, btw)There is definitely something broken here.
Jim, George-
We are seeing a problem in the idr layer. If we do idr_find(1024) when, say, a low valued idr, like, zero, is unallocated, NULL is returned.
This readily manifests itself in inotify, where we recently switched to using idr_get_new_above() with our last allocated token.
Robert Love
Quoted 4 lines · Robert Love
Quoted 2 lines · John McCutchan
On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote:
~
Quoted 29 lines
Quoted 29 lines
Quoted 11 lines
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): removing wd 1022 from inotify fd 4
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly):inotify_add_watchreturned 1023
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly):inotify_add_watchreturned 1024
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): removing wd 1024 from inotify fd 4
dovecot: Aug 25 19:31:27 Error: IMAP(gilly):inotify_rm_watch()failed: Invalid argument
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): removing wd 1023 from inotify fd 4
dovecot: Aug 25 19:31:28 Warning: IMAP(gilly):inotify_add_watchreturned 1024
dovecot: Aug 25 19:31:28 Warning: IMAP(gilly):inotify_add_watchreturned 1024Note the incrementing wd value even though we are removing them as we go..
What kernel are you running? The wd's should ALWAYS be incrementing, you should never get the same wd as you did before. From your log, you are getting the same wd (after you
inotify_rm_watchit). I can reproduce this bug on 2.6.13-rc7.
idr_get_new_aboveisn't returning something above.
Also, the idr layer seems to be breaking when we pass in 1024. I can reproduce that on my 2.6.13-rc7 system as well.
Quoted 3 lines
This is using latest CVS of dovecot code and with 2.6.12-rc6-mm(1|2) kernel.
Robert, John, what do you think? Is this possibly related to the oops seen in the log that I reported earlier? (Which is still showing up 2-3 times per day, btw)
Quoted 7 lines
Quoted 1 line
There is definitely something broken here.
Jim, George-
We are seeing a problem in the idr layer. If we do
idr_find(1024)when, say, a low valued idr, like, zero, is unallocated, NULL is returned.
I think the best thing is to take idr into user space and emulate the problem usage. To this end, from the log it appears that you _might_ be moving between 0, 1 and 2 entries increasing the number each time. It also appears that the failure happens here:
add 1023
add 1024
find 1024 or is it the remove that fails? It also looks like 1024 got allocated twice. Am I reading the log correctly?
So, is it correct to assume that the tree is empty save these two at this time? I am just trying to figure out what the test program needs to do.
Signature · 2 lines
George Anzinger [email protected] HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
Quoted 2 lines · George Anzinger
I think the best thing is to take idr into user space and emulate the problem usage.
Good plan, I guess. Do you think that's easy?
Quoted 7 lines
To this end, from the log it appears that you _might_ be moving between 0, 1 and 2 entries increasing the number each time. It also appears that the failure happens here:
add 1023
add 1024
find 1024 or is it the remove that fails? It also looks like 1024 got allocated twice. Am I reading the log correctly?
Remove 1024 fails, but add(please make it >1024) seems to return 1024, and find(1024) also seems to fail. Well, remove() probably has to find(), but I'm not really sure what inotify does (maybe find first, to see if it's valid).
Quoted 3 lines
So, is it correct to assume that the tree is empty save these two at this time? I am just trying to figure out what the test program needs to do.
Yes, but all the smaller ones have been in it at some point in time.
johannes
Quoted 15 lines · Johannes Berg
Quoted 2 lines · George Anzinger
I think the best thing is to take idr into user space and emulate the problem usage.
Good plan, I guess. Do you think that's easy?
Quoted 7 lines
To this end, from the log it appears that you _might_ be moving between 0, 1 and 2 entries increasing the number each time. It also appears that the failure happens here:
add 1023
add 1024
find 1024 or is it the remove that fails? It also looks like 1024 got allocated twice. Am I reading the log correctly?
Quoted 4 lines
Remove 1024 fails, but add(please make it >1024) seems to return 1024, and find(1024) also seems to fail. Well,
remove()probably has tofind(), but I'm not really sure what inotify does (maybe find first, to see if it's valid).
Just to clarify, the remove() he is talking about isn't idr_remove, it is inotify's remove. idr_find() is failing at 1024 which causes inotify's remove to fail.
Signature · 1 line
John McCutchan <[email protected]>
Quoted 46 lines · George Anzinger
Quoted 4 lines · Robert Love
Quoted 2 lines · John McCutchan
On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote:
~
Quoted 32 lines
Quoted 32 lines
Quoted 10 lines
dovecot: Aug 25 19:31:26 Warning: IMAP(gilly): removing wd 1022 from inotify fd 4
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly):inotify_add_watchreturned 1023
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly):inotify_add_watchreturned 1024
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): removing wd 1024 from inotify fd 4
dovecot: Aug 25 19:31:27 Error: IMAP(gilly):inotify_rm_watch()failed:
Invalid argument
dovecot: Aug 25 19:31:27 Warning: IMAP(gilly): removing wd 1023 from inotify fd 4
dovecot: Aug 25 19:31:28 Warning: IMAP(gilly):inotify_add_watchreturned 1024
dovecot: Aug 25 19:31:28 Warning: IMAP(gilly):inotify_add_watchreturned 1024Quoted 2 lines
Note the incrementing wd value even though we are removing them as we go..
What kernel are you running? The wd's should ALWAYS be incrementing, you should never get the same wd as you did before. From your log, you are getting the same wd (after you
inotify_rm_watchit). I can reproduce this bug on 2.6.13-rc7.
idr_get_new_aboveisn't returning something above.
Also, the idr layer seems to be breaking when we pass in 1024. I can reproduce that on my 2.6.13-rc7 system as well.
Quoted 5 lines
This is using latest CVS of dovecot code and with 2.6.12-rc6-mm(1|2) kernel.
Robert, John, what do you think? Is this possibly related to the oops seen
in the log that I reported earlier? (Which is still showing up 2-3 times per day, btw)Quoted 7 lines
Quoted 1 line
There is definitely something broken here.
Jim, George-
We are seeing a problem in the idr layer. If we do
idr_find(1024)when, say, a low valued idr, like, zero, is unallocated, NULL is returned.
Quoted 8 lines
I think the best thing is to take idr into user space and emulate the problem usage. To this end, from the log it appears that you _might_ be moving between 0, 1 and 2 entries increasing the number each time. It also appears that the failure happens here:
add 1023
add 1024
find 1024 or is it the remove that fails? It also looks like 1024 got allocated twice. Am I reading the log correctly?
You are reading the log correctly. There are two bugs. One is that if we pass X to idr_get_new_above, it can return X again (doesn't ever seem to return < X). The other problem is that the find fails on 1024 (and 2048 if we skip 1024).
Quoted 4 lines
So, is it correct to assume that the tree is empty save these two at this time? I am just trying to figure out what the test program needs to do.
Yes that is the exact scenario. Only 2 id's are used at any given time, and once we hit 1024 things break. This doesn't happen when the tree is not empty.
Thanks for looking at this!
Signature · 1 line
John McCutchan <[email protected]>
Quoted 8 lines · John McCutchan
Quoted 6 lines · George Anzinger
Quoted 4 lines · Robert Love
Quoted 1 line · John McCutchan
On Thu, 2005-08-25 at 22:07 +1200, Reuben Farrelly wrote:
~
Quoted 10 lines
Quoted 4 lines
I think the best thing is to take idr into user space and emulate the problem usage. To this end, from the log it appears that you _might_ be moving between 0, 1 and 2 entries increasing the number each time. It also appears that the failure happens here:
add 1023
add 1024
find 1024 or is it the remove that fails? It also looks like 1024 got allocated twice. Am I reading the log correctly?You are reading the log correctly. There are two bugs. One is that if we pass X to
idr_get_new_above, it can return X again (doesn't ever seem to return < X). The other problem is that the find fails on 1024 (and 2048 if we skip 1024).
That IS strange. 1024 is on a "level" boundry, but then next level is 2**15, not 2**11. I will take a look.
Quoted 10 lines
Quoted 3 lines
So, is it correct to assume that the tree is empty save these two at this time? I am just trying to figure out what the test program needs to do.
Yes that is the exact scenario. Only 2 id's are used at any given time, and once we hit 1024 things break. This doesn't happen when the tree is not empty.
Thanks for looking at this!
Signature · 2 lines
George Anzinger [email protected] HRT (High-res-timers): http://sourceforge.net/projects/high-res-timers/
Quoted 2 lines · George Anzinger
That IS strange. 1024 is on a "level" boundry, but then next level is 2**15, not 2**11. I will take a look.
Remember that the level is never filled, so maybe the smallest level just gets an offset or something? Well, you're the expert I suppose, so apologies if this didn't make sense. Just crossed my mind :)
johannes