Many open/close on same files yeilds "No such file or directory".
Subsystems (2)
Hi list.
I have a "fairly" reproducible problem. When a program opens and closes the same file many times, it eventually ends up with a "no such file or directory". Test program that can reproduce the problem on my setup:
root@hest:~# cat test-file-c.c #include <stdlib.h> #include <stdio.h> #include <fcntl.h> #include <unistd.h>
int main(int argc, char *argv[]) {
unsigned long i=0;
int fh;
char *filename;
filename=argv[1];
while(1) {
fh=open(filename, O_RDONLY);
if (fh==-1) {
printf("Failed to open %s\n", filename);
printf("Open number: %ld\n",i);
exit(10);
}
close(fh);
i++;
} exit(0);
}
root@hest:~# ./test-file-c /z/bio/databases/online/index/index-by-accno Failed to open /z/bio/databases/online/index/index-by-accno Open number: 61785000 root@hest:~# ./test-file-c /z/bio/databases/online/index/index-by-accno Failed to open /z/bio/databases/online/index/index-by-accno Open number: 120929685 (The problem is not isolate to a single file on the filesystem).
strace on the program reviel that the system indeed return a "No such file or directory" to the program.
This is run on an Ubuntu Gutsy (vendor kernel): 2.6.22-14-server on an 4.5TB ext3 filesystem on an LVM volume. The volume was created on a dapper (2 releases back) and has just followed with during upgrades. I cannot reproduce it on other disks attached to the same server or on other servers attached to similar disksystems.
The filesystem was taken offline yesterday for a forced fsck and it was found to be clean.
The diskarray is a quite old Fibrenetix FX1200 with 12xPATA disk
in raid5 (with hotspare) exposed to the OS as 3 SCSI-disks of
2+2+0.5TB assembled with LVM afterwards. The SCSI-controller is a:
05:08.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev c1)
What suggestions do you have to solve this problem?
I'm about to mkfs.ext3 the volume and spool it back in from the backup, but somehow I'm not convinced that it will solve the problem at all. It may just be a hardware problem, but dmesg doesnt tell anything.
We actually got the problem from a perl-script, but this seems to be the minimal program that reproduces the problem.
Jesper
Signature · 1 line
Jesper
Quoted 31 lines · On Thu, 01 May 2008 17:34:46 +0200 Jesper Krogh <[email protected]> wrote:
Hi list.
I have a "fairly" reproducible problem. When a program opens and closes the same file many times, it eventually ends up with a "no such file or directory". Test program that can reproduce the problem on my setup:
root@hest:~# cat test-file-c.c #include <stdlib.h> #include <stdio.h> #include <fcntl.h> #include <unistd.h>int main(int argc, char *argv[]) { unsigned long i=0; int fh; char *filename; filename=argv[1]; while(1) { fh=open(filename, O_RDONLY); if (fh==-1) { printf("Failed to open %s\n", filename); printf("Open number: %ld\n",i); exit(10); } close(fh); i++; } exit(0);
Quoted 9 lines
} root@hest:~# ./test-file-c /z/bio/databases/online/index/index-by-accno Failed to open /z/bio/databases/online/index/index-by-accno Open number: 61785000 root@hest:~# ./test-file-c /z/bio/databases/online/index/index-by-accno Failed to open /z/bio/databases/online/index/index-by-accno Open number: 120929685 (The problem is not isolate to a single file on the filesystem).
What an amazing bug.
Quoted 6 lines
strace on the program reviel that the system indeed return a "No such file or directory" to the program.
This is run on an Ubuntu Gutsy (vendor kernel): 2.6.22-14-server on an 4.5TB ext3 filesystem on an LVM volume. The volume was created on a dapper (2 releases back) and has just followed with during upgrades.
The test program is (almost) all in RAM and won't care about the hardware.
Quoted 2 lines
I cannot reproduce it on other disks attached to the same server or on other servers attached to similar disksystems.
hmm.
I guess it would be interesting to remount that filesystem with `noatime' to eliminate the last bit of I/O and block-=realted code.
Quoted 9 lines
The filesystem was taken offline yesterday for a forced fsck and it was found to be clean.
The diskarray is a quite old Fibrenetix FX1200 with 12xPATA disk in raid5 (with hotspare) exposed to the OS as 3 SCSI-disks of 2+2+0.5TB assembled with LVM afterwards. The SCSI-controller is a:
05:08.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev c1)
Quoted 8 lines
What suggestions do you have to solve this problem?
I'm about to mkfs.ext3 the volume and spool it back in from the backup, but somehow I'm not convinced that it will solve the problem at all. It may just be a hardware problem, but dmesg doesnt tell anything.
We actually got the problem from a perl-script, but this seems to be the minimal program that reproduces the problem.
I'd suspect that after 1e8 loops your CPU got too hot and started to misbehave.
Quoted 7 lines · Andrew Morton
Quoted 2 lines
I cannot reproduce it on other disks attached to the same server or on other servers attached to similar disksystems.
hmm.
I guess it would be interesting to remount that filesystem with `noatime' to eliminate the last bit of I/O and block-=realted code.
It is allready mounted noatime:/dev/mapper/fx1200_vg-fx1200_lv on /z/fx1200 type ext3 (rw,noatime)
Quoted 9 lines
Quoted 6 lines
I'm about to mkfs.ext3 the volume and spool it back in from the backup, but somehow I'm not convinced that it will solve the problem at all. It may just be a hardware problem, but dmesg doesnt tell anything.
We actually got the problem from a perl-script, but this seems to be the minimal program that reproduces the problem.
I'd suspect that after 1e8 loops your CPU got too hot and started to misbehave.
Hardware is an Sun Fire X4600 (8xdual-core AMD64 processors). The problem seem to be tied to this filesystem. (I cannot havent been able to reproduce it on the /-mounted disk of the same system. So if a cpu problem.. then it shouldn't be tied to a specific filesystem?
This is the only activity on the system .. so a load of 1 / 16cpus.
The system are generally rock-solid.
Jesper
Signature · 1 line
Jesper
Quoted 6 lines · On Fri, 02 May 2008 10:20:20 +0200 Jesper Krogh <[email protected]> wrote:
Hardware is an Sun Fire X4600 (8xdual-core AMD64 processors). The problem seem to be tied to this filesystem. (I cannot havent been able to reproduce it on the /-mounted disk of the same system. So if a cpu problem.. then it shouldn't be tied to a specific filesystem?
This is the only activity on the system .. so a load of 1 / 16cpus.
have you run fsck on the filesystem?
(and it might be interesting to use dump the metadata to a file first to save the broken state for later analysis)
Re: Many open/close on same files yeilds
Quoted 5 lines · Arjan van de Ven <arjan <at> infradead.org> writes:
Quoted 1 line · On Fri, 02 May 2008 10:20:20 +0200 Jesper Krogh <jesper <at> krogh.cc> wrote:
This is the only activity on the system .. so a load of 1 / 16cpus.
have you run fsck on the filesystem?
Yes.. the first thing i did was a forced fsck .. and it was clean.
Quoted 2 lines
(and it might be interesting to use dump the metadata to a file first to save the broken state for later analysis)
All 4.5TB?
Signature · 1 line
Jesper
Re: Many open/close on same files yeilds
Quoted 6 lines · On Fri, 2 May 2008 11:03:16 +0000 (UTC) Jesper Krogh <[email protected]> wrote:
Quoted 5 lines · Arjan van de Ven <arjan <at> infradead.org> writes:
Quoted 2 lines · On Fri, 02 May 2008 10:20:20 +0200 Jesper Krogh <jesper <at> krogh.cc> wrote:
This is the only activity on the system .. so a load of 1 / 16cpus.
Quoted 8 lines
Quoted 1 line
have you run fsck on the filesystem?
Yes.. the first thing i did was a forced fsck .. and it was clean.
Quoted 2 lines
(and it might be interesting to use dump the metadata to a file first to save the broken state for later analysis)
All 4.5TB?
metdata only; the "e2image" program only saves metadata, not data... so it'll be a lot less than 4.5tb..
Quoted 1 line
Quoted 9 lines · Jesper Krogh
Quoted 2 lines
I'd suspect that after 1e8 loops your CPU got too hot and started to misbehave.
Hardware is an Sun Fire X4600 (8xdual-core AMD64 processors). The problem seem to be tied to this filesystem. (I cannot havent been able to reproduce it on the /-mounted disk of the same system. So if a cpu problem.. then it shouldn't be tied to a specific filesystem?
This is the only activity on the system .. so a load of 1 / 16cpus.
I've tried to explore this suggestion (the best I could).
There are 2 ext3 filesystems locally mounted. / and this one. Running 16 parallel runs of this program on a file on the /-mounted filesystem cannot reproduce the problem. If it was linked to hot hardware, I believe I should be able to reproduce it this way. The servers are in a 17 degress serverroom.
It changes alot when.. it actually happens. The "earliest ones" has been from 200000 cycles.
Ive tried to upgrade.. so the system is now running a 2.6.24-16-server (Ubuntu/Hardy) kernel. But execpt from differnet output in uname, everything else seems to be the same.
All suggestions/ideas are welcome? The general feeling is that it wont change a thing to reformat the filesystem?
Jesper
Signature · 1 line
Jesper
Quoted 17 lines · Jesper Krogh
Quoted 5 lines · Jesper Krogh
Quoted 4 lines
I'd suspect that after 1e8 loops your CPU got too hot and started to misbehave.
Quoted 7 lines
Hardware is an Sun Fire X4600 (8xdual-core AMD64 processors). The problem seem to be tied to this filesystem. (I cannot havent been able to reproduce it on the /-mounted disk of the same system. So if a cpu problem.. then it shouldn't be tied to a specific filesystem?
This is the only activity on the system .. so a load of 1 / 16cpus.
I've tried to explore this suggestion (the best I could).
Quoted 8 lines
There are 2 ext3 filesystems locally mounted. / and this one. Running 16
parallel runs of this program on a file on the /-mounted filesystem cannot reproduce the problem. If it was linked to hot hardware, I believe I should be able to reproduce it this way. The servers are in a 17 degress serverroom.It changes alot when.. it actually happens. The "earliest ones" has been
from 200000 cycles.
Run 16 in parallel on /, and another 16 simultaneously on the trouble filesystem? If you continue to get errors only on the 'trouble' filesystem, and no errors start occurring on / coincident, then it sounds pretty localized.
BTW, I may have missed this earlier, but does it happen *anywhere* on the troublesome filesystem (ie, in a newly created subdirectory)?
Quoted 21 lines · Ray Lee
Quoted 20 lines · Jesper Krogh
Quoted 10 lines · Jesper Krogh
Quoted 3 lines
I'd suspect that after 1e8 loops your CPU got too hot and started to misbehave.
Hardware is an Sun Fire X4600 (8xdual-core AMD64 processors). The problem seem to be tied to this filesystem. (I cannot havent been able to reproduce it on the /-mounted disk of the same system. So if a cpu problem.. then it shouldn't be tied to a specific filesystem?
This is the only activity on the system .. so a load of 1 / 16cpus.
I've tried to explore this suggestion (the best I could).
There are 2 ext3 filesystems locally mounted. / and this one. Running 16
parallel runs of this program on a file on the /-mounted filesystem cannot reproduce the problem. If it was linked to hot hardware, I believe I should be able to reproduce it this way. The servers are in a 17 degress serverroom.
Quoted 7 lines
Quoted 2 lines
It changes alot when.. it actually happens. The "earliest ones" has been
from 200000 cycles.Run 16 in parallel on /, and another 16 simultaneously on the trouble filesystem? If you continue to get errors only on the 'trouble' filesystem, and no errors start occurring on / coincident, then it sounds pretty localized.
That test has been done. I can only reproduce it on this filesystem. But I cannot really conclude that it is only present there.. since sometimes my testprogram just goes on .. and dies past 1 billion cycles. But I have never gotten errors from the / filesystem on the same installation.
Quoted 2 lines
BTW, I may have missed this earlier, but does it happen *anywhere* on the troublesome filesystem (ie, in a newly created subdirectory)?
I'll run that test now.
Signature · 1 line
Jesper
Quoted 43 lines · Jesper Krogh
Quoted 15 lines · Ray Lee
Quoted 13 lines · Jesper Krogh
Quoted 10 lines · Jesper Krogh
Quoted 5 lines
I'd suspect that after 1e8 loops your CPU got too hot and started to misbehave.
Hardware is an Sun Fire X4600 (8xdual-core AMD64 processors). The problem seem to be tied to this filesystem. (I cannot havent been able to reproduce it on the /-mounted disk of the same system. So if a cpu problem.. then it shouldn't be tied to a specific filesystem?
Quoted 6 lines
Quoted 6 lines
Quoted 3 lines
This is the only activity on the system .. so a load of 1 / 16cpus.
I've tried to explore this suggestion (the best I could).
There are 2 ext3 filesystems locally mounted. / and this one. Running
16
Quoted 1 line
Quoted 1 line
parallel runs of this program on a file on the /-mounted filesystem
cannot
Quoted 1 line
Quoted 1 line
reproduce the problem. If it was linked to hot hardware, I believe I
should
Quoted 4 lines
Quoted 4 lines
be able to reproduce it this way. The servers are in a 17 degress serverroom.
It changes alot when.. it actually happens. The "earliest ones" has
been
Quoted 8 lines
Quoted 2 lines
from 200000 cycles.
Run 16 in parallel on /, and another 16 simultaneously on the trouble filesystem? If you continue to get errors only on the 'trouble' filesystem, and no errors start occurring on / coincident, then it sounds pretty localized.
Quoted 4 lines
That test has been done. I can only reproduce it on this filesystem. But I
cannot really conclude that it is only present there.. since sometimes my testprogram just goes on .. and dies past 1 billion cycles. But I have never gotten errors from the / filesystem on the same installation.
Sorry for belaboring the point, but reading up-thread I see you ran both / and /troublefs one after the other, but you're saying you also ran them at the same time? If so, that would seem to conclusively rule out hardware overheating.
Quoted 4 lines · Ray Lee
Sorry for belaboring the point, but reading up-thread I see you ran both / and /troublefs one after the other, but you're saying you also ran them at the same time? If so, that would seem to conclusively rule out hardware overheating.
I've run up to 32 instances of the program at the same time. Both on the / volumes and the "trouble" volume. Only programs from the trouble volume failed.
Jesper
Signature · 1 line
Jesper
Quoted 2 lines · Ray Lee
BTW, I may have missed this earlier, but does it happen *anywhere* on the troublesome filesystem (ie, in a newly created subdirectory)?
Yes. It is reproducible on a newly created subdirectory on the filesystem.
Jesper
Signature · 1 line
Jesper
Quoted 6 lines · Jesper Krogh
Quoted 2 lines · Ray Lee
BTW, I may have missed this earlier, but does it happen *anywhere* on the troublesome filesystem (ie, in a newly created subdirectory)?
Yes. It is reproducible on a newly created subdirectory on the filesystem.
Now I've created a new filesystem. (resized the original LVM-volume to half and created a new ext3 one next to it). The problem persists on this new filesystem.
Just guessing, can there be something timing related in clearing the FS-cache for the filesystem?
Any other suggestions/pointers.
Jesper
Signature · 1 line
Jesper
Quoted 8 lines · Jesper Krogh
Quoted 6 lines · Jesper Krogh
Quoted 2 lines · Ray Lee
BTW, I may have missed this earlier, but does it happen *anywhere* on the troublesome filesystem (ie, in a newly created subdirectory)?
Yes. It is reproducible on a newly created subdirectory on the filesystem.
Quoted 8 lines
Now I've created a new filesystem. (resized the original LVM-volume to half and created a new ext3 one next to it). The problem persists on this new filesystem.
Just guessing, can there be something timing related in clearing the FS-cache for the filesystem?
Any other suggestions/pointers.
Hi,
Has anyone else been able to reproduce this problem?
I ran the test program overnight with no problem.
Signature · 1 line
~Randy
Quoted 12 lines · Randy.Dunlap
Quoted 10 lines · Jesper Krogh
Quoted 5 lines · Jesper Krogh
Quoted 2 lines · Ray Lee
BTW, I may have missed this earlier, but does it happen *anywhere* on the troublesome filesystem (ie, in a newly created subdirectory)?
Yes. It is reproducible on a newly created subdirectory on the filesystem.
Now I've created a new filesystem. (resized the original LVM-volume to half and created a new ext3 one next to it). The problem persists on this new filesystem.
Quoted 8 lines
Quoted 4 lines
Just guessing, can there be something timing related in clearing the FS-cache for the filesystem?
Any other suggestions/pointers.
Hi,
Has anyone else been able to reproduce this problem?
No, and out of my systems. I only have this external SCSI-IDE-raid device I can reproduce it on. The internal system disks doesnt allow me to reproduce it on the same installation.
Signature · 1 line
Jesper Krogh
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 32 lines · Ray Lee
Quoted 31 lines · Jesper Krogh
Quoted 8 lines · Randy.Dunlap
Quoted 5 lines · Jesper Krogh
Quoted 3 lines · Jesper Krogh
Quoted 1 line · Ray Lee
BTW, I may have missed this earlier, but does it happen *anywhere*
on
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
Quoted 2 lines
the troublesome filesystem (ie, in a newly created subdirectory)?
Yes. It is reproducible on a newly created subdirectory on the filesystem.
Now I've created a new filesystem. (resized the original LVM-volume to
half
Quoted 4 lines
Quoted 4 lines
and created a new ext3 one next to it). The problem persists on this new filesystem.
Just guessing, can there be something timing related in clearing the
FS-cache
Quoted 8 lines
Quoted 4 lines
for the filesystem?
Any other suggestions/pointers.
Hi,
Has anyone else been able to reproduce this problem?
Quoted 7 lines
Quoted 3 lines
No, and out of my systems. I only have this external SCSI-IDE-raid device I
can reproduce it on. The internal system disks doesnt allow me to reproduce it on the same installation.I'd been meaning to ask what the topology was. External, eh? Are you sure the enclosure, cabling, and card/connectors are all good? Have you tried swapping out cables?
It is new SCSI-controller, new cable and new terminator put onto it. But (just enlighten me), if I had problems at this level I'd expect the serverlog to be full of SCSI/FS-related errors and not just a single syscall, that doesn't even touch the array due to caching, to be failing.
Jesper
Signature · 1 line
Jesper
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · Ray Lee
Quoted 4 lines · Jesper Krogh
Quoted 4 lines
I'd been meaning to ask what the topology was. External, eh? Are you sure the enclosure, cabling, and card/connectors are all good? Have you tried swapping out cables?
Quoted 7 lines
Quoted 5 lines
It is new SCSI-controller, new cable and new terminator put onto it. But (just enlighten me), if I had problems at this level I'd expect the serverlog to be full of SCSI/FS-related errors and not just a single syscall, that doesn't even touch the array due to caching, to be failing.
Borderline hardware does not always create logged errors.
Ok. I think this _really_ point to a kernel problem. (or just some broken hardware from Sun in multiple copies)
Quoted 3 lines
If I understood you correctly earlier, identical hardware on another system does not show the error. That, quite honestly, rules out the software.
Now I've moved the data to fresh ext3 filesystems on a storage-array based on iscsi. Mounted the filesystems to another, similar server and I can still reproduce the problem.
Both servers are 16 cores. The problem wasn't there on a different server with only 2 cores. (or I didn't run into it).
The 3 setups above has both been tested with a 2.6.22-14-server and 2.6.24-17-server (towards the iscsi volume).
Doing more testing show that I have 3 machines (all X4600, 16 cores/32GB ram that I can reproduce it on against different filesystem)
The more processes running on the system (accessing the FS volume), the easier it seems to get into the problem.
Quoted 2 lines
What's left, however unlikely, has to be the issue. And what's left is your scsi controller, the cable, and the external disk array.
Now I've removed all of them.. and still got the problem.
Signature · 1 line
Jesper
Quoted 3 lines · Jesper Krogh
Both servers are 16 cores. The problem wasn't there on a different server with only 2 cores. (or I didn't run into it).
The 3 setups above has both been tested with a 2.6.22-14-server and 2.6.24-17-server (towards the iscsi volume).
I can confirm that the problem persists on a 2.6.25.2 kernel.
Signature · 1 line
Jesper
Quoted 9 lines · Jesper Krogh
Quoted 5 lines · Andrew Morton
Quoted 2 lines
I cannot reproduce it on other disks attached to the same server or on other servers attached to similar disksystems.
I guess it would be interesting to remount that filesystem with `noatime' to eliminate the last bit of I/O and block-=realted code.
It is allready mounted noatime:
/dev/mapper/fx1200_vg-fx1200_lvon /z/fx1200 type ext3 (rw,noatime)
Watching vmstat 1 while running the program confirms that there are absolutely no block i/o. Everything happens purely in the cache of the OS.
Jesper
Signature · 1 line
Jesper
Hi.
A week has passed the problem still persist and I have done a fair amount of testing.
I have now been able to reproduce it on 2 different servers, a dual-core,8 cpu Sun X4600 amd64 and a 2 cpu single-core Sun V20Z.
It has been reproduced against 4 different diskarrays one of them being the IDE-SCSI(We have 2 of them) array mentioned before. The other one is an iSCSI SAN. And lately when trying to move of the "trouble systems" we hit the bug on a FC-SAN.
I belive that we can rule out hardware now.
I have both reproduced it on "old" ext3 volumes and freshly created ones.
I havent been able to reproduce it on the internal system disks of the servers, but thinking a bit more about the setup it seems that the filesystem need to have a significant load on the filesystem-structures (not data) to exploit the bug. The typical environment where I found it was serving (the same) ~5GB of files of the NFS-server to 48 dual-core machines, this bacically never hits the actual disk (with 32GB of ram in the machine). In this setup a single process could hit the bug on the server (the problem could also be seen over NFS from the clients).
It is worth keeping in mind that the problem is "rare". Thus all applications only opening and reading a single file every 10 minutes or so probably never hits it.
Altering the test-script so it just tries again to pick up the file, it succeeds on the second pass.
From my perspective this defininately has something to do with how the OS caches/invalidates the filesystem structures it has in memory (or somewhere near that).
I still haven't been able to produce a small pack-and-shit test that knocks the problem out on every system. But please come with suggestions about how to write a piece of code that tries to hit the problem from my descriptions.
My feeling is that the script below may reveal the bug on any "busy" volume, where busy is lots of activity in the OS-cache of the volume, not on the actual drives.
All suggestions are welcome.
I have tried 4 different kernel versions:
2.6.20, 2.6.22, 2.6.24, 2.6.25.2
Jesper
Quoted 68 lines · Jesper Krogh
Hi list.
I have a "fairly" reproducible problem. When a program opens and closes the same file many times, it eventually ends up with a "no such file or directory". Test program that can reproduce the problem on my setup:
root@hest:~# cat test-file-c.c #include <stdlib.h> #include <stdio.h> #include <fcntl.h> #include <unistd.h>int main(int argc, char *argv[]) { unsigned long i=0; int fh; char *filename; filename=argv[1]; while(1) { fh=open(filename, O_RDONLY); if (fh==-1) { printf("Failed to open %s\n", filename); printf("Open number: %ld\n",i); exit(10); } close(fh); i++; } exit(0); } root@hest:~# ./test-file-c /z/bio/databases/online/index/index-by-accno Failed to open /z/bio/databases/online/index/index-by-accno Open number: 61785000 root@hest:~# ./test-file-c /z/bio/databases/online/index/index-by-accno Failed to open /z/bio/databases/online/index/index-by-accno Open number: 120929685 (The problem is not isolate to a single file on the filesystem).strace on the program reviel that the system indeed return a "No such file or directory" to the program.
This is run on an Ubuntu Gutsy (vendor kernel): 2.6.22-14-server on an 4.5TB ext3 filesystem on an LVM volume. The volume was created on a dapper (2 releases back) and has just followed with during upgrades. I cannot reproduce it on other disks attached to the same server or on other servers attached to similar disksystems.
The filesystem was taken offline yesterday for a forced fsck and it was found to be clean.
The diskarray is a quite old Fibrenetix FX1200 with 12xPATA disk
in raid5 (with hotspare) exposed to the OS as 3 SCSI-disks of
2+2+0.5TB assembled with LVM afterwards. The SCSI-controller is a:
05:08.0 SCSI storage controller: LSI Logic / Symbios Logic 53c1030 PCI-X Fusion-MPT Dual Ultra320 SCSI (rev c1)What suggestions do you have to solve this problem?
I'm about to mkfs.ext3 the volume and spool it back in from the backup, but somehow I'm not convinced that it will solve the problem at all. It may just be a hardware problem, but dmesg doesnt tell anything.
We actually got the problem from a perl-script, but this seems to be the minimal program that reproduces the problem.
Jesper
Quoted 35 lines · On Fri, 09 May 2008 07:22:46 +0200 Jesper Krogh <[email protected]> wrote:
Hi.
A week has passed the problem still persist and I have done a fair amount of testing.
I have now been able to reproduce it on 2 different servers, a dual-core,8 cpu Sun X4600 amd64 and a 2 cpu single-core Sun V20Z.
It has been reproduced against 4 different diskarrays one of them being the IDE-SCSI(We have 2 of them) array mentioned before. The other one is an iSCSI SAN. And lately when trying to move of the "trouble systems" we hit the bug on a FC-SAN.
I belive that we can rule out hardware now.
I have both reproduced it on "old" ext3 volumes and freshly created ones.
I havent been able to reproduce it on the internal system disks of the servers, but thinking a bit more about the setup it seems that the filesystem need to have a significant load on the filesystem-structures (not data) to exploit the bug. The typical environment where I found it was serving (the same) ~5GB of files of the NFS-server to 48 dual-core machines, this bacically never hits the actual disk (with 32GB of ram in the machine). In this setup a single process could hit the bug on the server (the problem could also be seen over NFS from the clients).
It is worth keeping in mind that the problem is "rare". Thus all applications only opening and reading a single file every 10 minutes or so probably never hits it.
Altering the test-script so it just tries again to pick up the file, it succeeds on the second pass.
Quoted 8 lines
From my perspective this defininately has something to do with how the
OS caches/invalidates the filesystem structures it has in memory (or somewhere near that).I still haven't been able to produce a small pack-and-shit test that knocks the problem out on every system. But please come with suggestions about how to write a piece of code that tries to hit the problem from my descriptions.
It's weird.
Quoted 3 lines
My feeling is that the script below may reveal the bug on any "busy" volume, where busy is lots of activity in the OS-cache of the volume, not on the actual drives.
By this do you mean that there has to be a lot of other activity on the system to reproduce it? Stuff which is turning over memory?
Because one possiblity is that the cached dentry got reclaimed by memory pressure and we have some race bug which causes us to think that the file doesn't exist.
(That still shouldn't happen because the dentry should be marked recently-accessed, but perhaps the underlying inode gets reclaimed or something. Grasping at straws here)
Quoted 40 lines
All suggestions are welcome.
I have tried 4 different kernel versions:
2.6.20, 2.6.22, 2.6.24, 2.6.25.2Jesper
Quoted 32 lines · Jesper Krogh
Hi list.
I have a "fairly" reproducible problem. When a program opens and closes the same file many times, it eventually ends up with a "no such file or directory". Test program that can reproduce the problem on my setup:
root@hest:~# cat test-file-c.c #include <stdlib.h> #include <stdio.h> #include <fcntl.h> #include <unistd.h>int main(int argc, char *argv[]) { unsigned long i=0; int fh; char *filename; filename=argv[1]; while(1) { fh=open(filename, O_RDONLY); if (fh==-1) { printf("Failed to open %s\n", filename); printf("Open number: %ld\n",i); exit(10); } close(fh); i++; } exit(0); }
gee.
Quoted 5 lines · Andrew Morton
Quoted 2 lines
My feeling is that the script below may reveal the bug on any "busy" volume, where busy is lots of activity in the OS-cache of the volume, not on the actual drives.
By this do you mean that there has to be a lot of other activity on the system to reproduce it? Stuff which is turning over memory?
Yes, something like that. (sorry for not being able to be more concrete). The applications has "high activity" on a few files, not spread activity throughout the volume.
Quoted 3 lines
Because one possiblity is that the cached dentry got reclaimed by memory pressure and we have some race bug which causes us to think that the file doesn't exist.
What can i do to explore this theory? Can I disable caching of dentries
and see it go away? Does it fit the pattern that it is only the "open"-syscall that is hit (not read for example)?
Quoted 3 lines
(That still shouldn't happen because the dentry should be marked recently-accessed, but perhaps the underlying inode gets reclaimed or something. Grasping at straws here)
When I disabled the NFS-server and rand my "real-world" program on a single processor (make -j 1). It ran through fine. It basically gets around 20 million chunks out of differnet file and assemble the chuncks in a few other files. This processes more or less 5 individual sections, so make can run effectively with a concurrency of 5.
I dont know if there can be any technical reasons for not seeing it on internal attached disks? (other than I just hadn't been able to reproduce the same error conditions there.
Jesper
Signature · 1 line
Jesper
Quoted 12 lines · On Fri, 09 May 2008 08:09:34 +0200 Jesper Krogh <[email protected]> wrote:
Quoted 6 lines · Andrew Morton
Quoted 3 lines
My feeling is that the script below may reveal the bug on any "busy" volume, where busy is lots of activity in the OS-cache of the volume, not on the actual drives.
By this do you mean that there has to be a lot of other activity on the system to reproduce it? Stuff which is turning over memory?
Yes, something like that. (sorry for not being able to be more concrete). The applications has "high activity" on a few files, not spread activity throughout the volume.
Quoted 5 lines
Quoted 3 lines
Because one possiblity is that the cached dentry got reclaimed by memory pressure and we have some race bug which causes us to think that the file doesn't exist.
What can i do to explore this theory?
Watch /proc/vmstat while the tests is running. If the "*steal*" numbers are going up, the system is reclaiming memory.
Quoted 2 lines
Can I disable caching of dentries
and see it go away?
Nope.
Quoted 2 lines
Does it fit the pattern that it is only the "open"-syscall that is hit (not read for example)?
Yes. open() will look up the filename in the dentry cache, read() wil not.
If name lookup has a race agaisnt dentry cache reclaim, something like
this might happen.
But it'd be damned odd.
Quoted 10 lines
Quoted 3 lines
(That still shouldn't happen because the dentry should be marked recently-accessed, but perhaps the underlying inode gets reclaimed or something. Grasping at straws here)
When I disabled the NFS-server and rand my "real-world" program on a single processor (make -j 1). It ran through fine. It basically gets around 20 million chunks out of differnet file and assemble the chuncks in a few other files. This processes more or less 5 individual sections, so make can run effectively with a concurrency of 5.
Quoted 3 lines
I dont know if there can be any technical reasons for not seeing it on internal attached disks? (other than I just hadn't been able to reproduce the same error conditions there.
I can't think of any reason.
I guess a suitable test would be to run your little test app and then read large files as fast as you can from as many disks as you can, to force memory reclaim. If that triggers the bug, and the bug is more likely to trigger the faster you read those files, then we have a theory.
Damned odd though.
Quoted 6 lines · On Friday May 9, [email protected] wrote:
When I disabled the NFS-server and rand my "real-world" program on a single processor (make -j 1). It ran through fine. It basically gets around 20 million chunks out of differnet file and assemble the chuncks in a few other files. This processes more or less 5 individual sections, so make can run effectively with a concurrency of 5.
(For linux-nfs readers: the problem is that repeatedly opening a given
file sometimes returns a ENOENT - http://lkml.org/lkml/2008/5/9/15).
The mention of an NFS-server made my ears prick up...
Do I understand correctly that the problem only occurs when you have 48 clients hammering away at the filesystem in question?
Could the clients be accessing the same file that you are experiencing problems with? Or one of the directories in the path (if so, how deep).
How many different files to these 20 million chunks come from? And how does that number compare with the first number from
grep dentry /proc/slabinfo
??
The NFS server does some slighty strange things with the dcache if the object being access is not in the cache.
Also, can get a few instances of
grep '^fh' /proc/nfs/rpc/nfsd
while things are going strange. The numbers are:
* fh <stale> <total-lookups> <anonlookups> <dir-not-in-dcache> <nondir-not-in-dcache>
That will show us if it is looking for things that aren't in the dcache.
Finally, if the filesystem export with "subtree_check" or "nosubtree_check"?
Does it make a difference if you switch the setting of this flag and re-export?
NeilBrown
Quoted 25 lines · Neil Brown
Quoted 6 lines · On Friday May 9, [email protected] wrote:
When I disabled the NFS-server and rand my "real-world" program on a single processor (make -j 1). It ran through fine. It basically gets around 20 million chunks out of differnet file and assemble the chuncks in a few other files. This processes more or less 5 individual sections, so make can run effectively with a concurrency of 5.
(For linux-nfs readers: the problem is that repeatedly opening a given
file sometimes returns a ENOENT - http://lkml.org/lkml/2008/5/9/15).The mention of an NFS-server made my ears prick up...
Do I understand correctly that the problem only occurs when you have 48 clients hammering away at the filesystem in question?
Could the clients be accessing the same file that you are experiencing problems with? Or one of the directories in the path (if so, how deep).
How many different files to these 20 million chunks come from? And how does that number compare with the first number from
grep dentry/proc/slabinfo
??
Quoted 5 lines
The NFS server does some slighty strange things with the dcache if the object being access is not in the cache.
Also, can get a few instances of
grep '^fh'/proc/nfs/rpc/nfsd
I think you meant /proc/net/rpc/nfsd.
--b.
Quoted 18 lines
while things are going strange. The numbers are:
* fh <stale> <total-lookups> <anonlookups> <dir-not-in-dcache> <nondir-not-in-dcache>That will show us if it is looking for things that aren't in the dcache.
Finally, if the filesystem export with "
subtree_check" or "nosubtree_check"?
Does it make a difference if you switch the setting of this flag and re-export?NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 9 lines · Neil Brown
Quoted 5 lines · On Friday May 9, [email protected] wrote:
When I disabled the NFS-server and rand my "real-world" program on a single processor (make -j 1). It ran through fine. It basically gets around 20 million chunks out of differnet file and assemble the chuncks in a few other files. This processes more or less 5 individual sections, so make can run effectively with a concurrency of 5.
(For linux-nfs readers: the problem is that repeatedly opening a given
file sometimes returns a ENOENT - http://lkml.org/lkml/2008/5/9/15).
This thing really, really irritated me, but I must admit that Andrew Morton was very correct about this "not being very likely"- a kernel bug.
It seem that our central configuration handling system (slack) was being way to aggressive about updating symlinks in paths of the filesystems that I was testing upon, that explains why I couldn't reproduce it on the internal volumes, and not on any of the volumes I created only for testing purposes. Sometimes you just get too blind..
(I haven't been able to reproduce for 12 hours now)
Just to answer your questions, yes, the 48 clients do hammer on NFS and now it seems to work excellent.
Sorry for all the noise.
Signature · 1 line
Jesper
Quoted 1 line · On Mon, 12 May 2008 08:41:02 +0200 Jesper Krogh…
Sorry for all the noise.
It happens. Reporting a possible kernel bug is better than remaining silent or working around it or waiting indefinitely until you're sure it's a kernel bug, etc.
Thanks for the update.