Annoying rename() problem
Subsystems (2)
Hello,
I am using NFS to store Courier mailboxes in Maildir format on a NFS exported disk. As some of you might already know, Courier has two main folders inside each Maildir, new/ and cur/. The new/ directory one is used to incoming messages and the cur/ directory is used to store already seen messages for a given folder. When an user access its mailbox, Courier IMAP will rename all the files found in new/ to the cur/ directory.
Now, when a user has more than 50 files inside the new/ directory and Courier IMAP rename those files to cur/ directory, I get strange errors inside the Courier log complaining that the rename() has failed with the errno=ENOENT (no such file or directory). The most amazing part is that the files are moved just fine from new/ to cur/ and I just can not understand why I keep getting those rename() errors and most important, how can I fix this.
My setup follows:
The NFS server runs under Debian Sarge with custom 2.6.18 kernel, NFSv4 support with NFSv3 support. The psyhical machine is an AMD Opteron 64 and the NFS exports are stored on a RAID-5 array. /etc/exports contains:
/nfs-exported 192.168.0.2(rw,all_squash,anonuid=10000,anongid=10000,sync)
The versions of Debian packages:
portmap : Version: 5-9 / Architecture: amd64
nfs-kernel-server: Version: 1:1.0.6-3.1 / Architecture: amd64 nfs-common : Version: 1:1.0.6-3.1 / Architecture: amd64
The NFS client runs under Debian Sarge with custom 2.6.18 kernel, NFSv4 support with NFSv3 support inside a VMWare virtual machine with bridged networking and vmware-tools correctly installed.
Client mount options from /etc/fstab:
192.168.0.1:/nfs-exported /nfs nfs
rw,hard,intr,nfsvers=3,sync,noac,nodev,nosuid 0 0
The nfs-common and portmap packages have the same version like on the NFS server. On the client, I've tried most of the mont nfs's combinations and tried using the pseudo-fileystem nfs4. No luck, the errors are still there.
My problem perfectly fits this description:
http://lists.freebsd.org/pipermail/freebsd-bugs/2003-October/003461.html but the bug reports dates from 2003.
Anyone has a suggestion of what I could try or do ? I've searched the NFS mailinglist, nothing, nada. Any suggestion is more than welcome.
Thank you,
George
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · Antoniu-George SAVU
Now, when a user has more than 50 files inside the new/ directory and Courier IMAP rename those files to cur/ directory, I get strange errors inside the Courier log complaining that the
rename()has failed with the errno=ENOENT (no such file or directory). The most amazing part is that the files are moved just fine from new/ to cur/ and I just can not understand why I keep getting thoserename()errors and most important, how can I fix this.
+1
I ran across this message while trying to understand other rename over NFS failures. On a company software project a test framework would sporadically show test failures where the application would fail reporting rename failures of NFS. But debugging would always show that the rename had actually happened. This seems to need fairly heavy activity load in order to trigger this problem. I cannot reproduce it on demand.
The application replaces an existing file on disk by writing a new file off to the side in a temporary file and then after successfully close() operation rename()s the file to the final name, replacing the file that is already on disk. The rename() returns a failure. Subsequent inspection showed that the new file was renamed okay.
Quoted 5 lines
My setup follows:
The NFS server runs under Debian Sarge with custom 2.6.18 kernel, NFSv4 support with NFSv3 support. The psyhical machine is an AMD Opteron 64 and the NFS exports are stored on a RAID-5 array.
My NFS client is a Debian Sarge system with custom 2.6.12 kernel and the server is an HP-UX 11.23 host.
Quoted 3 lines
My problem perfectly fits this description:
http://lists.freebsd.org/pipermail/freebsd-bugs/2003-October/003461.html but the bug reports dates from 2003.
In my case the new temporary file and the previously existing file are both in the same directory. Here would be the pseudo-code for my case. The application is a compiled C program.
touch foo.d~
mv foo.d~ foo.db
Quoted 2 lines
Anyone has a suggestion of what I could try or do ? I've searched the NFS mailinglist, nothing, nada. Any suggestion is more than welcome.
To work around this problem we have stopped checking the return code of rename()! Instead we now stat() the file before and after the rename to verify the operation. This is less efficient but avoided the spurious error message.
Bob
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 8 lines · Bob Proulx
Quoted 7 lines · Antoniu-George SAVU
Now, when a user has more than 50 files inside the new/ directory and Courier IMAP rename those files to cur/ directory, I get strange errors inside the Courier log complaining that the
rename()has failed with the errno=ENOENT (no such file or directory). The most amazing part is that the files are moved just fine from new/ to cur/ and I just can not understand why I keep getting thoserename()errors and most important, how can I fix this.
...
Quoted 5 lines
The application replaces an existing file on disk by writing a new file off to the side in a temporary file and then after successfully
close()operation rename()s the file to the final name, replacing the file that is already on disk. Therename()returns a failure. Subsequent inspection showed that the new file was renamed okay.
I'd have guessed the client is replaying the rename request and only seeing the reply to the second attempt (even though the first also succeeded).
Are you using tcp? (And if not, does switching help?) Does adjusting the timeout times (see timeo, retrans in "man nfs") help?
--b.
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 3 lines · J. Bruce Fields
I'd have guessed the client is replaying the rename request and only seeing the reply to the second attempt (even though the first also succeeded).
That is possible. In a busy network environment it would be possible for the udp packets to be duplicated. If the nfs system did not handle this duplication then it could be fooled into thinking it had failed when it had not. But I thought that nfs handled udp with checksumming and retries, no?
Quoted 1 line
Are you using tcp?
I am using udp. I don't know about the OP.
Quoted 1 line
(And if not, does switching help?)
I don't know if switching to tcp would help. For various reasons I can't really make that change at this time.
Quoted 2 lines
Does adjusting the timeout times (see timeo, retrans in "man nfs") help?
Various people in my lab have tried an assortment of nfs timeout and retransmit options, with regards to nfs performance, not to this particular problem.
I don't know if the original poster Antoniu-George SAVU will be able to benefit from the suggestions here. In my case the problem is already worked around with code changes in the application. Therefore my test case has dissappeared.
I was just adding a sighting confirmation to the original poster Antoniu-George SAVU's report of this problem that I have also seen a problem that looks identical. So I don't think this is an isolated problem. Unfortunately the test case has gone away for me.
Bob
-------------------------------------------------------------------------
Using Tomcat but need to do more? Need to support web services, security? Get stuff done quickly with pre-integrated technology to make your job easier Download IBM WebSphere Application Server v.1.0.1 based on Apache Geronimo
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=120709&bid=263057&dat=121642
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs