Skip to contentSkip to conversation

open O_EXCL works?

DISCUSSION Filipe Brandenburger 3 participants 16 messages Mar 13, 2005 – Mar 23, 2005

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,

I'm trying to find out if open(O_CREAT|O_EXCL) works or not in NFS. I read the RFC1813, and it says NFSv3 must implement it on the server, but even so I don't know if the client is doing it reliably or not.

My server is a Netapp FAS940, my client is Linux RH7.3 with kernel 2.4.9-e.38smp (the kernel is from RHAS2.1). I'm mounting the NFS with vers=3, the line that is output in the "mount" command for this filesystem is:

10.10.4.7:/vol/vol3/mail7n03 on /var/spool/mail7n03-01 type nfs (rw,intr,vers=3,proto=udp,nolock,rsize=8192,wsize=8192,addr=10.10.4.7)

The fact is that I'm using tcpdump to find it out, but it seems to do an "access" operation before the "create" operation to determine if the file already exists (and fail the O_EXCL in that case), but this is not atomic.

I also tried to test it with O_EXCL and without O_EXCL and take tcpdump's, but they seem to be equal to me.

For example, when I executed this code:

[root@marova mail7n03-01]# rm -f branden.txt; FILE=branden.txt perl -MFcntl -le'sysopen(H,$ENV{FILE},O_RDWR|O_CREAT|O_EXCL)||die "$!"'

I got this tcpdump for the create operation:

10:46:35.381897 10.10.105.244.1111971850 > 10.10.4.7.2049: 192 create fh Unknown/1 "branden.txt" (DF)

0x0000   4500 00dc 0000 4000 4011 b802 0a0a 69f4        E.....@[email protected].
0x0010   0a0a 0407 031b 0801 00c8 a9f0 4247 580a        ............BGX.
0x0020   0000 0000 0000 0002 0001 86a3 0000 0003        ................
0x0030   0000 0008 0000 0001 0000 0044 0019 adcf        ...........D....
0x0040   0000 0013 6d61 726f 7661 2e74 6572 7261        ....marova.terra
0x0050   2e63 6f6d 2e62 7200 0000 0000 0000 0000        .com.br.........
0x0060   0000 0007 0000 0000 0000 0001 0000 0002        ................
0x0070   0000 0003 0000 0004 0000 0006 0000 000a        ................
0x0080   0000 0000 0000 0000 0000 0020 7700 0000        ............w...
0x0090   e74f a100 2000 0000 0000 0077 e74f a100        .O.........w.O..
0x00a0   b4f6 0012 7700 0000 e74f a100 0000 000b        ....w....O......
0x00b0   6272 616e 6465 6e2e 7478 7400 0000 0000        branden.txt.....
0x00c0   0000 0001 0000 81a4 0000 0000 0000 0000        ................
0x00d0   0000 0000 0000 0000 0000 0000                  ............
10:46:35.382105 10.10.4.7.2049 > 10.10.105.244.1111971850: reply ok 272 create fh Unknown/1
0x0000   4500 012c 26df 0000 4011 d0d3 0a0a 0407        E..,&...@.......
0x0010   0a0a 69f4 0801 031b 0118 91eb 4247 580a        ..i.........BGX.
0x0020   0000 0001 0000 0000 0000 0000 0000 0000        ................
0x0030   0000 0000 0000 0000 0000 0001 0000 0020        ................
0x0040   7700 0000 e74f a100 2000 0000 00ac 8d20        w....O..........
0x0050   b481 aa00 b4f6 0012 7700 0000 e74f a100        ........w....O..
0x0060   0000 0001 0000 0001 0000 01a4 0000 0001        ................
0x0070   0000 0000 0000 0000 0000 0000 0000 0000        ................
0x0080   0000 0000 0000 0000 0000 0000 0000 0000        ................
0x0090   0000 0000 1200 f6b4 0000 0000 00ac 8d20        ................
0x00a0   4234 443b 2c39 f1f8 4234 443b 2c39 f1f8        B4D;,9..B4D;,9..
0x00b0   4234 443b 2c39 f1f8 0000 0001 0000 0000        B4D;,9..........
0x00c0   0000 3000 4234 43bb 25f7 d170 4234 43bb        ..0.B4C.%..pB4C.
0x00d0   25f7 d170 0000 0001 0000 0002 0000 01ed        %..p............
0x00e0   0000 000f 0000 0208 0000 0208 0000 0000        ................
0x00f0   0000 3000 0000 0000 0000 3000 0000 0000        ..0.......0.....
0x0100   0000 0000 0000 0000 1200 f6b4 0000 0000        ................
0x0110   0000 0077 4234 442c 00d5 93c8 4234 443b        ...wB4D,....B4D;
0x0120   2c39 f1f8 4234 443b 2c39 f1f8                  ,9..B4D;,9..

And when I executed this code without the O_EXCL:

[root@marova mail7n03-01]# rm -f branden.txt; FILE=branden.txt perl -MFcntl -le'sysopen(H,$ENV{FILE},O_RDWR|O_CREAT)||die "$!"'

I got this tcpdump for the create operation, that seemed the same, except of course for being a different run (different timestamps, and other insignificant differences):

10:51:44.966999 10.10.105.244.1196120330 > 10.10.4.7.2049: 192 create fh Unknown/1 "branden.txt" (DF)

0x0000   4500 00dc 0000 4000 4011 b802 0a0a 69f4        E.....@[email protected].
0x0010   0a0a 0407 031a 0801 00c8 a2b8 474b 590a        ............GKY.
0x0020   0000 0000 0000 0002 0001 86a3 0000 0003        ................
0x0030   0000 0008 0000 0001 0000 0044 0019 af04        ...........D....
0x0040   0000 0013 6d61 726f 7661 2e74 6572 7261        ....marova.terra
0x0050   2e63 6f6d 2e62 7200 0000 0000 0000 0000        .com.br.........
0x0060   0000 0007 0000 0000 0000 0001 0000 0002        ................
0x0070   0000 0003 0000 0004 0000 0006 0000 000a        ................
0x0080   0000 0000 0000 0000 0000 0020 7700 0000        ............w...
0x0090   e74f a100 2000 0000 0000 0077 e74f a100        .O.........w.O..
0x00a0   b4f6 0012 7700 0000 e74f a100 0000 000b        ....w....O......
0x00b0   6272 616e 6465 6e2e 7478 7400 0000 0000        branden.txt.....
0x00c0   0000 0001 0000 81a4 0000 0000 0000 0000        ................
0x00d0   0000 0000 0000 0000 0000 0000                  ............
10:51:44.967211 10.10.4.7.2049 > 10.10.105.244.1196120330: reply ok 272 create fh Unknown/1
0x0000   4500 012c f3e0 0000 4011 03d2 0a0a 0407        E..,....@.......
0x0010   0a0a 69f4 0801 031a 0118 56a8 474b 590a        ..i.......V.GKY.
0x0020   0000 0001 0000 0000 0000 0000 0000 0000        ................
0x0030   0000 0000 0000 0000 0000 0001 0000 0020        ................
0x0040   7700 0000 e74f a100 2000 0000 00ac 8d20        w....O..........
0x0050   b581 aa00 b4f6 0012 7700 0000 e74f a100        ........w....O..
0x0060   0000 0001 0000 0001 0000 01a4 0000 0001        ................
0x0070   0000 0000 0000 0000 0000 0000 0000 0000        ................
0x0080   0000 0000 0000 0000 0000 0000 0000 0000        ................
0x0090   0000 0000 1200 f6b4 0000 0000 00ac 8d20        ................
0x00a0   4234 4571 1575 2a00 4234 4571 1575 2a00        B4Eq.u*.B4Eq.u*.
0x00b0   4234 4571 1575 2a00 0000 0001 0000 0000        B4Eq.u*.........
0x00c0   0000 3000 4234 4562 2eca 0ae8 4234 4562        ..0.B4Eb....B4Eb
0x00d0   2eca 0ae8 0000 0001 0000 0002 0000 01ed        ................
0x00e0   0000 000f 0000 0208 0000 0208 0000 0000        ................
0x00f0   0000 3000 0000 0000 0000 3000 0000 0000        ..0.......0.....
0x0100   0000 0000 0000 0000 1200 f6b4 0000 0000        ................
0x0110   0000 0077 4234 442c 00d5 93c8 4234 4571        ...wB4D,....B4Eq
0x0120   1575 2a00 4234 4571 1575 2a00                  .u*.B4Eq.u*.

I also tried to find something from the kernel source, I looked into the 2.4.18 source that I had in my hands, and I found out inside fs/nfs/nfs3proc.c this comments:

/*
* Create a regular file.
* For now, we don't implement O_EXCL.
*/
static int
nfs3_proc_create(struct inode *dir, struct qstr *name, struct iattr *sattr,
int flags, struct nfs_fh *fhandle, struct nfs_fattr *fattr)
{
[...]

But the strange is that the code then goes and implements exactly what O_EXCL is to do:

[...]

        arg.createmode = NFS3_CREATE_UNCHECKED;
        if (flags & O_EXCL) {
                arg.createmode  = NFS3_CREATE_EXCLUSIVE;
                arg.verifier[0] = jiffies;
                arg.verifier[1] = current->pid;
        }

[...]

I don't know who is calling this function, and if who is calling this funcion is never using O_EXCL and implementing that check at that level. Is it the case?

Well, I hope you can answer my questions, I can send you more information about the problem if that's the case (tcpdump's, etc.).

Thanks in advance,
Filipe

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#1 lore raw

su den 13.03.2005 Klokka 14:23 (-0300) skreiv Filipe Brandenburger:

Quoted 10 lines

Hello,

I'm trying to find out if open(O_CREAT|O_EXCL) works or not in NFS. I read the RFC1813, and it says NFSv3 must implement it on the server, but even so I don't know if the client is doing it reliably or not.

My server is a Netapp FAS940, my client is Linux RH7.3 with kernel 2.4.9-e.38smp (the kernel is from RHAS2.1). I'm mounting the NFS with vers=3, the line that is output in the "mount" command for this filesystem is:

2.4.x kernels do not implement O_EXCL opens in NFS due to VFS limitations. In 2.6.x we fixed the VFS limitations by adding "intents" and so those kernels implement O_EXCL correctly for both NFSv3 and NFSv4.

Cheers,
Trond

Signature · 12 lines
Trond Myklebust <[email protected]>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#2 lore raw

Hi, Trond.

Thank you very much for your very prompt answer.

I would like to get some more advice, since I will need locks in my application and I'm wondering what will be the best way to implement it, regarding NFS.

I found this post where you state that the best for locking is a hard link:
http://marc.theaimsgroup.com/?l=linux-nfs&m=87941837303944&w=2

But the fact is that I need my locks to have a timestamp, so that other processes can detect that another process created a lock and died before removing it (in that case I can invalidate all my data).

The problem is that by using hard links, I would have to create a new file and then create a hard link to this file with the lock file name, that would be a little more expensive on the NFS side than I'm expecting to.

I had two other ideas: symlink and mkdir.

However, I made some tcpdumps, and I found out that almost all operations make a "lookup" before making the operation itself, and if the file/directory already exists, it skips the main RPC call (link/symlink/mkdir). The tcpdumps are at the end of the post.

My question is to know what would happen if:
a) at the time the "lookup" call is made, the hardlink/symlink/directory does not exist yet, so that I can go to the next call;
b) when I call the main function (link/symlink/mkdir), the hardlink/symlink/directory was already created by another process (maybe on another host).

What I mean is, apparently (to me at least) there's a race condition there, that can only be circumvented if mkdir is using the error return from the mkdir RPC call besides the result of the lookup. Is that the case?

What I mean is, if I have processes using mkdir to create a lock directory, can I be sure that no two processes will believe they created the directory at the same time?

Thanks a lot,
Filipe

* Sun, 13 Mar 2005 13:44:38 -0500, Trond Myklebust <[email protected]>:

Quoted 10 lines

2.4.x kernels do not implement O_EXCL opens in NFS due to VFS limitations. In 2.6.x we fixed the VFS limitations by adding "intents" and so those kernels implement O_EXCL correctly for both NFSv3 and NFSv4.

Cheers,
Trond

Signature · 1 line
Trond Myklebust <[email protected]>

Hardlink:

[popmgr@marova mail7n03-01]$ ln branden.txt teste.txt

07:55:50.749005 10.10.105.244.1636967181 > 10.10.4.7.2049: 136 lookup fh Unknown/1 "teste.txt" (DF) 07:55:50.749219 10.10.105.244.1653744397 > 10.10.4.7.2049: 136 lookup fh Unknown/1 "branden.txt" (DF) 07:55:50.749435 10.10.105.244.1670521613 > 10.10.4.7.2049: 172 link fh Unknown/1 -> fh Unknown/1 "teste.txt" (DF)

[popmgr@marova mail7n03-01]$ ln branden.txt teste.txt
ln: `teste.txt': File exists

07:56:28.974885 10.10.105.244.244720397 > 10.10.4.7.2049: 136 lookup fh Unknown/1 "teste.txt" (DF) 07:56:28.975094 10.10.105.244.261497613 > 10.10.4.7.2049: 136 lookup fh Unknown/1 "branden.txt" (DF)

Symlink:

[popmgr@marova mail7n03-01]$ ln -s branden.txt teste.txt

07:58:18.621391 10.10.105.244.2309366541 > 10.10.4.7.2049: 136 lookup fh Unknown/1 "teste.txt" (DF) 07:58:18.621587 10.10.105.244.2326143757 > 10.10.4.7.2049: 180 symlink fh Unknown/1 "teste.txt" ->"branden.txt" (DF)

[popmgr@marova mail7n03-01]$ ln -s branden.txt teste.txt
ln: `teste.txt': File exists

07:58:20.530707 10.10.105.244.3819315981 > 10.10.4.7.2049: 136 lookup fh Unknown/1 "teste.txt" (DF) 07:58:20.530906 10.10.105.244.3836093197 > 10.10.4.7.2049: 120 readlink fh Unknown/1 (DF) 07:58:20.531040 10.10.105.244.3852870413 > 10.10.4.7.2049: 136 lookup fh Unknown/1 "branden.txt" (DF)

Mkdir:

[popmgr@marova mail7n03-01]$ mkdir teste.dir

07:59:54.027575 10.10.105.244.4240122637 > 10.10.4.7.2049: 136 lookup fh Unknown/1 "teste.dir" (DF) 07:59:54.027798 10.10.105.244.4256899853 > 10.10.4.7.2049: 164 mkdir fh Unknown/1 "teste.dir" (DF)

[popmgr@marova mail7n03-01]$ mkdir teste.dir
mkdir: cannot create directory `teste.dir': File exists

07:59:56.316678 10.10.105.244.96215821 > 10.10.4.7.2049: 136 lookup fh Unknown/1 "teste.dir" (DF)

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#3 lore raw

m=E5 den 14.03.2005 Klokka 08:18 (-0300) skreiv Filipe Brandenburger:

Quoted 7 lines

My question is to know what would happen if:
a) at the time the "lookup" call is made, the hardlink/symlink/directory does not exist yet, so that I can go to the next call;
b) when I call the main function (link/symlink/mkdir), the hardlink/symlink/directory was already created by another process (maybe on another host).
=20

Quoted 8 lines

What I mean is, apparently (to me at least) there's a race condition there, that can only be circumvented if mkdir is using the error return from the mkdir RPC call besides the result of the lookup. Is that the case?
=20
What I mean is, if I have processes using mkdir to create a lock directory, can I be sure that no two processes will believe they created the directory at the same time?

Both symlink and mkdir return errors that close the race. In fact, we shouldn't have to do that lookup at all (and one of these days, I will get round to setting up the intent flags for all these cases).

Cheers,
Trond

--=20
Trond Myklebust <[email protected]>

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#4 lore raw

Hi, there.

I'm trying to make my program work correctly over NFS, but apparently I still have a race condition, because I'm getting inconsistencies. I still can't find out exactly what is happening. This is what I'm doing:

f =3D open("myfile", O_CREAT|O_WRONLY|O_APPEND)

[repeat the loop below for the data]
hardlink("myfile", "myfile.lock")
-> if it fails, invalidate information
write(f, "mydata", sizeof(mydata))
unlink("myfile.lock")
[end loop]

close(f)

I would expect to get consistent data, as "hardlink" should be an atomic operation over NFS (according to the previous posts on this thread). But I'm still having problems.

What I think is happening is that one process in one host is silently overwriting data just written by another process on another host (this overwriting happens on the same second interval), instead of appending to the end of the file, just after that other process record...

I mean, I found out that append writes are not atomic in Linux 2.4, but if I have a hardlink operation just before the write, it should fix the issue, shouldn't it?

Or somehow isn't the hardlink operation atomic? Or is the information of the file size (for the append) being held in some cache, and not being looked up just before each write (and just after we assure the file is ours with the lockfile hardlink)? Could you explain what is happening? Could you suggest a way to fix this and reliably append data to the file?

I can provide more information upon request. By the way, the NFS client platform is Linux RedHat 7.3, with kernel from RHAS 2.1, uname returns 2.4.9-e.49enterprise, NFS servers are a mix of EMC Celerra, EMC NS-700 (Clariion) and Netapp FAS940.

Thanks a lot,
Filipe

* Mon, 14 Mar 2005 10:24:23 -0500, Trond Myklebust <[email protected].= no>:

Quoted 3 lines

m=E5 den 14.03.2005 Klokka 08:18 (-0300) skreiv Filipe Brandenburger:

Quoted 2 lines

My question is to know what would happen if:
a) at the time the "lookup" call is made, the hardlink/symlink/director=

y

Quoted 3 lines
Quoted 3 lines

does not exist yet, so that I can go to the next call;
b) when I call the main function (link/symlink/mkdir), the hardlink/symlink/directory was already created by another process (mayb=

e

Quoted 11 lines
Quoted 1 line

on another host).

=20
Both symlink and mkdir return errors that close the race. In fact, we shouldn't have to do that lookup at all (and one of these days, I will get round to setting up the intent flags for all these cases). =20
Cheers,
Trond
=20
--=20
Trond Myklebust <[email protected]>

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#5 lore raw

m=E5 den 21.03.2005 Klokka 18:31 (-0300) skreiv Filipe Brandenburger:

Quoted 23 lines

Hi, there.
=20
I'm trying to make my program work correctly over NFS, but apparently I still have a race condition, because I'm getting inconsistencies. I still can't find out exactly what is happening. This is what I'm doing:
=20
=20
f =3D open("myfile", O_CREAT|O_WRONLY|O_APPEND)
=20
[repeat the loop below for the data]
hardlink("myfile", "myfile.lock")
-> if it fails, invalidate information
write(f, "mydata", sizeof(mydata))
unlink("myfile.lock")
[end loop]
=20
close(f)
=20
=20
I would expect to get consistent data, as "hardlink" should be an atomic operation over NFS (according to the previous posts on this thread). But I'm still having problems.
=20

Quoted 8 lines

What I think is happening is that one process in one host is silently overwriting data just written by another process on another host (this overwriting happens on the same second interval), instead of appending to the end of the file, just after that other process record... =20
I mean, I found out that append writes are not atomic in Linux 2.4, but if I have a hardlink operation just before the write, it should fix the issue, shouldn't it?

Append writes are *never* atomic on NFS for any platform. They would need additional protocol support.

For newer kernels, you should read all about the NFS cache consistency model on

http://nfs.sourceforge.net/#faq_a8

That will tell you why you should create the lock file before opening the file.
On an old kernel, like 2.4.20, you will also have to set the "noac" mount option. It makes everything very slow, but it is necessary in order to compensate for the lack of close-to-open cache consistency.

Cheers,
Trond

--=20
Trond Myklebust <[email protected]>

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#6 lore raw

Hi again.

* Mon, 21 Mar 2005 19:01:15 -0500, Trond Myklebust <[email protected]>:

Quoted 4 lines

For newer kernels, you should read all about the NFS cache consistency model on

   http://nfs.sourceforge.net/#faq_a8

Which versions? 2.6 and newer? As I read it on the FAQ, close-to-open consistency was introduced in 2.4.20 (although I'm with 2.4.9, but I believe it's really patched, as it's a RedHat thing).

Quoted 3 lines

On an old kernel, like 2.4.20, you will also have to set the "noac" mount option. It makes everything very slow, but it is necessary in order to compensate for the lack of close-to-open cache consistency.

Gee, do I need to completely disable attribute caching for that? I really can't afford it, the perofmance penalty for it is too expensive... Isn't the "nocto" enough already (as I read it, "cto" is for "close-to-open")?

nocto Suppress the retrieval of new attributes when creating a file.

Isn't there a safe and cheap way to use a lock file that works on NFSv3 on a 2.4.9 kernel?

Thanks,
Filipe

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#7 lore raw

ty den 22.03.2005 Klokka 07:46 (-0300) skreiv Filipe Brandenburger:

Quoted 9 lines

* Mon, 21 Mar 2005 19:01:15 -0500, Trond Myklebust <[email protected]>:

Quoted 4 lines

For newer kernels, you should read all about the NFS cache consistency model on

   http://nfs.sourceforge.net/#faq_a8

Which versions? 2.6 and newer? As I read it on the FAQ, close-to-open consistency was introduced in 2.4.20 (although I'm with 2.4.9, but I believe it's really patched, as it's a RedHat thing).

I just downloaded the source code for 2.4.9-e49 from RedHat, and the dentry revalidation code looks pretty much untouched compared to stock 2.4.9. No true cto there...

However, I had forgotten how the 2.4.9 code worked. If the file has been modified within the last hour, then the kernel should revalidate the attribute cache on open. This scheme very much relies on you using ntp or something like that to ensure that the clock on the server and client are in reasonable sync.

Quoted 10 lines
Quoted 3 lines

On an old kernel, like 2.4.20, you will also have to set the "noac" mount option. It makes everything very slow, but it is necessary in order to compensate for the lack of close-to-open cache consistency.

Gee, do I need to completely disable attribute caching for that? I really can't afford it, the perofmance penalty for it is too expensive... Isn't the "nocto" enough already (as I read it, "cto" is for "close-to-open")?

nocto Suppress the retrieval of new attributes when creating a file.

Err... We need to rewrite that manpage. The above doesn't even make sense... It should read "Suppress the retrieval of new attributes when opening a cached file"

Which is exactly the opposite of what you really want.

Quoted 2 lines

Isn't there a safe and cheap way to use a lock file that works on NFSv3 on a 2.4.9 kernel?

The best way to ensure cache consistency is to put a posix (i.e. fcntl()) lock on the file after you have created the lock file. Doing so will flush the stale caches.

Cheers,
Trond

Signature · 12 lines
Trond Myklebust <[email protected]>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#8 lore raw

Hi again,

* Tue, 22 Mar 2005 07:27:15 -0500, Trond Myklebust <[email protected]>:

Quoted 6 lines
Quoted 2 lines

Isn't there a safe and cheap way to use a lock file that works on NFSv3 on a 2.4.9 kernel?

The best way to ensure cache consistency is to put a posix (i.e. fcntl()) lock on the file after you have created the lock file. Doing so will flush the stale caches.

Let me see if I understand: the lock (with the hardlink) is working fine, my problem is actually with the position where to append, is that it? Somehow, the information of the size of the file (that is where the append will write) is being cached in my clients, and if two clients append at the same time, the cache may cause them to overwrite each other, right?

If so, you suggest me to use fcntl on the file where I'm writing before the write operation, right? This would be the code:

f = open("myfile", O_CREAT|O_WRONLY|O_APPEND)

[repeat the loop below for the data]
hardlink("myfile", "myfile.lock")
-> if it fails, invalidate information
fcntl(f, F_SETLK, F_WRLCK)
-> if it fails, invalidate information
write(f, "mydata", sizeof(mydata))
fcntl(f, F_SETLK, F_UNLCK)
unlink("myfile.lock")
[end loop]

close(f)

Would this work? Also, using fcntl(F_SETLK) implies mounting filesystems without the "nolock" option (that is what I'm using now), and implies running lockd, right? Also, wouldn't I have problems if some process dies between locking and unlocking with fcntl?

Would change the position of "open" and "close" to inside the hardlink lock solve the problem? What I mean is, would this code work as I expect, with appends being atomic between different clients?

[repeat the loop below for the data]
hardlink("myfile", "myfile.lock")
-> if it fails, invalidate information
f = open("myfile", O_CREAT|O_WRONLY|O_APPEND)
write(f, "mydata", sizeof(mydata))
close(f)
unlink("myfile.lock")
[end loop]

Thanks a lot,
Filipe

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#9 lore raw

ty den 22.03.2005 Klokka 10:28 (-0300) skreiv Filipe Brandenburger:

Quoted 4 lines

Would this work? Also, using fcntl(F_SETLK) implies mounting filesystems without the "nolock" option (that is what I'm using now), and implies running lockd, right? Also, wouldn't I have problems if some process dies between locking and unlocking with fcntl?

Not necessarily. Cache invalidation should happen whether or not you use the nolock option.

Quoted 5 lines

Would change the position of "open" and "close" to inside the hardlink lock solve the problem? What I mean is, would this code work as I expect, with appends being atomic between different clients?

Quoted 8 lines

[repeat the loop below for the data]
hardlink("myfile", "myfile.lock")
-> if it fails, invalidate information
f = open("myfile", O_CREAT|O_WRONLY|O_APPEND)
write(f, "mydata", sizeof(mydata))
close(f)
unlink("myfile.lock")
[end loop]

Yes. This would work if close-to-open works correctly.

Cheers,
Trond

Signature · 12 lines
Trond Myklebust <[email protected]>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#10 lore raw

Hello, once again! :-)

* Tue, 22 Mar 2005 08:41:43 -0500, Trond Myklebust <[email protected]>:

Quoted 10 lines
Quoted 8 lines

[repeat the loop below for the data]
hardlink("myfile", "myfile.lock")
-> if it fails, invalidate information
f = open("myfile", O_CREAT|O_WRONLY|O_APPEND)
write(f, "mydata", sizeof(mydata))
close(f)
unlink("myfile.lock")
[end loop]

Yes. This would work if close-to-open works correctly.

Alright, but in:

Quoted 1 line
   http://nfs.sourceforge.net/#faq_a8

It's saying that close-to-open was introduced in 2.4.20, and I'm still with 2.4.9. Will it work with 2.4.9 too? The FAQ mentions something about "old behavior", what was that exactly? You mentioned too that in 2.4.9 it depends a lot on the synchronization of time between client and server, how is it?

What I really need to know is if that code above will solve my problem on 2.4.9 too, or only on 2.4.20 and 2.6 or superior.

Thanks,
Filipe

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#11 lore raw

ty den 22.03.2005 Klokka 10:51 (-0300) skreiv Filipe Brandenburger:

Quoted 14 lines

Hello, once again! :-)

* Tue, 22 Mar 2005 08:41:43 -0500, Trond Myklebust <[email protected]>:

Quoted 10 lines
Quoted 8 lines

[repeat the loop below for the data]
hardlink("myfile", "myfile.lock")
-> if it fails, invalidate information
f = open("myfile", O_CREAT|O_WRONLY|O_APPEND)
write(f, "mydata", sizeof(mydata))
close(f)
unlink("myfile.lock")
[end loop]

Yes. This would work if close-to-open works correctly.

Quoted 7 lines

Alright, but in:

Quoted 1 line
   http://nfs.sourceforge.net/#faq_a8

It's saying that close-to-open was introduced in 2.4.20, and I'm still with 2.4.9. Will it work with 2.4.9 too? The FAQ mentions something about "old behavior", what was that exactly? You mentioned too that in 2.4.9 it depends a lot on the synchronization of time between client and server, how is it?

I wrote:
However, I had forgotten how the 2.4.9 code worked. If the file has been modified within the last hour, then the kernel should revalidate the attribute cache on open. This scheme very much relies on you using ntp or something like that to ensure that the clock on the server and client are in reasonable sync.

Quoted 2 lines

What I really need to know is if that code above will solve my problem on 2.4.9 too, or only on 2.4.20 and 2.6 or superior.

As you can see from the above, the answer is _no guarantees_.

It may, assuming that your file changes frequently, and client and server are in sync timewise.

Signature · 12 lines
Trond Myklebust <[email protected]>



-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#12 lore raw

rename works? (was: open O_EXCL works?)

In reply to Trond Myklebust

Hi, there.

I found another problem just now. I have two processes calling "rename" to move all files from one directory to another, they're both calling the same actions, for example, if process 1 on client 1 calls rename("a/x","b/x"), process 2 on client 2 will call rename("a/x","b/x").

Apparently, somehow, I had one process trying to move 120 files (all that were on directory "a"). It said it managed to move 112 files, 8 of them failed. The second process entered the directory when there were 17 files, it tried to move them, and reported that moved successfully 16 of them, 1 of them failed. But that means that both together moved 112 + 16 = 128 files! That "8" files are reflected in my other accounting, that says there are 8 files more than there really are.

What I mean is, is it possible that both processes called the rename for the same file, and that both of them succeeded? Is it that the success of the rename operation depends on the destination file being there instead of the success of the rename operation itself? If that's it, is there a way to circumvent this restriction? If I use, for example, hardlink and unlink, it would solve the concurrency problems?

I can provide more information on the problem upon request.

Thanks again,
Filipe

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#13 lore raw

rename works? (was: open O_EXCL works?)

In reply to Trond Myklebust

Hi, there.

I found another problem just now. I have two processes calling "rename" to move all files from one directory to another, they're both calling the same actions, for example, if process 1 on client 1 calls rename("a/x","b/x"), process 2 on client 2 will call rename("a/x","b/x").

Apparently, somehow, I had one process trying to move 120 files (all that were on directory "a"). It said it managed to move 112 files, 8 of them failed. The second process entered the directory when there were 17 files, it tried to move them, and reported that moved successfully 16 of them, 1 of them failed. But that means that both together moved 112 + 16 = 128 files! That "8" files are reflected in my other accounting, that says there are 8 files more than there really are.

What I mean is, is it possible that both processes called the rename for the same file, and that both of them succeeded? Is it that the success of the rename operation depends on the destination file being there instead of the success of the rename operation itself? If that's it, is there a way to circumvent this restriction? If I use, for example, hardlink and unlink, it would solve the concurrency problems?

I can provide more information on the problem upon request.

Thanks again,
Filipe

-------------------------------------------------------
This SF.net email is sponsored by: 2005 Windows Mobile Application Contest Submit applications for Windows Mobile(tm)-based Pocket PCs or Smartphones for the chance to win $25,000 and application distribution. Enter today at
http://ads.osdn.com/?ad_id=6882&alloc_id=15148&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

#14 lore raw
In reply to Filipe Brandenburger
Quoted 38 lines · Filipe Brandenburger

Hi, there.

I'm trying to make my program work correctly over NFS, but apparently I still have a race condition, because I'm getting inconsistencies. I still can't find out exactly what is happening. This is what I'm doing:

f = open("myfile", O_CREAT|O_WRONLY|O_APPEND)

[repeat the loop below for the data]
hardlink("myfile", "myfile.lock")
-> if it fails, invalidate information
write(f, "mydata", sizeof(mydata))
unlink("myfile.lock")
[end loop]

close(f)

I would expect to get consistent data, as "hardlink" should be an atomic operation over NFS (according to the previous posts on this thread). But I'm still having problems.

What I think is happening is that one process in one host is silently overwriting data just written by another process on another host (this overwriting happens on the same second interval), instead of appending to the end of the file, just after that other process record...

I mean, I found out that append writes are not atomic in Linux 2.4, but if I have a hardlink operation just before the write, it should fix the issue, shouldn't it?

Or somehow isn't the hardlink operation atomic? Or is the information of the file size (for the append) being held in some cache, and not being looked up just before each write (and just after we assure the file is ours with the lockfile hardlink)? Could you explain what is happening? Could you suggest a way to fix this and reliably append data to the file?

Quoted 7 lines

I can provide more information upon request. By the way, the NFS client platform is Linux RedHat 7.3, with kernel from RHAS 2.1, uname returns 2.4.9-e.49enterprise, NFS servers are a mix of EMC Celerra, EMC NS-700 (Clariion) and Netapp FAS940.

Thanks a lot,
Filipe

hi filipe-

i won't even pretend to be an expert on this list, but i have successfully implemented a locking scheme using link(2) on nfs and can tell you what i did.

from the readme, here is a snippet:

...
...
...

BASIC ALGORITHIM

* create a globally uniq filename in the same filesystem as the desired lockfile - this can be nfs mounted

* link(2) this file to the desired lockfile, ignore all errors

* stat the uniq filename and desired lockfile to determine is they are the same, use only stat.rdev and stat.ino - ignore stat.nlink as NFS can cause this report incorrect values

* iff same, you have lock. either return or run optional code block with optional refresher thread keeping lockfile fresh during execution of code block, ensuring that the lockfile is removed..

* iff not same try again a few times in rapid succession (poll), then, iff this fails, sleep using incremental backoff time. optionally remove lockfile if it is older than a certain time (stale), timeout if more than a certain amount of time has passed attempting to lock file.

...
...
...

you can download the package, which includes both a ruby library and commandline tool for safely performing nfs link based locks at

http://www.codeforpeople.com/lib/ruby/lockfile/

you can read the code for the latest version here

http://www.codeforpeople.com/lib/ruby/lockfile/lockfile-1.3.0/lib/lockfile.rb

the commandline tool lets you lock a file and run another program while running it, eg

  rlock foo.lock -- myprog.exe --option-for-myprog

this runs the the code while holding the lock.

in any case, i suspect your issue is assuming the return code for link(2) is correct - you have to do the stats and check fields...

at least, that is what people on this list helped me to do and it works under quite heavy testing.

kind regards.

-a

Signature · 16 lines
===============================================================================
| EMAIL   :: Ara [dot] T [dot] Howard [at] noaa [dot] gov
| PHONE   :: 303.497.6469
| When you do something, you should burn yourself completely, like a good
| bonfire, leaving no trace of yourself.  --Shunryu Suzuki
===============================================================================


-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
#15 lore raw

Hi, there.

Are the stat's really necessary? I mean, in another post in this same thread, Trond told that for hardlink, symlink and mkdir, both of them are "atomic" in NFS meaning that they will succeed only for one client and fail for all others if called at the same time. For me at least, that means the stat's would not be needed... Isn't that so?

On another quite old post, I have read that even on NFS 2 hardlink works in an atomic way, so that it could be used to implement a lock. I mean, stat's certainly are safe, but this seems too expensive for me! Is this extra safety really needed?

Thanks,
Filipe

* Mon, 21 Mar 2005 21:13:49 -0700 (MST), "Ara.T.Howard" <[email protected]>:

Quoted 12 lines · Filipe Brandenburger

BASIC ALGORITHIM

* create a globally uniq filename in the same filesystem as the desired lockfile - this can be nfs mounted

* link(2) this file to the desired lockfile, ignore all errors

* stat the uniq filename and desired lockfile to determine is they are the same, use only stat.rdev and stat.ino - ignore stat.nlink as NFS can cause this report incorrect values

Quoted 8 lines

* iff same, you have lock. either return or run optional code block with optional refresher thread keeping lockfile fresh during execution of code block, ensuring that the lockfile is removed..

* iff not same try again a few times in rapid succession (poll), then, iff this fails, sleep using incremental backoff time. optionally remove lockfile if it is older than a certain time (stale), timeout if more than a certain amount of time has passed attempting to lock file.

-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs