inode caching

DISCUSSION Timo Sirainen 4 participants 8 messages May 26, 2008 – May 28, 2008

Subsystems (2)

NFS client NFS server

lore

Share & tools

lore

Timo Sirainen

author

2008-05-26 21:25 UTC #0 lore raw

NFS server: Linux 2.6.25
NFS client: Linux debian 2.6.25-2 (or 2.6.23.1)

If I do:

NFS client: fd1 = creat("foo"); write(fd1, "xx", 2); fsync(fd1); NFS server: unlink("foo"); creat("foo");
NFS client: fd2 = open("foo"); fstat(fd1, &st1); fstat(fd2, &st2); fstat(fd1, &st3);

The result is usually that the fstat(fd1) fails with ESTALE. But sometimes the result is st1.st_ino == st2.st_ino == st3.st_ino and st1.st_size == 2 but st2.st_size == 0. So I see two different files using the same inode number. I'd really want to avoid seeing that condition.

So what I'd want to know is:

a) Why does this happen only sometimes? I can't really figure out from the code what invalidates the fd1 inode. Apparently the second open() somehow, but since it uses the new "foo" file with a different struct inode, where does the old struct inode get invalidated?

b) Can this be fixed? Or is it just luck that it works as well as it does now?

Attached a test program. Usage:

NFS client: Mount with actimeo=2
NFS client: ./t
(Run the next two commands within 2 seconds)
NFS server: rm -f foo;touch foo
NFS client: hit enter

Once in a while the result will be:
1a: ino=15646940 size=2
1b: ino=15646940 size=2
1c: ino=15646940 size=2
2: ino=15646940 size=0
1d: ino=15646940 size=2

#include <errno.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <sys/stat.h>

int main(void) {
	struct stat st;
	int fd, fd2;
	char buf[100];

	fd = open("foo", O_RDWR | O_CREAT, 0666);
	write(fd, "xx", 2); fsync(fd);
	if (fstat(fd, &st) < 0) perror("fstat()");
	printf("1a: ino=%ld size=%ld\n", (long)st.st_ino, st.st_size);

	fgets(buf, sizeof(buf), stdin);
	if (fstat(fd, &st) < 0) perror("fstat()");
	else printf("1b: ino=%ld size=%ld\n", (long)st.st_ino, st.st_size);

	fd2 = open("foo", O_RDWR);
	if (fstat(fd, &st) < 0) perror("fstat()");
	else printf("1c: ino=%ld size=%ld\n", (long)st.st_ino, st.st_size);
	if (fstat(fd2, &st) < 0) perror("fstat()");
	else printf("2: ino=%ld size=%ld\n", (long)st.st_ino, st.st_size);
	if (fstat(fd, &st) < 0) perror("fstat()");
	else printf("1d: ino=%ld size=%ld\n", (long)st.st_ino, st.st_size);
	return 0;
}

Peter Staubach

2008-05-27 12:48 UTC #1 lore raw

Quoted 5 lines · Timo Sirainen

NFS server: Linux 2.6.25
NFS client: Linux debian 2.6.25-2 (or 2.6.23.1)
If I do:

Quoted 7 lines

NFS client: fd1 =3D creat("foo"); write(fd1, "xx", 2); fsync(fd1); NFS server: unlink("foo"); creat("foo");
NFS client: fd2 =3D open("foo"); fstat(fd1, &st1); fstat(fd2, &st2); fstat(fd1, &st3);
The result is usually that the fstat(fd1) fails with ESTALE. But sometimes the result is st1.st_ino =3D=3D st2.st_ino =3D=3D st3.st_in=

o and

Quoted 1 line

st1.st_size =3D=3D 2 but st2.st_size =3D=3D 0. So I see two different=

files

Quoted 4 lines

using the same inode number. I'd really want to avoid seeing that condition.
=20

This is really up the file system on the server. It is the one that selects the inode number when creating a new file.

Quoted 3 lines

So what I'd want to know is:
a) Why does this happen only sometimes? I can't really figure out fro=

Quoted 5 lines

the code what invalidates the fd1 inode. Apparently the second open() somehow, but since it uses the new "foo" file with a different struct inode, where does the old struct inode get invalidated?
=20

This will happen always, but you may see occasional successful fstat() calls on the client due to attribute caching and/or dentry caching.

Quoted 4 lines

b) Can this be fixed? Or is it just luck that it works as well as it does now?
=20

This can be fixed, somewhat. I have some changes to address the ESTALE situation in system calls that take filename as arguments, but I need to work with some more people to get them included. The system calls which do not take file names as arguments can not be recovered from because the file they are referring is really gone or at least not accessible anymore.

The reuse of the inode number is just a fact of life and that way that file systems work. I would suggest rethinking your application in order to reduce or eliminate any dependence that it might have.

All this said, making changes on both the server and the client is dangerous and can easily to lead to consistency and/or performance issues.

Thanx...

Quoted 15 lines

=EF=BB=BFAttached a test program. Usage:
NFS client: Mount with actimeo=3D2
NFS client: ./t
(Run the next two commands within 2 seconds)
NFS server: rm -f foo;touch foo
NFS client: hit enter=20
Once in a while the result will be:
1a: ino=3D15646940 size=3D2
1b: ino=3D15646940 size=3D2
1c: ino=3D15646940 size=3D2
2: ino=3D15646940 size=3D0
1d: ino=3D15646940 size=3D2

Quoted 2 lines

=20
---------------------------------------------------------------------=

---

Quoted 8 lines

#include <errno.h>
#include <string.h>
#include <unistd.h>
#include <fcntl.h>
#include <stdio.h>
#include <sys/stat.h>

Quoted 13 lines

int main(void) {
	struct stat st;
	int fd, fd2;
	char buf[100];

	fd =3D open("foo", O_RDWR | O_CREAT, 0666);
	write(fd, "xx", 2); fsync(fd);
	if (fstat(fd, &st) < 0) perror("fstat()");
	printf("1a: ino=3D%ld size=3D%ld\n", (long)st.st_ino, st.st_size);

	fgets(buf, sizeof(buf), stdin);
	if (fstat(fd, &st) < 0) perror("fstat()");
	else printf("1b: ino=3D%ld size=3D%ld\n", (long)st.st_ino, st.st_siz=

e);

Quoted 4 lines

	fd2 =3D open("foo", O_RDWR);
	if (fstat(fd, &st) < 0) perror("fstat()");
	else printf("1c: ino=3D%ld size=3D%ld\n", (long)st.st_ino, st.st_siz=

e);

Quoted 2 lines

	if (fstat(fd2, &st) < 0) perror("fstat()");
	else printf("2: ino=3D%ld size=3D%ld\n", (long)st.st_ino, st.st_size=

);

Quoted 2 lines

	if (fstat(fd, &st) < 0) perror("fstat()");
	else printf("1d: ino=3D%ld size=3D%ld\n", (long)st.st_ino, st.st_siz=

e);

Quoted 3 lines

	return 0;
}

=20

Timo Sirainen

author

2008-05-27 15:40 UTC #2 lore raw

Quoted 19 lines · Peter Staubach

Quoted 10 lines · Timo Sirainen
NFS server: Linux 2.6.25
NFS client: Linux debian 2.6.25-2 (or 2.6.23.1)
If I do:
NFS client: fd1 = creat("foo"); write(fd1, "xx", 2); fsync(fd1); NFS server: unlink("foo"); creat("foo");
NFS client: fd2 = open("foo"); fstat(fd1, &st1); fstat(fd2, &st2); fstat(fd1, &st3);
Quoted 7 lines
The result is usually that the fstat(fd1) fails with ESTALE. But sometimes the result is st1.st_ino == st2.st_ino == st3.st_ino and st1.st_size == 2 but st2.st_size == 0. So I see two different files using the same inode number. I'd really want to avoid seeing that condition.

Quoted 2 lines

This is really up the file system on the server. It is the one that selects the inode number when creating a new file.

I don't mind that the inode gets reused, I mind that I can't reliably detect that situation.

Quoted 9 lines

Quoted 8 lines
So what I'd want to know is:
a) Why does this happen only sometimes? I can't really figure out from the code what invalidates the fd1 inode. Apparently the second open() somehow, but since it uses the new "foo" file with a different struct inode, where does the old struct inode get invalidated?

Quoted 3 lines

This will happen always, but you may see occasional successful fstat() calls on the client due to attribute caching and/or dentry caching.

I would understand if it always failed or always succeeded, but it seems to be somewhat random now. And it's not "occational successful fstat()", but it's "occational failed fstat()". The difference shouldn't be because of attribute caching, because I specify it explicitly to two seconds and run the test within that 2 second. So the test should always hit the attribute cache, and according to you that should always cause it to succeed (but it rarely does). I think dentry caching also more or less depends on attribute cache timeout?

Quoted 12 lines

Quoted 4 lines
b) Can this be fixed? Or is it just luck that it works as well as it does now?
This can be fixed, somewhat. I have some changes to address the ESTALE situation in system calls that take filename as arguments, but I need to work with some more people to get them included. The system calls which do not take file names as arguments can not be recovered from because the file they are referring is really gone or at least not accessible anymore.

Quoted 3 lines

The reuse of the inode number is just a fact of life and that way that file systems work. I would suggest rethinking your application in order to reduce or eliminate any dependence that it might have.

The problem I have is that I need to reliably find out if a file has been replaced with a new file. So I first flush the dentry cache (chowning parent directory), stat() the file and fstat() the opened file. If fstat() fails with ESTALE or if the inodes don't match, I know that the file has been replaced and I need to re-open and re-read it. This seems to work nearly always.

Peter Staubach

2008-05-27 18:09 UTC #3 lore raw

Quoted 22 lines · Timo Sirainen

Quoted 20 lines · Peter Staubach
Quoted 18 lines · Timo Sirainen
NFS server: Linux 2.6.25
NFS client: Linux debian 2.6.25-2 (or 2.6.23.1)
If I do:
NFS client: fd1 = creat("foo"); write(fd1, "xx", 2); fsync(fd1); NFS server: unlink("foo"); creat("foo");
NFS client: fd2 = open("foo"); fstat(fd1, &st1); fstat(fd2, &st2); fstat(fd1, &st3);
The result is usually that the fstat(fd1) fails with ESTALE. But sometimes the result is st1.st_ino == st2.st_ino == st3.st_ino and st1.st_size == 2 but st2.st_size == 0. So I see two different files using the same inode number. I'd really want to avoid seeing that condition.

Quoted 8 lines

Quoted 3 lines
This is really up the file system on the server. It is the one that selects the inode number when creating a new file.
I don't mind that the inode gets reused, I mind that I can't reliably detect that situation.

Outside of shortening up the attribute cache timeout values, with the current implementation, I don't think that you are going to be able to reliably detect when the file on the server has been removed and recreated.

Quoted 14 lines

Quoted 9 lines
Quoted 9 lines
So what I'd want to know is:
a) Why does this happen only sometimes? I can't really figure out from the code what invalidates the fd1 inode. Apparently the second open() somehow, but since it uses the new "foo" file with a different struct inode, where does the old struct inode get invalidated?
Quoted 4 lines
This will happen always, but you may see occasional successful fstat() calls on the client due to attribute caching and/or dentry caching.

Quoted 10 lines

I would understand if it always failed or always succeeded, but it seems to be somewhat random now. And it's not "occational successful fstat()", but it's "occational failed fstat()". The difference shouldn't be because of attribute caching, because I specify it explicitly to two seconds and run the test within that 2 second. So the test should always hit the attribute cache, and according to you that should always cause it to succeed (but it rarely does). I think dentry caching also more or less depends on attribute cache timeout?

How did you specify the attribute cache to be 2 seconds?

The dentry based caching is also subject to timeout based verification, but typically on much longer time scales.

Quoted 17 lines

Quoted 12 lines
Quoted 5 lines
b) Can this be fixed? Or is it just luck that it works as well as it does now?
This can be fixed, somewhat. I have some changes to address the ESTALE situation in system calls that take filename as arguments, but I need to work with some more people to get them included. The system calls which do not take file names as arguments can not be recovered from because the file they are referring is really gone or at least not accessible anymore.
Quoted 4 lines
The reuse of the inode number is just a fact of life and that way that file systems work. I would suggest rethinking your application in order to reduce or eliminate any dependence that it might have.

Quoted 7 lines

The problem I have is that I need to reliably find out if a file has been replaced with a new file. So I first flush the dentry cache (chowning parent directory), stat() the file and fstat() the opened file. If fstat() fails with ESTALE or if the inodes don't match, I know that the file has been replaced and I need to re-open and re-read it. This seems to work nearly always.

This would seem to be quite implementation specific and also has some timing dependencies built-in. These would seem to me to be dangerous assumptions and heuristics to be depending upon.

Have you considered making the contents of the file itself versioned in some fashion and thus, removing dependencies on how the NFS client works and/or the file system on the NFS server?

Thanx...

Timo Sirainen

author

2008-05-27 19:13 UTC #4 lore raw

Quoted 24 lines · Peter Staubach

Quoted 13 lines
Quoted 8 lines
Quoted 8 lines
So what I'd want to know is:
a) Why does this happen only sometimes? I can't really figure out from the code what invalidates the fd1 inode. Apparently the second open() somehow, but since it uses the new "foo" file with a different struct inode, where does the old struct inode get invalidated?
Quoted 4 lines
This will happen always, but you may see occasional successful fstat() calls on the client due to attribute caching and/or dentry caching.
Quoted 10 lines
I would understand if it always failed or always succeeded, but it seems to be somewhat random now. And it's not "occational successful fstat()", but it's "occational failed fstat()". The difference shouldn't be because of attribute caching, because I specify it explicitly to two seconds and run the test within that 2 second. So the test should always hit the attribute cache, and according to you that should always cause it to succeed (but it rarely does). I think dentry caching also more or less depends on attribute cache timeout?

Quoted 1 line

How did you specify the attribute cache to be 2 seconds?

mount -o actimeo=2

Quoted 24 lines

Quoted 16 lines
Quoted 11 lines
Quoted 4 lines
b) Can this be fixed? Or is it just luck that it works as well as it does now?
This can be fixed, somewhat. I have some changes to address the ESTALE situation in system calls that take filename as arguments, but I need to work with some more people to get them included. The system calls which do not take file names as arguments can not be recovered from because the file they are referring is really gone or at least not accessible anymore.
Quoted 4 lines
The reuse of the inode number is just a fact of life and that way that file systems work. I would suggest rethinking your application in order to reduce or eliminate any dependence that it might have.
Quoted 7 lines
The problem I have is that I need to reliably find out if a file has been replaced with a new file. So I first flush the dentry cache (chowning parent directory), stat() the file and fstat() the opened file. If fstat() fails with ESTALE or if the inodes don't match, I know that the file has been replaced and I need to re-open and re-read it. This seems to work nearly always.

Quoted 7 lines

This would seem to be quite implementation specific and also has some timing dependencies built-in. These would seem to me to be dangerous assumptions and heuristics to be depending upon.
Have you considered making the contents of the file itself versioned in some fashion and thus, removing dependencies on how the NFS client works and/or the file system on the NFS server?

I guess one possibility would be to link() the file elsewhere for "a while", so that the inode wouldn't get reused until everyone's attribute caches have become flushed. That feels a bit dirty solution too though. (This is about handling Dovecot IMAP/POP3's metadata files.)

I'd still like to understand why exactly this happens though. Maybe there's a chance that this is just a bug in the current NFS implementation so I could keep using my current code (which is actually very difficult to break even with stress testing, so if this doesn't get fixed on kernel side I'll probably just leave my code as it is). I guess I'll start debugging the NFS code to find out what's really going on.

Benny Halevy

2008-05-28 05:39 UTC #5 lore raw

Quoted 64 lines · Timo Sirainen

Quoted 30 lines · Peter Staubach
Quoted 15 lines
Quoted 11 lines
Quoted 11 lines
So what I'd want to know is:
a) Why does this happen only sometimes? I can't really figure out from
the code what invalidates the fd1 inode. Apparently the second open()
somehow, but since it uses the new "foo" file with a different struct
inode, where does the old struct inode get invalidated?
Quoted 4 lines
This will happen always, but you may see occasional successful fstat() calls on the client due to attribute caching and/or dentry caching.
Quoted 15 lines
I would understand if it always failed or always succeeded, but it seems
to be somewhat random now. And it's not "occational successful fstat()",
but it's "occational failed fstat()". The difference shouldn't be because of attribute caching, because I specify it explicitly to two seconds and run the test within that 2 second. So the test should always
hit the attribute cache, and according to you that should always cause
it to succeed (but it rarely does). I think dentry caching also more or
less depends on attribute cache timeout?
Quoted 1 line
How did you specify the attribute cache to be 2 seconds?
mount -o actimeo=2
Quoted 28 lines
Quoted 16 lines
Quoted 12 lines
Quoted 5 lines
b) Can this be fixed? Or is it just luck that it works as well as it
does now?
This can be fixed, somewhat. I have some changes to address the ESTALE situation in system calls that take filename as arguments, but I need to work with some more people to get them included. The system calls which do not take file names as arguments can not be recovered from because the file they are referring is really gone or at least not accessible anymore.
Quoted 4 lines
The reuse of the inode number is just a fact of life and that way that file systems work. I would suggest rethinking your application in order to reduce or eliminate any dependence that it might have.
Quoted 8 lines
The problem I have is that I need to reliably find out if a file has been replaced with a new file. So I first flush the dentry cache (chowning parent directory), stat() the file and fstat() the opened file. If fstat() fails with ESTALE or if the inodes don't match, I know
that the file has been replaced and I need to re-open and re-read it. This seems to work nearly always.
This would seem to be quite implementation specific and also has some timing dependencies built-in. These would seem to me to be dangerous assumptions and heuristics to be depending upon.

Quoted 8 lines

Quoted 3 lines
Have you considered making the contents of the file itself versioned in some fashion and thus, removing dependencies on how the NFS client works and/or the file system on the NFS server?
I guess one possibility would be to link() the file elsewhere for "a while", so that the inode wouldn't get reused until everyone's attribute caches have become flushed. That feels a bit dirty solution too though. (This is about handling Dovecot IMAP/POP3's metadata files.)

The NFS (v2/v3) server can't guarantee you traditional Unix semantics where the inode is kept around until last close. Hard linking it to keep it around is the cleanest way you can go IMO.

Quoted 8 lines

I'd still like to understand why exactly this happens though. Maybe there's a chance that this is just a bug in the current NFS implementation so I could keep using my current code (which is actually very difficult to break even with stress testing, so if this doesn't get fixed on kernel side I'll probably just leave my code as it is). I guess I'll start debugging the NFS code to find out what's really going on.

My guess would be that the new incarnation of the inode generates the same filehandle as the old one, not just the same inode number.

Benny

J. Bruce Fields

2008-05-28 14:00 UTC #6 lore raw

Quoted 11 lines · Benny Halevy

Quoted 7 lines · Timo Sirainen
I'd still like to understand why exactly this happens though. Maybe there's a chance that this is just a bug in the current NFS implementation so I could keep using my current code (which is actually very difficult to break even with stress testing, so if this doesn't get fixed on kernel side I'll probably just leave my code as it is). I guess I'll start debugging the NFS code to find out what's really going on.
My guess would be that the new incarnation of the inode generates the same filehandle as the old one, not just the same inode number.

That sounds like a server bug (either in the server itself, or the filesystem it's exporting); the generation number is supposed to prevent this.

--b.

Timo Sirainen

author

2008-05-28 15:21 UTC #7 lore raw

Quoted 10 lines · J. Bruce Fields

Quoted 9 lines · Benny Halevy
Quoted 7 lines · Timo Sirainen
I'd still like to understand why exactly this happens though. Maybe there's a chance that this is just a bug in the current NFS implementation so I could keep using my current code (which is actually very difficult to break even with stress testing, so if this doesn't get fixed on kernel side I'll probably just leave my code as it is). I guess I'll start debugging the NFS code to find out what's really going on.

Quoted 6 lines

Quoted 2 lines
My guess would be that the new incarnation of the inode generates the same filehandle as the old one, not just the same inode number.
That sounds like a server bug (either in the server itself, or the filesystem it's exporting); the generation number is supposed to prevent this.

And if it happens, wouldn't the struct inode on NFS client be the same for both files then? I'm seeing different results for fstat() calls (just with the same st_ino).