nanosecond file timestamp resolution in filesystems, GNU make, etc.
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): 200210280947.g9S9l9H01162@sic.twinsun.com.suse.lists.linux.kernel
Quoted 6 lines · Paul Eggert
Quoted 2 lines
Another way would be to round on flush, but that also has some problems :-
Rounding is even worse. GNU Make assumes truncation, i.e. it assumes that a timestamp is truncated (floored, actually) when it is stored on a non-nanosecond-aware filesystem.
That is what my patchkit does currently, so I guess it should work fine.
Quoted 8 lines
Quoted 4 lines
In my current patchkit I just chose to truncate because that was the easiest and the other more complicated solutions didn't offer any compeling advantage.
Can't you truncate/floor to filesystem timestamp resolution immediately, i.e., before the inode is flushed? That would address the problems that I see.
That would complicate the timestamp management in the kernel considerably. I'm not sure if I want to do that, probably not.
-Andi
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 3 lines · Jamie Lokier
Unfortunately that application code breaks when the filesystem may have timestamps with resolution better than 1 second, but worse than 1 nanosecond.
The current resolution is jiffies, which tends to be 1ms
Then the application just can't do the right thing,
Quoted 2 lines
unless it knows what rounding was applied by the kernel/filesystem, so it can change that rounding in a safe direction.
The rounding is always truncation. So the application can just assume that.
-Andi
Quoted 4 lines · Andi Kleen
Quoted 3 lines
Unfortunately that application code breaks when the filesystem may have timestamps with resolution better than 1 second, but worse than 1 nanosecond.
Quoted 8 lines
The current resolution is jiffies, which tends to be 1ms
Then the application just can't do the right thing,
Quoted 2 lines
unless it knows what rounding was applied by the kernel/filesystem, so it can change that rounding in a safe direction.
The rounding is always truncation. So the application can just assume that.
This is fine when you are comparing two files with the same timestamp resolution, but when the resolutions are different you need to know what they are.
Come to think of it, rounding up is no better than rounding down when comparing two files. The application needs to round one of them up and one of them down, in order to make reliable tests of the form "is this file definitely newer than this other file".
For those kinds of tests, the application needs to know a lower bound of the resolution. Note that a jiffie is not suitable as the lower bound, because that part of the timestamp is dropped when the inode is dropped from memory.
The other kind of test is a comparison of one file against against its own modification time when something derived from the file was last cached. (This is appropriate for server requests and JIT compiler launching, for example).
This time there is only one resolution. Nevertheless, to make a reliable test of the form "have the contents of the file definitely not been modified since mtime T", neither form of rounding on the kernel side is sufficient: the application needs to know the resolution.
I think that in all cases, for the application to make useful decisions it needs to know the resolution of the timestamps in any particular struct stat. If those resolutions change when an inode is flushed from memory: that should change the resolution returned by struct stat.
So I propose: add a field to struct stat indicating the resolution of the timestamps in it. It can go on the end.
-- Jamie
Quoted 2 lines
So I propose: add a field to struct stat indicating the resolution of the timestamps in it. It can go on the end.
It's impossible. There is no space left in struct stat64 And adding a new syscall just for that would be severe overkill.
But what you could do if you really wanted that: implement kernel POSIX pathconf()/fpathconf() and implement it as a parameter to that.
kernel pathconf would be needed for some other reasons anyways, e.g. to return proper max hard link counts (currently glibc hardcodes the parameters for various fs in user space and it always breaks the LSB test suite for new file systems). Ulrich Drepper could probably give you other reasons on why it is needed if you ask him nicely.
I personally have no plans to implement it, however, because it looks like kernel bloat to me :-)
-Andi
Quoted 5 lines · Andi Kleen
Quoted 2 lines
So I propose: add a field to struct stat indicating the resolution of the timestamps in it. It can go on the end.
It's impossible. There is no space left in struct stat64 And adding a new syscall just for that would be severe overkill.
Well, possibly more stuff could benefit from new stat syscalls, like ast_gen member for inode generations. And as someone suggested, a version number or a length could be specified by the calls this time to permit less disturbing expansion in the future.
Signature · 3 lines
+ Maciej W. Rozycki, Technical University of Gdansk, Poland + +--------------------------------------------------------------+ + e-mail: [email protected], PGP key available +
Quoted 7 lines · Maciej W. Rozycki
Quoted 2 lines
It's impossible. There is no space left in struct stat64 And adding a new syscall just for that would be severe overkill.
Well, possibly more stuff could benefit from new stat syscalls, like a
st_genmember for inode generations. And as someone suggested, a version number or a length could be specified by the calls this time to permit less disturbing expansion in the future.
It's already there. The kernel stat64() syscall has a flags argument, which is unused at the moment. I presume it's for this purpose.
Glibc aleady uses a version number for its stat() calls, to permit binary compatible extensions on the user side.
So all the mechanism is there AFAIK.
-- Jamie
Quoted 7 lines · Jamie Lokier
Quoted 4 lines
Well, possibly more stuff could benefit from new stat syscalls, like a
st_genmember for inode generations. And as someone suggested, a version number or a length could be specified by the calls this time to permit less disturbing expansion in the future.It's already there. The kernel
stat64()syscall has a flags argument, which is unused at the moment. I presume it's for this purpose.
Hmm, I haven't thought of this argument to be used this way. Actually it
isn't currently initialized by glibc in any way, which makes its utility questionable.
Quoted 2 lines
Glibc aleady uses a version number for its
stat()calls, to permit binary compatible extensions on the user side.
Well, it used to use xstat() functions that provided versioning since the
old days and now ELF symbol versioning is used, too, so the userland is long prepared.
Signature · 3 lines
+ Maciej W. Rozycki, Technical University of Gdansk, Poland + +--------------------------------------------------------------+ + e-mail: [email protected], PGP key available +
Quoted 6 lines · Maciej W. Rozycki
Quoted 2 lines
It's already there. The kernel
stat64()syscall has a flags argument, which is unused at the moment. I presume it's for this purpose.Hmm, I haven't thought of this argument to be used this way. Actually it
isn't currently initialized by glibc in any way, which makes its utility questionable.
You are right. I just checked dietlibc and uclibc - neither of them initialise the flags argument. It should be deleted from the kernel, because nobody can use it.
On the bright side (for my specific request of st_resolution), it seems every architecture has a different size reserved for st_dev and st_rdev in struct stat64:
- i386: 12 bytes (8 bytes used by Glibc)
- SPARC: 8 bytes (all needed for 8 byte dev_t, but space elsewhere)
- ARM: 12 bytes
- MIPS: 16 bytes (!)
- S390: 12 bytes
- Alpha: not obvious what's used - is int 64 bits wide on Alpha? if it's not, other changes need for 64-bit dev_t anyway.
All the architectures I've looked at have two words available in struct stat64, if they have struct stat64, but the available space is in different places for each architecture.
-- Jamie
Quoted 4 lines · Andi Kleen
Quoted 2 lines
So I propose: add a field to struct stat indicating the resolution of the timestamps in it. It can go on the end.
It's impossible. There is no space left in struct stat64 And adding a new syscall just for that would be severe overkill.
There's the 4 bytes following dev_t in Glibc's definition (call __pad1 in Glibc) which isn't used by Glibc or anything else. That's still free even when the kernel changes to 64-bit dev_t.
And if you don't like that to do that, i.e you want to guarantee that the unused word continues to be zero for older programs (though I can't think of any reason why it would matter), that's what the (currently unused) flags argument to stat64() is for.
Quoted 3 lines
But what you could do if you really wanted that: implement kernel POSIX pathconf()/fpathconf() and implement it as a parameter to that.
Ugh.
Quoted 2 lines
I personally have no plans to implement it [pathconf], however, because it looks like kernel bloat to me :-)
Given the choice of using fpathconf, I'd rather accept that the nanoseconds field is not reliable, hence I must ignore it. It does seem a waste though - sometimes you are returning good information I can use, other times you're not, and I can't tell the difference :-(
I'd much rather do this right. What do you think of storing the resolution in the unused word called __pad1 in Glibc?
(Btw, it wouldn't bloat the in-core kernel inode, because only a couple of flag bits are needed there to distinguish known resolution values).
-- Jamie