Skip to contentSkip to conversation

Nanosecond resolution for stat(2)

DISCUSSION Andi Kleen 5 participants 5 messages Sep 23, 2002 – Sep 24, 2002

lore
Share & tools
lore

Andi Kleen

author
#0 lore raw

[The original message for this included the patch and didn't make it to l-k likely because it was too big. Reposted with out-of-line patch]

Linux currently uses second resolution (time_t) for st_[cam]time in stat(2). This is a problem for highly parallel make -j, which cannot detect cases when a make rule runs for less than a second. GNU make supports finer grained timestamps for this on other OS, but Linux doesn't support it so far. This patch changes that. We also have several filesystems in tree now that support better than second resolution for [cam]time in their on disk inode (XFS,JFS,NFSv3 and VFAT).

This patch extends the VFS and stat(2) to add nsec timestamps.

Why nsecs? First to be compatible with Solaris and then when you add a new 32bit field then there is no reason to stop at msec. It just uses a POSIX struct timespec. This matches what the filesystems (NFSv3,JFS,XFS) do.

The real resolution is a jiffie current because it just uses xtime instead of calling gettimeofday. In 2.5 that is 1ms, which should be hopefully good enough. If not we can change it later to use do_gettimeofday. do-gettimeofday unfortunately takes a readlock currently on most architectures, so before doing it it would be a good idea to fix at least i386 to use lockless gettimeofday (implementations of that exist already). But xtime should be good enough for now.

I chose to reuse the "reserved for year 2036" fields in stat64 for nsec, because y2036 will need many other system call and glibc changes anyways (e.g. new time, new gettimeofday, glibc support) so adding a new stat64 by then won't be a big deal. The newer architectures have enough

The current kernels fill the fields now reused for nsec always with 0, so there is perfect compatibility.

On stat64 these fields are always there because everybody uses the glibc layout. With stat on 64bit architectures it is unfortunately mixed. The newer 64bit architectures use the stat64 layout. The older ones unfortunately didn't reserve fields for this (this is mainly alpha) I think. For now alpha has no way to get at the nsec values. Fixing it probably requires a new stat64 call for alpha.

I had to add a preprocessor symbol for this case.

I fixed all the architectures for it.

The old utimes system call already supported timeval, so it works fine (that is ms instead of ns resolution, but should be good enough for now)

I changed the inode and iattr fields to struct timespec. and fixed all the file systems and other code that accessed it. The rounding in general is a bit crude from seconds - it should round, but they are currently just truncated.

Some drivers (like mouse drivers or tty) do dubious inode [mac] time accesses of the on disk inode and without even marking it dirty. This is likely a bug. I fixed some of them but left others of these alone for now, but should probably be all fixed.

[Linus noted that the tty drivers does this to keep 'w' updated. The patch keeps this. It's probably nonsense for the mouse drivers and partly removed there.]

I didn't fix Intermezzo completely because it didn't compile at all.

This patch could in theory affect benchmarks a bit. Andrew Morton previously did an optimization to put inodes only once a second onto the dirty list when their [mca]time change. With this patch they will be put on the dirty list each jiffie (1ms), so in the worst case 1000 times as often. The cost in this is mainly in taken the locks and putting the inode onto the dirty list. On many FS which do not have better than a second resolution this makes no sense, because they only change the value once a second anyways. If this should be a problem a new update_time file/inode operation may need to be added. I didn't do this for now.

The kernel internally always keeps the nsec (or rather 1ms) resolution stamp. When a filesystem doesn't support it in its inode (like ext2) and the inode is flushed to disk and then reloaded then an application that is nanosecond aware could in theory see a backwards jumping time. I didn't do anything anything against that yet, because it looks more like a theoretical problem for me. If it should be one in practice it could be fixed by rounding the time up in this case.

Patch for 2.5.38 can be found at ftp://ftp.firstfloor.org/pub/ak/v2.5/nsec-2.5.38-1.gz

-Andi

#1 lore raw
Quoted 6 lines · Andi Kleen

The kernel internally always keeps the nsec (or rather 1ms) resolution stamp. When a filesystem doesn't support it in its inode (like ext2) and the inode is flushed to disk and then reloaded then an application that is nanosecond aware could in theory see a backwards jumping time. I didn't do anything anything against that yet, because it looks more like a theoretical problem for me.

I assume you mean "theoretical" that an application would care, not that it would happen. (Unless I misunderstand, it is nearly guaranteed to happen every time an inode is evicted after a [mac]time update.)

I fear that there are applications that will be harmed by any spurious change in [mac]time, even if it's not backwards. Apps that trigger on any change in mtime may trigger twice for every change. Eg, I suspect there is some scenario in which an rsync-like application that supports nanoseconds could suffer (just in performance, but still).

Quoted 2 lines

If it should be one in practice
it could be fixed by rounding the time up in this case.

This would mean that even "legacy" programs that only use second resolution would be affected, which seems worse. At least programs that recognize the nanosecond field are more likely to know about the issue.

Andrew

#2 lore raw
Quoted 7 lines · Andrew Pimlott
Quoted 6 lines · Andi Kleen

The kernel internally always keeps the nsec (or rather 1ms) resolution stamp. When a filesystem doesn't support it in its inode (like ext2) and the inode is flushed to disk and then reloaded then an application that is nanosecond aware could in theory see a backwards jumping time. I didn't do anything anything against that yet, because it looks more like a theoretical problem for me.

Quoted 7 lines

...
I fear that there are applications that will be harmed by any spurious change in [mac]time, even if it's not backwards. Apps that trigger on any change in mtime may trigger twice for every change. Eg, I suspect there is some scenario in which an rsync-like application that supports nanoseconds could suffer (just in performance, but still).

The behaviour does seem wrong. Resolution should not be faked to be more accurate than the granularity offered by the underlying file system. Timestamps can be persistently stored, or stored for longer periods of times, for all sorts of reasons beyond 'make', each with consequence that cannot be determined here.

What would it take to get microsecond or better time stored in ext[23]?

mark

Signature · 9 lines
[email protected]/[email protected]/[email protected] __________________________
.  .  _  ._  . .   .__    .  . ._. .__ .   . . .__  | Neighbourhood Coder
|\/| |_| |_| |/    |_     |\/|  |  |_  |   |/  |_   | 
|  | | | | \ | \   |__ .  |  | .|. |__ |__ | \ |__  | Ottawa, Ontario, Canada

  One ring to rule them all, one ring to find them, one ring to bring them all
                       and in the darkness bind them...

                           http://mark.mielke.cc/
#3 lore raw
Quoted 7 lines · Mark Mielke

The behaviour does seem wrong. Resolution should not be faked to be more accurate than the granularity offered by the underlying file system. Timestamps can be persistently stored, or stored for longer periods of times, for all sorts of reasons beyond 'make', each with consequence that cannot be determined here.

What would it take to get microsecond or better time stored in ext[23]?

Not very much. We have been thinking about this for a while already.

The microsecond-resolution times would be stored in a "large inode" or in an extended attribute if the inode is a regular-sized one. The latter would be a pretty big performance hit for most applications if it were only the u-second data that were being stored in the EA space. We are also looking at a better method of storing the EA data so that it is more efficient than the current EA implementation, but that is mostly tangential to your concerns.

Cheers, Andreas
--
Andreas Dilger
http://www-mddsp.enel.ucalgary.ca/People/adilger/
http://sourceforge.net/projects/ext2resize/

#4 lore raw
In reply to Andi Kleen
Quoted 8 lines · On Monday September 23, [email protected] wrote:

The kernel internally always keeps the nsec (or rather 1ms) resolution stamp. When a filesystem doesn't support it in its inode (like ext2) and the inode is flushed to disk and then reloaded then an application that is nanosecond aware could in theory see a backwards jumping time. I didn't do anything anything against that yet, because it looks more like a theoretical problem for me. If it should be one in practice it could be fixed by rounding the time up in this case.

Would it make sense, when loading a time from disk, for the low order, non-stored bits of the time to be initialised high rather than low. i.e. to 999,999,999 rather than 0.
This way time stamps would never seem to jump backwards, only forwards, which seems less likely to cause confusion and will mean that a change is not missed (I'm thinking NFS here where cache correctness depends heavily on mtime).

Also, would it make sense, for filesystems that don't store the full resolution, to make that forward jump appear as early as possible. i.e. if the mtime (ctime/atime) is earlier than the current time at the resoltion of the filesystem, then make the mtime appear to be what it would be if reloaded from storage... Maybe an example would help.

Assuming an internal resolution on 1millisecond (to save on digits) and a stored resolution of 1 second

time change is made Apparent timestamp

23.100         X                      23.100
23.200                                23.100
23.300         X                      23.300
23.500         X                      23.500
23.900                                23.500
24.001                                23.999
25.000                                23.999

Thus the only incorrect observation that an application can make is that there is an extra change at the end of a second when other changes were made. I think this is better than an apparent change suddenly becoming visible many minutes after the time of that apparent change, and definately better than a timestamp moving backwards.

NeilBrown