Nanosecond fs timestamp support: sad

This means I can touch a file something like 70k times per second and get only 300 distinct timestamps on my laptop. And only 100 distinct timestamps on a typical distro server kernel.

Meanwhile, I can call gettimeofday 35M times per second and get ~1M distinct responses.

Given that we can do gettimeofday three orders of magnitude faster than we can do file transactions and it has four orders of magnitude better resolution, shouldn't we be using it for filesystem time when sb->s_time_gran is less than 1/HZ?

Quoted 3 lines · Matt Mackall

This means I can touch a file something like 70k times per second and get only 300 distinct timestamps on my laptop. And only 100 distinct timestamps on a typical distro server kernel.

You should use the inode generation number if you really want to see every update.

Quoted 3 lines

Meanwhile, I can call gettimeofday 35M times per second and get ~1M distinct responses.

They key word here is "I".

Quoted 4 lines

Given that we can do gettimeofday three orders of magnitude faster than we can do file transactions and it has four orders of magnitude better resolution, shouldn't we be using it for filesystem time when sb->s_time_gran is less than 1/HZ?

Some systems have a quite slow gettimeofday() That was the primary motivation for using jiffies.

Also adding more granuality makes it more expensive, because there's additional work every time it changes. Even jiffies already caused regressions.

-Andi

Quoted 7 lines · On Thu, 21 Jul 2011 23:01:24 -0700 Andi Kleen <[email protected]> wrote:

Quoted 3 lines · Matt Mackall

This means I can touch a file something like 70k times per second and get only 300 distinct timestamps on my laptop. And only 100 distinct timestamps on a typical distro server kernel.

Quoted 2 lines

You should use the inode generation number if you really want to see every update.

I assume you mean i_version which gets incremented (under a spinlock) if the filesystem asks for it.

This doesn't let you compare the ages of two files. I wonder if that is important. Is it important to you Matt?

I imagine a scheme where 'stat' would set a flag if it wasn't set, and file_update_time would:
- if the flag is set, use gettimeofday and clear the flag
- if the flag is not set, use jiffies

so if you are looking, you will see i_mtime changing precisely but if not, you don't pay the price.

This wouldn't allow precise ordering of distinct files either of course.

Indeed. Only usefully exists on ext4 and requires extra system calls.

Quoted 1 line

Indeed. Only usefully exists on ext4 and requires extra system calls.

Not sure what you mean? It's in stat(2), just like the timestamps.

Quoted 3 lines · Andi Kleen

Quoted 1 line

Indeed. Only usefully exists on ext4 and requires extra system calls.

Not sure what you mean? It's in stat(2), just like the timestamps.

I don't see anything that looks like a version or generation number in either the man pages, the asm-generic/stat.h, or glibc's asm/stat.h. Pointer?

Quoted 5 lines · Matt Mackall

Quoted 3 lines · Andi Kleen

Quoted 1 line

Indeed. Only usefully exists on ext4 and requires extra system calls.

Not sure what you mean? It's in stat(2), just like the timestamps.

Quoted 3 lines

I don't see anything that looks like a version or generation number in either the man pages, the asm-generic/stat.h, or glibc's asm/stat.h. Pointer?

Hmm you're right. I thought it was in there, but apparently not. I think it should be added there though. We still have some unused fields.

Quoted 6 lines · Andi Kleen

Quoted 5 lines · Matt Mackall

Quoted 3 lines · Andi Kleen

Quoted 1 line

Indeed. Only usefully exists on ext4 and requires extra system calls.

Not sure what you mean? It's in stat(2), just like the timestamps.

Quoted 7 lines

Quoted 3 lines

I don't see anything that looks like a version or generation number in either the man pages, the asm-generic/stat.h, or glibc's asm/stat.h. Pointer?

Hmm you're right. I thought it was in there, but apparently not. I think it should be added there though. We still have some unused fields.

But last I checked I thought it was only ext4 that actually incremented the i_version on IO, and even then only when given a (non-default) mount option.

My notes on what needs to be done there:

- collect data to determine whether turning on i_version causes any significant performance regressions.
- Last I talked to him, Ted Tso recommended running Bonnie on a local disk, since it does a lot of little writes, which is somewhat of a worst case, as it will generate extra metadata updates for each write. Compare total wall-clock time, number of iops, and number of bytes (using some kind of block tracing).
- If there aren't any problems, turn it on by default, and we're done.

If there are unfixable problems, consider something more complicated (like turning on i_version automatically when someone asks for it).
- We need to check that i_version is also doing something sensible on directory as well as on file inodes.
- We also need to think about what it does after reboots. (E.g. what is an nfs server to do if clients see the i_version go backwards (and hence possible repeat old values) after a reboot?)
- Double-check the order that data updates and i_version updates are done in. (Ideal would be if they were atomic, but for nfsd's purposes at least it should be adequate if the i_version comes after, and no later than the next commit.)

--b.

Quoted 21 lines · bfields

Quoted 6 lines · Andi Kleen

Quoted 5 lines · Matt Mackall

Quoted 3 lines · Andi Kleen

Quoted 1 line

Indeed. Only usefully exists on ext4 and requires extra system calls.

Not sure what you mean? It's in stat(2), just like the timestamps.

Quoted 7 lines

Quoted 3 lines

I don't see anything that looks like a version or generation number in either the man pages, the asm-generic/stat.h, or glibc's asm/stat.h. Pointer?

Hmm you're right. I thought it was in there, but apparently not. I think it should be added there though. We still have some unused fields.

But last I checked I thought it was only ext4 that actually incremented the i_version on IO, and even then only when given a (non-default) mount option.

My notes on what needs to be done there:

Quoted 10 lines

- collect data to determine whether turning on i_version causes any significant performance regressions.
- Last I talked to him, Ted Tso recommended running Bonnie on a local disk, since it does a lot of little writes, which is somewhat of a worst case, as it will generate extra metadata updates for each write. Compare total wall-clock time, number of iops, and number of bytes (using some kind of block tracing).
- If there aren't any problems, turn it on by default, and we're done.

(Well,and talk the other filesystem implementors into doing it.)

Quoted 33 lines · On Fri, 22 Jul 2011 18:31:58 -0400 "J. Bruce Fields" <[email protected]>…

Quoted 31 lines · bfields

Quoted 6 lines · Andi Kleen

Quoted 5 lines · Matt Mackall

Quoted 3 lines · Andi Kleen

Quoted 1 line

Indeed. Only usefully exists on ext4 and requires extra system calls.

Not sure what you mean? It's in stat(2), just like the timestamps.

Quoted 7 lines

Quoted 3 lines

I don't see anything that looks like a version or generation number in either the man pages, the asm-generic/stat.h, or glibc's asm/stat.h. Pointer?

Hmm you're right. I thought it was in there, but apparently not. I think it should be added there though. We still have some unused fields.

But last I checked I thought it was only ext4 that actually incremented the i_version on IO, and even then only when given a (non-default) mount option.

My notes on what needs to be done there:

- collect data to determine whether turning on i_version causes any significant performance regressions.
- Last I talked to him, Ted Tso recommended running Bonnie on a local disk, since it does a lot of little writes, which is somewhat of a worst case, as it will generate extra metadata updates for each write. Compare total wall-clock time, number of iops, and number of bytes (using some kind of block tracing).
- If there aren't any problems, turn it on by default, and we're done.

Quoted 2 lines

(Well,and talk the other filesystem implementors into doing it.)

But does anyone apart from NFSv4 actually *want* i_version as opposed to the more-generally-useful precise timestamps?

If not, we probably should tell NFSv4 to use timestamps and focus on making them work well.
??

The timestamp used doesn't need to update ever nanosecond. I think if it were just updated on every userspace->kernel transition (or effective equivalents inside kernel threads) that would be enough capture all causality. I wonder how that would be achieved.. I wonder if RCU machinery could help - doesn't it keep track of when threads schedule ... or something?

NeilBrown

Quoted 2 lines · NeilBrown

But does anyone apart from NFSv4 actually *want* i_version as opposed to the more-generally-useful precise timestamps?

It *seems* like a generally useful idea, but I don't know of any other users.

Quoted 6 lines · J. Bruce Fields

Quoted 2 lines · NeilBrown

But does anyone apart from NFSv4 actually *want* i_version as opposed to the more-generally-useful precise timestamps?

It *seems* like a generally useful idea, but I don't know of any other users.

(Out of curiosity: what actually *needs* real timestamps?:
- They're generally useful to people, of course; ("what did I change last tuesday?")

- Make uses them, though in theory perhaps it could do the same job by caching records like "object X was built from versions a, b, and c of objects A, B, and C respectively".

But a lot of uses are probably just to answer the question "did this file change since the last time I looked at it"?

Of course, however theoretically useful, there's always the argument that linux-specific interfaces are unlikely to be used by anyone except Lennart Poettering.)

--b.

Quoted 33 lines · On Fri, 22 Jul 2011 18:31:58 -0400 "J. Bruce Fields" <[email protected]>…

Quoted 31 lines · bfields

Quoted 6 lines · Andi Kleen

Quoted 5 lines · Matt Mackall

Quoted 3 lines · Andi Kleen

Quoted 1 line

Indeed. Only usefully exists on ext4 and requires extra system calls.

Not sure what you mean? It's in stat(2), just like the timestamps.

Quoted 7 lines

Quoted 3 lines

I don't see anything that looks like a version or generation number in either the man pages, the asm-generic/stat.h, or glibc's asm/stat.h. Pointer?

Hmm you're right. I thought it was in there, but apparently not. I think it should be added there though. We still have some unused fields.

But last I checked I thought it was only ext4 that actually incremented the i_version on IO, and even then only when given a (non-default) mount option.

My notes on what needs to be done there:

- collect data to determine whether turning on i_version causes any significant performance regressions.
- Last I talked to him, Ted Tso recommended running Bonnie on a local disk, since it does a lot of little writes, which is somewhat of a worst case, as it will generate extra metadata updates for each write. Compare total wall-clock time, number of iops, and number of bytes (using some kind of block tracing).
- If there aren't any problems, turn it on by default, and we're done.

Quoted 2 lines

(Well,and talk the other filesystem implementors into doing it.)

But does anyone apart from NFSv4 actually *want* i_version as opposed to the more-generally-useful precise timestamps?

If not, we probably should tell NFSv4 to use timestamps and focus on making them work well.
??

The timestamp used doesn't need to update ever nanosecond. I think if it were just updated on every userspace->kernel transition (or effective equivalents inside kernel threads) that would be enough capture all causality. I wonder how that would be achieved.. I wonder if RCU machinery could help - doesn't it keep track of when threads schedule ... or something?

Quoted 36 lines · NeilBrown

Quoted 33 lines · On Fri, 22 Jul 2011 18:31:58 -0400 "J. Bruce Fields" <[email protected]>…

Quoted 31 lines · bfields

Quoted 6 lines · Andi Kleen

Quoted 5 lines · Matt Mackall

Quoted 3 lines · Andi Kleen

Quoted 1 line

Indeed. Only usefully exists on ext4 and requires extra system calls.

Not sure what you mean? It's in stat(2), just like the timestamps.

Quoted 7 lines

Quoted 3 lines

I don't see anything that looks like a version or generation number in either the man pages, the asm-generic/stat.h, or glibc's asm/stat.h. Pointer?

Hmm you're right. I thought it was in there, but apparently not. I think it should be added there though. We still have some unused fields.

But last I checked I thought it was only ext4 that actually incremented the i_version on IO, and even then only when given a (non-default) mount option.

My notes on what needs to be done there:

- collect data to determine whether turning on i_version causes any significant performance regressions.
- Last I talked to him, Ted Tso recommended running Bonnie on a local disk, since it does a lot of little writes, which is somewhat of a worst case, as it will generate extra metadata updates for each write. Compare total wall-clock time, number of iops, and number of bytes (using some kind of block tracing).
- If there aren't any problems, turn it on by default, and we're done.

Quoted 5 lines

Quoted 2 lines

(Well,and talk the other filesystem implementors into doing it.)

But does anyone apart from NFSv4 actually *want* i_version as opposed to the more-generally-useful precise timestamps?

In theory, a microsecond timestamp (ie gtod) may already not be good enough for all applications. But i_version also doesn't allow comparing across files.

Quoted 9 lines

If not, we probably should tell NFSv4 to use timestamps and focus on making them work well.
??

The timestamp used doesn't need to update ever nanosecond. I think if it were just updated on every userspace->kernel transition (or effective equivalents inside kernel threads) that would be enough capture all causality. I wonder how that would be achieved.. I wonder if RCU machinery could help - doesn't it keep track of when threads schedule ... or something?

Sort of.

Some observations:

- we only need to go to higher resolution when two events happen in the same time quantum
- this applies at both the level of seconds and jiffies
- if the only file touched in a given quantum gets touched ago, we don't need to update its timestamp if stat wasn't also called on it in this quantum
- we never need to use a higher resolution than the global min(s_time_gran)

For instance, if a machine is idle, except for writing to a single file once a second, 1s resolution suffices.

If a machine is idle, except for writing to the same file 1000 times per second, and no one is watching it, 1s still suffices (inode is dirtied once per second).

Any time two files are touched in the same second, the second one (and later files) needs jiffies resolution. Similarly, any time two files are touched in the same jiffy, the second one should use gtod().

The global status bits needed to track this could be managed fairly efficiently with cmpxchg.

(Arguably, we should supply > 1s resolution whether they're strictly needed or not on filesystems with nanosecond support, so that people casually inspecting timestamps don't wonder where their nanoseconds went.)

Signature · 1 line

Mathematics is the supreme nostalgia of our time.

Quoted 61 lines · Matt Mackall

Quoted 36 lines · NeilBrown

Quoted 33 lines · On Fri, 22 Jul 2011 18:31:58 -0400 "J. Bruce Fields" <[email protected]>…

Quoted 31 lines · bfields

Quoted 6 lines · Andi Kleen

Quoted 5 lines · Matt Mackall

Quoted 3 lines · Andi Kleen

Quoted 1 line

Indeed. Only usefully exists on ext4 and requires extra system calls.

Not sure what you mean? It's in stat(2), just like the timestamps.

Quoted 7 lines

Quoted 3 lines

I don't see anything that looks like a version or generation number in either the man pages, the asm-generic/stat.h, or glibc's asm/stat.h. Pointer?

Hmm you're right. I thought it was in there, but apparently not. I think it should be added there though. We still have some unused fields.

But last I checked I thought it was only ext4 that actually incremented the i_version on IO, and even then only when given a (non-default) mount option.

My notes on what needs to be done there:

- collect data to determine whether turning on i_version causes any significant performance regressions.
- Last I talked to him, Ted Tso recommended running Bonnie on a local disk, since it does a lot of little writes, which is somewhat of a worst case, as it will generate extra metadata updates for each write. Compare total wall-clock time, number of iops, and number of bytes (using some kind of block tracing).
- If there aren't any problems, turn it on by default, and we're done.

Quoted 5 lines

Quoted 2 lines

(Well,and talk the other filesystem implementors into doing it.)

But does anyone apart from NFSv4 actually *want* i_version as opposed to the more-generally-useful precise timestamps?

In theory, a microsecond timestamp (ie gtod) may already not be good enough for all applications. But i_version also doesn't allow comparing across files.

Quoted 9 lines

If not, we probably should tell NFSv4 to use timestamps and focus on making them work well.
??

The timestamp used doesn't need to update ever nanosecond. I think if it were just updated on every userspace->kernel transition (or effective equivalents inside kernel threads) that would be enough capture all causality. I wonder how that would be achieved.. I wonder if RCU machinery could help - doesn't it keep track of when threads schedule ... or something?

Sort of.

Some observations:

Quoted 8 lines

- we only need to go to higher resolution when two events happen in the same time quantum
- this applies at both the level of seconds and jiffies
- if the only file touched in a given quantum gets touched ago, we don't need to update its timestamp if stat wasn't also called on it in this quantum
- we never need to use a higher resolution than the global min(s_time_gran)

Right, so there was a rough algorithm hashed out somewhere around here:

	http://thread.gmane.org/gmane.linux.kernel/1022866/focus=1024624

that depended on those observations.

NFS presents a worst-case as the standard NFSv3 read and write operations include timestamps in the result. So every single IO comes with a stat. So either you have a clock good enough to give a distinct timestamp for all of those, or you fall back on a global counter that ends up serializing all IO. I think. I admit I'm not sure I understand your proposal below.

with a stat. So either you have a clock good enough to give a distinct timestamp for all of those, or you fall back on a global counter that ends up serializing all IO. I think. I admit I'm not sure I understand

Quoted 9 lines

If not, we probably should tell NFSv4 to use timestamps and focus on making them work well.
??

The timestamp used doesn't need to update ever nanosecond. I think if it were just updated on every userspace->kernel transition (or effective equivalents inside kernel threads) that would be enough capture all causality. I wonder how that would be achieved.. I wonder if RCU machinery could help - doesn't it keep track of when threads schedule ... or something?

Sort of.

Some observations:

- we only need to go to higher resolution when two events happen in the same time quantum
- this applies at both the level of seconds and jiffies
- if the only file touched in a given quantum gets touched ago, we don't need to update its timestamp if stat wasn't also called on it in this quantum

- we never need to use a higher resolution than the global min(s_time_gran)

For instance, if a machine is idle, except for writing to a single file once a second, 1s resolution suffices.

Any time two files are touched in the same second, the second one (and later files) needs jiffies resolution. Similarly, any time two files are touched in the same jiffy, the second one should use gtod().

Hi!

Quoted 14 lines

Quoted 9 lines

If not, we probably should tell NFSv4 to use timestamps and focus on making them work well.
??

The timestamp used doesn't need to update ever nanosecond. I think if it were just updated on every userspace->kernel transition (or effective equivalents inside kernel threads) that would be enough capture all causality. I wonder how that would be achieved.. I wonder if RCU machinery could help - doesn't it keep track of when threads schedule ... or something?

Sort of.

Some observations:

Quoted 6 lines

- we only need to go to higher resolution when two events happen in the same time quantum
- this applies at both the level of seconds and jiffies
- if the only file touched in a given quantum gets touched ago, we don't need to update its timestamp if stat wasn't also called on it in this quantum

parse error aroound 'ago'.

Quoted 6 lines

- we never need to use a higher resolution than the global min(s_time_gran)

For instance, if a machine is idle, except for writing to a single file once a second, 1s resolution suffices.

Are you sure? As soon as you get network communication...

Quoted 3 lines

Any time two files are touched in the same second, the second one (and later files) needs jiffies resolution. Similarly, any time two files are touched in the same jiffy, the second one should use gtod().

For make. I don't see how this is globally true.

I do

( date; > stamp; date ) | ( sleep 5; cat > counterexample )

I know timestamp should be between two dates, but it is not.

Quoted 33 lines · On Fri, 22 Jul 2011 18:31:58 -0400 "J. Bruce Fields" <[email protected]>…

Quoted 21 lines · bfields

Quoted 6 lines · Andi Kleen

Quoted 5 lines · Matt Mackall

Quoted 3 lines · Andi Kleen

Quoted 1 line

Indeed. Only usefully exists on ext4 and requires extra system calls.

Not sure what you mean? It's in stat(2), just like the timestamps.

Quoted 7 lines

Quoted 3 lines

I don't see anything that looks like a version or generation number in either the man pages, the asm-generic/stat.h, or glibc's asm/stat.h. Pointer?

Hmm you're right. I thought it was in there, but apparently not. I think it should be added there though. We still have some unused fields.

But last I checked I thought it was only ext4 that actually incremented the i_version on IO, and even then only when given a (non-default) mount option.

My notes on what needs to be done there:

Quoted 10 lines

- collect data to determine whether turning on i_version causes any significant performance regressions.
- Last I talked to him, Ted Tso recommended running Bonnie on a local disk, since it does a lot of little writes, which is somewhat of a worst case, as it will generate extra metadata updates for each write. Compare total wall-clock time, number of iops, and number of bytes (using some kind of block tracing).
- If there aren't any problems, turn it on by default, and we're done.

Quoted 2 lines

(Well,and talk the other filesystem implementors into doing it.)

But does anyone apart from NFSv4 actually *want* i_version as opposed to the more-generally-useful precise timestamps?

If not, we probably should tell NFSv4 to use timestamps and focus on making them work well.
??

The timestamp used doesn't need to update ever nanosecond. I think if it were just updated on every userspace->kernel transition (or effective equivalents inside kernel threads) that would be enough capture all causality. I wonder how that would be achieved.. I wonder if RCU machinery could help - doesn't it keep track of when threads schedule ... or something?

NeilBrown
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Quoted 33 lines · On Fri, 22 Jul 2011 18:31:58 -0400 "J. Bruce Fields" <[email protected]>…

Quoted 21 lines · bfields

Quoted 6 lines · Andi Kleen

Quoted 5 lines · Matt Mackall

Quoted 3 lines · Andi Kleen

Quoted 1 line

Indeed. Only usefully exists on ext4 and requires extra system calls.

Not sure what you mean? It's in stat(2), just like the timestamps.

Quoted 7 lines

Quoted 3 lines

I don't see anything that looks like a version or generation number in either the man pages, the asm-generic/stat.h, or glibc's asm/stat.h. Pointer?

Hmm you're right. I thought it was in there, but apparently not. I think it should be added there though. We still have some unused fields.

But last I checked I thought it was only ext4 that actually incremented the i_version on IO, and even then only when given a (non-default) mount option.

My notes on what needs to be done there:

Quoted 10 lines

- collect data to determine whether turning on i_version causes any significant performance regressions.
- Last I talked to him, Ted Tso recommended running Bonnie on a local disk, since it does a lot of little writes, which is somewhat of a worst case, as it will generate extra metadata updates for each write. Compare total wall-clock time, number of iops, and number of bytes (using some kind of block tracing).
- If there aren't any problems, turn it on by default, and we're done.

Quoted 2 lines

(Well,and talk the other filesystem implementors into doing it.)

But does anyone apart from NFSv4 actually *want* i_version as opposed to the more-generally-useful precise timestamps?

If not, we probably should tell NFSv4 to use timestamps and focus on making them work well.
??

The timestamp used doesn't need to update ever nanosecond. I think if it were just updated on every userspace->kernel transition (or effective equivalents inside kernel threads) that would be enough capture all causality. I wonder how that would be achieved.. I wonder if RCU machinery could help - doesn't it keep track of when threads schedule ... or something?