statx, inode: document the new STATX_INO_VERSION field
Subsystems (6)
Jeff Layton
maintainerRFC PATCH v2 1/1 statx, inode: document the new STATX_INO_VERSION field
I'm proposing to expose the inode change attribute via statx [1]. Document what this value means and what an observer can infer from it changing.
[1]: https://lore.kernel.org/linux-nfs/[email protected]/T/#t
--- man2/statx.2 | 17 +++++++++++++++++ man7/inode.7 | 12 ++++++++++++ 2 files changed, 29 insertions(+)
v2: revised the definition to be more strict, since that seemed to be
consensus on desired behavior. Spurious i_version bumps would now be considered bugs, by this definition.
man2/statx.2 +17 -0
@@ -62,6 +62,7 @@ struct statx { __u32 stx_dev_major; /* Major ID */ __u32 stx_dev_minor; /* Minor ID */ __u64 stx_mnt_id; /* Mount ID */ + __u64 stx_ino_version; /* Inode change attribute */ }; .EE .in
@@ -247,6 +248,7 @@ STATX_BTIME Want stx_btime STATX_ALL The same as STATX_BASIC_STATS | STATX_BTIME. It is deprecated and should not be used. STATX_MNT_ID Want stx_mnt_id (since Linux 5.8) +STATX_INO_VERSION Want stx_ino_version (DRAFT) .TE .in .PP
@@ -411,6 +413,21 @@ and corresponds to the number in the first field in one of the records in For further information on the above fields, see .BR inode (7). .\" +.TP +.I stx_ino_version +The inode version, also known as the inode change attribute. This +value must change any time there is an inode status change. Any +operation that would cause the +.I stx_ctime +to change must also cause +.I stx_ino_version +to change, even when there is no apparent change to the +.I stx_ctime +due to coarse timestamp granularity. +.IP +An observer cannot infer anything about the nature or magnitude of the change +from the value of this field. A change in this value only indicates that +there has been an explicit change in the inode. .SS File attributes The .I stx_attributes
man7/inode.7 +12 -0
@@ -184,6 +184,18 @@ Last status change timestamp (ctime) This is the file's last status change timestamp. It is changed by writing or by setting inode information (i.e., owner, group, link count, mode, etc.). +.TP +Inode version (i_version) +(not returned in the \fIstat\fP structure); \fIstatx.stx_ino_version\fP +.IP +This is the inode change attribute. Any operation that would result in a change +to \fIstatx.stx_ctime\fP must result in a change to this value. The value must +change even in the case where the ctime change is not evident due to coarse +timestamp granularity. +.IP +An observer cannot infer anything from the returned value about the nature or +magnitude of the change. If the returned value is different from the last time +it was checked, then something has made an explicit change to the inode. .PP The timestamp fields report time measured with a zero point at the .IR Epoch ,
Signature · 1 line
2.37.2
* Jeff Layton:
Quoted 9 lines
man2/statx.2 +5
-0@@ -411,6 +413,21 @@ and corresponds to the number in the first field in one of the records in For further information on the above fields, see .BR inode (7). .\" +.TP +.I stx_ino_version +The inode version, also known as the inode change attribute. This +value must change any time there is an inode status change. Any +operation that would cause the
man2/statx.2 and corresponds to the number in the first field in one of the records in · line 430
Quoted 10 lines
man2/statx.2 +10
-0+.I stx_ctime +to change must also cause +.I stx_ino_version +to change, even when there is no apparent change to the +.I stx_ctime +due to coarse timestamp granularity. +.IP +An observer cannot infer anything about the nature or magnitude of the change +from the value of this field. A change in this value only indicates that +there has been an explicit change in the inode.
What happens if the file system does not support i_version?
Quoted 10 lines
man7/inode.7 +2
-0@@ -184,6 +184,18 @@ Last status change timestamp (ctime) This is the file's last status change timestamp. It is changed by writing or by setting inode information (i.e., owner, group, link count, mode, etc.). +.TP +Inode version (i_version)
man7/inode.7 Last status change timestamp (ctime) · line 198
Quoted 10 lines
man7/inode.7 +10
-0+(not returned in the \fIstat\fP structure); \fIstatx.stx_ino_version\fP +.IP +This is the inode change attribute. Any operation that would result in a change +to \fIstatx.stx_ctime\fP must result in a change to this value. The value must +change even in the case where the ctime change is not evident due to coarse +timestamp granularity. +.IP +An observer cannot infer anything from the returned value about the nature or +magnitude of the change. If the returned value is different from the last time +it was checked, then something has made an explicit change to the inode.
What is the wraparound behavior for i_version? Does it use the full 64-bit range?
If the system crashes without flushing disks, is it possible to observe new file contents without a change of i_version?
Thanks,
Florian
Jeff Layton
maintainerQuoted 15 lines · Florian Weimer
* Jeff Layton:
Quoted 13 lines
man2/statx.2 +9
-0@@ -411,6 +413,21 @@ and corresponds to the number in the first field in one of the records in For further information on the above fields, see .BR inode (7). .\" +.TP +.I stx_ino_version +The inode version, also known as the inode change attribute. This +value must change any time there is an inode status change. Any +operation that would cause the +.I stx_ctime +to change must also cause +.I stx_ino_version +to change, even when there is no apparent change to the
Quoted 9 lines
man2/statx.2 and corresponds to the number in the first field in one of the records in · line 430
Quoted 6 lines
man2/statx.2 +6
-0+.I stx_ctime +due to coarse timestamp granularity. +.IP +An observer cannot infer anything about the nature or magnitude of the change +from the value of this field. A change in this value only indicates that +there has been an explicit change in the inode.What happens if the file system does not support
i_version?
The STATX_INO_VERSION bit will not be set in stx_mask field of the response.
Quoted 15 lines
Quoted 15 lines
man7/inode.7 +7
-0@@ -184,6 +184,18 @@ Last status change timestamp (ctime) This is the file's last status change timestamp. It is changed by writing or by setting inode information (i.e., owner, group, link count, mode, etc.). +.TP +Inode version (i_version) +(not returned in the \fIstat\fP structure); \fIstatx.stx_ino_version\fP +.IP +This is the inode change attribute. Any operation that would result in a change +to \fIstatx.stx_ctime\fP must result in a change to this value. The value must +change even in the case where the ctime change is not evident due to coarse
Quoted 9 lines
man7/inode.7 Last status change timestamp (ctime) · line 198
Quoted 5 lines
man7/inode.7 +5
-0+timestamp granularity. +.IP +An observer cannot infer anything from the returned value about the nature or +magnitude of the change. If the returned value is different from the last time +it was checked, then something has made an explicit change to the inode.What is the wraparound behavior for
i_version? Does it use the full 64-bit range?
All of the existing implementations use all 64 bits. If you were to increment a 64 bit value every nanosecond, it will take >500 years for it to wrap. I'm hoping that's good enough. ;)
The implementation that all of the local Linux filesystems use track whether the value has been queried using one bit, so there you only get 63 bits of counter.
My original thinking here was that we should leave the spec "loose" to allow for implementations that may not be based on a counter. E.g. could some filesystem do this instead by hashing certain metadata?
It's arguable though that the NFSv4 spec requires that this be based on a counter, as the client is required to increment it in the case of write delegations.
Quoted 2 lines
If the system crashes without flushing disks, is it possible to observe new file contents without a change of
i_version?
Yes, I think that's possible given the current implementations.
We don't have a great scheme to combat that at the moment, other than looking at this in conjunction with the ctime. As long as the clock doesn't jump backward after the crash and it takes more than one jiffy to get the host back up, then you can be reasonably sure that i_version+ctime should never repeat.
Maybe that's worth adding to the NOTES section of the manpage?
Signature · 1 line
Jeff Layton <[email protected]>
* Jeff Layton:
Quoted 11 lines
All of the existing implementations use all 64 bits. If you were to increment a 64 bit value every nanosecond, it will take >500 years for it to wrap. I'm hoping that's good enough. ;)
The implementation that all of the local Linux filesystems use track whether the value has been queried using one bit, so there you only get 63 bits of counter.
My original thinking here was that we should leave the spec "loose" to allow for implementations that may not be based on a counter. E.g. could some filesystem do this instead by hashing certain metadata?
Hashing might have collisions that could be triggered deliberately, so probably not a good idea. It's also hard to argue that random collisions are unlikely.
Quoted 3 lines
It's arguable though that the NFSv4 spec requires that this be based on a counter, as the client is required to increment it in the case of write delegations.
Yeah, I think it has to be monotonic.
Quoted 5 lines
Quoted 2 lines
If the system crashes without flushing disks, is it possible to observe new file contents without a change of
i_version?Yes, I think that's possible given the current implementations.
Quoted 7 lines
We don't have a great scheme to combat that at the moment, other than looking at this in conjunction with the ctime. As long as the clock doesn't jump backward after the crash and it takes more than one jiffy to get the host back up, then you can be reasonably sure that i_version+ctime should never repeat.
Maybe that's worth adding to the NOTES section of the manpage?
I'd appreciate that.
Thanks,
Florian
Jeff Layton
maintainerQuoted 10 lines · Florian Weimer
* Jeff Layton:
Quoted 8 lines
All of the existing implementations use all 64 bits. If you were to increment a 64 bit value every nanosecond, it will take >500 years for it to wrap. I'm hoping that's good enough. ;)
The implementation that all of the local Linux filesystems use track whether the value has been queried using one bit, so there you only get 63 bits of counter.
Quoted 8 lines
Quoted 3 lines
My original thinking here was that we should leave the spec "loose" to allow for implementations that may not be based on a counter. E.g. could some filesystem do this instead by hashing certain metadata?
Hashing might have collisions that could be triggered deliberately, so probably not a good idea. It's also hard to argue that random collisions are unlikely.
In principle, if a filesystem could guarantee enough timestamp resolution, it's possible collisions could be hard to achieve. It's also possible you could factor in other metadata that wasn't necessarily visible to userland to try and ensure uniqueness in the counter.
Still...
Quoted 6 lines
Quoted 3 lines
It's arguable though that the NFSv4 spec requires that this be based on a counter, as the client is required to increment it in the case of write delegations.
Yeah, I think it has to be monotonic.
I think so too. NFSv4 sort of needs that anyway.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
If the system crashes without flushing disks, is it possible to observe new file contents without a change of
i_version?Yes, I think that's possible given the current implementations.
We don't have a great scheme to combat that at the moment, other than looking at this in conjunction with the ctime. As long as the clock doesn't jump backward after the crash and it takes more than one jiffy to get the host back up, then you can be reasonably sure that i_version+ctime should never repeat.
Quoted 3 lines
Quoted 1 line
Maybe that's worth adding to the NOTES section of the manpage?
I'd appreciate that.
Ok! New version of the manpage patch sent. If no one has strong objections to the proposed docs, I'll send out new kernel patches in the next day or two.
Thanks!
Signature · 1 line
Jeff Layton <[email protected]>
Jeff Layton
maintainerQuoted 20 lines · Jeff Layton
Quoted 10 lines · Florian Weimer
* Jeff Layton:
Quoted 8 lines
All of the existing implementations use all 64 bits. If you were to increment a 64 bit value every nanosecond, it will take >500 years for it to wrap. I'm hoping that's good enough. ;)
The implementation that all of the local Linux filesystems use track whether the value has been queried using one bit, so there you only get 63 bits of counter.
Quoted 8 lines
Quoted 3 lines
My original thinking here was that we should leave the spec "loose" to allow for implementations that may not be based on a counter. E.g. could some filesystem do this instead by hashing certain metadata?
Hashing might have collisions that could be triggered deliberately, so probably not a good idea. It's also hard to argue that random collisions are unlikely.
Quoted 7 lines
In principle, if a filesystem could guarantee enough timestamp resolution, it's possible collisions could be hard to achieve. It's also possible you could factor in other metadata that wasn't necessarily visible to userland to try and ensure uniqueness in the counter.
Still...
Actually, Bruce brought up a good point on IRC. The main danger here is that we might do this:
Start (i_version is at 1)
write data (i_version goes to 2)
statx+read data (observer associates data with i_version of 2) Crash, but before new i_version made it to disk Machine comes back up (i_version back at 1)
write data (i_version goes to 2)
statx (observer assumes his cache is valid)
We can mitigate this by factoring in the ctime when we do the statx. Another option though would be to factor in the ctime when we generate the new value and store it.
Here's what nfsd does today:
chattr = stat->ctime.tv_sec;
chattr <<= 30;
chattr += stat->ctime.tv_nsec;
chattr += inode_query_iversion(inode);Instead of doing this after we query it, we could do that before storing it. After a crash, we might see the value go backward, but if a new write later happens, the new value would be very unlikely to match the one that got lost.
That seems quite doable, and might be better for userland consumers overall.
Quoted 29 lines
Quoted 6 lines
Quoted 3 lines
It's arguable though that the NFSv4 spec requires that this be based on a counter, as the client is required to increment it in the case of write delegations.
Yeah, I think it has to be monotonic.
I think so too. NFSv4 sort of needs that anyway.
Quoted 11 lines
Quoted 11 lines
Quoted 2 lines
If the system crashes without flushing disks, is it possible to observe new file contents without a change of
i_version?Yes, I think that's possible given the current implementations.
We don't have a great scheme to combat that at the moment, other than looking at this in conjunction with the ctime. As long as the clock doesn't jump backward after the crash and it takes more than one jiffy to get the host back up, then you can be reasonably sure that i_version+ctime should never repeat.
Quoted 3 lines
Quoted 1 line
Maybe that's worth adding to the NOTES section of the manpage?
I'd appreciate that.
Ok! New version of the manpage patch sent. If no one has strong objections to the proposed docs, I'll send out new kernel patches in the next day or two.
Thanks!
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 21 lines · Jeff Layton
Quoted 20 lines · Jeff Layton
Quoted 18 lines · Florian Weimer
* Jeff Layton:
Quoted 11 lines
All of the existing implementations use all 64 bits. If you were to increment a 64 bit value every nanosecond, it will take >500 years for it to wrap. I'm hoping that's good enough. ;)
The implementation that all of the local Linux filesystems use track whether the value has been queried using one bit, so there you only get 63 bits of counter.
My original thinking here was that we should leave the spec "loose" to allow for implementations that may not be based on a counter. E.g. could some filesystem do this instead by hashing certain metadata?
Hashing might have collisions that could be triggered deliberately, so probably not a good idea. It's also hard to argue that random collisions are unlikely.
Quoted 6 lines
Quoted 6 lines
In principle, if a filesystem could guarantee enough timestamp resolution, it's possible collisions could be hard to achieve. It's also possible you could factor in other metadata that wasn't necessarily visible to userland to try and ensure uniqueness in the counter.
Still...
I've got one other nagging worry, about the ordering of change attribute updates with respect to their corresponding changes. I think with current implementations it's possible that the only change attribute update(s) may happen while the old file data is still visible, which means a concurrent reader could cache the old data with the new change attribute, and be left with a stale cache indefinitely.
For the purposes of close-to-open semantics I think that's not a problem, though.
There may be some previous discussion of this in mailing list archives.
--b.
Jeff Layton
maintainerQuoted 27 lines · J. Bruce Fields
Quoted 26 lines · Jeff Layton
Quoted 25 lines · Jeff Layton
Quoted 10 lines · Florian Weimer
* Jeff Layton:
Quoted 8 lines
All of the existing implementations use all 64 bits. If you were to increment a 64 bit value every nanosecond, it will take >500 years for it to wrap. I'm hoping that's good enough. ;)
The implementation that all of the local Linux filesystems use track whether the value has been queried using one bit, so there you only get 63 bits of counter.
Quoted 8 lines
Quoted 3 lines
My original thinking here was that we should leave the spec "loose" to allow for implementations that may not be based on a counter. E.g. could some filesystem do this instead by hashing certain metadata?
Hashing might have collisions that could be triggered deliberately, so probably not a good idea. It's also hard to argue that random collisions are unlikely.
In principle, if a filesystem could guarantee enough timestamp resolution, it's possible collisions could be hard to achieve. It's also possible you could factor in other metadata that wasn't necessarily visible to userland to try and ensure uniqueness in the counter.
Quoted 9 lines
Quoted 1 line
Quoted 1 line
Still...
I've got one other nagging worry, about the ordering of change attribute updates with respect to their corresponding changes. I think with current implementations it's possible that the only change attribute update(s) may happen while the old file data is still visible, which means a concurrent reader could cache the old data with the new change attribute, and be left with a stale cache indefinitely.
Yeah, that's a potential issue. The i_version is updated in inode_update_time, which does happen before the write to the pagecache.
We should probably add a note to the manpage that one should not expect any sort of atomicity between the change to the inode and the change in the value. I'm not sure we can offer much in the way of mitigation for that problem, otherwise.
Quoted 5 lines
For the purposes of close-to-open semantics I think that's not a problem, though.
There may be some previous discussion of this in mailing list archives.
Signature · 1 line
Jeff Layton <[email protected]>