Clarifying XFS behaviour for dates before 1901 and after 2038
Hi all
Today I was running some file timestamping tests to get a feel for the bigtime XFS option and to confirm that it was doing what I expected. Everything made sense until a certain point.
There are two systems:
* PC-1: Slackware64 15.0, xfsprogs 5.13.0, 5.15.27 kernel
* PC-2: Slackware64 14.2, xfsprogs 4.3.0, 4.4.19 kernel
On PC-1 with an xfs created many years ago, xfs_info reports bigtime=0 (as expected). Two tests were run:
* > touch -d '1800/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 jwoithe users 0 1901-12-14 06:15:52.000000000 +0930 foobar
* > touch -d '2100/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 jwoithe users 0 2038-01-19 13:44:07.000000000 +1030 foobar
Both results are entirely as expected: the times are clamped to the minimum and maximum values. The +0930 timezone in the 1901 date is due to there being no daylight saving in operation in 1901.
A newly created xfs is also on PC-1 where bigtime was requested during mkfs.xfs. Bigtime is confirmed set according to xfs_info. Three tests were run:
* > touch -d '1800/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 radar users 0 1901-12-14 06:15:52.000000000 +0930 foobar
* > touch -d '2100/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 radar users 0 2100-01-01 02:23:45.670000000 +1030 foobar
* > touch -d '2800/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 radar users 0 2486-07-03 05:50:24.000000000 +0930 foobar
Again, everything is as expected. Bigtime expands the maximum time out to 2486 as advertised. The +0930 timezone in the last result is due to there being no daylight saving in July.
Turning to PC-2, things became confusing. This older enviroment also has an xfs created many years ago. Two tests were run:
* > touch -d '1800/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 jwoithe users 0 1800-01-01 02:23:45.670000000 +0914 foobar
* > touch -d '2100/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 jwoithe users 0 2100-01-01 02:23:45.670000000 +1030 foobar
Since the kernel on PC-2 is way earlier than 5.10 and its xfs filesystems predate bigtime, I would have expected the times to be clamped between 1901 and 2038. However, it seems that the system somehow manages to store the out-of-bound years. Doing so has an interesting effect on the timezone offset for the pre-1901 years, but for years beyond 2038 there is no directly observable problem. Incidently, running
stat foobar
happily reports the extended date in this case too:
Access: 2100-01-01 02:23:45.670000000 +1030
Modify: 2100-01-01 02:23:45.670000000 +1030
For a giggle I tried
Quoted 1 line
touch -d '21000/01/01 02:23:45.67' foobar
and the system still managed to store the 5-digit year:
-rw-r--r-- 1 jwoithe users 0 21000-01-01 02:23:45.670000000 +1030 foobar
This isn't what I expected. Given an old userspace with an old kernel and old xfs filesystem, dates outside the 1901-2038 range should not be possible. Given the apparent corruption of the timezone field when a year before 1901 is set, one naive thought is that the apparent success of these extended years on PC-2 (the old system) is due to a lack of bounds checking on the time value and (presumedly) some overflow within on-disc structures as a result. This would have been noticed way before now though.
I am therefore curious about the reason for the above behaviour. What subtlety am I missing?
While this may be a well known quirk, it is rather difficult to search for online. I've tried a few things but they haven't turned up any relevant matches. Apologies if this is an FAQ that I can't locate.
Regards
jonathan
Quoted 88 lines · Jonathan Woithe
Hi all
Today I was running some file timestamping tests to get a feel for the bigtime XFS option and to confirm that it was doing what I expected. Everything made sense until a certain point.
There are two systems:
* PC-1: Slackware64 15.0, xfsprogs 5.13.0, 5.15.27 kernel
* PC-2: Slackware64 14.2, xfsprogs 4.3.0, 4.4.19 kernel
On PC-1 with an xfs created many years ago,
xfs_inforeports bigtime=0 (as expected). Two tests were run:* > touch -d '1800/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 jwoithe users 0 1901-12-14 06:15:52.000000000 +0930 foobar
* > touch -d '2100/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 jwoithe users 0 2038-01-19 13:44:07.000000000 +1030 foobar
Both results are entirely as expected: the times are clamped to the minimum and maximum values. The +0930 timezone in the 1901 date is due to there being no daylight saving in operation in 1901.
A newly created xfs is also on PC-1 where bigtime was requested during mkfs.xfs. Bigtime is confirmed set according to
xfs_info. Three tests were run:* > touch -d '1800/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 radar users 0 1901-12-14 06:15:52.000000000 +0930 foobar
* > touch -d '2100/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 radar users 0 2100-01-01 02:23:45.670000000 +1030 foobar
* > touch -d '2800/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 radar users 0 2486-07-03 05:50:24.000000000 +0930 foobar
Again, everything is as expected. Bigtime expands the maximum time out to 2486 as advertised. The +0930 timezone in the last result is due to there being no daylight saving in July.
Turning to PC-2, things became confusing. This older enviroment also has an xfs created many years ago. Two tests were run:
* > touch -d '1800/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 jwoithe users 0 1800-01-01 02:23:45.670000000 +0914 foobar
* > touch -d '2100/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 jwoithe users 0 2100-01-01 02:23:45.670000000 +1030 foobar
Since the kernel on PC-2 is way earlier than 5.10 and its xfs filesystems predate bigtime, I would have expected the times to be clamped between 1901 and 2038. However, it seems that the system somehow manages to store the out-of-bound years. Doing so has an interesting effect on the timezone offset for the pre-1901 years, but for years beyond 2038 there is no directly observable problem. Incidently, running
stat foobar
happily reports the extended date in this case too:
Access: 2100-01-01 02:23:45.670000000 +1030
Modify: 2100-01-01 02:23:45.670000000 +1030For a giggle I tried
Quoted 1 line
touch -d '21000/01/01 02:23:45.67' foobar
and the system still managed to store the 5-digit year:
-rw-r--r-- 1 jwoithe users 0 21000-01-01 02:23:45.670000000 +1030 foobar
This isn't what I expected. Given an old userspace with an old kernel and old xfs filesystem, dates outside the 1901-2038 range should not be possible. Given the apparent corruption of the timezone field when a year before 1901 is set, one naive thought is that the apparent success of these extended years on PC-2 (the old system) is due to a lack of bounds checking on the time value and (presumedly) some overflow within on-disc structures as a result. This would have been noticed way before now though.
Quoted 2 lines
I am therefore curious about the reason for the above behaviour. What subtlety am I missing?
Older kernels (pre-5.4 I think?) permitted userspace to store arbitrary 64-bit timestamps in the in-memory inode. The fs would truncate (== rip off the upper bits) them when writing them to disk, and then you'd get the shattered remnants the next time the inode got reloaded from disk.
Nowadays, filesystems advertise the timestamp range they support, and the VFS clamps the in-memory timestamp to that range.
--D
Quoted 6 lines
While this may be a well known quirk, it is rather difficult to search for online. I've tried a few things but they haven't turned up any relevant matches. Apologies if this is an FAQ that I can't locate.
Regards
jonathan
Hi Darrick
Quoted 19 lines · Darrick J. Wong
Quoted 15 lines · Jonathan Woithe
Since the kernel on PC-2 is way earlier than 5.10 and its xfs filesystems predate bigtime, I would have expected the times to be clamped between 1901 and 2038. However, it seems that the system somehow manages to store the out-of-bound years. Doing so has an interesting effect on the timezone offset for the pre-1901 years, but for years beyond 2038 there is no directly observable problem. ...
:
This isn't what I expected. Given an old userspace with an old kernel and old xfs filesystem, dates outside the 1901-2038 range should not be possible. Given the apparent corruption of the timezone field when a year before 1901 is set, one naive thought is that the apparent success of these extended years on PC-2 (the old system) is due to a lack of bounds checking on the time value and (presumedly) some overflow within on-disc structures as a result. This would have been noticed way before now though.Quoted 2 lines
I am therefore curious about the reason for the above behaviour. What subtlety am I missing?
Quoted 7 lines
Older kernels (pre-5.4 I think?) permitted userspace to store arbitrary 64-bit timestamps in the in-memory inode. The fs would truncate (== rip off the upper bits) them when writing them to disk, and then you'd get the shattered remnants the next time the inode got reloaded from disk.
Nowadays, filesystems advertise the timestamp range they support, and the VFS clamps the in-memory timestamp to that range.
Thanks so much for taking the time to respond. That all makes sense and is useful information to keep in mind.
Regards
jonathan
Hi all
Quoted 7 lines · Jonathan Woithe
Turning to PC-2, things became confusing. This older environment also has an xfs created many years ago. Two tests were run:
* > touch -d '1800/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 jwoithe users 0 1800-01-01 02:23:45.670000000 +0914 foobar
Quoted 7 lines
* > touch -d '2100/01/01 02:23:45.67' foobar
Quoted 1 line
ls --full-time foobar
-rw-r--r-- 1 jwoithe users 0 2100-01-01 02:23:45.670000000 +1030 foobar
... However, it seems that the system somehow manages to store the out-of-bound years. Doing so has an interesting effect on the timezone offset for the pre-1901 years, ...
Following the information provided by Darrick I still wanted to understand the "+0914" timezone offset. I've done some research and it turns out that prior to 1 February 1895 the time zone in this part of the world really was UT+0914 (or, to be entirely accurate, UT+0914.5) [1]. In those days noon was defined to be the time that sun was directly overhead, hense the odd-ball timezone. The "+0914" timezone reported in the year-1800 case is therefore correct.
Regards
jonathan