ext4 - getting at birth time (file create time) and getting/setting nanosecond time stamps and utime
Subsystems (2)
A new version of the format for Samba server related file system extended attributes is expected in the next version of Samba so the topic of create time and nanosecond timestamps for Linux has come up again. Since some file systems don't support storing create time (birth time), and none support updating create time, or for that matter for storing any nanosecond timestamps at all (a millisecond seems like a much longer time today than when the stat structure was defined), and dos attributes, Samba server stores these in extended attributes, which is awkward on those file systems which store (different) versions of these on disk.
In the absence of any syscalls for utimes (nanosecond granularity) and no syscall for returning (or setting) create time (birth time), and ext4 likely to become popular (ext4 stores nanosecond timestamps, and create time), Samba developers (and probably other apps such as backup applications) need a way to modify the inode's nanosecond timestamps and get at file create time, perhaps via ioctl and/or xattr.
I think the chance of adding new syscalls for these and libc getting modified for this is near zero, so any opinions about the best way for Samba server to get at this information?
Quoted 5 lines · Mingming Cao
1) Ext4 adds support for date-created timestamps. But, current stat()interface doesn't support the file creation time/delete time. As far as I know, it is hard to modify or add the necessary systemcall to export this info out to user programs for now. These changes would require coordination of many projects.
2) from the linux man page utimes seems only allow to set microsecond timestamp
Signature · 3 lines
Thanks, Steve
Quoted 10 lines · Steve French
A new version of the format for Samba server related file system extended attributes is expected in the next version of Samba so the topic of create time and nanosecond timestamps for Linux has come up again. Since some file systems don't support storing create time (birth time), and none support updating create time, or for that matter for storing any nanosecond timestamps at all (a millisecond seems like a much longer time today than when the stat structure was defined), and dos attributes, Samba server stores these in extended attributes, which is awkward on those file systems which store (different) versions of these on disk.
I had proposed in the past that these file attributes be exposed to userspace as virtual xattrs, e.g. user.cr_time returning a struct timespec, but the data is stored internal to the filesystem in whatever format is most efficient for it. We only strictly need user.crtime for this, but I wouldn't object to exporting other inode attributes in this way (e.g. user.mtime, user.atime, etc).
I believe that nanosecond timestamps ARE supported with newer filesystems using stat(). There is the utimensat() system call, which takes a timespec parameter, so it looks to be suitable for setting nanosecond timestamps from userspace. I don't know if glibc() supports this, but I do see it in the kernel.
As for _updating_ create time, isn't that sort of defeating the purpose of create time?
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 3 lines · Andreas Dilger
As for _updating_ create time, isn't that sort of defeating the purpose of create time?
Yes, you would think so, wouldn't you. Unfortunately that's the way Windows behaves (create time can be updated) so that's what we have to emulate. If does indeed make no sense (it's just another timestamp meta-data that can't be used for auditing at that point).
Jeremy.
Quoted 10 lines · Jeremy Allison
Quoted 3 lines · Andreas Dilger
As for _updating_ create time, isn't that sort of defeating the purpose of create time?
Yes, you would think so, wouldn't you. Unfortunately that's the way Windows behaves (create time can be updated) so that's what we have to emulate. If does indeed make no sense (it's just another timestamp meta-data that can't be used for auditing at that point).
Updating create time does make it harder to use for auditing, but I expect that 20 years ago when this was added someone was thinking about making sure you could preserve all attributes on backup/restore especially for a system restore after disk failure - and end up with the same file metadata as when you started.
Signature · 3 lines
Thanks, Steve
Quoted 3 lines
As for _updating_ create time, isn't that sort of defeating the purpose of create time?
Maybe one would want to restore the original create times when restoring from backups?
- z
Quoted 5 lines · Zach Brown
Quoted 2 lines
As for _updating_ create time, isn't that sort of defeating the purpose of create time?
Maybe one would want to restore the original create times when restoring from backups?
Ideally that can be restricted to the special privilege that only backup programs have? I know Windows has something like that, but I'm not sure how it's used.
-- Jamie
Quoted 13 lines · Andreas Dilger
Quoted 11 lines · Steve French
A new version of the format for Samba server related file system extended attributes is expected in the next version of Samba so the topic of create time and nanosecond timestamps for Linux has come up again. Since some file systems don't support storing create time (birth time), and none support updating create time, or for that matter for storing any nanosecond timestamps at all (a millisecond seems like a much longer time today than when the stat structure was defined), and dos attributes, Samba server stores these in extended attributes, which is awkward on those file systems which store (different) versions of these on disk.
Quoted 6 lines
I had proposed in the past that these file attributes be exposed to userspace as virtual xattrs, e.g. user.cr_time returning a struct timespec, but the data is stored internal to the filesystem in whatever format is most efficient for it. We only strictly need user.crtime for this, but I wouldn't object to exporting other inode attributes in this way (e.g. user.mtime, user.atime, etc).
Yes - the fake xattr approach was the approach that I preferred (since I could do this trivially to cifs, and for ext4 it seems equally easy), but I wonder if this "path based" approach is slightly harder for Samba, since for most cases Samba maps to "handle based" equivalents (when the same operation can come in eitherpath based or
handle based - Samba mostly maps to handle based). In cifs
I also store dos attributes (as presumably vfat and ntfs could too) which if requested could return the same way (network backup application writers have asked me for an xattr for this before, but I would prefer to use a standard xattr name rather than something cifs specific). As we move to SMB2, all network file operations become handle based (except open).
It may be easier for Samba as an ioctl (handle based) rather than xattr (path based) - not sure.
Signature · 7 lines
Thanks, Steve -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 8 lines · Steve French
Quoted 6 lines · Andreas Dilger
I had proposed in the past that these file attributes be exposed to userspace as virtual xattrs, e.g. user.cr_time returning a struct timespec, but the data is stored internal to the filesystem in whatever format is most efficient for it. We only strictly need user.crtime for this, but I wouldn't object to exporting other inode attributes in this way (e.g. user.mtime, user.atime, etc).
Quoted 6 lines
Yes - the fake xattr approach was the approach that I preferred (since I could do this trivially to cifs, and for ext4 it seems equally easy), but I wonder if this "path based" approach is slightly harder for Samba, since for most cases Samba maps to "handle based" equivalents (when the same operation can come in eitherpath based or
handle based - Samba mostly maps to handle based).
Could you please elaborate what you mean by "path based" for xattrs? They can be gotten with sys_fgetxattr() from a file handle just as easily as from the filename.
I definitely agree with having a common attribute name for these.
As for being able to write to the "create time" attribute, I would prefer that this be a filesystem mount option. For some users (myself included) I don't care whether Windows is unhappy that it can't update this creation time - I'd prefer to know when a file is actually created.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 19 lines · Andreas Dilger
Quoted 10 lines · Steve French
Quoted 7 lines · Andreas Dilger
I had proposed in the past that these file attributes be exposed to userspace as virtual xattrs, e.g. user.cr_time returning a struct timespec, but the data is stored internal to the filesystem in whatever format is most efficient for it. We only strictly need user.crtime for this, but I wouldn't object to exporting other inode attributes in this way (e.g. user.mtime, user.atime, etc).
Quoted 7 lines
Yes - the fake xattr approach was the approach that I preferred (since I could do this trivially to cifs, and for ext4 it seems equally easy),
but I wonder if this "path based" approach is slightly harder for Samba, since for most cases Samba maps to "handle based" equivalents (when the same operation can come in eitherpath based or handle based - Samba mostly maps to handle based).
Quoted 3 lines
Could you please elaborate what you mean by "path based" for xattrs? They can be gotten with
sys_fgetxattr()from a file handle just as easily as from the filename.
Although handle based interfaces are there for xattrs in userspace, internally the xattr calls are path based, so handle based calls get converted to path based in kernel and you could lose some of the performance benefit - path based could be slightly slower (due to path revalidation, and access checks) than handle based calls.
int (*setxattr) (struct dentry *, const char *,const void *,size_t,int);
ssize_t (*getxattr) (struct dentry *, const char *, void *, size_t);
ssize_t (*listxattr) (struct dentry *, char *, size_t);
int (*removexattr) (struct dentry *, const char *);Quoted 7 lines
I definitely agree with having a common attribute name for these.
As for being able to write to the "create time" attribute, I would prefer that this be a filesystem mount option. For some users (myself included) I don't care whether Windows is unhappy that it can't update this creation time - I'd prefer to know when a file is actually created.
I agree - if create time could be overwritten - that behavior hould be configurable (another post mentioned the alternative to mount option - a flag for this perhaps along the lines of the a "backup intent" flag - although somewhat different than what Windows uses that for).
Signature · 7 lines
Thanks, Steve -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 23 lines · Steve French
Quoted 9 lines · Andreas Dilger
Quoted 7 lines · Steve French
Yes - the fake xattr approach was the approach that I preferred (since I could do this trivially to cifs, and for ext4 it seems equally easy),
but I wonder if this "path based" approach is slightly harder for Samba, since for most cases Samba maps to "handle based" equivalents (when the same operation can come in eitherpath based or handle based - Samba mostly maps to handle based).Quoted 3 lines
Could you please elaborate what you mean by "path based" for xattrs? They can be gotten with
sys_fgetxattr()from a file handle just as easily as from the filename.Although handle based interfaces are there for xattrs in userspace, internally the xattr calls are path based, so handle based calls get converted to path based in kernel and you could lose some of the performance benefit - path based could be slightly slower (due to path revalidation, and access checks) than handle based calls.
int (*setxattr) (struct dentry *, const char *,const void
*,size_t,int);
ssize_t(*getxattr) (struct dentry *, const char *, void *,
Quoted 3 lines
size_t);
ssize_t (*listxattr) (struct dentry *, char *, size_t);
int (*removexattr) (struct dentry *, const char *);I'm not sure I understand your point. sys_fgetxattr() maps directly to vfs_getxattr(), and while it still does permission checks, that doesn't have anything to do with pathname revalidation AFAICS.
There are an equal number of permission checks in sys_fstat->vfs_getattr() as in sys_fgetxattr->vfs_getxattr().
Quoted 9 lines
Quoted 4 lines
As for being able to write to the "create time" attribute, I would prefer that this be a filesystem mount option. For some users (myself included) I don't care whether Windows is unhappy that it can't update this creation time - I'd prefer to know when a file is actually created.
I agree - if create time could be overwritten - that behavior hould be configurable (another post mentioned the alternative to mount option - a flag for this perhaps along the lines of the a "backup intent" flag - although somewhat different than what Windows uses that for).
If this is a flag that a user can configure/select themselves, then it is completely useless to me. If it is a mount option and/or possibly an additional process capability that would be more useful.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 10 lines · Andreas Dilger
Quoted 8 lines · Steve French
some of the performance benefit - path based could be slightly slower (due to path revalidation, and access checks) than handle based calls.
int (*setxattr) (struct dentry *, const char *,const void *,size_t,int);
ssize_t(*getxattr) (struct dentry *, const char *, void *,size_t);ssize_t(*listxattr) (struct dentry *, char *,size_t); int (*removexattr) (struct dentry *, const char *);
Quoted 6 lines
I'm not sure I understand your point. sys_fgetxattr() maps directly to
vfs_getxattr(), and while it still does permission checks, that doesn't have anything to do with pathname revalidation AFAICS.There are an equal number of permission checks in
sys_fstat->vfs_getattr()as insys_fgetxattr->vfs_getxattr().
Good point. Looking more carefully I think you are right - the path revalidation
is skipped in fgetxattr so it probably isn't much slower if at all (than a handle based ioctl call would be).
Quoted 8 lines
Quoted 5 lines
Quoted 5 lines
As for being able to write to the "create time" attribute, I would prefer that this be a filesystem mount option. For some users (myself included) I don't care whether Windows is unhappy that it can't update this creation
time - I'd prefer to know when a file is actually created.If this is a flag that a user can configure/select themselves, then it is completely useless to me. If it is a mount option and/or possibly an additional process capability that would be more useful.
Yes ... there is a precedent for a process capability for something similar (e.g. a Windows user must have both the "backup privilege" in his process capability and specify a flag indicating he wants to
use "backup intent" for certain operations). There are already
33 capabilities in include/linux/capability.h, some as narrow as this.
Signature · 7 lines
Thanks, Steve -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Andreas Dilger
Quoted 6 lines · Steve French
Quoted 4 lines · Andreas Dilger
As for being able to write to the "create time" attribute, I would prefer that this be a filesystem mount option. For some users (myself included) I don't care whether Windows is unhappy that it can't update this creation time - I'd prefer to know when a file is actually created.
Quoted 8 lines
Quoted 4 lines
I agree - if create time could be overwritten - that behavior hould be configurable (another post mentioned the alternative to mount option - a flag for this perhaps along the lines of the a "backup intent" flag - although somewhat different than what Windows uses that for).
If this is a flag that a user can configure/select themselves, then it is completely useless to me. If it is a mount option and/or possibly an additional process capability that would be more useful.
Restricting the modification of create time is pointless.
A number of filesystems and operating systems besides windows have create time and it MUST be settable on restore just like the access and modification times.
There is no value in it needing "special privileges". What you are forgetting is *NO* timestamp on a filesystem can be trusted to be correct. All times are just fields set based on what the system thinks the time is now (not to mention that any disk editor can change the field). The "now time" can be intentionally or accidentally set wrong, or as has happened to me, a totally crap value from a bad RTC battery. :)
I did legal records storage a long time ago, and I can guarantee you disk timestamps are meaningless.
On the other hand, if you just want stamps for debugging and internal purposes, then you do what I did, put versions and times inside on-disk metadata structures the filesystem does not expose. As long as you remember all stamps are still at the mercy of those who can patch your clock, your code, and your raw disk blocks.
jim
Quoted 12 lines · jim owens
Quoted 11 lines · Andreas Dilger
Quoted 10 lines · Steve French
Quoted 4 lines · Andreas Dilger
As for being able to write to the "create time" attribute, I would prefer that this be a filesystem mount option. For some users (myself included) I don't care whether Windows is unhappy that it can't update this creation time - I'd prefer to know when a file is actually created.
I agree - if create time could be overwritten - that behavior hould be configurable (another post mentioned the alternative to mount option - a flag for this perhaps along the lines of the a "backup intent" flag - although somewhat different than what Windows uses that for).
Quoted 5 lines
Quoted 3 lines
If this is a flag that a user can configure/select themselves, then it is completely useless to me. If it is a mount option and/or possibly an additional process capability that would be more useful.
Restricting the modification of create time is pointless.
You contradict yourself later. What we have NOW is creation timestamps inside on-disk metadata structures the filesystem does not expose. Going a step further - to expose this in a read-only manner does not remove the usefulness of this field, but making it read-write does.
For those users/admins that want to be able to change this (due to Samba, or whatever), great, we can give them this ability, but for users who do NOT want regular users to be able to change the file creation timestamp, that should be possible as well.
Quoted 3 lines
A number of filesystems and operating systems besides windows have create time and it MUST be settable on restore just like the access and modification times.
It seems those other systems can't choke _too_ badly, given that they don't have creation timestamps today and seem to work fine. Since no Unix tools or applications depend on the presence of this field, I would say that the default be to NOT allow this timestamp to be changed.
Quoted 8 lines
There is no value in it needing "special privileges". What you are forgetting is *NO* timestamp on a filesystem can be trusted to be correct. All times are just fields set based on what the system thinks the time is now (not to mention that any disk editor can change the field). The "now time" can be intentionally or accidentally set wrong, or as has happened to me, a totally crap value from a bad RTC battery. :)
Quoted 6 lines
On the other hand, if you just want stamps for debugging and internal purposes, then you do what I did, put versions and times inside on-disk metadata structures the filesystem does not expose. As long as you remember all stamps are still at the mercy of those who can patch your clock, your code, and your raw disk blocks.
Sure, if you give every user write permission to the disk, then every bit of security is meaningless, but I don't expect that many sysadmins will do this. For a single-user system, they can do whatever they want in the end, so preventing regular users from changing the timestamp is no hindrance to them in the end either.
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 4 lines · Andreas Dilger
For those users/admins that want to be able to change this (due to Samba, or whatever), great, we can give them this ability, but for users who do NOT want regular users to be able to change the file creation timestamp, that should be possible as well.
So, are we ready for Mingming or one of the ext4 developers to propose a patch for this via xattrs (I can do a similar one for cifs). Sounds like various have said:
1) xattrs instead of ioctl
2) get of create time allowed by default, but set of create time limited
But for the second part of this - how should we set nanosecond timestamps for the others (or in Samba's case, rounded to 100 nanoseconds for DCE time) ... a millisecond granularity for utimes is a very long time these days.
Signature · 3 lines
Thanks, Steve
Quoted 11 lines · Steve French
So, are we ready for Mingming or one of the ext4 developers to propose a patch for this via xattrs (I can do a similar one for cifs). Sounds like various have said:
1) xattrs instead of ioctl
2) get of create time allowed by default, but set of create time limitedBut for the second part of this - how should we set nanosecond timestamps for the others (or in Samba's case, rounded to 100 nanoseconds for DCE time) ... a millisecond granularity for utimes is a very long time these days.
What's wrong with stat(2)? AFAIK, it supports nanoseconds.
Quoted 9 lines · Sunil Mushran
Quoted 8 lines · Steve French
So, are we ready for Mingming or one of the ext4 developers to propose a patch for this via xattrs (I can do a similar one for cifs). Sounds like various have said:
1) xattrs instead of ioctl
2) get of create time allowed by default, but set of create time limited
Quoted 6 lines
Quoted 4 lines
But for the second part of this - how should we set nanosecond timestamps for the others (or in Samba's case, rounded to 100 nanoseconds for DCE time) ... a millisecond granularity for utimes is a very long time these days.
What's wrong with stat(2)? AFAIK, it supports nanoseconds.
An earlier post from Mingming IIRC noted that utimes does not support setting nanosecond timestamps (e.g. write time).
"The function utime() allows specification of time stamps with a resolution of 1 second. ... The function utimes() is similar, but the times argument allows a reso lution of 1 microsecond for the
timestamps. " 1 millisecond is not sufficient, and as a result Samba
has to store duplicate time stamps (in Samba specific xattrs) which we would like to avoid wherever possible
Signature · 3 lines
Thanks, Steve
Quoted 18 lines · Steve French
Quoted 5 lines · Sunil Mushran
Quoted 4 lines · Steve French
So, are we ready for Mingming or one of the ext4 developers to propose a patch for this via xattrs (I can do a similar one for cifs). Sounds like various have said:
Quoted 8 lines
Quoted 7 lines
1) xattrs instead of ioctl
2) get of create time allowed by default, but set of create time limitedBut for the second part of this - how should we set nanosecond timestamps for the others (or in Samba's case, rounded to 100 nanoseconds for DCE time) ... a millisecond granularity for utimes is a very long time these days.
What's wrong with stat(2)? AFAIK, it supports nanoseconds.
An earlier post from Mingming IIRC noted that utimes does not support setting nanosecond timestamps (e.g. write time).
Quoted 6 lines
"The function
utime()allows specification of time stamps with a resolution of 1 second. ... The functionutimes()is similar, but the times argument allows a reso lution of 1 microsecond for the
timestamps. " 1 millisecond is not sufficient, and as a result Samba
has to store duplicate time stamps (in Samba specific xattrs) which we would like to avoid wherever possible
If you are trying to update it, you can use utimensat(2).
Quoted 24 lines · Sunil Mushran
Quoted 23 lines · Steve French
Quoted 10 lines · Sunil Mushran
Quoted 8 lines · Steve French
So, are we ready for Mingming or one of the ext4 developers to propose a patch for this via xattrs (I can do a similar one for cifs). Sounds like various have said:
1) xattrs instead of ioctl
2) get of create time allowed by default, but set of create time limitedQuoted 6 lines
Quoted 4 lines
But for the second part of this - how should we set nanosecond timestamps for the others (or in Samba's case, rounded to 100 nanoseconds for DCE time) ... a millisecond granularity for utimes is a very long time these days.
What's wrong with stat(2)? AFAIK, it supports nanoseconds.
An earlier post from Mingming IIRC noted that utimes does not support setting nanosecond timestamps (e.g. write time).
Quoted 8 lines
Quoted 6 lines
"The function
utime()allows specification of time stamps with a resolution of 1 second. ... The functionutimes()is similar, but the times argument allows a reso lution of 1 microsecond for the timestamps. " 1 millisecond is not sufficient, and as a result Samba has to store duplicate time stamps (in Samba specific xattrs) which we would like to avoid wherever possibleIf you are trying to update it, you can use utimensat(2).
Looks like that should work based on what I see in fs/utimes.c ... but I have heard from two that think it doesn't work, so time to test it ... (unfortunately
the libc is not new enough on this system ... ugh).
Signature · 3 lines
Thanks, Steve
Quoted 10 lines · Steve French
Quoted 2 lines · Sunil Mushran
If you are trying to update it, you can use utimensat(2).
Looks like that should work based on what I see in
fs/utimes.c... but I have heard from two that think it doesn't work, so time to test it ... (unfortunately
the libc is not new enough on this system ... ugh).
I just ran the following on my box. (2.6.28-15-generic #52-Ubuntu SMP) This is on ocfs2.
$ stat DSCN1417.jpg File: `DSCN1417.jpg' Size: 60991 Blocks: 120 IO Block: 4096 regular file Device: 807h/2055d Inode: 9031712 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( xxxxx/xxxxxxx) Gid: ( xxxx/xxxxxxx) Access: 2009-09-10 10:49:08.508507794 -0700 Modify: 2006-07-05 18:27:05.000000000 -0700 Change: 2007-05-29 15:23:16.000058283 -0700
$ touch DSCN1417.jpg
$ stat DSCN1417.jpg File: `DSCN1417.jpg' Size: 60991 Blocks: 120 IO Block: 4096 regular file Device: 807h/2055d Inode: 9031712 Links: 1 Access: (0664/-rw-rw-r--) Uid: ( xxxx/xxxxxxx) Gid: ( xxxx/xxxxxx) Access: 2009-10-20 14:41:51.224405489 -0700 Modify: 2009-10-20 14:41:51.224405489 -0700 Change: 2009-10-20 14:41:51.224405489 -0700
Quoted 6 lines · Sunil Mushran
Quoted 5 lines · Steve French
Quoted 1 line · Sunil Mushran
Quoted 7 lines
Quoted 7 lines
Quoted 2 lines
If you are trying to update it, you can use utimensat(2).
Looks like that should work based on what I see in
fs/utimes.c... but I have heard from two that think it doesn't work, so time to test it ... (unfortunately
the libc is not new enough on this system ... ugh).
strace shows that touch on my ubuntu system still calls utimesat
(touch --version displays v6.10 2008). What version of touch are you
running?
Signature · 3 lines
Thanks, Steve
Quoted 7 lines · Steve French
Quoted 6 lines · Sunil Mushran
Quoted 5 lines · Steve French
Quoted 2 lines · Sunil Mushran
If you are trying to update it, you can use utimensat(2).
Quoted 8 lines
Quoted 4 lines
Quoted 4 lines
Looks like that should work based on what I see in
fs/utimes.c... but I have heard from two that think it doesn't work, so time to test it ... (unfortunately
the libc is not new enough on this system ... ugh).strace shows that touch on my ubuntu system still calls utimesat
(touch --version displays v6.10 2008). What version of touch are you
running?
$ touch --version touch (GNU coreutils) 6.10
I am running the same and it calls utimensat(). Time to look at the source.
Quoted 13 lines · Sunil Mushran
Quoted 12 lines · Steve French
Quoted 10 lines · Sunil Mushran
Quoted 9 lines · Steve French
Quoted 2 lines · Sunil Mushran
If you are trying to update it, you can use utimensat(2).
Looks like that should work based on what I see in
fs/utimes.c... but I have heard from two that think it doesn't work, so time to test it ... (unfortunately
the libc is not new enough on this system ... ugh).
Quoted 8 lines
Quoted 3 lines
strace shows that touch on my ubuntu system still calls utimesat
(touch --version displays v6.10 2008). What version of touch are you
running?$ touch --version touch (GNU coreutils) 6.10I am running the same and it calls
utimensat(). Time to look at the source.
Just to confirm that I am running the same version of touch too, version 6.10, looks like it calls utimensat() and sets the nanosecond timestamps.
Mingming
Quoted 4 lines
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
tis 2009-10-20 klockan 16:11 -0500 skrev Steve French:
Quoted 4 lines
"The function
utime()allows specification of time stamps with a resolution of 1 second. ... The functionutimes()is similar, but the times argument allows a reso lution of 1 microsecond for the
timestamps. " 1 millisecond is not sufficient
But it isn't 1 millisecond, it's granulatity is 1 microsecond (1/1000 millisecond, or 1000 nanoseconds)
utime -> second (time_t), same granularity as old stat(), time() and friends.
utimes -> microsecond (struct timeval) (1/1000000), same granularity as gettimeofday() and friends.
utimen -> nanosecond (struct timespec) (1/1000000000), same granularity as current stat(), clock_gettime() and friends.
Regards
Henrik
Quoted 2 lines · Henrik Nordstrom
utimen -> nanosecond (struct timespec) (1/1000000000), same granularity as current
stat(),clock_gettime()and friends.
Yes. I have opened bug 6836 in samba bugzilla to track having smbd
use utimensat (but not sure whether they will be able to detect when the file system can support nanosecond timestamps through - even with use of utimensat - so there may be questions about how to do runtime detection of whether the filesystem is one like ext4, xfs, jfs etc. that can store such timestamps, if not they probably still have to store 100 nanosecond "DCE time" granulariy timestamps in xattrs)
The getting (and optionally setting) create time (via xattr) is still an open question, but am hopeful that that won't be hard to add for ext4.
Signature · 7 lines
Thanks, Steve -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 7 lines · Steve French
Yes. I have opened bug 6836 in samba bugzilla to track having smbd
use utimensat (but not sure whether they will be able to detect when the file system can support nanosecond timestamps through - even with use of utimensat - so there may be questions about how to do runtime detection of whether the filesystem is one like ext4, xfs, jfs etc. that can store such timestamps, if not they probably still have to store 100 nanosecond "DCE time" granulariy timestamps in xattrs)
I think the right way to do it would be to implement a system call for pathconf/fpathconf and add a selector for timestamp granularity. This obviously isn't an instant fix since it's not there already, but I think it's the right long-term interface. We already have the functions available in libc, but currently they don't call into the kernel for anything. If there was a system call and an operation somewhere in the VFS code to call into the individual drivers, then we could export this as well as lots of other useful information. It would also be based on a POSIX standard interface.
Brad Boyer
[email protected]
On 2009-10-21 at 10:36 -0500 Steve French sent off:
Quoted 5 lines
Quoted 2 lines · Henrik Nordstrom
utimen -> nanosecond (struct timespec) (1/1000000000), same granularity as current
stat(),clock_gettime()and friends.
Quoted 7 lines
Yes. I have opened bug 6836 in samba bugzilla to track having smbd
use utimensat (but not sure whether they will be able to detect when the file system can support nanosecond timestamps through - even with use of utimensat - so there may be questions about how to do runtime detection of whether the filesystem is one like ext4, xfs, jfs etc. that can store such timestamps, if not they probably still have to store 100 nanosecond "DCE time" granulariy timestamps in xattrs)
up to these days Samba just supports the granularity the undelying filesystem supports and I think there is no need to change this. The utimensat documentation says: "The file's relevant timestamp shall be set to the greatest value supported by the file system that is not greater than the specified time." So just using this call (where available) is all Samba will have to do - I see nothing more to care about.
Unfortunately the designers of utimensat() didn't think of setting of birth times on systems that have it. It would have been nice if this brand new call had covered this, too. The FreeBSD folks thought about adding a system call to set birth time. All they currently have is a rather hackish[1] way to modify it timewise in one direction.
Cheers
Björn
[1] http://fuse4bsd.creo.hu/localcgi/man-cgi.cgi?utimes+2
... For file systems that support file birth (creation) times (such as UFS2), the birth time will be set to the value of the second element if the second element is older than the currently set birth time.
Quoted 14 lines · Björn Jacke
On 2009-10-21 at 10:36 -0500 Steve French sent off:
Quoted 5 lines
Quoted 2 lines · Henrik Nordstrom
utimen -> nanosecond (struct timespec) (1/1000000000), same granularity as current
stat(),clock_gettime()and friends.Quoted 7 lines
Yes. I have opened bug 6836 in samba bugzilla to track having smbd use utimensat (but not sure whether they will be able to detect when the file system can support nanosecond timestamps through - even with use of utimensat - so there may be questions about how to do runtime detection of whether the filesystem is one like ext4, xfs, jfs etc. that can store such timestamps, if not they probably still have to store 100 nanosecond "DCE time" granulariy timestamps in xattrs)
Quoted 6 lines
up to these days Samba just supports the granularity the undelying filesystem supports and I think there is no need to change this. The utimensat documentation says: "The file's relevant timestamp shall be set to the greatest value supported by the file system that is not greater than the specified time." So just using this call (where available) is all Samba will have to do - I see nothing more to care about.
Time granularity of 1 second on ext3 is so atrociously bad, and ext3 is so common,
that I see obvious reason why Samba server would want to store 100 nanosecond time stamps (in xattrs) for this very common case.
It would be very confusing to some Windows apps to have time on files seem to move backwards (due to rounding to 1 second granularity) - to set the time to something and have the time when next retrieved (on average) a 1/2 second earlier than what they just set.
Signature · 7 lines
Thanks, Steve -- To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 6 lines · Steve French
Quoted 4 lines · Andreas Dilger
For those users/admins that want to be able to change this (due to Samba, or whatever), great, we can give them this ability, but for users who do NOT want regular users to be able to change the file creation timestamp, that should be possible as well.
Quoted 6 lines
So, are we ready for Mingming or one of the ext4 developers to propose a patch for this via xattrs (I can do a similar one for cifs). Sounds like various have said:
1) xattrs instead of ioctl
2) get of create time allowed by default, but set of create time limited
A wildly untested/uncompiled patch is attached for purposes of discussion. One issue that I've already come across is if we are storing a raw "timespec" we will get different xattrs on disk in the fallback case where the on-disk inodes are not large enough to store the crtime inside the inode.
This version of the patch will fall back to storing the timespec on disk in the supplied size/endianness as an xattr, but it definitely needs to be handled more appropriately (endian/word size). It _would_ be nice to store a full 64-bit timespec on disk, but given that the in-inode format only can handle 34 bits of seconds + 30 bits of nanoseconds, it isn't clear whether the "fallback" should do better than that.
The main question is whether storing a "user.crtime" as an on-disk xattr is a desirable fallback, or if this should just fail? Similarly, should the lack of on-disk crtime return "0.0" as the creation time, or should it return -ENODATA or similar?
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
fs/ext4/ext4.h renamed from fs/ext4/ext4.h.orig +1 -0
@@ -560,6 +699,7 @@ do { \#define EXT4_MOUNT_DELALLOC 0x8000000/* Delalloc support */ #define EXT4_MOUNT_DATA_ERR_ABORT 0x10000000/* Abort on file data write*/ #define EXT4_MOUNT_BLOCK_VALIDITY 0x20000000/* Block validity checking */ +#define EXT4_MOUNT_USER_CRTIME 0x40000000/* Users can set crtime */ #define clear_opt (o, opt) o &= ~EXT4_MOUNT_##opt#define set_opt (o, opt) o |= EXT4_MOUNT_##opt
fs/ext4/xattr_user.c renamed from fs/ext4/xattr_user.c.orig +39 -0
@@ -36,8 +36,25 @@ ext4_xattr_user_get(struct inode *inode, {if (strcmp (name,"" ) ==0 )return -EINVAL ; + +if (strcmp (name,"crtime" ) ==0 && +EXT4_FITS_IN_INODE (EXT4_I (inode), inode, i_crtime)) { +struct timespec *crtime = &EXT4_I (inode)->i_crtime; + +if (size ==0 ) +return sizeof (*crtime); + +if (size <sizeof (*crtime)) +return -ERANGE ; + +memcpy (buffer, crtime,sizeof (*crtime)); + +return sizeof (*crtime); + } +if (!test_opt (inode->i_sb,XATTR_USER ))return -EOPNOTSUPP ; +return ext4_xattr_get (inode,EXT4_XATTR_INDEX_USER , name, buffer, size); }
@@ -47,8 +64,30 @@ ext4_xattr_user_set(struct inode *inode, {if (strcmp (name,"" ) ==0 )return -EINVAL ; + +/* Prevent regular users from modifying the file creation time */ +if (strcmp (name,"crtime" ) ==0 && + !(capable (CAP_DAC_OVERRIDE ) ||test_opt (inode->i_sb,USER_CRTIME ))) +return -EACCES ; + +/* If there is room in the large inode, store the create time there, +* otherwise store it as a regular extended attribute. */ +if (strcmp (name,"crtime" ) ==0 && +EXT4_FITS_IN_INODE (EXT4_I (inode), inode, i_crtime)) { +struct timespec *crtime = &EXT4_I (inode)->i_crtime; + +if (size <sizeof (*crtime)) +return -EINVAL ; + +memcpy (crtime, value,sizeof (*crtime)); +ext4_dirty_inode (inode); + +return sizeof (*crtime); + } +if (!test_opt (inode->i_sb,XATTR_USER ))return -EOPNOTSUPP ; +return ext4_xattr_set (inode,EXT4_XATTR_INDEX_USER , name, value, size, flags); }
fs/ext4/super.c renamed from fs/ext4/super.c.orig +9 -0
@@ -1023,6 +1049,7 @@ enum { Opt_stripe, Opt_delalloc, Opt_nodelalloc, Opt_block_validity, Opt_noblock_validity, Opt_inode_readahead_blks, Opt_journal_ioprio, + Opt_user_crtime, Opt_nouser_crtime }; static const match_table_t tokens = {
@@ -1088,6 +1116,8 @@ static const match_table_t tokens = { {Opt_auto_da_alloc, "auto_da_alloc=%u"}, {Opt_auto_da_alloc, "auto_da_alloc"}, {Opt_noauto_da_alloc, "noauto_da_alloc"}, + {Opt_user_crtime, "user_crtime" }, + {Opt_nouser_crtime, "nouser_crtime" }, {Opt_err, NULL}, };
@@ -1552,6 +1554,12 @@ set_qf_format:else set_opt (sbi->s_mount_opt,NO_AUTO_DA_ALLOC );break ; +case Opt_user_crtime: +set_opt (sbi->s_mount_opt,USER_CRTIME ); +break ; +case Opt_nouser_crtime: +clear_opt (sbi->s_mount_opt,USER_CRTIME ); +break ;default :ext4_msg (sb,KERN_ERR , "Unrecognized mount option \"%s\" "
Quoted 25 lines · Andreas Dilger
Quoted 7 lines · Steve French
Quoted 4 lines · Andreas Dilger
For those users/admins that want to be able to change this (due to Samba, or whatever), great, we can give them this ability, but for users who do NOT want regular users to be able to change the file creation timestamp, that should be possible as well.
Quoted 7 lines
So, are we ready for Mingming or one of the ext4 developers to propose a patch for this via xattrs (I can do a similar one for cifs). Sounds like various have said:
1) xattrs instead of ioctl
2) get of create time allowed by default, but set of create time limitedA wildly untested/uncompiled patch is attached for purposes of discussion.
One issue that I've already come across is if we are storing a raw "timespec"
we will get different xattrs on disk in the fallback case where the on- disk
inodes are not large enough to store the crtime inside the inode.
Quoted 11 lines
This version of the patch will fall back to storing the timespec on disk in the supplied size/endianness as an xattr, but it definitely needs to be
handled more appropriately (endian/word size). It _would_ be nice to store
a full 64-bit timespec on disk, but given that the in-inode format only can
handle 34 bits of seconds + 30 bits of nanoseconds, it isn't clear whether
the "fallback" should do better than that.
Could we transparently convert user provided raw creation time and store the same format as whose in-inode creation timestamp in the fallback case? Just to be consistent. User doesn't have to know what type/size of ext4 inode it is accessing...
Quoted 2 lines
The main question is whether storing a "user.crtime" as an on-disk xattr is a desirable fallback, or if this should just fail?
I think it would be nice to have this fallback, just be consistent, that for all type of ext4 inodes there is a way to set file creation time for allowed users(whether default 256 bytes large inode which could store the 34+30bit creation time, or the 128 bytes smaller inode, which doesn't have space to store the creation in the inode)
Regards,
Mingming
Quoted 8 lines
Similarly, should
the lack of on-disk crtime return "0.0" as the creation time, or should it return-ENODATAor similar?
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.
Quoted 7 lines · Andreas Dilger
Quoted 1 line · jim owens
Restricting the modification of create time is pointless.
You contradict yourself later. What we have NOW is creation timestamps inside on-disk metadata structures the filesystem does not expose. Going a step further - to expose this in a read-only manner does not remove the usefulness of this field, but making it read-write does.
The apparent contradiction is that I forgot to explicitly say there are two kinds of create time stamps in filesystems:
- "user create time"
- "internal filesystem create time"
My assumption was we were *adding* a "user create time" which must be settable by things like star that archive and restore.
Otherwise we have "modification time" before "user create time" when a user restores their files.
An "internal filesystem create time" that is not exposed or is read-only is different. And AFAIK most linux filesystems do not have either create time today so unless we want to say Samba only uses ext4, we need to think broader in scope.
Quoted 3 lines
Since no Unix tools or applications depend on the presence of this field, I would say that the default be to NOT allow this timestamp to be changed.
If nobody uses the field, what is the need to protect it? Seems you are now contradicting yourself.
jim
Quoted 19 lines · Andreas Dilger
Quoted 10 lines · Steve French
A new version of the format for Samba server related file system extended attributes is expected in the next version of Samba so the topic of create time and nanosecond timestamps for Linux has come up again. Since some file systems don't support storing create time (birth time), and none support updating create time, or for that matter for storing any nanosecond timestamps at all (a millisecond seems like a much longer time today than when the stat structure was defined), and dos attributes, Samba server stores these in extended attributes, which is awkward on those file systems which store (different) versions of these on disk.
I had proposed in the past that these file attributes be exposed to userspace as virtual xattrs, e.g. user.cr_time returning a struct timespec, but the data is stored internal to the filesystem in whatever format is most efficient for it. We only strictly need user.crtime for this, but I wouldn't object to exporting other inode attributes in this way (e.g. user.mtime, user.atime, etc).
Quoted 8 lines
I believe that nanosecond timestamps ARE supported with newer filesystems
usingstat(). There is theutimensat()system call, which takes a timespec
parameter, so it looks to be suitable for setting nanosecond timestamps from userspace. I don't know ifglibc()supports this, but I do see it in the kernel.
How we are going to set the nanotimestamps today? utimes(2) man page seems indicating it only support setting time at microseond level.
Quoted 3 lines
As for _updating_ create time, isn't that sort of defeating the purpose of create time?
Perphas the backup/restore wants to preserve the original creation timestamp after the file being backuped?
Quoted 10 lines
Cheers, Andreas
--
Andreas Dilger
Sr. Staff Engineer, Lustre Group
Sun Microsystems of Canada, Inc.--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html