FAT, NTFS, CIFS and DOS attributes
Hello all,
I recently posted to LKML a patch to get or set DOS attribute flags for fatfs. That patch used ioctl(). It was suggested that a better way would be using xattrs, although the xattr mechanism seems clumsy to me, and has namespace issues.
I also think it would be good to have a unified interface for FAT, NTFS and CIFS for these attributes.
I noticed that CIFS has a placeholder "user.DosAttrib" in cifs/xattr.c, although it doesn't seem to be implemented.
Questions:
a) is xattr the right thing? It seems to be a fairly complex and ill-thought-out mechanism all along, especially the whole namespace business (what is a system attribute to one filesystem is a user attribute to another, for example.)
b) if xattr is the right thing, shouldn't this be in the system namespace rather than the user namespace?
c) What should the representation be? Binary byte? String containing a subset of "rhsvda67" (barf)?
-hpa
Quoted 9 lines · H. Peter Anvin
I noticed that CIFS has a placeholder "user.DosAttrib" in
cifs/xattr.c, although it doesn't seem to be implemented.Questions:
a) is xattr the right thing? It seems to be a fairly complex and ill-thought-out mechanism all along, especially the whole namespace business (what is a system attribute to one filesystem is a user attribute to another, for example.)
Ahh, just go with xattrs Pete :-> I don't see the namespace issue to be a big deal. The interface does seem a *little* overdesigned. It would have been adequate to just use the dev:ino pair from stat(2) and dump namespaces altogether since the real performance critical apps will have stat'd the living daylights out of the path trying to canonicalize the case so the last thing you want to do is a path lookup.
Quoted 2 lines
b) if xattr is the right thing, shouldn't this be in the system namespace rather than the user namespace?
If we're just thinking about MS-oriented discretionary access control then I think the owner of the file is basically king and should be the only normal user to that can read and write it's xattrs. So whatever namespace that is (not system).
Quoted 2 lines
c) What should the representation be? Binary byte? String containing a subset of "rhsvda67" (barf)?
Definitely binary.
Mike
Quoted 9 lines · Michael B Allen
Quoted 3 lines
b) if xattr is the right thing, shouldn't this be in the system namespace rather than the user namespace?
If we're just thinking about MS-oriented discretionary access control then I think the owner of the file is basically king and should be the only normal user to that can read and write it's xattrs. So whatever namespace that is (not system).
system namespace means that it's a name defined by the kernel as opposed to a name defined by the user. One of the most glaring design errors in this whole thing, in my opinion, but if we're going to use xattrs we probably should stick with it.
Thus, I'd propose:
system.dosattrib - DOS attributes (single byte)
system.dosshortname - DOS short name (e.g. for VFAT)
-hpa
Quoted 18 lines · H. Peter Anvin
Quoted 11 lines · Michael B Allen
Quoted 3 lines
b) if xattr is the right thing, shouldn't this be in the system namespace rather than the user namespace?
If we're just thinking about MS-oriented discretionary access control then
I think the owner of the file is basically king and should be the only normal user to that can read and write it's xattrs. So whatever namespace
that is (not system).system namespace means that it's a name defined by the kernel as opposed to a name defined by the user. One of the most glaring design errors in this whole thing, in my opinion, but if we're going to use xattrs we probably should stick with it.
Quoted 4 lines
Thus, I'd propose:
system.dosattrib - DOS attributes (single byte)
system.dosshortname - DOS short name (e.g. for VFAT)
Oh, from the man page I thought 'system' attributes were "extended attributes to which ordinary processes should not have access". Is that not true? If it is true, how would an ordinary user change these attrs?
Mike
Quoted 11 lines · Michael B Allen
Quoted 5 lines
Thus, I'd propose:
system.dosattrib - DOS attributes (single byte)
system.dosshortname - DOS short name (e.g. for VFAT)Oh, from the man page I thought 'system' attributes were "extended attributes to which ordinary processes should not have access". Is that not true? If it is true, how would an ordinary user change these attrs?
Nah:
Extended system attributes
Extended system attributes are used by the kernel to store sys- tem objects such as Access Control Lists and Capabilities. Read and write access permissions to system attributes depend on the policy implemented for each system attribute implemented by filesystems in the kernel.
It's not *that* broken...
-hpa
Quoted 2 lines
system.dosattrib - DOS attributes (single byte)
system.dosshortname - DOS short name (e.g. for VFAT)
you need more than one byte for DOS attrib. These are the bits Samba4 defines:
/* FileAttributes (search attributes) field */
#define FILE_ATTRIBUTE_READONLY 0x0001
#define FILE_ATTRIBUTE_HIDDEN 0x0002
#define FILE_ATTRIBUTE_SYSTEM 0x0004
#define FILE_ATTRIBUTE_VOLUME 0x0008
#define FILE_ATTRIBUTE_DIRECTORY 0x0010
#define FILE_ATTRIBUTE_ARCHIVE 0x0020
#define FILE_ATTRIBUTE_DEVICE 0x0040
#define FILE_ATTRIBUTE_NORMAL 0x0080
#define FILE_ATTRIBUTE_TEMPORARY 0x0100
#define FILE_ATTRIBUTE_SPARSE 0x0200
#define FILE_ATTRIBUTE_REPARSE_POINT 0x0400
#define FILE_ATTRIBUTE_COMPRESSED 0x0800
#define FILE_ATTRIBUTE_OFFLINE 0x1000
#define FILE_ATTRIBUTE_NONINDEXED 0x2000
#define FILE_ATTRIBUTE_ENCRYPTED 0x4000
while most apps don't care about the bits beyond 0xFF at the moment, I think that might change, especially for win32 clients accessing linux filesystems via wine and Samba.
Also, there are many other bits of windows meta-data that matter for apps that care about dos attributes, including the extra 1.5 time fields (windows has 4 settable time fields, posix has 2 settable and 3 readable time fields), the 8.3 name, the allocation size and all the DOS EAs (important for OS/2 clients).
We use the following xattrs in Samba4:
user.DosAttrib : structure holding basic non-privileged attribute information user.DosEAs : all the DOS (OS/2) style EAs user.DosStreams : list of alternate data steams, flagged as internal or external
user.DosStream.name: the stream data itself for internal streams
security.NTACL : the NT ACL
the rationale for making most of them in the user namespace is that it is 'mostly harmless' to allow the owner of the file to change those ones. The NTACL needs to be in the security namespace as it contains elements that the user must not be able to control without system permission (everyone can read it, but only root can write).
see xattr.idl for the full definitions of the above.
Cheers, Tridge
Quoted 6 lines · [email protected] wrote:
We use the following xattrs in Samba4:
user.DosAttrib : structure holding basic non-privileged attribute information user.DosEAs : all the DOS (OS/2) style EAs user.DosStreams : list of alternate data steams, flagged as internal or external
Quoted 7 lines
user.DosStream.name: the stream data itself for internal streams
security.NTACL : the NT ACLthe rationale for making most of them in the user namespace is that it is 'mostly harmless' to allow the owner of the file to change those ones.
Right, it's the "design is broken so everything ends up in user.*". Now, I clearly dislike the StudlyCaps used here, but if it's already deployed it's probably too late to fix this :(
Does Samba have any way do deal with VFAT short names?
-hpa
Quoted 3 lines
Right, it's the "design is broken so everything ends up in user.*". Now, I clearly dislike the StudlyCaps used here, but if it's already deployed it's probably too late to fix this :(
Samba4 is only deployed by a very few brave sites (such as my wifes server) who all know that things might change in non-compatible ways. Still, I'd want a slightly stronger reason than dislike of studly caps to change it :-)
Quoted 1 line
Does Samba have any way do deal with VFAT short names?
Samba doesn't take advantage of the fact that VFAT can store short names directly in the filesystem, and instead deals with short names completely in userspace using a hash based name mangling scheme. It treats VFAT as just another unix filesystem. People who want to deploy a serious Samba server tend to want journaling, ACLs etc, so VFAT isn't a candidate.
I currently don't store short file names in xattrs for Samba4 as there has just no advantage to doing so. Without a "open by xattr contents" call that doesn't have to scan the entire directory it is much more efficient to store the 8.3 names in user-space where we can look them up more efficiently. The scheme we use is to store a cache of 8.3->longname mappings, and when we get a 8.3 name that isn't in cache we fall back to a directory scan, re-forming the 8.3 name using a hash for each directory entry.
I'm also much less concerned about 8.3 names these days than I was a few years ago, as the number of applications that need them is rapidly dropping. There are some obvious exceptions (such as the idiotic API calls that cmd.exe uses to implement "del *.*"), but we have worked out ways to cope with those in a reasonable manner.
Cheers, Tridge
Quoted 9 lines · [email protected] wrote:
Quoted 3 lines
Right, it's the "design is broken so everything ends up in user.*". Now, I clearly dislike the StudlyCaps used here, but if it's already deployed it's probably too late to fix this :(
Samba4 is only deployed by a very few brave sites (such as my wifes server) who all know that things might change in non-compatible ways. Still, I'd want a slightly stronger reason than dislike of studly caps to change it :-)
The slightly stronger reason is basically the same reason why we don't stuff a bunch of things into a struct stat and call a single system call to change a bunch of attributes; you don't want to have to change them all every time, and by putting them all in the same structure that's your only option, since setxattr() doesn't allow you to mask and merge.
Incidentally, the document you pointed me to wasn't clear on the endianness convention.
-hpa
Quoted 3 lines
The slightly stronger reason is basically the same reason why we don't stuff a bunch of things into a struct stat and call a single system call to change a bunch of attributes
yes, and if we want this to be a good API then we'd use something like a bitmask to indicate what fields to update so we can update them in groups in a raceless fashion, but that would require that the kernel understand the internals of these structures. I didn't have that luxury, so I grouped them in the way that best matched the common use of the attributes.
Quoted 3 lines
you don't want to have to change them all every time, and by putting them all in the same structure that's your only option, since
setxattr()doesn't allow you to mask and merge.
can you tell me who you imagining will be using these attributes apart from Samba, Wine and backup/restore apps?
Quoted 3 lines
Incidentally, the document you pointed me to wasn't clear on the endianness convention.
It's little-endian NDR. For a full description of NDR in all its gory details see http://www.opengroup.org/onlinepubs/9629399/chap14.htm
NDR seems like overkill at first, until you start to look at security descriptors. They are very complex beasts, and using IDL/NDR makes it much easier (in fact, security descriptors need some minor enhancements to NDR to encode them the same way windows does).
Cheers, Tridge
Quoted 1 line · H. Peter Anvin
Right, it's the "design is broken so everything ends up in user.*".
The design isn't broken, you're just missing an important detail of what the system namespace entails:
xattrs in the system namespace have a format defined by the kernel and (more importantly -- this is the important detail) modify kernel behavior.
If the xattr namespace was flat, I would have no way of knowing whether or not the kernel will set the Archived bit in fatattrs (or DosAttrib) xattr when I write to a file that has that xattr or whether or not the kernel will choose to enforce the ACL I store in the posix_acl_access xattr.
With the system namespace, I can rely on the fact that xattrs in that namespace actually have a meaning and are in sync with what the kernel believes to be true about the file.
If I cannot rely on this to be true, than at best that's a bug and at worst it's a gaping security hole. (NT ACLs can specify Allow and Deny, if I create a NT ACL xattr that denies somebody access, the kernel damn well better enforce it.)
Earlier you mentioned a desire to be able to backup the various pieces of metadata that a filesystem exports via xattrs simply by copying the files to another filesystem, and the fact that the destination filesystem may not allow you to store the same attributes in the system namespace as the source prevented you from being able to do this.
This is akin to complaining that you cannot make an accurate backup of an ext3 filesystem simply by copying it's files to a vfat filesystem, because vfat doesn't support the same notions of timestamps, ownership or permissions that ext3 does.
Tools such as tar or cpio exist to store Unix files and their metadata in a flat format, and they can be extended to understand the extra metadata made available in Linux xattrs.
Signature · 1 line
Nicholas Miell <[email protected]>
Quoted 14 lines · Nicholas Miell
The design isn't broken, you're just missing an important detail of what the system namespace entails:
xattrs in the system namespace have a format defined by the kernel and (more importantly -- this is the important detail) modify kernel behavior.
If the xattr namespace was flat, I would have no way of knowing whether or not the kernel will set the Archived bit in fatattrs (or DosAttrib) xattr when I write to a file that has that xattr or whether or not the kernel will choose to enforce the ACL I store in the
posix_acl_accessxattr.
Quoted 4 lines
With the system namespace, I can rely on the fact that xattrs in that namespace actually have a meaning and are in sync with what the kernel believes to be true about the file.
What you're neglecting is that there is a LARGE class of metadata where the important thing is that you store them; if you don't know what they are you merely ignore them and keep them as-is.
There is no place for those in the current design.
-hpa
Quoted 5 lines · H. Peter Anvin
What you're neglecting is that there is a LARGE class of metadata where the important thing is that you store them; if you don't know what they are you merely ignore them and keep them as-is.
There is no place for those in the current design.
That's what the user namespace is for -- the kernel has no interest in its contents, it just stores it as-is for apps that do.
Signature · 1 line
Nicholas Miell <[email protected]>
Quoted 4 lines · On Tue, 4 Jan 2005 [email protected] wrote:
you need more than one byte for DOS attrib. These are the bits Samba4 defines:
/* FileAttributes (search attributes) field */
[ ... ]
Quoted 1 line
#define
FILE_ATTRIBUTE_SPARSE0x0200
[ ... ]
Quoted 1 line
#define
FILE_ATTRIBUTE_COMPRESSED0x0800
[ ... ]
Quoted 4 lines
while most apps don't care about the bits beyond
0xFFat the moment, I think that might change, especially for win32 clients accessing linux filesystems via wine and Samba.
Setting the above two attributes from a Linux client accessing NTFS via cifsfs pops up more often too. One scenario would be
ntfsclone --fs-compression --output /mnt/cifsfs/backup.img /dev/foo
and the transparently compressed backup.img should be loopback mountable by whatever preferred NTFS driver for whatever reason.
Szaka
Mike,
Quoted 4 lines
If we're just thinking about MS-oriented discretionary access control then I think the owner of the file is basically king and should be the only normal user to that can read and write it's xattrs. So whatever namespace that is (not system).
for the DACL the owner is king (the owner gets the WRITE_DAC, READ_CONTROL and STD_DELETE access bits forced on), but for the other parts of the full security descriptor this is not true. The owner doesn't get to arbitrarily write to the owner_sid or SACL. Thats why I used security.NTACL not user.NTACL.
I suppose we could have a separate user.DACL attribute, but given that there is just one API that sets all 4 elements of the SD (with a bitmask to say which bits to set), it made more sense to me to group them all together. The disadvantage is that Samba needs to gain/lose root privileges for the "set SD" call even if the client is only asking to set the DACL.
Cheers, Tridge
Quoted 13 lines · [email protected] wrote:
Mike,
Quoted 4 lines
If we're just thinking about MS-oriented discretionary access control then I think the owner of the file is basically king and should be the only normal user to that can read and write it's xattrs. So whatever namespace that is (not system).
for the DACL the owner is king (the owner gets the
WRITE_DAC,READ_CONTROLandSTD_DELETEaccess bits forced on), but for the other parts of the full security descriptor this is not true. The owner doesn't get to arbitrarily write to theowner_sidor SACL. Thats why I used security.NTACL not user.NTACL.
Quoted 7 lines
I suppose we could have a separate user.DACL attribute, but given that there is just one API that sets all 4 elements of the SD (with a bitmask to say which bits to set), it made more sense to me to group them all together. The disadvantage is that Samba needs to gain/lose root privileges for the "set SD" call even if the client is only asking to set the DACL.
Even more so a reason for this not to be a general API.
-hpa
Quoted 2 lines
I noticed that CIFS has a placeholder "user.DosAttrib" in
cifs/xattr.c, although it doesn't seem to be implemented.
Thats taken from Samba4, where it is fully implemented. I guess Steve is planning on integrating cifsfs with the Samba4 way of handling EAs, NT ACLs, attribs, streams etc at some stage.
See
http://samba.org/ftp/unpacked/samba4/source/librpc/idl/xattr.idl
for a full definition of the structures we use.
I used a NDR encoding in each of the xattrs to provide a well defined architecture independent encoding, and an easy way to extend the structure in the future (thats why DosAttrib is a union with a version switch).
The place where this will really interact a lot with the kernel is in the Samba LSM module that tpot and myself have been looking at writing. That module will provide the in-kernel implementation of these attributes that is needed to make them raceless (especially for NT ACLs).
These xattr structures are also the key to solving the case-insensitivity problem that has been plaguing Samba for so long. We have come up with a very simple scheme for making case-insensiitive filenames work very efficiently on any filesystem that supports xattrs. It requires no additional kernel support beyond xattrs, and gets rid of the need for a large user-space name cache.
Cheers, Tridge
Quoted 5 lines · [email protected] wrote:
Thats taken from Samba4, where it is fully implemented. I guess Steve is planning on integrating cifsfs with the Samba4 way of handling EAs, NT ACLs, attribs, streams etc at some stage.
Quoted 9 lines
See
http://samba.org/ftp/unpacked/samba4/source/librpc/idl/xattr.idl
for a full definition of the structures we use.I used a NDR encoding in each of the xattrs to provide a well defined architecture independent encoding, and an easy way to extend the structure in the future (thats why DosAttrib is a union with a version switch).
Oh geez. Couldn't you have split out the various data items into separate xattrs? This seems to be a really bad user interface, especially for writing (can't chmod the file without poking at all the other data items), except for a non-DOS-based filesystem to keep data for Samba itself. Samba clearly has other needs than other users, although of course it would be unfortunate if Samba then can't export this information.
In other words, I'm inclined to define simple system attributes or just go back to the original ioctl() patch for the DOS filesystems as seen by the kernel.
-hpa
Quoted 2 lines
Oh geez. Couldn't you have split out the various data items into separate xattrs?
fetching and setting this stuff is _really_ common, so I have arranged to store them in a way to make it as efficient as possible. For example, when a remote client asks for a directory listing we need to fetch just one xattr from the kernel per file (directory listings in the windows world usually return the equivalent of a full stat structure for every name).
Similarly, when these are set the client tends to set more than one at a time. The most commonly used set call takes this structure:
/* RAW_SFILEINFO_BASIC_INFO and
RAW_SFILEINFO_BASIC_INFORMATION interfaces */
struct {
enum smb_setfileinfo_level level;
union setfileinfo_file file;
struct {
NTTIME create_time;
NTTIME access_time;
NTTIME write_time;
NTTIME change_time;
uint32_t attrib;
} in;
} basic_info;Having the items that tend to get read/written together grouped together allowed me to make it all quite efficient, while still having a reasonable chance of the EAs all fitting in-inode on filesystems that support that.
Obviously it is racy when dealt with from user space, but there really is no way to avoid all these races without a user space accessible "lock the files meta-data" call and that is why I'm looking forward to having a Samba LSM module to avoid these races.
Quoted 3 lines
Samba clearly has other needs than other users, although of course it would be unfortunate if Samba then can't export this information.
I think you'll find that all users of dos attributes on Linux will have very similar needs to Samba, and will want these things grouped together. For example:
- backup/restore apps will want to backup/restore these attributes as lumps
- wine implements essentially the same APIs as Samba, just in a different form, and so tends to get the same groupings of attributes get/set calls that Samba does (the SMB protocol is to a large degree a on-the-wire version of Win32).
Are there any other significant users of DOS attributes on Linux that want something different?
Cheers, Tridge
Quoted 12 lines · [email protected] wrote:
I think you'll find that all users of dos attributes on Linux will have very similar needs to Samba, and will want these things grouped together. For example:
- backup/restore apps will want to backup/restore these attributes as lumps
- wine implements essentially the same APIs as Samba, just in a different form, and so tends to get the same groupings of attributes get/set calls that Samba does (the SMB protocol is to a large degree a on-the-wire version of Win32).
Quoted 3 lines
Are there any other significant users of DOS attributes on Linux that want something different?
I don't know, but it certainly doesn't match my application (which is pretty simple... needing to frob DOS attribute bits while writing DOS files) or expectation thereof... plus it's not very unixy :-/
More or less what you seem to want is an ioctl() that takes a mask of what to write, similar to the way notify_change() works inside the kernel. This is a legitimate API, but it requires knowledge of the internals, and isn't setxattr(). The big thing here is the need for a mask.
Of course, one way one can do this is to expose user.dos.attrib or something like that as a synthesized block form of these, and still have separate system.* xattrs that control the individual options on the filesystems which actually store this stuff.
I certainly see why *you* want it, but it still seems to me to be bad interface design for anything other than backup and file repository programs. Also see my previous note about endianness of structures carried from place to place.
At this point I think I'm going to let my ioctl() patch stand as-is, solving the immediate problem, and let people who are more directly affected delve into this particular morass.
-hpa
Quoted 4 lines
More or less what you seem to want is an
ioctl()that takes a mask of what to write, similar to the waynotify_change()works inside the kernel. This is a legitimate API, but it requires knowledge of the internals, and isn'tsetxattr(). The big thing here is the need for a mask.
That API would make sense, but I didn't really expect the kernel to provide it. What I expected to happen was for Samba4 to use the xattr blobs like it does now, hopefully for Wine to learn to interpret those same blobs, for backup/restore apps to learn to backup/restore them (as blobs, with no interpretation) and for the proposed Samba LSM module to do the dirty work of interpreting the contents of these blobs in-kernel to provide raceless windows file serving.
The LSM module would then expose a richer API to a user space library via some yet to be determined mechanism (netlink? ioctl? sysfs? proc? dunno yet). That API would include the ability to tell the LSM module what "nt token" (the windows equivalent of euid, egid and supplementary groups) to use for operations, thus allowing the module to correctly interpret the NT ACLs for read/write access to each of these attributes. The module would also cache the xattr blob contents, in unpacked form, to allow access decisions to be made very fast.
The whole design was based on the idea that proposing anything more intrusive would (quite rightly) get smacked down as "this is not NT, go away".
Quoted 2 lines
Also see my previous note about endianness of structures carried from place to place.
I specifically chose NDR (and specifically little-endian NDR) as it solves the endianness and 32/64 bit problems. You can take these blobs and put them on any platform and they will be interpreted the same.
We also have a mechanism (an external tdb) for storing these xattrs on filesystems that have no xattr support. That allows Samba4 to be fully functional on any platform, but just much more efficient and scalable on platforms that do have xattrs.
Cheers, Tridge
Quoted 9 lines · [email protected] wrote:
That API would make sense, but I didn't really expect the kernel to provide it. What I expected to happen was for Samba4 to use the xattr blobs like it does now, hopefully for Wine to learn to interpret those same blobs, for backup/restore apps to learn to backup/restore them (as blobs, with no interpretation) and for the proposed Samba LSM module to do the dirty work of interpreting the contents of these blobs in-kernel to provide raceless windows file serving.
Right, but here we're talking about exposing the guts of a DOS-based filesystem so they can be manipulated by an application which isn't necessarily a bulk file handler. The API doesn't really work for that, especially since some of the attributes have different access properties from the others.
-hpa
Quoted 5 lines
Right, but here we're talking about exposing the guts of a DOS-based filesystem so they can be manipulated by an application which isn't necessarily a bulk file handler. The API doesn't really work for that, especially since some of the attributes have different access properties from the others.
right. Samba doesn't care much about VFAT, and you don't care about all the other attributes, so we should get along fine without treading on each others toes too much.
I explained what Samba4 does as you asked about the user.DosAttrib xattr that Steve put a placeholder for in cifsfs. That came from Samba4, so if you suddenly started using it in a different way I would get a sore toe :-)
Once the Samba LSM module is done and Wine and Samba start working more together on all these extra bits of meta-data then we could consider making your ioctl work on all filesystems when the LSM module is loaded.
Cheers, Tridge
Quoted 10 lines · [email protected] wrote:
right. Samba doesn't care much about VFAT, and you don't care about all the other attributes, so we should get along fine without treading on each others toes too much.
I explained what Samba4 does as you asked about the user.DosAttrib xattr that Steve put a placeholder for in cifsfs. That came from Samba4, so if you suddenly started using it in a different way I would get a sore toe :-)
Quoted 5 lines
Once the Samba LSM module is done and Wine and Samba start working more together on all these extra bits of meta-data then we could consider making your ioctl work on all filesystems when the LSM module is loaded.
Sounds reasonable.
-hpa
Quoted 4 lines · On Jan 03, 2005, at 21:05, [email protected] wrote:
Once the Samba LSM module is done and Wine and Samba start working more together on all these extra bits of meta-data then we could consider making your ioctl work on all filesystems when the LSM module is loaded.
IMHO, it would be really nice (Although I realize this would be a humongous task) if there was a "Linux" ACL system supported by the VFS that is backwards compatible with UNIX perms, POSIX ACLs, and NTFS ACLs (One of the few things that Microsoft got right). The fact stands that NT ACLs are much more powerful and useful than their POSIX counterparts (Although some of the subdirectory inheritance stuff is rather uselessly algorithmically complicated.)
I think it would be a major achievement if some kind of design and API could be decided upon, because at least that way there would be some kind of foundation upon which to build patches.
I'd like to work on a patch for this, but I'm afraid my kernel coding skills are a bit rusty at the moment. Sorry :-D
Cheers,
Kyle Moffett
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------
Kyle,
Quoted 4 lines
IMHO, it would be really nice (Although I realize this would be a humongous task) if there was a "Linux" ACL system supported by the VFS that is backwards compatible with UNIX perms, POSIX ACLs, and NTFS ACLs
The problem is that apps/users that care about posix ACLs and NT ACLs tend to care about them being _exactly_ right, so half baked mappings between the different systems don't really help much. We have learnt this the hard way in Samba3, where we attempt to map NT ACLs to posix ACLs on the fly, and we get a never ending stream of bug reports that NT ACLs aren't working as expected for windows users. Thats why I stuck the full NT ACL in an xattr for Samba4.
The plan for mapping between the systems in Samba4 goes like this:
- when a NT ACL is needed for access control, or the client asks for the ACL via a query file info call, smbd first looks in the NTACL xattr. If it finds it then all is well with the world.
- if the NTACL xattr isn't there, then it produces one using an approximate mapping. The mapping for mode_t is in the function pvfs_default_acl() in pvfs_acl.c. The mapping when there is an existing posix ACL isn't done yet (we have a mapping for that in Samba3, but it will be a bit different in Samba4).
- if its a file/directory create, then it also obeys the inheritance rules, pulling the NTACL from the parent using the above method then applying to the child.
I also plan to catch the posix ACL set call in the Samba LSM module, and wipe the NTACL when that happens. In that case the NTACL will be overridden by the new posix ACL.
Similarly, when a posix application asks to read the posix ACL, then a reverse mapping would be supplied if there is a current NT ACL on the file.
The important thing is that if your file access is either only from unix apps that know about posix ACLs or only from windows apps that know about NT ACLs, then you need to provide perfect lossless storage of those native ACLs for those applications. It is only when you get a transition from one scheme to the other on the one file that a mapping should happen.
Cheers, Tridge
Quoted 6 lines · On Jan 03, 2005, at 21:49, [email protected] wrote:
The important thing is that if your file access is either only from unix apps that know about posix ACLs or only from windows apps that know about NT ACLs, then you need to provide perfect lossless storage of those native ACLs for those applications. It is only when you get a transition from one scheme to the other on the one file that a mapping should happen.
I was thinking something more along the lines of a more complex and detailed scheme that is a superset of both NT ACLs and POSIX ACLs. Then that scheme could be used for DAC within the kernel, although it would still continue to support operations using the POSIX API by translating losslessly from POSIX to "Linux", although it might not work the other way around. Such an ACL would need to be well designed internally to be compatible with future ACL structures and such, but it would make your work a lot easier. There will always be data loss when translating between incompatible formats, but if we can come up with some kind of system that can handle both data-sets losslessly, then we could have the VFS use that, even if the only programs that understand it are the ACL tools.
Cheers,
Kyle Moffett
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------
Kyle,
Quoted 2 lines
I was thinking something more along the lines of a more complex and detailed scheme that is a superset of both NT ACLs and POSIX ACLs.
superset is hard, as a uid_t/gid_t is only superfically similar to a windows SID. Samba has to do quite a lot of complex stuff to map between general SIDs and posix IDs. It can't be done in any reasonable fashion without being able to talk MSRPC to domain controllers, or at least having a (potentially quite large) persistent database of mappings.
The schemes that attempt to do general SID -> uid/gid mappings via fixed algorithmic mappings are hopeless. They are great for toy demos, but useless for real deployments.
Cheers, Tridge
Quoted 6 lines · On Jan 03, 2005, at 22:56, [email protected] wrote:
superset is hard, as a uid_t/gid_t is only superfically similar to a windows SID. Samba has to do quite a lot of complex stuff to map between general SIDs and posix IDs. It can't be done in any reasonable fashion without being able to talk MSRPC to domain controllers, or at least having a (potentially quite large) persistent database of mappings.
I only have a couple points of response here, really. Primarily, I didn't say that I thought this was possible, let alone easy, only that it would be closer to the mythical "ideal" than the system what we currently have.
My personal opinion is that perhaps this (ideally) should be integrated somehow with an additional "credentials" system, perhaps like that which dhowells added. An "SID" for a particular local NTFS (or maybe even remote via CIFS too) filesystem would be another "credential" given to users. Heck, if we _really_ wanted to go far, the "user" and "group" identifiers originating in UNIX could be "credentials" too, and passed on from parent to child processes. In such a system, many of the core process identity syscalls would need to be changed, but it would be the framework for a much better network environment integration, because the UID and GIDs would be local to one specific computer, not global like NFS pretends they are. Someone could even have differing UIDs for local ext3 filesystems and remote NFSv3 and CIFS filesystems, like this:
/ "bobdoe@localhost" (32)/mnt/nfs "bdoe@cronos" (1054)/mnt/cifs "Bob Doe@ATLAS" (S-1-3-21-123123-456456-789789-32)
Of course, the standard disclaimer is that all this is far far down the road, mainly because of how extensive such changes would be :-\. It's always nice to dream, though :-D.
Cheers,
Kyle Moffett
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCM/CS/IT/U d- s++: a18 C++++>$ UB/L/X/*++++(+)>$ P+++(++++)>$ L++++(+++) E W++(+) N+++(++) o? K? w--- O? M++ V? PS+() PE+(-) Y+ PGP+++ t+(+++) 5 X R? tv-(--) b++++(++) DI+ D+ G e->++++$ h!*()>++$ r !y?(-) ------END GEEK CODE BLOCK------
Quoted 3 lines · H. Peter Anvin
In other words, I'm inclined to define simple system attributes or just go back to the original
ioctl()patch for the DOS filesystems as seen by the kernel.
Well if you're just interested in DOS attributes you don't care about how Samba handles ACLs. You're only interested in about 3 bits. Anyway I have an idea...
If the attrib member in xattr_DosInfo2 was moved to the beginning of the structure the all attrib members would always be encoded at the same offset within the data returned by getxattr regardless of what info strucutre was used. So you could get and set these attributes as user.DosAttrib in a fairly compatible way.
Mike
Quoted 13 lines · H. Peter Anvin
Hello all,
I recently posted to LKML a patch to get or set DOS attribute flags for fatfs. That patch used
ioctl(). It was suggested that a better way would be using xattrs, although the xattr mechanism seems clumsy to me, and has namespace issues.I also think it would be good to have a unified interface for FAT, NTFS and CIFS for these attributes.
I noticed that CIFS has a placeholder "user.DosAttrib" in
cifs/xattr.c, although it doesn't seem to be implemented.
Quoted 6 lines
Questions:
a) is xattr the right thing? It seems to be a fairly complex and ill-thought-out mechanism all along, especially the whole namespace business (what is a system attribute to one filesystem is a user attribute to another, for example.)
More importantly, what has a defined meaning for one filesystem and is interpreted and generated on demand by the kernel is irrelevant or unsupported on other filesystems.
So, yes, you can't just copy a bunch of files from vfat to ext3 and preserve the vfat attributes, but you should be able to stuff a bunch of vfat files into a tar file and then restore them to a vfat filesystem with all their original attributes intact.
Quoted 2 lines
b) if xattr is the right thing, shouldn't this be in the system namespace rather than the user namespace?
Yes.
Quoted 2 lines
c) What should the representation be? Binary byte? String containing a subset of "rhsvda67" (barf)?
ASCII strings require no special tools to manipulate from shell scripts (or even for the end user to interpret).
Signature · 1 line
Nicholas Miell <[email protected]>
Quoted 18 lines · H. Peter Anvin
I recently posted to LKML a patch to get or set DOS attribute flags for fatfs. That patch used
ioctl(). It was suggested that a better way would be using xattrs, although the xattr mechanism seems clumsy to me, and has namespace issues.I also think it would be good to have a unified interface for FAT, NTFS and CIFS for these attributes.
I noticed that CIFS has a placeholder "user.DosAttrib" in
cifs/xattr.c, although it doesn't seem to be implemented.Questions:
a) is xattr the right thing? It seems to be a fairly complex and ill-thought-out mechanism all along, especially the whole namespace business (what is a system attribute to one filesystem is a user attribute to another, for example.)
Quoted 5 lines
b) if xattr is the right thing, shouldn't this be in the system namespace rather than the user namespace?
c) What should the representation be? Binary byte? String containing a subset of "rhsvda67" (barf)?
Definitely not c!
In NTFS, the "dos attribute flags" are part of the system information attribute which is an entity in its own right, totally separate from extended attributes (and named streams for that matter). So if I were to be thinking in an NTFS-only world I would be inclined to use an ioctl() to access/modify them (i.e. not b either). So if you implement an ioctl() for vfat I will probably be able to provide the same in NTFS with almost zero effort (we already have the code to read and write the attribute flags in the kernel ntfs driver, we just do not provide an interface for it).
But please note that it would be best if you could use 32-bits for the flags. At the very least 16-bits though as on NTFS there are currently in use 16-bits in the standard information but the field is u32 sized on disk (little endian) and two of the higher bits are in use in the file name attribute as well and I would not be surprised if more bits get used in future NTFS releases. Tridge already gave you a list of all the Samba dos attributes so here is the full list for NTFS (note they are 100% compatible to the Samba ones and also note in NTFS we always keep the flags in little endian and just define all the constants to be little endian as well - makes life much easier):
/*
* File attribute flags (32-bit).
*/
enum {
/*
* The following flags are only present in theSTANDARD_INFORMATION
* attribute (in the field file_attributes).
*/
FILE_ATTR_READONLY = const_cpu_to_le32(0x00000001),
FILE_ATTR_HIDDEN = const_cpu_to_le32(0x00000002),
FILE_ATTR_SYSTEM = const_cpu_to_le32(0x00000004),
/* Old DOS volid. Unused in NT. = const_cpu_to_le32(0x00000008),*/
FILE_ATTR_DIRECTORY = const_cpu_to_le32(0x00000010),
/* Note, FILE_ATTR_DIRECTORY is not considered valid in NT. It
is
reserved for the DOS SUBDIRECTORY flag. */
FILE_ATTR_ARCHIVE = const_cpu_to_le32(0x00000020),
FILE_ATTR_DEVICE = const_cpu_to_le32(0x00000040),
FILE_ATTR_NORMAL = const_cpu_to_le32(0x00000080),
FILE_ATTR_TEMPORARY = const_cpu_to_le32(0x00000100),
FILE_ATTR_SPARSE_FILE = const_cpu_to_le32(0x00000200),
FILE_ATTR_REPARSE_POINT = const_cpu_to_le32(0x00000400),
FILE_ATTR_COMPRESSED = const_cpu_to_le32(0x00000800),
FILE_ATTR_OFFLINE = const_cpu_to_le32(0x00001000),
FILE_ATTR_NOT_CONTENT_INDEXED = const_cpu_to_le32(0x00002000),
FILE_ATTR_ENCRYPTED = const_cpu_to_le32(0x00004000),
FILE_ATTR_VALID_FLAGS = const_cpu_to_le32(0x00007fb7),
/* Note, FILE_ATTR_VALID_FLAGS masks out the old DOS VolId and
the
FILE_ATTR_DEVICE and preserves everything else. This mask is
used
to obtain all flags that are valid for reading. */
FILE_ATTR_VALID_SET_FLAGS = const_cpu_to_le32(0x000031a7),
/* Note, FILE_ATTR_VALID_SET_FLAGS masks out the old DOS VolId,the
F_A_DEVICE, F_A_DIRECTORY, F_A_SPARSE_FILE,F_A_REPARSE_POINT,
F_A_COMPRESSED, and F_A_ENCRYPTED and preserves the rest.
This mask
is used to to obtain all flags that are valid for setting. */
/*
* The following flags are only present in the FILE_NAME
attribute (in
* the field file_attributes).
*/
FILE_ATTR_DUP_FILE_NAME_INDEX_PRESENT =const_cpu_to_le32(0x10000000), /* Note, this is a copy of the
corresponding bit from the mft record,
telling us whether this is a directory or not, i.e. whether
it has
an index root attribute or not. */
FILE_ATTR_DUP_VIEW_INDEX_PRESENT =const_cpu_to_le32(0x20000000), /* Note, this is a copy of the
corresponding bit from the mft record,
telling us whether this file has a view index present (eg.
object id
index, quota index, one of the security indexes or the
encrypting
file system related indexes). */
};
typedef le32 FILE_ATTR_FLAGS;
Best regards,
Anton
Signature · 4 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/
Quoted 19 lines · Anton Altaparmakov
Quoted 18 lines · H. Peter Anvin
I recently posted to LKML a patch to get or set DOS attribute flags for fatfs. That patch used
ioctl(). It was suggested that a better way would be using xattrs, although the xattr mechanism seems clumsy to me, and has namespace issues.I also think it would be good to have a unified interface for FAT, NTFS and CIFS for these attributes.
I noticed that CIFS has a placeholder "user.DosAttrib" in
cifs/xattr.c, although it doesn't seem to be implemented.Questions:
a) is xattr the right thing? It seems to be a fairly complex and ill-thought-out mechanism all along, especially the whole namespace business (what is a system attribute to one filesystem is a user attribute to another, for example.)
Quoted 7 lines
Quoted 5 lines
b) if xattr is the right thing, shouldn't this be in the system namespace rather than the user namespace?
c) What should the representation be? Binary byte? String containing a subset of "rhsvda67" (barf)?
Definitely not c!
I meant definitely not the second suggestion of c (i.e. the string subset). That is just horrible...
Quoted 10 lines
In NTFS, the "dos attribute flags" are part of the system information attribute which is an entity in its own right, totally separate from extended attributes (and named streams for that matter). So if I were to be thinking in an NTFS-only world I would be inclined to use an
ioctl()to access/modify them (i.e. not b either). So if you implement anioctl()for vfat I will probably be able to provide the same in NTFS with almost zero effort (we already have the code to read and write the attribute flags in the kernel ntfs driver, we just do not provide an interface for it).
Quoted 10 lines
But please note that it would be best if you could use 32-bits for the flags. At the very least 16-bits though as on NTFS there are currently in use 16-bits in the standard information but the field is u32 sized on disk (little endian) and two of the higher bits are in use in the file name attribute as well and I would not be surprised if more bits get used in future NTFS releases. Tridge already gave you a list of all the Samba dos attributes so here is the full list for NTFS (note they are 100% compatible to the Samba ones and also note in NTFS we always keep the flags in little endian and just define all the constants to be little endian as well - makes life much easier):
Another reason to want 32-bits is that this could be used to add more Linux specific bits like "FILE_ATTR_CHAR_DEV" for character device special files and use the existing "FILE_ATTR_DEVICE" for block devices. Windows unfortunately does not distinguish between the two in the flags. )-:
Quoted 68 lines
/*
* File attribute flags (32-bit).
*/
enum {
/*
* The following flags are only present in theSTANDARD_INFORMATION* attribute (in the field file_attributes). */ FILE_ATTR_READONLY = const_cpu_to_le32(0x00000001), FILE_ATTR_HIDDEN = const_cpu_to_le32(0x00000002), FILE_ATTR_SYSTEM = const_cpu_to_le32(0x00000004), /* Old DOS volid. Unused in NT. = const_cpu_to_le32(0x00000008),*/
FILE_ATTR_DIRECTORY = const_cpu_to_le32(0x00000010), /* Note, FILE_ATTR_DIRECTORY is not considered valid in NT. It is reserved for the DOS SUBDIRECTORY flag. */ FILE_ATTR_ARCHIVE = const_cpu_to_le32(0x00000020), FILE_ATTR_DEVICE = const_cpu_to_le32(0x00000040), FILE_ATTR_NORMAL = const_cpu_to_le32(0x00000080), FILE_ATTR_TEMPORARY = const_cpu_to_le32(0x00000100), FILE_ATTR_SPARSE_FILE = const_cpu_to_le32(0x00000200), FILE_ATTR_REPARSE_POINT = const_cpu_to_le32(0x00000400), FILE_ATTR_COMPRESSED = const_cpu_to_le32(0x00000800), FILE_ATTR_OFFLINE = const_cpu_to_le32(0x00001000), FILE_ATTR_NOT_CONTENT_INDEXED = const_cpu_to_le32(0x00002000), FILE_ATTR_ENCRYPTED = const_cpu_to_le32(0x00004000), FILE_ATTR_VALID_FLAGS = const_cpu_to_le32(0x00007fb7), /* Note, FILE_ATTR_VALID_FLAGS masks out the old DOS VolId and the FILE_ATTR_DEVICE and preserves everything else. This mask is used to obtain all flags that are valid for reading. */ FILE_ATTR_VALID_SET_FLAGS = const_cpu_to_le32(0x000031a7), /* Note, FILE_ATTR_VALID_SET_FLAGS masks out the old DOS VolId,the
F_A_DEVICE,F_A_DIRECTORY,F_A_SPARSE_FILE,F_A_REPARSE_POINT,
F_A_COMPRESSED, andF_A_ENCRYPTEDand preserves the rest.
This mask
is used to to obtain all flags that are valid for setting. *//*
* The following flags are only present in theFILE_NAME
attribute (in
* the fieldfile_attributes).
*/
FILE_ATTR_DUP_FILE_NAME_INDEX_PRESENT=const_cpu_to_le32(0x10000000), /* Note, this is a copy of the
corresponding bit from the mft record,
telling us whether this is a directory or not, i.e. whether
it has
an index root attribute or not. */
FILE_ATTR_DUP_VIEW_INDEX_PRESENT=const_cpu_to_le32(0x20000000), /* Note, this is a copy of the
corresponding bit from the mft record,
telling us whether this file has a view index present (eg.
object id
index, quota index, one of the security indexes or the
encrypting
file system related indexes). */
};
Quoted 1 line
typedef le32
FILE_ATTR_FLAGS;
Best regards,
Anton
Signature · 4 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/
Quoted 19 lines · Anton Altaparmakov
Quoted 18 lines · H. Peter Anvin
I recently posted to LKML a patch to get or set DOS attribute flags for fatfs. That patch used
ioctl(). It was suggested that a better way would be using xattrs, although the xattr mechanism seems clumsy to me, and has namespace issues.I also think it would be good to have a unified interface for FAT, NTFS and CIFS for these attributes.
I noticed that CIFS has a placeholder "user.DosAttrib" in
cifs/xattr.c, although it doesn't seem to be implemented.Questions:
a) is xattr the right thing? It seems to be a fairly complex and ill-thought-out mechanism all along, especially the whole namespace business (what is a system attribute to one filesystem is a user attribute to another, for example.)
Quoted 7 lines
Quoted 5 lines
b) if xattr is the right thing, shouldn't this be in the system namespace rather than the user namespace?
c) What should the representation be? Binary byte? String containing a subset of "rhsvda67" (barf)?
Definitely not [a string]!
Why not? It makes special setfatattr and getfatattr (or setntfsattr and getnetfsattr) tools completely unnecessary. All you need is setfattr and getfattr, which already exist.
Quoted 11 lines
In NTFS, the "dos attribute flags" are part of the system information attribute which is an entity in its own right, totally separate from extended attributes (and named streams for that matter). So if I were to be thinking in an NTFS-only world I would be inclined to use an
ioctl()to access/modify them (i.e. not b either). So if you implement anioctl()for vfat I will probably be able to provide the same in NTFS with almost zero effort (we already have the code to read and write the attribute flags in the kernel ntfs driver, we just do not provide an interface for it).
An ioctl would be wrong. Remember, the point of this exercise is to expose these attributes in such a way that tools don't have to have any special knowledge to correctly preserve them.
If I want to be able to copy files from one NTFS volume to another (preserving all their NTFS attributes), I don't want to have to teach cp to run a Linux-specific and NTFS-specific ioctl on each file on the source and destination for it to work, it should be able to see the xattrs and just do the right thing.
The fact that the NTFS "dos attribute flags" are seperate from real extended attributes isn't a problem, either. Real extended attributes can be exported in the user namespace, just like ext2/3 does. (Or are the real extended attributes something other than inert blobs of data -- does Windows care about their contents at all, or does it just store them for users who do?)
Quoted 11 lines
But please note that it would be best if you could use 32-bits for the flags. At the very least 16-bits though as on NTFS there are currently in use 16-bits in the standard information but the field is u32 sized on disk (little endian) and two of the higher bits are in use in the file name attribute as well and I would not be surprised if more bits get used in future NTFS releases. Tridge already gave you a list of all the Samba dos attributes so here is the full list for NTFS (note they are 100% compatible to the Samba ones and also note in NTFS we always keep the flags in little endian and just define all the constants to be little endian as well - makes life much easier):
Using anything other than 8-bits to represent the FAT attributes would be wrong, too. It's better to separate the xattr holding NTFS DOS attributes (what genius at Microsoft named this...) from the xattr holding FAT attributes so that, again, cp can do the right thing without any special knowledge about filesystems.
If the attributes aren't separated into different xattrs, what would happen when I copy files from an NTFS volume to a VFAT volume?
cp will attempt to copy xattrs containing NTFS-specific bits that VFAT can't store, now what? VFAT could either silently discard the bits that it doesn't support, or it could fail the entire operation. Either way, it has to do the wrong thing.
Now, if the attributes have been seperated into system.fatattrs and system.ntfsattrs, cp will be able to completely preserve system.fatattrs with no trouble at all and inform the user that it could not create system.ntfsattrs (similar to what it does now when it cannot preserve ownership or permissions when copying from ext3 to VFAT).
Quoted 4 lines
Best regards,
Anton
Signature · 1 line
Nicholas Miell <[email protected]>
Quoted 23 lines · Nicholas Miell
Quoted 22 lines · Anton Altaparmakov
Quoted 21 lines · H. Peter Anvin
I recently posted to LKML a patch to get or set DOS attribute flags for fatfs. That patch used
ioctl(). It was suggested that a better way would be using xattrs, although the xattr mechanism seems clumsy to me, and has namespace issues.I also think it would be good to have a unified interface for FAT, NTFS and CIFS for these attributes.
I noticed that CIFS has a placeholder "user.DosAttrib" in
cifs/xattr.c, although it doesn't seem to be implemented.Questions:
a) is xattr the right thing? It seems to be a fairly complex and ill-thought-out mechanism all along, especially the whole namespace business (what is a system attribute to one filesystem is a user attribute to another, for example.)
b) if xattr is the right thing, shouldn't this be in the system namespace rather than the user namespace?
Quoted 8 lines
Quoted 4 lines
Quoted 2 lines
c) What should the representation be? Binary byte? String containing a subset of "rhsvda67" (barf)?
Definitely not [a string]!
Why not? It makes special setfatattr and getfatattr (or setntfsattr and getnetfsattr) tools completely unnecessary. All you need is setfattr and getfattr, which already exist.
For one because seeing things like "rhsvda67" makes me want to throw up. (-; Also strings are no use for internationalisation. Binary is much shorter and already well defined. If you want text, just have a library or even do it in the application. The lib or app can decode the binary to text and output it in any language it desires and in any shape it desires. Also, binary bit testing is a hell of a lot faster than strncmp() on each and every string...
Quoted 10 lines
Quoted 9 lines
In NTFS, the "dos attribute flags" are part of the system information attribute which is an entity in its own right, totally separate from extended attributes (and named streams for that matter). So if I were to be thinking in an NTFS-only world I would be inclined to use an
ioctl()to access/modify them (i.e. not b either). So if you implement anioctl()for vfat I will probably be able to provide the same in NTFS with almost zero effort (we already have the code to read and write the attribute flags in the kernel ntfs driver, we just do not provide an interface for it).
Quoted 3 lines
An ioctl would be wrong. Remember, the point of this exercise is to expose these attributes in such a way that tools don't have to have any special knowledge to correctly preserve them.
Certainly not for me. I don't care about backup programs. We have ntfsclone for that. You can never use xattrs to do full backups of ntfs anyway so no point in trying. How do you deal with compression or encryption for example? Do you just allow everyone to read the crypto keys so they can back them up? What about ACLs? And quotas? And reparse points (sym links and mount points and more complex stuff like DFS/HFS). The only way you can really backup ntfs IMO is using ntfsclone or if the ntfs driver and/or libntfs were to provide it, the MS BackupAPI or something equivalent. Perhaps the "Linux Backup API"? That could be defined and then exported by the VFS with appropriate hooks for each fs that does the nitty gritty work. (-;
Quoted 6 lines
If I want to be able to copy files from one NTFS volume to another (preserving all their NTFS attributes), I don't want to have to teach cp to run a Linux-specific and NTFS-specific ioctl on each file on the source and destination for it to work, it should be able to see the xattrs and just do the right thing.
Quoted 6 lines
The fact that the NTFS "dos attribute flags" are seperate from real extended attributes isn't a problem, either. Real extended attributes can be exported in the user namespace, just like ext2/3 does. (Or are the real extended attributes something other than inert blobs of data -- does Windows care about their contents at all, or does it just store them for users who do?)
I do not believe windows cares about the EAs at all.
Quoted 11 lines
Quoted 10 lines
But please note that it would be best if you could use 32-bits for the flags. At the very least 16-bits though as on NTFS there are currently in use 16-bits in the standard information but the field is u32 sized on disk (little endian) and two of the higher bits are in use in the file name attribute as well and I would not be surprised if more bits get used in future NTFS releases. Tridge already gave you a list of all the Samba dos attributes so here is the full list for NTFS (note they are 100% compatible to the Samba ones and also note in NTFS we always keep the flags in little endian and just define all the constants to be little endian as well - makes life much easier):
Quoted 8 lines
Using anything other than 8-bits to represent the FAT attributes would be wrong, too. It's better to separate the xattr holding NTFS DOS attributes (what genius at Microsoft named this...) from the xattr holding FAT attributes so that, again, cp can do the right thing without any special knowledge about filesystems.
If the attributes aren't separated into different xattrs, what would happen when I copy files from an NTFS volume to a VFAT volume?
Simply loose the bits that cannot be preserved. It makes no sense to try to preserve them on VFAT as they have no meaning there. It is just pointless.
Quoted 4 lines
cp will attempt to copy xattrs containing NTFS-specific bits that VFAT can't store, now what? VFAT could either silently discard the bits that it doesn't support, or it could fail the entire operation. Either way, it has to do the wrong thing.
So according to your arguments all FS now need to support NTFS based compression and encryption as well then? So no information gets lots when copying from one to the other? Unlikely... Far more likely is that you just loose this information and you copy the file uncompressed / unencrypted. You then also want to loose the compressed/encrypted bits on the file attributes anyway as the file is no longer compressed/encrypted. That would certainly be the element of least surprise. You simply loose unsupported bits since you also loose all the features that go with their meaning. Makes perfect sense to me anyway. YMMV (-;
Best regards,
Anton
Signature · 4 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/ & http://www-stu.christs.cam.ac.uk/~aia21/
Quoted 12 lines · Anton Altaparmakov
Quoted 10 lines · Nicholas Miell
Quoted 5 lines · Anton Altaparmakov
Quoted 2 lines · H. Peter Anvin
c) What should the representation be? Binary byte? String containing a subset of "rhsvda67" (barf)?
Definitely not [a string]!
Why not? It makes special setfatattr and getfatattr (or setntfsattr and getntfsattr) tools completely unnecessary. All you need is setfattr and getfattr, which already exist.
Quoted 7 lines
For one because seeing things like "rhsvda67" makes me want to throw up. (-; Also strings are no use for internationalisation. Binary is much shorter and already well defined. If you want text, just have a library or even do it in the application. The lib or app can decode the binary to text and output it in any language it desires and in any shape it desires. Also, binary bit testing is a hell of a lot faster than
strncmp()on each and every string...
Eh. I only suggested stringifying it in the first place to make the "use text in /proc and /sys for shell scripts!! and people!!" faction happy, I'm not particularly attached to the idea. (Although, I have become a bit enamored with how the string representation needs no special tools to use.)
Quoted 15 lines
Quoted 11 lines
Quoted 10 lines
In NTFS, the "dos attribute flags" are part of the system information attribute which is an entity in its own right, totally separate from extended attributes (and named streams for that matter). So if I were to be thinking in an NTFS-only world I would be inclined to use an
ioctl()to access/modify them (i.e. not b either). So if you implement anioctl()for vfat I will probably be able to provide the same in NTFS with almost zero effort (we already have the code to read and write the attribute flags in the kernel ntfs driver, we just do not provide an interface for it).Quoted 3 lines
An ioctl would be wrong. Remember, the point of this exercise is to expose these attributes in such a way that tools don't have to have any special knowledge to correctly preserve them.
Quoted 11 lines
Certainly not for me. I don't care about backup programs. We have ntfsclone for that. You can never use xattrs to do full backups of ntfs anyway so no point in trying. How do you deal with compression or encryption for example? Do you just allow everyone to read the crypto keys so they can back them up? What about ACLs? And quotas? And reparse points (sym links and mount points and more complex stuff like DFS/HFS). The only way you can really backup ntfs IMO is using ntfsclone or if the ntfs driver and/or libntfs were to provide it, the MS BackupAPI or something equivalent. Perhaps the "Linux Backup API"? That could be defined and then exported by the VFS with appropriate hooks for each fs that does the nitty gritty work. (-;
I'm not talking about backing up NTFS, I'm talking about preserving data that is trivial to preserve. If it can be done, it should be done.
Quoted 8 lines
Quoted 6 lines
The fact that the NTFS "dos attribute flags" are seperate from real extended attributes isn't a problem, either. Real extended attributes can be exported in the user namespace, just like ext2/3 does. (Or are the real extended attributes something other than inert blobs of data -- does Windows care about their contents at all, or does it just store them for users who do?)
I do not believe windows cares about the EAs at all.
That's good, then they can just be exported in the user namespace.
Quoted 6 lines
Quoted 6 lines
Using anything other than 8-bits to represent the FAT attributes would be wrong, too. It's better to separate the xattr holding NTFS DOS attributes (what genius at Microsoft named this...) from the xattr holding FAT attributes so that, again, cp can do the right thing without any special knowledge about filesystems.
Quoted 6 lines
Quoted 2 lines
If the attributes aren't separated into different xattrs, what would happen when I copy files from an NTFS volume to a VFAT volume?
Simply loose the bits that cannot be preserved. It makes no sense to try to preserve them on VFAT as they have no meaning there. It is just pointless.
Wouldn't you want the user to know that the bits are being thrown away? Silently discarding data is bound to piss somebody off.
Quoted 5 lines
Quoted 4 lines
cp will attempt to copy xattrs containing NTFS-specific bits that VFAT can't store, now what? VFAT could either silently discard the bits that it doesn't support, or it could fail the entire operation. Either way, it has to do the wrong thing.
Quoted 9 lines
So according to your arguments all FS now need to support NTFS based compression and encryption as well then? So no information gets lots when copying from one to the other? Unlikely... Far more likely is that you just loose this information and you copy the file uncompressed / unencrypted. You then also want to loose the compressed/encrypted bits on the file attributes anyway as the file is no longer compressed/encrypted. That would certainly be the element of least surprise. You simply loose unsupported bits since you also loose all the features that go with their meaning. Makes perfect sense to me anyway. YMMV (-;
I never suggested that all filesystems should support every feature that every filesystem has ever supported, and I don't know where you got such a crazy idea.
I am advocating two things:
1) If information can be accurately preserved, it should be.
2) If information cannot be accurately preserved, the user should be notified of this fact.
My suggestion of creating two seperate xattrs meets both of these requirements nicely.
The system.fatattrs xattr is the interface to the bits supported by every Microsoft filesystem, whether it be msdos, vfat, CIFS, or NTFS.
The system.ntfsattrs xattr is the interface to the bits only supported by NTFS (and CIFS, too, I suppose).
Thus, I know for a fact that if I am able to use the system.fatattr xattr with a file, that that filesystem accurately supports FAT attributes. I also know that if I am able to use system.ntfsattr, that the filesystem accurately supports NTFS DOS attributes. I can can use neither, than I know that I will lose information, and if I can use one and not the other, I know which information will be lost.
[ Note to audience: the following is a long (and largely irrelevant to the subject) discussion of how NTFS could implement reparse points and encryption on Linux. Feel free to ignore it. ]
Of course, reparse points and encrypted files make this difficult because those are NTFS DOS attribute bits that require extra information and cannot be explicitly set. I'm not sure how you'd want to implement that, but here's my suggestion (although, you're the NTFS expert, so these may be completely wrong).
Reparse points fall into two categories: those that Linux supports (afaik, only symlinks) and those that Linux doesn't support.
Reparse points that Linux doesn't support are (conceptually) easy: just add a system.ntfs_reparse_point xattr that exports the reparse tag, reparse GUID, and the data buffer.
If something attempts set the reparse bit in the NTFS DOS attr on a file without the system.ntfs_reparse_point xattr, silently clear that bit, if something attempts to clear that bit on a file with the xattr, silently set the bit, when something creates the xattr (i.e. when something turns a file/directory into a reparse point), set the bit, when something removes the xattr, clear the bit.
Reparse points that Linux does support (symlinks) are more difficult. You could pretend that they aren't actually reparse points and just make them look like symlinks to the Linux VFS with no bit set in the NTFS DOS attributes and no system.ntfs_reparse_point xattr.
Or you could include the bit in the attributes and the system.ntfs_reparse_point xattr, but silently resist any attempt to change them via the xattr interface.
Finally, you could allow arbitrary modification of the system.ntfs_reparse_point xattr, but then you have to deal with the possibility that something could come along and turn a normal file into a symlink right under the VFS's nose, which is bound to cause problems.
Encryption doesn't look to be all that hard, actually. When an app sets the Encrypted bit, you can grab the appropriate NTFS-encrypting-key from their keyring and use that to encrypt the file.
Arranging for copying files to use the right key is harder. If you're happy with Win2k semantics, you can live with the fact that setting the Encrypted bit just causes the file to be encrypted with the current key. If you want XP semantics, where copying a file causes the copy to be encrypted with the same key if possible (or the default key, if not), you could add a system.ntfs_key_ids xattr that lists the ids of the keys used and require that apps create all xattrs before writing to the file for copying to work.
Those are just my ideas, I've no intention of implementing them and you can probably come up with better ones.
Of course, I don't use NTFS (or, for that matter, FAT and CIFS), so I don't care all that much what you do, beyond my general desire for Linux to be stellar instead of merely adequate.
Quoted 3 lines
Best regards,
Anton
Signature · 1 line
Nicholas Miell <[email protected]>
Quoted 3 lines · Nicholas Miell
[ Note to audience: the following is a long (and largely irrelevant to the subject) discussion of how NTFS could implement reparse points and encryption on Linux. Feel free to ignore it. ]
Sorry for the reply to my own message, but I forget to mention in the original that the right thing to do for the Encrypted and Reparse bits may be just be to silently discard them and not support the manipulation of these rather esoteric NTFS features through a general API at all.
Signature · 1 line
Nicholas Miell <[email protected]>