silent semantic changes in reiser4 (brief attempt to document the idea of what reiser4 wants to do with metafiles and why
The Idea
You should be able to access metadata about a file the same way you access the file's data, but with a name based on the filename followed by a name to select the metadata of interest.
Examples:
cat song_of_silence/metas/owner
cat song_of_silence/metas/permissions
cat 10 > song_of_silence/metas/mixer_defaults/volume cat song_of_silence/metas/license
But wait! This idea has flaws. Let us list them and address them:
Problem: Often we want to access more than one metafile at a time, and a separate system call for each of them is less efficient than stat or xattrs.
Solution: Create a system call that can access more than one file at a time, whether it is a metafile or a data file should be irrelevant.
Problem: Many kinds of metadata need to be updated atomically.
Solution: Atomic filesystem, which allows all files, including data files, to be updated atomically.
Problem: Many kinds of metadata need to be constrained in their allowed values.
Solution: Allow general purpose constraints to constrain all files, including data files also.
Problem: Many kinds of metadata should not be visible to the user unless he wants to see them.
Solution: Allow all files to be hidden from the user unless he wants to see them.
Example: metas subdirectory is invisible to readdir
Problem: this approach can pollute the namespace
Answer: choose obscure names
Problem (all credit to Mr. Demidov for identifying this problem, I argued the other viewpoint, and I can only claim the wisdom to know that I lost the argument): names like "..metas" are ugly to new users, who don't really care for languages that use punctuation in their keywords.
Answer
don't make them too obscure, experienced namespace developers know that the problem of polluting the namespace is not really as big a deal as beginners think it is, and Clearcase and the WAFL filesystem manage to get by just fine, whereas the problem of putting punctuation marks into names and syntax is a big deal for newbies to the system. Name it "metas" not "..metas", and users will never experience it as a real problem, and newbies will never be annoyed by a-rhythmic punctuation. Note: if Linus disagrees, it is not the most important thing in this design, "..metas" isn't the end of the world.
Problem: Many kinds of metadata, such as acls, need to be inherited.
Solution: Allow all files, metafiles and data files both, to be inherited from other files.
The Idea 2:
When implementing a file plugin, very often there are methods supplied by that plugin that are unique to that plugin. These methods are often well interfaced with as files that can be written to or read from. Allow file plugins to define pseudo files that perform these methods.
Examples:
cat complex_document/pseudos/glued gives an archive of the directory "complex_document" and everything below it, suitable for emailing.
cat filename/pseudos/backup gives a set of commands that a backup command can use to create the file "filename" with all of its state preserved, without the backup program having to understand the plugin's internals. (If there is going to be a proliferation of file plugins, then backup plugin methods are a necessity.
cat password_directory/pseudos/cat gives a concatenation of all the files in the directory "password_directory".
Why Is This Design Important?
It is all about namespace fragmentation. If you can access everything in the OS from the same namespace, the OS is much more powerful.
The expressive power of an OS is not proportional to how many objects are in the OS, but rather to how many different ways you can combine them. Separate namespaces for different types of objects is like having different railroad track gauges for the different peoples of the world.
If there are different namespaces for 2 sets of objects, then the number of possible combinations of the objects is (N/2)^2, which is less than the number of combinations of the objects if they are in one namespace, which is N^2.
Why Openat Sucks:
because you can't go cat filenameA/metas/permissions > filenameB/permissions
If cat doesn't work, then we are suffering exactly the problem with namespace fragmentation that this whole scheme was invented to avoid.
The Origins Of This Idea
I disliked stat() way back in 1984. This is just one small piece of my efforts to build tools to allow OS architects to unify namespaces throughout the OS. DARPA funded this under the CHATS program, and it was one of the main components of my proposal.
Schedule:
One piece at a time, I would like to let this mature as a reiser4 only prototype throughout the 2.7 kernel series, and then have it reviewed for use by other filesystems after that. Let us not rush it, please. I have been studying namespace design since 1984, let someone who is deeply familiar with namespace design principles have a try at a coherent design with all the pieces in place first before committee redesign is attempted. Please consider that there are advantages to letting the guy who had the idea and did the work to make it happen be the maintainer.
My approach is to first produce a working prototype, and only then to try to make a standard. It is not an unusual approach in our community. Let me take that path, it works better for persons like me. I have a half working prototype. A little more time please. After 30 years of Unix file system semantics stagnation, there is not a great rush for this little piece of mild innovation I think.
Conclusion:
Namespace fragmentation is the most important single determinant of OS expressive power, just as trade fragmentation is the most important single determinant of national wealth (see Adam Smith's The Wealth of Nations). Making objects combine easily is what it is all about.
Hans Reiser wrote on Sun, 29 Aug 2004 13:21:44 -0700:
Quoted 1 line
The Idea [...] Conclusion
Good summary!
Quoted 3 lines
The Idea 2: [...]
cat filename/pseudos/backup gives a set of commands that a backup command can use to create the file "filename" [...]
Are you sneaking in file types there? Just how does a file know which plugins it supports?
- Alex
Quoted 17 lines · Alexander G. M. Smith
Hans Reiser wrote on Sun, 29 Aug 2004 13:21:44 -0700:
Quoted 3 lines
The Idea [...] Conclusion
Good summary!
Quoted 5 lines
The Idea 2: [...]
cat filename/pseudos/backup gives a set of commands that a backup command can use to create the file "filename" [...]
Quoted 8 lines
Are you sneaking in file types there? Just how does a file know which plugins it supports?
- Alex
we have plugins with pluginids, is that what you mean by file type? I think they are a bit different from file types.
Hans Reiser wrote on Mon, 30 Aug 2004 23:43:13 -0700:
Quoted 6 lines
Quoted 2 lines · Alexander G. M. Smith
Are you sneaking in file types there? Just how does a file know which plugins it supports?
we have plugins with pluginids, is that what you mean by file type? I think they are a bit different from file types.
From your white-paper: "Every file possesses a plugin id, and every directory possesses a plugin id. This plugin id will identify a set of methods."
Functionally this is very close to a file type. It classifies the files into related groups, maybe not as finely as a MIME file type which can distinguish between multiple varieties of text files.
A file type would tell us that a file is a text file and can be opened by certain applications (text/e-mail can be opened by e-mail reader, text editor, etc) and have other properties (lists of standard attributes, default icon). A plugin ID says that the file can use text related plugins (like a word count, or XML structure as a subdirectory).
In both cases there is a global repository (I assume) that associates the file type or plugin ID with a list of things about it.
You could combine the two concepts, just have a file type ID that in the global repository specifies what plugins it can use as well as the userland properties (MIME string, etc) of that kind of file. Or at least make the type ID available to userland so it can be used there.
Or is this binding irrelevant? How often does the file type the user sees not match the plugins desired for the file? Or can a new subtype be defined for just that file? Which may mean that we need something better than MIME strings for types (something which has inheritance).
- Alex
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Alexander G. M. Smith wrote:
[...]
| A file type would tell us that a file is a text file and can be opened | by certain applications (text/e-mail can be opened by e-mail reader, | text editor, etc) and have other properties (lists of standard | attributes, default icon). A plugin ID says that the file can use text | related plugins (like a word count, or XML structure as a subdirectory). | | In both cases there is a global repository (I assume) that associates | the file type or plugin ID with a list of things about it.
I'm not sure how automatic this is. Standard things like directories, block devices, files, symlinks, and so on are all separate plugins.
Other than that, it doesn't seem so automatic. I have to explicitly turn on encryption, but I can do it to any kind of file or directory -- assuming the cryptocompress plugin worked yet.
[...]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTVIGHgHNmZLgCUhAQKzGg//U3bd6yn4LtArriZY5ldBtEah37MidQMR SZC+jljIdNdwVap2H5m9dC8P4UbN6FLV/3ZVx4K/sel+E35oSM5QWhGyL0DMsTXw M2irNqwDzLQr7iY7tOm0sqmAc1vMHjyk+w50tIQ0Kjbe6n4Rg3e9ZPf9iPvhJxx2 tlYcGlqcoJ8C1hfI0yHHWTZIx63TE4eOTDgZ7n2EkmI+23STOmRTKrPT7QXQPgPV 0eaC+xSUVcF+78KBTfg1j4nq4OVdgNUS0dbyfHqKZBwUn76zcfHz6aVwFKLOaMNH lZ+jzpWYykIFhol/HJOmzO8W7GQfHnpKLNC2rDAlh7eKbCdo7mk2WvyDU/ObSenr fYyhrDwdUG/WKzAa6WySOWuMnABB16mHiUggc2qMcqkyrkCfwq7X13Ryjis0cmCJ mkIKZWglZaHZUbOOBKrXnT9zyd8T1DfmxwJ7eg66xnk0m5gQfMYk0Yg6z0VQ6KYv AmBAkRV70YnC87rVTINN4byqMfTWyySu9EqiKaH0STXFNR/cBA87comIOYp0JPZ8 blCfIR5dIw8e5iPEUhPE623qKT4ljKmiw+xJZcbOczi0dymlXQOagJXE0/NH4IdK vwBgHUhtoxFWxsDeGW4xEF+mE8YLjXJC5AIxvFIdlOv3N27rvm3w1dPwensj5uFC iFCQ80xftW8=
=2vbH
-----END PGP SIGNATURE-----
Quoted 14 lines · Alexander G. M. Smith
Hans Reiser wrote on Mon, 30 Aug 2004 23:43:13 -0700:
Quoted 10 lines
Quoted 4 lines · Alexander G. M. Smith
Are you sneaking in file types there? Just how does a file know which plugins it supports?
we have plugins with pluginids, is that what you mean by file type? I think they are a bit different from file types.
Quoted 6 lines
Quoted 1 line
From your white-paper: "Every file possesses a plugin id, and every
directory possesses a plugin id. This plugin id will identify a set of methods."
Functionally this is very close to a file type.
A file type is based on semantics of the contents, and a plugin id is based on methods of the object.
Quoted 18 lines
It classifies the
files into related groups, maybe not as finely as a MIME file type which can distinguish between multiple varieties of text files.A file type would tell us that a file is a text file and can be opened by certain applications (text/e-mail can be opened by e-mail reader, text editor, etc) and have other properties (lists of standard attributes, default icon). A plugin ID says that the file can use text related plugins (like a word count, or XML structure as a subdirectory).
In both cases there is a global repository (I assume) that associates the file type or plugin ID with a list of things about it.
You could combine the two concepts, just have a file type ID that in the global repository specifies what plugins it can use as well as the userland properties (MIME string, etc) of that kind of file. Or at least make the type ID available to userland so it can be used there.
Quoted 10 lines
Or is this binding irrelevant? How often does the file type the user sees not match the plugins desired for the file? Or can a new subtype be defined for just that file? Which may mean that we need something better than MIME strings for types (something which has inheritance).
- Alex
I think that there may be some good use for a filename/metas/type metafile.
Quoted 16 lines · Hans Reiser
Quoted 14 lines · Alexander G. M. Smith
Hans Reiser wrote on Mon, 30 Aug 2004 23:43:13 -0700:
Quoted 9 lines
Quoted 3 lines · Alexander G. M. Smith
Are you sneaking in file types there? Just how does a file know which plugins it supports?
we have plugins with pluginids, is that what you mean by file type? I think they are a bit different from file types.
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
From your white-paper: "Every file possesses a plugin id, and every
directory possesses a plugin id. This plugin id will identify a set of methods."
Functionally this is very close to a file type.
A file type is based on semantics of the contents, and a plugin id is based on methods of the object.
Today I read this info on an article published by an excellent magazine, that will appears on October (we are Future Technologies :-) , you know)
Hans, like I comment to you time ago, we plan to implement the un-delete for ReiserFS 4, like an external plug-in in 2005.
Thanks,
Giovanni
Quoted 33 lines
Quoted 21 lines
It classifies the
files into related groups, maybe not as finely as a MIME file type which can distinguish between multiple varieties of text files.A file type would tell us that a file is a text file and can be opened by certain applications (text/e-mail can be opened by e-mail reader, text editor, etc) and have other properties (lists of standard attributes, default icon). A plugin ID says that the file can use text related plugins (like a word count, or XML structure as a subdirectory).
In both cases there is a global repository (I assume) that associates the file type or plugin ID with a list of things about it.
You could combine the two concepts, just have a file type ID that in the global repository specifies what plugins it can use as well as the userland properties (MIME string, etc) of that kind of file. Or at least make the type ID available to userland so it can be used there.
Quoted 10 lines
Or is this binding irrelevant? How often does the file type the user sees not match the plugins desired for the file? Or can a new subtype be defined for just that file? Which may mean that we need something better than MIME strings for types (something which has inheritance).
- Alex
I think that there may be some good use for a filename/metas/type metafile.
Signature · 11 lines
-- -- Check FT Websites ... http://www.futuretg.com - ftp://ftp.futuretg.com http://www.FTLinuxCourse.com http://www.FTLinuxCourse.com/Certification http://www.rpmparadaise.org http://GNULinuxUtilities.com http://www.YourPersonalOperatingSystem.com --
Hi!
Quoted 10 lines
Answer: choose obscure names
Problem (all credit to Mr. Demidov for identifying this problem, I argued the other viewpoint, and I can only claim the wisdom to know that I lost the argument): names like "..metas" are ugly to new users, who don't really care for languages that use punctuation in their keywords.
Answer
Quoted 9 lines
don't make them too obscure, experienced namespace developers know that the problem of polluting the namespace is not really as big a deal as beginners think it is, and Clearcase and the WAFL filesystem manage to get by just fine, whereas the problem of putting punctuation marks into names and syntax is a big deal for newbies to the system. Name it "metas" not "..metas", and users will never experience it as a real problem, and newbies will never be annoyed by a-rhythmic punctuation. Note: if Linus disagrees, it is not the most important thing in this design, "..metas" isn't the end of the world.
What about choosing just "..." instead of "metas"? "metas" is string that needs translation etc, while "..." is nicely neutral.
cat /sound_of_silence.mp3/.../author
does not look bad, either...
Pavel
Signature · 2 lines
People were complaining that M$ turns users into beta-testers... ...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Quoted 4 lines · Pavel Machek
What about choosing just "..." instead of "metas"? "metas" is string that needs translation etc, while "..." is nicely neutral.
cat /sound_of_silence.mp3/.../author
Feel free to read these rather lengthy threads:
http://marc.theaimsgroup.com/?l=reiserfs&m=108039830904541&w=2
http://marc.theaimsgroup.com/?l=reiserfs&m=108112390327773&w=2
Signature · 1 line
lg, Chris
Why not a kernel option (accessible through /proc if needed for scripts or the like) ?
Quoted 13 lines · Christian Mayrhuber
Quoted 4 lines · Pavel Machek
What about choosing just "..." instead of "metas"? "metas" is string that needs translation etc, while "..." is nicely neutral.
cat /sound_of_silence.mp3/.../author
Feel free to read these rather lengthy threads:
http://marc.theaimsgroup.com/?l=reiserfs&m=108039830904541&w=2
http://marc.theaimsgroup.com/?l=reiserfs&m=108112390327773&w=2
T�rnqvist
Quoted 2 lines · Mikaël Cluseau
Why not a kernel option (accessible through /proc if needed for scripts or the like) ?
What if there has to be a patch for find so it accesses metadata as well, it seems sort of ugly that it has to check /proc or /sys or somesuch to access the name.
Same thing with the kernel exporting a read-only environment variable.
Hans is pretty stubborn about this and I guess we'll just have to live with some custom-patched or default name until we see progs that are dependant on a single name...
Signature · 1 line
mjt
Quoted 26 lines · Pavel Machek
Hi!
Quoted 10 lines
Answer: choose obscure names
Problem (all credit to Mr. Demidov for identifying this problem, I argued the other viewpoint, and I can only claim the wisdom to know that I lost the argument): names like "..metas" are ugly to new users, who don't really care for languages that use punctuation in their keywords.
Answer
Quoted 11 lines
don't make them too obscure, experienced namespace developers know that the problem of polluting the namespace is not really as big a deal as beginners think it is, and Clearcase and the WAFL filesystem manage to get by just fine, whereas the problem of putting punctuation marks into names and syntax is a big deal for newbies to the system. Name it "metas" not "..metas", and users will never experience it as a real problem, and newbies will never be annoyed by a-rhythmic punctuation. Note: if Linus disagrees, it is not the most important thing in this design, "..metas" isn't the end of the world.
Quoted 9 lines
What about choosing just "..." instead of "metas"? "metas" is string that needs translation etc, while "..." is nicely neutral.
cat /sound_of_silence.mp3/.../author
does not look bad, either...
Pavel
"..." is pretty good, but I think it has been used by others, but I really forget who. I could live with "...", but I think "metas" and "..metas" will collide less often. Apparently Meta is a finnish name or something, so Linus does not like it. The exact string is really not very important to me. I agree that "..." is elegant.
Hans
Hi!
Quoted 5 lines
Quoted 5 lines
What about choosing just "..." instead of "metas"? "metas" is string that needs translation etc, while "..." is nicely neutral.
cat /sound_of_silence.mp3/.../author
Quoted 7 lines
Quoted 1 line
does not look bad, either...
"..." is pretty good, but I think it has been used by others, but I really forget who. I could live with "...", but I think "metas" and "..metas" will collide less often. Apparently Meta is a finnish name or something, so Linus does not like it. The exact string is really not very important to me. I agree that "..." is elegant.
Well, "..." is mostly used by script kiddies -- they usually have their rootkit collection there :-).
It would be nice to decide on one escape into "meta" namespace, uservfs and similar projects probably should be converted to use same escape.
[Uservfs currently uses things like cat /tmp/foo.tgz#utar/bar.gz#ugz ... essentially using # as another separator. It should probably be converted to use same meta escape].
Pavel
Quoted 2 lines · Pavel Machek
Well, "..." is mostly used by script kiddies -- they usually have their rootkit collection there :-).
Wasn't that ".. "? :-)
Romano / Sorry. couldn't resist
Signature · 2 lines
Romano Giannetti - Univ. Pontificia Comillas (Madrid, Spain) Electronic Engineer - phone +34 915 422 800 ext 2416 fax +34 915 596 569
Good day, all,
Quoted 9 lines · Hans Reiser
Quoted 7 lines
What about choosing just "..." instead of "metas"? "metas" is string that needs translation etc, while "..." is nicely neutral.
cat /sound_of_silence.mp3/.../author
does not look bad, either...
"..." is pretty good, but I think it has been used by others, but I really forget who. I could live with "...", but I think "metas" and
Some trojans and rootkits have used it to store files;
/usr/lib/... , /usr/doc/... , /usr/sbin/... , /usr/local/bin/... , /dev/... , and /usr/include/... are used by bobkit.
This might trigger false positives for rootkit detection tools
like chkrootkit and rootkit-hunter.
Cheers,
- Bill
---------------------------------------------------------------------------
"What is the most effective Windows NT remote management tool?
A car."
-- Network Intrusion Detection, An Analyst's Handbook
2nd Edition, 2000
Stephen Northcutt et al, page 147
(Courtesy of Rodrigo Goya <[email protected]>)
--------------------------------------------------------------------------
William Stearns ([email protected]). Mason, Buildkernel, freedups, p0f,
rsync-backup, ssh-keyinstall, dns-check, more at: http://www.stearns.org
--------------------------------------------------------------------------
Quoted 3 lines · William Stearns
somone wrote
Quoted 2 lines
"..." is pretty good, but I think it has been used by others, but I really forget who. I could live with "...", but I think "metas" and
In MS-DOS you can do "cd ..." as a shortcut for ..\.. but only in the cd command. VMS used it with fileneme pattern matching to indicate a recursive search.
Quoted 7 lines
Some trojans and rootkits have used it to store files;
/usr/lib/... ,/usr/doc/... ,/usr/sbin/... ,/usr/local/bin/... ,/dev/... , and/usr/include/... are used by bobkit.
This might trigger false positives for rootkit detection tools
like chkrootkit and rootkit-hunter.
Cheers,
- Bill
Rootkit detection tools need to be updated and improved regularly anyway. If you use old rootkit detection tools, it might be a bonus if you are reminded of that fact. That's a feature, not a bug.
Maybe file/./attribute then. /. on a file is currently meaningless. That does not avoid the unpleasant fact that has been brought up by others (only to be ignored), that the directory syntax does not allow metadata on directories.
I'm not convinced that totally transparent access to meta-data actually benefits anyone. If metadata is that useful (which I believe) it may well be worth fixing those apps that need, and can use them. The rest should just ignore it, even loose it.
There are too many apps that will not work properly with the metadata as file semantics anyway. Implementing something that only works sometimes is not a good idea. A new API would work where you expect it to work, i.e. in new and fixed apps, just like ACL's and EA's. (Ok, the ACL's are effective everywhere, but they aren't copied to new files or even preserved when editing, so emacs and others needs some fixing anyway).
-- robin
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 3 lines · Robin Rosenberg
Maybe file/./attribute then. /. on a file is currently meaningless. That does not avoid the unpleasant fact that has been brought up by others (only to be ignored), that the directory syntax does not allow metadata on directories.
*Not* that I am endorsing the idea of being able to access metadata via a standard pathname --- I continue to believe that named streams are a bad idea that will be an attractive nuisance to application developers, and if we must do them, then Solaris's openat(2) API is the best way to proceed --- HOWEVER, if people are insistent on being able to do this via standard pathnames, and not introducing a new system call, I would suggest /|/ as the separator as the third least worst option. Why?
Any such scheme will violate POSIX and SUS, since we are stealing from the filename namespace, and thus could cause a previously working program to stop working --- however, assuming that we don't care about this, the virtical bar is the least likely to collide with existing file usages, because of its status as a shell meta-character (i.e., pipe). This means that in order to use it on the shell command line, programs will have to quote it:
cat /home/tytso/word.doc/\|/meta/silly-stupid-metadata-or-named-stream
This may seem to be inconvenient, but one very good thing about this is that PHP and existing Perl scripts already already treat pathnames that contain pipes with a certain amount of suspicion --- and this is a good thing! Otherwise, programs that take input from untrusted sources (say, URL's or http form posts), may convert such input into a metadata access, and that may be a very, very, very bad thing. (For example, it may mean that you will have accidentally allowed a web user to read or possibly modify an ACL with whatever privileges of the CGI-perl or php script.) By using a pipe character, it avoids this problem, since secure CGI scripts must be already checking for the pipe character anyway.
Quoted 4 lines
I'm not convinced that totally transparent access to meta-data actually benefits anyone. If metadata is that useful (which I believe) it may well be worth fixing those apps that need, and can use them. The rest should just ignore it, even loose it.
Totally agreed. As I said above, I would prefer openat(2) to trying to do this within a standard pathname, and I would prefer not doing it all since aside from Samba, which is simply trying to maintain backwards compatibility with a Really Bad Idea, the number of protocols and data formats (ftp, tar, zip, gzip, cpio, etc., etc., etc.) that would need to be revamped is huge.
- Ted
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 4 lines · Robin Rosenberg
Quoted 4 lines
Maybe file/./attribute then. /. on a file is currently meaningless. That does not avoid the unpleasant fact that has been brought up by others (only to be ignored), that the directory syntax does not allow metadata on directories.
Quoted 8 lines · Theodore Ts'o
*Not* that I am endorsing the idea of being able to access metadata via a standard pathname --- I continue to believe that named streams are a bad idea that will be an attractive nuisance to application developers, and if we must do them, then Solaris's openat(2) API is the best way to proceed --- HOWEVER, if people are insistent on being able to do this via standard pathnames, and not introducing a new system call, I would suggest /|/ as the separator as the third least worst option. Why?
I believe this debate is counterproductive while there are far more basic and serious issues with reiser4, such as architecture-neutrality of the interpretation of the on-disk format, still pending.
-- wli
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
In reply to Theodore Ts'oQuoted 2 lines · Theodore Ts'o
Any such scheme will violate POSIX and SUS, since we are stealing from the filename namespace,
Not forcibly. POSIX does not give guarantees that everybody can access existing files with arbitrary names. If there is a metafile in a directory which can be looked up using the regular pathname hierarchy but requires certain conditions to be accessed, there is principally nothing wrong with this. It would likely be wrong if accessing the name for a non-existent metafile using one of the POSIX interfaces (e.g. creat()) would result in other than one of the defined actions.
Quoted 1 line
and thus could cause a previously working program to stop working
POSIX holds a barrier against this. It just does not work using the pathname resolution specification alone. It works by defining certain actions for certain file type/system interface combinations. For example, it is demanded that chdir() fails with ENOTDIR if the target name is not a directory. Thus if the target is a regular file, chdir() is required to fail.
If, however, the type of the file in question is neither S_IFREG nor S_IFDIR nor another of the standardized file types, there are no prescriptions about what system interfaces must do on them. POSIX (IEEE Std 1003.1, 2004 Edition) explicitly allows to support non-standard file types in its Base Definitions, 3. 'Definitions', 3.163 'File'.
Also (Base Definitions, 2.1 'Implementation Conformance', 2.1.1 'Requirements'):
# 4. The system may provide additional utilities, functions, or facilities
# not required by IEEE Std 1003.1-2001. Non-standard extensions of the # utilities, functions, or facilities specified in IEEE Std 1003.1-2001 # should be identified as such in the system documentation. Non-standard # extensions, when used, may change the behavior of utilities, functions, # or facilities defined by IEEE Std 1003.1-2001. The conformance document # shall define an environment in which an application can be run with the # behavior specified by IEEE Std 1003.1-2001. In no case shall such an # environment require modification of a Strictly Conforming POSIX # Application (see Strictly Conforming POSIX Application ).
For example, Unix domain sockets were not part of POSIX until 2001, but none of the existing systems violated POSIX by refusing to open() them.
I'm not sure about the results of pathname lookup using the names of such implementation-defined file types with slashes attached, but it would probably be worth to file an interpretation request for this once there is sufficient demand to support it in Linux.
Quoted 7 lines
--- however, assuming that we don't care about
this, the virtical bar is the least likely to collide with existing file usages, because of its status as a shell meta-character (i.e., pipe). This means that in order to use it on the shell command line, programs will have to quote it:cat /home/tytso/word.doc/\|/meta/silly-stupid-metadata-or-named-stream
POSIX certainly requires this to fail with ENOTDIR if 'word.doc' is S_IFREG, but if it is something like S_IFSTR, there might be a realistic chance to have it as an implementation extension without violating POSIX.
Gunnar
Signature · 1 line
http://omnibus.ruf.uni-freiburg.de/~gritter
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
In reply to Theodore Ts'oPutting \ into filenames makes windows compatibility less trivial. Putting | into filenames seems like asking for trouble with shells. Asking users to keep track of multiple levels of escapes imposed by shells and such is hard on them.
If you think \| is user friendly, oh god, people like you are the reason why Unix is hated by many.
Having to explain filename/metas/owner or filename/.../owner or filename/..metas/owner (I don't deeply care what string is used in place of "metas") is hard enough.
All of that said, if "|" was what people preferred, I could live with it.
Stealing from the namespace has a long history behind it (see WAFL, Clearcase, many others), and has never been a real world problem. It is not so bad. If you manage to find a historical case where someone made a mistake in the past, go ahead and name it, but I think moderate caution in such thievery is enough, paranoia is not required. Frankly I think the people who get paranoid about stealing a little bit from the namespace just aren't experienced enough in such matters.
Making an omelette requires breaking eggs. Making a new semantic layer (or adding features to languages generally) requires stealing from the namespace. POSIX is a least common denominator of operating systems, not something for innovators to follow.
Ted, I encourage you to not innovate and stick with POSIX.;-)
(Oh, and yes, I understand that minimizing the cost of change by being artful is desirable.)
Streams are a bad idea. The additional features required to emulate streams using files and directories are interesting though. Putting metafiles in the fs namespace is an increase in closure for the OS, and thus a good thing, because more closure means more connectivity between OS components.
Rather few people understand closure though, so I don't expect to do well in the politics of this. It is a bit like being for free trade, most people will never understand why it is so important because their mental gifts are in other matters, and the notion that people need to be well connected and free to interact is just way too abstract. That it is the single most important determinant of a nation's wealth, oh well.
Namespace connectivity is the single most important determinant of an OS's expressive power.
Hans
Quoted 63 lines · Theodore Ts'o
Quoted 3 lines · Robin Rosenberg
Maybe file/./attribute then. /. on a file is currently meaningless. That does not avoid the unpleasant fact that has been brought up by others (only to be ignored), that the directory syntax does not allow metadata on directories.
*Not* that I am endorsing the idea of being able to access metadata via a standard pathname --- I continue to believe that named streams are a bad idea that will be an attractive nuisance to application developers, and if we must do them, then Solaris's openat(2) API is the best way to proceed --- HOWEVER, if people are insistent on being able to do this via standard pathnames, and not introducing a new system call, I would suggest /|/ as the separator as the third least worst option. Why?
Any such scheme will violate POSIX and SUS, since we are stealing from the filename namespace, and thus could cause a previously working program to stop working --- however, assuming that we don't care about this, the virtical bar is the least likely to collide with existing file usages, because of its status as a shell meta-character (i.e., pipe). This means that in order to use it on the shell command line, programs will have to quote it:
cat /home/tytso/word.doc/\|/meta/silly-stupid-metadata-or-named-stream
This may seem to be inconvenient, but one very good thing about this is that PHP and existing Perl scripts already already treat pathnames that contain pipes with a certain amount of suspicion --- and this is a good thing! Otherwise, programs that take input from untrusted sources (say, URL's or http form posts), may convert such input into a metadata access, and that may be a very, very, very bad thing. (For example, it may mean that you will have accidentally allowed a web user to read or possibly modify an ACL with whatever privileges of the CGI-perl or php script.) By using a pipe character, it avoids this problem, since secure CGI scripts must be already checking for the pipe character anyway.
Quoted 5 lines
I'm not convinced that totally transparent access to meta-data actually benefits anyone. If metadata is that useful (which I believe) it may well be worth fixing those apps that need, and can use them. The rest should just
ignore it, even loose it.Totally agreed. As I said above, I would prefer openat(2) to trying to do this within a standard pathname, and I would prefer not doing it all since aside from Samba, which is simply trying to maintain backwards compatibility with a Really Bad Idea, the number of protocols and data formats (ftp, tar, zip, gzip, cpio, etc., etc., etc.) that would need to be revamped is huge.
- Ted
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 1 line · Hans Reiser
Putting \ into filenames makes windows compatibility less trivial.
Err, I think Ted used \ as an example of how to escape |. It is not part of the filename.
Quoted 1 line
Putting | into filenames seems like asking for trouble with shells.
I think that was Ted's precise reason for arguing that | be used. Did you even read his rationale? :)
Quoted 1 line
If you think \| is user friendly, oh god, people like you are the reason why Unix is hated by many.
I think he was arguing | (not \|) is the least worst seperator to use.
Quoted 1 line
Rather few people understand closure though, so I don't expect to do well in the politics of this. It is a bit like being for free trade, most people will never understand why it is so important because their mental gifts are in other matters,
Lots of people understand why free-trade is important. It's taught in introductory economics/business classes in secondary school.
If you are similarly underestimating the understanding of those who are debating merits of in-name-space streams with you, and overestimating your own understanding, you're not going make progress in convincing people of their merit (if at all).
regards,
Signature · 5 lines
Paul Jakma [email protected] [email protected] Key ID: 64A2FF6A Fortune: The only way to keep your health is to eat what you don't want, drink what you don't like, and do what you'd rather not. -- Mark Twain
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 6 lines · Paul Jakma
Quoted 1 line · Hans Reiser
Putting \ into filenames makes windows compatibility less trivial.
Err, I think Ted used \ as an example of how to escape |. It is not part of the filename.
It is not part of it at one level, but in the shell it is part of it.
Quoted 5 lines
Quoted 2 lines
Putting | into filenames seems like asking for trouble with shells.
I think that was Ted's precise reason for arguing that | be used. Did you even read his rationale? :)
That trouble is desirable? Yes, I can understand why he might not want things to work well.;-)
Quoted 10 lines
Quoted 2 lines
If you think \| is user friendly, oh god, people like you are the reason why Unix is hated by many.
I think he was arguing | (not \|) is the least worst seperator to use.
Quoted 1 line
Rather few people understand closure though, so I don't expect to do well in the politics of this. It is a bit like being for free trade, most people will never understand why it is so important because their mental gifts are in other matters,
Lots of people understand why free-trade is important. It's taught in introductory economics/business classes in secondary school.
Have you looked at the political process at all? Or by lots of people, do you mean a sizable minority?
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 1 line · Hans Reiser
It is not part of it at one level, but in the shell it is part of it.
Just one of many applications. Watch Joe-user save their word processing file sometime, they'll use spaces, quotes, etc.
Quoted 1 line
Have you looked at the political process at all? Or by lots of people, do you mean a sizable minority?
Kernel development does require deep understanding by the majority of computer users. Only kernel developers need deep understanding. ;)
The real question though is: Have you given Al Viro technical answers to his technical questions?
regards,
Signature · 4 lines
Paul Jakma [email protected] [email protected] Key ID: 64A2FF6A Fortune: Malek's Law: Any simple idea will be worded in the most complicated way.
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 6 lines · Paul Jakma
Quoted 1 line · Hans Reiser
It is not part of it at one level, but in the shell it is part of it.
Just one of many applications. Watch Joe-user save their word processing file sometime, they'll use spaces, quotes, etc.
With great unhappiness they will.
Quoted 5 lines
Quoted 2 lines
Have you looked at the political process at all? Or by lots of people, do you mean a sizable minority?
Kernel development does
did you mean to have a "not" here?
Quoted 1 line
require deep understanding by the majority of computer users. Only kernel developers need deep understanding. ;)
What makes you think kernel developers have a deep understanding of the value of connectivity in the OS? They don't. The average kernel developer is not particularly bright. Just ask Ted why htrees are slower than reiser4, or ext2 tail combining is slower, and, well, he has no clue. He is happy to explain how architects don't do real work and should not attend the Linux Kernel Summit, and then when reiser4 blows htrees away he undoubtedly still thinks I just take the credit away from the programmers who do the real work. They did real work, and they are the best in the field, but architecture also matters --- quite a lot actually.
Maybe instead of free trade, I should have used anti-trust laws as my example. The percentage of persons who analytically understand both that free trade is vital and that anti-trust laws are a good thing is very small (and especially small at Harvard Law School). Average people can understand freedom. Understanding that one is not really free to choose to not purchase from a cartel is hard for many. Understanding that free markets are only a first approximation and that is why we need anti-trust laws is beyond perhaps even most economics PhDs.
This is not due to a lack of education. I once had a boss explain to me how many people have trouble understanding orders of magnitude and ratios. He particularly meant his boss, who was having trouble understanding my report.
We all have mental defects, we just have them in different areas from each other. (Forgive me for not enumerating mine....;-) ) Some technical matters are understood by much less than 50% of the population. Closure is one of them. For most people the value of closure can only be understood by using and liking a system that has it, and they are not capable of wanting it in advance during the design stages. Codd understood the importance of closure. You could sense his frustration at being unable to convey it to others in his writings. The search engine industry completely misses the importance of closure.
This is why I just want to be left alone to tinker with reiser4. It is faster than other filesystems. People should assume I know what I am doing, and leave me to tinker in my little fs. 5 years later others will follow, or not, I don't care.
Quoted 2 lines
The real question though is: Have you given Al Viro technical answers to his technical questions?
Yes, I did. Got no response. Would you like me to post something nice and technical to this thread?;-) I can send a summary of my design, and the answers I sent to Viro and Linus.
Quoted 2 lines
regards,
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 4 lines · Hans Reiser
Quoted 2 lines
The real question though is: Have you given Al Viro technical answers to his technical questions?
Yes, I did. Got no response.
Liar.
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 15 lines · [email protected] wrote:
Quoted 6 lines · Hans Reiser
Quoted 3 lines
The real question though is: Have you given Al Viro technical answers to his technical questions?
Yes, I did. Got no response.
Liar.
What was your technical response then?
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 13 lines · Hans Reiser
Quoted 11 lines · [email protected] wrote:
Quoted 7 lines · Hans Reiser
Quoted 4 lines
The real question though is: Have you given Al Viro technical answers to his technical questions?
Yes, I did. Got no response.
Quoted 6 lines
Quoted 5 lines
Liar.
What was your technical response then?
[email protected], written in assumption that the only reply I've got regaring the Message-ID of your "answers" had been correct.
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 33 lines · [email protected] wrote:
Quoted 19 lines · Hans Reiser
Quoted 15 lines · [email protected] wrote:
Quoted 10 lines · Hans Reiser
Quoted 5 lines
The real question though is: Have you given Al Viro technical answers to his technical questions?
Yes, I did. Got no response.
Quoted 10 lines
Quoted 7 lines
Liar.
What was your technical response then?
Quoted 12 lines
[email protected], written in assumption that the only reply I've got regaring the Message-ID of your "answers" had been correct.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Not found in my folder, perhaps you could just forward it.....
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 14 lines · Hans Reiser
Quoted 13 lines · [email protected] wrote:
Quoted 11 lines · Hans Reiser
Quoted 9 lines · [email protected] wrote:
Quoted 5 lines · Hans Reiser
Quoted 3 lines
The real question though is: Have you given Al Viro technical answers to his technical questions?
Yes, I did. Got no response.
Liar.
Quoted 8 lines
Quoted 7 lines
Quoted 1 line
What was your technical response then?
[email protected], written in assumption that the only reply I've got regaring the Message-ID of your "answers" had been correct.
Not found in my folder, perhaps you could just forward it.....
There was a list outage lasting several hours at the height of the reiser4 thread. So, before you start with the name calling, please check the archives to see if your post made it.
Lee
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 3 lines · Lee Revell
There was a list outage lasting several hours at the height of the reiser4 thread. So, before you start with the name calling, please check the archives to see if your post made it.
http://marc.theaimsgroup.com/?l=linux-kernel&m=109463636524917&w=2
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
In reply to [email protected]Quoted 15 lines · [email protected] wrote:
Quoted 6 lines · Hans Reiser
Quoted 3 lines
The real question though is: Have you given Al Viro technical answers to his technical questions?
Yes, I did. Got no response.
Liar.
I don't think that "Liar." is an appropriate response. If you sent a response, just quote it.
I must say that your attitude towards persons contributing to Linux (of which this email is the least of it) has over the years lost Linux persons much more talented than yourself. We lost the opportunity to have one of DARPAs hot young security researchers contribute to us because his experience was that Linux maintainers are a collection of assholes hostile to new contributors, and he had too much self respect to deal with the likes of you. He was a very nice and talented young man. Frankly, I look at Linux, and I see all the reasons why I decided not to develop for BSD coming to life in Linux now that Linux is more successful than BSD. Inner circle, hostility to newcomers, patch acceptance based on whose nose is in whose ass, etc., etc.
Hans
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 1 line · Hans Reiser
I don't think that "Liar." is an appropriate response.
To a bold-faced lie? Yes, it is.
Quoted 2 lines
If you sent a
response, just quote it.
I've already posted Message-Id, but if you prefer a quote, fine, here it is:
============================================================================
Quoted 1 line · Sriram Karra
Perhaps this is one? Message-ID: <[email protected]>
OK...
One note before replying: current code deadlocks even if you make ->link() *ALWAYS* return an error. It doesn't get to calling the method. No amount of "disallow hard links to <something>" is going to help here, obviously.
<quote>
Cycle detection:
We should either 1) make hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks (best short term answer), or 2) ask Alexander Smith to help us with applying his cycle detection algorithm and gain the benefit of being able to hard link to directories (if it works well, best long term answer).
</quote>
... which doesn't address the problem at all. The question is what to do with seeing directory "aspect..." in more than one place when we have many links to file in question. So much for (1). And (2) is not feasible with on-disk fs both due to memory, CPU and IO costs _and_ due to exclusion from hell you'll need to make it safe.
Re: ambiguity - lots and lots of handwaving on both sides. FWIW, I agree with Hans in one (and only one) respect here - openat() as a primary API (and not a convenient libc function) is an atrocity. Simply because it doesn't address operations beyond open (unlinkat(2), anyone?).
However, I still haven't seen any strong arguments for need of this "metas" stuff _or_ the need to export mode/ownership as files, both for regular files and for directories. Aside of "we can do that" [if we solve the locking issues] and "xattrs are atrocious" [yes, they are; it doesn't make alternative mechanism any better] there was nothing that even pretended to be a technical reason.
Note that we also have fun issues with device nodes (Linus' "show partitions" vs. "show metadata from hosting filesystem"), which makes it even more dubious. We also have symlinks to deal with (do they have attributes? where should that be exported?).
Reserved names have one more problem: to be useful, they'd have to be hardcoded into applications. And that will create hell with use of such applications on existing filesystems. Again, no feasible scheme to deal with that in userland code had been proposed so far, AFAICS.
Locking: see above - links to regular files would create directories seen in many places. With all related issues...
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Well I didn't get this response, so whether or not you sent it, it was not a lie. Drink less coffee.
Quoted 42 lines · [email protected] wrote:
Quoted 3 lines · Hans Reiser
I don't think that "Liar." is an appropriate response.
To a bold-faced lie? Yes, it is.
Quoted 3 lines
If you sent a response, just quote it.
I've already posted Message-Id, but if you prefer a quote, fine, here it is:
============================================================================
Quoted 3 lines · Sriram Karra
Perhaps this is one? Message-ID: <[email protected]>
OK...
One note before replying: current code deadlocks even if you make ->link() *ALWAYS* return an error. It doesn't get to calling the method. No amount of "disallow hard links to <something>" is going to help here, obviously.
<quote>
Cycle detection:We should either 1) make hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks (best short term answer), or 2) ask Alexander Smith to help us with applying his cycle detection algorithm and gain the benefit of being able to hard link to directories (if it works well, best long term answer).
</quote>
Quoted 4 lines
... which doesn't address the problem at all. The question is what to do with seeing directory "aspect..." in more than one place when we have many links to file in question.
You don't allow people to see the directory aspect in more than one place.....
Quoted 5 lines
So much for (1). And (2) is not feasible with
on-disk fs both due to memory, CPU and IO costs _and_ due to exclusion from hell you'll need to make it safe.
Your statement regarding 2) is unsupported by technical argument and I think wrong as well.
Quoted 5 lines
Re: ambiguity - lots and lots of handwaving on both sides. FWIW, I agree with Hans in one (and only one) respect here -
openat()as a primary API (and not a convenient libc function) is an atrocity. Simply because it doesn't address operations beyond open (unlinkat(2), anyone?).
Quoted 8 lines
However, I still haven't seen any strong arguments for need of this "metas" stuff _or_ the need to export mode/ownership as files, both for regular files and for directories. Aside of "we can do that" [if we solve the locking issues] and "xattrs are atrocious" [yes, they are; it doesn't make alternative mechanism any better] there was nothing that even pretended to be a technical reason.
Closure is very technical as a reason. It seems to be above your head though. I can say more about it, but not before bed....
Quoted 11 lines
Note that we also have fun issues with device nodes (Linus' "show partitions" vs. "show metadata from hosting filesystem"), which makes it even more dubious. We also have symlinks to deal with (do they have attributes? where should that be exported?).
Reserved names have one more problem: to be useful, they'd have to be hardcoded into applications. And that will create hell with use of such applications on existing filesystems. Again, no feasible scheme to deal with that in userland code had been proposed so far, AFAICS.
How is this different from adding any new feature to any program (library, kernel, fs, daemon) with competitors, that other programs interact with? If you can't cope with change, don't user reiser4..... reiser4 still supports stat(),....
Quoted 3 lines
Locking: see above - links to regular files would create directories seen in many places.
No, it would only be seen from one parent, unless Smith's solution is used.
Quoted 5 lines
With all related issues...
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 7 lines · Hans Reiser
Quoted 7 lines
file-directory duality, and persons who want to link to the directory aspect must use symlinks (best short term answer), or 2) ask Alexander Smith to help us with applying his cycle detection algorithm and gain the benefit of being able to hard link to directories (if it works well, best long term answer).
</quote>
Quoted 6 lines
Quoted 4 lines
... which doesn't address the problem at all. The question is what to do with seeing directory "aspect..." in more than one place when we have many links to file in question.
You don't allow people to see the directory aspect in more than one place.....
And which place would that be? Concrete example: we have 4 links to the same inode in /bin - /bin/gzip, /bin/gunzip, /bin/zcat and /bin/uncompress. What should happen upon attempts to look at the metadata of /bin/gzip and /bin/gunzip simultaneously from completely unrelated processes? Where can these guys be found and in case if you say "whoever looks first wins, another guy gets -EBUSY or some other error" keep in mind that user-triggerable DoS tend to make sysadmins quite unhappy.
Quoted 7 lines
Quoted 5 lines
So much for (1). And (2) is not feasible with
on-disk fs both due to memory, CPU and IO costs _and_ due to exclusion from hell you'll need to make it safe.Your statement regarding 2) is unsupported by technical argument and I think wrong as well.
Uhh... Hans, think for a second - any algorithm would have to be able to tell if adding an edge to graph would create a loop. Consider the following
graph: take two full binary trees of depth N, order their leaves, revert the edges in one of them and for each leaf of the first tree add an edge leading to corresponding leaf of the second one. (IOW, for N=2 you'll get 14 nodes with the following edges: A->A0, A->A1, A0->A00, A0->A01, A1->A10, A1->A11, A00->B00, A01->B01, A10->B10, A11->B11, B00->B0, B01->B0, B10->B1, B11->B1, B0->B, B1->B). Now, give that tree to your algorithm and start asking if adding an edge between given two nodes would add a loop. You'll get exponential time complexity. Moreover, the number of nodes you would have to examine is also exponential.
Note that guy specifically mentioned that his filesystem had been in-core one. With disk-based fs you'll either have to keep the entire graph in-core *or* you will have to walk the damn thing pulling it from disk.
And you need an exclusion against graph modifications while you are looking through it. Have fun...
Quoted 13 lines
Quoted 5 lines
Re: ambiguity - lots and lots of handwaving on both sides. FWIW, I agree with Hans in one (and only one) respect here -
openat()as a primary API (and not a convenient libc function) is an atrocity. Simply because it doesn't address operations beyond open (unlinkat(2), anyone?).Quoted 8 lines
However, I still haven't seen any strong arguments for need of this "metas" stuff _or_ the need to export mode/ownership as files, both for regular files and for directories. Aside of "we can do that" [if we solve the locking issues] and "xattrs are atrocious" [yes, they are; it doesn't make alternative mechanism any better] there was nothing that even pretended to be a technical reason.
Quoted 2 lines
Closure is very technical as a reason. It seems to be above your head though. I can say more about it, but not before bed....
The word "closure" has more than enough meanings, so I am afraid that unless you care to specify what exactly you are talking about it will remain above my head - I am not a telepath.
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
In reply to Hans ReiserQuoted 3 lines · Hans Reiser
Quoted 2 lines
Just one of many applications. Watch Joe-user save their word processing file sometime, they'll use spaces, quotes, etc.
With great unhappiness they will.
Its only problematic for the command line users. The GUI doesn't have some mysterious notion of meta-characters, it provides out of band information on boundaries.
Quoted 4 lines
This is why I just want to be left alone to tinker with reiser4. It is faster than other filesystems. People should assume I know what I am doing, and leave me to tinker in my little fs. 5 years later others will follow, or not, I don't care.
See I don't care if you tinker with reiser4. I don't care if it turns out to be a crap fs or a great fs. If its a great fs and scales and unlike reiser3 can recover well from disk errors then one year I might even use it.
I do care if you ask me to suffer core API changes for your research, that in your economics world is an externality. Its a large negative externality on the part of the userbase so the userbase objects. It doesn't take a PhD in economics to understand this.
Alan
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 6 lines · Alan Cox
Quoted 3 lines · Hans Reiser
Quoted 3 lines
Just one of many applications. Watch Joe-user save their word processing file sometime, they'll use spaces, quotes, etc.
Quoted 9 lines
Quoted 3 lines
With great unhappiness they will.
Its only problematic for the command line users. The GUI doesn't have some mysterious notion of meta-characters, it provides out of band information on boundaries.
Forgive me, what is out of band information on boundaries?
Most people I know don't use the GUI for executing commands, perhaps this is because the existing guis are not good enough yet.
Quoted 12 lines
Quoted 5 lines
This is why I just want to be left alone to tinker with reiser4. It is faster than other filesystems. People should assume I know what I am doing, and leave me to tinker in my little fs. 5 years later others will follow, or not, I don't care.
See I don't care if you tinker with reiser4. I don't care if it turns out to be a crap fs or a great fs. If its a great fs and scales and unlike reiser3 can recover well from disk errors then one year I might even use it.
Is there a technical basis for your claim that we have trouble with disk errors?
Do you mean badblocks support or what?
Quoted 11 lines
I do care if you ask me to suffer core API changes for your research, that in your economics world is an externality. Its a large negative externality on the part of the userbase so the userbase objects. It doesn't take a PhD in economics to understand this.
Alan
I think it would be reasonable for people to say that our approach currently has bugs, we should turn metafiles off until we make the bugs go away.
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 4 lines · Hans Reiser
Is there a technical basis for your claim that we have trouble with disk errors?
Do you mean badblocks support or what?
I mean probability of gettng your data back after a disk loses data. And the technical basis for my claims is twofold - painful experience is one, and shooting random blocks of zeros onto a disk and run fsck tools is another.
Quoted 3 lines
I think it would be reasonable for people to say that our approach currently has bugs, we should turn metafiles off until we make the bugs go away.
Well reiserfs4 is a lot more than metafiles and new vfs layer concepts. Clearly those parts of the the fs that don't require core fs changes belong in the kernel as soon as everyone is happy they are clean enough and look correct.
Metafiles and openat() are an argument we can all have later.
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
In reply to Hans ReiserQuoted 8 lines · Hans Reiser
Quoted 5 lines · Paul Jakma
Quoted 1 line · Hans Reiser
It is not part of it at one level, but in the shell it is part of it.
Just one of many applications. Watch Joe-user save their word processing file sometime, they'll use spaces, quotes, etc.
With great unhappiness they will.
Given the right GUI, they won't even notice.
Quoted 5 lines
Quoted 4 lines
Quoted 2 lines
Have you looked at the political process at all? Or by lots of people, do you mean a sizable minority?
Kernel development does
Quoted 7 lines
did you mean to have a "not" here?
Quoted 2 lines
require deep understanding by the majority of computer users. Only kernel developers need deep understanding. ;)
What makes you think kernel developers have a deep understanding of the value of connectivity in the OS?
Enlighten us. Never heard of "conectivity in the OS" before.
Quoted 2 lines
They don't. The average kernel
developer is not particularly bright.
Right.
Quoted 3 lines
Just ask Ted why htrees are slower
than reiser4, or ext2 tail combining is slower, and, well, he has no clue.
Great PR move, again: Go on publicly insulting the people who you are trying to convince.
Quoted 6 lines
He is happy to explain how architects don't do real work and
should not attend the Linux Kernel Summit, and then when reiser4 blows htrees away he undoubtedly still thinks I just take the credit away from the programmers who do the real work. They did real work, and they are the best in the field, but architecture also matters --- quite a lot actually.
Right you are about architecture.
[...]
Quoted 4 lines
This is why I just want to be left alone to tinker with reiser4. It is faster than other filesystems. People should assume I know what I am doing, and leave me to tinker in my little fs. 5 years later others will follow, or not, I don't care.
Great idea.
Quoted 4 lines
Quoted 2 lines
The real question though is: Have you given Al Viro technical answers to his technical questions?
Yes, I did.
Haven't seen them.
Quoted 2 lines
Got no response. Would you like me to post something nice
and technical to this thread?;-)
That is what this list is supposed to be about: Technical discussion of the Linux kernel development.
Quoted 2 lines
I can send a summary of my design, and
the answers I sent to Viro and Linus.
Please do. Together with the questions, so we can see the whole picture.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
In reply to Hans ReiserQuoted 2 lines · Hans Reiser
Quoted 1 line · Paul Jakma
On Thu, 9 Sep 2004, Hans Reiser wrote:
[...]
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Putting \ into filenames makes windows compatibility less trivial.
Err, I think Ted used \ as an example of how to escape |. It is not part of the filename.
It is not part of it at one level, but in the shell it is part of it.
What are you talking about?
Quoted 6 lines
Quoted 4 lines
Quoted 1 line
Putting | into filenames seems like asking for trouble with shells.
I think that was Ted's precise reason for arguing that | be used. Did you even read his rationale? :)
That trouble is desirable?
He didn't say that at all. He said that '|' in filenames is currently troublesome, so nobody will do it. Besides, '|' is dangerous in some contexts, so it will probably already be filtered out where it might do damage without any further work. This is important, you can't expect everybody rewrite all their applications/web sites just because somebody might be fooling around with Reiser4 to check it out.
Quoted 2 lines
Yes, I can understand why he might not want
things to work well.;-)
It is very clear that you don't even try to understand criticism.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
If you think \| is user friendly, oh god, people like you are the reason why Unix is hated by many.
It isn't. Ted's point is that the alternatives are much worse. Perhaps using "meta" or something like that is friendlier at first sight, but if your box is throroughly 0wn3d as a result, I'm sure your impression of friendliness will radically change.
Quoted 1 line
Quoted 1 line
I think he was arguing | (not \|) is the least worst seperator to use.
Exactly.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Rather few people understand closure though, so I don't expect to do well in the politics of this.
Sorry for you, but either you convince the head kernel hackers (including Ted, BTW) that you are right. And they are very hard to convince, only hope is to present clear, clean solutions to the problems they raise. Nobody has done so in this flamewar, AFAICS. Politics has very little to do with this.
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
It is a bit like being for free trade,
most people will never understand why it is so important because their mental gifts are in other matters,
Here you aren't talking to the counterpart of Joe Sixpack, you are trying to convince the Economics Department of an ivy-league university that your ideas on economics are right and they don't know what they are talking about. Possible, but I'd call it somewhat unlikely.
Quoted 5 lines
Quoted 2 lines
Lots of people understand why free-trade is important. It's taught in introductory economics/business classes in secondary school.
Have you looked at the political process at all? Or by lots of people, do you mean a sizable minority?
Again, no political process in sight.
Please stop the handwawing, and address the core issues: Locking, POSIX compatibility (or convince people that POSIX is wrong), costs of implementing this (no, not just kernel implementation; also new applications, application changes required, user/sysadmin retraining, etc).
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
In reply to Hans ReiserQuoted 11 lines · Hans Reiser
Putting \ into filenames makes windows compatibility less trivial. Putting | into filenames seems like asking for trouble with shells. Asking users to keep track of multiple levels of escapes imposed by shells and such is hard on them.
If you think \| is user friendly, oh god, people like you are the reason why Unix is hated by many.
Having to explain filename/metas/owner or filename/.../owner or filename/..metas/owner (I don't deeply care what string is used in place of "metas") is hard enough.
[Sorry if the tone of this e-mail is particularly heated. I used to use reiserfs and I am really looking towards the insertion of the final version of reiser4 in the linux kernel, but I wish to publish my 2 cents as user (not as kernel developer). TIA for your attention and patience in advance...]
The idea of using "metas" (instead of ".metas" and "..metas") however is ATROCIOUS, as it would mean to have a fine "metas" entry popping up in every dir, be undeletable, and having some random user pissed at his random ${OS} because "it puts stuff that does not belong in the system"...
The main idea about metadata is that you should be able to use metadata only if you care. And if you use something, THEN you need to be bugged by its presence but in a way that does not bug everyone's else... 99% of people out there only care to use metadatas only via those kind of applications that _really_ support it, like Mp3 players, Torrentlike apps and CVS-like programs (if they do and really care to have their metadatas consistent with the reality, that is [1]).
It has no sense for a human being to go and manage thousands of metadata directories by hand, hence having "metas" right there in the filesystem is not calling for user-friendlyness... it is only desperately calling for some n00b# to dd inside it "just for fun" or "just to make it go away". Putting in the spotlight a potentially unreliable OR fragile source of informations is only asking for trouble and is also harrassing for the end-user that couldn't care less about that "metas" stuff. (what is this metas stuff that is between my mariah999.jpg and morena001.jpg in my pr0n/ folder? [2])
In the end, using a UNIX-flavoured system matters because things are shaped to help sysadmin's jobs, and sysadmins are asked for their task to Read The Finely-written (?) Manuals before inserting the root password at the login prompts.
The end-users are to be protected from themselves until they are really ready to read the documentation.
In conclusion: I don't know WAFL, but I won't make Clearcase example a too forward example... Clearcase naming is useful only inside a Clearcase-enabled environment and using Clearcase-enabled tools... I don't think that people at IBM are going to flame me or handle me a DMCA takedown notice if I put on my filesystem filenames that could possibly conflict with their system. Instead Reiserfs4 is going to ask me to sacrifice a viable word for my filenames (no, I don't call my filenames .something), and to create a workaround for my scripts [3] while an alternative like .metas or a ..metas would make even more sense and be more streamlined with Linux/UNIX customs.
Just my 2 cents as random n00b.
Ciao :)
Signature · 5 lines
Davide Inglima [1] http://www.well.com/~doctorow/metacrap.htm [2] whoops... [3] ls -1 | wc -l
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
In reply to Theodore Ts'oQuoted 7 lines · Theodore Ts'o
Quoted 3 lines · Robin Rosenberg
Maybe file/./attribute then. /. on a file is currently meaningless. That does not avoid the unpleasant fact that has been brought up by others (only to be ignored), that the directory syntax does not allow metadata on directories.
Quoted 10 lines
*Not* that I am endorsing the idea of being able to access metadata via a standard pathname --- I continue to believe that named streams are a bad idea that will be an attractive nuisance to application developers, and if we must do them, then Solaris's openat(2) API is the best way to proceed --- HOWEVER, if people are insistent on being able to do this via standard pathnames, and not introducing a new system call, I would suggest /|/ as the separator as the third least worst option. Why?
What's wrong with using / as the separator? It is already used to separate components of pathnames. Named streams are very much like files in a subdirectory.
This scheme makes for very little change to existing tools, users may then do a "gimp somefile/icon.jpg" for example. Or "ls somefile/*" to see all the named streams/forks.
Helge Hafting
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 17 lines · Helge Hafting
Quoted 7 lines · Theodore Ts'o
Quoted 4 lines · Robin Rosenberg
Maybe file/./attribute then. /. on a file is currently meaningless. That does not avoid the unpleasant fact that has been brought up by others (only to be ignored), that the directory syntax does not allow metadata on directories.
Quoted 8 lines
*Not* that I am endorsing the idea of being able to access metadata via a standard pathname --- I continue to believe that named streams are a bad idea that will be an attractive nuisance to application developers, and if we must do them, then Solaris's openat(2) API is the best way to proceed --- HOWEVER, if people are insistent on being able to do this via standard pathnames, and not introducing a new system call, I would suggest /|/ as the separator as the third least worst option. Why?
Quoted 3 lines
What's wrong with using / as the separator? It is already used to separate components of pathnames. Named streams are very much like files in a subdirectory.
/ is separator for directories, POSIX mandates its exact use. No, POSIX isn't broken here, and even if it was, you have to remain compatible.
Quoted 1 line
This scheme makes for very little change to existing tools,
... while breaking fundamental assumptions by all programs in a major way, and no sane solution for legacy applications in sight, with unknown (probably huge) correctness and security implications...
Quoted 2 lines
users may then do a "gimp somefile/icon.jpg" for example. Or "ls somefile/*" to see all the named streams/forks.
Please don't rehash this one. It is fundamentally broken.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
A message that is not in the archive (sent off-list or not yet received): [email protected]
Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why
Quoted 5 lines · Elladan
Quoted 5 lines
What's wrong with using / as the separator? It is already used to separate components of pathnames. Named streams are very much like files in a subdirectory.
Quoted 9 lines
Quoted 5 lines
This scheme makes for very little change to existing tools, users may then do a "gimp somefile/icon.jpg" for example. Or "ls somefile/*" to see all the named streams/forks.
Directories may have metadata as well.
They can. That doesn't stand in the way of using "/" to separate the named stream's name from the file (or directory) that have the attribute. "Directories may have metadata" pops up now and then, and the solution is so blindlingly obvious that nobody sees it.
A file-as-dir can be implemented as a rather normal directory attached to the file's name. The stuff inside may be interpreted as "attributes" or as something more file-like, such as the often mentioned thumbnails.
What about a directory then? It _is_ a directory, so it support named streams already. They are usually called "files". :-) So, if you really want a thumb for your directory, just store a thumb.jpg in it.
Helge Hafting
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 9 lines · Hans Reiser
Why Openat Sucks:
because you can't go cat filenameA/metas/permissions > filenameB/permissions
If cat doesn't work, then we are suffering exactly the problem with namespace fragmentation that this whole scheme was invented to avoid.
Sure you can. All you need to do is fix your shell.
- --
Mike Waychison
Sun Microsystems, Inc.
1 (650) 352-5299 voice
1 (416) 202-8336 voice
http://www.sun.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE: The opinions expressed in this email are held by me, and may not represent the views of Sun Microsystems, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
iD8DBQFBNIavdQs4kOxk3/MRAmvAAJ9sZfEgIOfINZsT23pa+Y3nQ4lfMACgmN3q sdgPzBUg9i/c+roWDrXq5BI=
=poSJ
-----END PGP SIGNATURE-----
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Mike" == Mike Waychison <[email protected]> writes:
Quoted 8 lines · Mike> Hans Reiser wrote:
Quoted 1 line
Why Openat Sucks:
Quoted 6 lines
because you can't go cat filenameA/metas/permissions > filenameB/permissions
If cat doesn't work, then we are suffering exactly the problem with namespace fragmentation that this whole scheme was invented to avoid.
Mike> Sure you can. All you need to do is fix your shell.
And you'll have to fix every other program that you want to use to edit metadata/attributes/streams.
The whole point of avoiding namespace fragmentation is to avoid having to fix programs.
Like I said in another post, the world is bigger than just the command line. The command line just gives really easy examples of why Hans (and others) thinks about things the way he does.
Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Quoted 6 lines · Hans Reiser
The Idea
You should be able to access metadata about a file the same way you access the file's data, but with a name based on the filename followed by a name to select the metadata of interest.
Quoted 6 lines
Examples:
cat song_of_silence/metas/owner
cat song_of_silence/metas/permissions
cat 10 > song_of_silence/metas/mixer_defaults/volume cat song_of_silence/metas/license
Maybe I'm crazy but:
You're talking about a major change in the way filesystems work if this is
going to be used by other FSs too. If I understand this correctly it is a completely new thing and trying to do it by patching existing well-known 'primitives' may be wrong.
AFAIK and AFAICS the metadata are not files or directories. You can look at
them as files/dirs but they are not, just like a tar is not a directory. I believe that the correct thing to do (tm) is to add a new 'concept' named 'metadata' (which already exists). This way you'll have files, directories and metadata (or whatever you call them). So, each directory can have metadatas and files and each file can have metadatas. Then you have to provide some new methods of accessing them and not to use chdir() etc. (lets say chdir_meta() to enter the meta dir which will work for files too). After entering the 'metadir' you'll be able to use existing methods etc to access its 'files'.
This approach doesn't mess with existing things and can be extended for
other filesystems too.
(Just a thought)
<<V13>>
Quoted 35 lines · Hans Reiser
Quoted 6 lines
The Idea
You should be able to access metadata about a file the same way you access the file's data, but with a name based on the filename followed by a name to select the metadata of interest.
Quoted 6 lines
Examples:
cat song_of_silence/metas/owner
cat song_of_silence/metas/permissions
cat 10 > song_of_silence/metas/mixer_defaults/volume cat song_of_silence/metas/licenseMaybe I'm crazy but:
You're talking about a major change in the way filesystems work if this is
going to be used by other FSs too. If I understand this correctly it is a completely new thing and trying to do it by patching existing well-known 'primitives' may be wrong.AFAIK and AFAICS the metadata are not files or directories. You can look at
them as files/dirs but they are not, just like a tar is not a directory. I believe that the correct thing to do (tm) is to add a new 'concept' named 'metadata' (which already exists). This way you'll have files, directories and metadata (or whatever you call them). So, each directory can have metadatas and files and each file can have metadatas. Then you have to provide some new methods of accessing them and not to usechdir()etc. (lets saychdir_meta()to enter the meta dir which will work for files too). After entering the 'metadir' you'll be able to use existing methods etc to access its 'files'.
Quoted 4 lines
This approach doesn't mess with existing things and can be extended for
other filesystems too.(Just a thought)
It is a good thought. However I think they are trying to figure out a way to have the metadata and streams to be accesible with legacy applications.
The file-as-directory concept is one way, which still seem to have issues.
How are things done on Windows platforms when there are files and directories with the same name? In Unix that is imposible. How does it work for environments like Cygwin etc? What happen to tools that run in them?
Quoted 1 line
<<V13>>
Salut,
Quoted 4 lines · Spam
How are things done on Windows platforms when there are files and directories with the same name? In Unix that is imposible. How does it work for environments like Cygwin etc? What happen to tools that run in them?
In NTFS it's illegal IIRC. At least the fs correction utilities complain about a block being assigned to two files.
Same with HFS+.
Sometimes there seem to be several things with the same name. But that's because of hidden extensions (.lnk for example).
I'm talking out of the book here, maybe the real-world implementations of Windows are different. I can't tell, I only used Windows once to ssh into a screwed-up router.
Tonnerre
Quoted 5 lines
Salut,
Quoted 8 lines
Quoted 4 lines · Spam
How are things done on Windows platforms when there are files and directories with the same name? In Unix that is imposible. How does it work for environments like Cygwin etc? What happen to tools that run in them?
In NTFS it's illegal IIRC. At least the fs correction utilities complain about a block being assigned to two files.
I meant a file and a directory with the same name, not two files with the same name :) subtle but important difference.
ie, you can have a file named "foo" and a directory named "foo" and they won't collide.
Quoted 10 lines
Same with HFS+.
Sometimes there seem to be several things with the same name. But that's because of hidden extensions (.lnk for example).
I'm talking out of the book here, maybe the real-world implementations of Windows are different. I can't tell, I only used Windows once to ssh into a screwed-up router.
Tonnerre
Quoted 8 lines · Spam
Quoted 8 lines
Salut,
Quoted 4 lines · Spam
How are things done on Windows platforms when there are files and directories with the same name? In Unix that is imposible. How does it work for environments like Cygwin etc? What happen to tools that run in them?
Quoted 8 lines
Quoted 2 lines
In NTFS it's illegal IIRC. At least the fs correction utilities complain about a block being assigned to two files.
I meant a file and a directory with the same name, not two files with the same name :) subtle but important difference.
ie, you can have a file named "foo" and a directory named "foo" and they won't collide.
You can't have a file and a directory with the same name in W*.
Alternative data streams don't appear in a normal directory. You need special API:s and tools to see them. If you know the name of and ADS you can however access them with standard tools, including cygwin.
echo foo >a.txt
echo bar >a.txt:b.txt
You can also have ADS's on directories. Not sure how Reiserfs4 does that.
mkdir foo
echo bar >foo:a.txt
-- robin
Quoted 24 lines · Robin Rosenberg
Quoted 8 lines · Spam
Quoted 8 lines
Salut,
Quoted 4 lines · Spam
How are things done on Windows platforms when there are files and directories with the same name? In Unix that is imposible. How does it work for environments like Cygwin etc? What happen to tools that run in them?
Quoted 8 lines
Quoted 2 lines
In NTFS it's illegal IIRC. At least the fs correction utilities complain about a block being assigned to two files.
I meant a file and a directory with the same name, not two files with the same name :) subtle but important difference.
ie, you can have a file named "foo" and a directory named "foo" and they won't collide.
You can't have a file and a directory with the same name in W*.
Alternative data streams don't appear in a normal directory. You need special API:s and tools to see them. If you know the name of and ADS you can however access them with standard tools, including cygwin.
Quoted 7 lines
echo foo >a.txt
echo bar >a.txt:b.txtYou can also have ADS's on directories. Not sure how Reiserfs4 does that.
mkdir foo
echo bar >foo:a.txt
The problem with this approach is that some tools will not recognize the file:ADS syntax, and complain. This is what happens under Windows, at least. Would the same happen under Linux, I wonder?
Say that we choose a reserved character (say, #; is # a reserved character?) to mark the frontier between the file and the ADS/attributes/whatever. Would old programs still be able to parse the (commandline or anywhere else) "URL" specification and get to the ADS/attribute/whatever?
Say, would cat somefile#an_ads work, or would cat (as it is now) complain in some way?
Signature · 7 lines
Giuseppe "Oblomov" Bilotta
Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
(Roger Waters)Quoted 32 lines · Spam
Quoted 32 lines · Hans Reiser
Quoted 6 lines
The Idea
You should be able to access metadata about a file the same way you access the file's data, but with a name based on the filename followed by a name to select the metadata of interest.
Quoted 6 lines
Examples:
cat song_of_silence/metas/owner
cat song_of_silence/metas/permissions
cat 10 > song_of_silence/metas/mixer_defaults/volume cat song_of_silence/metas/licenseMaybe I'm crazy but:
You're talking about a major change in the way filesystems work if this
is going to be used by other FSs too. If I understand this correctly it is a completely new thing and trying to do it by patching existing well-known 'primitives' may be wrong.AFAIK and AFAICS the metadata are not files or directories. You can
look at them as files/dirs but they are not, just like a tar is not a directory. I believe that the correct thing to do (tm) is to add a new 'concept' named 'metadata' (which already exists). This way you'll have files, directories and metadata (or whatever you call them). So, each directory can have metadatas and files and each file can have metadatas. Then you have to provide some new methods of accessing them and not to usechdir()etc. (lets saychdir_meta()to enter the meta dir which will work for files too). After entering the 'metadir' you'll be able to use existing methods etc to access its 'files'.
Quoted 8 lines
Quoted 4 lines
This approach doesn't mess with existing things and can be extended for
other filesystems too.(Just a thought)
It is a good thought. However I think they are trying to figure out a way to have the metadata and streams to be accesible with legacy applications.
They will be since after chdir_meta() the user will be able to look at the metadata just like Hans described it. The only thing that changes (from the userland POV) is the way someone can enter the 'metadata directory'. This way you don't have to have a special name, just a special function and no existing application (like tar) can possibly break because it will not know how to enter this 'metadata directory'.
<<V13>>
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"V13" == V13 <[email protected]> writes:
Quoted 3 lines · Spam
Quoted 3 lines
It is a good thought. However I think they are trying to figure out a way to have the metadata and streams to be accesible with legacy applications.
V13> They will be since after chdir_meta() the user will be able to look V13> at the metadata just like Hans described it.
Uh, no. Legacy means no changes made. It shouldn't need to be modified to use chdir_meta.
From what I can see, your proposal isn't much different from the openat proposal.
V13> The only thing that changes (from the userland POV) is the way V13> someone can enter the 'metadata directory'. This way you don't have V13> to have a special name, just a special function and no existing V13> application (like tar) can possibly break because it will not know V13> how to enter this 'metadata directory'.
tar won't be able to backup the metadata. That's the major breakage of tar that we're worried about.
Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
V13>> The only thing that changes (from the userland POV) is the way V13>> someone can enter the 'metadata directory'. This way you don't have V13>> to have a special name, just a special function and no existing V13>> application (like tar) can possibly break because it will not know V13>> how to enter this 'metadata directory'.
Quoted 2 lines
tar won't be able to backup the metadata. That's the major breakage of tar that we're worried about.
However, if we do a "cp fileA fileB" then the metadata and streams ought to be copied too, even if "cp" does not support them. This is the real challenge. Backup tools like tar can be patched just like it has so many times before.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Spam" == Spam <[email protected]> writes:
V13> The only thing that changes (from the userland POV) is the way V13> someone can enter the 'metadata directory'. This way you don't have V13> to have a special name, just a special function and no existing V13> application (like tar) can possibly break because it will not know V13> how to enter this 'metadata directory'.
Quoted 2 lines
Quoted 2 lines
tar won't be able to backup the metadata. That's the major breakage of tar that we're worried about.
Spam> However, if we do a "cp fileA fileB" then the metadata and
Spam> streams ought to be copied too, even if "cp" does not support Spam> them.
Huh? How do you plan on pulling that off? Unless cp uses the not-yet-existing copy syscall, if cp can't see the metadata or streams, how is it going to copy it?
Spam> This is the real challenge. Backup tools like tar can be patched Spam> just like it has so many times before.
Yes. And if we can get file-as-dir, then we only need to patch tar once, since everything can be exported through that interface.
Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
"Spam" == Spam <[email protected]> writes:
V13>> The only thing that changes (from the userland POV) is the way V13>> someone can enter the 'metadata directory'. This way you don't have V13>> to have a special name, just a special function and no existing V13>> application (like tar) can possibly break because it will not know V13>> how to enter this 'metadata directory'.
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
tar won't be able to backup the metadata. That's the major breakage of tar that we're worried about.
Spam>> However, if we do a "cp fileA fileB" then the metadata and
Spam>> streams ought to be copied too, even if "cp" does not support Spam>> them.
Quoted 3 lines
Huh? How do you plan on pulling that off? Unless cp uses the not-yet-existing copy syscall, if cp can't see the metadata or streams, how is it going to copy it?
My first thought would change the API that cp uses to copy the file. But in an earlier response on this message thread I have found out that there is no such API (well there is, but it is linked into the program) and cp in fact itself is doing the copying. this is also what I objected against before. It is a bad design and should be attended much higher interest to change than just adding adding a new type of file-as-dir.
Spam>> This is the real challenge. Backup tools like tar can be patched Spam>> just like it has so many times before.
Quoted 2 lines
Yes. And if we can get file-as-dir, then we only need to patch tar once, since everything can be exported through that interface.
Yes. This seem to be an acceptable way to do things. But next time someone comes and want to do changes like this we need to start patching things again. If there was an API that was separate from the programs then new features could be included much more easily as things could be done behind the scenes. ie the "cp fileA fileB" would succeed and all the extended attributes, metas, streams etc would be copied too. Nothing would ever be lost unless copying to a filesystem that doesn't support the special stuff. (as with NTFS file streams).
~S
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Spam" == Spam <[email protected]> writes:
Yes. And if we can get file-as-dir, then we only need to patch tar once, since everything can be exported through that interface.
Spam> Yes. This seem to be an acceptable way to do things. But next
Spam> time someone comes and want to do changes like this we need to Spam> start patching things again. ...
But that's my point. Anything that I can imagine anyone ever wanting to extend the filesystem to do can be exported through this file-as-dir system. ACLs (or insert your favourite access-control scheme here), xattrs, streams. If you want to add something new, just make sure that it can be seen through the file-as-dir interface, and you don't need to patch tar again. Want to use mandatory access control instead of acl's? Just export through foo.txt/metas/mac_level, and tar will catch it with no problems. Want to let your streams have substreams too? export through foo.txt/streams/bar/streams/baz, and tar will still catch it.
And you get the added bonus that you can use any program to edit these extra things, no modifications needed.
Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"V13" == V13 <[email protected]> writes:
V13> Maybe I'm crazy but:
V13> You're talking about a major change in the way filesystems work if V13> this is going to be used by other FSs too. If I understand this V13> correctly it is a completely new thing and trying to do it by V13> patching existing well-known 'primitives' may be wrong.
V13> AFAIK and AFAICS the metadata are not files or directories. You
V13> can look at them as files/dirs but they are not, just like a tar is V13> not a directory. I believe that the correct thing to do (tm) is to V13> add a new 'concept' named 'metadata' (which already exists). ...
And you'll need a new 'concept' named 'streams'. And a new concept for every extension you want/need to add to the filesystem. By exposing everything through the normal filesystem interface, you don't need to make any modifications to your programs to use the new concepts. Everything just works as expected. (With the possible exception of backup-like software, but it is easier to fix just the backup software than every other program that's out there.)
The hypothetical object /usr/bin/xterm/icon.png looks like a file, smells like a file, and quacks like a file. Why shouldn't it be a file?
V13> This way you'll have files, directories and metadata (or whatever V13> you call them). So, each directory can have metadatas and files and V13> each file can have metadatas. Then you have to provide some new V13> methods of accessing them and not to use chdir() etc. (lets say V13> chdir_meta() to enter the meta dir which will work for files V13> too). After entering the 'metadir' you'll be able to use existing V13> methods etc to access its 'files'.
V13> This approach doesn't mess with existing things and can be
V13> extended for other filesystems too.
Either way, filesystems that want to implement that interface will need to be changed. The question is just what kind of interface we will expose to the users/programs. And since that's what the question is, it seems to me that we want to expose it through an interface that the users are already used to.
Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
On Tue, 2004-08-31 at 18:55, V13 wrote:
...
Quoted 10 lines
AFAIK and AFAICS the metadata are not files or directories. You can look at
them as files/dirs but they are not, just like a tar is not a directory. I believe that the correct thing to do (tm) is to add a new 'concept' named 'metadata' (which already exists). This way you'll have files, directories and metadata (or whatever you call them). So, each directory can have metadatas and files and each file can have metadatas. Then you have to provide some new methods of accessing them and not to usechdir()etc. (lets saychdir_meta()to enter the meta dir which will work for files too). After entering the 'metadir' you'll be able to use existing methods etc to access its 'files'.
I think this is what Hans is trying to avoid -- it results in creating a separate namespace for the metadata.
I would agree with the notion that metadata nodes are light-weight files in that they themselves have no metadata associated with them, but I don't see why they need separate system-calls (if we can help it).
Chris Dawes.
Quoted 7 lines
This approach doesn't mess with existing things and can be extended for
other filesystems too.(Just a thought)
<<V13>>
Quoted 4 lines · V13 wrote:
AFAIK and AFAICS the metadata are not files or directories.
Yes they are. In reiser4. They might be stored different, but their interface is what counts.