Skip to contentSkip to conversation

silent semantic changes with reiser4

DISCUSSION Christoph Hellwig 141 participants 1,073 messages Aug 24, 2004 – Sep 17, 2004 7 messages not in the archive VFS Page 2 of 5

lore
Share & tools
lore
#271 lore raw

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

In reply to Frank van Maarseveen
Quoted 6 lines · Frank van Maarseveen
Quoted 3 lines · Tonnerre
$ cat fs_header owner_root flags_with_suid evil_program > evil.iso
$ ls -l evil.iso/evil_program

It should of course be equivalent to a user mount: nodev nosuid etc.

hmm, sounds reasonable, but what if root accesses it? (or somebody with the 'right' capability)

- it might be strange if even root is not able to open device nodes or execute files from an archive

- it might lead to interesting situations if the archive is opened by root, but accessed by an user (thinking of caches and such)

best,
Herbert

Quoted 6 lines
Signature · 5 lines
Frank
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#272 lore raw

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

Quoted 5 lines · Herbert Poetzl

hmm, sounds reasonable, but what if root accesses it? (or somebody with the 'right' capability)

- it might be strange if even root is not able to open device nodes or execute files from an archive

Yes, but if the file is owned by or writable for non-root then you've got a security problem. So, unless owned by root and not writable (readable, executable?) for anyone else "nodev" and 'nosuid" are mandatory.

Quoted 4 lines

- it might lead to interesting situations if the archive is opened by root, but accessed by an user (thinking of caches and such)

See the above.
Alternatively, each process could have its own vfsmount (please don't shoot me for suggesting this ;-)

Signature · 1 line
Frank
#273 lore raw

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

In reply to Linus Torvalds

Hi!

Quoted 3 lines

I bet you could write a small library to test this out for a few types. See if it's useful to you. And only if it's useful (and would make a huge performance difference) would it be worth putting in the kernel.

It seemed really usefull in uservfs incarnation. Unfortunately the daemon was not multihreaded at that time, so it was not really usefull on multiuser systems :-(

Quoted 4 lines

Implementation of the _user_space_ library would be something like this:

	#define MAXNAME 1024
	int open_cached_view(int base_fd, char *type, char *subname)

Well, you'd need more than simple open. For caching tar (etc), you'd need stat_cached_view and opendir_cached_view and ...

And this really works, only that its called mc_open(), mc_stat() etc.

Gnome actually uses newer incarnation of mc_open etc, but they had to introduce rather ugly interface to make it asynchronous.
Pavel

Signature · 1 line
When do you have heart between your knees?
#274 lore raw

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

In reply to Linus Torvalds

Salut,

Quoted 29 lines · Linus Torvalds
	#define MAXNAME 1024
	int open_cached_view(int base_fd, char *type, char *subname)
	{
	        struct stat st;
	        char filename[PATH_MAX];
	        char name[MAXNAME];
	        int len, cachefd;
	
	        if (fstat(base_fd, &st) < 0)
	                return -1;
	        sprintf(name, "/proc/self/fd/%d", base_fd);
	        len = readlink(name, filename, sizeof(filename)-1);
	        if (len < 0)
	                return -1;
	        filename[len] = 0;
	
	        /* FIXME! Replace '/' with '#' in "type" and "subname" */
	        len = snprintf(name, sizeof(name),
	                        "%04llx/%04llx/%s/%s/%s",
	                        (unsigned long long) st.st_dev,
	                        (unsigned long long) st.st_ino,
	                        type ? : "default",
	                        subname,
	                        filename);
	        errno = ENAMETOOLONG;
	        if (len >= sizeof(name))
	                return -1;
	        cachefd = open(name, O_RDONLY);
	        if (cachefd >= 0) {
Quoted 8 lines
	                /* Check mtime here - maybe we could have kernel support */
	                return cachefd;
	        }
	        if (errno != ENOENT)
	                return -1;
	        /*
	        .. try to generate cache file here ..
	         */

Around of what I've had in mind. Only that one might use libmagic instead of the type argument. The rest can be done by a corresponding MIME plugin.

Tonnerre

#275 lore raw

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

In reply to Jamie Lokier
Quoted 2 lines · Jamie Lokier

I'm going to explain why filesystem support for .tar.gz or other "document container"-like formats is useful.

Nobody disagrees there (I think), the disagreement is on whether the usefullness is worth the hassle.

Quoted 4 lines

This does _not_ mean tar
in the kernel (I know someone who can't read will think that if I don't say it isn't); it does mean hooks in the kernel for maintaining coherency between different views and filesystem support for cacheing.

"Coherency" and "different views" implies atomic transactions, and being able to tell that an access to the file requieres updating random junk about it. It requires being able to guess if it is worth updating now (given that the file might be modified a dozen times more before the junk being checked).

Quoted 4 lines

The vision I'm going for here is:

1. OpenOffice and similar programs store compound documents in standard file formats, such as .tar.gz, compressed XML and such.

And they are doing fine AFAICS. Besides, they won't exactly jump on the possibility of leaving behind all other OSes on which they run to become a Linux-only format.

Quoted 2 lines

Fs support can reduce CPU time handling these sorts of files, as I explain below, while still working with the standard file formats.

I don't buy this one. A tar.gz must be uncompressed and unpacked, and whatever you could save is surely dwarfed by those costs.

Quoted 3 lines

With appropriate userspace support, programs can be written which have access to the capabilities on all platforms, but reduced CPU time occurs only on platforms with the fs support.

Userspace support isn't there on any of the platforms right now, if ever it will be a strange-Linux-installation thing for quite some time to come. Not exactly attractive for application writers.

Quoted 1 line

2. Real-time indexing and local search engine tools.

Sure! Gimme the CPU power and disk throughput for that, pretty please. [No, I won't tell I have better use for those right now...]

Quoted 6 lines

This isn't
just things like local Google; it's also your MP3 player scanning for titles & artists, your email program scanning for subject lines to display the summary fast, your blog server caching built pages, your programming environment scanning for tags, and your file transfer program scanning for shared deltas to reduce bandwidth.

With no description on how this is supposed to work, this is pure science fiction/wet dreams.

Quoted 4 lines

I won't explain how these work as it would make this mail too long. It should be clear to anyone who thinks about it why the coherency mechanism is essential for real-time, and a consistent interface to container internals helps with performance.

Coherency is essential, but it isn't free. Far from it. The easiest way of getting coherency is having _one_ authoritative source. That way you don't need coherency, and don't pay for it. Anything in this class must by force be just hints, to be recomputed at a moment's notice. I.e., let the application who might use it check and recompute as needed.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#276 lore raw

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

Quoted 6 lines · Horst von Brand
Quoted 4 lines

This does _not_ mean tar
in the kernel (I know someone who can't read will think that if I don't say it isn't); it does mean hooks in the kernel for maintaining coherency between different views and filesystem support for cacheing.

"Coherency" and "different views" implies atomic transactions,

Wrong. They imply ordering.

Quoted 2 lines

and being able to tell that an access to the file requieres updating random junk about it.

Correct.

Quoted 3 lines

It requires being able to guess if it is worth updating now (given that the file might be modified a dozen times more before the junk being checked).

Wrong. Lazy and deterministic are compatible.

Quoted 9 lines
Quoted 5 lines

The vision I'm going for here is:

1. OpenOffice and similar programs store compound documents in standard file formats, such as .tar.gz, compressed XML and such.

And they are doing fine AFAICS. Besides, they won't exactly jump on the possibility of leaving behind all other OSes on which they run to become a Linux-only format.

You replied before reading the rest of the mail again, didn't you? The whole purpose of the idea is to work with formats which are _not_ Linux-only.

Quoted 5 lines
Quoted 2 lines

Fs support can reduce CPU time handling these sorts of files, as I explain below, while still working with the standard file formats.

I don't buy this one. A tar.gz must be uncompressed and unpacked, and whatever you could save is surely dwarfed by those costs.

Didn't they explain what "lazy" means in your computer science class? Google for "lazy evaluation".

In this context it means _unnecessary_ uncomressions and compressions, i.e. repeated ones, or ones where you never really needed the compressed form, are eliminated. Necessary ones are of course done.

Those are things which are currently wasting your CPU because apps cannot eliminate them: to do so implies synchronisation between different apps running at different times. Surprise surprise, that's something filesystems and kernels are good for.

Quoted 7 lines
Quoted 3 lines

With appropriate userspace support, programs can be written which have access to the capabilities on all platforms, but reduced CPU time occurs only on platforms with the fs support.

Userspace support isn't there on any of the platforms right now, if ever it will be a strange-Linux-installation thing for quite some time to come. Not exactly attractive for application writers.

Again, you have a strange vision of what userspace support means. I say it means a portable and simple library for accessing components inside a compressed container file in a standard format.

Something I believe OpenOffice et al. already has - so to say it doesn't exist is both incorrect and missing the point. The point is to change that library which already exists, so it uses the filesystem facility when available. And to make the library better in standalone form, so that other tools which manipulate container-like formats are inclined to work with it as plugins, instead of creating their own interface as cli tools.

_At no point is portability an issue_ - that library is already portable to everything - and it's certainly not a strange-Linux thing nor meant to become one.

Quoted 4 lines
Quoted 1 line

2. Real-time indexing and local search engine tools.

Sure! Gimme the CPU power and disk throughput for that, pretty please. [No, I won't tell I have better use for those right now...]

You still haven't grasped the idea of an algorithm which is more complex but reduces CPU time and disk throughput, have you?

*Today*, when I open Rhythmbox I have to wait 5 minutes while it wastes my CPU power and disk throughput scanning all the files in my Music directory.

The entire point of these indexing schemes is so that programs like Rhythmbox will display the same data without having to scan the disk for 5 minutes every time they start up. That's a near infinite _saving_ of CPU power and time.

If you don't see that I'm just going to have to suggest you google for "cache" and learn a bit about them.

Quoted 9 lines
Quoted 6 lines

This isn't
just things like local Google; it's also your MP3 player scanning for titles & artists, your email program scanning for subject lines to display the summary fast, your blog server caching built pages, your programming environment scanning for tags, and your file transfer program scanning for shared deltas to reduce bandwidth.

With no description on how this is supposed to work, this is pure science fiction/wet dreams.

Sigh. If you must continue to hurl your blunt instruments around, here is a description. I didn't want to write this becuase it is off topic, and you insist on not understanding the basic of algorithm complexity and CPU usage, so you'll probably not understand this either, but I'll give it a go.

1. Local Google (by which I mean a search engine on your local machine), Real-time (by which I mean the results are always up to date):

Every file modified since last search is known to the query engine. This is a reality: BeOS does it; WinFS is expected to do it.

We know that it's possible to update free text indexes with small amounts of known changes quickly, at the time of a query.

Thus we have real-time local free text search engine, and other features like searching inside files and for file names. The point is the real-time nature of it: the results you get correspond to exactly the contents of the filesystem at the time of the query (writes which occur _during_ a query are obviously not coherent with this, but writes which complete before the query, even immediately before, appear in the results).

Note that file write timing need not be affected - all the work can happen during queries (although it is best to have a nightly re-index or whatever so that the delay at query times is kept moderate).

2. MP3 player scanning artists & titles:

Easy. MP3 player does what it does, shows the extracted ID tags from all .mp3 files in your home directory. They do this already! The difference is that with fs coherency hooks, they can _store_ that ID information, retrieve it later without having to scan all the .mp3s again (see Rhythmbox earlier), and keep their lists updated on the screen as soon as any .mp3 is changed or even any new ones created anywhere.

Technically it's a simpler subset of real-time queries in 1.

3. Email program scanning for subject lines fast:

See Evolution; the only difference is stat() on a thousand files won't be required, it'll be exposed through a standard query instead of an Evolution-only cache method so other mail programs may use it as well as shell commands, and you can pretend you have mbox instead of maildir (mbox is a container format...).

4. Blog server caching built pages:

It's a reality already, obviously. The difference is it'll make sense to built the pages through an indexed query on files instead of a database, e.g. one file per article, and (independent of the first point) the process of building can keep track of all the prerequisite files and scripts and templates used to produce the page, and actually expect to know, _with coherent accuracy_, if any of those prerequisites is different the next time the page is needed.

In other words, complex script-generated web pages, cached output, and with no overhead when serving each page to check prerequisites (i.e. no 100 stat() calls), because we know the filesystem would have told us if any prerequisite had changed prior to the moment we begin serving the cached output.

You can do something close to this already with dnotify (if you ignore that it doesn't tell you about new hard links, which is a dnotify bug), although doing the query part is unrealistic with dnotify and stat() prerequisites unless the directory names are structured thoughtfully.

5. Programming environment scanning for tags:

By now this should be obvious. No need to run "exuberant ctags" or whatever class-hierarchy-extraction and documentation-extraction program every so often after making changes, and yet your IDE's clickable list of tags and notes stays up to date with file modifications in real time. I think some IDEs do this already with moderate size trees, using stat(), but it's not realistic when you have tens of thousands of source files.

6. File transfer program scanning for shared deltas.

This is nothing more than searching all files for common subsequences which match strong hashes which are expected to be common during file transfer operations. E.g. the GPL header at the start of many source files would have one such hash. A hash of every whole file is another one, along with the name of the file (it is a key which indicates a "likely to match the corresponding hash" condition), as is a hash of every aligned 64k subsequence, or whatever is appropriate to reduce disk I/O.

Having an index of these can speed up some file transfer protocols, in the manner of rsync but comparing among a large group of files instead of just two. The point is that kind of protocol can be more efficient than rsync (sometimes dramatically so), but it's only a net gain to use that kind of algorithm if you have a handy _and reliable_ index of likely common subsequences and whole-file hashes, otherwise it uses far too much disk I/O to check the group. The index needs filesystem coherency support, otherwise it is not reliable enough for this bandwidth optimisation.

Quoted 10 lines
Quoted 4 lines

I won't explain how these work as it would make this mail too long. It should be clear to anyone who thinks about it why the coherency mechanism is essential for real-time, and a consistent interface to container internals helps with performance.

Coherency is essential, but it isn't free. Far from it. The easiest way of getting coherency is having _one_ authoritative source. That way you don't need coherency, and don't pay for it. Anything in this class must by force be just hints, to be recomputed at a moment's notice. I.e., let the application who might use it check and recompute as needed.

(a) Without some kind of minimal hook, there is no way for the application to check without re-reading all the files it has used for a computation, starting at the moment where it's about to potentially recompute the result.

Sometimes you can get away with stat() on the files (e.g. Evolution and fontconfig do that), but often you can't because that's not trustworthy enough (security checks, protocol optimisations, transparent "as if I didn't cache anything" semantics), and sometimes stat() calls are far too slow anyway (anything that involves a lot of files).

(The trustworthiness problem is likely to be solved, for some applications, by an xattr or such which is guaranteed to be deleted or changed when a file is modified. That is really a very minimal hook, but it is one).

(b) Even with all the misgivings of stat(), you can't realistically update real time displays, e.g. the lists of artists & albums on the screen in your MP3 player whenever you modify anything in your music collection using another program. At best, you have to expect the display to not notice the change for a while (minutes in the mp3 case), or until you click an "Explicit Refresh" button. Neither should be necessary.

It's true that the simplest coherency is one authoratitive source which you have to check all the time (although even that doesn't work for some things). But checking all the time does rule out, due to its high algorithmic complexity, a lot of very interesting applications and even some interesting simple unixish tools which you might like, such as real-time "locate".

-- Jamie

#277 lore raw

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

Quoted 6 lines · Jamie Lokier

1. Local Google (by which I mean a search engine on your local machine), Real-time (by which I mean the results are always up to date):

Every file modified since last search is known to the query engine. This is a reality: BeOS does it; WinFS is expected to do it.

Quoted 7 lines

Thus we have real-time local free text search engine, and other features like searching inside files and for file names. The point is the real-time nature of it: the results you get correspond to exactly the contents of the filesystem at the time of the query (writes which occur _during_ a query are obviously not coherent with this, but writes which complete before the query, even immediately before, appear in the results).

Can be done with dnotify/inotify and a cache daemon keeping track of mtime. Yes, this will need a kernel change to make sure mtime always changed when the file changes, but it does not require anything else.

Quoted 1 line

2. MP3 player scanning artists & titles:

Same.

Quoted 1 line

3. Email program scanning for subject lines fast:

Same here.

Quoted 3 lines

4. Blog server caching built pages:
5. Programming environment scanning for tags:
6. File transfer program scanning for shared deltas.

And so on.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#278 lore raw

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

Quoted 3 lines · Christer Weinigel

Can be done with dnotify/inotify and a cache daemon keeping track of mtime. Yes, this will need a kernel change to make sure mtime always changed when the file changes, but it does not require anything else.

- Can the daemon keep track of _every_ file on my disk like this? That's more than a million files, and about 10^5 directories. dnotify would require the daemon to open all the directories. I'm not sure what inotify offers.

- What happens at reboot - I guess the daemon has to call stat() on every file to verify its indexes? Have you any idea how long it takes to call stat() on every file in my home directory?

- The ordering problem: I write to a file, then the program returns. System is very busy compiling. 2 minutes later, I execute a search query. The file I wrote two minute ago doesn't appear in the search results. What's wrong?

Due to scheduling, the daemon hasn't caught up yet. Ok, we can accept that's just hard life. Sometimes it takes a while for something I write to appear in search results.

But! That means I can't use these optimised queries as drop-in replacements for calling grep and find, or for making Make-like programs run faster (by eliminating parsing and stat() calls). That's a shame, it would have been nice to have a mechanism that could transparently optimise prorgrams that do calculations....

Do you see what I'm getting at? There's building some nice GUI and search engine like functionality, where changes made by one program _eventually_ show up in another (i.e. not synchronously).

That's easy.

And then there's optimising things like grep, find, perl, gcc, make, httpd, rsync, in a way that's semantically transparent, but executes faster _as if_ they had recalculated everything they need to every time. That's harder.

Quoted 9 lines
Quoted 1 line

3. Email program scanning for subject lines fast:

Same here.

Quoted 3 lines

4. Blog server caching built pages:
5. Programming environment scanning for tags:
6. File transfer program scanning for shared deltas.

And so on.

No, not 3, 4 or 6. For correct behaviour those require synchronous query results. Think about 6, where one important cached query is "what is the MD5 sum of this file", and another critical one, which can only work through indexing, is "give me the name of any file whose MD5 sum matches $A_SPECIFIC_MD5". Trusting the async results for those kind of queries from your daemon would occasionally result in data loss due to race conditions. So you wouldn't trust the async results, and you fail to get those CPU-saving and bandwidth-saving optimisations.

-- Jamie

#279 lore raw

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

Quoted 4 lines · Jamie Lokier

- Can the daemon keep track of _every_ file on my disk like this? That's more than a million files, and about 10^5 directories. dnotify would require the daemon to open all the directories. I'm not sure what inotify offers.

This is currently a real issue for both desktop search and for virus scanners. They want a "what changed and where" system wide (or at least per namespace/mount).

#280 lore raw

Re: Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

Quoted 9 lines · Alan Cox
Quoted 4 lines · Jamie Lokier

- Can the daemon keep track of _every_ file on my disk like this? That's more than a million files, and about 10^5 directories. dnotify would require the daemon to open all the directories. I'm not sure what inotify offers.

This is currently a real issue for both desktop search and for virus scanners. They want a "what changed and where" system wide (or at least per namespace/mount).

In the database work everything is a transaction and the transactions are logged. Reiser4 is fully atomic and logged. So to get the "what changed and where" you just process the transaction logs from the point of your last marked checkpoint. Hot backup db servers work this way too by listening to the transaction log stream. You don't need a daemon in this model.

Jon Smirl
[email protected]

#281 lore raw

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

In reply to Jamie Lokier
Quoted 9 lines · Jamie Lokier
Quoted 3 lines · Christer Weinigel

Can be done with dnotify/inotify and a cache daemon keeping track of mtime. Yes, this will need a kernel change to make sure mtime always changed when the file changes, but it does not require anything else.

- Can the daemon keep track of _every_ file on my disk like this? That's more than a million files, and about 10^5 directories. dnotify would require the daemon to open all the directories. I'm not sure what inotify offers.

I don't think dnotify/inotify handles subdirectories well yet, so I suppose that they would have to be extended.

Quoted 3 lines

- What happens at reboot - I guess the daemon has to call stat() on every file to verify its indexes? Have you any idea how long it takes to call stat() on every file in my home directory?

The daemon saves state before it shuts down and reloads the state after a reboot. You have to make sure that it is started first and stopped last during the boot process. How would a kernel plugin handle things that happen before or after the plugin module has been loaded? It's the same problem.

Quoted 9 lines

- The ordering problem: I write to a file, then the program returns. System is very busy compiling. 2 minutes later, I execute a search query. The file I wrote two minute ago doesn't appear in the search results. What's wrong?

Due to scheduling, the daemon hasn't caught up yet. Ok, we can accept that's just hard life. Sometimes it takes a while for something I write to appear in search results.

Quoted 5 lines

But! That means I can't use these optimised queries as drop-in replacements for calling grep and find, or for making Make-like programs run faster (by eliminating parsing and stat() calls). That's a shame, it would have been nice to have a mechanism that could transparently optimise prorgrams that do calculations....

Sure you can. First of all, you can just wait for the daemon to finish indexing any files that it has been notified about changes in. This is no different from you having to wait for the kernel to finish indexing the files. Or are you suggesting that the kernel should stop all other processes until the indexing is done?

Quoted 10 lines

Do you see what I'm getting at? There's building some nice GUI and search engine like functionality, where changes made by one program _eventually_ show up in another (i.e. not synchronously).

That's easy.

And then there's optimising things like grep, find, perl, gcc, make, httpd, rsync, in a way that's semantically transparent, but executes faster _as if_ they had recalculated everything they need to every time. That's harder.

If you have a good notify, it's not harder.

Quoted 9 lines

No, not 3, 4 or 6. For correct behaviour those require synchronous query results. Think about 6, where one important cached query is "what is the MD5 sum of this file", and another critical one, which can only work through indexing, is "give me the name of any file whose MD5 sum matches $A_SPECIFIC_MD5". Trusting the async results for those kind of queries from your daemon would occasionally result in data loss due to race conditions. So you wouldn't trust the async results, and you fail to get those CPU-saving and bandwidth-saving optimisations.

So how do you calculate the MD5 sum of a file that is in the process of being modified? It's not possible to do that unless you block all other access to that file and recalculate the MD5 sum after each write. With a notifier that tells the daemon that it has stale data and needs to reprocess the file, it's no different.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#282 lore raw

Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)

On Mon, Sep 06, 2004 at 05:55:21PM +0200, Christer Weinigel wrote:

Could ya'll take this to slashdot for a bit just to improve the level of commentary there?
http://slashdot.org/article.pl?sid=04/09/06/1235236

Signature · 4 lines
Chris Dukes
Warning: Do not use the reflow toaster oven to prepare foods after
it has been used for solder paste reflow. 
http://www.stencilsunlimited.com/stencil_article_page5.htm
#283 lore raw
In reply to Jamie Lokier

Jamie Lokier wrote:
[snip

Quoted 11 lines

When a simple "cd" into .tar.gz or .iso is implemented properly, it will have _no_ performance penalty after you have first looked in the file, so long as it remains in the on-disk cache. And, the filesystem will manage that cache intelligently.

Imagine: for looking at source files and such, you probably won't bother untarring in future, and you won't bother keeping untarred source trees in your home directory for easy access to things you look at often. Why waste the space? You could install whole applications as a .tar and run them from within it, with no performance penalty.

Quoted 6 lines

Similarly, the filesystem will be able to archive directories automatically that haven't been touched in a long time, with no visible change except increased storage space. "grep" will be a bit slower, but you'll have a useful search tool by then (using coherent indexes) which will be more useful than grep, and much faster.

[Again, pardon me for being 5000 messages behind.]

Anyhow, I recall reading an article about 'unified name spaces', and I'm not referring to what's on namesys.com. It mentioned how putting device nodes into the file system is a very powerful innovation of UNIX. Having files and devices in the same name space simplifies tools and makes the environment much more powerful, because you can connect data sources and targets together more arbitarily.

If I recall correctly, the article mentioned something about Plan9 taking things to a greater extreme than UNIX.

Going along with what you said, Jamie, if "containers" like tar files could be accessed like directories without being first extracted, it would increase the power and flexibility of the whole system. Windows XP does something like this with the way it presents ZIP files as directories, and although I'm sure the Windows way isn't nearly as efficient and universal as how Linux would do it, I find the feature to be INCREDIBLY useful.

Of course, we don't necessarily want codecs for every archive format ever invented to be embedded in the kernel. Instead, we need to do something like how you can mount an ISO on a loopback device, only more transparently and more flexibly, and with the less performance-critical codecs being implemented as daemons in userspace.

Also, there would be limitations. For instance, many such pseudo-directories would be read-only, and some would be writable only when on some file systems (like writing to a compressed archive might be much easier to implement on Reiser4 than something else, or perhaps the ext3 version would have to do a lot more copying, while the Reiser version could take advantage of Reiser-specific features like inserting in the middle of a file).

#284 lore raw

A friend asked me a question, and because he is very bright it reminded me that I have not done a good job of reviewing the history of the design's evolution.

He asked me, why not just access a filename's size as filename/size?

So, the original idea was to access metafiles as just files within a directory, and it actually remains that way. However, I first decided to make the standard unix metafiles less intrusive on the namespace. So that led to calling it filename/..size and filename/..owner, etc. In this scheme, the use of '..' was just a style convention for metafiles, and not a requirement in anyway.

This was actually pretty decent as a design, but then a user on the mailing list suggested replacing the '..' prefix with a subdirectory prefix. I forget who suggested this and what the prefix was exactly. So we then created a '..metas' subdirectory, and this had the advantage that one could ls it to see all the builtins supported by a given plugin. This is not an important advantage, and I encourage others to critique it.

So, instead of filename/size we have filename/..metas/size. The only thing gained by this is that 'size' has a rarer name of '..metas/size'. The use of the '..metas/' prefix is purely a non-mandated style convention. File plugins that dislike it are free to violate the convention. There is no deep semantic to it, just a cowardly aversion to intruding on current namespace usage with something as common as 'size'. It has the significant disadvantage of being a longer name than 'size' or '..size'. I could be talked out of it.

Hans

#285 lore raw
Quoted 1 line · Hans Reiser

He asked me, why not just access a filename's size as filename/size?

I now understand that you need a way to distinguish between something like

shoe/size

and

shoe/.../size (or shoe/..size)

The first one is the size of the shoe, the second is the automatically generated size of the file (object). You would get into trouble if you would not allow the user to use shoe/size for shoe size. Peter

#286 lore raw
Quoted 12 lines · Peter Foldiak
Quoted 3 lines · Hans Reiser

He asked me, why not just access a filename's size as filename/size?

I now understand that you need a way to distinguish between something like

shoe/size

Quoted 13 lines

and

shoe/.../size (or shoe/..size)

The first one is the size of the shoe, the second is the automatically generated size of the file (object). You would get into trouble if you would not allow the user to use shoe/size for shoe size. Peter

exactly.

Of course, problem/shoe/size could refer to shoe size in centimeters of a problem shoe or the size of the problem relating to a shoe in units of reporters providing press coverage of it or....

So there are lots of opportunities for ambiguity in semantics....

Still, widely used builtins seem like they should be moderately evasive of commonly used names.

#287 lore raw
Quoted 27 lines · Hans Reiser
Quoted 12 lines · Peter Foldiak
Quoted 2 lines · Hans Reiser

He asked me, why not just access a filename's size as filename/size?

I now understand that you need a way to distinguish between something like

shoe/size

Quoted 13 lines

and

shoe/.../size (or shoe/..size)

The first one is the size of the shoe, the second is the automatically generated size of the file (object). You would get into trouble if you would not allow the user to use shoe/size for shoe size. Peter

Quoted 7 lines

exactly.

Of course, problem/shoe/size could refer to shoe size in centimeters of a problem shoe or the size of the problem relating to a shoe in units of reporters providing press coverage of it or....

So there are lots of opportunities for ambiguity in semantics....

Still, widely used builtins seem like they should be moderately evasive of commonly used names.

You know, if tools all need to be rewritten anyway to deal with the file metadata "directory", then why not change the symbol that delimits the metadata key?

Everyone likes ':', so we'd have "problem/shoe:size". (Don't bother to complain about files which have : in them, because I already know it sucks, but it's an example.)

See, unless you can come up with a way to seamlessly make old tools work with the new semantics, then there's no reason not to make more than one change to tools at at the same time.

Also, if you take the ':' example literally, then the file system would need to be able to figure out that a file whose literal name is "C:\MYPR0NDIR\BODY_PARTS.JPG" isn't referring to metadata for a file named "C". :)

#288 lore raw

From: Timothy Miller <[email protected]>

Quoted 3 lines

Everyone likes ':', so we'd have "problem/shoe:size". (Don't bother to complain about files which have : in them, because I already know it sucks, but it's an example.)

[[ I just joined this discussion, so pardon if this is already known.]]

One advantage of ':' is that portable programs already have to avoid it because of NTFS alternate data streams:
http://www.diamondcs.com.au/index.php?page=archive&id=ntfs-streams

For example on an XP box with NTFS:

$ mkdir j
$ cd j
$ echo hi > foo:bar
$ ls -l
total 0
-rw-r--r--    1 wscott   Administ        0 Sep 10 10:45 foo
$ cat foo
$ cat foo:bar
hi
$ rm foo
$ cat foo:bar
cat: foo:bar: No such file or directory

-Wayne

#289 lore raw
Quoted 8 lines · Timothy Miller
Quoted 3 lines

Everyone likes ':', so we'd have "problem/shoe:size". (Don't bother to complain about files which have : in them, because I already know it sucks, but it's an example.)

[[ I just joined this discussion, so pardon if this is already known.]]

One advantage of ':' is that portable programs already have to avoid it because of NTFS alternate data streams:

Portable programs working on POSIX don't give a damn that C:, CON:, etc are reserved in MS-DOS.

Windows warts shouldn't infect Linux.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#291 lore raw
In reply to Timothy Miller
Quoted 34 lines · Timothy Miller
Quoted 30 lines · Hans Reiser
Quoted 13 lines · Peter Foldiak
Quoted 2 lines · Hans Reiser

He asked me, why not just access a filename's size as filename/size?

I now understand that you need a way to distinguish between something like

shoe/size

Quoted 13 lines

and

shoe/.../size (or shoe/..size)

The first one is the size of the shoe, the second is the automatically generated size of the file (object). You would get into trouble if you would not allow the user to use shoe/size for shoe size. Peter

exactly.

Quoted 8 lines
Quoted 5 lines

Of course, problem/shoe/size could refer to shoe size in centimeters of a problem shoe or the size of the problem relating to a shoe in units of reporters providing press coverage of it or....

So there are lots of opportunities for ambiguity in semantics....

Still, widely used builtins seem like they should be moderately evasive of commonly used names.

You know, if tools all need to be rewritten anyway to deal with the file metadata "directory", then why not change the symbol that delimits the metadata key?

because it is useful that it is only a style convention. Changing the symbol makes it mandatory to distinguish metafiles from files.

Quoted 9 lines

Everyone likes ':', so we'd have "problem/shoe:size". (Don't bother to complain about files which have : in them, because I already know it sucks, but it's an example.)

See, unless you can come up with a way to seamlessly make old tools work with the new semantics, then there's no reason not to make more than one change to tools at at the same time.

Also, if you take the ':' example literally, then the file system would need to be able to figure out that a file whose literal name is "C:\MYPR0NDIR\BODY_PARTS.JPG" isn't referring to metadata for a file named "C". :)

#292 lore raw
Quoted 1 line · Hans Reiser

Timothy Miller wrote:

[...]

Quoted 6 lines
Quoted 3 lines

You know, if tools all need to be rewritten anyway to deal with the file metadata "directory", then why not change the symbol that delimits the metadata key?

because it is useful that it is only a style convention. Changing the symbol makes it mandatory to distinguish metafiles from files.

If they are really different than directories, it should be clear what is what, IMVHO. If they aren't different, they have no place here. Trying to break hoary Unix tradition (files and directories are separate) while pretending nothing has changed (see, a file can be handled just like a directory) just doesn't cut it.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#293 lore raw
Quoted 11 lines · Horst von Brand
Quoted 1 line · Hans Reiser

Timothy Miller wrote:

[...]

Quoted 1 line
Quoted 1 line

You know, if tools all need to be rewritten anyway to deal with the file metadata "directory", then why not change the symbol that delimits the metadata key?

Quoted 8 lines
Quoted 1 line

because it is useful that it is only a style convention. Changing the symbol makes it mandatory to distinguish metafiles from files.

If they are really different than directories, it should be clear what is what, IMVHO. If they aren't different, they have no place here. Trying to break hoary Unix tradition (files and directories are separate) while pretending nothing has changed (see, a file can be handled just like a directory) just doesn't cut it.

I think that sums up what I was trying to say better than what I said did. :)

#294 lore raw
In reply to Hans Reiser

Hans Reiser wrote on Thu, 09 Sep 2004 22:22:51 -0700:

Quoted 7 lines

This was actually pretty decent as a design, but then a user on the mailing list suggested replacing the '..' prefix with a subdirectory prefix. I forget who suggested this and what the prefix was exactly. So we then created a '..metas' subdirectory, and this had the advantage that one could ls it to see all the builtins supported by a given plugin. This is not an important advantage, and I encourage others to critique it.

Like I said before, it breaks orthogonality by introducing a directory which isn't a file/directory like the others. ..metas can't have attributes attached to it like other files can. In practical use, that makes it difficult to display in a GUI view, where you might want to attach attributes describing the ..metas directory window size and position. The GUI code would have to special case it, for those times when you want to drill down and see what's attached to a file.

On the other hand, it does satisfy the Human need to group things together neatly, like you said. But then so does Tetris :-).

- Alex

#295 lore raw
Quoted 11 lines · Alexander G. M. Smith

Hans Reiser wrote on Thu, 09 Sep 2004 22:22:51 -0700:

Quoted 3 lines

This was actually pretty decent as a design, but then a user on the mailing list suggested replacing the '..' prefix with a subdirectory prefix. I forget who suggested this and what the prefix was exactly. So we then created a '..metas' subdirectory, and this had the advantage that one could ls it to see all the builtins supported by a given plugin. This is not an important advantage, and I encourage others to critique it.

Like I said before, it breaks orthogonality by introducing a directory which isn't a file/directory like the others. ..metas can't have attributes attached to it like other files can.

That does not have to be true. Whether ..metas contains only builtins can be nothing more than a style convention, or it can be a plugin property for plugins that don't want other files to pretend to be their builtins.

Quoted 14 lines

In practical use,
that makes it difficult to display in a GUI view, where you might want to attach attributes describing the ..metas directory window size and position. The GUI code would have to special case it, for those times when you want to drill down and see what's attached to a file.

On the other hand, it does satisfy the Human need to group things together neatly, like you said. But then so does Tetris :-).

- Alex

#296 lore raw
In reply to Hans Reiser
Quoted 3 lines · Hans Reiser

A friend asked me a question, and because he is very bright it reminded me that I have not done a good job of reviewing the history of the design's evolution.

The design's evolution, and your friend's question, are irrelevant. Might be a nice story, but not relevant to discuss the _current_ design, its shortcommings, and possible solutions.

Quoted 1 line

He asked me, why not just access a filename's size as filename/size?

Urgh.

Quoted 6 lines

So, the original idea was to access metafiles as just files within a directory, and it actually remains that way. However, I first decided to make the standard unix metafiles less intrusive on the namespace. So that led to calling it filename/..size and filename/..owner, etc. In this scheme, the use of '..' was just a style convention for metafiles, and not a requirement in anyway.

Oh, so metafiles could have any names at all. Awful.

Quoted 1 line

This was actually pretty decent as a design,

Could have been worse, true. Not very much, anyway.

Quoted 4 lines

but then a user on the
mailing list suggested replacing the '..' prefix with a subdirectory prefix. I forget who suggested this and what the prefix was exactly. So we then created a '..metas' subdirectory,

Bad design. As Ted explained, _any_ "normal" name in there will lead to legacy applications do the wrong thing unaware of the special status of this stuff, and this is sure to break things and/or to lead to nasty security problems.

Quoted 3 lines

and this had the advantage
that one could ls it to see all the builtins supported by a given plugin.

What kind of builtins? What is a "plugin"?

Quoted 2 lines

This is not an important advantage, and I encourage others to
critique it.

If it is not important, let's postpone any discussion on it until (if ever) the important problems are solved.

Quoted 2 lines

So, instead of filename/size we have filename/..metas/size. The only thing gained by this is that 'size' has a rarer name of '..metas/size'.

English, please.

Quoted 2 lines

The use of the '..metas/' prefix is purely a non-mandated style convention.

So it isn't mandated, and I could decide placing my own stuff under other names, willy-nilly? See the comment above on "normal names" being dangerous; if it hasn't a builtin way of ensuring legacy (or even new applications) don't trip over this stuff, it is a moby nightmare security and correctness wise.

Quoted 3 lines

File plugins that dislike it are free to violate the
convention. There is no deep semantic to it, just a cowardly aversion to intruding on current namespace usage

"Common" and "current usage" is important to the average user, "normal" is crucial for correctness and security,

Quoted 3 lines

with something as common as 'size'.
It has the significant disadvantage of being a longer name than 'size' or '..size'. I could be talked out of it.

So far, I've seen no real reason to introduce this. If it is scrapped, there will be no need to talk anybody out of silly ideas.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#297 lore raw
In reply to Hans Reiser
Quoted 1 line · Hans Reiser

Yes, this was part of the plan, tar file-directory plugins would be cute.

Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute
directory or a directory in the tarball named "metas"?

Joel

Signature · 10 lines
"I'm so tired of being tired,
 Sure as night will follow day.
 Most things I worry about
 Never happen anyway."

Joel Becker
Senior Member of Technical Staff
Oracle Corporation
E-mail: [email protected]
Phone: (650) 506-8127
#298 lore raw
Quoted 5 lines · Joel Becker
Quoted 1 line · Hans Reiser

Yes, this was part of the plan, tar file-directory plugins would be cute.

Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute
directory or a directory in the tarball named "metas"?

This needs to be designed.

Perhaps /foo/bar/baz.tar.gz/tar/metas is the directory in the tarball named "metas".

Or perhaps /foo/bar/baz.tar.gz/x/metas is: it's independent of archive format, and I personally tend to extract things into a directory called "x". [*]

Or perhaps /foo/bar/baz.tar.gz/metas is, and the attribute directory is /foo/bar/baz.tar.gz/../metas, to be perverse ;)

I prefer the second one, ("x/metas"), but not with any conviction.

-- Jamie

[*] Actually I prefer:

/foo/bar/baz.tar.gz/content/metas /foo/bar/baz-0.01.tar.gz/content/baz-0.01/metas

Archives always in "content". One layer of decompression always tried for .tar files and other uncompressed archive formats.

/foo/bar/baz.tar.gz/x -> content/ /foo/bar/baz-0.01.tar.gz/x -> content/baz-0.01/

If the root of the archive contains a single directory, "x" is a symlink to it. Otherwise "x" is a symlink to the root directory of the archive. This is comfortable with the common practice by which archives are distributed, without making a mess when someone forgets to put everything in a top-level directory.

#299 lore raw
In reply to Joel Becker
Quoted 2 lines · Joel Becker

Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute
directory or a directory in the tarball named "metas"?

This has been fought over on the reiserfs-list ad nauseaum, but it's a valid point.

That's why I tend to rename metas/ into ..metas/ to avoid name clashes, even if I've never had a directory named metas/ apart from what Reiser4 ships.

It is then debatable if it should be renamed before it's too late, have it renamable in the kernel configs (and the name exported via /sys or something) or just leave it be.

Signature · 1 line
mjt
#300 lore raw
Quoted 3 lines · Markus Törnqvist

That's why I tend to rename metas/ into ..metas/ to avoid name clashes, even if I've never had a directory named metas/ apart from what Reiser4 ships.

http://packages.debian.org/cgi-bin/search_contents.pl?word=metas&searchmode=searchfilesanddirs&case=insensitive&version=unstable&arch=i386

OK, those are capital METAS rather than junior metas, but it does show this is not a unique word to reiser4.

Signature · 10 lines
"Next the statesmen will invent cheap lies, putting the blame upon 
the nation that is attacked, and every man will be glad of those
conscience-soothing falsities, and will diligently study them, and refuse
to examine any refutations of them; and thus he will by and by convince 
himself that the war is just, and will thank God for the better sleep 
he enjoys after this process of grotesque self-deception." -- Mark Twain
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#301 lore raw
In reply to Jamie Lokier
Quoted 2 lines · Jamie Lokier

One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way.

Arguably this belongs in userspace --- and people have put it there.

--cw

#302 lore raw
Quoted 6 lines · Chris Wedgwood
Quoted 2 lines · Jamie Lokier

One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way.

Arguably this belongs in userspace --- and people have put it there.

I agree that these belong in userspace, and that there's plenty* of userspace code doing a similar thing already. I don't think there's any argument over it.

However, as far as I know it's not accessible in a file-as-directory form as yet. In my opinion that is the most natural form and it would be very intuitive to use. I hope we can pick a useful semantics for them, and also provide filesystem-independent plugins with GNU Hurd-like per-user extensibility.

-- Jamie

* plenty == too much.
Gnome, KDE, Emacs and Bash all see different virtual filesystems. (All but Bash implement their own virtual filesystem extensions). That makes them much less useful than they could be.

#303 lore raw
Quoted 20 lines
Quoted 6 lines · Chris Wedgwood
Quoted 2 lines · Jamie Lokier

One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way.

Arguably this belongs in userspace --- and people have put it there.

I agree that these belong in userspace, and that there's plenty* of userspace code doing a similar thing already. I don't think there's any argument over it.

However, as far as I know it's not accessible in a file-as-directory form as yet. In my opinion that is the most natural form and it would be very intuitive to use. I hope we can pick a useful semantics for them, and also provide filesystem-independent plugins with GNU Hurd-like per-user extensibility.

Quoted 6 lines

-- Jamie

* plenty == too much.
Gnome, KDE, Emacs and Bash all see different virtual filesystems. (All but Bash implement their own virtual filesystem extensions). That makes them much less useful than they could be.

Exactly, and I doubt they have the ability to join together and create a common uniform standard out of it either. They have just far to different views on Linux and how they want things to be done.

#304 lore raw
Quoted 1 line · Spam
Quoted 1 line

Chris Wedgwood wrote:

[...]

Quoted 7 lines
Quoted 3 lines

Gnome, KDE, Emacs and Bash all see different virtual filesystems. (All but Bash implement their own virtual filesystem extensions). That makes them much less useful than they could be.

Exactly, and I doubt they have the ability to join together and create a common uniform standard out of it either. They have just far to different views on Linux and how they want things to be done.

And the solution is... inventing yet another, incompatible way of doing things that nobody outside its followers will ever use. Inside the kernel. Should have thought of that before.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#305 lore raw
In reply to Jamie Lokier
Quoted 6 lines · Jamie Lokier

However, as far as I know it's not accessible in a file-as-directory form as yet. In my opinion that is the most natural form and it would be very intuitive to use. I hope we can pick a useful semantics for them, and also provide filesystem-independent plugins with GNU Hurd-like per-user extensibility.

Quoted 6 lines

-- Jamie

* plenty == too much.
Gnome, KDE, Emacs and Bash all see different virtual filesystems. (All but Bash implement their own virtual filesystem extensions). That makes them much less useful than they could be.

It has always bugged me that Gnome and KDE implement their own VFS layers.

It seems to me that having a standard userspace filesystem layer like FUSE (http://sourceforge.net/projects/avf) would provide a lot of the benefit that HURD users (all 5 of them) see from filesystem "translators". Now, safely allowing unprivileged users to run arbitrary userspace filesystems would be a real trick. But, if it were possible, it could be combined with files-as-directories in some rather neat ways.

Signature · 2 lines
Will Dyson
"Back off man, I'm a scientist!" -Dr. Peter Venkman
#306 lore raw
Quoted 8 lines · Will Dyson
Quoted 6 lines · Jamie Lokier

However, as far as I know it's not accessible in a file-as-directory form as yet. In my opinion that is the most natural form and it would be very intuitive to use. I hope we can pick a useful semantics for them, and also provide filesystem-independent plugins with GNU Hurd-like per-user extensibility.

Quoted 9 lines
Quoted 6 lines

-- Jamie

* plenty == too much.
Gnome, KDE, Emacs and Bash all see different virtual filesystems. (All but Bash implement their own virtual filesystem extensions). That makes them much less useful than they could be.

It has always bugged me that Gnome and KDE implement their own VFS layers.

Same here. This always seemed like something the kernel should be able to handle. It seems to me that if reiser4 had been available at the time the Gnome and KDE developers would not have needed to do this.

Lee

#307 lore raw

Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell:

Quoted 5 lines
Quoted 1 line

It has always bugged me that Gnome and KDE implement their own VFS layers.

Same here. This always seemed like something the kernel should be able to handle. It seems to me that if reiser4 had been available at the time the Gnome and KDE developers would not have needed to do this.

Well, the kernel doesn't have a filesystem that speaks http, scp and those things. GnomeVFS is URL-based. It has some pseudo-protocols that extract a pseudo directory-tree for all installed applications + the changes the used made, created on the fly from a set of XML files that are read-only and system-wide and the user-overridden changes. I don't know if all of these things would really make sense inside the kernel.

#308 lore raw
Quoted 8 lines · Christophe Saout

Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell:

Quoted 5 lines
Quoted 1 line

It has always bugged me that Gnome and KDE implement their own VFS layers.

Same here. This always seemed like something the kernel should be able to handle. It seems to me that if reiser4 had been available at the time the Gnome and KDE developers would not have needed to do this.

Quoted 7 lines

Well, the kernel doesn't have a filesystem that speaks http, scp and those things. GnomeVFS is URL-based. It has some pseudo-protocols that extract a pseudo directory-tree for all installed applications + the changes the used made, created on the fly from a set of XML files that are read-only and system-wide and the user-overridden changes. I don't know if all of these things would really make sense inside the kernel.

True. FWIW, I never use most of those features. It's just too damn slow. Windows seems to implement all of the useful features of GnomeVFS, and they are 10x faster.

Lee

#309 lore raw
Quoted 6 lines · Lee Revell
Quoted 2 lines · Christophe Saout

are read-only and system-wide and the user-overridden changes. I don't know if all of these things would really make sense inside the kernel.

True. FWIW, I never use most of those features. It's just too damn slow. Windows seems to implement all of the useful features of GnomeVFS, and they are 10x faster.

Are they in the kernel in Windows?

Signature · 1 line
mjt
#310 lore raw

Hello Markus,

Quoted 9 lines · you
Quoted 6 lines · Lee Revell
Quoted 2 lines · Christophe Saout

are read-only and system-wide and the user-overridden changes. I don't know if all of these things would really make sense inside the kernel.

True. FWIW, I never use most of those features. It's just too damn slow. Windows seems to implement all of the useful features of GnomeVFS, and they are 10x faster.

Are they in the kernel in Windows?

Some are, some aren't :)

Many of the features are between the FS and the application layers. Therefore they will work with all programs that use the Windows API to access files etc. I do not think that many programs actually
access disks directly at the FS or below level. Exceptions would
perhaps be file-recovery tools etc.

Others are simply plugins to Windows Explorer.

I don't know if all, any, or none, are installed at the kernel level. I doubt they are.

But the problem with Linux seem to be that programs actually do access the FS directly. Therefore everything will break if you change the FS structures to much.

Like I said, this is IMHO something that Linux should somehow solve. Applications should really not access the filesystems directly. Perhaps Grub and stuff like it should, but not many other.

~S

#311 lore raw
In reply to Lee Revell
Quoted 3 lines · Lee Revell

Same here. This always seemed like something the kernel should be able to handle. It seems to me that if reiser4 had been available at the time the Gnome and KDE developers would not have needed to do this.

reiser4 has prompted the discussion again, but it doesn't provide the virtual filesystem capabilities that we're talking about, which Gnome and KDE implement in userspace. The FUSE project is much more relevant for that.

We had file-as-directory discussions years ago, and 5 years ago I was experimenting with doing virtual filesystems with that capability over NFS.

podfuk offered something close to Gnome and KDE but in kernel space:
virtual filesytems accessible through the kernel, but it didn't get too popular. It was based on Midnight Commander's VFS (which has evolved into Gnome's VFS), using the CODA filesystem interface to provide the kernel hooks. It had some problems, and was a bit fiddly to use. We still don't have something that's particularly nice to use for ordinary users.

-- Jamie

#312 lore raw
In reply to Jamie Lokier

Hi!

Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way.

Arguably this belongs in userspace --- and people have put it there.

Quoted 7 lines

I agree that these belong in userspace, and that there's plenty* of userspace code doing a similar thing already. I don't think there's any argument over it.

However, as far as I know it's not accessible in a file-as-directory form as yet. In my opinion that is the most natural form and it would be very intuitive to use.

It does not work. .deb file is ar archive. It is also debian package. How do you know if I want to see it as a package or as a archive? How do you identify file types, anyway?

Same happens for .tar.gz.

uservfs does

cd foo.deb#uar
vs.
cd foo.deb#udeb

and

cd foo.tgz#utar
vs.
cat foo.tgz#ugz

Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         
#313 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pavel Machek wrote:
[...]

| uservfs does
|
| cd foo.deb#uar

cd foo.deb/ar
| vs.
| cd foo.deb#udeb
cd foo.deb/deb

and why would you want that, instead of just:
cd foo.deb # for the ar
dpkg -i foo.deb # for the deb

|
| and
|
| cd foo.tgz#utar

cd foo.tgz
| vs.
| cat foo.tgz#ugz
cat foo.tgz

Just want to extract the tar file? Maybe something like
cat foo.tgz/gunzip
In which case (of course) foo.tgz/gunzip has exactly the same directory contents as foo.tgz

Looks different, that's all.

In fact, for just about any syntax anyone could suggest, I can't really see why you can't just replace all weird symbols with a slash and a symbol. Instead of
foo.tgz#utar
you have

	foo.tgz/#/utar

Only difference is, some things which used to require special tools can now be serviced by less than what's in busybox.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTPNpngHNmZLgCUhAQJ0PA//RmPgHL4MgiQn7Li5b52DI/0p+Zp9JUs7 OaA+k4wHeXEIdBProtBQ+3541noEg086bVriYRnajsA9xgd/P9ncE7ci/d9Kagdl 8KmH4eWnCjNkL4x8GnDeR+EM5EfcgoSxR7ezO1KO9uMMcPFnCvmeB1yB5U+Fr9Q1 HVPHfGsh+ZWU2CFJHoJx27Q07UK+bFvzGr4JkhLbRMbPeEROrkHJHp3wj/N2yXXb 5icLgFa0g/b6wM8SuBAaO5xVM8jkmECI/P5Jo3n/d/CTUumb/BoKMIgbMurR61CG hrZ40mizpCMFAnLrgm9FIvjKtZUsigG+oM/2xTtku2Z2nSM16/ChKtJOAj6MoMGT xvWzLdTY6kEL30VDaHtcVvIyUFtUe5oYxEasnrcKseV7hJtccgjTYKB21PnWyZFz /KIdh8sBg6i1nuYOtHncL6agD1M2bg1vBYwbMvIRfa1YbAFUbW1A6hDTftxJIhLn Beso6d68SyfNMdQ8yWloR7sIufmbDut2fx1SHS3wPt2Z5W1e0XBexcpSqdD8KPns lvmC0ggPDuQzXjPCIsS17Uk5vrCtVEdrsCCaj62a8UE2d1mCqu4CyLwMYCFv3Sve SwZ36iOv4fzdvgjPOUg27tMwbpa/WiDAcZJ/31BfZfTDl4E9LbkQjWYT/YgsJuaQ wp3Hi/Cgh/4=
=8YVp
-----END PGP SIGNATURE-----

#314 lore raw

Hi!

Quoted 11 lines
| uservfs does
|
| cd foo.deb#uar

cd foo.deb/ar
| vs.
| cd foo.deb#udeb
cd foo.deb/deb

and why would you want that, instead of just:
cd foo.deb # for the ar
dpkg -i foo.deb # for the deb

Because I want to see contents of that .deb package, nicely parsed?

Quoted 4 lines

Just want to extract the tar file? Maybe something like
cat foo.tgz/gunzip
In which case (of course) foo.tgz/gunzip has exactly the same directory contents as foo.tgz

Yes, that would work.

Quoted 8 lines

In fact, for just about any syntax anyone could suggest, I can't really see why you can't just replace all weird symbols with a slash and a symbol. Instead of
foo.tgz#utar
you have

	foo.tgz/#/utar

Only difference is, some things which used to require special tools can now be serviced by less than what's in busybox.

That would work, too. I do not get your comment about busybox.
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#315 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Pavel" == Pavel Machek <[email protected]> writes:

Pavel> Hi!

Quoted 7 lines
Quoted 7 lines
| uservfs does
|
| cd foo.deb#uar cd foo.deb/ar | vs.  | cd foo.deb#udeb cd

foo.deb/deb

and why would you want that, instead of just: cd foo.deb # for the ar dpkg -i foo.deb # for the deb

Pavel> Because I want to see contents of that .deb package, nicely Pavel> parsed?

What nicely-parsed information do you want to see? A deb package is an ar archive of three files:
- debian-binary, which contains the deb format version
- control.tar.gz, which contains the control data and related things
- data.tar.gz, which contains the actual files to be installed

So to get at the format version, you can "cat foo.deb/debian-binary". To look at the control data, you can "ls foo.deb/control.tar.gz". To browse the files, "cd foo.deb/data.tar.gz".

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#316 lore raw
In reply to Pavel Machek
Quoted 1 line · Pavel Machek

David Masover <[email protected]> said:

[...]

Quoted 16 lines
Quoted 4 lines

Just want to extract the tar file? Maybe something like
cat foo.tgz/gunzip
In which case (of course) foo.tgz/gunzip has exactly the same directory contents as foo.tgz

Yes, that would work.

Quoted 8 lines

In fact, for just about any syntax anyone could suggest, I can't really see why you can't just replace all weird symbols with a slash and a symbol. Instead of
foo.tgz#utar
you have

	foo.tgz/#/utar

Only difference is, some things which used to require special tools can now be serviced by less than what's in busybox.

Quoted 1 line

That would work, too. I do not get your comment about busybox.

You do need extra tools anyway, placing them in the kernel is cheating (and absolutely pointless, IMHO).

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#317 lore raw
Quoted 3 lines · Horst von Brand

You do need extra tools anyway, placing them in the kernel is cheating (and absolutely pointless, IMHO).

I agree.

There's no point to having the kernel export information that is already inherent in the main stream.

I've seen all these examples of exposing MP3 ID information as a "side stream", and that's TOTALLY POINTLESS! The information is already there, it's in a standard format, and exporting it as a stream buys you absolutely nothing.

Where named attributes make sense is when they are _independent_ data. Data that is only tangentially related to the main stream itself, and which the main stream _cannot_ encompass because of some real technical issue.

In a graphical environment, the "icon" stream is a good example of this. It literally has _nothing_ to do with the data in the main stream. The only linkage is a totally non-technical one, where the user wanted to associate a secondary stream with the main stream _without_ altering the main one. THAT is where named streams make sense.

But if you want to look at one particular file inside a tar-file, do so in user space. There are zero advantages to exposing it as a side-stream, and there are absolutely _tons_ of disadvantages.

In short, named streams only make sense if:
- they are tied to the file some way that is _independent_ of the file contents (since if it's dependent on the file contents, you're just a ton better off regenerating it with a caching server)
- there are serious reasons to keep the lookup synchronized (since if there isn't such a reason, you're just better off with a separate shadow tree in user space)

And realize that the "separate shadow tree" actually works very well. That's how version control systems like CVS have always worked. It's certainly how you can make icon information work too. If you use a tool for accessing the data, the tool can maintain coherency and you'll never care about the side stream.

Which means that normally we really don't _want_ named streams. In 99% of all cases we can use equally good - and _much_ simpler - tool-based solutions.

Which means that the only _real_ technical issue for supporting named streams really ends up being things like samba, which want named streams just because the work they do fundamentally is about them, for externally dictated reasons. Doing named streams for any other reason is likely just being stupid.

Once you do decide that you have to do named streams, you might then decide to use them for convenient things like icons. But it should very much be a secondary issue at that point.

Linus

#318 lore raw
Quoted 9 lines · Horst von Brand
Quoted 3 lines

You do need extra tools anyway, placing them in the kernel is cheating (and absolutely pointless, IMHO).

I agree.

There's no point to having the kernel export information that is already inherent in the main stream.

There are two things you are saying in the same sentence here. One I do not agree with. There is point to export information from the main stream to secondary file streams. Just it doesn't have to be done by the kernel.

Just do not limit things to a few defined types of things that can be stored in a stream. IMHO a file stream should be able to contain just about anything the user wishes. That said, the kernel should not have to know about the contents.

Quoted 4 lines

I've seen all these examples of exposing MP3 ID information as a "side stream", and that's TOTALLY POINTLESS! The information is already there, it's in a standard format, and exporting it as a stream buys you absolutely nothing.

True. But this is the case for this specific example. I have stated previously about what I think would be legitimate uses as examples. Of course these apply to me and what I work with. One of the
examples were to add streams that contained descriptions or
keywords.

Quoted 11 lines

Where named attributes make sense is when they are _independent_ data. Data that is only tangentially related to the main stream itself, and which the main stream _cannot_ encompass because of some real technical issue.

In a graphical environment, the "icon" stream is a good example of this. It literally has _nothing_ to do with the data in the main stream. The only linkage is a totally non-technical one, where the user wanted to associate a secondary stream with the main stream _without_ altering the main one. THAT is where named streams make sense.

Quoted 3 lines

But if you want to look at one particular file inside a tar-file, do so in user space. There are zero advantages to exposing it as a side-stream, and there are absolutely _tons_ of disadvantages.

The tar archive support seem (is) to be useless because tar is also so supported and implemented in every type of application out there. It is also easy to script and otherwise work with.

But the idea to be able to extend the filesystem to show/represent data as file streams and meta directores is very good.

Someone might come up with an idea to allow browsing and changing of data inside XML files. Of course this may already be done with sed and other tools.

Quoted 8 lines

In short, named streams only make sense if:
- they are tied to the file some way that is _independent_ of the file contents (since if it's dependent on the file contents, you're just a ton better off regenerating it with a caching server)
- there are serious reasons to keep the lookup synchronized (since if there isn't such a reason, you're just better off with a separate shadow tree in user space)

Quoted 5 lines

And realize that the "separate shadow tree" actually works very well. That's how version control systems like CVS have always worked. It's certainly how you can make icon information work too. If you use a tool for accessing the data, the tool can maintain coherency and you'll never care about the side stream.

Just that with shadow trees (I assume they are stored simply in another folder beside the file) you won't be able to "cp filea /mnt/backup" and keep the extra stuff.

Sure icons could also easily be stores as file.icon, just like on the Amiga :). It just isn't as nice.

Quoted 9 lines

Which means that normally we really don't _want_ named streams. In 99% of all cases we can use equally good - and _much_ simpler - tool-based solutions.

Which means that the only _real_ technical issue for supporting named streams really ends up being things like samba, which want named streams just because the work they do fundamentally is about them, for externally dictated reasons. Doing named streams for any other reason is likely just being stupid.

I doubt it would be stupid. Perhaps today just not needed for many things.

Quoted 3 lines

Once you do decide that you have to do named streams, you might then decide to use them for convenient things like icons. But it should very much be a secondary issue at that point.

I think we focus to much on the actual contents of these streams. Do we really have to know what will in the future be stored inside these streams?

For example. Applications may use file streams for locking instead of having a .lock file. The future isn't known, and we shouldn't try to limit what we can put in the streams. You said 99% of the uses won't need them. True. The same is with xattr, ACLs and so many other things.

Even if 0.1% of the users today would use it I think we should have the support for named streams.

~S

Quoted 1 line

Linus

#319 lore raw
In reply to Linus Torvalds

Hi!

Quoted 5 lines
Quoted 2 lines

You do need extra tools anyway, placing them in the kernel is cheating (and absolutely pointless, IMHO).

I agree.

Quoted 7 lines

There's no point to having the kernel export information that is already inherent in the main stream.

I've seen all these examples of exposing MP3 ID information as a "side stream", and that's TOTALLY POINTLESS! The information is already there, it's in a standard format, and exporting it as a stream buys you absolutely nothing.

It buys me caching. I do quite often

bzcat patch.2.6.8.bz2 | less (read the patch)
(sometimes repeat that few times because I hit ^c when I should not have etc).
cd ...clean; bzcat patch.2.6.8.bz2 | patch -Esp1
cd ...linux; bzcat patch.2.6.8.bz2 | patch -Esp1

Now... that's total waste of cpu. bzip2 decompression takes quite some time.

I could do

bzcat patch.2.6.8.bz2 > /tmp/delete.me.when.you.are.done

...but I'd probably forget to delete that one and anyway, it requires me to think about it. Nicest way would be

cat patch.2.6.8.bz2/ubz | less
cd ...clean; cat patch.2.6.8.bz2/ubz | patch -Esp1 cd ...linux; cat patch.2.6.8.bz2/ubz | patch -Esp1

with kernel intelligently caching uncompressed data. I believe this can not be done completely in userspace.
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#320 lore raw
Quoted 2 lines · Pavel Machek

It buys me caching.

I'll actually buy into that. If only because I consider caching to be one of the more important things that the kernel does (caches are a _classic_ case of "shared data that needs synchronization").

However, that said, user space can trivially cache things in the filesystem, so while this may be a convenient feature, I think you should look at perhaps doing it in the _shell_ instead..

Linus

#321 lore raw

Hi!

Quoted 9 lines
Quoted 1 line

It buys me caching.

I'll actually buy into that. If only because I consider caching to be one of the more important things that the kernel does (caches are a _classic_ case of "shared data that needs synchronization").

However, that said, user space can trivially cache things in the filesystem, so while this may be a convenient feature, I think you should look at perhaps doing it in the _shell_ instead..

That cache should disappear as soon as I need disk
space. I.e. userspace should never see -ENOSPC because of this kind of caching. This need some kernel support. Ouch and cached file should atomically go away as soon as main file changes, otherwise I do not see how multiple processes could cooperate on caching...

chattr +kill-this-file-when-low-on-disk-space patch.bz2...ubz

would solve first problem. Not sure how to do the second one.

Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#322 lore raw
Quoted 10 lines · Pavel Machek
Quoted 4 lines

However, that said, user space can trivially cache things in the filesystem, so while this may be a convenient feature, I think you should look at perhaps doing it in the _shell_ instead..

That cache should disappear as soon as I need disk
space. I.e. userspace should never see -ENOSPC because of this kind of caching. This need some kernel support. Ouch and cached file should atomically go away as soon as main file changes, otherwise I do not see how multiple processes could cooperate on caching...

Well, what other projects have done is to just reserve a certain amount of disk for caching. See "ccache", which solves both of the above problems (it doesn't shrink the cache on ENOSPC, but reserving diskspace is accepted practice for things like this..)

Linus

#323 lore raw

Hi!

Quoted 10 lines
Quoted 9 lines
Quoted 3 lines

However, that said, user space can trivially cache things in the filesystem, so while this may be a convenient feature, I think you should look at perhaps doing it in the _shell_ instead..

That cache should disappear as soon as I need disk
space. I.e. userspace should never see -ENOSPC because of this kind of caching. This need some kernel support. Ouch and cached file should atomically go away as soon as main file changes, otherwise I do not see how multiple processes could cooperate on caching...

Quoted 4 lines

Well, what other projects have done is to just reserve a certain amount of disk for caching. See "ccache", which solves both of the above problems (it doesn't shrink the cache on ENOSPC, but reserving diskspace is accepted practice for things like this..)

Okay, that does work, it just is not really nice. Just as reserving fixed ammount of space for disk cache is bad, reserving fixed ammount of space for ccache (and similar) is bad. When there are few of such caches, balancing between them starts to matter...
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#324 lore raw
Quoted 4 lines · Pavel Machek

Okay, that does work, it just is not really nice. Just as reserving fixed ammount of space for disk cache is bad, reserving fixed ammount of space for ccache (and similar) is bad. When there are few of such caches, balancing between them starts to matter...

So try to convince them to use the same cache daemon or the same shared cache manager library then. It isn't all that different from implementing a kernel interface that everyone is supposed to use.

A cache manager daemon could sit and watch the free space on the disk every other second and start deleting the cached files (according to some LCU heuristics or whatever) whenever free space is getting low.

To see if the original file is newer than the cached file, good old mtime can be used.

I can't remember who right now, but someone mentioned a few problems with timestamps:

1. The resolutions is too low in current kernels, but that just means that someone(tm) needs to implement nanosecond timestamps in the kernel and for the filesystems.

2. Even nanosecond timestams may become to coarse in the future. One way of getting around this is to keep track of the latest timestamp written and make sure the next timestamp is one higher than the latest one. This way timestamps are always increasing, and I doubt that we can have modifications each nanosecond for a long time with the hardware available for the next ten years.

3. If time goes backwards beacuse somebody changes the clock, the caching (or makefile rules) can get confued. This is a hard one, but I actually we can use the doctors response "don't do that then". With NTP we can guarantee a monotonically increasing clock.

I belive the kernel could give some assistance to make it easier to see if a file has been modified, I remember that a few suggestions were thrown around the last time Samba and dcache aliases were discussed on l-k. I definitely belive that kind of infrastructure belongs in the kernel. But the cache manager itself, no.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#325 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christer Weinigel wrote:

| Pavel Machek <[email protected]> writes:
|
|
|>Okay, that does work, it just is not really nice. Just as reserving
|>fixed ammount of space for disk cache is bad, reserving fixed ammount
|>of space for ccache (and similar) is bad. When there are few of such
|>caches, balancing between them starts to matter...
|
|
| So try to convince them to use the same cache daemon or the same
| shared cache manager library then.  It isn't all that different from
| implementing a kernel interface that everyone is supposed to use.

You're right, it's not. Is there some huge performance loss by having a kernel interface that then turns around and calls a userland daemon? If yes, I can understand the argument. If no, then the kernel interface should ultimately be able to replace gnome-vfs -- and it should rely heavily on userland utilities.

Caching is very generic. It's so generic that there should be exactly one cache manager, the way there is exactly one VM manager for Linux. If there's a choice, choose it at boot or compile-time.

Abiword uses gzipped files. So does gnumeric, gnucash, cube (a cute little game engine), and many others. There's no way every single one will suddenly use a new library/daemon. But with plugins and intelligent caching, any kind of compression could be supported without the app needing to support it natively -- either it goes in a compressed folder or the user browses to it and changes an attribute.

| A cache manager daemon could sit and watch the free space on the disk
| every other second and start deleting the cached files (according to
| some LCU heuristics or whatever) whenever free space is getting low.

Agreed. The daemon should go in userspace. But the interface should be in the kernel. Just as no one suggested putting tar in the kernel, I don't think anyone would really insist that the daemon needs to go in the kernel.

That is, unless critical system files start actually being cached streams of the actual package from the distro... But this way leads to madness. Any serious issues here are better solved by an initrd.

[...]

| I belive the kernel could give some assistance to make it easier to
| see if a file has been modified, I remember that a few suggestions
| were thrown around the last time Samba and dcache aliases were
| discussed on l-k.  I definitely belive that kind of infrastructure
| belongs in the kernel.  But the cache manager itself, no.

Isn't there already dnotify?

Dnotify is wrong here, because I don't think it would be sane in a sufficiently large setup. But wouldn't it be nice if the kernel not only told you whether a file has been modified, but told you immediately after it was modified?

Yet another reason why there has to be a common cache and significant kernel management -- if I've got a drive of less than, say, 5 gigs, I am simply not willing to reserve any large amount of it for cache. Take the example of a source tarball being automatically extracted and the contents cached.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUpPHgHNmZLgCUhAQLhIg/9G87W4OCt8hOvTK+XniPgWGn+YC7t4fKy mi4QowX1eLmnFEovprjSUzPgKRMeDSChZ8M1pvXXWJQ/iULE44WH8vNiH8H974zw h5O8dNI/AczuzaCalQ+BYXEGM8pasaCGCI/sLrr+WBy97NOpM2VUUKY+qKxSFWFt x7u7/XCGGWSHVG3t3KzpkqD2K8u3IGSrV/vHIR2PLatEXu3G8K8sUAhfXMgFsTFp gQAuQN/YzQKGUphnMzFWLDMYyEYjAA4VACZFqFz9QJIYeWzP+YmGdLfZ5mzN91JF g5ACGHUxbYdjZsjTh6RbalVr1HpAAy8mvGNydbwUvuOxaXBvH44vUfRLnvIUoZs9 BB0Tq4xPqAwAtP0Z7b5Qem1ef3gqGXo1hfsEJWJZPv1lssYzn1+Jpn99p6N6VkXI fbAHnypbHeYbf30lC5lvID0yngb+s5EUaxzxOYqFA8QpEJD/bl3S9Qx6bpom/9AL U8kdK8QjNbhHSMrOpl8mg77VRLXdINoV7R5iSXXUMdIcA1j+4R1AIPh62JAMwQDF /Vzek8KNyxvJDgF/y78LiGETZcG/l2W4szEsJFFF4iIKLri5+OFGiLFz8jVNViKu Lr7Q7u7TgWSSykdTrHFcR/WRupAHeL5RB5DA9HBZErqgAgJWz/Rjv743G8PYFKs8 oPhRyePnFkE=
=cQLE
-----END PGP SIGNATURE-----

#326 lore raw
In reply to Christer Weinigel

Hi!

Quoted 5 lines

I belive the kernel could give some assistance to make it easier to see if a file has been modified, I remember that a few suggestions were thrown around the last time Samba and dcache aliases were discussed on l-k. I definitely belive that kind of infrastructure belongs in the kernel. But the cache manager itself, no.

Well, callback "filesystem full" would be nice, too; but I guess that is "support user cache", too...
Pavel

Signature · 1 line
When do you have heart between your knees?
#327 lore raw
In reply to Christer Weinigel

Salut,

Quoted 2 lines · Christer Weinigel

To see if the original file is newer than the cached file, good old mtime can be used.

I'd not use a daemon or any mtime for it. I'd just put it into xattrs and let it up to the applications to poke the values in case they changed. And copy them on cp, since it would have to be changed either way.

Tonnerre

#328 lore raw
In reply to Linus Torvalds
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Linus" == Linus Torvalds <[email protected]> writes:

[...]

Linus> I've seen all these examples of exposing MP3 ID information as a Linus> "side stream", and that's TOTALLY POINTLESS! The information is Linus> already there, it's in a standard format, and exporting it as a Linus> stream buys you absolutely nothing.

Exporting it through the filesystem would mean that I could use my favourite editor to edit MP3/OGG tags, instead of having to use some other program that has a piece-of-crap interface. It's a user-interface issue, and not a technical issue.

Someone might then tell me to write my own MP3-tag editor that has an interface that I like. How may MP3 tag editors do we need? Besides, I have better things to do with my time.

Actually, I would argue that things like ID3 tags for MP3 files, and EXIF for JPEG exist only because filesystems in general can't support associating metadata/attributes with files. If filesystems were built the "right way" to begin with, all those tagging schemes would never have come into existence. Of course, that's just a dream, but one can dream, can't they?

[...]

Linus> And realize that the "separate shadow tree" actually works very Linus> well. That's how version control systems like CVS have always Linus> worked.

VMS (or some other OS -- too lazy to look it up) exposed version control through the filesystem. Their naming sucked (or so says Pike), but that's what they did.

Yes, CVS works, but I wouldn't say that it works very well. If I want to get a specific version of a file, I need to drop to the command line and type in some cvs command that I don't remember and then open the file. I can't just do "C-x C-f foo.c/version/2.1" in emacs.

Yes, emacs has version-control support, so that I can open up a specific version. But if v-c isn't exposed through the filesystem, we would have to add support to emacs for every v-c system out there. And do the same for vi, gedit, nano, bluefish, conglomerate, Mozilla Composer, glade, etc.

Linus> It's certainly how you can make icon information work too. If you Linus> use a tool for accessing the data, the tool can maintain Linus> coherency and you'll never care about the side stream.

The point is that I shouldn't need to use a special tool. I should be able to use my favourite text/image/sound/etc. editor.

Using a special tool may work for you, but it isn't the optimal user interface in general.

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#329 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

In a graphical environment, the "icon" stream is a good example of this. It literally has _nothing_ to do with the data in the main stream. The only linkage is a totally non-technical one, where the user wanted to associate a secondary stream with the main stream _without_ altering the main one. THAT is where named streams make sense.

I think that the "icon" argument for named streams is a silly argument, since different users may want to have different icons for the same file. Say that I want /usr/bin/emacs to have the enterprise icon and someone else wants the gnu head icon. And besides, root owns the file anyways, so neither of us mortal users should be able to add a stream to it.

Another reason for named streams that usually crops up is the ability set a "preferred application" for a certain file, so that when I double click on a document I want to open it with antiword instead of openoffice. But the same contra-argument applies here, different users have different preferences.

I can see the argument for having the equivalent of Content-type or Content-transfer-encoding as a named stream though.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#330 lore raw
Quoted 11 lines
Quoted 5 lines · Linus Torvalds

In a graphical environment, the "icon" stream is a good example of this. It literally has _nothing_ to do with the data in the main stream. The only linkage is a totally non-technical one, where the user wanted to associate a secondary stream with the main stream _without_ altering the main one. THAT is where named streams make sense.

Quoted 6 lines

I think that the "icon" argument for named streams is a silly argument, since different users may want to have different icons for the same file. Say that I want /usr/bin/emacs to have the enterprise icon and someone else wants the gnu head icon. And besides, root owns the file anyways, so neither of us mortal users should be able to add a stream to it.

Yet again are we thinking in blocking ways. Firstly this was an example. Usually, though, most users accept the default icon for a file. If they do not they can still change the icon for the link they make on their start-menu/home folder/etc.

Quoted 5 lines

Another reason for named streams that usually crops up is the ability set a "preferred application" for a certain file, so that when I double click on a document I want to open it with antiword instead of openoffice. But the same contra-argument applies here, different users have different preferences.

I can make the same argument as for the icons.

Quoted 2 lines

I can see the argument for having the equivalent of Content-type or Content-transfer-encoding as a named stream though.

That would be a nice thing.

Quoted 1 line

/Christer

#331 lore raw
In reply to Christer Weinigel
Quoted 8 lines · Christer Weinigel

I think that the "icon" argument for named streams is a silly argument, since different users may want to have different icons for the same file. Say that I want /usr/bin/emacs to have the enterprise icon and someone else wants the gnu head icon. And besides, root owns the file anyways, so neither of us mortal users should be able to add a stream to it.

Quoted 8 lines

Another reason for named streams that usually crops up is the ability set a "preferred application" for a certain file, so that when I double click on a document I want to open it with antiword instead of openoffice. But the same contra-argument applies here, different users have different preferences.

I can see the argument for having the equivalent of Content-type or Content-transfer-encoding as a named stream though.

You still have the same problem with that. What if I want to override the content-type for some reason. (I have had to do this is the past, but not for root owned files. In the end though I had to edit /etc/mailcap to do what I wanted :( )

You still have problems with icons. What format should they be stored in? If all this effort is to make samba work then would this mean a windows icon? That is not going to make your X11 app happy.

This could probably all be solved with magic plugins in the filesystem (reiserfs style) but it is probably better to return to simple approach of "everything is a file". The executable /usr/bin/emacs is a file. The way it looks on my desktop is a file ~/.desktop/emacs etc. This is what most apps do currently and it works quite well.

Just my 2c

Signature · 5 lines
John Newbigin - Computer Systems Officer
School of Information Technology
Swinburne University of Technology
Melbourne, Australia
http://www.it.swin.edu.au/staff/jnewbigin
#332 lore raw
In reply to Christer Weinigel

Salut,

Quoted 2 lines · Christer Weinigel

I can see the argument for having the equivalent of Content-type or Content-transfer-encoding as a named stream though.

Why having them as named streams if we can get them as xattrs for no additional pain? (Since fileutils would have to be changed anyway, we can even make cp copy and emacs update xattrs.)

Tonnerre

#333 lore raw
Quoted 9 lines · Tonnerre

Salut,

Quoted 2 lines · Christer Weinigel

I can see the argument for having the equivalent of Content-type or Content-transfer-encoding as a named stream though.

Why having them as named streams if we can get them as xattrs for no additional pain? (Since fileutils would have to be changed anyway, we can even make cp copy and emacs update xattrs.)

Because some of as requested to be able to open tar,iso, and few other formats this way. So I can simply access it with cat (as a dir/file).

Signature · 1 line
GJ
#334 lore raw
In reply to Tonnerre
Quoted 5 lines

Salut,

Quoted 7 lines
Quoted 2 lines · Christer Weinigel

I can see the argument for having the equivalent of Content-type or Content-transfer-encoding as a named stream though.

Why having them as named streams if we can get them as xattrs for no additional pain? (Since fileutils would have to be changed anyway, we can even make cp copy and emacs update xattrs.)

What if I do not use emacs, but vim, mcedit, gedit, or some other editor? It doesn't seem logical to have to patch every application that uses files.

~S

Quoted 1 line

Tonnerre

#335 lore raw

Salut,

Quoted 3 lines · Spam

What if I do not use emacs, but vim, mcedit, gedit, or some other editor? It doesn't seem logical to have to patch every application that uses files.

We would have to do that in either case, so let's patch them to do it in a nonintrusive way. And as to reading and writing inside tar files, write and/or use a really nice userspace library to do it. (As does MacOS/X, as does KDE, etc.)

Tonnerre

#336 lore raw
Quoted 6 lines

Salut,

On Sun, Sep 05, 2004 at 01:57:49PM +0200, Spam wrote:

Quoted 8 lines
Quoted 3 lines

What if I do not use emacs, but vim, mcedit, gedit, or some other editor? It doesn't seem logical to have to patch every application that uses files.

We would have to do that in either case, so let's patch them to do it in a nonintrusive way. And as to reading and writing inside tar files, write and/or use a really nice userspace library to do it. (As does MacOS/X, as does KDE, etc.)

No, not with the current setup with file-as-dir. It also works in Gnome/Natilus. Just tested:

mkdir test
chmod +x test
cd test
dir metas
echo 0777>rwx

In Nautilus I just typed in in the address bar /test/metas/ I could open meta-data and change with GEdit too.

I do not see that I have had to patch any programs to get this to work.

~S

Quoted 1 line

Tonnerre

#337 lore raw
In reply to Tonnerre
Quoted 6 lines

Salut,

On Sun, Sep 05, 2004 at 01:57:49PM +0200, Spam wrote:

Quoted 8 lines
Quoted 3 lines

What if I do not use emacs, but vim, mcedit, gedit, or some other editor? It doesn't seem logical to have to patch every application that uses files.

We would have to do that in either case, so let's patch them to do it in a nonintrusive way. And as to reading and writing inside tar files, write and/or use a really nice userspace library to do it. (As does MacOS/X, as does KDE, etc.)

The problem with the userspace library is standardization. What would be needed is a userspace library that has a extensible plugin interface that is standardized. Otherwise we would need lots of different libraries, and I seriously doubt that 1) this will happen and 2) we get all Linux programs to be patched to use it.

~S

Quoted 1 line

Tonnerre

#338 lore raw

Hi!

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

What if I do not use emacs, but vim, mcedit, gedit, or some other editor? It doesn't seem logical to have to patch every application that uses files.

We would have to do that in either case, so let's patch them to do it in a nonintrusive way. And as to reading and writing inside tar files, write and/or use a really nice userspace library to do it. (As does MacOS/X, as does KDE, etc.)

Quoted 5 lines

The problem with the userspace library is standardization. What would be needed is a userspace library that has a extensible plugin interface that is standardized. Otherwise we would need lots of different libraries, and I seriously doubt that 1) this will happen and 2) we get all Linux programs to be patched to use it.

libvfs from midnight commander (and anything build on the top of it) already is very extensible.
Pavel

#339 lore raw
Quoted 14 lines

Hi!

Quoted 9 lines
Quoted 8 lines
Quoted 3 lines

What if I do not use emacs, but vim, mcedit, gedit, or some other editor? It doesn't seem logical to have to patch every application that uses files.

We would have to do that in either case, so let's patch them to do it in a nonintrusive way. And as to reading and writing inside tar files, write and/or use a really nice userspace library to do it. (As does MacOS/X, as does KDE, etc.)

Quoted 8 lines
Quoted 5 lines

The problem with the userspace library is standardization. What would be needed is a userspace library that has a extensible plugin interface that is standardized. Otherwise we would need lots of different libraries, and I seriously doubt that 1) this will happen and 2) we get all Linux programs to be patched to use it.

libvfs from midnight commander (and anything build on the top of it) already is very extensible.

This wasn't my point. My point was about all applications using it. Most aren't.

Quoted 1 line

Pavel

#340 lore raw
Quoted 11 lines · Spam
Quoted 8 lines
Quoted 5 lines

The problem with the userspace library is standardization. What would be needed is a userspace library that has a extensible plugin interface that is standardized. Otherwise we would need lots of different libraries, and I seriously doubt that 1) this will happen and 2) we get all Linux programs to be patched to use it.

libvfs from midnight commander (and anything build on the top of it) already is very extensible.

This wasn't my point. My point was about all applications using it. Most aren't.

So why do you think that just because you put an interface into the everyone will automatically start using it and use it without any conflicts (i.e. is foo/icon a windows icon, a gif, png, or jpeg file, what size is it, 16x16, 48x48...).

When you add a new shared namespace, userspace must be taught about it anyways. So start with a userspace library, convince people to use it, and when you know what people want, _then_ put it into the kernel to increase performance or security. Don't start with "wouldn't it be nice if" without any thought behind it, it will just mean that we get another useless and misdesigned interface in the kernel that we have to live with for years.

Do you know how long it took to get rid of /dev/cua0 and the locking mess associated with it? (Deity! I just checked on my FC2 system, it's still there. I thought it would be dead by now.)

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#341 lore raw
Quoted 22 lines
Quoted 11 lines · Spam
Quoted 8 lines
Quoted 5 lines

The problem with the userspace library is standardization. What would be needed is a userspace library that has a extensible plugin interface that is standardized. Otherwise we would need lots of different libraries, and I seriously doubt that 1) this will happen and 2) we get all Linux programs to be patched to use it.

libvfs from midnight commander (and anything build on the top of it) already is very extensible.

This wasn't my point. My point was about all applications using it. Most aren't.

So why do you think that just because you put an interface into the everyone will automatically start using it and use it without any conflicts (i.e. is foo/icon a windows icon, a gif, png, or jpeg file, what size is it, 16x16, 48x48...).

Quoted 4 lines

When you add a new shared namespace, userspace must be taught about it anyways. So start with a userspace library, convince people to use it, and when you know what people want, _then_ put it into the kernel to increase performance or security.

Several aspects of this;

1) Programs will need to actually be coded against this shared library which probably will be more advanced that just usning normal file access code.

2) Then we redo it all again to fit in the kernel, as a kernel plugin or user-level plugin.

Quoted 4 lines

Don't start with "wouldn't it be
nice if" without any thought behind it, it will just mean that we get another useless and misdesigned interface in the kernel that we have to live with for years.

I am not saying that there should not be any thought behind implementing something. But from many hear I just hear a big no, without any thought about it either.

Named streams and meta files are just one example of a general access to extra data that belongs to files. It seem to me as a much simpler way for program to use those than to link in several various libraries to do the same thing.

Plugins were just another thing that could extend the functionality of these streams and meta data. Reiser4 has a plugin architecture, although not yet any run-time support for loading them. Is this so bad that we have to prevent it?

No one has to use them. maybe at some point it would be possible to specify which classes of plugins that could be loaded by a user and his files.

What are the complaints about. The technical challenge to make things like this secure, or the concept to have this functionality at all. I do not think that because something is hard to do, it should be dismissed.

~S

Quoted 5 lines

Do you know how long it took to get rid of /dev/cua0 and the locking mess associated with it? (Deity! I just checked on my FC2 system, it's still there. I thought it would be dead by now.)

/Christer

#342 lore raw
Quoted 10 lines · Spam
Quoted 4 lines

When you add a new shared namespace, userspace must be taught about it anyways. So start with a userspace library, convince people to use it, and when you know what people want, _then_ put it into the kernel to increase performance or security.

Several aspects of this;

1) Programs will need to actually be coded against this shared library which probably will be more advanced that just usning normal file access code.

Not at all. As many others have already shown, trying to use files as directories won't work since then you can't have named streams or attributes for real directories. What does foo/icon refer to when foo is a directory? There must be some way of differentiating between them, which in turn means that we need a new way of referring to the stream. So the simple, hackish, solution really won't fly.

Besides, it will probably break a lot of security critical assumptions in todays software. What if you have a filter in your web server that excludes executable files or files that end with .php, can you imagine the mess if you could just open "foo.php/main-stream" instead?

Quoted 2 lines

2) Then we redo it all again to fit in the kernel, as a kernel plugin or user-level plugin.

No, we use the same user space library, but let the user space library use helper functions in the kernel.

We could for example add a function that opens a named stream based on a filename which uses a .xattr directory in the same directory as the file, or in database in our home directory. If we add a solaris-like openat(3) syscall we modify the library to take advantage of it.

Quoted 4 lines

Plugins were just another thing that could extend the functionality of these streams and meta data. Reiser4 has a plugin architecture, although not yet any run-time support for loading them. Is this so bad that we have to prevent it?

Take an example: "expose a tar-file as streams below the file" which as been suggested here is IMNSHO plain silly. I'm not saying anything about mounting a tar file via userfs somewhere else in the file system, that is quite ok, but trying to mount it on top of the same file which suddenly and automagically turns into a sort-of-directory breaks too many thing. Let your file manager do the choice instead, based on the users preferences. For example, with a tar.gz-file, I'd like to have a choice of "open file as a seekable file" which would do:

mount -t userfs -o driver=gunzip foo.tar.gz /tmp/xyzzy

Then I can work with /tmp/xyzzy as a normal file (maybe even with write access if the userfs driver can handle this). Another choice in the same file manager would be "open file as a directory" which would mount the same file in _another place_ as a directory, and I can even have different views of the same file mounted at the same time. With the named streams junk that have been suggested here, having two different views would be impossible.

Sure, we could say that we add another level of indirection to the named streams, so that we specify the driver as the first component of te magic file-as-directory, i.e. foo.tar.gz/ungzipped would refer to the ungzipped stream and foo.tar.gz/ungzipped-and-untarred would show the tar file as a directory, but really, this isn't any more useful than doing a userfs mount. The userfs mount does not break existing semantics (anymore than mount -o loop does today), and it will work with the existing infrastructure in the linux kernel. The only advantage of files-as-directories with magic plugins in the kernel is that one can look at it and say "look, how neat, the filenames look almost the same".

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#343 lore raw

Hi!

Quoted 11 lines

Sure, we could say that we add another level of indirection to the named streams, so that we specify the driver as the first component of te magic file-as-directory, i.e. foo.tar.gz/ungzipped would refer to the ungzipped stream and foo.tar.gz/ungzipped-and-untarred would show the tar file as a directory, but really, this isn't any more useful than doing a userfs mount. The userfs mount does not break existing semantics (anymore than mount -o loop does today), and it will work with the existing infrastructure in the linux kernel. The only advantage of files-as-directories with magic plugins in the kernel is that one can look at it and say "look, how neat, the filenames look almost the same".

Who is going to umount it when application crashes, etc? Plus mount required root priviledges last time I checked.

Pavel

#344 lore raw
Quoted 1 line · Pavel Machek

Who is going to umount it when application crashes, etc?

Who removes temporary files left behind when an application crashes? I'd guess a daemon such as autofs could do a very good job here.

Quoted 1 line

Plus mount required root priviledges last time I checked.

bash$ ls -l /bin/mount
-rwsr-xr-x 1 root root 78504 May 4 23:34 /bin/mount

with proper policies in userspace it allows users to perform mounts.

I'm not suggesting that the kernel should be unchanged, but really, some of the proposals here, to put a hell of a lot of complexity into the kernel it just wet dreams with not much thought of how it affects the kernel. What happened to the philosophy of putting complexity and policy in userspace? Look at khttpd and tux, they were hacks in the kernel to try things out. But what ended up in the kernel is generic infrastructure that is useful for a lot more applications than just a web server. That is the right way to do things.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#345 lore raw

Hi!

Quoted 6 lines
Quoted 1 line

Plus mount required root priviledges last time I checked.

bash$ ls -l /bin/mount
-rwsr-xr-x 1 root root 78504 May 4 23:34 /bin/mount

with proper policies in userspace it allows users to perform mounts.

You want to do exec() each time you enter a archive? I do not think that is good idea.

Quoted 8 lines

I'm not suggesting that the kernel should be unchanged, but really, some of the proposals here, to put a hell of a lot of complexity into the kernel it just wet dreams with not much thought of how it affects the kernel. What happened to the philosophy of putting complexity and policy in userspace? Look at khttpd and tux, they were hacks in the kernel to try things out. But what ended up in the kernel is generic infrastructure that is useful for a lot more applications than just a web server. That is the right way to do things.

Well, generic infrastructure for uservfs would be enough for me... And it is few lines in kernel.
Pavel

#346 lore raw
In reply to Pavel Machek

Salut,

Quoted 1 line · Pavel Machek

Who is going to umount it when application crashes, etc?

This has been discussed along with the HAL people a while
ago. Actually, file systems can introduce a refcount, where we need a decrement function which automatically unmounts the filesystem if we decrement the use count to zero. Kind of an automatic umount.

How do you tell which file systems shall be autoumounted? HAL
suggestion: introduce a no-op mount option. For some classes of fs'es we might as well make that obligatory, such as tar.

Tonnerre

#347 lore raw
Quoted 4 lines · Tonnerre

This has been discussed along with the HAL people a while
ago. Actually, file systems can introduce a refcount, where we need a decrement function which automatically unmounts the filesystem if we decrement the use count to zero. Kind of an automatic umount.

We've supported file system garbage collection when they become unused since umm about 2.4.10 I think.

#348 lore raw
In reply to Christer Weinigel
Quoted 16 lines
Quoted 5 lines

Plugins were just another thing that could extend the functionality of these streams and meta data. Reiser4 has a plugin architecture, although not yet any run-time support for loading them. Is this so bad that we have to prevent it?

Take an example: "expose a tar-file as streams below the file" which as been suggested here is IMNSHO plain silly. I'm not saying anything about mounting a tar file via userfs somewhere else in the file system, that is quite ok, but trying to mount it on top of the same file which suddenly and automagically turns into a sort-of-directory breaks too many thing. Let your file manager do the choice instead, based on the users preferences. For example, with a tar.gz-file, I'd like to have a choice of "open file as a seekable file" which would do:

Quoted 1 line

mount -t userfs -o driver=gunzip foo.tar.gz /tmp/xyzzy

Where is the difference? Both would be handled by a specific driver or module and export the same semantics (files+dirs) with permissions etc to the user. With your idea you still would need the userfs kernel module, and with "magic plugins", as you said, you will need a vfs/reiser4 kernel plugin.

Quoted 8 lines

Then I can work with /tmp/xyzzy as a normal file (maybe even with write access if the userfs driver can handle this). Another choice in the same file manager would be "open file as a directory" which would mount the same file in _another place_ as a directory, and I can even have different views of the same file mounted at the same time. With the named streams junk that have been suggested here, having two different views would be impossible.

Quoted 11 lines

Sure, we could say that we add another level of indirection to the named streams, so that we specify the driver as the first component of te magic file-as-directory, i.e. foo.tar.gz/ungzipped would refer to the ungzipped stream and foo.tar.gz/ungzipped-and-untarred would show the tar file as a directory, but really, this isn't any more useful than doing a userfs mount. The userfs mount does not break existing semantics (anymore than mount -o loop does today), and it will work with the existing infrastructure in the linux kernel. The only advantage of files-as-directories with magic plugins in the kernel is that one can look at it and say "look, how neat, the filenames look almost the same".

No there are _usability_ differences. I cirtanly do not want to mount lots of files with mount -t userfs, just to see extra meta-data that I want to quickly be able to use. And it also wouldn't work generically (searchable) with tools like find, grep, etc either.

~S

Quoted 1 line

/Christer

#349 lore raw
In reply to Christer Weinigel

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christer Weinigel wrote:

| Spam <[email protected]> writes:
|
|
|>>When you add a new shared namespace, userspace must be taught about it
|>>anyways.  So start with a userspace library, convince people to use
|>>it, and when you know what people want, _then_ put it into the kernel
|>>to increase performance or security.
|>
|>  Several aspects of this;
|>
|>  1) Programs will need to actually be coded against this shared
|>  library which probably will be more advanced that just usning normal
|>  file access code.
|
|
| Not at all.  As many others have already shown, trying to use files as
| directories won't work since then you can't have named streams or
| attributes for real directories.  What does foo/icon refer to when foo

Oh yes, you can. That's what "metas" is for. I wanted to call it ... | is a directory? There must be some way of differentiating between

When foo is a directory, foo/icon is a normal file. But foo/metas/icon (or foo/.../icon if you like) is the metadata.

This already works. I can use it right now for file permissions:

echo 755 > /bin/bash/metas/rwx
echo 755 > ./metas/rwx

| Besides, it will probably break a lot of security critical assumptions
| in todays software.  What if you have a filter in your web server that
| excludes executable files or files that end with .php, can you imagine
| the mess if you could just open "foo.php/main-stream" instead?

Why would you have a foo.php/main-stream?

And what would be so hard about allowing read access to foo, but not access to foo's metas? This is essentially the same as having mode 6 on a directory.

[...]

|>  Plugins were just another thing that could extend the functionality
|>  of these streams and meta data. Reiser4 has a plugin architecture,
|>  although not yet any run-time support for loading them. Is this so
|>  bad that we have to prevent it?
|
|
| Take an example: "expose a tar-file as streams below the file" which
| as been suggested here is IMNSHO plain silly.  I'm not saying anything
| about mounting a tar file via userfs somewhere else in the file
| system, that is quite ok, but trying to mount it on top of the same
| file which suddenly and automagically turns into a sort-of-directory
| breaks too many thing.  Let your file manager do the choice instead,

What things? It's just dangerously different thinking. I don't see it breaking anything on my system right now. If anything, the nvidia binary drivers are a bigger issue...

| based on the users preferences.  For example, with a tar.gz-file, I'd
| like to have a choice of "open file as a seekable file" which would
| do:
|
|     mount -t userfs -o driver=gunzip foo.tar.gz /tmp/xyzzy
|
| Then I can work with /tmp/xyzzy as a normal file (maybe even with
| write access if the userfs driver can handle this).  Another choice in
| the same file manager would be "open file as a directory" which would
| mount the same file in _another place_ as a directory, and I can even
| have different views of the same file mounted at the same time.  With
| the named streams junk that have been suggested here, having two
| different views would be impossible.
|
| Sure, we could say that we add another level of indirection to the
| named streams, so that we specify the driver as the first component of
| te magic file-as-directory, i.e. foo.tar.gz/ungzipped would refer to
| the ungzipped stream and foo.tar.gz/ungzipped-and-untarred would show
| the tar file as a directory, but really, this isn't any more useful
| than doing a userfs mount.  The userfs mount does not break existing

Yes, it is more useful than doing a userfs mount. The userfs mount has to be done manually or by a file manager. I do not use a file manager. ~ I am not alone here. Unless you're going to have the entire FS mounted userfs and have that be the file manager, which wouldn't be nearly as fast as plugins.

I've heard autofs mentioned -- which only works on specific files which it's been preconfigured for. Not good here.

| semantics (anymore than mount -o loop does today), and it will work | with the existing infrastructure in the linux kernel. The only

That's true -- it's easier to implement if you ignore the number of apps which need patching.

There's also the fact that most people here won't be patching apps, they'll just be breathing a sigh of relief that they didn't have to patch the kernel.

Just because someone else has to do it doesn't make it any less work.

| advantage of files-as-directories with magic plugins in the kernel is
| that one can look at it and say "look, how neat, the filenames look
| almost the same".

And this is a usability issue. Not all newbies use Nautilus. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTy3MngHNmZLgCUhAQJ/lQ//UmQOJLZAmsFFD8TJehD8WitMrImrcrBC SJ/mCkIaPq5JNo9R/ovd1UoGlf47JUvaaT5G1d/yOMA+/0kPNqjJL49Km9A3Ze+C 00Uq12yP8yp6NjBzq9HKAjvyHVghl1A5ZllIdiwuc8Fywbl6UKUHlN1D4PpUp1uA znKPjmGpBOsTzgNWJYbAfjns60C7DHZoD/S+ldt3af2PsJKOpx0v4fusV8Y2+XRG kWybZLl1YPwPLfJLCaa3KpbY1KkSuk6TSeut90+zRV6s1WJ8LE98NpDBGUDt/+cm AXcp/zyrUmbpvQtZBaUFYwZ8imizspiSHYnBwdhVm9tfERFOFI9diBidvanHk/s0 D+FSJhnQSeo/hMnbHlfv4E6p4k1jDvJk06XJ/U45sEym4hCp9roDPB0tUic7bOnE mxNxzogp92KHfsVPuqkidq255ztAjRbzdnt8zsBOPImqTgfsO3xVfT3svIixNxWv +qaAlrR7dAri9bi3wNLCjzRhp3id0fspetLJrrCHI/8hMlu7a6nsIHLocuvG069W OzdXtTPRYsV8YlElsdL5d/3grZSL3CWNpQxoRb9gzj0hKHHPjrXw9ACcQthiknNc DkjjcLxV2gPqhKq3WgMTKXHhLDWEfJpFVQ8PccJHsKaaUgGtDUg0ETBT54h8Mo8E I0bkcYOLtHk=
=o3vQ
-----END PGP SIGNATURE-----

#350 lore raw
In reply to Spam

Spam <[email protected]> said:

[...]

Quoted 5 lines

The problem with the userspace library is standardization. What would be needed is a userspace library that has a extensible plugin interface that is standardized. Otherwise we would need lots of different libraries, and I seriously doubt that 1) this will happen and 2) we get all Linux programs to be patched to use it.

What is the difference with a kernel implementation? Not by being in-kernel will it make all the incompatible ways of doing this magically vanish, and give outstanding performance. Plus handling and maintaining the in-kernel stuff is _much_ harder than userspace libraries.

I'd go the other way around: Get userspace to agree on a common framework, make it work in userspace; if (extensive, hopefully) experience shows that a pure userspace solution has issues that can't be solved except by kernel assistance, so be it.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#351 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:

| Spam <[email protected]> said:
|
| [...]
|
|
|>  The problem with the userspace library is standardization. What
|>  would be needed is a userspace library that has a extensible plugin
|>  interface that is standardized. Otherwise we would need lots of
|>  different libraries, and I seriously doubt that 1) this will happen
|>  and 2) we get all Linux programs to be patched to use it.
|
|
| What is the difference with a kernel implementation? Not by being
in-kernel
| will it make all the incompatible ways of doing this magically vanish, and
| give outstanding performance. Plus handling and maintaining the in-kernel
| stuff is _much_ harder than userspace libraries.

First of all, only the interface has to be in the kernel. I haven't heard anyone suggest otherwise.

Second, there are quite a few things which I might want to do, which can be done with this interface and without patching programs, but would require massive patches to userspace. There have been numerous examples.

There are some things which can't be solved without patching. Version control is one such thing. But then there can be more generic patches
- -- as soon as the transaction API is done, you only have to patch apps to use that, and have a version control reiser4 plugin.

| I'd go the other way around: Get userspace to agree on a common framework,
| make it work in userspace; if (extensive, hopefully) experience shows that
| a pure userspace solution has issues that can't be solved except by kernel
| assistance, so be it.

We already have such a framework -- it's called "VFS". -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTyyGXgHNmZLgCUhAQLUeQ/8D3tWL9l/zQeGylpVbOe6SkcSPOOmlIFR tcjh0y1Y4ET17ATFKbKbzQYYDgd49AqU/gZnro27jYun3Yi6U0fWGGQFfi1A9O5E gSCmsjSWjfDfx4gu3EU1x0Bhkd6Mo8GCrC7L5gj5C+L4c5ZAnffeGloF8nM4hCex Wsb0PgOSxXuoQcd2EELVEXYdq0RCnxrmuszl1B2SE6w1ImONWMoXJ9fDGDf0aIUu rwrrZnlH4zp0bQ0dXDGXqUYYT5h5DAhbh96IWLrPbWMB0vLBqIP+95P2/vTHb7EL RwVKBV1UuuZ2ANPbImoIuxHWF+PCx/HwFs/mUolw0D2Yn3u2HgmPVFemyPnlCfeX yGPhJgnieRuGntgUZcfbqk7ZO3y0y5eRDq6N4eMHMlWYV9LC5kyP7OxcQ8SAF8P6 Sk4iylYN1AMiy5Bp9odScauST0NT9CLmi1Ps0DYwgVN1H+ldS1l+4ITokb1Ex1+4 ZLq1HhPNaYYWoA4VPuwxl0XrB4wGrMbOt1w4+TNM3AG9MvzqTGgSrh2rXfXkPHGZ 7LNHuinRyJt3dcF0vPS4WHG6FtVsO8XVPaY55tYQIYZEBtZl3mattBb9gM3WDJmw M/pxbAQTDZHloR9/7TGEF8gD3AjPBexTfvojqVHK2VvRu4/2Ku17wvK82v68LuyU bFxxxgj9IgY=
=BxAT
-----END PGP SIGNATURE-----

#352 lore raw
Quoted 4 lines · David Masover

Second, there are quite a few things which I might want to do, which can be done with this interface and without patching programs, but would require massive patches to userspace. There have been numerous examples.

Please give one example? There is a lot of handwaving here on all sides.

Quoted 6 lines
| I'd go the other way around: Get userspace to agree on a common framework,
| make it work in userspace; if (extensive, hopefully) experience shows that
| a pure userspace solution has issues that can't be solved except by kernel
| assistance, so be it.

We already have such a framework -- it's called "VFS".

That you'd like to break the semantics of. Great.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#353 lore raw
In reply to David Masover
Quoted 19 lines · David Masover

Horst von Brand wrote:

| Spam <[email protected]> said:
|
| [...]
|
|
|>  The problem with the userspace library is standardization. What
|>  would be needed is a userspace library that has a extensible plugin
|>  interface that is standardized. Otherwise we would need lots of
|>  different libraries, and I seriously doubt that 1) this will happen
|>  and 2) we get all Linux programs to be patched to use it.
|
|
| What is the difference with a kernel implementation? Not by being
| in-kernel will it make all the incompatible ways of doing this
| magically vanish, and give outstanding performance. Plus handling
| and maintaining the in-kernel stuff is _much_ harder than userspace
| libraries.
Quoted 2 lines

First of all, only the interface has to be in the kernel. I haven't heard anyone suggest otherwise.

Right. But it is _another_ interface in the kernel, plus special userland code supporting it.

Quoted 2 lines

Second, there are quite a few things which I might want to do, which can be done with this interface and without patching programs,

Such as?

Quoted 2 lines

but would
require massive patches to userspace. There have been numerous examples.

Haven't seen any that made sense to me, sorry.

Quoted 1 line

There are some things which can't be solved without patching.

Maybe. Question is, is it worth it (kernel modifications + userland support, or just userland support, or leave it alone). Sure, it might make your particular application easier to write (at a cost for _all_ filesystem hackers!), perhaps even a bit faster; but is _your_ particular convenience worth the cost for _everybody_?

Quoted 2 lines

Version
control is one such thing.

bk, cvs, svn, rcs, ... are working just fine here, thank you so much. Used to work on SunOS and Solaris, even SCO Unix (I used at least rcs and cvs there). No Reiser4 in sight.

Quoted 3 lines

But then there can be more generic patches
- -- as soon as the transaction API is done, you only have to patch apps to use that, and have a version control reiser4 plugin.

Again, _what_ version control exactly? Will the above packages be able to make use of it (remember they all are cross-platform (at least cross-Unix), and so quite unlikely to make use of a Reiser4 on Linux whackiness...)?

Quoted 6 lines
| I'd go the other way around: Get userspace to agree on a common framework,
| make it work in userspace; if (extensive, hopefully) experience shows that
| a pure userspace solution has issues that can't be solved except by kernel
| assistance, so be it.

We already have such a framework -- it's called "VFS".

Right. It offers what applications need to build their own stuff. It is minimalistic (well, sort of) and time-proven.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#354 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:
[...]

|>Second, there are quite a few things which I might want to do, which can
|>be done with this interface and without patching programs,
|
|
| Such as?

They've been mentioned.

|
|
|>                                                           but would
|>require massive patches to userspace.  There have been numerous examples.
|
|
| Haven't seen any that made sense to me, sorry.

Sorry if they don't make sense to you, but I don't feel like discussing them now. Either you get it or you don't, either you agree or you don't. Read the archives.

|
|
|>There are some things which can't be solved without patching.
|
|
| Maybe. Question is, is it worth it (kernel modifications + userland
| support, or just userland support, or leave it alone). Sure, it might make
| your particular application easier to write (at a cost for _all_

filesystem
| hackers!), perhaps even a bit faster; but is _your_ particular convenience | worth the cost for _everybody_?

There are far more userland developers than filesystem hackers. If you're a filesystem hacker, it's easier to understand how much work it is to do something in the filesystem/kernel, and harder to understand how it works in userland, or for the actual user.

|
|
|>                                                               Version
|>control is one such thing.
|
|
| bk, cvs, svn, rcs, ... are working just fine here, thank you so much. Used
| to work on SunOS and Solaris, even SCO Unix (I used at least rcs and cvs
| there). No Reiser4 in sight.

Transparently?

It _works_ to do
zcat file.gz > /tmp/file
vim /tmp/file
gzip -c /tmp/file > file.gz
rm /tmp/file
You can even do that as a script, call it zvim. You could even do it as a generic script, where "vim" is replaced with "$1". But is it as elegent as transparently compressed files?

I'm not sure about the version control thing -- I don't think people have hit every conceptual issue about it yet. But the point is:

Moving complexity from kernel space to user space (or the other way) doesn't make it any less complex.

|>                            But then there can be more generic patches
|>- -- as soon as the transaction API is done, you only have to patch apps
|>to use that, and have a version control reiser4 plugin.
|
|
| Again, _what_ version control exactly? Will the above packages be able to
| make use of it (remember they all are cross-platform (at least

cross-Unix),
| and so quite unlikely to make use of a Reiser4 on Linux whackiness...)?

Probably. It wasn't specified what version control would be used -- that's currently just as abstract as what compression algorithm to use to transparently compress files.

Maybe something new. Version control just doesn't look that complex once you've got the interface and the data storage done. I do incremental backups using rsync and hardlinks -- why would version control be so much more complex than that?

Said backup system uses hardlinks, btw, which not all systems support. And rsync is fairly broken on some platforms.

|>| I'd go the other way around: Get userspace to agree on a common framework,
|>| make it work in userspace; if (extensive, hopefully) experience shows that
|>| a pure userspace solution has issues that can't be solved except by kernel

|>| assistance, so be it.
|
|
|>We already have such a framework -- it's called "VFS".
|
|
| Right. It offers what applications need to build their own stuff. It is
| minimalistic (well, sort of) and  time-proven.

x86 assemby is minimalistic and time-proven.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQT1MGHgHNmZLgCUhAQLxoQ/+Mglz6tr0A4wQmAnP1y9q7WVqQQ8s3Cwg b3VyOHaIIZSw3lod+bWDgE9LJJsQOxO4SThlWLyMEHipFJT0iHVIYjLSrC5e4DOb Wl5E51yz3ZVx11mRTM7uEw+Ez6wOcaWIFLTvarDxzmDaxS6oh2ZpN2Ibnf8h/1lT 0YJnz6C964TkaA8jYIsQljIWkMMk7EGzP6UlrQOBbo4xMJNT+wIZMJWX5JAsULc2 EGzTO8dgZB0MbJ5STufS4h5tudny/lz4TO9iTv9CRGhusf2am7k7PVFPG4RcjXKU iMgLTHKsXbDjncj39JZqwQtBZP8nuf72pMzbEe5iiYYIHGWu4Mm/JSxUmuJZ/YXX yZIS6DKwSrKzDf/p1chvLVCaScxsaIWuetoe4ODFUoWMRUfCdxK+r7+6j9tKn+hn LK5iYVAs3/xpU64jpBKvrKlRlISm8++GvGD+tdnZKAnetmaS0QHb2DrbwSbONvC1 4RvABUYC2IoVoDuAsueRDQxqTJjGPWn7DBvVUI5SCQZVlJPMavHmrv9qRVtNA4Y4 LBzfYK6aCprbmmX2Axs8FS4ptNGdwGpUhwuVKfSzlOgUS5gIaWlMKOGxWOTgDx7U Q53mVhdZIinHH+/h4xBpcP3Q1fk8nTQ1gqfYVTseNlrMZvgFAax7E2VINiJWohis KH6z9bFgkZA=
=aNM7
-----END PGP SIGNATURE-----

#355 lore raw

I just want to remind that no one has been able to offer a good way of handling attributes/streams/metafiles other than reiser4 in which CTRL-XCTRL-Ffilename within emacs to edit the stream/attribute/metafile will work without modifying the emacs source. David Masover is right that there are far more application writers than kernel hackers, and we should make the kernel more complicated if it makes a few thousand apps simpler.

#356 lore raw
Quoted 7 lines · Hans Reiser

I just want to remind that no one has been able to offer a good way of handling attributes/streams/metafiles other than reiser4 in which CTRL-XCTRL-Ffilename within emacs to edit the stream/attribute/metafile will work without modifying the emacs source. David Masover is right that there are far more application writers than kernel hackers, and we should make the kernel more complicated if it makes a few thousand apps simpler.

This thread is getting a bit soft on technical details and/or value.

Hans, please post the results of fsstress (the bits from ext3 cvs) on reiser4 on SMP machines of non-i386 architectures (e.g. ppc64, sparc64, ia64) as well as the results of transferring reiser4 filesystems between machines whose PAGE_SIZE, wordsize and endianness vary.

-- wli

#357 lore raw
Quoted 5 lines · William Lee Irwin III

This thread is getting a bit soft on technical details and/or value. Hans, please post the results of fsstress (the bits from ext3 cvs) on reiser4 on SMP machines of non-i386 architectures (e.g. ppc64, sparc64, ia64) as well as the results of transferring reiser4 filesystems between machines whose PAGE_SIZE, wordsize and endianness vary.

Okay, sounds like time to run these tests and post the results myself.

Hans, these things should be part of your standard QA. It would likely make a better impression if there were some record of these kinds of things having been successfully tested prior to your releases. For future reference, Andrew, Christoph, myself, and others can provide more detailed references to suites of stress tests and various kinds of tests filesystems should pass before being considered stable, and we (it's a relatively safe presumption that I speak for all of us when I say this) would appreciate this kind of testing in the future.

-- wli

#358 lore raw
Quoted 9 lines · William Lee Irwin III

Okay, sounds like time to run these tests and post the results myself. Hans, these things should be part of your standard QA. It would likely make a better impression if there were some record of these kinds of things having been successfully tested prior to your releases. For future reference, Andrew, Christoph, myself, and others can provide more detailed references to suites of stress tests and various kinds of tests filesystems should pass before being considered stable, and we (it's a relatively safe presumption that I speak for all of us when I say this) would appreciate this kind of testing in the future.

Step 1, the tools are very broken. This level of nonfunctionality of the reiser4 toolchain precludes any kind of exposure to the kind of testing I've asked about. I would very strongly prefer not to have to become a reiser4 implementor and furthermore fix numerous bugs just to have the smallest bit of assurance that this thing won't generate bug reports en masse once merged.

The following is from an UltraSPARC system (64-bit wordsize, big-endian, 8KB pagesize, 32-bit userspace, including 32-bit compiled reiser4 tools) running 2.6.9-rc1-mm3:

# /usr/local/sbin/mkfs.reiser4 -f -b 4096 /dev/loop0
/usr/local/sioctl32(mkfs.reiser4:1949): Unknown cmd fd(3) cmd(40081272){00} arg(efffc810) on /dev/loop0
4boicnt/lm3k2f(sm.krfesi.sreeri4s e1r.40:.109
 9C)o:p yUrnikgnhotw n( Cc)m d2 0f0d1(,3 )2 0c0m2d,( 420000831,2 7220)0{40 0b}y raHragn(se fRfefics8e1r0,)  loinc e/ndseivn/gl ogoopv0e
 ned by
reiser4progs/COPYING.

Block size 4096 will be used.
Linux 2.6.9-rc1-mm3 is detected.
Reiser4 is going to be created on /dev/loop0.
(Yes/No): Yes
ioctl32(mkfs.reiser4:1949): Unknown cmd fd(3) cmd(40081272){00} arg(efffc738) on /dev/loop0
i o c t l 3 2 ( m k f s . r e i s e r 4 : 1 9 4 9 ) : U n k n o w n c m d sC r/edaetvi/nlgo orpe0i0 8 1 2 7 2 ) { 0 0 } a r g ( e f f f c 7 3 8 )
er4 on /dev/loop0 ... Bus error
#

With the below patch, I get:

# /usr/local/sbin/mkfs.reiser4 -f -b 4096 /dev/loop0                            /usr/local/sbin/mkfs.reiser4 1.0.0
Copyright (C) 2001, 2002, 2003, 2004 by Hans Reiser, licensing governed by
reiser4progs/COPYING.

Block size 4096 will be used.
Linux 2.6.9-rc1-mm3 is detected.
Reiser4 is going to be created on /dev/loop0.
(Yes/No): Yes
Creating reiser4 on /dev/loop0 ... Bus error
#

The S_ISBLK() check is so that other assumptions about the file being a block device elsewhere won't be tripped up; it's not strictly necessary, only retained so as not to expose unaudited code to a new situation for which it's not prepared.

strace(1) before and after the patch is included here. The backtrace from the core dump is:

#0 0x7005e464 in cde40_insert_units () from /usr/local/lib/libreiser4-1.0.so.0 (gdb) bt

#0  0x7005e464 in cde40_insert_units () from /usr/local/lib/libreiser4-1.0.so.0
#1  0x7004fa44 in node40_modify () from /usr/local/lib/libreiser4-1.0.so.0
#2  0x7003c608 in cb_node_insert () from /usr/local/lib/libreiser4-1.0.so.0
#3  0x7003c58c in reiser4_node_modify ()
   from /usr/local/lib/libreiser4-1.0.so.0
#4  0x7003ee7c in reiser4_tree_modify ()
   from /usr/local/lib/libreiser4-1.0.so.0
#5  0x7006bc10 in obj40_insert () from /usr/local/lib/libreiser4-1.0.so.0
#6  0x7006d9e8 in dir40_create () from /usr/local/lib/libreiser4-1.0.so.0
#7  0x70041610 in reiser4_object_create ()

from /usr/local/lib/libreiser4-1.0.so.0
#8 0x00012160 in main ()

-- wli

src/file.c +16 -44
Index: libaal-1.0.0/src/file.c
===================================================================
@@ -193,59 +193,31 @@
 	return !aal_strncmp(file1, file2, aal_strlen(file1));
 }
 
-#if defined(__linux__) && defined(_IOR) && !defined(BLKGETSIZE64)
-#   define BLKGETSIZE64 _IOR(0x12, 114, uint64_t)
-#endif
-
 /* Handler for "len" operation for use with file device. See bellow for
    understanding where it is used. */
 static count_t file_len(
 	aal_device_t *device)	    /* file device, lenght will be obtained from */
 {
-	uint64_t size;
-	off_t max_off = 0;
+	off_t off, size;
+	struct stat stat_buf;
+	int fd;
 
-	if (!device) 
+	if (!device)
 		return INVAL_BLK;
-    
-#ifdef BLKGETSIZE64
-	if ((int)ioctl(*((int *)device->entity), BLKGETSIZE64, &size) >= (int)0) {
-		uint32_t block_count;
-		
-		size = (size / 4096) * 4096 / device->blksize;
-		block_count = size;
-		
-		if ((uint64_t)block_count != size) {
-			aal_fatal("The partition size is too big.");
-			return INVAL_BLK;
-		}
-		
-		return (count_t)block_count;
-	}
-    
-#endif
-
-#ifdef BLKGETSIZE    
-	{
-		unsigned long l_size;
-		
-		if (ioctl(*((int *)device->entity), BLKGETSIZE, &l_size) >= 0) {
-			size = l_size;
-			return (count_t)((size * 512 / 4096) * 4096 / 
-				device->blksize);
-		}
-	}
-    
-#endif
-    
-	if ((max_off = lseek(*((int *)device->entity), 
-			     0, SEEK_END)) == (off_t)-1) 
-	{
-		file_error(device);
+	fd = *(int *)device->entity;
+	if (fstat(fd, &stat_buf))
+		return INVAL_BLK;
+	if (!S_ISBLK(stat_buf.st_mode))
+		return INVAL_BLK;
+	off = lseek(fd, 0, SEEK_CUR);
+	size = lseek(fd, 0, SEEK_END);
+	if (lseek(fd, off, SEEK_SET) == (off_t)-1)
+		errno = 0;
+	if (size == (off_t)-1) {
+		errno = 0;
 		return INVAL_BLK;
 	}
-    
-	return (count_t)(max_off / device->blksize);
+	return (size & ~4096ULL)/device->blksize;
 }
 
 /* Initializing the file device operations. They are used when any operation of
#359 lore raw

src/file.c line 220

Quoted 2 lines · William Lee Irwin III
src/file.c +1 -1
-	return (count_t)(max_off / device->blksize);
+	return (size & ~4096ULL)/device->blksize;

Correcting this to return (size & ~4095ULL)/device->blksize does not fix the coredumps.

-- wli

#360 lore raw
In reply to David Masover
Quoted 9 lines · David Masover
|>Second, there are quite a few things which I might want to do, which can
|>be done with this interface and without patching programs,
| Such as?

They've been mentioned.

| Haven't seen any that made sense to me, sorry.
Sorry if they don't make sense to you, but I don't feel like discussing them now. Either you get it or you don't, either you agree or you don't. Read the archives.

Great argument. Not. There has been so much shit thrown around here so that it's impossible to keep track of all examples.

Could you please try summarize a few of the arguments that you find especially compelling? This thread has gotten very confused since there are a bunch of different subjects all being intermixed here.

What are we discussing?

1. Do we want support for named streams?

I belive the answer is yes, since both NTFS and HFS (that's the MacOS filesystem, isn't it?) supports streams we want Linux to support this if possible.

Anyone disagreeing?

2. How do we want to expose named streams?

One suggestion is file-as-directory in some form.

Another suggestion made is to expose named streams somewhere under /proc/self/fd.

Yet another suggestion is to use the openat(3) API from solaris.

Some filesystems exposes extra data in a special directory in the same directory as the file, such as netapps .snapshot directories or the extra directories that netatalk expects. This has the advantage that it even works on non-named stream capable filesystems, but it has a lot of problems too.

Linux already has limited support for names streams via the xattr interface, but it's not a good interface for people wanting to have large files as named streams.

4. What belongs in the generic VFS, what belongs in Reiser4?

Some things reiser4 do, such as files-as-directories need changes to the VFS because it breaks assumptions that the VFS makes (i.e. a deadlock or an oops when doing a hard link out of one).

Some other things reiser4 can do would be better if they were in the VFS since other filesystems might want to support the same functionality.

Or Linux may not support some of the things reiserfs at all.

5. What belongs in the kernel, what belongs in userspace?

This is mostly what I have been trying to argue about.

So, to try to summarize my opinion, regarding file-as-directory, I belive it's fatally flawed because it breaks a lot of assumptions that existing code make. One example of an application that will break is a web server that tries to filter out accesses to "bad" files, files-as-directories suddenly means that part of those files will be accessible (and there are a _lot_ of CERT reports on just this kind of problems with Windows web servers due to access to named streams not being restricted or ways to access files with non-canonical names that also managed to bypass access restrictions).

Files-as-directories also does not give us named streams on directories. The suggestion to have dir/metas access the named streams means that if someone already has a file named metas in a directory that file will be lost. (Does anyone remember the discussions about the linux kernel having a directory named "core" and the problems this caused for some people?)

All this suggests to me that named streams must live in another namespace than the normal one. To me, openat(3) seems like a good choice for an API and it has the advantage that someone else, Solaris, already has implemented it.

Additionally, files-as-directores does not solve the problem of "cp a b" losing named streams. There is curently no copyfile syscall in the Linux kernel, "cp a b" essentially does "cat a >b". So unless cp is modified we don't gain anything. If cp is modified to know about named streams, it really does not matter if named streams are accessed as file-as-directories, via openat(3) or via a shared library with some other interface.

Regarding the kernel or userspace discussion. In my opinion anything that can be done in user space should be done in userspace. If the performance sucks, or it has security problems, or needs caching that cant be solved in userspace it can be moved to the kernel, but in that case the smallest and cleanest API possible should be implemented.

If, for historical reasons, an API must be in the kernel, there is not much we can do about it either. It'll have to stay there, but we can avoid making the same mistakes again.

So, for all the examples of the kernel having plugins that automatically lets an application see a tar-file as a directory, I really, really don't belive this belongs in the kernel. First of all, this is the file-as-directory discussion again, I belive it is a mistake to expose the contents as a directory on top of the file because it breaks a lot of assumptions that unix programs make.

It's much better to expose the contents at another place in the filesystem by doing a temporary mount of the file with the proper filesystem. As Pavel Machek pointed out, this has the problem of who cleans up the mount if the application crashes. One way to handle this could be something like this:

    mount -t tarfs -o loop bar.tar /tmp/bar-fabb50509
    chdir /tmp/bar-fabb50509
    umount -f /tmp/bar-fabb50509

This will require the ability to unmount busy filesystems (but I belive Alexander Viro already has implemented the infrastructure needed for this).

Or for files that we don't have a real filesystem driver (or on other systems where userspace mounts are not allowed), we could just unpack the contents into /tmp. For cleanup we could let whatever cleans up /tmp anyways handle it, or have a cache daemon that keeps track of untarred directories and removes them after a while.

Another way is to completely forget about presenting the contents of a tar file as a real files, and just use a shared library to get at the contents (now we just have to convince everyone to use the shared library). This would also be portable to other systems.

If we do this right, it could all be hidden in a shared library, and if the system below it supports more advanced features, it can use it.

Regarding the "I want a realtime index of all files". I belive that a notifier that can tell me when a file has been changed and a userspace daemon ought to handle most of the cases that have been mentioned. The suggested problems of not getting an up to date query response can be handled by just asking the daemon "are you done with indexing yet". The design of such a daemon and the support it needs from the kernel can definitely be discussed. But to put the indexer itself in the kernel sounds like a bad idea. Even adding an API to query the indexer into the kernel sounds pointless, why do that instead of just opening a Unix socket to the indexer and asking it directly?

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#361 lore raw
Quoted 7 lines · Christer Weinigel

Additionally, files-as-directores does not solve the problem of "cp a b" losing named streams. There is curently no copyfile syscall in the Linux kernel, "cp a b" essentially does "cat a >b". So unless cp is modified we don't gain anything. If cp is modified to know about named streams, it really does not matter if named streams are accessed as file-as-directories, via openat(3) or via a shared library with some other interface.

You cannot just 'modify cp'. cp is a programming interface standardized in POSIX.1. You can of course add non-standard extensions to some cp implementations, but it seems hardly evitable then that you either have to use cp in a non-standard manner regularly with Linux or risk to lose data.

This is even more severe with tar/pax. Just patching GNU tar for file streams, as it was suggested earlier in this discussion, is still far away from a real solution because it neither solves the issues with the POSIX.1 pax standard nor those with other implementations of it.

Given these facts, it does not seem so clear to me that adding named streams for Windows and Mac OS interoperability would be a win to Linux in the end. The loss of interoperability to Unix/POSIX/today's Linux might have much worse effects.

The current xattr extension is much less of a problem because it only holds metadata, which is mostly not applicable to other environments anyway.

Gunnar

Signature · 1 line
http://omnibus.ruf.uni-freiburg.de/~gritter
#362 lore raw
Quoted 4 lines · Gunnar Ritter

You cannot just 'modify cp'.

People who think that POSIX is the objective rather than the least common denominator of OS design have had their head screwed on backwards to better look at where their competitors used to be.

However, I agree that streams suck. That is why reiser4 just has files and directories and not streams. Our files and directories just happen to be able to do all that streams can do.

Hans

#363 lore raw
Quoted 5 lines · Hans Reiser
Quoted 2 lines · Gunnar Ritter

You cannot just 'modify cp'.

People who think that POSIX is the objective rather than the least common denominator of OS design

I am not principally adversed against extensions to POSIX. My mailx implementation 'nail' has e.g. perhaps more extensions than there are commands and options in the POSIX standard for it.

POSIX is also not against extensions. In fact, POSIX development generally works as follows: One vendor creates something as an extension, other vendors follow to implement it, and later on it is discussed if it is desirable to integrate the feature into the standard itself. It is absolutely possible that Sun's openat() might be in POSIX.1-2010 one day, for example. Useful extensions are thus welcome to POSIX.

This does not mean, however, that one should not clearly distinct between standard and extensions, and that extensions should be created at will without carefully weighting pros and cons.

I did not say: POSIX forbids to handle streams or directory/file mixes. This would not even have been true. However, POSIX restricts the choice of possible interfaces for them. One of those restrictions is that 'cp' may not copy streams if used in strict accordance with POSIX. As you acknowledged in your reply, POSIX is the least common denominator. Thus 'cp' implementations should not be modified in a way that violates it.

This means, in effect, that a strictly conforming POSIX application (i.e. something like a shell script that uses no POSIX extensions or methods which are not clearly defined in the standard) will very likely be unable to copy streams, unless some other, conforming, method is found. Which is a problem one should know about when discussing this.

Quoted 2 lines

have had their head screwed on backwards
to better look at where their competitors used to be.

But there are not only forwards and backwards directions. Sideways might lead to nowhere.

Gunnar

Signature · 1 line
http://omnibus.ruf.uni-freiburg.de/~gritter
#364 lore raw
In reply to Christer Weinigel
Quoted 19 lines · Christer Weinigel
Quoted 9 lines · David Masover
|>Second, there are quite a few things which I might want to do, which can
|>be done with this interface and without patching programs,
| Such as?

They've been mentioned.

| Haven't seen any that made sense to me, sorry.
Sorry if they don't make sense to you, but I don't feel like discussing them now. Either you get it or you don't, either you agree or you don't. Read the archives.

Great argument. Not. There has been so much shit thrown around here so that it's impossible to keep track of all examples.

Could you please try summarize a few of the arguments that you find especially compelling? This thread has gotten very confused since there are a bunch of different subjects all being intermixed here.

Quoted 7 lines

What are we discussing?

1. Do we want support for named streams?

I belive the answer is yes, since both NTFS and HFS (that's the MacOS filesystem, isn't it?) supports streams we want Linux to support this if possible.

well, yes HFS has this, is it advantageous, no it's kind of heritage ...

Quoted 1 line

Anyone disagreeing?

yes, MacOS X allows to use UFS instead of HFS+ which doesn't support the fancy/confusing streams

I, for my part, do not like the idea of multiple streams for one file, IMHO all features can be provided by using directories instead, which does not break any userspace tools _and_ sounds natural to me ...

best,
Herbert

Quoted 132 lines

2. How do we want to expose named streams?

One suggestion is file-as-directory in some form.

Another suggestion made is to expose named streams somewhere under /proc/self/fd.

Yet another suggestion is to use the openat(3) API from solaris.

Some filesystems exposes extra data in a special directory in the same directory as the file, such as netapps .snapshot directories or the extra directories that netatalk expects. This has the advantage that it even works on non-named stream capable filesystems, but it has a lot of problems too.

Linux already has limited support for names streams via the xattr interface, but it's not a good interface for people wanting to have large files as named streams.

4. What belongs in the generic VFS, what belongs in Reiser4?

Some things reiser4 do, such as files-as-directories need changes to the VFS because it breaks assumptions that the VFS makes (i.e. a deadlock or an oops when doing a hard link out of one).

Some other things reiser4 can do would be better if they were in the VFS since other filesystems might want to support the same functionality.

Or Linux may not support some of the things reiserfs at all.

5. What belongs in the kernel, what belongs in userspace?

This is mostly what I have been trying to argue about.

So, to try to summarize my opinion, regarding file-as-directory, I belive it's fatally flawed because it breaks a lot of assumptions that existing code make. One example of an application that will break is a web server that tries to filter out accesses to "bad" files, files-as-directories suddenly means that part of those files will be accessible (and there are a _lot_ of CERT reports on just this kind of problems with Windows web servers due to access to named streams not being restricted or ways to access files with non-canonical names that also managed to bypass access restrictions).

Files-as-directories also does not give us named streams on directories. The suggestion to have dir/metas access the named streams means that if someone already has a file named metas in a directory that file will be lost. (Does anyone remember the discussions about the linux kernel having a directory named "core" and the problems this caused for some people?)

All this suggests to me that named streams must live in another namespace than the normal one. To me, openat(3) seems like a good choice for an API and it has the advantage that someone else, Solaris, already has implemented it.

Additionally, files-as-directores does not solve the problem of "cp a b" losing named streams. There is curently no copyfile syscall in the Linux kernel, "cp a b" essentially does "cat a >b". So unless cp is modified we don't gain anything. If cp is modified to know about named streams, it really does not matter if named streams are accessed as file-as-directories, via openat(3) or via a shared library with some other interface.

Regarding the kernel or userspace discussion. In my opinion anything that can be done in user space should be done in userspace. If the performance sucks, or it has security problems, or needs caching that cant be solved in userspace it can be moved to the kernel, but in that case the smallest and cleanest API possible should be implemented.

If, for historical reasons, an API must be in the kernel, there is not much we can do about it either. It'll have to stay there, but we can avoid making the same mistakes again.

So, for all the examples of the kernel having plugins that automatically lets an application see a tar-file as a directory, I really, really don't belive this belongs in the kernel. First of all, this is the file-as-directory discussion again, I belive it is a mistake to expose the contents as a directory on top of the file because it breaks a lot of assumptions that unix programs make.

It's much better to expose the contents at another place in the filesystem by doing a temporary mount of the file with the proper filesystem. As Pavel Machek pointed out, this has the problem of who cleans up the mount if the application crashes. One way to handle this could be something like this:

    mount -t tarfs -o loop bar.tar /tmp/bar-fabb50509
    chdir /tmp/bar-fabb50509
    umount -f /tmp/bar-fabb50509

This will require the ability to unmount busy filesystems (but I belive Alexander Viro already has implemented the infrastructure needed for this).

Or for files that we don't have a real filesystem driver (or on other systems where userspace mounts are not allowed), we could just unpack the contents into /tmp. For cleanup we could let whatever cleans up /tmp anyways handle it, or have a cache daemon that keeps track of untarred directories and removes them after a while.

Another way is to completely forget about presenting the contents of a tar file as a real files, and just use a shared library to get at the contents (now we just have to convince everyone to use the shared library). This would also be portable to other systems.

If we do this right, it could all be hidden in a shared library, and if the system below it supports more advanced features, it can use it.

Regarding the "I want a realtime index of all files". I belive that a notifier that can tell me when a file has been changed and a userspace daemon ought to handle most of the cases that have been mentioned. The suggested problems of not getting an up to date query response can be handled by just asking the daemon "are you done with indexing yet". The design of such a daemon and the support it needs from the kernel can definitely be discussed. But to put the indexer itself in the kernel sounds like a bad idea. Even adding an API to query the indexer into the kernel sounds pointless, why do that instead of just opening a Unix socket to the indexer and asking it directly?

/Christer

Signature · 8 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#365 lore raw

Am Dienstag, 7. September 2004 14:30 schrieb Herbert Poetzl:

Quoted 8 lines
Quoted 5 lines

1. Do we want support for named streams?

   I belive the answer is yes, since both NTFS and HFS (that's the MacOS filesystem, isn't it?) supports streams we want Linux to support this if possible.

well, yes HFS has this, is it advantageous, no
it's kind of heritage ...

HFS doesn't support _named_ streams. There are just two forks per file.

Regards
Oliver

#366 lore raw
In reply to Christer Weinigel
Quoted 14 lines
Quoted 9 lines · David Masover
|>Second, there are quite a few things which I might want to do, which can
|>be done with this interface and without patching programs,
| Such as?

They've been mentioned.

| Haven't seen any that made sense to me, sorry.
Sorry if they don't make sense to you, but I don't feel like discussing them now. Either you get it or you don't, either you agree or you don't. Read the archives.

Quoted 6 lines

Great argument. Not. There has been so much shit thrown around here so that it's impossible to keep track of all examples.

Could you please try summarize a few of the arguments that you find especially compelling? This thread has gotten very confused since there are a bunch of different subjects all being intermixed here.

Indeed. We are discussion changes to the heart of Linux. It is bound to get a little heated :/

Quoted 9 lines

What are we discussing?

1. Do we want support for named streams?

I belive the answer is yes, since both NTFS and HFS (that's the MacOS filesystem, isn't it?) supports streams we want Linux to support this if possible.

Anyone disagreeing?

     No :)
Quoted 35 lines

2. How do we want to expose named streams?

One suggestion is file-as-directory in some form.

Another suggestion made is to expose named streams somewhere under /proc/self/fd.

Yet another suggestion is to use the openat(3) API from solaris.

Some filesystems exposes extra data in a special directory in the same directory as the file, such as netapps .snapshot directories or the extra directories that netatalk expects. This has the advantage that it even works on non-named stream capable filesystems, but it has a lot of problems too.

Linux already has limited support for names streams via the xattr interface, but it's not a good interface for people wanting to have large files as named streams.

4. What belongs in the generic VFS, what belongs in Reiser4?

Some things reiser4 do, such as files-as-directories need changes to the VFS because it breaks assumptions that the VFS makes (i.e. a deadlock or an oops when doing a hard link out of one).

Some other things reiser4 can do would be better if they were in the VFS since other filesystems might want to support the same functionality.

Or Linux may not support some of the things reiserfs at all.

5. What belongs in the kernel, what belongs in userspace?

This is mostly what I have been trying to argue about.

Quoted 9 lines

So, to try to summarize my opinion, regarding file-as-directory, I belive it's fatally flawed because it breaks a lot of assumptions that existing code make. One example of an application that will break is a web server that tries to filter out accesses to "bad" files, files-as-directories suddenly means that part of those files will be accessible (and there are a _lot_ of CERT reports on just this kind of problems with Windows web servers due to access to named streams not being restricted or ways to access files with non-canonical names that also managed to bypass access restrictions).

But restrictions could be controlled. But exactly how depends on what solution for named streams is chosen. For example, there might be separate permissions for named streams other than the main stream.

Quoted 6 lines

Files-as-directories also does not give us named streams on directories. The suggestion to have dir/metas access the named streams means that if someone already has a file named metas in a directory that file will be lost. (Does anyone remember the discussions about the linux kernel having a directory named "core" and the problems this caused for some people?)

Yes, reserved keywords needs to be chosen very carefully if they are used. Other examples here has been ..metas or ...

Quoted 5 lines

All this suggests to me that named streams must live in another namespace than the normal one. To me, openat(3) seems like a good choice for an API and it has the advantage that someone else, Solaris, already has implemented it.

Quoted 7 lines

Additionally, files-as-directores does not solve the problem of "cp a b" losing named streams. There is curently no copyfile syscall in the Linux kernel, "cp a b" essentially does "cat a >b". So unless cp is modified we don't gain anything. If cp is modified to know about named streams, it really does not matter if named streams are accessed as file-as-directories, via openat(3) or via a shared library with some other interface.

One suggestion is missed. It is to provide system calls for copy. That would also solve the problem. Named streams and metas would then be handled correctly. It also allows further changes to filesystems without having to patch applications yet again.

A copy system call would also be large beneficial for networked filesystems (NFS, Samba, etc) as data wouldn't have to be transferred over the network and back.

Quoted 5 lines

Regarding the kernel or userspace discussion. In my opinion anything that can be done in user space should be done in userspace. If the performance sucks, or it has security problems, or needs caching that cant be solved in userspace it can be moved to the kernel, but in that case the smallest and cleanest API possible should be implemented.

Yes. I agree. I do not think most people arguing for plugins or usage for named streams actually wanted to put everything in the kernel, but rather to have a kernel interface to which user-level modules, plugins and applications could work.

Quoted 25 lines

If, for historical reasons, an API must be in the kernel, there is not much we can do about it either. It'll have to stay there, but we can avoid making the same mistakes again.

So, for all the examples of the kernel having plugins that automatically lets an application see a tar-file as a directory, I really, really don't belive this belongs in the kernel. First of all, this is the file-as-directory discussion again, I belive it is a mistake to expose the contents as a directory on top of the file because it breaks a lot of assumptions that unix programs make.

It's much better to expose the contents at another place in the filesystem by doing a temporary mount of the file with the proper filesystem. As Pavel Machek pointed out, this has the problem of who cleans up the mount if the application crashes. One way to handle this could be something like this:

    mount -t tarfs -o loop bar.tar /tmp/bar-fabb50509
    chdir /tmp/bar-fabb50509
    umount -f /tmp/bar-fabb50509

This will require the ability to unmount busy filesystems (but I belive Alexander Viro already has implemented the infrastructure needed for this).

Quoted 5 lines

Or for files that we don't have a real filesystem driver (or on other systems where userspace mounts are not allowed), we could just unpack the contents into /tmp. For cleanup we could let whatever cleans up /tmp anyways handle it, or have a cache daemon that keeps track of untarred directories and removes them after a while.

But now, as you said, we are not talking about named streams, but about a specic type of plugin and its implications.

Can we make a plugin infrastructure that will let user-space plugins to be loaded for certain directories or files? If we can, then it would present a much cleaner and easier way for the user to access data he wants. In this particular example it was a tar file.

Quoted 4 lines

Another way is to completely forget about presenting the contents of a tar file as a real files, and just use a shared library to get at the contents (now we just have to convince everyone to use the shared library). This would also be portable to other systems.

Yes. If it was possible. I do honestly think it will be easier and less man hour work to create a run-time loadable user-space plugin interface than to convince thousands of application developers to use this shared library.

Quoted 6 lines

If we do this right, it could all be hidden in a shared library, and if the system below it supports more advanced features, it can use it.

Regarding the "I want a realtime index of all files". I belive that a notifier that can tell me when a file has been changed and a userspace daemon ought to handle most of the cases that have been mentioned.

We have FAM (File Alteration Monitor) that notifies applications of changes to files they are interested in.

http://oss.sgi.com/projects/fam/

Quoted 7 lines

The suggested problems of not getting an up to date query response can be handled by just asking the daemon "are you done with indexing yet". The design of such a daemon and the support it needs from the kernel can definitely be discussed. But to put the indexer itself in the kernel sounds like a bad idea. Even adding an API to query the indexer into the kernel sounds pointless, why do that instead of just opening a Unix socket to the indexer and asking it directly?

I think that we do not need a indexer with Reiser4 anyway. It is already a database that could be queried directly. There shouldn't be any need to build a database on top of Reiser4 (like the updatedb) that holds the same information already existing in Reiser4.

Now, it is just a matter of having an interface to query the Reiser4 db. One of the suggested ways were through meta-data. For example:

ls -l ..metas/keyword/.pdf

~S

Quoted 1 line

/Christer

#367 lore raw
Quoted 8 lines · Spam
Quoted 7 lines

Additionally, files-as-directores does not solve the problem of "cp a b" losing named streams. There is curently no copyfile syscall in the Linux kernel, "cp a b" essentially does "cat a >b". So unless cp is modified we don't gain anything. If cp is modified to know about named streams, it really does not matter if named streams are accessed as file-as-directories, via openat(3) or via a shared library with some other interface.

Quoted 4 lines

One suggestion is missed. It is to provide system calls for copy. That would also solve the problem. Named streams and metas would then be handled correctly. It also allows further changes to filesystems without having to patch applications yet again.

But this still solves only part of the problem. A backup application won't have any use for a copyfile syscall, it will need to be taught about streams.

Quoted 3 lines

A copy system call would also be large beneficial for networked filesystems (NFS, Samba, etc) as data wouldn't have to be transferred over the network and back.

Definitely.

Quoted 4 lines

Can we make a plugin infrastructure that will let user-space plugins to be loaded for certain directories or files? If we can, then it would present a much cleaner and easier way for the user to access data he wants. In this particular example it was a tar file.

In that case I'd argue that:

mount -t userfs -o driver=tarfs foo /tmp/foo

is a rather good kernel interface for plugins. userfs (or something based on userfs) is the plugin API and tarfs is a plugin. :-)

To make this efficient, well have to allow non-root users to perform the mount syscall (with the limitation that they can only mount on top of directories they own and that the mounts have the nosuid and nodev flags set).

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#368 lore raw
Quoted 13 lines
Quoted 8 lines · Spam
Quoted 7 lines

Additionally, files-as-directores does not solve the problem of "cp a b" losing named streams. There is curently no copyfile syscall in the Linux kernel, "cp a b" essentially does "cat a >b". So unless cp is modified we don't gain anything. If cp is modified to know about named streams, it really does not matter if named streams are accessed as file-as-directories, via openat(3) or via a shared library with some other interface.

Quoted 8 lines
Quoted 4 lines

One suggestion is missed. It is to provide system calls for copy. That would also solve the problem. Named streams and metas would then be handled correctly. It also allows further changes to filesystems without having to patch applications yet again.

But this still solves only part of the problem. A backup application won't have any use for a copyfile syscall, it will need to be taught about streams.

Yes, but backup programs always needed to be taught about new features. Be it new type of files, attributes or meta-data. I think that teaching backup applications is far better than teaching every application.

Quoted 15 lines
Quoted 3 lines

A copy system call would also be large beneficial for networked filesystems (NFS, Samba, etc) as data wouldn't have to be transferred over the network and back.

Definitely.

Quoted 4 lines

Can we make a plugin infrastructure that will let user-space plugins to be loaded for certain directories or files? If we can, then it would present a much cleaner and easier way for the user to access data he wants. In this particular example it was a tar file.

In that case I'd argue that:

mount -t userfs -o driver=tarfs foo /tmp/foo

Quoted 7 lines

is a rather good kernel interface for plugins. userfs (or something based on userfs) is the plugin API and tarfs is a plugin. :-)

To make this efficient, well have to allow non-root users to perform the mount syscall (with the limitation that they can only mount on top of directories they own and that the mounts have the nosuid and nodev flags set).

Yes, this seem to be one solution. It isn't very dynamic in usage though. You can't use this directly from applications wihout manually doing the mount.

This is only a solution to browsing contents of files. It doesn't provide a solution for using meta-data streams or other things like this.

~S

Quoted 1 line

/Christer

#369 lore raw
Quoted 1 line · Spam

Christer Weinigel <[email protected]> said:

[...]

Quoted 8 lines
Quoted 3 lines

But this still solves only part of the problem. A backup application won't have any use for a copyfile syscall, it will need to be taught about streams.

Yes, but backup programs always needed to be taught about new features. Be it new type of files, attributes or meta-data. I think that teaching backup applications is far better than teaching every application.

Strange... tar(1) is quite capable of backing up .mp3 files, which weren't around when tar was born...

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#370 lore raw
In reply to Christer Weinigel
Quoted 6 lines · Christer Weinigel
Quoted 2 lines · Spam
Quoted 2 lines

Additionally, files-as-directores does not solve the problem of "cp a b" losing named streams.

reiser4 does not support streams, it supports files that can do what streams do. cp -r does not currently lose files.

#371 lore raw
Quoted 10 lines · Hans Reiser
Quoted 6 lines · Christer Weinigel
Quoted 3 lines · Spam
Quoted 3 lines

Additionally, files-as-directores does not solve the problem of "cp a b" losing named streams.

reiser4 does not support streams, it supports files that can do what streams do. cp -r does not currently lose files.

I'm not talking about reiser4 only. I'm interested in semantics that can be sanely apply to reiser4, NTFS, HFS, or any other file system that supports XATTRS.

I relize that resier4 is your baby and you're only interested in it, but I'm much more interested in linux in general.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#372 lore raw
In reply to Spam
Quoted 1 line · Spam

Christer Weinigel <[email protected]> said:

[...]

Quoted 6 lines
Quoted 3 lines

Could you please try summarize a few of the arguments that you find especially compelling? This thread has gotten very confused since there are a bunch of different subjects all being intermixed here.

Indeed. We are discussion changes to the heart of Linux. It is bound to get a little heated :/

True. But without any specific applications, just "this would be nice to have", the discussion can't go forward.

Quoted 11 lines
Quoted 9 lines

What are we discussing?

1. Do we want support for named streams?

I belive the answer is yes, since both NTFS and HFS (that's the MacOS filesystem, isn't it?) supports streams we want Linux to support this if possible.

Anyone disagreeing?

     No :)

There are many people around here who disagree (that is precisely the heart of the discussion). I for one don't think Linux has to get $RANDOM_FEATURE just because $SOME_OTHER_OS has got it. Either the feature stands on its own _in the context of POSIX/Unix/Linux_ (possibly as an extension or modification of said standards) or it isn't worth it.

Quoted 3 lines
Quoted 3 lines

2. How do we want to expose named streams?

One suggestion is file-as-directory in some form.

Which is broken, as it forbids hard links to files.

It also has quite interesting issues with moving streams around among containers, or renaming them. How would you handle permissions on streams inside containers? Links to streams? What happens if the principal stream is deleted? Renamed? Moved into another container?

Now you have 3 principal types of objects: Directories, containers (files with streams), and files (no streams). What is the essential difference between containers and directories? Same for containers vs plain files? If none, containers are just complicating stuff, and should not exist at all.

If you add streams to directories, you get 4 types. Again, tell what the exact differences are, and what each one can do none of the others can.

In the preceding two exercises, it is not enough to show something that would be dificult or messy to do with what we have now, it must also be some operation(s) that are natural and common enough to warrant the extra complexity (in kernel, in applications, in wetware). Consider support in form of existing or to-be-written tools for the task. What of all this can the GUI hide from (or fake for) the unsuspecting end-user?

Also, there have been suggestions of cached data in streams (like thumbnail version of graphic file, contents listing of archive, ...), where losing the stream is of no real consecuence (except performance). There have also been requests for metadata that can't be recovered from the file (author, comments on the file, older versions of the file (i.e., some form of file version control), permissions). And finally there have been requests for structured contents manipulable in other forms (f.ex. .tar.bz2 file in unpacked form, where you can directly read or even modify the contents). Some suggestions go for a limited number of small(ish), even standardized, extra streams with no further structure, others ask for full directories under streams (which could even be much larger than the principal stream, like the tar.bz2 case). And there is the possibility of recursive
application: If you want to be consecuent, the files under a .tar.bz2 should again be able to be .tar.bz2'es with their own streams, or files with streams in them.

What permissions should apply to streams? The ones of the file (i.e., I can see the icon stream of a file I can't read), or extra permissions (i.e., icon is writable to me (so I can adjust it to taste) but I can't write to the file; now I can screw up the icon for everybody...). Streams on a per-user basis (i.e., icon stream for each user)?

We need to sort out exactly how far it makes sense to go, by showing concrete, down to earth uses for whatever substructure we want. Then show the effect can't be easily gotten through tools for power users or faking it for unsuspecting users via GUI, and that overall the complexity and performance cost is less than the win. Note that the success of the Unix way is in large part due to its use of few, simple concepts that can be combined endlessly; and tools following the same strategy. Adding extra concepts that current tools can't naturally handle has to be considered with extreme care.

[-- much snippage --]

Quoted 8 lines
Quoted 7 lines

The suggested problems of not getting an up to date query response can be handled by just asking the daemon "are you done with indexing yet". The design of such a daemon and the support it needs from the kernel can definitely be discussed. But to put the indexer itself in the kernel sounds like a bad idea. Even adding an API to query the indexer into the kernel sounds pointless, why do that instead of just opening a Unix socket to the indexer and asking it directly?

Quoted 5 lines

I think that we do not need a indexer with Reiser4 anyway. It is already a database that could be queried directly. There shouldn't be any need to build a database on top of Reiser4 (like the updatedb) that holds the same information already existing in Reiser4.

Indexing/general queries of the filesystem are _hard_ (you need a complete database behind it for this, plus handling transactions for full functionality). Applications who _really_ need this use DBMSes, and are prepared to pay the ensuing cost. They are far in between, in any case.

Besides, this is completely orthogonal to the streams discussion.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#373 lore raw
Quoted 58 lines · Horst von Brand
Quoted 3 lines · Spam

Christer Weinigel <[email protected]> said:

[...]

Quoted 8 lines
Quoted 5 lines

Could you please try summarize a few of the arguments that you find especially compelling? This thread has gotten very confused since there are a bunch of different subjects all being intermixed here.

Quoted 4 lines

Indeed. We are discussion changes to the heart of Linux. It is bound to get a little heated :/

True. But without any specific applications, just "this would be nice to have", the discussion can't go forward.

Quoted 14 lines
Quoted 14 lines

What are we discussing?

1. Do we want support for named streams?

I belive the answer is yes, since both NTFS and HFS (that's the MacOS filesystem, isn't it?) supports streams we want Linux to support this if possible.

Quoted 6 lines
Quoted 3 lines

Anyone disagreeing?

    No :)

There are many people around here who disagree (that is precisely the heart of the discussion). I for one don't think Linux has to get $RANDOM_FEATURE just because $SOME_OTHER_OS has got it. Either the feature stands on its own _in the context of POSIX/Unix/Linux_ (possibly as an extension or modification of said standards) or it isn't worth it.

Quoted 10 lines
Quoted 6 lines
Quoted 6 lines

2. How do we want to expose named streams?

One suggestion is file-as-directory in some form.

Which is broken, as it forbids hard links to files.

No, it forbids hard links to the directory aspect of the file-directory duality.

Quoted 5 lines

Now you have 3 principal types of objects: Directories, containers (files with streams), and files (no streams).

No, the reiser4 design supports only files and directories, but makes them able to do what people use streams for.

The reiser4 design is based on a hatred of streams, and a desire to show that adding more features to files and directories makes streams unnecessary.

Hans

#374 lore raw
Quoted 3 lines · Hans Reiser
Quoted 2 lines · Horst von Brand
Quoted 1 line · Spam

Christer Weinigel <[email protected]> said:

[...]

Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Quoted 3 lines

2. How do we want to expose named streams?

One suggestion is file-as-directory in some form.

Which is broken, as it forbids hard links to files.

No, it forbids hard links to the directory aspect of the file-directory duality.

How do you distinguish a "hard link to the directory personality" from "hard link to the file personality"? And how do you make sure that only one of them can be followed if you go to the directory, while allowing several to the exact same file?!

Quoted 5 lines
Quoted 2 lines

Now you have 3 principal types of objects: Directories, containers (files with streams), and files (no streams).

No, the reiser4 design supports only files and directories, but makes them able to do what people use streams for.

I.e., you have files/directories with/without "use as streams" stuff. 4 types (or uses) of things.

Quoted 3 lines

The reiser4 design is based on a hatred of streams, and a desire to show that adding more features to files and directories makes streams unnecessary.

Trade one ugly wart for others...

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#375 lore raw
Quoted 35 lines · Horst von Brand
Quoted 9 lines · Hans Reiser
Quoted 6 lines · Horst von Brand
Quoted 3 lines · Spam

Christer Weinigel <[email protected]> said:

[...]

Quoted 15 lines
Quoted 15 lines
Quoted 9 lines
Quoted 9 lines

2. How do we want to expose named streams?

One suggestion is file-as-directory in some form.

Which is broken, as it forbids hard links to files.

Quoted 7 lines
Quoted 3 lines

No, it forbids hard links to the directory aspect of the file-directory duality.

How do you distinguish a "hard link to the directory personality" from "hard link to the file personality"?

Put in (undoubtedly overly) simple terms, if you can do it to a file you can do it to the file personality, but if you currently can only do it to a directory and get an error from attempting it to a file then in the new scheme doing it to the hard link only gives the same error.

Or, we can ask Alexander to help us use his deadlock detection algorithm and try to do things right....

#376 lore raw
Quoted 5 lines · Hans Reiser
Quoted 4 lines · Horst von Brand
Quoted 3 lines · Hans Reiser
Quoted 2 lines · Horst von Brand
Quoted 1 line · Spam

Christer Weinigel <[email protected]> said:

[...]

Quoted 8 lines
Quoted 7 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

2. How do we want to expose named streams? One suggestion is file-as-directory in some form.

Which is broken, as it forbids hard links to files.

No, it forbids hard links to the directory aspect of the file-directory duality.

Quoted 7 lines
Quoted 2 lines

How do you distinguish a "hard link to the directory personality" from "hard link to the file personality"?

Put in (undoubtedly overly) simple terms, if you can do it to a file you can do it to the file personality, but if you currently can only do it to a directory and get an error from attempting it to a file then in the new scheme doing it to the hard link only gives the same error.

Let me sort this out: If it can't be done POSIXly to a directory, it can't be done in Reiser4 to a file (which really is a directory too). So there can be exactly _one_ hard link to a file. Way borken.

Quoted 2 lines

Or, we can ask Alexander to help us use his deadlock detection algorithm and try to do things right....

Good luck with that one. I'd suspect if it can be made to work, it will have _huge_ overhead, so much that it is useless. I'd love to be proven wrong, but I won't hold my breath.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#377 lore raw
Quoted 23 lines
Quoted 5 lines · Hans Reiser
Quoted 4 lines · Horst von Brand
Quoted 3 lines · Hans Reiser
Quoted 2 lines · Horst von Brand
Quoted 1 line · Spam

Christer Weinigel <[email protected]> said:

[...]

Quoted 10 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

2. How do we want to expose named streams? One suggestion is file-as-directory in some form.

Which is broken, as it forbids hard links to files.

No, it forbids hard links to the directory aspect of the file-directory duality.

How do you distinguish a "hard link to the directory personality" from "hard link to the file personality"?

Quoted 8 lines
Quoted 4 lines

Put in (undoubtedly overly) simple terms, if you can do it to a file you can do it to the file personality, but if you currently can only do it to a directory and get an error from attempting it to a file then in the new scheme doing it to the hard link only gives the same error.

Let me sort this out: If it can't be done POSIXly to a directory, it can't be done in Reiser4 to a file (which really is a directory too). So there can be exactly _one_ hard link to a file. Way borken.

But you can make a hard link to a file in reiser4, and you can access the metadata in both. I did this test:
echo "moooo" > test
ln test moo
chmod +x moo test
echo "0700" > test/metas/rwx
dir moo test

      -rwx------  2 root root 10 Sep  7 23:17 moo*
      -rwx------  2 root root 10 Sep  7 23:17 test*

echo "0777" > test/metas/rwx
dir moo test

      -rwxrwxrwx  2 root root 10 Sep  7 23:17 moo*
      -rwxrwxrwx  2 root root 10 Sep  7 23:17 test*

rm moo test
mkdir test
ln test moo
ln: `test': hard link not allowed for directory

If this is the intended behaviour I do not know, but it shows that hard links works as normal.

You can also do linking to meta-data:

echo moo > moo
ln -s moo/metas/rwx test
dir

      -rwxrwxrwx   1 root    root      0 Sep  7 23:24 moo*
      lrwxrwxrwx   1 root    root     13 Sep  7 23:25 test -> moo/metas/rwx
  echo 0700 > test
  dir moo
      -rwx------   1 root    root      0 Sep  7 23:24 moo*
Quoted 6 lines
Quoted 2 lines

Or, we can ask Alexander to help us use his deadlock detection algorithm and try to do things right....

Good luck with that one. I'd suspect if it can be made to work, it will have _huge_ overhead, so much that it is useless. I'd love to be proven wrong, but I won't hold my breath.

#378 lore raw
Quoted 7 lines · Spam
Quoted 6 lines · Horst von Brand
Quoted 5 lines · Hans Reiser
Quoted 4 lines · Horst von Brand
Quoted 3 lines · Hans Reiser
Quoted 2 lines · Horst von Brand
Quoted 1 line · Spam

Christer Weinigel <[email protected]> said:

[...]

Quoted 16 lines
Quoted 15 lines
Quoted 15 lines
Quoted 10 lines
Quoted 7 lines
Quoted 4 lines
Quoted 2 lines
Quoted 2 lines

2. How do we want to expose named streams? One suggestion is file-as-directory in some form.

Which is broken, as it forbids hard links to files.

No, it forbids hard links to the directory aspect of the file-directory duality.

How do you distinguish a "hard link to the directory personality" from "hard link to the file personality"?

Put in (undoubtedly overly) simple terms, if you can do it to a file you can do it to the file personality, but if you currently can only do it to a directory and get an error from attempting it to a file then in the new scheme doing it to the hard link only gives the same error.

Quoted 5 lines
Quoted 3 lines

Let me sort this out: If it can't be done POSIXly to a directory, it can't be done in Reiser4 to a file (which really is a directory too). So there can be exactly _one_ hard link to a file. Way borken.

But you can make a hard link to a file in reiser4,

IIUC, this isn't safe (you might tie your filesystem into a knot this way, as LWN puts it). Sure, you can do it right now (just as I was able to create hard links to directories in some propietary Unix way back; untangling the resulting mess, complete with segfaulting fsck, was a whole world of fun... it just happened rarely enough that it did not matter very much); production versions will have to forbid it (or incurr in _huge_ overheads locking everything in sight, or detecting loops).

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#379 lore raw
In reply to Horst von Brand

Horst von Brand wrote on Tue, 07 Sep 2004 17:02:20 -0400:

Quoted 7 lines
Quoted 2 lines · Hans Reiser

Or, we can ask Alexander to help us use his deadlock detection algorithm and try to do things right....

Good luck with that one. I'd suspect if it can be made to work, it will have _huge_ overhead, so much that it is useless. I'd love to be proven wrong, but I won't hold my breath.

Depends if you consider it to be a huge overhead to lock all the objects that are ancestors (possibly through cyclical multiple parents) of the fildirute (file/directory/attribute/whatever) being deleted/renamed. It could definitely be a lot if you have some weird worst case directory layout with lots of parent directories. But that's rare (most users don't have that many directories anyway).

By the way, the deadlock system I was using is just a hack - a timeout on the locking semaphores for each file. If it fails to lock everything it needs, it backs off, waits a while, tries again, and eventually reports EDEADLK if it exhausts the retry count.

Well, actually it's a bit fancier - it read-locks the file nodes as it is traversing the graph to find all ancestors. Then when it has finished, it write-locks the ones it needs to change.

- Alex

#380 lore raw
In reply to Horst von Brand
Quoted 9 lines · Horst von Brand
Quoted 3 lines
Quoted 3 lines

1. Do we want support for named streams?

I belive the answer is yes [...]

There are many people around here who disagree (that is precisely the heart of the discussion). I for one don't think Linux has to get $RANDOM_FEATURE just because $SOME_OTHER_OS has got it. Either the feature stands on its own _in the context of POSIX/Unix/Linux_ (possibly as an extension or modification of said standards) or it isn't worth it.

Ok, noted. :-)

I myself am not very interested in generic named stream support in Linux. But since we have support for filesystems with named streams and people are interested in getting at those streams (if nothing else thanq to serve streams from an NTFS file system via Samba on a dual boot machine), we'd better do it well, instead of with ugly hacks.

Quoted 9 lines

We need to sort out exactly how far it makes sense to go, by showing concrete, down to earth uses for whatever substructure we want. Then show the effect can't be easily gotten through tools for power users or faking it for unsuspecting users via GUI, and that overall the complexity and performance cost is less than the win. Note that the success of the Unix way is in large part due to its use of few, simple concepts that can be combined endlessly; and tools following the same strategy. Adding extra concepts that current tools can't naturally handle has to be considered with extreme care.

Regarding this section and all you wrote before, I definitely agree. Simplicity wins hands down every time.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#381 lore raw
In reply to Spam
Quoted 2 lines · Spam

One suggestion is missed. It is to provide system calls for copy. That would also solve the problem.

No, it would not. If you read the POSIX.1 specification for cp carefully <http://www.unix.org/version3/online.html>, you will notice that the process for copying a regular file is carefully standardized. A POSIX.1-conforming cp implementation would not be allowed to copy additional streams, unless either additional options are given or the type of the file being copied is other than S_IFREG. And cp is just one example of a standardized file handling program.

Gunnar

Signature · 1 line
http://omnibus.ruf.uni-freiburg.de/~gritter
#382 lore raw
Quoted 8 lines
Quoted 2 lines · Spam

One suggestion is missed. It is to provide system calls for copy. That would also solve the problem.

Quoted 8 lines

No, it would not. If you read the POSIX.1 specification for cp carefully <http://www.unix.org/version3/online.html>, you will notice that the process for copying a regular file is carefully standardized. A POSIX.1-conforming cp implementation would not be allowed to copy additional streams, unless either additional options are given or the type of the file being copied is other than S_IFREG. And cp is just one example of a standardized file handling program.

It would solve the problem in Linux. However, it may not be POSIX.1 compatible. On the other hand I read that NTFS 5.0 is POSIX.1 compliant - and Windows uses copy system call. NTFS also has streams support using special character in the file names to select the streams.

Surely there must be a solution in Linux that will allow things like streams and meta-data(meta-streams) be visible to the user. ~S

Quoted 1 line

Gunnar

#383 lore raw
Quoted 11 lines · Spam
Quoted 11 lines
Quoted 2 lines · Spam

One suggestion is missed. It is to provide system calls for copy. That would also solve the problem.

No, it would not. If you read the POSIX.1 specification for cp carefully <http://www.unix.org/version3/online.html>, you will notice that the process for copying a regular file is carefully standardized. A POSIX.1-conforming cp implementation would not be allowed to copy additional streams, unless either additional options are given or the type of the file being copied is other than S_IFREG. And cp is just one example of a standardized file handling program.

Quoted 3 lines

It would solve the problem in Linux. However, it may not be POSIX.1 compatible. On the other hand I read that NTFS 5.0 is POSIX.1 compliant - and Windows uses copy system call.

You should obviously take a thorough read of at least the Base Definitions, section 2, 'Conformance' of POSIX.1 before you further comment on this issue.

Gunnar

#384 lore raw
In reply to Gunnar Ritter
Quoted 8 lines · Gunnar Ritter

No, it would not. If you read the POSIX.1 specification for cp carefully <http://www.unix.org/version3/online.html>, you will notice that the process for copying a regular file is carefully standardized. A POSIX.1-conforming cp implementation would not be allowed to copy additional streams, unless either additional options are given or the type of the file being copied is other than S_IFREG. And cp is just one example of a standardized file handling program.

We can safely ignore POSIX when it is too broken. cp could very well be modified to copy named streams except when the option --posix is specified or the environment variale POSIXLY_CORRECT is set.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#385 lore raw
Quoted 10 lines · Christer Weinigel
Quoted 8 lines · Gunnar Ritter

No, it would not. If you read the POSIX.1 specification for cp carefully <http://www.unix.org/version3/online.html>, you will notice that the process for copying a regular file is carefully standardized. A POSIX.1-conforming cp implementation would not be allowed to copy additional streams, unless either additional options are given or the type of the file being copied is other than S_IFREG. And cp is just one example of a standardized file handling program.

We can safely ignore POSIX when it is too broken.

Excuse me, but there's really nothing broken here with POSIX and cp. You're just making an insulting talk about a part of the specification which currently serves GNU/Linux and other Unix-like environments very well, and has done so for about twelve years now.

Quoted 2 lines

cp could very well be modified to copy named streams except when the option --posix is specified

Hey, you didn't ever even have a look at POSIX Shell & Utilities, did you? Then why are you making derogatory statements about it?

Quoted 1 line

or the environment variale POSIXLY_CORRECT is set.

Cool, data loss depending upon an environment variable which is even currently used by many programs unaware of such results. This really sounds like good engineering to me.

Gunnar

#386 lore raw
Quoted 4 lines · Gunnar Ritter

Excuse me, but there's really nothing broken here with POSIX and cp. You're just making an insulting talk about a part of the specification which currently serves GNU/Linux and other Unix-like environments very well, and has done so for about twelve years now.

"Broken" in the sense "POSIX mandates something that users wouldn't expect".

Quoted 5 lines
Quoted 2 lines

cp could very well be modified to copy named streams except when the option --posix is specified

Hey, you didn't ever even have a look at POSIX Shell & Utilities, did you? Then why are you making derogatory statements about it?

Those derogatory statments are really all in your mind.

Quoted 5 lines
Quoted 1 line

or the environment variale POSIXLY_CORRECT is set.

Cool, data loss depending upon an environment variable which is even currently used by many programs unaware of such results. This really sounds like good engineering to me.

How would you consider cp to cause "data loss" if it _besides_ copying the normal stream _also_ copied any named streams or xattrs belonging to the stream? How would it cause data loss if cp started using a theoretical copyfile syscall? It may not be 100% according to POSIX, but I'd definitely say that it does what the user expects.

Lots of GNU utilities already differ from POSIX mandated behaviour because the authors of those utilities belive that the POSIX mandated behaviour is confusing.

http://www.wlug.org.nz/POSIXLY_CORRECT

POSIXLY_CORRECT is an environment variable that some programs use to follow strict POSIX standards behaviour, where that isn't the default.

Probably the most well-known example of this is that POSIX states that filesystem blocks are 512 bytes per block, so the GNU fileutils such as df(1) and GNU tar(1) use 512 if the variable POSIXLY_CORRECT is set, and 1024 bytes per block by default.

Many of the GNU tools comply with POSIX by default, except for where the author thinks the POSIX standard is wrong or dumb. :) As a result, some programs also check if a variable named POSIX_ME_HARDER is set as an acceptable alias for POSIXLY_CORRECT. See Democracy Triumphs in Disk Units.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#387 lore raw
Quoted 7 lines · Christer Weinigel
Quoted 4 lines · Gunnar Ritter

Excuse me, but there's really nothing broken here with POSIX and cp. You're just making an insulting talk about a part of the specification which currently serves GNU/Linux and other Unix-like environments very well, and has done so for about twelve years now.

"Broken" in the sense "POSIX mandates something that users wouldn't expect".

Only if one breaks it by making extensions in an inappropriate way. This is not a fault of POSIX. POSIX usually allows a lot of sane ways to introduce extensions. There are usually valid interoperability arguments for behavior prescribed by POSIX. It is really not one of those standards where you want to ignore every second word because it is obviously nothing but committee nonsense.

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

or the environment variale POSIXLY_CORRECT is set.

Cool, data loss depending upon an environment variable which is even currently used by many programs unaware of such results. This really sounds like good engineering to me.

How would you consider cp to cause "data loss" if it _besides_ copying the normal stream _also_ copied any named streams or xattrs belonging to the stream?

You are reversing the argument. If additional streams are introduced inappropriately by extending the semantics of S_IFREG files, POSIX requires cp to lose the data. Your proposal would then make this loss of additional stream data dependent on an environment variable that is already in wide use. If it was set by accident, the data would be lost.

Besides, copying xattrs is usually permitted (POSIX.1-2004, XCU cp):

# If the implementation provides additional or alternate access control

# mechanisms (see the Base Definitions volume of IEEE Std 1003.1-2001,
# Section 4.4, File Access Permissions), their effect on copies of files
# is implementation-defined.

It is also permitted to add other S_IFXXX types and then let cp act in an implementation-defined manner on them (cf. my earlier message <[email protected]>).

The 'standardized' data loss would only occur if the standardized type of regular file, S_IFREG, was abused. This would really not be a fault of POSIX.

Quoted 3 lines

Lots of GNU utilities already differ from POSIX mandated behaviour because the authors of those utilities belive that the POSIX mandated behaviour is confusing.

Sure, but it is not the preferred method of adding features. In addition, most of the existing POSIXLY_CORRECT influences are nothing but cosmetical details in comparison to copying/not copying stream data.

Gunnar

#388 lore raw
Quoted 6 lines · Gunnar Ritter

Besides, copying xattrs is usually permitted (POSIX.1-2004, XCU cp):

# If the implementation provides additional or alternate access control

# mechanisms (see the Base Definitions volume of IEEE Std 1003.1-2001,
# Section 4.4, File Access Permissions), their effect on copies of files
# is implementation-defined.
Quoted 3 lines · In <[email protected]> you wrote:

A POSIX.1-conforming cp implementation would not be allowed to copy additional streams, unless either additional options are given or the type of the file being copied is other than S_IFREG.

I read this as that POSIX mandates that cp can absolutely not copy anything else but the file contents. That is what I called broken.

If we implement named streams as xattrs and that can be accessed with openat(..., O_XATTR) this means that cp is allowed to copy the xattrs (well, named streans don't neccesarily have to be "alternate access control mechanisms", but they can use the same xattr namespace).

That's quite ok in that case.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#389 lore raw
Quoted 9 lines · Christer Weinigel
Quoted 6 lines · Gunnar Ritter

Besides, copying xattrs is usually permitted (POSIX.1-2004, XCU cp):

# If the implementation provides additional or alternate access control

# mechanisms (see the Base Definitions volume of IEEE Std 1003.1-2001,
# Section 4.4, File Access Permissions), their effect on copies of files
# is implementation-defined.
Quoted 8 lines
Quoted 3 lines · In <[email protected]> you wrote:

A POSIX.1-conforming cp implementation would not be allowed to copy additional streams, unless either additional options are given or the type of the file being copied is other than S_IFREG.

I read this as that POSIX mandates that cp can absolutely not copy anything else but the file contents. That is what I called broken.

It would be really helpful if you read the specification before you comment on it or try to interpret my wording further.

Quoted 2 lines

If we implement named streams as xattrs and that can be accessed with openat(..., O_XATTR) this means that cp is allowed to copy the xattrs

No. Not if the file type of such beasts remains S_IFREG.

Quoted 2 lines

(well, named streans don't neccesarily have to be "alternate access control mechanisms", but they can use the same xattr namespace).

POSIX does not know anything about the 'xattr namespace'. It just allows 'additional or alternate access control mechanisms'. Which are, in turn, well-defined terms in the standard again. You can read about them too <http://www.unix.org/version3/>.

Gunnar

#390 lore raw
In reply to Christer Weinigel
Quoted 29 lines · Christer Weinigel
Quoted 9 lines · David Masover
|>Second, there are quite a few things which I might want to do, which can
|>be done with this interface and without patching programs,
| Such as?

They've been mentioned.

Quoted 6 lines

| Haven't seen any that made sense to me, sorry.
Sorry if they don't make sense to you, but I don't feel like discussing them now. Either you get it or you don't, either you agree or you don't. Read the archives.

Great argument. Not. There has been so much shit thrown around here so that it's impossible to keep track of all examples.

Could you please try summarize a few of the arguments that you find especially compelling? This thread has gotten very confused since there are a bunch of different subjects all being intermixed here.

What are we discussing?

Quoted 9 lines

1. Do we want support for named streams?

I belive the answer is yes, since both NTFS and HFS (that's the MacOS filesystem, isn't it?) supports streams we want Linux to support this if possible.

Anyone disagreeing?

No, we want files and directories that can do what streams can do. This means files that are also directories, plugins that aggregate the contents of a directory, files that inherit stat data, maybe I forget something ---- it is on my website.

Streams themselves are a bad idea because they are a filesystem inside of a file which is double the overhead, and they fragment the namespace.

Quoted 11 lines

4. What belongs in the generic VFS, what belongs in Reiser4?

Some things reiser4 do, such as files-as-directories need changes to the VFS because it breaks assumptions that the VFS makes (i.e. a deadlock or an oops when doing a hard link out of one).

Some other things reiser4 can do would be better if they were in the VFS since other filesystems might want to support the same functionality.

It is always better to lead by example. These ideas are too new for the other fs developers, they need 5 years to get used to them. Reiser4 should create something that works, and let others follow when they will.

Quoted 5 lines

Or Linux may not support some of the things reiserfs at all.

5. What belongs in the kernel, what belongs in userspace?

This is the wrong question. The right question is, what belongs in a unified namespace? Then having answered that, the extent to which it is easier to have all aspects of name resolution together in one body of code is an implementation detail that can change and evolve with time. A rapidly evolving namespace is easier to modify if it is all one body of code. Furthermore, the people doing the work should really be left to decide such implementation details themselves because they are more expert on their code than anyone else.

Hans

#391 lore raw
Quoted 8 lines · Hans Reiser

No, we want files and directories that can do what streams can do. This means files that are also directories, plugins that aggregate the contents of a directory, files that inherit stat data, maybe I forget something ---- it is on my website.

Streams themselves are a bad idea because they are a filesystem inside of a file which is double the overhead, and they fragment the namespace.

There really isn't a difference between your files are directories think and named streams that can be accessed with openat. Both are concepts that won't fit into the classical Unix entity "file".

As far as I can tell, windows named streams could be exposed via a file-as-directory model. In that case what windows sees as foo.txt:icon would be accessed as foo.txt/icon instead. The question in my mind is if that is a desirable concept. I can see that it is very a attractive concept because it's easy to explain to users, but at the same time, a lot of things suggest that the files-as- directories concept is a bad idea. I mentioned a few of the possible problems in the grand-grand-parent of this message.

Try to address those problems. It may be that your answer is "I don't give a shit about existing backup applications", or "I don't care about security in existing web-servers, they'll have to be taught about reiser4", but in that case say so. Or show how it's not a problem with existing applictions.

Quoted 4 lines

It is always better to lead by example. These ideas are too new for the other fs developers, they need 5 years to get used to them. Reiser4 should create something that works, and let others follow when they will.

Pretty please, give up the marketing bullshit. I already know that you belive reiser4 is the best thing since sliced bread. You say that "reiser4 should create something that works". So far you haven't convinced me that reiser4 actually "works". It's a nice proof of concept, but it definitely does not feel like something that I'd like in the mainstream kernel until those problems are addressed.

Have you looked at the VFS deadlocks that Alexander Viro pointed at?

Does tar work with files-as-directories? Can I use tar to make a backup of my resier4 directory. What changes would be required to the tar application or to the tar file format to make this work?

Can I do a cp -a /home/wingel to a NFS-mounted drive on some other computer to make a backup of my home directory or will I lose data that way? What changes would be required to cp to make this work?

Please remember that reiser4 does not exist in a vaccum. If you want to make a fundamental change to unix concepts that have been with us for three decades, it's really up to you to show that it doesn't break too many existing applications.

Quoted 8 lines

This is the wrong question. The right question is, what belongs in a unified namespace? Then having answered that, the extent to which it is easier to have all aspects of name resolution together in one body of code is an implementation detail that can change and evolve with time. A rapidly evolving namespace is easier to modify if it is all one body of code. Furthermore, the people doing the work should really be left to decide such implementation details themselves because they are more expert on their code than anyone else.

I still haven't seen a good answer to the namespace problem. You say that files-as-directories is great, I'm definitely not convinced.

The VFS is the body of code that handles most aspects of name resolution today, so it sounds as if reiser4 is supposed to be the new super-VFS, and you want to keep it that way and do not want to integrate with the current VFS. In that case resier4 really does not belong in the mainstream kernel since it's your pet research project.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#392 lore raw

so far the best answer that I've seen is a slight varient of what Hans is proposing for the 'file-as-a-directory'

make the base file itself be a serialized version of all the streams and if you want the 'main' stream open file/. (or some similar varient)

this doesn't address the hard-link issue, but it should handle the backup problems (your backup software just goes through the files and what it gets is suitable for backups).

you will ask what serializer to user, and my answer is to let one of the streams tell you, and have the kernel make a call out to userspace to execute the appropriate program (note that this means that tar is not put into the kernel)

in fact it may make sense to just open file/file to get at the 'main' stream of the file (there may be cases where the concept of a single main stream may not make sense)

so if this solves the tool/backup problem then we can look and figure out if there's a reasonable way to solve the hard-link problem

David Lang

Quoted 102 lines · Christer Weinigel

Date: 08 Sep 2004 00:13:12 +0200
From: Christer Weinigel <[email protected]>
To: Hans Reiser <[email protected]>
Cc: Christer Weinigel <[email protected]>,
David Masover <[email protected]>,
Horst von Brand <[email protected]>, Spam <[email protected]>, Tonnerre <[email protected]>, Linus Torvalds <[email protected]>, Pavel Machek <[email protected]>, Jamie Lokier <[email protected]>, Chris Wedgwood <[email protected]>, [email protected], Christoph Hellwig <[email protected]>, [email protected], [email protected], Alexander Lyamin aka FLX <[email protected]>, ReiserFS List <[email protected]>
Subject: Re: silent semantic changes with reiser4

Quoted 8 lines · Hans Reiser

No, we want files and directories that can do what streams can do. This means files that are also directories, plugins that aggregate the contents of a directory, files that inherit stat data, maybe I forget something ---- it is on my website.

Streams themselves are a bad idea because they are a filesystem inside of a file which is double the overhead, and they fragment the namespace.

There really isn't a difference between your files are directories think and named streams that can be accessed with openat. Both are concepts that won't fit into the classical Unix entity "file".

As far as I can tell, windows named streams could be exposed via a file-as-directory model. In that case what windows sees as foo.txt:icon would be accessed as foo.txt/icon instead. The question in my mind is if that is a desirable concept. I can see that it is very a attractive concept because it's easy to explain to users, but at the same time, a lot of things suggest that the files-as- directories concept is a bad idea. I mentioned a few of the possible problems in the grand-grand-parent of this message.

Try to address those problems. It may be that your answer is "I don't give a shit about existing backup applications", or "I don't care about security in existing web-servers, they'll have to be taught about reiser4", but in that case say so. Or show how it's not a problem with existing applictions.

Quoted 4 lines

It is always better to lead by example. These ideas are too new for the other fs developers, they need 5 years to get used to them. Reiser4 should create something that works, and let others follow when they will.

Pretty please, give up the marketing bullshit. I already know that you belive reiser4 is the best thing since sliced bread. You say that "reiser4 should create something that works". So far you haven't convinced me that reiser4 actually "works". It's a nice proof of concept, but it definitely does not feel like something that I'd like in the mainstream kernel until those problems are addressed.

Have you looked at the VFS deadlocks that Alexander Viro pointed at?

Does tar work with files-as-directories? Can I use tar to make a backup of my resier4 directory. What changes would be required to the tar application or to the tar file format to make this work?

Can I do a cp -a /home/wingel to a NFS-mounted drive on some other computer to make a backup of my home directory or will I lose data that way? What changes would be required to cp to make this work?

Please remember that reiser4 does not exist in a vaccum. If you want to make a fundamental change to unix concepts that have been with us for three decades, it's really up to you to show that it doesn't break too many existing applications.

Quoted 8 lines

This is the wrong question. The right question is, what belongs in a unified namespace? Then having answered that, the extent to which it is easier to have all aspects of name resolution together in one body of code is an implementation detail that can change and evolve with time. A rapidly evolving namespace is easier to modify if it is all one body of code. Furthermore, the people doing the work should really be left to decide such implementation details themselves because they are more expert on their code than anyone else.

I still haven't seen a good answer to the namespace problem. You say that files-as-directories is great, I'm definitely not convinced.

The VFS is the body of code that handles most aspects of name resolution today, so it sounds as if reiser4 is supposed to be the new super-VFS, and you want to keep it that way and do not want to integrate with the current VFS. In that case resier4 really does not belong in the mainstream kernel since it's your pet research project.

/Christer

--
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux Christer Weinigel <[email protected]> http://www.weinigel.se
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#393 lore raw
Quoted 10 lines · David Lang

so far the best answer that I've seen is a slight varient of what Hans is proposing for the 'file-as-a-directory'

make the base file itself be a serialized version of all the streams and if you want the 'main' stream open file/. (or some similar varient)

in fact it may make sense to just open file/file to get at the 'main' stream of the file (there may be cases where the concept of a single main stream may not make sense)

So what happens if I have a text file foo.txt and add an author attribute to it? When I read foo.txt the next time it's supposed to give me a serialized version with both the contents of foo.txt _and_ the author attribute?

That would definitely confuse me.

Or did I misunderstand something?

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#394 lore raw
Quoted 15 lines · Christer Weinigel

Subject: Re: silent semantic changes with reiser4

Quoted 10 lines · David Lang

so far the best answer that I've seen is a slight varient of what Hans is proposing for the 'file-as-a-directory'

make the base file itself be a serialized version of all the streams and if you want the 'main' stream open file/. (or some similar varient)

in fact it may make sense to just open file/file to get at the 'main' stream of the file (there may be cases where the concept of a single main stream may not make sense)

Quoted 9 lines

So what happens if I have a text file foo.txt and add an author attribute to it? When I read foo.txt the next time it's supposed to give me a serialized version with both the contents of foo.txt _and_ the author attribute?

That would definitely confuse me.

Or did I misunderstand something?

good point. under my scheme you would need to access foo.txt/foo.txt or foo.txt/. instead of just foo.txt

I guess my way would work if there is a way to know that a file has been extended (or if you just make it a habit of opening the file/file instead of just file) but not for random additions of streams to otherwise normal files.

Oh well, it seemed like a easy fix (and turned out to be to easy to be practical)

David Lang

Quoted 8 lines

/Christer

--
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux Christer Weinigel <[email protected]> http://www.weinigel.se

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#395 lore raw
Quoted 2 lines · David Lang
Quoted 1 line · Christer Weinigel

David Lang <[email protected]> writes:

[...]

Quoted 7 lines
Quoted 7 lines

So what happens if I have a text file foo.txt and add an author attribute to it? When I read foo.txt the next time it's supposed to give me a serialized version with both the contents of foo.txt _and_ the author attribute?

That would definitely confuse me.

Quoted 7 lines
Quoted 1 line

Or did I misunderstand something?

good point. under my scheme you would need to access foo.txt/foo.txt or foo.txt/. instead of just foo.txt

I guess my way would work if there is a way to know that a file has been extended

Out of the question if you want to use legacy tools.

Quoted 3 lines

(or if you just make it a habit of opening the file/file instead
of just file) but not for random additions of streams to otherwise normal files.

And then a file called foo inside directory foo will have to be refered to as foo/foo/foo...

Quoted 2 lines

Oh well, it seemed like a easy fix (and turned out to be to easy to be practical)

My gut feeling is that this whole concept is too messed up to be sorted out. That something can easily be done given whatever underlying structure there is doesn't make it worthwhile to do it. Just as it is absurdly easy to allow hard links to directories if you look at how they are implemented, but the consecuences aren't that easy to live with.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#396 lore raw
In reply to David Lang
Quoted 5 lines · David Lang

so far the best answer that I've seen is a slight varient of what Hans is proposing for the 'file-as-a-directory'

make the base file itself be a serialized version of all the streams and if you want the 'main' stream open file/. (or some similar varient)

As has been said previously, all such proposals except for those with two leading slashes would directly violate POSIX.1-2004, Base Definitions, 4.11 'Pathname Resolution'. In particular,

# A pathname that contains at least one non-slash character and that

# ends with one or more trailing slashes shall be resolved as if a
# single dot character ( '.' ) were appended to the pathname.

A regular file name with special semantics, in contrast, would not violate POSIX, in particular if a stat() on it would not return one of the standard S_IFXXX types. Inappropriate operations could just fail with EINVAL on such files, and it would be unspecified by the standard how they were handled by 'cp' or other standardized utilities.

Having a separate S_IFXXX type for streamed files would also give attention to portability issues easily. It might in fact be better to have existing applications fail explicitly with them than to make this as transparent as possible, but hope for good luck.

Utilities like 'tar' would just say 'Unknown file type' or the like and exclude such files from the archives they create. This would lead the user's explicit attention to the data loss. He might then choose to ignore the error, or to get a special version of 'tar' to handle it.

It would also be clear to programmers and users that such files are special things they don't normally need. Regular Linux installations might totally ignore them. If streamed files are only intended for use with CIFS or other special cases, that might be a more clean solution for this issue than watering down the existing portable file semantics.

Gunnar

#397 lore raw
Quoted 10 lines · Gunnar Ritter

Having a separate S_IFXXX type for streamed files would also give attention to portability issues easily. It might in fact be better to have existing applications fail explicitly with them than to make this as transparent as possible, but hope for good luck.

Utilities like 'tar' would just say 'Unknown file type' or the like and exclude such files from the archives they create. This would lead the user's explicit attention to the data loss. He might then choose to ignore the error, or to get a special version of 'tar' to handle it.

Quoted 6 lines

It would also be clear to programmers and users that such files are special things they don't normally need. Regular Linux installations might totally ignore them. If streamed files are only intended for use with CIFS or other special cases, that might be a more clean solution for this issue than watering down the existing portable file semantics.

You could actually use two S_IFXXX types - one for stream files which should be archived by backup programs because they contain real information, and a different one for stream files which should not be archived because they are merely a different "view" of data from the main file. This simplifies the archival problem because legacy programs would complain that the file types aren't handled - alerting the user to potential data loss, and updated programs would automatically backup only the attributes that are required.

Signature · 2 lines
-Julian Blake Kongslie
<[email protected]>
#398 lore raw
In reply to David Lang
Quoted 5 lines · David Lang

so far the best answer that I've seen is a slight varient of what Hans is proposing for the 'file-as-a-directory'

make the base file itself be a serialized version of all the streams and if you want the 'main' stream open file/. (or some similar varient)

Serialized how? cpio(1), tar(1), cat(1)ed together, ...? Ship it via FTP or something, how do you unpack on a traditional filesystem? On a wacky filesystem?

Note that this has the same performance implications as the targzfilesystem concept, just the other way around: Instead of unpacking on the fly it is packing on the fly. And paying a (possibly huge) cost for "simple, everyday operations" in some cases that can't be distinguished from ones where there is no such cost is very wrong to me.

Quoted 1 line

this doesn't address the hard-link issue,

It has to be solved!

Quoted 3 lines

but it should handle the backup
problems (your backup software just goes through the files and what it gets is suitable for backups).

But how do you recreate the original "files" later? You only get the serialized version back. Even worse, if you unpack on top of the original wacky file, you'd get a wacky file, where the main stream is the serialized version of the original... Rinse and repeat. [Shudder]

Quoted 4 lines

you will ask what serializer to user, and my answer is to let one of the streams tell you, and have the kernel make a call out to userspace to execute the appropriate program (note that this means that tar is not put into the kernel)

If I want to see it serialized via tar(1), and you via cpio(1), and somebody else via "take the icon stream, discard everything else", how do you handle that?

Quoted 3 lines

in fact it may make sense to just open file/file to get at the 'main' stream of the file (there may be cases where the concept of a single main stream may not make sense)

What do you do then?

Quoted 1 line

so if this solves the tool/backup problem

It makes it much worse, AFAICS.

Quoted 2 lines

then we can look and figure out
if there's a reasonable way to solve the hard-link problem

Right. And if none is found, can we drop this madness?

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#399 lore raw
Quoted 11 lines · Horst von Brand
Quoted 5 lines · David Lang

so far the best answer that I've seen is a slight varient of what Hans is proposing for the 'file-as-a-directory'

make the base file itself be a serialized version of all the streams and if you want the 'main' stream open file/. (or some similar varient)

Serialized how? cpio(1), tar(1), cat(1)ed together, ...? Ship it via FTP or something, how do you unpack on a traditional filesystem? On a wacky filesystem?

as I said elsewhere in the origional post the owner of the file sets the tool to use to serialize it. if you want tar set it to use tar, if you want cpio set that. in both cases it would be a call out to userspace

Quoted 5 lines

Note that this has the same performance implications as the targzfilesystem concept, just the other way around: Instead of unpacking on the fly it is packing on the fly. And paying a (possibly huge) cost for "simple, everyday operations" in some cases that can't be distinguished from ones where there is no such cost is very wrong to me.

this is a (huge) oppurtunity for the filesystem to provide for an optimization. if it is able to maintain a serialized version then this could turn into a simple read. no it's not always the simple obvious thing, but very few things are.

the problem of things having unexpected side effects in an uncontrolled server already are out there.

Quoted 3 lines
Quoted 1 line

this doesn't address the hard-link issue,

It has to be solved!

I never disputed that, in fact I pointed out that that issue remained, with the idea (since disproved) that this suggestion would settle the other issues sufficiantly to let everyone concentrate on the link issue.

Quoted 9 lines
Quoted 4 lines

but it should handle the backup
problems (your backup software just goes through the files and what it gets is suitable for backups).

But how do you recreate the original "files" later? You only get the serialized version back. Even worse, if you unpack on top of the original wacky file, you'd get a wacky file, where the main stream is the serialized version of the original... Rinse and repeat. [Shudder]

how smart if the filesystem plugin? can the filesystem recognise compound filetypes? if so then it can tell what's happening and do the right thing when recreating the file

Quoted 8 lines
Quoted 4 lines

you will ask what serializer to user, and my answer is to let one of the streams tell you, and have the kernel make a call out to userspace to execute the appropriate program (note that this means that tar is not put into the kernel)

If I want to see it serialized via tar(1), and you via cpio(1), and somebody else via "take the icon stream, discard everything else", how do you handle that?

the owner of the file decides (or whoever that owner grants the permissions to to change it to a different system)

Quoted 5 lines
Quoted 3 lines

in fact it may make sense to just open file/file to get at the 'main' stream of the file (there may be cases where the concept of a single main stream may not make sense)

What do you do then?

the backup doesn't really care. if you refrence files by file/file then you can't really say that file/foo is any more the 'main' stream then file/bar. but if you call it file/. then you do have to define what themain version is.

Quoted 10 lines
Quoted 1 line

so if this solves the tool/backup problem

It makes it much worse, AFAICS.

Quoted 2 lines

then we can look and figure out
if there's a reasonable way to solve the hard-link problem

Right. And if none is found, can we drop this madness?

solutions are known , it's just not known if any solution can be implemented with suitable performance (lock the entire filesystem when you go to do a dangerous action goes a _long_ way towards solving the problems, but it's not acceptable due to performance issues)

the fundamental problem with hard links and directories (maintaining a sane path) is the exact same problem that the *notify tools need solved so that a program can setup notification on a directory tree that it's interested in without tieing up a large number of file handles. it's a odd problem, but if it can be solved there are several doors that open up.

David Lang

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#400 lore raw
In reply to David Masover

Salut,

Quoted 10 lines · David Masover

Transparently?

It _works_ to do
zcat file.gz > /tmp/file
vim /tmp/file
gzip -c /tmp/file > file.gz
rm /tmp/file
You can even do that as a script, call it zvim. You could even do it as a generic script, where "vim" is replaced with "$1". But is it as elegent as transparently compressed files?

...or you get yourself a sane editor which supports
gzopen/gzread/gzwrite/gzclose. You can do that in userland, no kernel implementation needed.

Tonnerre

#401 lore raw
In reply to David Masover
Quoted 3 lines · David Masover

There are some things which can't be solved without patching. Version control is one such thing.

You can do most of the version control stuff without patching, for instance selecting a version by name is easy (see Clearcase). Clearcase users mostly do not patch user space binaries. But your point is generally true that there are features that knowing about them allows you to better employ them.

Quoted 10 lines

But then there can be more generic patches
-- as soon as the transaction API is done, you only have to patch apps to use that, and have a version control reiser4 plugin.

| I'd go the other way around: Get userspace to agree on a common framework,
| make it work in userspace; if (extensive, hopefully) experience shows that
| a pure userspace solution has issues that can't be solved except by kernel
| assistance, so be it.

We already have such a framework -- it's called "VFS".

If doing it in the kernel is so hard, why hasn't it stopped us yet? ;-)

I am not asking other people to contibute their labor to making this thing they view as infeasible work, I am just asking them to get out of the way please, and let the users decide whether they like it.

Nothing significant about the reiserfs project was thought likely to work by sensible people before it worked. I am a bit used to that. After all, Oracle's IFS and several similar projects proved filesystems on top of balanced trees cannot perform well....;-)

Anyone who complains about kernel bloat should first consider that reiser4 is not by any means the largest filesystem in the kernel.

Hans

#402 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds

In a graphical environment, the "icon" stream is a good example of this. It literally has _nothing_ to do with the data in the main stream. The only linkage is a totally non-technical one, where the user wanted to associate a secondary stream with the main stream _without_ altering the main one. THAT is where named streams make sense.

The icon doesn't belong in the document, that was a catastrophic disaster in early MacOS (although they only use an index). Users want to manage their icon choices, tags, tooltip notes and attached labels, and you cannot do that in the file if you don't own the file.

Also the icon is *not* unrelated to the file in a modern GUI, eg rox and nautilus uses scaled versions of the content for many media types and will show you pictures, frames from a movie etc to help you remember the content. On top they then add user specific annotations.

The things that are more independant are:

"This file was created by OpenOffice 1.2"
"This is a text/plain file in UTF-8"
"This file has a UUID of ...."

The type has dragons because you get heirarchical typing within documents (consider XML containing namespaces)

(UUIDs being one really useful thing we don't tag everywhere onto files that would be a godsend on the desktop providing they moved with the file on rename)

Alan

#403 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Alan" == Alan Cox <[email protected]> writes:

Alan> The icon doesn't belong in the document, that was a catastrophic Alan> disaster in early MacOS (although they only use an index). Users Alan> want to manage their icon choices, tags, tooltip notes and Alan> attached labels, and you cannot do that in the file if you don't Alan> own the file.

Most desktop environments provide defaults, but let the user customize their own settings. The substream *could* (I'm not saying if it's a good idea. It's just a possibility.) be used to provide a default icon.

Alan> Also the icon is *not* unrelated to the file in a modern GUI, eg Alan> rox and nautilus uses scaled versions of the content for many Alan> media types and will show you pictures, frames from a movie etc to Alan> help you remember the content.

For documents, the icon is not completely unrelated, but it isn't something that can be directly extracted. Generating thumbnails usually takes time, so the filemanagers need to cache it somewhere. Right now, they use ~/.thumbnails (at least that's what freedesktop.org recommends), but then you run into consistency issues -- when you delete the file, the thumbnail should be deleted too, and when you rename/move the file, the thumbnail should be attached to the new filename, and not have to be regenerated.

Alan> On top they then add user specific annotations.

And I think that would be a good use of substreams/attributes/metas too. You don't really want to store it in some separate location, because you run into the same consistency issues as I mentioned before. Sure, if you move a file using Nautilus, it may be smart enough to move the annotations, but I'd bet that mv doesn't know what to do.

Alan> The things that are more independant are:

Alan> "This file was created by OpenOffice 1.2" "This is a text/plain Alan> file in UTF-8" "This file has a UUID of ...."

Alan> The type has dragons because you get heirarchical typing within Alan> documents (consider XML containing namespaces)

Well, I think the web guys are going to have to figure something out, and we can just follow what they do ...

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#404 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

There's no point to having the kernel export information that is already inherent in the main stream.

As said before, it can be bounced :)

You have support for streams, someone could read the inherent information and write it to the stream. At the risk of these two streams being unsynced temporarily, requiring some syncer.

Quoted 2 lines

I've seen all these examples of exposing MP3 ID information as a "side stream", and that's TOTALLY POINTLESS! The information is already there,

No it's not pointless :)

Quoted 2 lines

it's in a standard format, and exporting it as a stream buys you absolutely nothing.

There is no nice standard way of finding MP3s by the ID information.

find music/ -name *mp3/..metas/id3/ | xargs grep -i "dream theater"

or something. Which reveals the metadata to find.
The kernel doesn't have to parse the ID3 tags, that can be handled in userspace by a daemon, updated based on mtimes or somesuch, yes?

Quoted 3 lines

Which means that normally we really don't _want_ named streams. In 99% of all cases we can use equally good - and _much_ simpler - tool-based solutions.

Sure, when someone goes through every application from ls to nautilus and rams a tool-based patch down the app maintainer's throat and then we're still dependent on the filesystem layout, which is easier to break than the filesystem itself :)

Quoted 2 lines

Which means that the only _real_ technical issue for supporting named streams really ends up being things like samba, which want named streams

So they should be implemented for samba and other guys can come up with alternative means for using them ;)

Signature · 1 line
mjt
#405 lore raw
Quoted 4 lines · On Wednesday September 1, [email protected] wrote:
Quoted 3 lines · Linus Torvalds

There's no point to having the kernel export information that is already inherent in the main stream.

Having read this quote a few times in this thread I finally figured out what was wrong with it.

It seems to imply that "iso9660" shouldn't be in the kernel. After all, it just exports information that is already in the underlying device.
It doesn't provide any "mediate multiple access" benefit as a read-only filesystem doesn't require any mediation between users. It might provide some caching benefit, but I somehow don't think that is the reason that it is in the kernel.

Now I'm happy to agree that there is a case for "iso9660" but not for "tarfs", but I don't think it is that "there's no point to having the kernel export information that is already [there]"

NeilBrown

#406 lore raw
Quoted 3 lines · Neil Brown

It seems to imply that "iso9660" shouldn't be in the kernel. After all, it just exports information that is already in the underlying device.

Some applications do isofs/udf in userspace, DVD playback for example.

Quoted 2 lines

It doesn't provide any "mediate multiple access" benefit as a read-only filesystem doesn't require any mediation between users.

Umm... you can have permissions for users and you need the kernel to determine who can access what.

--cw

#407 lore raw
In reply to Horst von Brand

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Horst von Brand wrote:

| Pavel Machek <[email protected]> said:
|
|>David Masover <[email protected]> said:
|
|
| [...]

[...]
| You do need extra tools anyway, placing them in the kernel is cheating (and
| absolutely pointless, IMHO).

Repeat after me: plugins in kernel does NOT equal tools in kernel.

It's not hard to, for instance, imagine a generic plugin for archive manipulation which talks to a userspace daemon/library. The kernel doesn't know anything other than (maybe) a list of extensions that are archives. All else is handled in userspace -- the idea that "this is a zipfile" and "zipfiles can be extracted with the 'zip' command" are all in userspace.

It's not about the kernel, it's about the interface. And see my other mail:
cat foo.zip/README
less foo.zip/contents/bar.c
is a lot easier than
lynx http://google.com/search?q=zip
emerge zip
man zip
unzip foo.zip
cat bar.c
which already assumes quite a lot of expertise.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUieXgHNmZLgCUhAQLj8Q//VEuRWg9RB/PVg/M1t36SraKIRdbrgGq6 0DYLZwwVWkfz+UKGhUkACNGQzhHG3ySn/lRjqXCsJhzMOzfgmte2CwgGxunHYqdp W69OSZ1vv0ZLpdpzEkVdMfRhhg8RBL54Ewj6fQu7G1ZzJ0J0HUFCEZIsVoL4zJfk +GFl4M+vvc+152HipEDsvrrPasc0dSxmryyzPbyABrfPRi3l2E3Okfewn/4MCC1E H/L04nmvJQ2OSZYu1eSMSpnLrgQdZJbYefRuSinpqfKhjpwQXjj2rnfcPm+xc3hh FDMlAUPA/DAlthTuhxd9+8iDPEARY57/FAMFTalYVhBEv0NWieMMn9V6R8BbnhZW 8y9EWu0Nvoowg9qElkVjWEt5M1Rbeuq4TDviyVyPYg/KcTKDGpjLuyQQbWv6hMVM WAx6THtRK8Ji/XLCj4Kh6qDOEYP1nuwqLYSWv13BtNAJ3bVeKryOJ4Er4AmuJomA uydBPQVsrtzTNmk7Eul2ZjrxXdESpUnUIAbu7H1Gbn51uP4rQTdlXkdPTv6s/wvG OSOMWKs28GBFsG3+IJLnoN2wxrffpgTBEAaPwbh9TRFKLIH+pkCFSx2cIkTj93nr RjETFObjYdJGhGygrq0xOmN5e7iprdxhxd3/+fvjQxnZGxXT3fZLUwhVX9/zhaXG cxau2Onr3jc=
=MrR9
-----END PGP SIGNATURE-----

#408 lore raw

Hi!

Quoted 13 lines

| You do need extra tools anyway, placing them in the kernel is cheating (and
| absolutely pointless, IMHO).

Repeat after me: plugins in kernel does NOT equal tools in kernel.

It's not hard to, for instance, imagine a generic plugin for archive manipulation which talks to a userspace daemon/library. The kernel doesn't know anything other than (maybe) a list of extensions that are archives. All else is handled in userspace -- the idea that "this is a zipfile" and "zipfiles can be extracted with the 'zip' command" are all in userspace.

Quoted 10 lines

It's not about the kernel, it's about the interface. And see my other mail:
cat foo.zip/README
less foo.zip/contents/bar.c
is a lot easier than
lynx http://google.com/search?q=zip
emerge zip
man zip
unzip foo.zip
cat bar.c
which already assumes quite a lot of expertise.

If you only want nice user interface, you can have that today. Its done using coda, and hosted at uservfs.sf.net.
Pavel

Signature · 1 line
When do you have heart between your knees?
#409 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Pavel" == Pavel Machek <[email protected]> writes:

Pavel> If you only want nice user interface, you can have that Pavel> today. Its done using coda, and hosted at uservfs.sf.net.

Hmmm. Last activity seems to be about three years ago. Any idea if it works with Linux 2.6? Is there any chance at all that it will be merged into the kernel?

Can anyone comment on the speed? I don't have the ability right now to try out any new kernel stuff.

The user interface isn't that nice either. There's no browsability for the methods. (Hmm. I have this strange .deb file. How do I look at the contents? #utar? #ugz? ...) And honestly, I think that "#" hack is pretty ugly. There's also possible name conflicts -- although it's highly unlikely -- if I have a file that just happens to be called foo.tar.gz#utar, I can't browse foo.tar.gz.

There may be some nice ideas in there, but from a quick lookover, it seems to be lacking in execution. I'll probably give it a try anyways, during my next round of kernel compiling/testing.

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#410 lore raw
In reply to Pavel Machek
Quoted 11 lines · Pavel Machek
Quoted 10 lines

It's not about the kernel, it's about the interface. And see my other mail:
cat foo.zip/README
less foo.zip/contents/bar.c
is a lot easier than
lynx http://google.com/search?q=zip
emerge zip
man zip
unzip foo.zip
cat bar.c
which already assumes quite a lot of expertise.

Quoted 2 lines

If you only want nice user interface, you can have that today. Its done using coda, and hosted at uservfs.sf.net.

Can I do these already using uservfs?

cd
less foo.zip/contents/bar.c
less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c grep -R obscure_label ~/RPMS

No I can't.

Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can only do the above with convenient paths, completion etc. if I mount my home directory and distfiles over uservfs, so that all I/O in those directories including ordinary accesses is through uservfs, and have the _real_ storage elsewhere. That's quite illogical, cumbersome to set up, and slows down normal I/O significantly.

I can do some of those things in a similar way if I do them inside Midnight Commander, Gnome, KDE, or Emacs -- but even then a path that works in one of those doesn't work in another, and the extensions don't even work in all programs, plugins or scripts for those systems.

As we're adding file-as-directories to VFS, I propose it would be good to have *hooks* in place, and non-metadata namespace reserved inside the directories, so that uservfs-like userspace filesystems can be auto-mounted at those places and synchronised with the file.

In other words, something that makes the next version of uservfs a whole lot more useful.

The good news is that a lot of the hooks are falling into place already.

Kernel support is needed to handle the interface, cacheing to trigger deletions, page cacheing so that I/O is a normal speed and mmap() works, moveable mounts so that renames work and the cached data followed them, and coherency so that if the underlying archive is changed, it invalidates cached data with the appropriate degree of synchronisation. Note that the current dnotify cannot provide the latter with any strength; it's too weak an interface (so is inotify). (You can omit a lot of these things, but then you get a flaky interface).

The FUSE project looks like it will provide an efficient interface to the page cache, avoiding round trips where possible and supporting mmap. If it doesn't, it should. (Similar VFS facilities are useful for distributed filesystems). The presently discussed file-as-directory semantics will provide a nice view, and force the necessary minimal tool changes, so _all_ tools will be fine with the new uservfs.

So all I am asking for is a facility to auto-mount with file-as-directory, and the ability for a userspace daemon to be notified of regular file modifications synchronously. Both can be added later, once file-as-directory and moveable mounts are established. (fcntl leases and dnotify _almost_ provide this, but they don't. Looks like incoherent hacks keep getting added all over the place for Samba... :).

That's all. Kernel space needs to provide very little that it doesn't already provide, for this rather nice and universally usable feature to work.

-- Jamie

#411 lore raw

Hi!

Quoted 11 lines
Quoted 11 lines
Quoted 10 lines

It's not about the kernel, it's about the interface. And see my other mail:
cat foo.zip/README
less foo.zip/contents/bar.c
is a lot easier than
lynx http://google.com/search?q=zip
emerge zip
man zip
unzip foo.zip
cat bar.c
which already assumes quite a lot of expertise.

Quoted 7 lines
Quoted 2 lines

If you only want nice user interface, you can have that today. Its done using coda, and hosted at uservfs.sf.net.

Can I do these already using uservfs?

cd
less foo.zip/contents/bar.c

less foo.zip#uzip/contents/bar.c

Quoted 1 line

less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c

less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c

Quoted 1 line

grep -R obscure_label ~/RPMS

I do not think you'd want this. How would you tell grep obscure label without entering archives, then?

Quoted 5 lines

No I can't.

Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can only do the above with convenient paths, completion etc. if I mount my

Well, you can... or at least I had version somewhere that automagically prepended /overlay when file was not found. That way, normal requests were fast and uservfs were used whenever it was needed... At the price of little uglyness and slowing down -ENOENTs.

Quoted 7 lines

As we're adding file-as-directories to VFS, I propose it would be good to have *hooks* in place, and non-metadata namespace reserved inside the directories, so that uservfs-like userspace filesystems can be auto-mounted at those places and synchronised with the file.

In other words, something that makes the next version of uservfs a whole lot more useful.

Yep, something less hacky would be nice.

Quoted 4 lines

So all I am asking for is a facility to auto-mount with file-as-directory, and the ability for a userspace daemon to be notified of regular file modifications synchronously. Both can be added later, once file-as-directory and moveable mounts are

If the userspace daemon is synchronously notified of file modification... will not that lead to very ugly deadlocks?
Pavel

Signature · 2 lines
Horseback riding is like software...
...vgf orggre jura vgf serr.
#412 lore raw
Quoted 7 lines · Pavel Machek
Quoted 4 lines

Can I do these already using uservfs?

cd
less foo.zip/contents/bar.c

less foo.zip#uzip/contents/bar.c

Quoted 8 lines
Quoted 1 line

less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c

less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c

Quoted 1 line

grep -R obscure_label ~/RPMS

I do not think you'd want this. How would you tell grep obscure label without entering archives, then?

Granted, this is where we need "grep --recurse-into-files" or, more generally useful, a find option.

Quoted 10 lines
Quoted 5 lines

No I can't.

Even using the right "#" names for uservfs ('foo.zip#zip' etc.), I can only do the above with convenient paths, completion etc. if I mount my

Well, you can... or at least I had version somewhere that automagically prepended /overlay when file was not found. That way, normal requests were fast and uservfs were used whenever it was needed... At the price of little uglyness and slowing down -ENOENTs.

There you go... a kernel hook to prepend /overlay works :) These hook I keep mentioning, they are really very simple :)

Quoted 7 lines
Quoted 4 lines

So all I am asking for is a facility to auto-mount with file-as-directory, and the ability for a userspace daemon to be notified of regular file modifications synchronously. Both can be added later, once file-as-directory and moveable mounts are

If the userspace daemon is synchronously notified of file modification... will not that lead to very ugly deadlocks?

Not if it's written properly.

-- Jamie

#413 lore raw
Quoted 12 lines · Jamie Lokier
Quoted 11 lines · Pavel Machek
Quoted 4 lines

Can I do these already using uservfs?

cd
less foo.zip/contents/bar.c

less foo.zip#uzip/contents/bar.c

Quoted 1 line

less /usr/portage/distfiles/perl-5.8.5.tar.gz/contents/toke.c

less /usr/portage/distfiles/perl-5.8.5.tar.gz#utar/contents/toke.c

Quoted 8 lines
Quoted 4 lines
Quoted 1 line

grep -R obscure_label ~/RPMS

I do not think you'd want this. How would you tell grep obscure label without entering archives, then?

Granted, this is where we need "grep --recurse-into-files" or, more generally useful, a find option.

FWIW, this is how Windows does it now. As of XP, 'Find files' has an option, enabled by default, to look inside archives. If you tell it to look for a driver in a given directory it will also look inside .cab and .zip files. It's extremely useful, I would imagine someone who uses XP a lot will come to expect this feature.

Of course, no idea how it's implemented, but users like it.

Lee

#414 lore raw

Lee Revell <[email protected]> said:

[...]

Quoted 5 lines

FWIW, this is how Windows does it now. As of XP, 'Find files' has an option, enabled by default, to look inside archives. If you tell it to look for a driver in a given directory it will also look inside .cab and .zip files. It's extremely useful, I would imagine someone who uses XP a lot will come to expect this feature.

It is trivial to implement this by looking inside the files. I.e., the way mc has done this for ages.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#415 lore raw
Quoted 4 lines · Horst von Brand

Lee Revell <[email protected]> said:

[...]

Quoted 8 lines
Quoted 5 lines

FWIW, this is how Windows does it now. As of XP, 'Find files' has an option, enabled by default, to look inside archives. If you tell it to look for a driver in a given directory it will also look inside .cab and .zip files. It's extremely useful, I would imagine someone who uses XP a lot will come to expect this feature.

It is trivial to implement this by looking inside the files. I.e., the way mc has done this for ages.

This requires a separate, MC-specific namespace. The point is to unify the namespace, not fragment it.

If Hans had a comprehensible web page, maybe more people would understand this aspect of his argument. Would you put digressions about chaos theory and King Arthur in a man page?

Lee

#416 lore raw
In reply to Horst von Brand
Quoted 7 lines

Lee Revell <[email protected]> said:

[...]

Quoted 8 lines
Quoted 5 lines

FWIW, this is how Windows does it now. As of XP, 'Find files' has an option, enabled by default, to look inside archives. If you tell it to look for a driver in a given directory it will also look inside .cab and .zip files. It's extremely useful, I would imagine someone who uses XP a lot will come to expect this feature.

It is trivial to implement this by looking inside the files. I.e., the way mc has done this for ages.

Difference is that you can't do "locate" or "find" or "Search".. You would have to open the files in an archive-supporting application such as mc.

#417 lore raw
Quoted 6 lines · Spam
Quoted 2 lines

It is trivial to implement this by looking inside the files. I.e., the way mc has done this for ages.

Difference is that you can't do "locate" or "find" or "Search".. You would have to open the files in an archive-supporting application such as mc.

And would you rather that logic was running swappable in shared library space or privileged and unswappable in kernel ?

Alan

#418 lore raw
Quoted 13 lines · Spam
Quoted 6 lines
Quoted 2 lines

It is trivial to implement this by looking inside the files. I.e., the way mc has done this for ages.

Difference is that you can't do "locate" or "find" or "Search".. You would have to open the files in an archive-supporting application such as mc.

And would you rather that logic was running swappable in shared library space or privileged and unswappable in kernel ?

I would rather have it as a filesystem/vfs plugin that would allow all my programs to use the features the plugin gives, even if it would mean being totally in kernel space.

Quoted 1 line

Alan

#419 lore raw
Quoted 6 lines · Spam
Quoted 2 lines

And would you rather that logic was running swappable in shared library space or privileged and unswappable in kernel ?

I would rather have it as a filesystem/vfs plugin that would allow all my programs to use the features the plugin gives, even if it would mean being totally in kernel space.

Thats not what I asked.

#420 lore raw
Quoted 12 lines · Spam
Quoted 6 lines
Quoted 2 lines

And would you rather that logic was running swappable in shared library space or privileged and unswappable in kernel ?

I would rather have it as a filesystem/vfs plugin that would allow all my programs to use the features the plugin gives, even if it would mean being totally in kernel space.

Thats not what I asked.

I failed to understand your question then. :(

~S

#421 lore raw
In reply to Alan Cox

Hi!

Quoted 9 lines
Quoted 6 lines
Quoted 2 lines

It is trivial to implement this by looking inside the files. I.e., the way mc has done this for ages.

Difference is that you can't do "locate" or "find" or "Search".. You would have to open the files in an archive-supporting application such as mc.

And would you rather that logic was running swappable in shared library space or privileged and unswappable in kernel ?

Well, having it in kernel mean that cache gets actually shared between different processes -- and that's usefull. With coda and similar tools you can put most of the logic into userspace (kernel impact is 20 lines). You *will* want it to run priviledged, because otherwise caches are useless.
Pavel

#422 lore raw
In reply to Spam

Hi!

Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

FWIW, this is how Windows does it now. As of XP, 'Find files' has an option, enabled by default, to look inside archives. If you tell it to look for a driver in a given directory it will also look inside .cab and .zip files. It's extremely useful, I would imagine someone who uses XP a lot will come to expect this feature.

Quoted 6 lines
Quoted 2 lines

It is trivial to implement this by looking inside the files. I.e., the way mc has done this for ages.

Difference is that you can't do "locate" or "find" or "Search".. You would have to open the files in an archive-supporting application such as mc.

You really need archive support in find. At the very least you need option "enter archives" vs. "do not enter archives". Entering archives automagically is seriously wrong.
Pavel

#423 lore raw
Quoted 11 lines · Pavel Machek

Hi!

Quoted 9 lines
Quoted 8 lines
Quoted 5 lines

FWIW, this is how Windows does it now. As of XP, 'Find files' has an option, enabled by default, to look inside archives. If you tell it to look for a driver in a given directory it will also look inside .cab and .zip files. It's extremely useful, I would imagine someone who uses XP a lot will come to expect this feature.

It is trivial to implement this by looking inside the files. I.e., the way mc has done this for ages.

Quoted 7 lines
Quoted 3 lines

Difference is that you can't do "locate" or "find" or "Search".. You would have to open the files in an archive-supporting application such as mc.

You really need archive support in find. At the very least you need option "enter archives" vs. "do not enter archives". Entering archives automagically is seriously wrong.

But is it efficient to make every application that reads files have to know how to get inside a tar file, just to read its contents? That seems like a massive duplication of effort. Better to have the contents accessible via a separate stream, in the same namespace. Fix it once in the kernel vs. fix it in umpteen apps.

The key point here is that the expressive power of the system is greatly reduced by having a fragmented namespace. Of course there are any number of ways for an app to find out what is in a tar file, but exporting all of that information in a unified namespace is nontrivial and much more interesting.

Lee

#424 lore raw
Quoted 5 lines · Lee Revell

But is it efficient to make every application that reads files have to know how to get inside a tar file, just to read its contents? That seems like a massive duplication of effort. Better to have the contents accessible via a separate stream, in the same namespace. Fix it once in the kernel vs. fix it in umpteen apps.

Thats how you get yourself a non useful OS. Fix it in a library and share it between the apps that care. Like say.. gnome-vfs2

#425 lore raw

Salut,

Quoted 2 lines · Alan Cox

Thats how you get yourself a non useful OS. Fix it in a library and share it between the apps that care. Like say.. gnome-vfs2

Even KIOslave has it. They even support sftp and stuff just by using shared files in /tmp in reality. That's a much saner interface than doing it all in the kernel.

I mean, the kernel is supposed to support access to the disk drives. Who says that it's got to be the uppermost VFS level? You can be perfectly happy to build your own VFS on top of it (or use other's implementations, that is.)

I can already see people moving to FreeBSD if this gets implemented into the kernel...

Tonnerre

#426 lore raw
Quoted 9 lines

Salut,

Quoted 2 lines · Alan Cox

Thats how you get yourself a non useful OS. Fix it in a library and share it between the apps that care. Like say.. gnome-vfs2

Quoted 8 lines

Even KIOslave has it. They even support sftp and stuff just by using shared files in /tmp in reality. That's a much saner interface than doing it all in the kernel.

I mean, the kernel is supposed to support access to the disk drives. Who says that it's got to be the uppermost VFS level? You can be perfectly happy to build your own VFS on top of it (or use other's implementations, that is.)

Yes, we have several VFS versions. KDE and Gnome has their own, mc another, and so on. None is compatible with the other and all data is unavailable if you do not use a program specifically coded to use the particular VFS you stored the data with.

Quoted 2 lines

I can already see people moving to FreeBSD if this gets implemented into the kernel...

Why? No one has to use it. you do not have to load FS/VFS plugins so you can extend the functionality of your filesystem with stuff like compression, encryption, sorting, filtering, etc etc..

Quoted 1 line

Tonnerre

#427 lore raw
In reply to Tonnerre

Hi!

Quoted 11 lines
Quoted 2 lines

Thats how you get yourself a non useful OS. Fix it in a library and share it between the apps that care. Like say.. gnome-vfs2

Even KIOslave has it. They even support sftp and stuff just by using shared files in /tmp in reality. That's a much saner interface than doing it all in the kernel.

I mean, the kernel is supposed to support access to the disk drives. Who says that it's got to be the uppermost VFS level? You can be perfectly happy to build your own VFS on top of it (or use other's implementations, that is.)

You can not reasonably do caching when you are in shared library. And you can not do caching across users at all.

Pavel

#428 lore raw
In reply to Tonnerre

Tonnerre <[email protected]> said:

[...]

Quoted 2 lines

I can already see people moving to FreeBSD if this gets implemented into the kernel...

If it works better for them, great! It is after all a free world. If it turns out via *BSD/MacOS XIX/Windows 2010/... that it is really useful, we can add it to Linux. Linux is _not_ in a feeping creaturitis race, thank you very much.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#429 lore raw

Salut,

Quoted 8 lines · Horst von Brand
Quoted 2 lines · Tonnerre

I can already see people moving to FreeBSD if this gets implemented into the kernel...

If it works better for them, great! It is after all a free world. If it turns out via *BSD/MacOS XIX/Windows 2010/... that it is really useful, we can add it to Linux. Linux is _not_ in a feeping creaturitis race, thank you very much.

Please read again what I just said. I'm against implementing it, because it is creeping featurism. On the contrary, I said that people might actually change to a less complicated/bloated OS (Whereas I would choose NetBSD over FreeBSD, but that's another story).

So you say the wrong things at the wrong people.

Tonnerre

#430 lore raw
Quoted 10 lines · Tonnerre
Quoted 8 lines · Horst von Brand
Quoted 2 lines · Tonnerre

I can already see people moving to FreeBSD if this gets implemented into the kernel...

If it works better for them, great! It is after all a free world. If it turns out via *BSD/MacOS XIX/Windows 2010/... that it is really useful, we can add it to Linux. Linux is _not_ in a feeping creaturitis race, thank you very much.

Quoted 4 lines

Please read again what I just said. I'm against implementing it, because it is creeping featurism. On the contrary, I said that people might actually change to a less complicated/bloated OS (Whereas I would choose NetBSD over FreeBSD, but that's another story).

Sorry, understood exactly the wrong way around. Fingers faster than the brain, again.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#431 lore raw
In reply to Lee Revell
Quoted 3 lines · Lee Revell

Better to have the contents accessible via a separate stream, in the same namespace. Fix it once in the kernel vs. fix it in umpteen apps.

This is ridiculous. We have shared libraries, 99% of applications manage to use libc for example --- this isn't that different.

--cw

#432 lore raw
Quoted 9 lines · Lee Revell
Quoted 4 lines

Better to have the contents accessible via a separate stream, in the same namespace. Fix it once in the kernel vs. fix it in umpteen apps.

This is ridiculous. We have shared libraries, 99% of applications manage to use libc for example --- this isn't that different.

Yes if that was the case then programs wouldn't brake if libc and its API was fixed. Sadly, I think it isn't.

~S

Quoted 1 line

--cw

#433 lore raw
In reply to Lee Revell

Hi!

Quoted 13 lines
Quoted 13 lines
Quoted 12 lines
Quoted 8 lines
Quoted 5 lines

FWIW, this is how Windows does it now. As of XP, 'Find files' has an option, enabled by default, to look inside archives. If you tell it to look for a driver in a given directory it will also look inside .cab and .zip files. It's extremely useful, I would imagine someone who uses XP a lot will come to expect this feature.

It is trivial to implement this by looking inside the files. I.e., the way mc has done this for ages.

Difference is that you can't do "locate" or "find" or "Search".. You would have to open the files in an archive-supporting application such as mc.

Quoted 6 lines
Quoted 3 lines

You really need archive support in find. At the very least you need option "enter archives" vs. "do not enter archives". Entering archives automagically is seriously wrong.

But is it efficient to make every application that reads files have to know how to get inside a tar file, just to read its contents? That

Application does not have to know how to handle tar/zip/etc, but it has to make distinction between "enter archives" and "do not enter archives". See uservfs.sf.net.
Pavel

#434 lore raw
Quoted 19 lines · Pavel Machek

Hi!

Quoted 17 lines
Quoted 16 lines
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines

FWIW, this is how Windows does it now. As of XP, 'Find files' has an option, enabled by default, to look inside archives. If you tell it to look for a driver in a given directory it will also look inside .cab and .zip files. It's extremely useful, I would imagine someone who uses XP a lot will come to expect this feature.

Quoted 6 lines
Quoted 2 lines

It is trivial to implement this by looking inside the files. I.e., the way mc has done this for ages.

Difference is that you can't do "locate" or "find" or "Search".. You would have to open the files in an archive-supporting application such as mc.

You really need archive support in find. At the very least you need option "enter archives" vs. "do not enter archives". Entering archives automagically is seriously wrong.

Quoted 6 lines
Quoted 2 lines

But is it efficient to make every application that reads files have to know how to get inside a tar file, just to read its contents? That

Application does not have to know how to handle tar/zip/etc, but it has to make distinction between "enter archives" and "do not enter archives". See uservfs.sf.net.

But how do you cache the information you had to look in the archive for in a way that other apps can use it? How do you synchronize access to the cache and maintain cache coherency in userspace?

Lee

#435 lore raw
Quoted 2 lines · Lee Revell

But how do you cache the information you had to look in the archive for in a way that other apps can use it?

~/.object-cache/ or whatever

Quoted 2 lines

How do you synchronize access to the cache and maintain cache coherency in userspace?

coherency doesn't exist for normal files for the most part anyhow

#436 lore raw
Quoted 7 lines · Chris Wedgwood
Quoted 2 lines · Lee Revell

But how do you cache the information you had to look in the archive for in a way that other apps can use it?

~/.object-cache/ or whatever

How are permissions handled? If root lists the contents of a tar file that is world readable, then joeuser comes along and does the same, can joeuser sees the cached listing? Would it depend on root's umask?

Lee

#437 lore raw
Quoted 3 lines · Lee Revell

How are permissions handled? If root lists the contents of a tar file that is world readable, then joeuser comes along and does the same, can joeuser sees the cached listing?

everyone has their own cache i guess, works well enough elsewhere

also, am i the only person scared by the code complexity when it comes to root and setuid code here? sounds like a disaster waiting to happen

#438 lore raw
In reply to Lee Revell

Hi!

Quoted 7 lines
Quoted 3 lines

Application does not have to know how to handle tar/zip/etc, but it has to make distinction between "enter archives" and "do not enter archives". See uservfs.sf.net.

But how do you cache the information you had to look in the archive for in a way that other apps can use it? How do you synchronize access to the cache and maintain cache coherency in userspace?

Uservfs.sf.net.

Unlike alan, I do not think that "do it all in library" is good idea. I put it in the userspace as "codafs" server, and let applications see it as a regular filesystem.
Pavel

#439 lore raw
Quoted 5 lines · Pavel Machek

Uservfs.sf.net.

Unlike alan, I do not think that "do it all in library" is good idea. I put it in the userspace as "codafs" server, and let applications see it as a regular filesystem.

That works for me too, providing someone has fixed all the user mode fs deadlocks with paging

#440 lore raw
Quoted 9 lines · Alan Cox
Quoted 5 lines · Pavel Machek

Uservfs.sf.net.

Unlike alan, I do not think that "do it all in library" is good idea. I put it in the userspace as "codafs" server, and let applications see it as a regular filesystem.

That works for me too, providing someone has fixed all the user mode fs deadlocks with paging

Aren't the deadlock scenarios only applicable for read/write mounted filesystems ?

Signature · 3 lines
// Gianni Tedesco (gianni at scaramanga dot co dot uk)
lynx --source www.scaramanga.co.uk/scaramanga.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D
#441 lore raw

Userspace framework (was: Re: silent semantic changes with reiser4)

Il Fri, 03 Sep 2004 04:22:48 +0100, Gianni Tedesco <[email protected]> ha scritto:

Quoted 8 lines · Alan Cox
Quoted 7 lines · Pavel Machek
Quoted 5 lines

Uservfs.sf.net.

Unlike alan, I do not think that "do it all in library" is good idea. I put it in the userspace as "codafs" server, and let applications see it as a regular filesystem.

Quoted 6 lines
Quoted 2 lines

That works for me too, providing someone has fixed all the user mode fs deadlocks with paging

Aren't the deadlock scenarios only applicable for read/write mounted filesystems ?

AFAIK deadlock arises when kernel manages buffers:
it has to free a buffer ==> choose a dirty one ==> if cleaning requires to make a call to network server and this last is waiting for a buffer (cleaning accomplished) ==> ==> deadlock.

I think buffers are involved even in reading file system access. I think also that NFS is threatened by deadlock, not CodaFS.

The problem with CodaFS is that it doesn't support random file access:
you have to download a whole file in coda cache directory even to read its first byte.

I'm writing my laurea thesis about userspace file system, and two Interesting papers i found about this, are:

http://a5.repetae.net/~frederik/avfs/frederik_surf_paper.pdf by Frederik Eaton and the more generic UFO paper
http://www.usenix.org/publications/library/proceedings/ana97/alexandrov.html

I think that LUFS (Linux User File System) and FUSE (Filesystem in USErspace) kernel modules are solutions to be considered for adding to Linux develop branch. They are similar, but FUSE is still maintained and improved, and it permit to create a custom daemon for each user filesystem (unlinke lufsd daemon which loads a shared library). Communication kernel-user space is done through a special file descriptor, unlike LUFS socket.
The only doubt is that FUSE is tuned for local file system performance, but I don't know if it is possible to develop network file system with this framework.
LUFS is slower but more generic in this sense.

http://sourceforge.net/projects/avf

There is more documentation in the tarball than in the home page.

Ciao
Luca

#442 lore raw

Re: Userspace framework (was: Re: silent semantic changes with reiser4)

Quoted 17 lines · Luca Ferroni

Il Fri, 03 Sep 2004 04:22:48 +0100, Gianni Tedesco <[email protected]> ha scritto:

Quoted 8 lines · Alan Cox
Quoted 7 lines · Pavel Machek
Quoted 5 lines

Uservfs.sf.net.

Unlike alan, I do not think that "do it all in library" is good idea. I put it in the userspace as "codafs" server, and let applications see it as a regular filesystem.

Quoted 6 lines
Quoted 2 lines

That works for me too, providing someone has fixed all the user mode fs deadlocks with paging

Aren't the deadlock scenarios only applicable for read/write mounted filesystems ?

Quoted 6 lines

AFAIK deadlock arises when kernel manages buffers:
it has to free a buffer ==> choose a dirty one ==> if cleaning requires to make a call to
network server and this last is waiting for a buffer (cleaning accomplished) ==>
==> deadlock.

so during page launder, we need to write to the filesystem, but if thats in userspace there is the possibility the page launder could have been caused in order that the filesystem daemon may run. AFAICS this problem only arises when file is being written.

The only deadlock I can think of for read-only filesystems is if the demon inadvertantly accesses one of the files that it is handling. That could be avoided quite simply by preventing the demon from doing that in the kernel.

I'm sure I'm missing something, I'd just like to know what ;)

Signature · 3 lines
// Gianni Tedesco (gianni at scaramanga dot co dot uk)
lynx --source www.scaramanga.co.uk/scaramanga.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D
#443 lore raw
In reply to Alan Cox

Hi!

Quoted 8 lines
Quoted 5 lines

Uservfs.sf.net.

Unlike alan, I do not think that "do it all in library" is good idea. I put it in the userspace as "codafs" server, and let applications see it as a regular filesystem.

That works for me too, providing someone has fixed all the user mode fs deadlocks with paging

Its okay: coda works on whole file granularity, and files are backed by real disk, so there are no deadlocks with paging. We only block open() and similar syscalls, and those are not in paging paths. It should be safe.
Pavel

#444 lore raw
In reply to Lee Revell
Quoted 13 lines · Lee Revell

To: Pavel Machek <[email protected]>
Cc: Spam <[email protected]>, Horst von Brand <[email protected]>,
Jamie Lokier <[email protected]>, David Masover <[email protected]>, Chris Wedgwood <[email protected]>, [email protected], Linus Torvalds <[email protected]>, Christoph Hellwig <[email protected]>, Hans Reiser <[email protected]>, [email protected], linux-kernel <[email protected]>,
Alexander Lyamin aka FLX <[email protected]>,
ReiserFS List <[email protected]>
X-Mailer: Ximian Evolution 1.4.6
Date: Thu, 02 Sep 2004 16:01:17 -0400

On Thu, 2004-09-02 at 15:49, Pavel Machek wrote:

[...]

Quoted 3 lines
Quoted 3 lines

You really need archive support in find. At the very least you need option "enter archives" vs. "do not enter archives". Entering archives automagically is seriously wrong.

I have used find(1) for quite some time now, and have never (or very rarely) missed this.

Quoted 2 lines

But is it efficient to make every application that reads files have to know how to get inside a tar file, just to read its contents?

Totally ridiculous, especially if you factor in .gz, .bz2, .zip, .a, .whatever.new.format.they.come.up.with.tomorrow. But then again, this would presumably reside in a (shared) library, so it isn't so bad...

Quoted 2 lines

That
seems like a massive duplication of effort.

Right. tar, gzip, bunzip, et al are already around.

Quoted 3 lines

Better to have the contents
accessible via a separate stream, in the same namespace. Fix it once in the kernel vs. fix it in umpteen apps.

Dead wrong. It is better to fix it in userspace (via a library, if required; could call random unpacking etc programs at will, even be configured on a user-by-user basis through ~/.wacky-file-handling or environment) than force this junk into the kernel. Kernel code is _always_ resident, extremely security critical, and hard to get right. Besides, not everybody will want to carry this around, and so it will forever stay a "weird Linux kernel configuration only" feature, i.e., useless in practice.

Quoted 5 lines

The key point here is that the expressive power of the system is greatly reduced by having a fragmented namespace. Of course there are any number of ways for an app to find out what is in a tar file, but exporting all of that information in a unified namespace is nontrivial and much more interesting.

I don't see how it is "nontrivial": "tar tf some.tar" is quite enough to find out what is inside, in a customary format.

Placing random junk in the kernel doesn't magically make it fast, right, or useful. Quite a lot of work is required for that, much more than for getting the same in userland.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#445 lore raw
Quoted 6 lines · Horst von Brand
Quoted 3 lines
Quoted 3 lines

You really need archive support in find. At the very least you need option "enter archives" vs. "do not enter archives". Entering archives automagically is seriously wrong.

I have used find(1) for quite some time now, and have never (or very rarely) missed this.

I've occasionally had the need to search all files on my system for the one file which contains a particular phrase -- all I remember is the phrase.

Just doing "grep -R" was a tedious job: at least half an hour.

Sometimes, I want to search all source files on my system for a particular word, for example to search for uses of a particular system call or library function.

That would require something that could search through all the .tar.gz files and .zip files (nested if necessary) as well as plain files. It would take so long -- hours at least, maybe more than a day -- that I've never bothered doing such a thing.

"find "that entered archives really wouldn't help (although sometimes "locate" that entered archives would be nice).

In other words, I'd use that capability if it was magically fast, but as we expect it to be insanely slow (just grepping gigabytes is slow) that makes it not so useful.

However, if we ever see that search engine index thing happen, it would be a most excellent capability if it searched inside archive files too. I would definitely use that. Not often, but occasionally I would.

-- Jamie

#446 lore raw
Quoted 3 lines · Jamie Lokier

However, if we ever see that search engine index thing happen, it would be a most excellent capability if it searched inside archive files too. I would definitely use that. Not often, but occasionally I would.

Thats an indexer decision, the search backend (which is the performance and complexity critical part) doesn't give a damn.

#447 lore raw
Quoted 10 lines · Jamie Lokier
Quoted 3 lines

However, if we ever see that search engine index thing happen, it would be a most excellent capability if it searched inside archive files too. I would definitely use that. Not often, but occasionally I would.

Thats an indexer decision, the search backend (which is the performance and complexity critical part) doesn't give a damn.

I am just talking general now, but it seems to me that there have been many suggestions on user-land solutions like shared librares and so forth just to say there is no need for file streams and plugins. Many of these ideas do exist in one way or another, but none is truly system wide and and as application independent as file streams+plugins would be. Would it not be much less effort to implement these in a good way, than trying to reinvent lots of new stuff in userland - that wouldn't be systemwide anyway?

~S

#448 lore raw
In reply to Jamie Lokier
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Jamie" == Jamie Lokier <[email protected]> writes:

Quoted 6 lines · Jamie> Horst von Brand wrote:
Quoted 6 lines
Quoted 3 lines
Quoted 3 lines

You really need archive support in find. At the very least you need option "enter archives" vs. "do not enter archives". Entering archives automagically is seriously wrong.

I have used find(1) for quite some time now, and have never (or very rarely) missed this.

Jamie> I've occasionally had the need to search all files on my system Jamie> for the one file which contains a particular phrase -- all I Jamie> remember is the phrase.

So use glimpse and run an indexer once a night to pick up changes.

Jamie> Just doing "grep -R" was a tedious job: at least half an hour.

Ugh, esp since you're not caching any results of all that work either, when you need to repeat 10 minutes down the line...

Jamie> Sometimes, I want to search all source files on my system for a Jamie> particular word, for example to search for uses of a particular Jamie> system call or library function.

I think we want to pull this back a bit more and define this better. Instead of 'system' we should be thinking 'namespace' or 'hierarchy'.

Jamie> That would require something that could search through all the Jamie> .tar.gz files and .zip files (nested if necessary) as well as Jamie> plain files. It would take so long -- hours at least, maybe Jamie> more than a day -- that I've never bothered doing such a thing.

Jamie> In other words, I'd use that capability if it was magically Jamie> fast, but as we expect it to be insanely slow (just grepping Jamie> gigabytes is slow) that makes it not so useful.

Jamie> However, if we ever see that search engine index thing happen, Jamie> it would be a most excellent capability if it searched inside Jamie> archive files too. I would definitely use that. Not often, Jamie> but occasionally I would.

Is the overhead of pre-indexing all files and their contents worth it though? And would the use of inotify allow us to efficiently update the index in a fairly quick time?

All the various arguements have been that we need a filesystem which magically indexes files, does queries and does it all atomically and without any thought on the part of the developer/user. It could happen, but then we'd have such a *slow* system in general, just to make 1% of the usage simpler (for some measure of simple) that it would be a waste.

Providing a simple, consistent set of semantics and syntax allows you to build on top of it the layer(s) you need to provide the guarenttees you need for your application.

I don't expect that Oracle has the same needs as does my mail spool, or as an image store. Sure, Oracle could do both of them, but are the overheads of oracle worth the slowdown in the common case?

John

#449 lore raw
In reply to Lee Revell

Salut,

Quoted 7 lines · Lee Revell

FWIW, this is how Windows does it now. As of XP, 'Find files' has an option, enabled by default, to look inside archives. If you tell it to look for a driver in a given directory it will also look inside .cab and .zip files. It's extremely useful, I would imagine someone who uses XP a lot will come to expect this feature.

Of course, no idea how it's implemented, but users like it.

The trick is that their tools are aware of it, and that the library the programs use supports on-the-fly decompression.

Same applies to Spotlight for Tiger.

Tonnerre

#450 lore raw
In reply to Jamie Lokier
Quoted 7 lines · Jamie Lokier

So all I am asking for is a facility to auto-mount with file-as-directory, and the ability for a userspace daemon to be notified of regular file modifications synchronously. Both can be added later, once file-as-directory and moveable mounts are established. (fcntl leases and dnotify _almost_ provide this, but they don't. Looks like incoherent hacks keep getting added all over the place for Samba... :).

How will synchronous file modification notifications be handled?

Programs listening for the notification will want to see the changes when they read the file but the reason this is synchronous is to prevent other applications seeing the changes before they are reflected in these indexing programs etc.. right?

Signature · 3 lines
// Gianni Tedesco (gianni at scaramanga dot co dot uk)
lynx --source www.scaramanga.co.uk/scaramanga.asc | gpg --import
8646BE7D: 6D9F 2287 870E A2C9 8F60 3A3C 91B5 7669 8646 BE7D
#451 lore raw
In reply to David Masover

Linus, you are looking at this like a lieutenant instead of an HQ staffer, which is unusual of you.

You are saying, 1-2% simpler and better, no biggie, why work so hard to get it?

And we are saying, 1-2% simpler and better, times thousands of applications, wow! That's a lot!

Yes, changing cat to use openat() is no big deal. 1-2% additional coding cost for cat, who cares? But if you add 1-2% coding cost to every application which might access an attribute/stream/whatever, well, that totals more than the effort of authoring emacs. Can you see that?

Namespace simplifications and empowerments are force multipliers. They don't add to Linux like writing a new app adds to Linux, they add to Linux like adding percentage improvements to every app in Linux adds to Linux. HQ staffer types know that if you collect enough itty bitty little force multipliers, you win the war. Whether the troops have to spend five minutes a day polishing their shoes because their type of shoe needs polishing, that matters more than losing a tank battalion, when you are a major power. Linux is a major power.

This is a software engineering issue. We are discussing improvements that because they are diffused throughout the OS in their impact, seem like no big deal. But they are a big deal. One of the major determinants of an organization's efficiency is whether the management can recognize widely diffused inefficiencies as well as it can recognize focused inefficiencies of the same magnitude.

David is so very right about the usability issues. Usability is all about trivia. Usability battles feature no dragons, they feature armies of spiders. If you add one step to what a user needs to do to get to the data with an app, that matters a lot. That's doubling the spider army. Making a 2 step process to access data into a one step process to access the data halves the time cost of accessing the data. Lifestyle efficiency is mostly about reducing the cost of trivia, because trivia is most of what we do. Most people have as much interest in reading the man page for tar as you have in learning how to turn the hand crank to start your car. Just the look of "tar -xf" turns them away. Crypto-Geek gobbledy-gook is what it is. Let's value their time, there are a lot of them.

Hans

Quoted 14 lines · David Masover

It's not about the kernel, it's about the interface. And see my other mail:
cat foo.zip/README
less foo.zip/contents/bar.c
is a lot easier than
lynx http://google.com/search?q=zip
emerge zip
man zip
unzip foo.zip
cat bar.c
which already assumes quite a lot of expertise.

#452 lore raw
Quoted 6 lines · Hans Reiser

You are saying, 1-2% simpler and better, no biggie, why work so hard to get it?

And we are saying, 1-2% simpler and better, times thousands of applications, wow! That's a lot!

But would thousands care? Seriously?

For example, you could make just _one_ program support "openat()", and you'd get most of the advantages, with no possibility of _breaking_ any of thousands of applications..

I know, you've ignored the "runat" program (which is just a wrapper around the openat() system call), but it _has_ been mentioned several times in this thread. Yes, you'd type a bit more to do

runat file ls -l

instead of

ls -l file/

but at least the openat/runat approach also works for directories, which does actually make it a lot more _generic_ than the "show in the regular filespace" approach. No special cases.

So your comparison isn't valid, because you're ignoring the people who shout "runat" at you. You've also not ever actually answered about the problems about directories with attributes.

Linus

#453 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Linus" == Linus Torvalds <[email protected]> writes:

[...]

Linus> I know, you've ignored the "runat" program (which is just a Linus> wrapper around the openat() system call), but it _has_ been Linus> mentioned several times in this thread. Yes, you'd type a bit Linus> more to do

Linus> runat file ls -l

Linus> instead of

Linus> ls -l file/

And how do I get GIMP (or insert your favourite image editing program here) to edit xterm/icon.png without having to open a new instance of GIMP? runat isn't going to help here. You'll have to modify every editing program if you want that type of functionality.

Linus> but at least the openat/runat approach also works for Linus> directories, which does actually make it a lot more _generic_ Linus> than the "show in the regular filespace" approach. No special Linus> cases.

Linus> So your comparison isn't valid, because you're ignoring the Linus> people who shout "runat" at you. You've also not ever actually Linus> answered about the problems about directories with attributes.

All attributes should be stuffed in a specially-named subdirectory, like Reiser4 puts all metadata in a metas subdirectory. That's one solution. The only trick here is picking a suitable name for the subdirectory that is both user-friendly and highly unlikely to generate clashes.

Both of these I've mentioned before in this thread, and nobody has brought up any objections yet.

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#454 lore raw
In reply to Linus Torvalds
Quoted 17 lines · Linus Torvalds
Quoted 5 lines · Hans Reiser

You are saying, 1-2% simpler and better, no biggie, why work so hard to get it?

And we are saying, 1-2% simpler and better, times thousands of applications, wow! That's a lot!

But would thousands care? Seriously?

For example, you could make just _one_ program support "openat()", and you'd get most of the advantages, with no possibility of _breaking_ any of thousands of applications..

I know, you've ignored the "runat" program (which is just a wrapper around
the openat() system call), but it _has_ been mentioned several times in this thread. Yes, you'd type a bit more to do

Quoted 9 lines

runat file ls -l

instead of

ls -l file/

but at least the openat/runat approach also works for directories, which does actually make it a lot more _generic_ than the "show in the regular filespace" approach. No special cases.

So you are saying I can do runat emacs my-compound-document/stream-name? and in place of CTRL-X CTRL-F to open a file in emacs I type what to open an attribute?

I think it does not work Linus....

Quoted 5 lines

So your comparison isn't valid, because you're ignoring the people who shout "runat" at you. You've also not ever actually answered about the problems about directories with attributes.

Which problem specifically? I will list them and hope I don't miss what you want addressed. I won't claim to be sure I have the right answers for all of these (because VFS has not been my interest in the past), but these look feasible to me.;-)

Cycle detection:

We should either 1) make hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks (best short term answer), or 2) ask Alexander Smith to help us with applying his cycle detection algorithm and gain the benefit of being able to hard link to directories (if it works well, best long term answer).

Ambiguity:

Linus said:

- how do you handle the ambiguity of

	//attr/usr/bin/emacs/icon

(is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the "emacs/icon" attribute on "/usr/bin").

If you want to avoid crowding the namespace with some piece of meta information that you want to be applicable to both directories and the files in them, and also widely used enough that you care about crowding the namespace, and unambiguous as to what level in the hierarchy it metas, then you put it into the metas subdirectory. There is no principled difference between a file that is a child of a directory and an attribute. The ambiguity is inherent in hiearchical semantics I think, and whether or not you have attributes you have this type of ambiguity.

Thus, you have either:

/usr/bin/metas/emacs/icon

or /usr/bin/emacs/metas/icon

and these are easily distinguished and unambiguous. This makes metas a reserved keyword, and there was a long discussion about whether to use "metas" or "..metas" as the reserved subdirectory name which I can summarize for you on request. There is also "pseudos", which contains files implementing special methods of special plugins. Feel free to invent more reserved subdirectories as needed, or to argue for fewer.

Seeing two trees, one with attrs and one without:

Linus said:

If you open a filename in some "secondary" tree (be it /proc or //attr or whatever) based on the filename in the primary one, you have two issues that you need to work out:

- how do you handle a name change in the primary tree at the same time as lookup
- how do you handle the ambiguity of

	//attr/usr/bin/emacs/icon

(is that the "icon" attribute on "/usr/bin/emacs", or is it perhaps the "emacs/icon" attribute on "/usr/bin").

The ambiguity can be handled by saying that attributes only have one component (ie only the _last_ component of a lookup is the attribute name). But the race between primary tree and secondary tree cannot be handled in a normal name-space.

Are you sure that you have to have two trees in order to see two trees? That is, can you not refuse to follow or see a link depending on whether some flag is set and what the type of the link is?

Locking:

Linus said:

let's say that you have filename "a" hard-linked to filename "b", and you have a directory structure of streams under there. So you have

	a/file1
	a/dir1/file2
	a/dir2/file3

and (through the hard-link with "b") you have aliases of all these same names available as "b/file1", "b/dir1/file2" etc).

Now, imagine that you have two processes doing

mv a/dir1 a/dir2/newdir

and

mv b/dir2 b/dir1/newdir

at the same time. Both of them MUST NOT SUCCEED, for pretty obvious reasons (you'd have moved two directories within each other, and now neither would be accessible any more).

How do you handle locking for this situation?

(Here is where my knowledge of VFS is shakiest.) Making "hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks" seems to solve this. If we want to use the Alexander Smith solution to cycle detection, and allow hard links to directories, then we must make sure that for the hard linked entity, there is a single lock for it somewhere (e.g. the directory inode) that can be taken. If you create two dentries for the same object, then flag the dentries as needing to lock a lock in their shared inode and not in the dentry. If you don't want to read in the inode from disk to do that, then create a pointer to a lock shared by them both.

Linus, is this a start on solving some of the issues?

#455 lore raw

Hans Reiser wrote on Wed, 01 Sep 2004 00:22:49 -0700:

Quoted 6 lines

Making "hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks" seems to solve this. If we want to use the Alexander Smith solution to cycle detection, and allow hard links to directories, then we must make sure that for the hard linked entity, there is a single lock for it somewhere (e.g. the directory inode) that can be taken.

Oddly enough that's what I did in my RAM file system. No matter how the file/directory node is found (the path can lead through different parents), it has a unique ID and a corresponding single lock for it.

Rename/delete goes and locks all the nodes that might change parents and/or children, then it does its operations (though it has to watch out for deadlocks, which are unavoidable when two rename/deletes both sneak up on the same file by different paths).

- Alex

#456 lore raw
In reply to Hans Reiser

I'm probably not the first to suggest this idea, and it's probably not a very good idea, but here's my idea anyhow:

You have a file "/usr/bin/emacs"
with a metadata property in the overlaid namespace "/usr/bin/emacs/[[..]metas/]icon"

According to some, this could cause some confusion. Howabout instead:

You have a file "/usr/bin/emacs"
with a metadata property in a slightly separated namespace "/metas/usr/bin/emacs/icon"

This has the advantage of still having the metadata in the filesystem namespace but without the confusion of having files-as-directories, ambiguity of filename, backup issues, etc. This is the reverse of having the namespaces overlaid with a "/nometas" view which is separate.

Furthermore, you can split things further like this:

You have a file "/usr/bin/emacs"
with an automatically-generated metadata property that you don't want to back up in "/autometas/usr/bin/emacs/modification_date" and a manually generated metadata property that you MAY want to backup in "/staticmetas/usr/bin/emacs/icon".

This is inelegant, I know. But if we do this, we can add the extra features of reiser4 without confusing existing apps or having to modify them to support the new functionality.

Furthermore, you can easily hide the extra features by not mounting the meta top-level directories (assuming they're mounted like separate filesystems, rather than just magically appearing there, which is okay too).

#457 lore raw
Quoted 10 lines · Timothy Miller

I'm probably not the first to suggest this idea, and it's probably not a very good idea, but here's my idea anyhow:

You have a file "/usr/bin/emacs"
with a metadata property in the overlaid namespace "/usr/bin/emacs/[[..]metas/]icon"

According to some, this could cause some confusion. Howabout instead:

You have a file "/usr/bin/emacs"
with a metadata property in a slightly separated namespace "/metas/usr/bin/emacs/icon"

And the problem with such a "solution" is
mv /usr/bin/emacs /usr/bin/old-emacs
Do this with an ordinary fs, and the /metas/usr/bin/emacs/icon won't move with it. Now metas might might not be an ordinary fs, so perhaps the move happens automatically there, but if so it will be unexpected.

Quoted 11 lines

This has the advantage of still having the metadata in the filesystem namespace but without the confusion of having files-as-directories, ambiguity of filename, backup issues, etc. This is the reverse of having the namespaces overlaid with a "/nometas" view which is separate.

Furthermore, you can split things further like this:

You have a file "/usr/bin/emacs"
with an automatically-generated metadata property that you don't want to back up in "/autometas/usr/bin/emacs/modification_date" and a manually generated metadata property that you MAY want to backup in "/staticmetas/usr/bin/emacs/icon".

This is inelegant, I know. But if we do this, we can add the extra features of reiser4 without confusing existing apps or having to modify them to support the new functionality.

Furthermore, you can easily hide the extra features by not mounting the meta top-level directories (assuming they're mounted like separate filesystems, rather than just magically appearing there, which is okay too).

Having to go up to root and then down a similar but different path to reach a file's metadata seems very counterintuitive to me. And you have to update all tools to do this automatically, or it'll be hopeless to actually use.

Helge Hafting

#458 lore raw
In reply to Linus Torvalds
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Linus" == Linus Torvalds <[email protected]> writes:

Linus> For example, you could make just _one_ program support Linus> "openat()", and you'd get most of the advantages, with no Linus> possibility of _breaking_ any of thousands of applications..

I'm starting to think that even if we had openat, it wouldn't solve much. Maybe I'm just not imaginative enough, but I still see problems. Let's pretend, just for the sake of argument, that every program all of a sudden magically was modified to support openat, which is a big assumption. But we'll make that assumption so that we can ignore _that_ problem for a second.

The question still remains, how do I tell the thing to open a stream? If cat knows about openat, how do I tell it to cat foo.txt, stream bar? I obviously can't do it using a normal filename, because if I could do that, we could just forget about the openat business. The only way I can think of is to give cat a flag that tells it what stream to open: "cat --stream bar foo.txt". Ugly. It also makes it impossible to do redirection, because you can't pass flags to "<".

And before someone yells "runat" at me (which, btw, won't work with redirection either), consider larger programs like GIMP or emacs. Let's look at GIMP first. I've already got an instance of GIMP open, and I don't want to waste memory with another instance. How do I tell it to open the icon.png stream of the file xterm? When I go to File | Open, up pops a dialog that only shows real files. Trying to design a dialog box that will show real files and substreams, and not confuse the users is not going to be fun. Please don't ask the GTK/GNOME developers to design another file selection dialog. There were enough people whining the last round. If they need to add more complexity, we'll have even more whining.

Now let's look at emacs. I want to open up the description.txt stream of the file xterm. So I do "C-x C-f", and it prompts me on the status bar for the file name. What do I type? Well, description.txt doesn't live in the normal namespace, so the only way around that is for emacs to ask for the filename, and then ask for the stream name. That's extra-annoying when I usually just want to open just the normal stream, and that second prompt just makes more work for me.

Linus> I know, you've ignored the "runat" program (which is just a Linus> wrapper around the openat() system call), but it _has_ been Linus> mentioned several times in this thread. Yes, you'd type a bit Linus> more to do

Linus> runat file ls -l

Linus> instead of

Linus> ls -l file/

Might not be just a "bit more" typing, because you lose handy things like tab completion. Say I want to "cat" file foo.txt, stream bar. Or was that baz? Instead of being able to type
cat foo/ba<tab>
and have the filename magically come up, I need to
runat foo ls ba*
runat foo cat baz
(which also requires more thinking, since I need to think about how to formulate the query, read the output, and write the new command, and so is much much slower than being able to mindlessly hit the <tab> key.)

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#459 lore raw
In reply to Hans Reiser
Quoted 2 lines · Hans Reiser

Yes, changing cat to use openat() is no big deal. 1-2% additional coding cost for cat, who cares?

Large cost on its own. The FSF would almost certainly reject such a change until someone had written portable emulation of the feature for every other platform which would mean vendor patches which would essentially mean it wouldn't happen.

I know I rarely agree with Hans but I think he's right on this one, it has to work with the existing open() API.

Alan

#460 lore raw
In reply to Hans Reiser
Quoted 3 lines · Hans Reiser

learning how to turn the hand crank to start your car. Just the look of "tar -xf" turns them away. Crypto-Geek gobbledy-gook is what it is. Let's value their time, there are a lot of them.

These kinds of people would use something like file-roller in GNOME, and they'd care not a jot whether file-rollers used built-in tar support, and/or openat() or via file-as-dir fs support.

All they care about is that they get a window with icons for their files that they can click on. How it's done does not matter - it only matters to those who must write the code.

Quoted 1 line

Hans

regards,

Signature · 3 lines
Paul Jakma	[email protected]	[email protected]	Key ID: 64A2FF6A
Fortune:
I haven't lost my mind; I know exactly where I left it.
#461 lore raw

Linus, did you get the email in which I (I think....) answered all the problems with files as directories that you and Viro had raised?

You and Viro did not respond to it....

Hans

#462 lore raw
Quoted 4 lines · Hans Reiser

Linus, did you get the email in which I (I think....) answered all the problems with files as directories that you and Viro had raised?

You and Viro did not respond to it....

Message-ID, please. There was a lot of mails from you in this thread and I probably have missed one that did address *any* problems. If such mail had ever existed, that is.

#463 lore raw
Quoted 10 lines · [email protected] writes:
Quoted 5 lines · Hans Reiser

Linus, did you get the email in which I (I think....) answered all the problems with files as directories that you and Viro had raised?

You and Viro did not respond to it....

Message-ID, please. There was a lot of mails from you in this thread and I probably have missed one that did address *any* problems. If such mail had ever existed, that is.

Perhaps this is one? Message-ID: <[email protected]>

#464 lore raw
Quoted 1 line · Sriram Karra

Perhaps this is one? Message-ID: <[email protected]>

OK...

One note before replying: current code deadlocks even if you make ->link() *ALWAYS* return an error. It doesn't get to calling the method. No amount of "disallow hard links to <something>" is going to help here, obviously.

<quote>
Cycle detection:

We should either 1) make hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks (best short term answer), or 2) ask Alexander Smith to help us with applying his cycle detection algorithm and gain the benefit of being able to hard link to directories (if it works well, best long term answer).
</quote>

... which doesn't address the problem at all. The question is what to do with seeing directory "aspect..." in more than one place when we have many links to file in question. So much for (1). And (2) is not feasible with on-disk fs both due to memory, CPU and IO costs _and_ due to exclusion from hell you'll need to make it safe.

Re: ambiguity - lots and lots of handwaving on both sides. FWIW, I agree with Hans in one (and only one) respect here - openat() as a primary API (and not a convenient libc function) is an atrocity. Simply because it doesn't address operations beyond open (unlinkat(2), anyone?).

However, I still haven't seen any strong arguments for need of this "metas" stuff _or_ the need to export mode/ownership as files, both for regular files and for directories. Aside of "we can do that" [if we solve the locking issues] and "xattrs are atrocious" [yes, they are; it doesn't make alternative mechanism any better] there was nothing that even pretended to be a technical reason.

Note that we also have fun issues with device nodes (Linus' "show partitions" vs. "show metadata from hosting filesystem"), which makes it even more dubious. We also have symlinks to deal with (do they have attributes? where should that be exported?).

Reserved names have one more problem: to be useful, they'd have to be hardcoded into applications. And that will create hell with use of such applications on existing filesystems. Again, no feasible scheme to deal with that in userland code had been proposed so far, AFAICS.

Locking: see above - links to regular files would create directories seen in many places. With all related issues...

#465 lore raw
In reply to David Masover

Salut,

Quoted 10 lines · David Masover

It's not about the kernel, it's about the interface. And see my other mail:
cat foo.zip/README
less foo.zip/contents/bar.c
is a lot easier than
lynx http://google.com/search?q=zip
emerge zip
man zip
unzip foo.zip
cat bar.c
which already assumes quite a lot of expertise.

Most distros ship Ziv's packer by default.

telnet dict.org 2628
DEFINE * "creeping featurism"
QUIT

Tonnerre

#466 lore raw
In reply to Pavel Machek

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Pavel Machek wrote:

| Hi!
|
|
|>| uservfs does
|>|
|>| cd foo.deb#uar
|>cd foo.deb/ar
|>| vs.
|>| cd foo.deb#udeb
|>cd foo.deb/deb
|>
|>and why would you want that, instead of just:
|>cd foo.deb	# for the ar
|>dpkg -i foo.deb	# for the deb
|
|
| Because I want to see contents of that .deb package, nicely parsed?
|
|
|>Just want to extract the tar file?  Maybe something like
|>cat foo.tgz/gunzip
|>In which case (of course) foo.tgz/gunzip has exactly the same directory
|>contents as foo.tgz
|
|
| Yes, that would work.
|
|
|>In fact, for just about any syntax anyone could suggest, I can't really
|>see why you can't just replace all weird symbols with a slash and a
|>symbol.  Instead of
|>	foo.tgz#utar
|>you have
|>	foo.tgz/#/utar
|>Only difference is, some things which used to require special tools can
|>now be serviced by less than what's in busybox.
|
|
| That would work, too. I do not get your comment about busybox.

Busybox is a bare minimum of unix tools, yet still includes things like tar/bz2. With such a system, you can use the most basic (sub-GNU) versions of tools like cp, cat, sh, ls, and so on, to do much more complex operations. What's more, you can see everything you can do to a file just by looking at it. I can certainly imagine doing
cat foo.tgz/metas/README
rather than a google search for tgz, followed by a "man tar" and "man gzip". Assuming I know nothing about tar/gzip files, it's a lot easier as a new user to have one common interface for everything -- including docs. Fewer commands to memorize, faster to learn new stuff. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTUha3gHNmZLgCUhAQKRJw/7BUEm41gKZGoVj7gOkcUVe/SRdTs9WtTL jkjCX8Zzr/1nnAKTHLqNJEiAJ68Ndf+dRhDfL5y6dDHhOy/Dx+38SzsWz+dQnMJ0 Fiq5fFDO/DTd0UPecXYPhBrtVeCdfjyyENhgRgJud6D8F39qvLWrZFUsIkjAA3lj nzZ/48jRPDWT/mF6JJmv0RfVvNtMWh9HUcW/i0+n4WZU1yFp6jyiF5ExCOP84z3x yXQ4Q9hnjgt52MDJTrGsCDvfCSQscQilxvXbl0iaozquIhG9Jz1XNYpij2XUygFi RDc1LA1k74VvbWahlkdfbTI+3P4qeMYGBblmfwXVDRwW0i4NH7X0/6xcnr1QjaK/ f/7EHBz0wnkIG4OpcGWzu9ik4WOR0WXapKrd938XC/mG+WzIluJlQsxG1A2HoRqg R66E/mVMVvHJWnJjrW6s10WIemPiKnuXgeIVPifsw81DLXlZ3wxjlZU3iuo7xked O6wNDFQBWQR8kv+EMDIt3gtbUdB+5KpxGR+Yli0AjxEdpdk3oZTfe5XQiNdJhcl5 xrcomVKXGvR96IlNUwHy1EsQk5LySMl101vJIS4z6i7lAIM5CQ1d6eOZrPp/DBOP Rw0KoQaiPd3SKNMgoyWSKQL6h85i1PRUN62WH0T2NRU1fZ3EazpiRZr+fuaFqFT9 Wk2lkDvm5yw=
=ShWo
-----END PGP SIGNATURE-----

#467 lore raw
In reply to Chris Wedgwood

Hi!

Quoted 5 lines
Quoted 2 lines

One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way.

Arguably this belongs in userspace --- and people have put it there.

uservfs.sf.net, using existing coda hooks. There are 2+ very similar projects.

Advantage is we can do ftpfs, too. Having ftp done by

Quoted 1 line

/ftp

cd /ftp/sunsite

would be too ugly for reiser4 plugins... Or not?

Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         
#469 lore raw
In reply to [email protected]

Hello,

This thread is very very very very big. I think lots of people spend hours talking here. Isn't it time for a summary of all this please ?

(for people like me who are quite busy and try to follow... but also for the health of the debate)

Thank!

Mikael.

#470 lore raw
In reply to [email protected]
Quoted 11 lines · [email protected] wrote:
Quoted 8 lines · Linus Torvalds
Quoted 5 lines · Christoph Hellwig

For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.

Quoted 9 lines
Quoted 5 lines

Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.

It's the UNIX way.

Not if you allow link(2) on them. And not if you design and market your stuff as a general-purpose backdoor into kernel.

What backdoor? Please spell it out. Plugins are not dynamically loadable.

Quoted 7 lines

Note how *EVERY* *DAMN*
*OPERATION* is made possible to override by "plugins". Which is the reason for deadlocks in question, BTW.

Don't fool yourself - that's what Hans is selling. Target market: ISV.

Hunh? No, target market is hackers who like to spend a weekend dreaming up funky new kinds of files. One guy (Jason Holt, clever guy) came up with an idea for write only files for which even root cannot read the parts of the file written prior to the time root was achieved, because the encryption key is changed in a forward computable only direction with every write, and the start key is kept on another computer. Lots of folks will have plugins I would never dream of. Think of photoshop plugins, and what plugins did for photoshop. Same thing will happen to reiser4 next year (or earlier).

Quoted 11 lines

Marketed product: a set of hooks, the wider the better, no matter how little sense it makes. The reason for doing that outside of core kernel:
bypassing any review and being able to control the product being sold (see above).

Shame that it got an actual filesystem mixed in with the marketing plans and general insanity...

I am one of those free software hippy-child anarchists who thinks that random people should come up with ideas and contribute them. You understand me well.;-)

#471 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Christoph Hellwig

For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.

Quoted 6 lines

Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.

It's the UNIX way.

I thought the UNIX way is "everything's a file", not "everything's a directory".

Quoted 3 lines

Will it potentially break something? Sure. Do we care? Me, I'll take that kind of extension _any_ day over xattrs, that are fundamentally flawed in my opinion and totally useless.

There's always the option that they're both broken.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#472 lore raw
Quoted 5 lines · Matt Mackall
Quoted 2 lines

It's the UNIX way.

I thought the UNIX way is "everything's a file", not "everything's a directory".

It really was. Directories were historically largely just files too, although with the special "lookup" operation.

Historic unix didn't have readdir/rmdir/mkdir/rename or really much _any_ special directory handling. Directories were just files, and you read them like files.

Of course, even in that early unix, "directories" were very much a reality even apart from the fact that they happened to be implemented pretty much like files. Nobody has ever claimed that the UNIX way is "everything is _one_ file", after all ;)

Quoted 5 lines
Quoted 3 lines

Will it potentially break something? Sure. Do we care? Me, I'll take that kind of extension _any_ day over xattrs, that are fundamentally flawed in my opinion and totally useless.

There's always the option that they're both broken.

Yes. Highly likely. However, something like that _does_ end up what a Windows fileserver wants. IOW, even if it's broken, _something_ is likely forced on us by that nasty thing we call "real users". Damn them.

Linus

#473 lore raw
In reply to Linus Torvalds
Quoted 15 lines · Linus Torvalds
Quoted 4 lines · Christoph Hellwig

For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.

Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.

It's the UNIX way.

And yes, the semantics can _easily_ be solved in very unixy ways.

Quoted 8 lines

One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is:

- without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
- with the slash, it won't open _without_ O_DIRECTORY (EISDIR)

Problem solved. Very user-friendly, and very intuitive.

Stupid question: who will use it? And why?

Anyone can write an userspace library, that implements function set_attribute(char *file, char *attribute, char *value), that creates directory ".attr/file" in file's directory and stores attribute there. (and you can get list of attributes from shell too:
ls echo "$filename" |sed 's/\/\([^\/]*\)$/\/\.attr\/\1/' ). There's no need to add extra functionality to kernel and filesystem.

Advantage:
- you don't add bloat to kernel or filesystem
- you don't need to teach tar/cp -a/mc about attributes
- you won't lose attributes after editing file in vim (it creates another file and renames it over original one)

Quoted 5 lines

Will it potentially break something? Sure. Do we care? Me, I'll take that kind of extension _any_ day over xattrs, that are fundamentally flawed in my opinion and totally useless. The argument that applications like "tar" won't understand the file-as-directory thing is _flawed_, since legacy apps won't understand xattrs either.

The only way xattrs are useful is that backup/restore software doesn't have to know about every filesystem with it's specific attributes and every magic ioctl for setting them. Instead it can save/restore filesystem-specific attributes without understanding what do they mean. However there's no need why application should use them. And no application does.

I can't imagine anyone shipping an application with "this app requires reiser4" prerequisite. Why should anyone use it if he can store attributes in ".attr" directory or whereever and make the application work on any OS and any filesystem?

Mikulas

#474 lore raw
Quoted 9 lines · Mikulas Patocka

Stupid question: who will use it? And why?

Anyone can write an userspace library, that implements function set_attribute(char *file, char *attribute, char *value), that creates directory ".attr/file" in file's directory and stores attribute there. (and you can get list of attributes from shell too:
ls echo "$filename" |sed 's/\/\([^\/]*\)$/\/\.attr\/\1/' ). There's no need to add extra functionality to kernel and filesystem.

..and the above is, roughly, what I understand samba etc falls back on.

The problem ends up being that the above isn't in any way safe from people moving files around (oops, where did those attributes go?) nor does it have any consistency guarantees. So it only works well if _one_ application does this, and that application follows all the locking rules.

Is it enough? It may have to be.

Quoted 6 lines

The only way xattrs are useful is that backup/restore software doesn't have to know about every filesystem with it's specific attributes and every magic ioctl for setting them. Instead it can save/restore filesystem-specific attributes without understanding what do they mean. However there's no need why application should use them. And no application does.

If no application does, then why back them up? Why implement them in the first place?

In other words - some apps obviously do want to use the. Sadly.

Linus

#475 lore raw
Quoted 11 lines
Quoted 6 lines

The only way xattrs are useful is that backup/restore software doesn't have to know about every filesystem with it's specific attributes and every magic ioctl for setting them. Instead it can save/restore filesystem-specific attributes without understanding what do they mean. However there's no need why application should use them. And no application does.

If no application does, then why back them up? Why implement them in the first place?

In other words - some apps obviously do want to use the. Sadly.

You can add more functionality to filesystem and use xattrs to control it. For example:
- acls
- compress file
- encrypt file (copy user's password into task_struct and use it to encrypt his files)
- preallocate file in 4MB contignuous chunks, becuase it needs real time multimedia access
- sync/append-only/immutable
etc.
However there's no need why an application should care whether the file is compressed, whether it has acls, or so. And applications don't.

And I think this is the only legitimate use for xattrs. Who else uses them except samba? I don't see how reiser4's hybrids would help.

Mikulas

#476 lore raw
In reply to Linus Torvalds

Streams are quite ugly. However, if you decompose streams into all of the little pieces that are needed to emulate them, the pieces are quite nice.

For instance, inheriting stat data from a common parent is nice, and inheritance is nice, and being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice, and, well, if you decompose all of the features of streams into little features you get a bunch of fun little features much nicer than streams.

Hans

#477 lore raw
Quoted 3 lines · Hans Reiser

being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice

Yes. Being able to cd into filename.tar.gz and filename.iso is also nice, but all of these features should be supported by the VFS generically, not in any specific filesystem, and there should be a hook to invoke the various fun filesystem-independent handlers by name.

-- Jamie

#478 lore raw
Quoted 9 lines · Jamie Lokier
Quoted 3 lines · Hans Reiser

being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice

Yes. Being able to cd into filename.tar.gz and filename.iso is also nice, but all of these features should be supported by the VFS generically, not in any specific filesystem, and there should be a hook to invoke the various fun filesystem-independent handlers by name.

It doesn't belong into the kernel at all. If at all it belongs into a userspace filesystems, but even in that case the magic detection of which one to use is kinda hard. You absoutely don't want to hardcode file formats in the kernel.

#479 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines
Quoted 3 lines

being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice

Yes. Being able to cd into filename.tar.gz and filename.iso is also nice, but all of these features should be supported by the VFS generically, not in any specific filesystem, and there should be a hook to invoke the various fun filesystem-independent handlers by name.

Quoted 4 lines

It doesn't belong into the kernel at all. If at all it belongs into a userspace filesystems, but even in that case the magic detection of which one to use is kinda hard. You absoutely don't want to hardcode file formats in the kernel.

Oh, I agree userspace should be involved.

-- Jamie

#480 lore raw
In reply to Christoph Hellwig
Quoted 13 lines · Jamie Lokier
Quoted 9 lines
Quoted 3 lines · Hans Reiser

being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice

Yes. Being able to cd into filename.tar.gz and filename.iso is also nice, but all of these features should be supported by the VFS generically, not in any specific filesystem, and there should be a hook to invoke the various fun filesystem-independent handlers by name.

Quoted 4 lines

It doesn't belong into the kernel at all. If at all it belongs into a userspace filesystems, but even in that case the magic detection of which one to use is kinda hard. You absoutely don't want to hardcode file formats in the kernel.

Do you mean user-level file system as a VFS handled by user applications, or a intermediate file system layer between any application and the real file system? The latter would be good enough as it would still be transparent to the applications.

~S

#481 lore raw
In reply to Mikulas Patocka
Quoted 7 lines · Mikulas Patocka
Quoted 3 lines · Linus Torvalds

One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is:

Stupid question: who will use it? And why?

I've got a stupid question too. How do you back up these things ?

If your backup program reads them as a file and restores them as a file, you might lose your directory-inside-the-file magic.

If your backup program dives into the file despite stat() saying it's a file and you restore your backup, how are the "file is a file" semantics preserved ?

Obviously this is something that needs to be sorted out at the VFS layer. A filesystem specific backup and restore program isn't desirable, if only because then there'd be no way for Hans's users to switch to reiser5 in 2010 ;)

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#482 lore raw
Quoted 31 lines · Rik van Riel
Quoted 6 lines · Mikulas Patocka

On Wed, 25 Aug 2004, Linus Torvalds wrote:

Quoted 8 lines
Quoted 5 lines

One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is:

Stupid question: who will use it? And why?

I've got a stupid question too. How do you back up these things ?

If your backup program reads them as a file and restores them as a file, you might lose your directory-inside-the-file magic.

If your backup program dives into the file despite stat() saying it's a file and you restore your backup, how are the "file is a file" semantics preserved ?

Quoted 7 lines

Obviously this is something that needs to be sorted out at the VFS layer. A filesystem specific backup and restore program isn't desirable, if only because then there'd be no way for Hans's users to switch to reiser5 in 2010 ;)

Sure, this sort of thing must be sorted out at the VFS layer. And a backup program working on such a filesystem
will need to know that something can be a file, a directory - or both.

So an old "tar" won't get this right as it will assume that an object is either file or directory. The change to get it right won't be that big - just notice that an object is both, then backup the ordinary file contents as usual, before recursing into the directory it also provides and backup stuff there as usual.

The resulting .tar can of course only be unpacked properly on a fs supporting file-as-directory, similiar to how a .tar of a fs with links only will unpack properly on a fs supporting links.

I don't see much problems for userland. Old apps will keep working, as the new features is a superset. Those who care about file-as-directory extras will provide patches for "tar" and friends, after that the extras become useable.

Helge Hafting

#483 lore raw
Quoted 2 lines · Helge

So an old "tar" won't get this right as it will assume that an object is either file or directory.

There are many backup apps, not just one. I've written a few myself, none of which will ever be worthy of notice. The sourceforge Topic.System.Archiving.Backup lists 335 projects at present.

I find the idea that most backup tools and scripts will silently stop working correctly to be pretty scary.

And then there's archiving, installation, distribution, administration, emulation, file system and partition managers, and on and on.

===

I wonder if we can make this "modal" somehow.

The one consistency I see is that apps that want the "enhanced" view need to ask for it, somehow. It is the new views of the data that are being added - let the app announce to the kernel (usually via specialized code in some shared library that the app is using to get the alternate views) that either per-task, or per-file descriptor, it is to see the "enhanced" view, as a side affect of trying to access it.

Old stuff, or even new stuff that is content to work with the "classic" view that a file is a single data stream, and that directories only have pathnames, not data, would by default see that view, and see _all_ the data, presented somehow in that view, perhaps as additional files with magic names.

This still leaves the breakage that such tools don't know, and don't preserve, the magic linkage between such magic files. But that is much less of an issue, in my view. Programs such as backups that are manipulating the files of apps they know nothing about already have to presume that all the files are important in inscrutable ways, and just be careful to preserve or copy or backup all of them.

Yeah - I realize that there will be a few followups denouncing modal architectures. I might even agree with some of them.

If this were easy, it would have been done years ago.

The onus should be on the new stuff to request the enhanced view, rather than on the old dogs to learn new tricks.

Signature · 3 lines
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <[email protected]> 1.650.933.1373
#484 lore raw
Quoted 2 lines · Paul Jackson

I find the idea that most backup tools and scripts will silently stop working correctly to be pretty scary.

Too late. We have xattrs already; many programs don't store them.

-- Jamie

#485 lore raw
Quoted 1 line

Too late. We have xattrs already; many programs don't store them.

If by your "too late" you mean we can stop worrying about any more breakage of file system utilities, because there exists an example in which some were already broken, then you are absolutely wrong.

Just because we caused some breakage doesn't give us license to cause even more.

Signature · 3 lines
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <[email protected]> 1.650.933.1373
#486 lore raw
In reply to Paul Jackson
Quoted 8 lines · Paul Jackson
Quoted 4 lines · Helge

So an old "tar" won't get this right as it will assume that an object is either file or directory.

Quoted 8 lines

There are many backup apps, not just one. I've written a few myself, none of which will ever be worthy of notice. The sourceforge Topic.System.Archiving.Backup lists 335 projects at present.

I find the idea that most backup tools and scripts will silently stop working correctly to be pretty scary.

They won't stop working, they will merely not support the new features. That is only a problem if you actually use those features. If all you have is plain files and plain directories - no problem. Not if files-as-directories are implemented right.

Helge Hafting

#487 lore raw
In reply to Rik van Riel
Quoted 9 lines · Rik van Riel

If your backup program dives into the file despite stat() saying it's a file and you restore your backup, how are the "file is a file" semantics preserved ?

Obviously this is something that needs to be sorted out at the VFS layer.

It needs to be sorted out, whether it is sorted out at the VFS layer is unimportant.

Quoted 6 lines

A filesystem specific backup and restore
program isn't desirable, if only because then there'd be no way for Hans's users to switch to reiser5 in 2010 ;)

It might be that we need a filenameA/metas/backup method for all of our file plugins, which if cat'd gives a set of instructions which if executed are adequate for restoring filenameA.

Hans

#488 lore raw
Quoted 5 lines · Hans Reiser
Quoted 3 lines

Obviously this is something that needs to be sorted out at the VFS layer.

It needs to be sorted out, whether it is sorted out at the VFS layer is unimportant.

So what exactly is wrong with sorting it out at the VFS layer, and why do you _insist_ on sorting it in the reiserfs4 core? I'm missing something, please fill me in on the details.

Sincerely,
Lars Marowsky-Brée <[email protected]>

Signature · 3 lines
High Availability & Clustering	   \\\  /// 
SUSE Labs, Research and Development \honk/ 
SUSE LINUX AG - A Novell company     \\// 
#489 lore raw
In reply to Rik van Riel
Quoted 3 lines · Rik van Riel

If your backup program reads them as a file and restores them as a file, you might lose your directory-inside-the-file magic.

Encode the magic in the names, by stealing a bit of the existing filename space to encode it.

Such works pretty well as part of the magic to map long filenames into DOS 8.3 names on my FAT partitions.

Apps linked with the appropriate Windows library see nice fancy long names.

The rest of the world, including DOS apps and my Unix backup scripts, see the primitive 8.3 names, including one or a few extra files per directory, which are nothing special to them.

So long as these other apps don't presume to know that they can keep some of the files in an apps directory, and drop others, then it works well enough. And no self-respecting general purpose backup program is going to presume such knowledge anyway.

Signature · 3 lines
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <[email protected]> 1.650.933.1373
#490 lore raw
In reply to Linus Torvalds
Quoted 20 lines · Linus Torvalds
Quoted 5 lines · Christoph Hellwig

For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.

Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.

It's the UNIX way.

And yes, the semantics can _easily_ be solved in very unixy ways.

One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is:

- without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
- with the slash, it won't open _without_ O_DIRECTORY (EISDIR)

Quoted 10 lines

Problem solved. Very user-friendly, and very intuitive.

Will it potentially break something? Sure. Do we care? Me, I'll take that kind of extension _any_ day over xattrs, that are fundamentally flawed in my opinion and totally useless. The argument that applications like "tar" won't understand the file-as-directory thing is _flawed_, since legacy apps won't understand xattrs either.

Oh, add a O_NOXATTRS flag to force a path lookup to only use regular directories, the same way we have O_NOFOLLOW and friends. That allows people to see the difference, if they care (ie a file server might decide that it doesn't want to expose things like this).

I think we should require people to care enough to supply an O_NOMETAS flag to see the difference.

Quoted 7 lines

I never liked the xattr stuff. It makes little sense, and is totally useless for 99.9999% of everything. I still don't see the point of it, except for samba. Ugly.

Linus

#491 lore raw
In reply to Linus Torvalds

If I may chime in here...

Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Christoph Hellwig

For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.

Quoted 6 lines

Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.

It's the UNIX way.

This is an issue that directly affects work I am doing in extended cryptfs:

http://www.linuxsymposium.org/2004/view_abstract.php?content_key=55
http://halcrow.us/~mhalcrow/ols2004.pdf
http://halcrow.us/~mhalcrow/ols_cryptfs.sxi

The basic idea is that the cryptographic context for every file is correlated with the individual file via xattr's. A file is a unit of data that should, as it stands, contain all the information requisite for the encrypting filesystem layer to transparently decrypt (and encrypt, when the file is written to). This allows for a key->file granularity, as opposed to a key->block device (dm-crypt) or a key->mount point (CFS) granularity.

My grand vision is to have a policy that determines whether or not the encrypted version of the file or the decrypted version of the file is read, dependent on whether or not the file is leaving the security domain (the storage device under the control of the currently running kernel). For example, if the ``cp'' command is copying a file from a filesystem mounted from /dev/hda1 to a filesystem mounted from /dev/fd0, then the policy would indicate that (unless otherwise noted in the .cryptfsrc file in the root of the filesystem mounted from /dev/fd0, which might also contain the default security context for that filesystem or directory - like whose public keys should be used to encrypt the symmetric key for data) the file is leaving the security domain, and the encrypted contents of the file should be given to cp. Same with mutt reading an email attachment (as opposed to, say, .muttrc, where, more likely than not, the unencrypted version is wanted).

The goal is to enable an ``encrypted by default'' policy, in which files on the storage devices are independent encrypted units that remain encrypted until an application that actually needs to see the decrypted contents opens them. Then the encryption and decryption is done transparently by the fs layer, as long as the user has the right keys. Extended attributes seem like a natural way to store this context.

Once you consider that you can have a crypto context for each file, you can start doing other neat tricks, like keyed hashes over extents within the file, to allow for dynamic integrity verification during the read. If an offset of 1.5 gigabytes into a 2-gigabyte has been tampered with, then that tampering will be caught when that portion of the file is read; you don't have to verify the hash of the entire 2-gigabyte file at the time of the open. Of course, this would very rapidly overrun the available xattr storage size. And so to realistically implement something like this, some new underlying file format is in order.

In any case, the issue of userspace applications supporting extended attributes is key to the viability of this approach. If cp, uuencode, tar, or what not do not preserve the extended attributes, then the crypto context is lost, and the file is unreadable. So the $64,000 question is, just how committed is the community to this whole concept of extended attributes? From this point, should I assume that good xattr support is forthcoming, or should I abandon the idea of using xattr's for this altogether?

One solution I've been kicking around is to make cryptfs GnuPG-compatible. Not only would this eliminate the need to store some of the crypto context in the xattr set, but it would also preserve the crypto context with apps that don't know about xattr's, and it would be possible for users who are not running cryptfs to read the files with gpg. Keyed hashes over extents would be doable if GnuPG allowed for opaque data blobs in the file that gpg would just ignore when decrypting the file (gnupg-dev list had technical issues last time I tried to post these ideas to it - any gpg guys around that can comment on this?).

Quoted 3 lines

I never liked the xattr stuff. It makes little sense, and is totally useless for 99.9999% of everything. I still don't see the point of it, except for samba. Ugly.

If xattr's wind up getting supported by a certain critical mass of applications, then they are somewhat useful for me, although, as currently implemented, are insufficient for what I really need (keyed hashes over extents require too much space).

BTW, early this week I migrated cryptfs over to use David Howell's new keyring, which is working out nicely.

Mike
.___________________________________________________________________.
Michael A. Halcrow
Security Software Engineer, IBM Linux Technology Center
GnuPG Fingerprint: 05B5 08A8 713A 64C1 D35D 2371 2D3C FDDA 3EB6 601D

#493 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

- without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
- with the slash, it won't open _without_ O_DIRECTORY (EISDIR)

Problem solved. Very user-friendly, and very intuitive.

Breaks bash and almost all other file completion 8)

#494 lore raw
In reply to Christoph Hellwig
Quoted 11 lines · Christoph Hellwig
Quoted 4 lines · Hans Reiser

You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles.

Actually in most of the discussion you simply didn't participate. While xattrs might not be the nicest interface they have the advantag of not breaking the SuS assumption of what directories vs files are, and they do not break the Linux O_DIRECTORY semantics that are defined and need to solve real-world races either.

Reiser4 may have a mount option for whoose who like or have to use traditional O_DIRECTORY semantics. There would be no /metas under non-directories, if user wants that.

[ ... ]

Signature · 1 line
Alex.
#495 lore raw
Quoted 3 lines · Alex Zarochentsev

Reiser4 may have a mount option for whoose who like or have to use traditional O_DIRECTORY semantics. There would be no /metas under non-directories, if user wants that.

Again, O_DIRECTORY was added to solve a real-world race, not just for the sake of it. If you really want to add that option I'll research the CAN number for you so you can named it after that - or just call it -o insecure directly.

#496 lore raw
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines

Reiser4 may have a mount option for whoose who like or have to use traditional O_DIRECTORY semantics. There would be no /metas under non-directories, if user wants that.

Again, O_DIRECTORY was added to solve a real-world race, not just for the sake of it. If you really want to add that option I'll research the CAN number for you so you can named it after that - or just call it -o insecure directly.

man open(2) explains that O_DIRECTORY is used by opendir() to prevent blocking when opening pipes and certain devices*, and should only by used by opendir (of course it isn't only used by opendir, as it's a handy optimisation).

In fact O_DIRECTORY is also used by Glibc to optimise away stat() before and fstat() after calls.

An O_NODEVICE flag would be equally secure, and more generally useful.

It's important that device nodes cannot be opened when O_DIRECTORY is set. This is compatible with reiser4 file-as-directory semantics, but I don't know if reiser4 actually implements this. If it does (and it should) then there is no device blocking problem.

That leaves only the optimisation of fstat() in opendir().

But that begs the question: do we want opendir() to succeed on a reiser4 file?

It's up to us to decide if we like that semantic, or prefer a different one such as the one I described before (path search enters file-as-directory, but opendir() directly on it fails), or the Sun syscalls someone mentioned.

-- Jamie

* Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on many systems. It's best if you can avoid opening devices at all though.

#497 lore raw
Quoted 25 lines · Jamie Lokier
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines

Reiser4 may have a mount option for whoose who like or have to use traditional O_DIRECTORY semantics. There would be no /metas under non-directories, if user wants that.

Again, O_DIRECTORY was added to solve a real-world race, not just for the sake of it. If you really want to add that option I'll research the CAN number for you so you can named it after that - or just call it -o insecure directly.

man open(2) explains that O_DIRECTORY is used by opendir() to prevent blocking when opening pipes and certain devices*, and should only by used by opendir (of course it isn't only used by opendir, as it's a handy optimisation).

In fact O_DIRECTORY is also used by Glibc to optimise away stat() before and fstat() after calls.

An O_NODEVICE flag would be equally secure, and more generally useful.

Quoted 10 lines

It's important that device nodes cannot be opened when O_DIRECTORY is set. This is compatible with reiser4 file-as-directory semantics, but I don't know if reiser4 actually implements this. If it does (and it should) then there is no device blocking problem.

That leaves only the optimisation of fstat() in opendir().

But that begs the question: do we want opendir() to succeed on a reiser4 file?

Yes, we do. Why not? Not is more complex.....

Quoted 14 lines

It's up to us to decide if we like that semantic, or prefer a different one such as the one I described before (path search enters file-as-directory, but opendir() directly on it fails), or the Sun syscalls someone mentioned.

-- Jamie

* Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on many systems. It's best if you can avoid opening devices at all though.

#498 lore raw
In reply to Jamie Lokier
Quoted 15 lines · Jamie Lokier
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines

Reiser4 may have a mount option for whoose who like or have to use traditional O_DIRECTORY semantics. There would be no /metas under non-directories, if user wants that.

Again, O_DIRECTORY was added to solve a real-world race, not just for the sake of it. If you really want to add that option I'll research the CAN number for you so you can named it after that - or just call it -o insecure directly.

man open(2) explains that O_DIRECTORY is used by opendir() to prevent blocking when opening pipes and certain devices*, and should only by used by opendir (of course it isn't only used by opendir, as it's a handy optimisation).

Quoted 4 lines

In fact O_DIRECTORY is also used by Glibc to optimise away stat() before and fstat() after calls.

An O_NODEVICE flag would be equally secure, and more generally useful.

It would prevent the open device issue but you'd need another syscall. Anyways, this is an existing API and chaging it now would suddenly making everything that relies on it for that purpose insecure, so it's completely out of option.

#499 lore raw
In reply to Christoph Hellwig
Quoted 12 lines · Christoph Hellwig
Quoted 5 lines · Alex Zarochentsev

Reiser4 may have a mount option for whoose who like or have to use traditional O_DIRECTORY semantics. There would be no /metas under non-directories, if user wants that.

Again, O_DIRECTORY was added to solve a real-world race, not just for the sake of it.

Can you supply more details and we will try to reply concretely? Thanks.

Quoted 8 lines

If you really want to add that option I'll research the
CAN number for you so you can named it after that - or just call it -o insecure directly.

#500 lore raw
Quoted 4 lines · Hans Reiser
Quoted 3 lines

Again, O_DIRECTORY was added to solve a real-world race, not just for the sake of it.

Can you supply more details and we will try to reply concretely? Thanks.

It's really userland programming 101 :)

you need to fstat after open to really check that no one swtiched the path below you to a fifo or device, O_DIRECTORY gets that directly in the open call, with the additional benefit of making sure you don't get any of the sideeffects at all that would happen if you're opening a device file.

the current reiser4 semantics break that and as soon as you're having a world-writeable (e.g. /tmp) dir on it and someone is doing an opendir on it he's lost.

#501 lore raw
Quoted 9 lines · Christoph Hellwig

you need to fstat after open to really check that no one swtiched the path below you to a fifo or device, O_DIRECTORY gets that directly in the open call, with the additional benefit of making sure you don't get any of the sideeffects at all that would happen if you're opening a device file.

the current reiser4 semantics break that and as soon as you're having a world-writeable (e.g. /tmp) dir on it and someone is doing an opendir on it he's lost.

How does the current reiser4 semantics break that?

In a reiser4 filesystem, a file _is_ a directory.
opendir() is supposed to succeed on it.

There's bound to be some security issue, but I'm not sure what you're getting at with /tmp. What sort of sort of security problem arises with a world-writeable directory such as /tmp, that cannot arise with the standard fs semantics?

-- Jamie

#502 lore raw
Quoted 6 lines · Jamie Lokier
Quoted 3 lines

the current reiser4 semantics break that and as soon as you're having a world-writeable (e.g. /tmp) dir on it and someone is doing an opendir on it he's lost.

How does the current reiser4 semantics break that?

Quoted 7 lines

In a reiser4 filesystem, a file _is_ a directory.
opendir() is supposed to succeed on it.

There's bound to be some security issue, but I'm not sure what you're getting at with /tmp. What sort of sort of security problem arises with a world-writeable directory such as /tmp, that cannot arise with the standard fs semantics?

Actually you are right on that issue because it would open the device/fifo as directory and not device/fifo (in fact I'd had to look at the code again to see whether they actually do this only for files or also for special files)

#503 lore raw
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines

There's bound to be some security issue, but I'm not sure what you're getting at with /tmp. What sort of sort of security problem arises with a world-writeable directory such as /tmp, that cannot arise with the standard fs semantics?

Actually you are right on that issue because it would open the device/fifo as directory and not device/fifo (in fact I'd had to look at the code again to see whether they actually do this only for files or also for special files)

Are you saying that with reiser4, you can open a device or fifo with O_DIRECTORY?

-- Jamie

#504 lore raw
Quoted 6 lines · Jamie Lokier
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines

There's bound to be some security issue, but I'm not sure what you're getting at with /tmp. What sort of sort of security problem arises with a world-writeable directory such as /tmp, that cannot arise with the standard fs semantics?

Quoted 7 lines
Quoted 4 lines

Actually you are right on that issue because it would open the device/fifo as directory and not device/fifo (in fact I'd had to look at the code again to see whether they actually do this only for files or also for special files)

Are you saying that with reiser4, you can open a device or fifo with O_DIRECTORY?

That's what I thought, but as far as I can follow the code this is not actually true.

#505 lore raw
Quoted 6 lines · Christoph Hellwig
Quoted 3 lines

Are you saying that with reiser4, you can open a device or fifo with O_DIRECTORY?

That's what I thought, but as far as I can follow the code this is not actually true.

It should be possible to do, though. There's nothing really different in making the "default" (unnamed) fork be a special device or a fifo.

And it would be perfectly ok for O_DIRECTORY to open such a file, as long as it opens the directory branch, not the special device.

I advocated (long ago) something like this for /dev handling, just because I think it would make sense to have

	/dev/hda	<- special file
	/dev/hda/part1	<- partition 1 (aka /dev/hda1)

which just seems like a very obvious and intuitive interface to me. Of course, we have so much legacy in /dev that there's no real point to doing this, but it's still an appealing approach, I think.

But I do take Al's concerns seriously. I like the notion of supporting "containers", but there are undoubtedly serious issues in the notion. I don't strictly know exactly _how_ to implement it sanely (I can talk about using the vfsmnt structure all I like, but the fact is, it's a different thing from a normal mount, and there may be serious problems indeed there).

Still, I really do like the idea of merging the notion of file and directory into one notion of "container". I absolutely _detest_ files with internal structure that tools have to know about (ie I hate seeing all those embedded formats that I can't use "grep" on - MIME being one case). I'd much rather see a "group of files" and a "file with a grouping of information".

(Now, flattening that "group of files" is obviously needed for serial protocols, so I think MIME/tar/xxxx are fine for _transporting_ data, but I'm saying that outside of transport I really prefer a "collection of files" approach).

Linus

#506 lore raw
Quoted 2 lines · Linus Torvalds

And it would be perfectly ok for O_DIRECTORY to open such a file, as long as it opens the directory branch, not the special device.

What about:

    cd /dev/into_directory_branch/
    run_setuid_program

        -> calls pwd
           pwd opens("."), (".."), ("../..") etc.
 
        -> the setuid program thus ends up opening a device or fifo,
           when it does pwd's path walk.  Yes it could use the getcwd
           syscall, but some programs do their own path walk.
Quoted 9 lines

I advocated (long ago) something like this for /dev handling, just because I think it would make sense to have

	/dev/hda	<- special file
	/dev/hda/part1	<- partition 1 (aka /dev/hda1)

which just seems like a very obvious and intuitive interface to me. Of course, we have so much legacy in /dev that there's no real point to doing this, but it's still an appealing approach, I think.

It also fits the container idea very well:

/dev/hda/part1/ <- the filesystem inside partition 1

That's not a contrivance, it's what I'd expect to happen if /dev/hda is an ordinary file containing a disk image:

image.bin <- hard disk image (a regular file)
image.bin/part1 <- partition 1
image.bin/part1/ <- the filesystem inside partition 1

That's assuming there's a format handler which recognises that image.

-- Jamie

#507 lore raw
Quoted 5 lines · Jamie Lokier
    run_setuid_program

        -> calls pwd
           pwd opens("."), (".."), ("../..") etc.

Ehh.. Not only does pwd not do that..

(hint: there's a getcwd() system call)

Quoted 3 lines

-> the setuid program thus ends up opening a device or fifo,
when it does pwd's path walk. Yes it could use the getcwd syscall, but some programs do their own path walk.

.. but even if it did that, it should use O_DIRECTORY when it did so. If it doesn't, it's broken.

So no, it would _not_ open the device or fifo when it did so.

The fact is, anything that expects to open a directory should already be opening it with O_DIRECTORY.

That said, ".." and "." are special already inside the kernel, and it migth be worth making them automatically imply O_DIRECTORY, since nothing else makes sense anyway. That would fix the case where somebody uses ".." _without_ using O_DIRECTORY.

Quoted 3 lines

It also fits the container idea very well:

/dev/hda/part1/ <- the filesystem inside partition 1

I don't think you can do that. The kernel has no idea how to mount the filesystem.

If it's already mounted somewhere else, that's a different issue. Although it might be mounted in several places (as a bind mount) with different writability, I guess, so even then it might be "interesting".

Linus

#508 lore raw
Quoted 4 lines · Linus Torvalds
Quoted 1 line

/dev/hda/part1/ <- the filesystem inside partition 1

I don't think you can do that. The kernel has no idea how to mount the filesystem.

... and what's more, fundamental thing for mount is fs type, what to do with device(s) (if any) and any other arguments is the filesystem's business.

#509 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 3 lines

-> the setuid program thus ends up opening a device or fifo,
when it does pwd's path walk. Yes it could use the getcwd syscall, but some programs do their own path walk.

.. but even if it did that, it should use O_DIRECTORY when it did so. If it doesn't, it's broken.

Didn't someone just say that O_DIRECTORY will succeed on a device, precisely because opendir() is supposed to succeed on the device?

I'm not sure. Let me refresh my memory.

Quoted 2 lines · Linus Torvalds

And it would be perfectly ok for O_DIRECTORY to open such a file, as long as it opens the directory branch, not the special device.

Oh, I see. You're right. ".." sees the path of directory branches.

Quoted 6 lines
Quoted 3 lines

It also fits the container idea very well:

/dev/hda/part1/ <- the filesystem inside partition 1

I don't think you can do that. The kernel has no idea how to mount the filesystem.

It is not the kernel which decides. The filesystem containing /dev/hda/part1 opens "the directory branch". If the filesystem is suitably configured, it's exactly the same as opening a .zip or ELF
file: it asks userspace for help. Userspace helps by either mounting it, or creating a container virtual directory -- in exactly the same way it helps with .zip and ELF files by creating a container virtual directory for those. It would mount it if the kernel supports the fs, and create a virtual directory if the kernel doesn't support the fs and userspace does.

This is exactly the same as when you cd into a filesystem image: the helper mounts (this time with a lookback file mount) if the kernel supports the image fs, otherwise the helper may be able to decode it in userspace.

By the way, these mechanisms could allow some of the old filesystems that nobody uses any more to be removed from the kernel altogether and kept in an archive of userspace-only supported filesystems. What do you think of that idea?

Quoted 3 lines

If it's already mounted somewhere else, that's a different issue. Although it might be mounted in several places (as a bind mount) with different writability, I guess, so even then it might be "interesting".

The obvious implementation has the userspace helper just mounting it, end of story. If the mount command fails, it fails. Much like autofs.

-- Jamie

#510 lore raw
Quoted 6 lines · Jamie Lokier
Quoted 3 lines

.. but even if it did that, it should use O_DIRECTORY when it did so. If it doesn't, it's broken.

Didn't someone just say that O_DIRECTORY will succeed on a device, precisely because opendir() is supposed to succeed on the device?

It will succeed on the _name_ of the device, but it won't open the device node. It would open the _directory_ node associated with that name.

Think of it this way: a pathname always points to one "container". That container is either a directory or a single node - and O_DIRECTORY ends up being the thing that chooses between them.

Quoted 5 lines
Quoted 2 lines

I don't think you can do that. The kernel has no idea how to mount the filesystem.

It is not the kernel which decides. The filesystem containing /dev/hda/part1 opens "the directory branch".

But that filesystem cannot know what the _other_ filesystem configurations are. And that's what you'd have to have to mount.

Quoted 2 lines

The obvious implementation has the userspace helper just mounting it, end of story. If the mount command fails, it fails. Much like autofs.

Yes, that would work, but it's of questionable use. If you want autofs, then just _use_ autofs.

Linus

#511 lore raw
Quoted 5 lines · Linus Torvalds
Quoted 2 lines

It is not the kernel which decides. The filesystem containing /dev/hda/part1 opens "the directory branch".

But that filesystem cannot know what the _other_ filesystem configurations are. And that's what you'd have to have to mount.

At which point, userspace comes in. Who knows what userspace chooses.

Quoted 5 lines
Quoted 2 lines

The obvious implementation has the userspace helper just mounting it, end of story. If the mount command fails, it fails. Much like autofs.

Yes, that would work, but it's of questionable use. If you want autofs, then just _use_ autofs.

I don't mean to imply it's useful behaviour: only that it seems to happen by default if you don't do anything special to prevent it, if you see what I mean.

It does seem useful to have _files_ mounted automagically (using loopback devices): for example, cd into a .iso file and it would be nice to see the contents. At this point, deciding on the preferred -EBUSY or not behaviour becomes relevant, as does deciding whether the .iso in this example is viewed using the kernel's isofs code, or using a userspace helper in the same way as we'd use one to browse a .zip file.

-- Jamie

#512 lore raw
In reply to Linus Torvalds
Quoted 17 lines · Linus Torvalds
Quoted 5 lines · Jamie Lokier
    run_setuid_program

        -> calls pwd
           pwd opens("."), (".."), ("../..") etc.

Ehh.. Not only does pwd not do that..

(hint: there's a getcwd() system call)

Quoted 3 lines

-> the setuid program thus ends up opening a device or fifo,
when it does pwd's path walk. Yes it could use the getcwd syscall, but some programs do their own path walk.

Quoted 7 lines

.. but even if it did that, it should use O_DIRECTORY when it did so. If it doesn't, it's broken.

So no, it would _not_ open the device or fifo when it did so.

The fact is, anything that expects to open a directory should already be opening it with O_DIRECTORY.

reiser4 files-as-dirs do not depend on that.

A file behaves as directory if one calls i_op->lookup() or f_op->readdir(). I am not sure that O_DIRECTORY should be a switch.

Quoted 17 lines

That said, ".." and "." are special already inside the kernel, and it migth be worth making them automatically imply O_DIRECTORY, since nothing else makes sense anyway. That would fix the case where somebody uses ".." _without_ using O_DIRECTORY.

Quoted 3 lines

It also fits the container idea very well:

/dev/hda/part1/ <- the filesystem inside partition 1

I don't think you can do that. The kernel has no idea how to mount the filesystem.

If it's already mounted somewhere else, that's a different issue. Although it might be mounted in several places (as a bind mount) with different writability, I guess, so even then it might be "interesting".

Linus

Signature · 1 line
Alex.
#513 lore raw
In reply to Linus Torvalds
Quoted 5 lines · On 26-Aug-2004 Linus Torvalds wrote:

I advocated (long ago) something like this for /dev handling, just because I think it would make sense to have

/dev/hda <- special file
/dev/hda/part1 <- partition 1 (aka /dev/hda1)

This breaks the r4 semantics if I understood it correctly. Because /partN are not simply associated to the file, they are part of the file. ie. when I modify /dev/hda I also change /dev/hda/partN and vice-versa. I don't see any pratical problem, though.

Quoted 6 lines

Still, I really do like the idea of merging the notion of file and directory into one notion of "container". I absolutely _detest_ files with internal structure that tools have to know about (ie I hate seeing all those embedded formats that I can't use "grep" on - MIME being one case). I'd much rather see a "group of files" and a "file with a grouping of information".

You're actually looking for a database with a legacy fs-like interface :)

--
Giuliano.

#514 lore raw
Quoted 14 lines · Giuliano Pochini
Quoted 5 lines · On 26-Aug-2004 Linus Torvalds wrote:

I advocated (long ago) something like this for /dev handling, just because I think it would make sense to have

/dev/hda <- special file
/dev/hda/part1 <- partition 1 (aka /dev/hda1)

This breaks the r4 semantics if I understood it correctly. Because /partN are not simply associated to the file, they are part of the file. ie. when I modify /dev/hda I also change /dev/hda/partN and vice-versa. I don't see any pratical problem, though.

Quoted 8 lines
Quoted 6 lines

Still, I really do like the idea of merging the notion of file and directory into one notion of "container". I absolutely _detest_ files with internal structure that tools have to know about (ie I hate seeing all those embedded formats that I can't use "grep" on - MIME being one case). I'd much rather see a "group of files" and a "file with a grouping of information".

You're actually looking for a database with a legacy fs-like interface :)

Hm. Isn't this exactly what MS has in store for Longhorn?

Signature · 7 lines
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)
#515 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Giuseppe" == Giuseppe Bilotta <[email protected]> writes:

Giuseppe> Giuliano Pochini wrote:

[...]

Quoted 2 lines
Quoted 2 lines

You're actually looking for a database with a legacy fs-like interface :)

Giuseppe> Hm. Isn't this exactly what MS has in store for Longhorn?

No. To access the database-like functionality in WinFS, you need to use a special API.

Hans as been planning this for a long time before WinFS was ever announced. See http://www.namesys.com/whitepaper.html

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#516 lore raw
Quoted 6 lines · Hubert Chan
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Giuseppe" == Giuseppe Bilotta <[email protected]> writes:

Giuseppe> Giuliano Pochini wrote:

[...]

Quoted 7 lines
Quoted 2 lines
Quoted 2 lines

You're actually looking for a database with a legacy fs-like interface :)

Giuseppe> Hm. Isn't this exactly what MS has in store for Longhorn?

No. To access the database-like functionality in WinFS, you need to use a special API.

I see, thanks. Might be interesting to know what they plan to make accessible of WinFS to legacy apps via the old API, then (especially since, at according to a /. article, they're planning to retrofit WinFS to WinXP ...)

Signature · 7 lines
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)
#517 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Giuseppe" == Giuseppe Bilotta <[email protected]> writes:

Quoted 2 lines · Giuseppe> Hubert Chan wrote:
Quoted 2 lines

No. To access the database-like functionality in WinFS, you need to use a special API.

Giuseppe> I see, thanks. Might be interesting to know what they plan to Giuseppe> make accessible of WinFS to legacy apps via the old API, then Giuseppe> (especially since, at according to a /. article, they're Giuseppe> planning to retrofit WinFS to WinXP ...)

AFAIK, all files are still accessible through normal NTFS. You just need the special API to do querying and such. So old apps can access all the files normally. They just can't take advantage of all the new whizbang features.

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#518 lore raw
In reply to Giuliano Pochini
Quoted 10 lines · Giuliano Pochini
Quoted 5 lines · On 26-Aug-2004 Linus Torvalds wrote:

I advocated (long ago) something like this for /dev handling, just because I think it would make sense to have

/dev/hda <- special file
/dev/hda/part1 <- partition 1 (aka /dev/hda1)

This breaks the r4 semantics if I understood it correctly.

Reiser4 currently links /metas/ directory under each reiser4 fs object. Reiser4 "silently" allows regular file to be a directory for that.

That is only an example how an fs object may link its internal hierarchical structure to the fs tree. /dev/hda/-as-a-dir is another example, but, I am afraid, it is not a reiser4 business to look into the devices. Devfs or its successor may do it.

Quoted 18 lines

Because
/partN are not simply associated to the file, they are part of the file. ie. when I modify /dev/hda I also change /dev/hda/partN and vice-versa. I don't see any pratical problem, though.

Quoted 6 lines

Still, I really do like the idea of merging the notion of file and directory into one notion of "container". I absolutely _detest_ files with internal structure that tools have to know about (ie I hate seeing all those embedded formats that I can't use "grep" on - MIME being one case). I'd much rather see a "group of files" and a "file with a grouping of information".

You're actually looking for a database with a legacy fs-like interface :)

--
Giuliano.

Signature · 1 line
Alex.
#519 lore raw
In reply to Christoph Hellwig
Quoted 12 lines · Christoph Hellwig
Quoted 11 lines · Jamie Lokier
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines

There's bound to be some security issue, but I'm not sure what you're getting at with /tmp. What sort of sort of security problem arises with a world-writeable directory such as /tmp, that cannot arise with the standard fs semantics?

Actually you are right on that issue because it would open the device/fifo as directory and not device/fifo (in fact I'd had to look at the code again to see whether they actually do this only for files or also for special files)

Quoted 5 lines
Quoted 2 lines

Are you saying that with reiser4, you can open a device or fifo with O_DIRECTORY?

That's what I thought, but as far as I can follow the code this is not actually true.

All reiser4 inodes have i_ops->lookup != NULL, so open(..., O_DIRECTORY) would succeed on them (thanks Nikita for reminding me that).

It may be better to pass the control of that to ->i_op->permission() through explicit MAY_LOOKUP flag. It can be possible to eliminate the race by some more complex logic (strict ownership check, for example) in ->permission().

Signature · 1 line
Alex.
#520 lore raw
In reply to Hans Reiser
Quoted 11 lines · Hans Reiser

You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles. When I said we should just add some nice optional features to files and directories so that they can do everything that attributes can do if they are used that way, you just didn't get it. You instead went for the quick ugly hack called xattrs. You then got that ugly hack done first, because quick hacks are, well, quick. I then went about doing it the right way for Reiser4, and got DARPA to fund doing it. I was never silent about it.

I don't want to comment on any of the technical issues about VFS etc. as I would be completely out of my depth, however I do want to say 2 things. Firstly, this is a feature that Samba users have been needing for many years to maintain compatibility with NTFS and Windows clients. Microsoft no longer sell any servers or clients without support for multiple data streams per file, and their latest XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad that Hans and Namesys have created something we can use to match this functionality - soon we will need it in order to be able to exist in a Microsoft client-dominated world.

My second point is the following. Hans - did you *really* have to reinvent the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001 for Solaris) ?

http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5
http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C
http://bama.ua.edu/cgi-bin/man-cgi?openat+2

I'm complaining here as someone who will have to write portable code to try and work on all these "files with streams" systems.

Jeremy.

#521 lore raw
Quoted 7 lines · Jeremy Allison
Quoted 3 lines · Hans Reiser

You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles. When I said we should just add some nice optional features to files and directories so that they can do everything that attributes can do if they are used that way, you just didn't get it. You instead went for the quick ugly hack called xattrs. You then got that ugly hack done first, because quick hacks are, well, quick. I then went about doing it the right way for Reiser4, and got DARPA to fund doing it. I was never silent about it.

Quoted 11 lines

I don't want to comment on any of the technical issues about VFS etc. as I would be completely out of my depth, however I do want to say 2 things. Firstly, this is a feature that Samba users have been needing for many years to maintain compatibility with NTFS and Windows clients. Microsoft no longer sell any servers or clients without support for multiple data streams per file, and their latest XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad that Hans and Namesys have created something we can use to match this functionality - soon we will need it in order to be able to exist in a Microsoft client-dominated world.

I agree that your work is important without agreeing that MS client domination will last.;-) It is indeed my desire to give you every single feature you need to emulate MS streams within files, but doing it using directories that are files. I would like to support you in emulating windows faster than windows.

Quoted 5 lines

My second point is the following. Hans - did you *really* have to reinvent the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001 for Solaris) ?

I interviewed for the file system architect job at Sun in, I think, 1999, and they offered me the job conditional on my giving up on my Linux work. (After much trying and failing to convince them that it would be okay for me to work on Linux also, I declined the job, much to my fiscal loss and work satisfaction.)

They do not do a pure job of implementing attributes in the file namespace though. There are far more distinctions between files and attributes than are necessary that are described in these man pages below, and those distinctions cause a loss of closure. I can say more on that if asked.

Quoted 12 lines

http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5
http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C
http://bama.ua.edu/cgi-bin/man-cgi?openat+2

I'm complaining here as someone who will have to write portable code to try and work on all these "files with streams" systems.

Jeremy.

#522 lore raw
Quoted 5 lines · Hans Reiser

I agree that your work is important without agreeing that MS client domination will last.;-) It is indeed my desire to give you every single feature you need to emulate MS streams within files, but doing it using directories that are files. I would like to support you in emulating windows faster than windows.

1) how do you back up and restore files with streams inside ?

2) how do standard unix utilities handle them ?

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#523 lore raw
Quoted 2 lines · Rik van Riel

1) how do you back up and restore files with streams inside ?

With a program that supports files with streams inside.

Quoted 1 line

2) how do standard unix utilities handle them ?

Most likely they don't ;) That is, until they are fixed or replaced. I've heard of people who want xattrs to be backed up so they use star, not gnu tar, already.

Signature · 1 line
mjt
#524 lore raw
Quoted 3 lines · Markus
Quoted 1 line

2) how do standard unix utilities handle them ?

Most likely they don't ;) That is, until they are fixed or replaced.

Not good, in my view.

Signature · 3 lines
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <[email protected]> 1.650.933.1373
#525 lore raw
In reply to Rik van Riel
Quoted 4 lines · Rik van Riel

On Thu, 26 Aug 2004, Hans Reiser wrote:

Quoted 10 lines
Quoted 3 lines

I agree that your work is important without agreeing that MS client domination will last.;-) It is indeed my desire to give you every single feature you need to emulate MS streams within files, but doing it using directories that are files. I would like to support you in emulating windows faster than windows.

1) how do you back up and restore files with streams inside ?

2) how do standard unix utilities handle them ?

To repeat, I think it would be nice to implement a filename/pseudos/backup method for all the plugins.

Guys, we just have the beginnings in place. One plugin method at a time it will all fall into place.

What we have now is useful now. The more methods come into existence, the more compelling it becomes --- standard network economic theory applies here.

Hans

#526 lore raw
In reply to Hans Reiser
Quoted 9 lines · Hans Reiser

I had not intended to respond to this because I have nothing positive to say, but Andrew said I needed to respond and suggested I should copy Linus. Sigh.

Dear Christoph,

Let me see if I can summarize what you and your contingent are saying, and if I misconstrue anything, let me know.;-)

Just for fun why don't we look at the way things are today:

1) reiser4 has semantics that do belong at the VFS level. They weren't implemented at the VFS level for a variety of reasons, none of which really matter right now.

2) new kernel patches that fragment the application developers between apis are a bad thing. There does need to be one interface here, and it is in Hans' best interest to unify his work by working with people to introduce new kernel wide apis.

This starts with exactly what Christoph described in writing a short summary of how you want things to work today. Since we can't resist, we'll also go ahead and rehash all the old flame wars over this, but try to include some new ideas about where you want to see the reiser4 interfaces in 6 months as well.

-chris

#527 lore raw
Quoted 16 lines · Chris Mason

This starts with exactly what Christoph described in writing a short summary of how you want things to work today. Since we can't resist, we'll also go ahead and rehash all the old flame wars over this, but try to include some new ideas about where you want to see the reiser4 interfaces in 6 months as well.

-chris

Did Christoph read the www.namesys.com URL that has for several years been hiding from him? ;-)

Hans

#528 lore raw
Quoted 2 lines · Hans Reiser

Did Christoph read the www.namesys.com URL that has for several years been hiding from him? ;-)

Yes, and I'm sick of your marketing bullshit Hans.

#529 lore raw
In reply to Hans Reiser
Quoted 10 lines · Hans Reiser

Making files into directories caused only two applications out of the entire OS to notice the change, and that was because of a bug in what error code we returned that we are going to fix. You think that was a disaster; I think it was a triumph.

Since we have such a performance lead, Namesys is about to change its
focus from the storage layer to semantics, look at www.namesys.com/whitepaper.html for details. Semantic enhancements are the important stuff, and finally Namesys is where we have all the storage layer prerequisites done right, and the real work can begin.

Just curious about your comments on Jamie Lokier's suggestions for enabling files-as-directories semantics without breaking existing apps.

Chris

#530 lore raw
In reply to Hans Reiser
Quoted 4 lines · Hans Reiser

I had not intended to respond to this because I have nothing positive to say, but Andrew said I needed to respond and suggested I should copy Linus.

Yes, but I didn't say "flame Christoph and ignore the issues" ;)

There are lots of little things to do with implementation, coding style, module exports, deadlocks, what code goes where, etc. These are all normal daily kernel business and we should set them aside for now and concentrate on the bigger issues.

And as I see it, there are two big issues:

a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not anticipate and

b) it does this within the context of just a single filesystem.

I see three possible responses:

a) accept the reiser4-only extensions as-is (possibly with post-review modifications, of course) or

b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or

c) reject the extensions.

My own order of preference is b) c) a). The fact that one filesystem will offer features which other filesystems do not and cannot offer makes me queasy for some reason.

b) means that at some time in the future we need to hoist the reiser4 extensions (at a conceptual level) (and probably with restrictions) up into the VFS. This will involve much thought, argument and work.

To get us started on this route it would really help me (and, probably, others) if you could describe what these API extensions are in a very simple way, without referring to incomprehehsible web pages, and without using terms which non-reiser4 people don't understand.

It would be best if each extension was addressed in a separate email thread.

We also need to discuss what a reiser4 "module" is, what its capabilities are, and what licensing implications they have.

Then, we can look at each one and say "yup, that makes sense - we want Linux to do that" and we can also think about how we would implement it at the VFS level.

If we follow the above route I believe we can make progress in a technical direction and not get deadlocked on personal/political stuff.

Now, an alternative to all the above is to just merge reiser4 as-is, after addressing all the lower-level coding issues. And see what happens. That may be a thing which Linus wishes to do. I'm easy.

#531 lore raw
Quoted 17 lines

And as I see it, there are two big issues:

a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not anticipate and

b) it does this within the context of just a single filesystem.

I see three possible responses:

a) accept the reiser4-only extensions as-is (possibly with post-review modifications, of course) or

b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or

Quoted 6 lines

c) reject the extensions.

My own order of preference is b) c) a). The fact that one filesystem will offer features which other filesystems do not and cannot offer makes me queasy for some reason.

This last sentence makes me wonder. Where is Linux heading? The idea
that a FS cannot contain features that no other FS has is very
scary.

I am all for uniformity, but not at the expense of shutting down advanced progress that Linux is so badly needing.

This talk about old UNIX seems like people want to still live in the 70'ies and not look forward. Please wake up!

~S

#532 lore raw
Quoted 9 lines

This last sentence makes me wonder. Where is Linux heading? The idea
that a FS cannot contain features that no other FS has is very
scary.

I am all for uniformity, but not at the expense of shutting down advanced progress that Linux is so badly needing.

This talk about old UNIX seems like people want to still live in the 70'ies and not look forward. Please wake up!

Just because semantics are at the VFS layer doesn't mean every filesystem has to implement them.