silent semantic changes with reiser4
After looking trough the code and mailinglists I'm quite unhappy with a bunch of user-visible changes that Hans sneaked in and make reiser4 incompatible with other filesystems and have a slight potential to break even in the kernel.
o files as directories
- O_DIRECTORY opens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely useless
- meaning of the -x permission. This one has different meanings on directories vs files on UNIX systems. If we want to support directories as files we'll probably have to find a way to work around this.
- dentry aliasing. I can't find a formal guarantee in the code this can't happen
o metafiles - ..metas as a magic name that's just taken out of the namespace doesn't sound like a good idea. If we want this it should be a VFS-level option and there should be a translation-layer to xattrs. Not doing this will again confuse applications greatly that expect uniform filesystem behaviour.
Given these problems I request that these interfaces are removed from reiser4 for the kernel merge, and if added later at the proper VFS level after discussion on linux-kernel and linux-fsdevel, like we did for xattrs.
Quoted 4 lines · Christoph Hellwig
-
O_DIRECTORYopens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely useless
So `find -type d' would list every file on the system?
Lee
Quoted 5 lines · Christoph Hellwig
o files as directories
-O_DIRECTORYopens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely useless
Ouch.
I would definitely classify this as a security hole, since userland definitely uses O_DIRECTORY to avoid races.
Jeff
Quoted 7 lines · Lee Revell
Quoted 4 lines · Christoph Hellwig
-
O_DIRECTORYopens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely uselessSo `find -type d' would list every file on the system?
the find I have here is using lstat and not open with O_DIRECTORY, so no.
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Lee Revell
Quoted 4 lines · Christoph Hellwig
-
O_DIRECTORYopens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely useless
Quoted 5 lines
Quoted 1 line
So `find -type d' would list every file on the system?
the find I have here is using lstat and not open with
O_DIRECTORY, so no.
Ugh, how embarrassing, I completely forgot about stat().
Lee
Quoted 8 lines · Jeff Garzik
Quoted 5 lines · Christoph Hellwig
o files as directories
-O_DIRECTORYopens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely useless
Quoted 4 lines
Ouch.
I would definitely classify this as a security hole, since userland definitely uses
O_DIRECTORYto avoid races.
Feh. That's far from the worst parts of the mess introduced by "hybrid" crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit higher on the suckitude scale, IMO.
Quoted 9 lines · Christoph Hellwig
Quoted 6 lines
Quoted 4 lines
-
O_DIRECTORYopens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely uselessSo `find -type d' would list every file on the system?
the find I have here is using lstat and not open with
O_DIRECTORY, so no.
The find-like program I use (called treescan) uses O_DIRECTORY as an optimisation. It assumes that O_DIRECTORY will only open objects which are directories and can be read using readdir().
However, if reiser4 returns d_type values, then it won't even attempt an open on non-DT_DIR objects, and that's a better optimisation. (reiserfs doesn't return d_type values, unfortunately).
So the list of files that treescan finds depends on whether reiser4 implements d_type.
This is nothing like a POSIX filesystem. You untar a tree, and then listing it recursively shows extra things created by reiser4.
I quite like the principle, but because it's not like POSIX and doesn't match some program's expectations, it's a problem in its present form.
xattrs aren't a complete solution as you can't store structured data in an xattr. For example, with reiser4's model, you can cd into a .tar, .zip, .mp3 or .xml file and list the internal structure along with the file's metadata. You can't do that with xattrs.
Programs exist which quite reasonably assume that when you create a file, you can't opendir() the file, and recursive listings (like find, ls -R et al.) won't automatically traverse into every file.
On the other hand, being able to enter a file in a directory-like way allows structured representations of the contents to be accessed in the very useful "everything's a file" way -- i.e. ordinary tools.
So here's a semantic proposal:
1. O_DIRECTORY won't open an ordinary file.
Corollary: opendir("file") won't open an ordinary file.
2. An ordinary file path followed by "/" won't open an ordinary file.
Corollary: opendir("file/") won't open an ordinary file.
This is because appending a trailing slash is an alternate way for userspace to get the same results as O_DIRECTORY.
3. An ordinary file path followed by "/" _and_ one or more path components will open the file as a directory and enter it.
Corollary: opendir("file/.") will open an ordinary file.
4. The type of "file/." shall be S_IFDIR, _not_ S_IFREG.
Corollary: stat("file/.") will return that it's a directory.
The intention here is that explicit requests to examine the metadata or alternate structure representations of a file will create such a view, but the view is only available if requested explicitly.
When such a view is created, the results of stat(), O_DIRECTORY and opendir() are absolutely consistent. This will minimise confusion. Programs which recurse over a directory tree won't look inside any of the files. However they can be explicitly asked to recurse starting from a path inside a file: then they'll recurse over a single file's metadata and structured data.
Regarding the problems of safe locking in the VFS. The VFS assumes that directories are not hard linked: i.e. that they cannot appear at more than one path in a filesystem. Files-as-directories breaks that.
However, VFS does support directories on multiple paths, using bind mounts.
So it wouldn't be out of the question if entering a file (as described above) effectively auto-mounted a bind mount at that point.
-- Jamie
Quoted 3 lines · On Tue, Aug 24, 2004 at 09:53:44PM +0100…
Feh. That's far from the worst parts of the mess introduced by "hybrid" crap - trivial
sys_link(2)deadlocks triggerable by any user rate a bit higher on the suckitude scale, IMO.
While we are at it - consider these hybrids vetoed until
a) sys_link()/sys_link() deadlock is fixed
b) sys_link()/sys_rename() deadlock is fixed
c) correctness proof of the locking scheme (inDocumentation/filesystems/directory-locking) is updated to match the presense of the file/directory hybrids.
Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince us that there's no more crap of that kind". IMO a reasonable request, seeing that the first look at the patches in -mm4 had turned up two exploits in that area, despite the *YEARS* of warnings about potential trouble and need to be careful there (actually, I've given Hans too much credit and assumed that link/link never happens since nobody would be dumb enough to provide ->link() method for non-directory inodes; turns out that somebody is dumb enough and link/link is as exploitable as link/rename).
I allowed myself to get talked out of a final top to bottom code audit, and obviously that was a mistake.
It will probably take about 6 weeks. Apologies for wasting your time before that was done.
Hans
Quoted 34 lines · [email protected] wrote:
Quoted 5 lines · On Tue, Aug 24, 2004 at 09:53:44PM +0100…
Feh. That's far from the worst parts of the mess introduced by "hybrid" crap - trivial
sys_link(2)deadlocks triggerable by any user rate a bit higher on the suckitude scale, IMO.While we are at it - consider these hybrids vetoed until
a) sys_link()/sys_link() deadlock is fixed
b) sys_link()/sys_rename() deadlock is fixed
c) correctness proof of the locking scheme (inDocumentation/filesystems/directory-locking) is updated to match the presense of the file/directory hybrids.Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince us that there's no more crap of that kind". IMO a reasonable request, seeing that the first look at the patches in -mm4 had turned up two exploits in that area, despite the *YEARS* of warnings about potential trouble and need to be careful there (actually, I've given Hans too much credit and assumed that link/link never happens since nobody would be dumb enough to provide ->link() method for non-directory inodes; turns out that somebody is dumb enough and link/link is as exploitable as link/rename).
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines · Hans Reiser
I allowed myself to get talked out of a final top to bottom code audit, and obviously that was a mistake.
It will probably take about 6 weeks. Apologies for wasting your time before that was done.
I don't think you'll get anywhere with auditing. We need to write down the semantics you want, define them at the VFS level and make sure they're not conflicting with defined userspace semantics or kernel assumptions.
I think you need to learn the basic distinction between the VFS layer and a lowlevel filesystem driver.
I had not intended to respond to this because I have nothing positive to say, but Andrew said I needed to respond and suggested I should copy Linus. Sigh.
Dear Christoph,
Let me see if I can summarize what you and your contingent are saying, and if I misconstrue anything, let me know.;-)
You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles. When I said we should just add some nice optional features to files and directories so that they can do everything that attributes can do if they are used that way, you just didn't get it. You instead went for the quick ugly hack called xattrs. You then got that ugly hack done first, because quick hacks are, well, quick. I then went about doing it the right way for Reiser4, and got DARPA to fund doing it. I was never silent about it.
Making files into directories caused only two applications out of the entire OS to notice the change, and that was because of a bug in what error code we returned that we are going to fix. You think that was a disaster; I think it was a triumph.
Now a cleanly architected filesystem with no attributes and just files and directories that can do everything attributes are used for exists. You don't want it to have the competitive advantage. Instead, you want it to have its clean design excised until you have something that duplicates it ready to go, and only then should it be allowed that users will use the features of your competitor's filesystem which you disdained implementing for so long.
Since you never studied or understood namespace design principles (or you would not have created and supported xattrs), you want to rename it to be called VFS, rewrite what we have done, and take over as the maintainer, mangling its design in a committee clusterfuck as you go.
We have just implemented very trivial semantic enhancements of the FS namespace, nothing like as ambitious as www.namesys.com/whitepaper.html or WinFS, and you are already pissing your pants.
Is that a fair summary?
Eat my dust.
Hans
PS
I should of course qualify what I have said. The use of files and directories in place of attributes is not a finished work. It has bugs, sys_reiser4() does not yet work, and there are little features still missing like having files readdir ignores.
Still, except for the bugs, what we have is usable, and there are a lot of happy reiser4 users right now even with the bugs. It will need a little bit more time, and then all the pieces will be in place.
PPS
If you implement your filesystems as reiser4 plugins, and rename reiser4's plugin code to be called "vfs", your filesystems will go faster. Not as fast as reiser4 though, because it has a better layout and that affects performance a lot, but faster is faster.... See www.namesys.com/benchmarks.html for details.
PPPS
Since we have such a performance lead, Namesys is about to change its focus from the storage layer to semantics, look at www.namesys.com/whitepaper.html for details. Semantic enhancements are the important stuff, and finally Namesys is where we have all the storage layer prerequisites done right, and the real work can begin. The gap between us is about to widen further.
Quoted 4 lines · Christoph Hellwig
After looking trough the code and mailinglists I'm quite unhappy with a bunch of user-visible changes that Hans sneaked in and make reiser4 incompatible with other filesystems
if we leave you in the dust, run faster.... not my problem....
Quoted 8 lines
Given these problems I request that these interfaces are removed from reiser4 for the kernel merge, and if added later at the proper VFS level after discussion on linux-kernel and linux-fsdevel, like we did for xattrs.
If you can't help fight WinFS, then get out of the way. Namesys is on the march. Read www.namesys.com/whitepaper.html.
Or, be smart, recognize that reiser4 is faster and more flexible than your storage layers because we are older and wiser and worked harder at it, join the team, and start contributing plugins that tap into the higher performance it offers.
Microsoft tried to build a storage layer that could handle small objects without losing performance, failed, and gave up at considerable cost to their architecture and pocketbook.
We just broke a hole in the enemy line. You could come swarming through it with us, but it sounds like you prefer complaining to HQ that we are getting too far in front of you.
That's a nice marketing talk. Get back to us when you have some technical contribution to make.
Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain
Quoted 4 lines · Hans Reiser
You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles.
Actually in most of the discussion you simply didn't participate. While xattrs might not be the nicest interface they have the advantag of not breaking the SuS assumption of what directories vs files are, and they do not break the Linux O_DIRECTORY semantics that are defined and need to solve real-world races either.
Quoted 7 lines
When I said we should just add some nice
optional features to files and directories so that they can do everything that attributes can do if they are used that way, you just didn't get it. You instead went for the quick ugly hack called xattrs. You then got that ugly hack done first, because quick hacks are, well, quick. I then went about doing it the right way for Reiser4, and got DARPA to fund doing it. I was never silent about it.
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Quoted 7 lines
Now a cleanly architected filesystem with no attributes and just files and directories that can do everything attributes are used for exists. You don't want it to have the competitive advantage. Instead, you want it to have its clean design excised until you have something that duplicates it ready to go, and only then should it be allowed that users will use the features of your competitor's filesystem which you disdained implementing for so long.
My competitors filesystem? If you look at MAINTAINERS I maintain only vxfs and sysvfs, neither of which I'd suggest anyone to run their system on.
Quoted 4 lines
Since you never studied or understood namespace design principles (or you would not have created and supported xattrs), you want to rename it to be called VFS, rewrite what we have done, and take over as the maintainer, mangling its design in a committee clusterfuck as you go.
Hans, please stop the personal crap or the black helicopters will kidnap
you. When was the last time you actually worked on kernel namespace
code instead of talking marketing bullshit and ignoring all real world problems.
Quoted 5 lines
If you implement your filesystems as reiser4 plugins, and rename reiser4's plugin code to be called "vfs", your filesystems will go faster. Not as fast as reiser4 though, because it has a better layout and that affects performance a lot, but faster is faster.... See www.namesys.com/benchmarks.html for details.
Could you pass on that crack pipe please?
Btw, I just got reminded you might take my saying as an "piss of you're idea stinks" or similar things. So let me clarify again the actual technical and project managment issues another time before we start getting really personal :)
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem. Your attributes are absoultely a VFS thing and as such should not happen at the filesystem layer, and no, that doesn't mean they're bad per se, I just think they are a rather bad fit for Linux.
So now go on and try to work together with the other peope doing VFS level work instead of hiding, or if you think you can't work together with us search a nice research OS where you can take over the VFS layer, if your ideas prove to be good I'm sure Linux will pick them up sooner or later.
Christoph
Quoted 11 lines · Hans Reiser
You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles. When I said we should just add some nice optional features to files and directories so that they can do everything that attributes can do if they are used that way, you just didn't get it. You instead went for the quick ugly hack called xattrs. You then got that ugly hack done first, because quick hacks are, well, quick. I then went about doing it the right way for Reiser4, and got DARPA to fund doing it. I was never silent about it.
I don't want to comment on any of the technical issues about VFS etc. as I would be completely out of my depth, however I do want to say 2 things. Firstly, this is a feature that Samba users have been needing for many years to maintain compatibility with NTFS and Windows clients. Microsoft no longer sell any servers or clients without support for multiple data streams per file, and their latest XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad that Hans and Namesys have created something we can use to match this functionality - soon we will need it in order to be able to exist in a Microsoft client-dominated world.
My second point is the following. Hans - did you *really* have to reinvent the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001 for Solaris) ?
http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5
http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C
http://bama.ua.edu/cgi-bin/man-cgi?openat+2
I'm complaining here as someone who will have to write portable code to try and work on all these "files with streams" systems.
Jeremy.
Quoted 9 lines · Hans Reiser
I had not intended to respond to this because I have nothing positive to say, but Andrew said I needed to respond and suggested I should copy Linus. Sigh.
Dear Christoph,
Let me see if I can summarize what you and your contingent are saying, and if I misconstrue anything, let me know.;-)
Just for fun why don't we look at the way things are today:
1) reiser4 has semantics that do belong at the VFS level. They weren't implemented at the VFS level for a variety of reasons, none of which really matter right now.
2) new kernel patches that fragment the application developers between apis are a bad thing. There does need to be one interface here, and it is in Hans' best interest to unify his work by working with people to introduce new kernel wide apis.
This starts with exactly what Christoph described in writing a short summary of how you want things to work today. Since we can't resist, we'll also go ahead and rehash all the old flame wars over this, but try to include some new ideas about where you want to see the reiser4 interfaces in 6 months as well.
-chris
Quoted 4 lines · Christoph Hellwig
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.
It's the UNIX way.
And yes, the semantics can _easily_ be solved in very unixy ways.
One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is:
- without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
- with the slash, it won't open _without_ O_DIRECTORY (EISDIR)
Problem solved. Very user-friendly, and very intuitive.
Will it potentially break something? Sure. Do we care? Me, I'll take that kind of extension _any_ day over xattrs, that are fundamentally flawed in my opinion and totally useless. The argument that applications like "tar" won't understand the file-as-directory thing is _flawed_, since legacy apps won't understand xattrs either.
Oh, add a O_NOXATTRS flag to force a path lookup to only use regular directories, the same way we have O_NOFOLLOW and friends. That allows people to see the difference, if they care (ie a file server might decide that it doesn't want to expose things like this).
I never liked the xattr stuff. It makes little sense, and is totally useless for 99.9999% of everything. I still don't see the point of it, except for samba. Ugly.
Linus
Quoted 10 lines · Hans Reiser
Making files into directories caused only two applications out of the entire OS to notice the change, and that was because of a bug in what error code we returned that we are going to fix. You think that was a disaster; I think it was a triumph.
Since we have such a performance lead, Namesys is about to change its
focus from the storage layer to semantics, look at www.namesys.com/whitepaper.html for details. Semantic enhancements are the important stuff, and finally Namesys is where we have all the storage layer prerequisites done right, and the real work can begin.
Just curious about your comments on Jamie Lokier's suggestions for enabling files-as-directories semantics without breaking existing apps.
Chris
Quoted 7 lines · Christoph Hellwig
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem. Your attributes are absoultely a VFS thing and as such should not happen at the filesystem layer, and no, that doesn't mean they're bad per se, I just think they are a rather bad fit for Linux.
Now this I agree with, in the sense that I think that if we want to support this, it should be supported at a VFS layer.
On the other hand, I think doing it inside the filesystem with ugly hacks is an acceptable way to prototype the idea before it's been proven to really be workable. Maybe it has more problems with legacy apps than we'd expect..
Linus
Quoted 7 lines · Linus Torvalds
Now this I agree with, in the sense that I think that if we want to support this, it should be supported at a VFS layer.
On the other hand, I think doing it inside the filesystem with ugly hacks is an acceptable way to prototype the idea before it's been proven to really be workable. Maybe it has more problems with legacy apps than we'd expect..
Oh, I'm the last person to tell anyone how to prototype things. I just don't want such inconsistancies in the mainline kernel.
Quoted 11 lines · Christoph Hellwig
Quoted 4 lines · Hans Reiser
You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles.
Actually in most of the discussion you simply didn't participate. While xattrs might not be the nicest interface they have the advantag of not breaking the SuS assumption of what directories vs files are, and they do not break the Linux
O_DIRECTORYsemantics that are defined and need to solve real-world races either.
Reiser4 may have a mount option for whoose who like or have to use traditional O_DIRECTORY semantics. There would be no /metas under non-directories, if user wants that.
[ ... ]
Signature · 1 line
Alex.
Quoted 10 lines · Linus Torvalds
And yes, the semantics can _easily_ be solved in very unixy ways.
One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is:
- without the slash, a file-as-dir won't open with
O_DIRECTORY(ENOTDIR)
- with the slash, it won't open _without_O_DIRECTORY(EISDIR)Problem solved. Very user-friendly, and very intuitive.
That would solve the O_DIRECTORY issue, the dentry aliasing still needs work though with the semantics for link/unlink/rename.
Maybe Hans & you should start 2.7 to work this out? :)
I just want to add that I AM capable of working with the other filesystem developers in a team-player way, and I am happy to cooperate with making portions more reusable where there is serious interest from other filesystems in that, but Christoph is a puppy who has never written or designed a major filesystem from scratch, and Nikita is a big dog who has written stuff very few projects are lucky enough to see the likes of, and when Christoph insults Nikita's code, or my design guidance for that code, it is not going to bring out my good side. The plugin and metafiles code needs many improvements, but Christoph does not have the expertise to understand what those needed improvements are because he hasn't invested the work into understanding the code.
Christoph is a bright and clever young fellow, who just hasn't had the years of study of the field yet. I wish him well, and away.;-)
Hans
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Christoph Hellwig
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Quoted 6 lines
Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.
It's the UNIX way.
Not if you allow link(2) on them. And not if you design and market your stuff as a general-purpose backdoor into kernel. Note how *EVERY* *DAMN* *OPERATION* is made possible to override by "plugins". Which is the reason for deadlocks in question, BTW.
Don't fool yourself - that's what Hans is selling. Target market: ISV. Marketed product: a set of hooks, the wider the better, no matter how little sense it makes. The reason for doing that outside of core kernel:
bypassing any review and being able to control the product being sold (see above).
Shame that it got an actual filesystem mixed in with the marketing plans and general insanity...
Quoted 4 lines · Hans Reiser
I just want to add that I AM capable of working with the other filesystem developers in a team-player way, and I am happy to cooperate with making portions more reusable where there is serious interest from other filesystems in that,
Prove it. Stop replying for today and come back tomorrow with some useful discussions. Christoph suggested that some of the v4 semantics belong in the VFS and therefore linux as a whole. He's helping you to make sure the semantics and fit nicely with the rest of kernel interfaces and are race free.
Take him up on the offer.
-chris
Quoted 3 lines · Alex Zarochentsev
Reiser4 may have a mount option for whoose who like or have to use traditional
O_DIRECTORYsemantics. There would be no /metas under non-directories, if user wants that.
Again, O_DIRECTORY was added to solve a real-world race, not just for the sake of it. If you really want to add that option I'll research the CAN number for you so you can named it after that - or just call it -o insecure directly.
Quoted 2 lines · Hans Reiser
Christoph is a bright and clever young fellow, who just hasn't had the years of study of the field yet. I wish him well, and away.;-)
I see this is as an opportunity where you can share some of your experience to Cristoph and many others and work to get the semantics into VFS.
Please make this work :)
Signature · 1 line
mjt
Quoted 11 lines · Linus Torvalds
Quoted 7 lines · Christoph Hellwig
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem. Your attributes are absoultely a VFS thing and as such should not happen at the filesystem layer, and no, that doesn't mean they're bad per se, I just think they are a rather bad fit for Linux.
Quoted 2 lines
Now this I agree with, in the sense that I think that if we want to support this, it should be supported at a VFS layer.
ACK. However, I'm still not seeing *ANYTHING* that would look like a workable scheme in presense of hardlinks. Show me how to make that deadlock- and race-free and we might very well do it in VFS.
_That_ is what's missing and it's needed no matter where it's implemented. You want hybrid objects - you want to solve that one. So far I've seen nothing workable.
Am Mittwoch, den 25.08.2004, 21:42 +0100 schrieb :
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Quoted 8 lines
Quoted 6 lines
Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.
It's the UNIX way.
Not if you allow link(2) on them.
That doesn't make sense anyway. (actually, I tried what happens and the result was an Oops ;))
It should be completely forbidden to link into a meta-directory or out of such a directory. You could think of those meta-directory as a sysfs for that inode. Of course it's not an own filesystem and that means that there need to be a lot of security precautions in the VFS layer. Where something like that belongs anyway, if done correctly.
Quoted 4 lines
And not if you design and market your
stuff as a general-purpose backdoor into kernel. Note how *EVERY* *DAMN* *OPERATION* is made possible to override by "plugins". Which is the reason for deadlocks in question, BTW.
What do you mean? If you tell that file that you want it to be compressed or encrypted or modify some attributes (like ACLs) this isn't necessarily a backdoor.
Quoted 5 lines
Don't fool yourself - that's what Hans is selling. Target market: ISV. Marketed product: a set of hooks, the wider the better, no matter how little sense it makes. The reason for doing that outside of core kernel:
bypassing any review and being able to control the product being sold (see above).
Yes, I don't think it was a good idea either. Probably someone should remove these features and make it a "normal" filesystem. The people who need it now can turn it on again and a real solution could be worked out in Linux 2.7.
I wouldn't use it on a public server anyway now because I'm not convinced some malicious guy could find a way to exploit that. What if you changed into a meta directory using ftp and some manage to break things? This might be very dangerous.
I personally think that the idea of doing something like this (I'm not speaking of the current implementation which I think is really bad) is the right way to go in the long term.
Quoted 6 lines · On Wed, 25 Aug 2004 [email protected] wrote:
Quoted 2 lines · Linus Torvalds
It's the UNIX way.
Not if you allow link(2) on them.
Heh. I don't think that's a very strong argument against being "unixy", considering how traditional unix _used_ to handle directories.
mkdir/rmdir/rename only came later. Now, obviously they did come later for a good reason, but still..
The interesting part is that thanks to the dcache, we should be perfectly able to actually _see_ circular links etc, so some of the problems with linking directories should actually be quite solvable - something that is _not_ true for a traditional UNIX VFS layer.
Of course, the dcache introduces some new problems of its own wrt directory aliasing, but I don't actually think that should be fundamental either. Treating them more as a "static mountpoint" from an FS angle and less as a traditional Unix hardlink should be doable, I'd have thought.
(Also, it's entirely possible that the filesystem may not support some of the more esoteric linking/renaming operations. For example, in a traditional xattrs setup where the xattr is linked on-disk with the file it is associated with, you simply _can't_ link it somewhere else, or rename it to any other directory. That's not a VFS layer issue, obviously, but I thought I'd bring up the point that file-as-dir cases may have limitations that normal files don't have).
Quoted 2 lines
And not if you design and market your stuff as a general-purpose
backdoor into kernel.
Now that's a separate argument, and not one I'm personally interested in arguing at least right now. I haven't actually looked at the reiser4 code, so I'm really _only_ arguing against special-case attributes.
Linus
Quoted 1 line · Hans Reiser
other filesystems in that, but Christoph is a puppy who has never
It's not Christoph who's shown more bark than bite in this thread.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 4 lines · Linus Torvalds
Of course, the dcache introduces some new problems of its own wrt directory aliasing, but I don't actually think that should be fundamental either. Treating them more as a "static mountpoint" from an FS angle and less as a traditional Unix hardlink should be doable, I'd have thought.
Yeah, if we ditch the "mountpoints are busy and untouchable" stuff. Which I'd love to, but it's a hell of a visible (and admin-visible) change.
FWIW, current deadlocks are unrelated to actual operation succeeding.
Look: we have sys_link() making sure that parent of target is a directory (PATH_LOOKUP, in a "it has ->lookup()" sense), then locking target's parent, then checking that it has ->link() (everyone on reiser4 does) and then checking that source (old link to file) is *not* a directory (in S_ISDIR sense). Then we lock source.
Note that currently it's OK - we get "all non-directories are always locked after all directories". With filesystem that provides hybrid objects with non-NULL ->link() it's not true and we are in deadlock country. Before we get anywhere near fs code.
I'm not saying that this particular instance is hard to fix, but it wasn't even looked at. All it would take is checking the description of current locking scheme and looking through the proof of correctness (present in the tree). That's the first point where said proof breaks if we have hybrids. And it's what, about 4 screenfuls of text?
I have no problems with discussing such stuff and no problems with having it merged if it actually works. But let's start with something better than "let's hope nothing breaks if we just add such objects and do nothing else, 'cause hybridi files/directories are good, mmmkay?"
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Christoph Hellwig
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Quoted 6 lines
Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.
It's the UNIX way.
I thought the UNIX way is "everything's a file", not "everything's a directory".
Quoted 3 lines
Will it potentially break something? Sure. Do we care? Me, I'll take that kind of extension _any_ day over xattrs, that are fundamentally flawed in my opinion and totally useless.
There's always the option that they're both broken.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 5 lines · Matt Mackall
Quoted 2 lines
It's the UNIX way.
I thought the UNIX way is "everything's a file", not "everything's a directory".
It really was. Directories were historically largely just files too, although with the special "lookup" operation.
Historic unix didn't have readdir/rmdir/mkdir/rename or really much _any_ special directory handling. Directories were just files, and you read them like files.
Of course, even in that early unix, "directories" were very much a reality even apart from the fact that they happened to be implemented pretty much like files. Nobody has ever claimed that the UNIX way is "everything is _one_ file", after all ;)
Quoted 5 lines
Quoted 3 lines
Will it potentially break something? Sure. Do we care? Me, I'll take that kind of extension _any_ day over xattrs, that are fundamentally flawed in my opinion and totally useless.
There's always the option that they're both broken.
Yes. Highly likely. However, something like that _does_ end up what a Windows fileserver wants. IOW, even if it's broken, _something_ is likely forced on us by that nasty thing we call "real users". Damn them.
Linus
Quoted 4 lines · Hans Reiser
I had not intended to respond to this because I have nothing positive to say, but Andrew said I needed to respond and suggested I should copy Linus.
Yes, but I didn't say "flame Christoph and ignore the issues" ;)
There are lots of little things to do with implementation, coding style, module exports, deadlocks, what code goes where, etc. These are all normal daily kernel business and we should set them aside for now and concentrate on the bigger issues.
And as I see it, there are two big issues:
a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not anticipate and
b) it does this within the context of just a single filesystem.
I see three possible responses:
a) accept the reiser4-only extensions as-is (possibly with post-review modifications, of course) or
b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or
c) reject the extensions.
My own order of preference is b) c) a). The fact that one filesystem will offer features which other filesystems do not and cannot offer makes me queasy for some reason.
b) means that at some time in the future we need to hoist the reiser4 extensions (at a conceptual level) (and probably with restrictions) up into the VFS. This will involve much thought, argument and work.
To get us started on this route it would really help me (and, probably, others) if you could describe what these API extensions are in a very simple way, without referring to incomprehehsible web pages, and without using terms which non-reiser4 people don't understand.
It would be best if each extension was addressed in a separate email thread.
We also need to discuss what a reiser4 "module" is, what its capabilities are, and what licensing implications they have.
Then, we can look at each one and say "yup, that makes sense - we want Linux to do that" and we can also think about how we would implement it at the VFS level.
If we follow the above route I believe we can make progress in a technical direction and not get deadlocked on personal/political stuff.
Now, an alternative to all the above is to just merge reiser4 as-is, after addressing all the lower-level coding issues. And see what happens. That may be a thing which Linus wishes to do. I'm easy.
Quoted 9 lines
This last sentence makes me wonder. Where is Linux heading? The idea
that a FS cannot contain features that no other FS has is very
scary.I am all for uniformity, but not at the expense of shutting down advanced progress that Linux is so badly needing.
This talk about old UNIX seems like people want to still live in the 70'ies and not look forward. Please wake up!
Just because semantics are at the VFS layer doesn't mean every filesystem has to implement them.
Quoted 17 lines
And as I see it, there are two big issues:
a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not anticipate and
b) it does this within the context of just a single filesystem.
I see three possible responses:
a) accept the reiser4-only extensions as-is (possibly with post-review modifications, of course) or
b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or
Quoted 6 lines
c) reject the extensions.
My own order of preference is b) c) a). The fact that one filesystem will offer features which other filesystems do not and cannot offer makes me queasy for some reason.
This last sentence makes me wonder. Where is Linux heading? The idea
that a FS cannot contain features that no other FS has is very
scary.
I am all for uniformity, but not at the expense of shutting down advanced progress that Linux is so badly needing.
This talk about old UNIX seems like people want to still live in the 70'ies and not look forward. Please wake up!
~S
Quoted 2 lines · Christophe Saout
It should be completely forbidden to link into a meta-directory or out of such a directory. [..]
agreed.
Quoted 1 line
Yes, I don't think it was a good idea either. Probably someone should
I personally would like plugins only if the API they use wouldn't allow them to corrupt the underlying fs, I'm not sure if this is the case with reiserfs4.
About the backwards compatibility, another option is to add a O_FILEDIR and have bash learn about it when you cd into a file. No magic with the slashes then.
Quoted 8 lines · Spam
Quoted 4 lines
My own order of preference is b) c) a). The fact that one filesystem will offer features which other filesystems do not and cannot offer makes me queasy for some reason.
This last sentence makes me wonder. Where is Linux heading? The idea
that a FS cannot contain features that no other FS has is very
scary.
That's not what Andrew said or meant.
Note the "cannot offer". As in "there is no way to offer them even if the filesystem could support it otherwise".
We have tons of filesystems that do things other filesystems cannot do. Most filesystems support writing to a file - despite the fact that some filesystems (iso9600 being an obvious one) cannot. The infrastructure is there in the VFS layer, and it becomes a _choice_ for the filesystem whether it offers certain capabilities.
So look at what Andrew said, again: his top choice would be (b). Let's see what that was again, shall we?
Quoted 3 lines
b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or
In other words, if reiserfs does something special, we should make standard interfaces for doing that special thing, so that everybody can do it without stepping on other peoples toes.
That doesn't mean that we'd _force_ everybody to do it. The same way we don't force iso9660 to write to a CD-ROM.
Linus
Quoted 28 lines · Spam
Quoted 8 lines
Quoted 4 lines
My own order of preference is b) c) a). The fact that one filesystem will offer features which other filesystems do not and cannot offer makes me queasy for some reason.
This last sentence makes me wonder. Where is Linux heading? The idea
that a FS cannot contain features that no other FS has is very
scary.That's not what Andrew said or meant.
Note the "cannot offer". As in "there is no way to offer them even if the filesystem could support it otherwise".
We have tons of filesystems that do things other filesystems cannot do. Most filesystems support writing to a file - despite the fact that some filesystems (iso9600 being an obvious one) cannot. The infrastructure is there in the VFS layer, and it becomes a _choice_ for the filesystem whether it offers certain capabilities.
So look at what Andrew said, again: his top choice would be (b). Let's see what that was again, shall we?
Quoted 7 lines
Quoted 3 lines
b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or
In other words, if reiserfs does something special, we should make standard interfaces for doing that special thing, so that everybody can do it without stepping on other peoples toes.
Agreed that would be the best. But how much time and effort will it be, and how much of the original ideas would be lost on the way to implement them in the VFS? Especially with new and very advanced FS's like Reiser4.
Isn't the line between the actual file system and the virtual one very hair thin? Where would the separation lay in Reiser4?
Quoted 4 lines
That doesn't mean that we'd _force_ everybody to do it. The same way we don't force iso9660 to write to a CD-ROM.
Linus
I got caught in the moment of flame war. My appologies.
~S
Quoted 7 lines · Spam
Quoted 4 lines
In other words, if reiserfs does something special, we should make standard interfaces for doing that special thing, so that everybody can do it without stepping on other peoples toes.
Agreed that would be the best. But how much time and effort will it be
Zero.
We can add these new features tomorrow, as reiser4-only features, with a plan in hand to generalise them later.
-->>__if__<<-- we think these are features which Linux should offer.
Quoted 5 lines · Andrew Morton
We can add these new features tomorrow, as reiser4-only features, with a plan in hand to generalise them later.
-->>__if__<<-- we think these are features which Linux should offer.
Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.
Jeremy.
Quoted 3 lines · Previously Jeremy Allison
Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.
Aside from samba, is there any other application that has a use for them?
Wichert.
Signature · 2 lines
Wichert Akkerman <[email protected]> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines
Reiser4 may have a mount option for whoose who like or have to use traditional
O_DIRECTORYsemantics. There would be no /metas under non-directories, if user wants that.Again,
O_DIRECTORYwas added to solve a real-world race, not just for the sake of it. If you really want to add that option I'll research the CAN number for you so you can named it after that - or just call it -o insecure directly.
man open(2) explains that O_DIRECTORY is used by opendir() to prevent blocking when opening pipes and certain devices*, and should only by used by opendir (of course it isn't only used by opendir, as it's a handy optimisation).
In fact O_DIRECTORY is also used by Glibc to optimise away stat() before and fstat() after calls.
An O_NODEVICE flag would be equally secure, and more generally useful.
It's important that device nodes cannot be opened when O_DIRECTORY is set. This is compatible with reiser4 file-as-directory semantics, but I don't know if reiser4 actually implements this. If it does (and it should) then there is no device blocking problem.
That leaves only the optimisation of fstat() in opendir().
But that begs the question: do we want opendir() to succeed on a reiser4 file?
It's up to us to decide if we like that semantic, or prefer a different one such as the one I described before (path search enters file-as-directory, but opendir() directly on it fails), or the Sun syscalls someone mentioned.
-- Jamie
* Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on many systems. It's best if you can avoid opening devices at all though.
Quoted 5 lines · Wichert Akkerman
Quoted 3 lines · Previously Jeremy Allison
Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.
Quoted 9 lines
Aside from samba, is there any other application that has a use for them?
Wichert.
--
Wichert Akkerman <[email protected]> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.
It's not so much about samba having a use for them as it is about other applications using them between platforms in the same way. I'm glad to see Jeremy is on top of this because it's not something I had even thought about having available on the linux platform in the sense that my work deals with data streams on NTFS, HFS, and whatever other filesystems join the venture.
NTFS had very poor support for them for a long time. At one point I was convinced it was going to be taken out eventually, but SP2 just showed with even better support for them, and more MS software has started using them. They can't turn around at this point and take them out, so they must be planning on using them to a wider extent.
Bryan
Quoted 2 lines · Previously Tierra
It's not so much about samba having a use for them as it is about other applications using them between platforms in the same way.
But what for? Personally I still haven't seen a good use for them. Apple's resource fork comes close though.
Quoted 4 lines
NTFS had very poor support for them for a long time. At one point I was convinced it was going to be taken out eventually, but SP2 just showed with even better support for them, and more MS software has started using them.
I wonder how they intend to shoehorn that into distribution across other media such as emails or zip files.
Wichert.
Signature · 2 lines
Wichert Akkerman <[email protected]> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
This message suggests a way to extend the VFS safe locking rules to include files-as-directories.
Quoted 4 lines · [email protected] wrote:
Note that currently it's OK - we get "all non-directories are always locked after all directories". With filesystem that provides hybrid objects with non-NULL ->link() it's not true and we are in deadlock country. Before we get anywhere near fs code.
Is this a problem if we treat entering a file-as-directory as crossing a mount point (i.e. like auto-mounting)?
Simply doing a path walk would lock the file and then cross the mount point to a directory.
A way to ensure that preserves the lock order is to require that the metadata is in a different filesystem to its file (i.e. not crossing a bind mount to the same filesystem).
That has the side effect of preventing hard links between metadata files and non-metadata, which in my opinion is fine.
Path walking will lock the file, and then lock the directory on a different filesystem. Lock order is still safe, provided a strict order is maintained between the two filesystems.
The strict order is ensured by preventing bind mounts which create a path cycle containing a file->metadata edge. One way to ensure that is to prevent mounts on the metadata filesystems, but the rule doesn't have to be that strict. This condition only needs to be checked in the mount() syscall.
-- Jamie
Quoted 15 lines · Linus Torvalds
Quoted 4 lines · Christoph Hellwig
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.
It's the UNIX way.
And yes, the semantics can _easily_ be solved in very unixy ways.
Quoted 8 lines
One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is:
- without the slash, a file-as-dir won't open with
O_DIRECTORY(ENOTDIR)
- with the slash, it won't open _without_O_DIRECTORY(EISDIR)Problem solved. Very user-friendly, and very intuitive.
Stupid question: who will use it? And why?
Anyone can write an userspace library, that implements function set_attribute(char *file, char *attribute, char *value), that creates directory ".attr/file" in file's directory and stores attribute there. (and you can get list of attributes from shell too:
ls echo "$filename" |sed 's/\/\([^\/]*\)$/\/\.attr\/\1/' ). There's no need to add extra functionality to kernel and filesystem.
Advantage:
- you don't add bloat to kernel or filesystem
- you don't need to teach tar/cp -a/mc about attributes
- you won't lose attributes after editing file in vim (it creates another file and renames it over original one)
Quoted 5 lines
Will it potentially break something? Sure. Do we care? Me, I'll take that kind of extension _any_ day over xattrs, that are fundamentally flawed in my opinion and totally useless. The argument that applications like "tar" won't understand the file-as-directory thing is _flawed_, since legacy apps won't understand xattrs either.
The only way xattrs are useful is that backup/restore software doesn't have to know about every filesystem with it's specific attributes and every magic ioctl for setting them. Instead it can save/restore filesystem-specific attributes without understanding what do they mean. However there's no need why application should use them. And no application does.
I can't imagine anyone shipping an application with "this app requires reiser4" prerequisite. Why should anyone use it if he can store attributes in ".attr" directory or whereever and make the application work on any OS and any filesystem?
Mikulas
Quoted 9 lines · Mikulas Patocka
Stupid question: who will use it? And why?
Anyone can write an userspace library, that implements function
set_attribute(char *file, char *attribute, char *value), that creates directory ".attr/file" in file's directory and stores attribute there. (and you can get list of attributes from shell too:
lsecho "$filename" |sed 's/\/\([^\/]*\)$/\/\.attr\/\1/'). There's no need to add extra functionality to kernel and filesystem.
..and the above is, roughly, what I understand samba etc falls back on.
The problem ends up being that the above isn't in any way safe from people moving files around (oops, where did those attributes go?) nor does it have any consistency guarantees. So it only works well if _one_ application does this, and that application follows all the locking rules.
Is it enough? It may have to be.
Quoted 6 lines
The only way xattrs are useful is that backup/restore software doesn't have to know about every filesystem with it's specific attributes and every magic ioctl for setting them. Instead it can save/restore filesystem-specific attributes without understanding what do they mean. However there's no need why application should use them. And no application does.
If no application does, then why back them up? Why implement them in the first place?
In other words - some apps obviously do want to use the. Sadly.
Linus
Quoted 2 lines · Jamie Lokier
Is this a problem if we treat entering a file-as-directory as crossing a mount point (i.e. like auto-mounting)?
Yes - mountpoints can't be e.g. unlinked. Moreover, having directory mounted on non-directory is also an interesting situation.
Quoted 2 lines
Simply doing a path walk would lock the file and then cross the mount point to a directory.
*Ugh*
What would happen if you open that directory or chdir there? If it's "underlying file stays locked" - we are in even more obvious deadlocks.
Quoted 6 lines
A way to ensure that preserves the lock order is to require that the metadata is in a different filesystem to its file (i.e. not crossing a bind mount to the same filesystem).
That has the side effect of preventing hard links between metadata files and non-metadata, which in my opinion is fine.
We don't actually need a different fs - different vfsmount will do just fine.
Quoted 5 lines
The strict order is ensured by preventing bind mounts which create a path cycle containing a
file->metadataedge. One way to ensure that is to prevent mounts on the metadata filesystems, but the rule doesn't have to be that strict. This condition only needs to be checked in themount()syscall.
You really don't want to lock mountpoint on path lookup, so I don't see how that would be relevant - it's a hell to clean up, for one thing (I've crossed ten mountpoints on the way, when do I unlock them and how do I prevent deadlocks from that?) Besides, different namespaces can have completely different mount trees, so tracking down all that stuff would be hell in its own right.
The main issue I see with all schemes in that direction (and something like that could be made workable) is the semantics of unlink() on mountpoints. *Especially* with users being able to see attributes of files they do not own (e.g. reiser4 mode/uid/gid stuff). Ability to pin down any damn file on the system and make it impossible to replace is not something you want to give to any user.
Quoted 2 lines · Wichert Akkerman
But what for? Personally I still haven't seen a good use for them. Apple's resource fork comes close though.
There's one way where it can really come in handy, and that's with specific application file types. Certain media could be saved in a better format and read more efficiently. The reason you haven't seen this done yet though is because multiple data streams isn't a standard and isn't available in most enviroments. There also isn't any file transfer protocols that will handle transfer of multiple data streams except through Windows File Sharing.
Take into account though that I have absolutely no experience or knowledge of how it all works in the backend, and don't know the implications involved in the implementation.
As for applications that already use data streams, you mentioned Apple's resource fork already, so I assume you know about file types there. IIS currently uses ADS for image thumbnailing and it's also possible to hide an alternate website right inline with a public one. I've heard of a couple other applications on Windows that uses them. But you are right, none of them _appropriately_ use data streams for any specific advantage over regular files except to keep system controlled information hidden.
Bryan
Quoted 8 lines · Wichert Akkerman
Quoted 3 lines · Previously Jeremy Allison
Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.
Aside from samba, is there any other application that has a use for them?
Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.
As to how to do it, I think the Solaris interface is reasonably decent. The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view
(An important detail for those who want to access their multiple-data-streams from non-MDS aware apps is the runat shell command, which basically does a chdir into the specified file's attribute directory and then runs a command. i.e. 'runat ~/blah ls' will list the ~/blah's attributes.)
The only real problem I have with their design is the calling them attributes and using "at" everywhere.
"Attributes", because it will get confused with the current Linux xattr implementation (which is still useful for things that actually are file attributes, like security labels, ACLs, weird attributess that FAT/NTFS/whatever have, etc.).
I don't like "at" because the API changes don't have anything to do with the actual attributes. It's a general set of changes to allow paths relative to a fd instead of the cwd, and doesn't really have anything specifically to do with attributes (with the exception of the O_XATTR flag).
Replace "at" with "rel" and O_XATTR with O_FORK or O_MULTI or something, and it's all good.
Quoted 11 lines
Quoted 6 lines
The only way xattrs are useful is that backup/restore software doesn't have to know about every filesystem with it's specific attributes and every magic ioctl for setting them. Instead it can save/restore filesystem-specific attributes without understanding what do they mean. However there's no need why application should use them. And no application does.
If no application does, then why back them up? Why implement them in the first place?
In other words - some apps obviously do want to use the. Sadly.
You can add more functionality to filesystem and use xattrs to control it. For example:
- acls
- compress file
- encrypt file (copy user's password into task_struct and use it to encrypt his files)
- preallocate file in 4MB contignuous chunks, becuase it needs real time multimedia access
- sync/append-only/immutable
etc.
However there's no need why an application should care whether the file is compressed, whether it has acls, or so. And applications don't.
And I think this is the only legitimate use for xattrs. Who else uses them except samba? I don't see how reiser4's hybrids would help.
Mikulas
Quoted 7 lines · Mikulas Patocka
Quoted 3 lines · Linus Torvalds
One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is:
Stupid question: who will use it? And why?
I've got a stupid question too. How do you back up these things ?
If your backup program reads them as a file and restores them as a file, you might lose your directory-inside-the-file magic.
If your backup program dives into the file despite stat() saying it's a file and you restore your backup, how are the "file is a file" semantics preserved ?
Obviously this is something that needs to be sorted out at the VFS layer. A filesystem specific backup and restore program isn't desirable, if only because then there'd be no way for Hans's users to switch to reiser5 in 2010 ;)
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 5 lines · [email protected] wrote:
Quoted 2 lines
Is this a problem if we treat entering a file-as-directory as crossing a mount point (i.e. like auto-mounting)?
Yes - mountpoints can't be e.g. unlinked. Moreover, having directory mounted on non-directory is also an interesting situation.
Ok, so can we make it so mountpoints can be unlinked? :)
The mount would continue to exist, but with no name, until its last user disappears.
Quoted 7 lines
Quoted 2 lines
Simply doing a path walk would lock the file and then cross the mount point to a directory.
*Ugh*
What would happen if you open that directory or chdir there? If it's "underlying file stays locked" - we are in even more obvious deadlocks.
I think the underlying file does not stay locked, and once you've entered it as a directory, it can be unlinked.
If you have the directory open or chdir into it, then it _may_ have the effect of keeping the file's storage allocated when you unlink it -- just like when a file is unlinked while opened. As that is not a user-visible property, it's a filesystem-specific implementation detail as to whether it keeps the file's data in existence while the metadata directories and/or files are open.
Quoted 15 lines
Quoted 6 lines
A way to ensure that preserves the lock order is to require that the metadata is in a different filesystem to its file (i.e. not crossing a bind mount to the same filesystem).
That has the side effect of preventing hard links between metadata files and non-metadata, which in my opinion is fine.
We don't actually need a different fs - different vfsmount will do just fine.
Quoted 5 lines
The strict order is ensured by preventing bind mounts which create a path cycle containing a
file->metadataedge. One way to ensure that is to prevent mounts on the metadata filesystems, but the rule doesn't have to be that strict. This condition only needs to be checked in themount()syscall.
Quoted 6 lines
You really don't want to lock mountpoint on path lookup, so I don't see how that would be relevant - it's a hell to clean up, for one thing (I've crossed ten mountpoints on the way, when do I unlock them and how do I prevent deadlocks from that?) Besides, different namespaces can have completely different mount trees, so tracking down all that stuff would be hell in its own right.
I didn't mean locking a chain of mountpoints, I meant the temporary state where two dentries and/or inodes are locked, parent and child, during a path walk. However I'm not very familiar with that part of the VFS and I see that the current RCU dcache might not lock that much during a path walk.
Quoted 6 lines
The main issue I see with all schemes in that direction (and something like that could be made workable) is the semantics of
unlink()on mountpoints. *Especially* with users being able to see attributes of files they do not own (e.g. reiser4 mode/uid/gid stuff). Ability to pin down any damn file on the system and make it impossible to replace is not something you want to give to any user.
I agree, users shouldn't be able to pin down a file.
I think unlink() should succeed on a file while something is visiting inside its metadata directory.
It's a filesystem quality-of-implementation feature whether that actually releases the file's data. It's a desirable feature because one user shouldn't be able to pin another user's quota'd data if they don't have permission to open the file, but if it's not implemented by a filesystem then it doesn't break anything fundamental.
It's a semantics question whether unlinking a file makes the metadata (i.e. "uid", "mode", "content-type" etc.) disappear at the same time, or if the metadata stays around until the last visitor leaves it. A filesystem might be able to keep the metadata in existence even if it deletes the file's storage on unlink(), but it would be nice for the VFS to declare which semantic is preferred.
One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way. In those cases, if you open one of the virtually generated "archive content" files, then you might expect the data to continue to exist after the underlying file is unlinked. I think that's reasonable:
being inside an archive is very similar to having it open. There is no quota pinning problem with this, because "archive content" files should inherit permission restrictions from the underlying file. If you can't read the file, then you can't read it's unpacked contents.
(reiser4 doesn't offer that last feature, but any of the myriad userspace filesystem hooks could offer it if the VFS has approprate auto-mounting file-as-directory hooks).
-- Jamie
Quoted 3 lines · Nicholas Miell
Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.
Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed.
Whereas there are some other kinds of metadata which should not be deleted if the file is modified.
-- Jamie
Quoted 5 lines · * [email protected]…
Quoted 2 lines · Jamie Lokier
Is this a problem if we treat entering a file-as-directory as crossing a mount point (i.e. like auto-mounting)?
Yes - mountpoints can't be e.g. unlinked.
Could it be essentially MNT_DETACH'd?
Quoted 4 lines · Nicholas Miell
Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.
Which is exactly how Windows apps have started to use streams.
Quoted 2 lines
As to how to do it, I think the Solaris interface is reasonably decent. The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view
I agree. This is an interface Samba can live with I think. I was thinking of implementing it anyway, just so I could piss off the Linux kernel developers by saying "oh if you need proper Windows semantics on Samba you have to use an advanced OS like Solaris, Linux doesn't cut it" :-) :-).
Quoted 2 lines
The only real problem I have with their design is the calling them attributes and using "at" everywhere.
Yep - they're different from xattrs. The easiest way to remember this is that file streams are seekable and get a fd, xattrs aren't and don't :-).
Jeremy.
Quoted 5 lines · Jamie Lokier
Quoted 3 lines · Nicholas Miell
Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.
Quoted 8 lines
Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed.
Whereas there are some other kinds of metadata which should not be deleted if the file is modified.
-- Jamie
Presumably the app which uses the metadata will be smart enough to compare the st_mtime of the MDS/stream/attribute/whatever (can we choose a name for these things now?) to the st_mtime of the file and do the right thing.
thumbnail - regenerate it
summary - keep it, it's relatively independent of the file's exact contents
signature - always verify it
etc.
Quoted 7 lines · Nicholas Miell
Quoted 7 lines
Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed.
Whereas there are some other kinds of metadata which should not be deleted if the file is modified.
Quoted 6 lines
Quoted 1 line
-- Jamie
Presumably the app which uses the metadata will be smart enough to compare the
st_mtimeof the MDS/stream/attribute/whatever (can we choose a name for these things now?) to thest_mtimeof the file and do the right thing.
[This is straying off the topic of files-as-directories].
st_mtime tests are weak. They break sometimes, and are not suitable for strong data models such as transparently caching generated data from a file's contents.
They're especially breakable if you change a file and then read it within a second. Sometimes, more than a second.
As has been raised before, nanosecond timestamps (a) don't solve this unless they're stored in the filesystem; (b) even then they will fail one day, on a sufficiently fast box; (c) don't work when there are changes to wall clock time.
They're fine where you don't care if the generated data is wrong sometimes.
If you want the _equivalent_ behaviour to opening and parsing the file, but fast, then they're no good.
Modification serial numbers would work, though.
-- Jamie
Quoted 29 lines · Jamie Lokier
Quoted 7 lines · Nicholas Miell
Quoted 7 lines
Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed.
Whereas there are some other kinds of metadata which should not be deleted if the file is modified.
Quoted 6 lines
Quoted 1 line
-- Jamie
Presumably the app which uses the metadata will be smart enough to compare the
st_mtimeof the MDS/stream/attribute/whatever (can we choose a name for these things now?) to thest_mtimeof the file and do the right thing.[This is straying off the topic of files-as-directories].
st_mtimetests are weak. They break sometimes, and are not suitable for strong data models such as transparently caching generated data from a file's contents.They're especially breakable if you change a file and then read it within a second. Sometimes, more than a second.
As has been raised before, nanosecond timestamps (a) don't solve this unless they're stored in the filesystem; (b) even then they will fail one day, on a sufficiently fast box; (c) don't work when there are changes to wall clock time.
Quoted 8 lines
They're fine where you don't care if the generated data is wrong sometimes.
If you want the _equivalent_ behaviour to opening and parsing the file, but fast, then they're no good.
Modification serial numbers would work, though.
-- Jamie
Remember that the Solaris attribute model is just another filesystem tree rooted in a hidden directory associated with a regular file. If you can come up with semantics that work in general for all files, that's fine.
inodes that delete themselves when other inodes are changed creep me out.
Quoted 8 lines · Jamie Lokier
Quoted 5 lines · [email protected] wrote:
Quoted 2 lines
Is this a problem if we treat entering a file-as-directory as crossing a mount point (i.e. like auto-mounting)?
Yes - mountpoints can't be e.g. unlinked. Moreover, having directory mounted on non-directory is also an interesting situation.
Ok, so can we make it so mountpoints can be unlinked? :)
User-visible change of behaviour and IIRC a SuS violation on top of that.
Quoted 2 lines
I think the underlying file does not stay locked, and once you've entered it as a directory, it can be unlinked.
So why lock it at all in that case?
Quoted 5 lines
I didn't mean locking a chain of mountpoints, I meant the temporary state where two dentries and/or inodes are locked, parent and child, during a path walk. However I'm not very familiar with that part of the VFS and I see that the current RCU dcache might not lock that much during a path walk.
Never had been needed on crossing mountpoints, actually.
Quoted 4 lines
I agree, users shouldn't be able to pin down a file.
I think
unlink()should succeed on a file while something is visiting inside its metadata directory.
See above. Again, the fundamental problem with that is allowing unlink and friends on a mountpoint. I would love to do that, but it always generated -EBUSY on all Unices. Linux got a bit more users and userland code than Plan 9 - they can afford such changes, but...
And yes, from the kernel POV it's trivial to do - witness the MNT_DETACH codepath in umount - it's much simpler than "normal" umount exactly because it doesn't try to emulate old "it's busy, can't umount" behaviour.
With umount we could introduce "don't bother with that shit" flag. With unlink() we would have to make that default behaviour to be useful.
Quoted 6 lines · Nicholas Miell
Quoted 5 lines · Wichert Akkerman
Quoted 3 lines · Previously Jeremy Allison
Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.
Quoted 7 lines
Quoted 3 lines
Aside from samba, is there any other application that has a use for them?
Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.
That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already.
While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE.
A file is a stream of bytes.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 21 lines · Matt Mackall
Quoted 6 lines · Nicholas Miell
Quoted 5 lines · Wichert Akkerman
Quoted 3 lines · Previously Jeremy Allison
Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.
Quoted 7 lines
Quoted 3 lines
Aside from samba, is there any other application that has a use for them?
Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.
That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already.
Quoted 6 lines
While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE.
A file is a stream of bytes.
"OMG! It breaks tar and email!!!" argument doesn't fly. Things break all the time and are fixed. It's called progress.
cp, grep, cat, and tar will continue to work just fine on files with multiple streams.
tar and cp will lose the extra streams until somebody fixes them, but they lose ACLs and xattrs right now, and I don't hear anybody suggesting that ACLs and xattrs be removed from the kernel because of this.
Fixing programs that do recursive filesystem traversals is a matter of a glibc patch to nftw(3) and a modification to their option processing.
Holding back a useful feature because you don't want to upgrade coreutils is just plain dumb.
(BTW, for email, multipart/parallel is a start, but a specific multipart content type for multi-stream file attachments would probably be more appropriate.)
Quoted 3 lines · Nicholas Miell
(BTW, for email, multipart/parallel is a start, but a specific multipart content type for multi-stream file attachments would probably be more appropriate.)
I don't understand why something like this is so important that it needs underlying VFS support. Multipart messages seem to have survived fine up until now. Where is the justification?
- James
Signature · 2 lines
James Morris <[email protected]>
Quoted 10 lines · Matt Mackall
Quoted 3 lines
Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.
That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already.
Quoted 6 lines
While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE.
A file is a stream of bytes.
I couldn't agree more. Metadata which is easily lost is a terrible place to put a "document summary".
However, it's not a bad place to put thumbnails which are generated from the file contents, if they can be regenerated after transporting. It's slightly better than a ".thumbnails" directory because the latter won't follow renames and things like that.
It's a very good place to put metadata which is semantically bound to the file in special ways not intended for transport, such as security attributes, auto-invalidated digests of the file's contents, an auto-unpacked view of an archive, a virtual tree of an XML or other structured file, or an auto-generated textual representation of a binary file.
The key concept is "not intended for transport".
Generally that means permissions metadata, and things which are re-generated from the file's contents on demand.*
It's a very bad place to put an abstract of a text document, or the names of authors, or the MIME content type, or the character encoding of the document, or the name of the shell to run the file as a script, unless these things are also deducible from the file's contents. However, sometimes the content doesn't give any clue, and then something like the character encoding of a text file is usefully stored in the metadata simply because that's better than nothing.
Of course the facility can be abused. What does Windows XP do? Does it only store the appropriate kind of metadata in the alternate streams? Even Windows XP users expect files transfered over HTTP or WebDAV to work, don't they?
-- Jamie
* - By my definition, modification time arguably shouldn't be metadata. Personally I like downloads to be stamped with their source's modification time, which is why I don't use Mozilla's download manager but cut-and-paste URLs to "wget -N" instead.
Quoted 27 lines · Nicholas Miell
Quoted 26 lines · Matt Mackall
Quoted 6 lines · Nicholas Miell
Quoted 5 lines · Wichert Akkerman
Quoted 3 lines · Previously Jeremy Allison
Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.
Quoted 7 lines
Quoted 3 lines
Aside from samba, is there any other application that has a use for them?
Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.
That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already.
While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE.
Quoted 4 lines
Quoted 1 line
A file is a stream of bytes.
"OMG! It breaks tar and email!!!" argument doesn't fly. Things break all the time and are fixed. It's called progress.
What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was.
Progress? No, this has been done before. Various dead operating systems have done it or similar and regretted it. Most recently MacOS, which jumped through major hurdles to begin purging themselves of resource forks when they went to OS X. They're still there, but heavily deprecated.
Quoted 2 lines
cp, grep, cat, and tar will continue to work just fine on files with multiple streams.
Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?
I like cat < a > b. You can keep your progress.
Signature · 1 line
Mathematics is the supreme nostalgia of our time.
Quoted 2 lines
"OMG! It breaks tar and email!!!" argument doesn't fly. Things break all the time and are fixed. It's called progress.
Yes - we break things all the time. But we take notice, when we are breaking things, of how long standing and deeply embedded they are.
The deeper the roots, the more respect we show it, and the harder it will be to change.
Heaping scorn on someone who is reluctant to change something as deeply embedded as "a file is a byte stream" does not further the discussion.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373Quoted 19 lines
Quoted 4 lines
Quoted 1 line
A file is a stream of bytes.
"OMG! It breaks tar and email!!!" argument doesn't fly. Things break all the time and are fixed. It's called progress.
What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was.
Progress? No, this has been done before. Various dead operating systems have done it or similar and regretted it. Most recently MacOS, which jumped through major hurdles to begin purging themselves of resource forks when they went to OS X. They're still there, but heavily deprecated.
Quoted 2 lines
cp, grep, cat, and tar will continue to work just fine on files with multiple streams.
Quoted 7 lines
Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?
I like cat < a > b. You can keep your progress.
cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions.
Signature · 1 line
vda
Quoted 31 lines · Rik van Riel
Quoted 6 lines · Mikulas Patocka
On Wed, 25 Aug 2004, Linus Torvalds wrote:
Quoted 8 lines
Quoted 5 lines
One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is:
Stupid question: who will use it? And why?
I've got a stupid question too. How do you back up these things ?
If your backup program reads them as a file and restores them as a file, you might lose your directory-inside-the-file magic.
If your backup program dives into the file despite
stat()saying it's a file and you restore your backup, how are the "file is a file" semantics preserved ?
Quoted 7 lines
Obviously this is something that needs to be sorted out at the VFS layer. A filesystem specific backup and restore program isn't desirable, if only because then there'd be no way for Hans's users to switch to reiser5 in 2010 ;)
Sure, this sort of thing must be sorted out at the VFS layer. And a backup program working on such a filesystem
will need to know that something can be a file, a directory - or both.
So an old "tar" won't get this right as it will assume that an object is either file or directory. The change to get it right won't be that big - just notice that an object is both, then backup the ordinary file contents as usual, before recursing into the directory it also provides and backup stuff there as usual.
The resulting .tar can of course only be unpacked properly on a fs supporting file-as-directory, similiar to how a .tar of a fs with links only will unpack properly on a fs supporting links.
I don't see much problems for userland. Old apps will keep working, as the new features is a superset. Those who care about file-as-directory extras will provide patches for "tar" and friends, after that the extras become useable.
Helge Hafting
Quoted 5 lines · Matt Mackall
What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was.
It breaks the concept of a file. In ways that offer more versatility, challenge the imagination to make even better progress and keeps Linux competing with competitors who are implementing this stuff as we speak.
I for one would truly welcome the coming of thumbnails and descriptions in picture files, because I have a real-life project going on where that would be extremely handy to have in the actual file. Were I any richer, I'd pay Namesys to have this work for me :)
Quoted 5 lines
Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?
Then I guess OS X ships a broken implementation of cp, yes?
On the cat example, what if cat < a > b simply copies the "main stream" and not the metadata, as a feature. The key being, "as a feature"
The metadata streams could get file descriptors of their own OR another program, streamcat or something, could be written to compensate.
Quoted 1 line
I like cat < a > b. You can keep your progress.
With all due respect, I hope not too many people agree with you :)
Signature · 1 line
mjt
Quoted 25 lines · Matt Mackall
Quoted 11 lines · Nicholas Miell
Quoted 8 lines · Wichert Akkerman
Quoted 5 lines · Previously Jeremy Allison
Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.
Quoted 10 lines
Quoted 5 lines
Aside from samba, is there any other application that has a use for them?
Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.
Quoted 7 lines
That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already.
Utilities that works on files only, such as cat, should keep working. No problem there. If you cat a file that also is a directory, then the file contents is all you get - by design.
Utilities that _only_ traverse the directory tree, such as find, should keep working too. Perhaps with a very minor update so they don't mistake a file-directory for a file only. I.e. find must recurse into anything that support directory semantics.
Something that both recurse and operate on files (cp -a, tar, grep, ... will need minor updating. They already knows how to handle files and directories, now they will need an update so they're open for objects that are both. I.e. let grep scan the file as usual, then recurse into its directory part in the usual way too. Performance problems should be avoided by not supporting directory operations on files with no directory content, which I believe will be many of them.
The "file-as-directory" thing will not be that useful before the tools gets these relatively simple updates. Till then it'll be a toy, which shouldn't stop it from getting into the VFS. Updating the tools will be a task for the file-as-dir fans.
(I don't know wether reiser4 does things this way - it is certainly they way I would want file-as-directory though.)
Quoted 6 lines
While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE.
Stick the file in an attachment and you get the file only. No problem, it is designed that way. An app that really wants everything in a single file should use a file structured
for that, not file-as-dir. File-as-dir attach stuff to a file in a
more loose way.
If you want to attach the directory contents too, do what you usually do when you want to mail someone a directory tree. You can't stick a directory in an attachment because it is not a file. So you either attach every file in the tree, or use tar. In this case, an updated tar.
The ftp server shouldn't be a problem. It supports both files and directories already. It may need a minor update in order to not mistake directory for file or vice-versa when someone request an operation.
ftp> get filename #Will get you the contents of the file part only - by design.
ftp> cd filename #Will change into the directory (if the file indeed provides one.)
Quoted 3 lines
A file is a stream of bytes.
Sure. And a file with a directory support both directory and file operations. You can get the stream of bytes as usual - that's the file part. Or you can cd into the directory as usual. There isn't much overlap between file operations and directory operations, so there is little conflict the way I see it. Merely letting the tools know that being a file no longer rule out the possibilities of directory operations. Getting the VFS right is another matter of course, but I don't worry about userland tools.
Helge Hafting
Matt Mackall wrote:
[...]
Quoted 18 lines
What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was.
Progress? No, this has been done before. Various dead operating systems have done it or similar and regretted it. Most recently MacOS, which jumped through major hurdles to begin purging themselves of resource forks when they went to OS X. They're still there, but heavily deprecated.
Quoted 4 lines
cp, grep, cat, and tar will continue to work just fine on files with multiple streams.
Quoted 5 lines
Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_?
It is what I'd expect. Now, use cp -R to copy the file _with its directory_, and see if that fares better. If not - bad implementation of fs and/or cp. The way I see file-as -directory is that _file_ operations (like the reads issued by cat) only work on the file part. You want the directory part? Use directory operations such as those "cp -R" use.
Quoted 4 lines
Now try cat < a > b on a file with a
fork. How is that ever going to work?
It is going to copy the _file_ part, of course. I wouldn't expect anything else - "cat" doesn't deal with directories.
This also raise the question of when to use file-as-directory. A usage where you need everything to follow the file, even when using "cat" calls out for an application that puts everything into one file. Directory-as-file is the wrong tool for that job, so don't worry about such problems.
Sticking thumbnails in a file-as-dir is another story though. Move the file (with "mv a b", not "cat a>b;rm a") moves the image file _and_ the thumbnail. No time wasted on regenerating thumbnails, no disk space or cleanup time wasted on dangling thumbs. Use the image file for its intended purposes (view it, mail it off, serve it on the web, edit it, print it) and the thumbnail doesn't get in the way because it isn't embedded in the file format. Embedding it in the file might work for jpeg which support generic embedded data, it surely won't work for every image format out there.
This is my idea of how file-as-directory should work.
Helge Hafting
Helge Hafting
Quoted 10 lines · Andrew Morton
Quoted 3 lines · Hans Reiser
I had not intended to respond to this because I have nothing positive to say, but Andrew said I needed to respond and suggested I should copy Linus.
Yes, but I didn't say "flame Christoph and ignore the issues" ;)
Oh....;-)
Quoted 6 lines
There are lots of little things to do with implementation, coding style, module exports, deadlocks, what code goes where, etc. These are all normal daily kernel business and we should set them aside for now and concentrate on the bigger issues.
Yes, you are right, but I am not sure Viro will go along with that..... ;-)
Quoted 12 lines
And as I see it, there are two big issues:
a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not anticipate and
b) it does this within the context of just a single filesystem.
I see three possible responses:
a) accept the reiser4-only extensions as-is (possibly with post-review modifications, of course) or
Quoted 9 lines
b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or
c) reject the extensions.
My own order of preference is b) c) a).
I don't object to b), though I think b) should wait for 2.7 and reiser4 should not.
Quoted 5 lines
The fact that one filesystem will
offer features which other filesystems do not and cannot offer makes me queasy for some reason.
Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem for Apple, and that means we need to put search engine and database functionality into the filesystem. It takes 11 years of serious research to build a clean storage layer able to handle doing that. Reiser4 has done that, finally. None of the other Linux filesystems have. The next major release of ReiserFS is going to be bursting with semantic enhancements, because the prerequisites for them are in place now. None of the other Linux filesystems have those prerequisites. They won't be able to keep up with the semantic enhancements. This metafiles and file-directories stuff is actually fairly trivial stuff.
Look guys, in 1993 I anticipated the battle would be here, and I build the foundation for a defensive tower right at the spot MS and Apple are now maneuvering towards. Help me get the next level on the tower before they get here. It is one hell of a foundation, they won't be able to shake it, their trees are not as powerful. Don't move reiser4 into vfs, use reiser4 as the vfs. Don't write filesystems, write file plugins and disk format plugins and all the other kinds of plugins, and you won't be missing any expressive power that you really want....
Give Saveliev and I some credit. 10 years of hard work at an ivory tower nobody thought mattered. Now the battle leaves the browser and swings our way. Don't duplicate that infrastructure, use it.
There is so much we could use help with if talented people like you chose to contribute.
Quoted 9 lines
b) means that at some time in the future we need to hoist the reiser4 extensions (at a conceptual level) (and probably with restrictions) up into the VFS. This will involve much thought, argument and work.
To get us started on this route it would really help me (and, probably, others) if you could describe what these API extensions are in a very simple way, without referring to incomprehehsible web pages,
what is not comprehensible....?
Quoted 4 lines
and without
using terms which non-reiser4 people don't understand.
Well, I agree that there is value in defining things in more detail than we have.
Quoted 21 lines
It would be best if each extension was addressed in a separate email thread.
We also need to discuss what a reiser4 "module" is, what its capabilities are, and what licensing implications they have.
Then, we can look at each one and say "yup, that makes sense - we want Linux to do that" and we can also think about how we would implement it at the VFS level.
If we follow the above route I believe we can make progress in a technical direction and not get deadlocked on personal/political stuff.
Now, an alternative to all the above is to just merge reiser4 as-is, after addressing all the lower-level coding issues. And see what happens. That may be a thing which Linus wishes to do. I'm easy.
Quoted 12 lines · Andrea Arcangeli
Quoted 4 lines · Christophe Saout
It should be completely forbidden to link into a meta-directory or out of such a directory. [..]
agreed.
Quoted 9 lines
Quoted 3 lines
Yes, I don't think it was a good idea either. Probably someone should
I personally would like plugins only if the API they use wouldn't allow them to corrupt the underlying fs, I'm not sure if this is the case with reiserfs4.
You compile crap into the kernel, you are screwed. We have not and will not change that.
Reiser4 plugins are not for end users to download from amazon.com, they are for weekend hackers to send me a cool plugin for me to review, assign a plugin id to, and send to Linus in the next release. Sometimes being less ambitious works better, and dynamically loaded plugins would have made the infrastructure too bulky to be fun and fast.
Streams are quite ugly. However, if you decompose streams into all of the little pieces that are needed to emulate them, the pieces are quite nice.
For instance, inheriting stat data from a common parent is nice, and inheritance is nice, and being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice, and, well, if you decompose all of the features of streams into little features you get a bunch of fun little features much nicer than streams.
Hans
Quoted 9 lines · Rik van Riel
If your backup program dives into the file despite
stat()saying it's a file and you restore your backup, how are the "file is a file" semantics preserved ?Obviously this is something that needs to be sorted out at the VFS layer.
It needs to be sorted out, whether it is sorted out at the VFS layer is unimportant.
Quoted 6 lines
A filesystem specific backup and restore
program isn't desirable, if only because then there'd be no way for Hans's users to switch to reiser5 in 2010 ;)
It might be that we need a filenameA/metas/backup method for all of our file plugins, which if cat'd gives a set of instructions which if executed are adequate for restoring filenameA.
Hans
Quoted 9 lines · Jamie Lokier
Quoted 5 lines · Nicholas Miell
Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.
Quoted 12 lines
Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed.
Whereas there are some other kinds of metadata which should not be deleted if the file is modified.
-- Jamie
Yes, I agree.
Actually we plan to have a whole link taxonomy, and one expected feature is that some links don't count towards the refcount needed to keep an object in existence (For instance, the links between key words and text documents, you don't want to have to explicitly unlink every keyword in order to delete a document).
Quoted 5 lines · Matthew Wilcox
That's a nice marketing talk. Get back to us when you have some technical contribution to make.
www.namesys.com/download.html
Quoted 6 lines · Christoph Hellwig
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem.
VFS is the common filesystem layer. The only reason you think semantics belong in the common filesystem layer is that you are not innovating in your semantics, and feel content with stasis.
I don't. I expect that semantics will get radically changed over the next few years as we compete with Giampaolo and whatever lesser lights are working at Microsoft.
Quoted 4 lines
Your attributes are absoultely a VFS thing and as such
should not happen at the filesystem layer, and no, that doesn't mean they're bad per se, I just think they are a rather bad fit for Linux.
Quoted 11 lines
So now go on and try to work together with the other peope doing VFS level work instead of hiding, or if you think you can't work together with us search a nice research OS where you can take over the VFS layer, if your ideas prove to be good I'm sure Linux will pick them up sooner or later.
Christoph
I tell you what, use xattrs for all the half speed filesystems, and the users and I will use metafiles.
Quoted 7 lines · Jeremy Allison
Quoted 3 lines · Hans Reiser
You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles. When I said we should just add some nice optional features to files and directories so that they can do everything that attributes can do if they are used that way, you just didn't get it. You instead went for the quick ugly hack called xattrs. You then got that ugly hack done first, because quick hacks are, well, quick. I then went about doing it the right way for Reiser4, and got DARPA to fund doing it. I was never silent about it.
Quoted 11 lines
I don't want to comment on any of the technical issues about VFS etc. as I would be completely out of my depth, however I do want to say 2 things. Firstly, this is a feature that Samba users have been needing for many years to maintain compatibility with NTFS and Windows clients. Microsoft no longer sell any servers or clients without support for multiple data streams per file, and their latest XP SP2 code *does* use this feature. Whatever the kernel issues I'm really glad that Hans and Namesys have created something we can use to match this functionality - soon we will need it in order to be able to exist in a Microsoft client-dominated world.
I agree that your work is important without agreeing that MS client domination will last.;-) It is indeed my desire to give you every single feature you need to emulate MS streams within files, but doing it using directories that are files. I would like to support you in emulating windows faster than windows.
Quoted 5 lines
My second point is the following. Hans - did you *really* have to reinvent the wheel w.r.t userspace API calls ? Did you look at this work (done in 2001 for Solaris) ?
I interviewed for the file system architect job at Sun in, I think, 1999, and they offered me the job conditional on my giving up on my Linux work. (After much trying and failing to convince them that it would be okay for me to work on Linux also, I declined the job, much to my fiscal loss and work satisfaction.)
They do not do a pure job of implementing attributes in the file namespace though. There are far more distinctions between files and attributes than are necessary that are described in these man pages below, and those distinctions cause a loss of closure. I can say more on that if asked.
Quoted 12 lines
http://bama.ua.edu/cgi-bin/man-cgi?fsattr+5
http://bama.ua.edu/cgi-bin/man-cgi?attropen+3C
http://bama.ua.edu/cgi-bin/man-cgi?openat+2I'm complaining here as someone who will have to write portable code to try and work on all these "files with streams" systems.
Jeremy.
Quoted 16 lines · Chris Mason
This starts with exactly what Christoph described in writing a short summary of how you want things to work today. Since we can't resist, we'll also go ahead and rehash all the old flame wars over this, but try to include some new ideas about where you want to see the reiser4 interfaces in 6 months as well.
-chris
Did Christoph read the www.namesys.com URL that has for several years been hiding from him? ;-)
Hans
Quoted 20 lines · Linus Torvalds
Quoted 5 lines · Christoph Hellwig
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.
It's the UNIX way.
And yes, the semantics can _easily_ be solved in very unixy ways.
One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is:
- without the slash, a file-as-dir won't open with
O_DIRECTORY(ENOTDIR)
- with the slash, it won't open _without_O_DIRECTORY(EISDIR)
Quoted 10 lines
Problem solved. Very user-friendly, and very intuitive.
Will it potentially break something? Sure. Do we care? Me, I'll take that kind of extension _any_ day over xattrs, that are fundamentally flawed in my opinion and totally useless. The argument that applications like "tar" won't understand the file-as-directory thing is _flawed_, since legacy apps won't understand xattrs either.
Oh, add a
O_NOXATTRSflag to force a path lookup to only use regular directories, the same way we haveO_NOFOLLOWand friends. That allows people to see the difference, if they care (ie a file server might decide that it doesn't want to expose things like this).
I think we should require people to care enough to supply an O_NOMETAS flag to see the difference.
Quoted 7 lines
I never liked the xattr stuff. It makes little sense, and is totally useless for 99.9999% of everything. I still don't see the point of it, except for samba. Ugly.
Linus
Quoted 12 lines · Linus Torvalds
Quoted 8 lines · Christoph Hellwig
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem. Your attributes are absoultely a VFS thing and as such should not happen at the filesystem layer, and no, that doesn't mean they're bad per se, I just think they are a rather bad fit for Linux.
Quoted 5 lines
Now this I agree with, in the sense that I think that if we want to support this, it should be supported at a VFS layer.
On the other hand, I think doing it inside the filesystem with ugly hacks
what is ugly? ;-/
Quoted 7 lines
is an acceptable way to prototype the idea before it's been proven to really be workable. Maybe it has more problems with legacy apps than we'd expect..
Linus
Quoted 11 lines · [email protected] wrote:
Quoted 8 lines · Linus Torvalds
Quoted 5 lines · Christoph Hellwig
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Quoted 9 lines
Quoted 5 lines
Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.
It's the UNIX way.
Not if you allow link(2) on them. And not if you design and market your stuff as a general-purpose backdoor into kernel.
What backdoor? Please spell it out. Plugins are not dynamically loadable.
Quoted 7 lines
Note how *EVERY* *DAMN*
*OPERATION* is made possible to override by "plugins". Which is the reason for deadlocks in question, BTW.Don't fool yourself - that's what Hans is selling. Target market: ISV.
Hunh? No, target market is hackers who like to spend a weekend dreaming up funky new kinds of files. One guy (Jason Holt, clever guy) came up with an idea for write only files for which even root cannot read the parts of the file written prior to the time root was achieved, because the encryption key is changed in a forward computable only direction with every write, and the start key is kept on another computer. Lots of folks will have plugins I would never dream of. Think of photoshop plugins, and what plugins did for photoshop. Same thing will happen to reiser4 next year (or earlier).
Quoted 11 lines
Marketed product: a set of hooks, the wider the better, no matter how little sense it makes. The reason for doing that outside of core kernel:
bypassing any review and being able to control the product being sold (see above).Shame that it got an actual filesystem mixed in with the marketing plans and general insanity...
I am one of those free software hippy-child anarchists who thinks that random people should come up with ideas and contribute them. You understand me well.;-)
Quoted 12 lines · Christoph Hellwig
Quoted 5 lines · Alex Zarochentsev
Reiser4 may have a mount option for whoose who like or have to use traditional
O_DIRECTORYsemantics. There would be no /metas under non-directories, if user wants that.Again,
O_DIRECTORYwas added to solve a real-world race, not just for the sake of it.
Can you supply more details and we will try to reply concretely? Thanks.
Quoted 8 lines
If you really want to add that option I'll research the
CAN number for you so you can named it after that - or just call it -o insecure directly.
Quoted 11 lines · Spam
Quoted 6 lines
The fact that one filesystem will
offer features which other filesystems do not and cannot offer makes me queasy for some reason.This last sentence makes me wonder. Where is Linux heading? The idea
that a FS cannot contain features that no other FS has is very
scary.
Quoted 13 lines
I am all for uniformity, but not at the expense of shutting down advanced progress that Linux is so badly needing.
This talk about old UNIX seems like people want to still live in the 70'ies and not look forward. Please wake up!
~S
well said.
Quoted 25 lines · Jamie Lokier
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines
Reiser4 may have a mount option for whoose who like or have to use traditional
O_DIRECTORYsemantics. There would be no /metas under non-directories, if user wants that.Again,
O_DIRECTORYwas added to solve a real-world race, not just for the sake of it. If you really want to add that option I'll research the CAN number for you so you can named it after that - or just call it -o insecure directly.man open(2) explains that
O_DIRECTORYis used byopendir()to prevent blocking when opening pipes and certain devices*, and should only by used by opendir (of course it isn't only used by opendir, as it's a handy optimisation).In fact
O_DIRECTORYis also used by Glibc to optimise awaystat()before andfstat()after calls.An
O_NODEVICEflag would be equally secure, and more generally useful.
Quoted 10 lines
It's important that device nodes cannot be opened when
O_DIRECTORYis set. This is compatible with reiser4 file-as-directory semantics, but I don't know if reiser4 actually implements this. If it does (and it should) then there is no device blocking problem.That leaves only the optimisation of
fstat()inopendir().But that begs the question: do we want
opendir()to succeed on a reiser4 file?
Yes, we do. Why not? Not is more complex.....
Quoted 14 lines
It's up to us to decide if we like that semantic, or prefer a different one such as the one I described before (path search enters file-as-directory, but
opendir()directly on it fails), or the Sun syscalls someone mentioned.-- Jamie
* Aside: O_NONBLOCK|O_NOCTTY is an effective way to prevent blocking on many systems. It's best if you can avoid opening devices at all though.
Quoted 6 lines · Hans Reiser
Quoted 5 lines
To get us started on this route it would really help me (and, probably, others) if you could describe what these API extensions are in a very simple way, without referring to incomprehehsible web pages,
what is not comprehensible....?
Pretty much anything at www.namesys.com. The amount of time which is needed to extract the technical info which one is looking for vastly exceeds a gnat-like attention span.
As a starting point, please prepare a bullet-point list of userspace-visible changes which the filesystem introduces, or is planned to introduce.
And describe the "plugin" system. Why does the filesystem need such a thing (other filesystems get their features via `patch -p1')?
And what are the licensing implications of plugins? Are they derived works? Must they be GPL'ed?
Quoted 6 lines · Jamie Lokier
One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way.
Yes, this was part of the plan, tar file-directory plugins would be cute.
Quoted 4 lines · Rik van Riel
On Wed, 25 Aug 2004, Hans Reiser wrote:
Quoted 8 lines
Quoted 3 lines
other filesystems in that, but Christoph is a puppy who has never
It's not Christoph who's shown more bark than bite in this thread.
The bite is at www.namesys.com/download.html
Hans
Quoted 14 lines · Christoph Hellwig
I don't think you'll get anywhere with auditing. We need to write down the semantics you want, define them at the VFS level and make sure they're not conflicting with defined userspace semantics or kernel assumptions.
I think you need to learn the basic distinction between the VFS layer and a lowlevel filesystem driver.
How old are you? I thought you were the guy at Linux Tag with fashion oriented hair who gave a talk on his XFS work? Did I confuse you with someone else?
Hans
Quoted 9 lines · Andrew Morton
Quoted 7 lines · Spam
Quoted 4 lines
In other words, if reiserfs does something special, we should make standard interfaces for doing that special thing, so that everybody can do it without stepping on other peoples toes.
Agreed that would be the best. But how much time and effort will it be
Quoted 6 lines
Zero.
We can add these new features tomorrow, as reiser4-only features, with a plan in hand to generalise them later.
-->>__if__<<-- we think these are features which Linux should offer.
Please don't forget that if the reiser4 features are merged as they are now, then we will likely be stuck with the API reiser4 chooses. There will be tools that will rely on it springing up no doubt.
Moving the reiser4 features to VFS later is fine and good, but what if the VFS doesn't want the same API for those features? Either we would have to allow reiser4 to continue providing the old API even though the VFS now provides a new, shiny API or we would have to break all existing API users on reiser4. Things like "I rebooted into the latest kernel and my computer failed to boot because essential app FOO failed to access the reiser4 API - Help!" spring to mind.
Yes, I know I am painting a rather black picture here and I know you might well say "screw apps", its been done plenty of times in Linux kernel development before...
Best regards,
Anton
Signature · 4 lines
Anton Altaparmakov <aia21 at cam.ac.uk> (replace at with @) Unix Support, Computing Service, University of Cambridge, CB2 3QH, UK Linux NTFS maintainer / IRC: #ntfs on irc.freenode.net WWW: http://linux-ntfs.sf.net/, http://www-stu.christs.cam.ac.uk/~aia21/
Quoted 2 lines · Helge
So an old "tar" won't get this right as it will assume that an object is either file or directory.
There are many backup apps, not just one. I've written a few myself, none of which will ever be worthy of notice. The sourceforge Topic.System.Archiving.Backup lists 335 projects at present.
I find the idea that most backup tools and scripts will silently stop working correctly to be pretty scary.
And then there's archiving, installation, distribution, administration, emulation, file system and partition managers, and on and on.
===
I wonder if we can make this "modal" somehow.
The one consistency I see is that apps that want the "enhanced" view need to ask for it, somehow. It is the new views of the data that are being added - let the app announce to the kernel (usually via specialized code in some shared library that the app is using to get the alternate views) that either per-task, or per-file descriptor, it is to see the "enhanced" view, as a side affect of trying to access it.
Old stuff, or even new stuff that is content to work with the "classic" view that a file is a single data stream, and that directories only have pathnames, not data, would by default see that view, and see _all_ the data, presented somehow in that view, perhaps as additional files with magic names.
This still leaves the breakage that such tools don't know, and don't preserve, the magic linkage between such magic files. But that is much less of an issue, in my view. Programs such as backups that are manipulating the files of apps they know nothing about already have to presume that all the files are important in inscrutable ways, and just be careful to preserve or copy or backup all of them.
Yeah - I realize that there will be a few followups denouncing modal architectures. I might even agree with some of them.
If this were easy, it would have been done years ago.
The onus should be on the new stuff to request the enhanced view, rather than on the old dogs to learn new tricks.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373Quoted 8 lines · Hans Reiser
Quoted 6 lines · Christoph Hellwig
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem.
Quoted 7 lines
VFS is the common filesystem layer. The only reason you think semantics belong in the common filesystem layer is that you are not innovating in your semantics, and feel content with stasis.
I don't. I expect that semantics will get radically changed over the next few years as we compete with Giampaolo and whatever lesser lights are working at Microsoft.
Hans, please stop the gooddamn personal attack bullshit.
How do you for example suggestion exporting your semantics over the network if they're not done at the VFS level? How do you want some clusterfilesystem support them or tmpfs?
Quoted 15 lines · Andrew Morton
Quoted 7 lines · Hans Reiser
Quoted 4 lines
To get us started on this route it would really help me (and, probably, others) if you could describe what these API extensions are in a very simple way, without referring to incomprehehsible web pages,
what is not comprehensible....?
Pretty much anything at www.namesys.com. The amount of time which is needed to extract the technical info which one is looking for vastly exceeds a gnat-like attention span.
Quoted 8 lines
As a starting point, please prepare a bullet-point list of userspace-visible changes which the filesystem introduces, or is planned to introduce.
And describe the "plugin" system. Why does the filesystem need such a thing (other filesystems get their features via `patch -p1')?
It takes 6 months or more to become competent to change a usual filesystem. Creating a new reiser4 plugin is a weekend programmer fun hack to do. Weekend programmers matter, because they tend to have clever ideas based on understanding a need they have. How many people can easily add new features to ext3 or reiserfs V3? Very few.
What happens if you need a disk format change?
Well, in V4, you can easily compose a plugin from plugin methods of other plugins, write a little piece of code with the one thing you want different, and add it in. Disk format changes, no big deal, add a new disk format plugin, or a new item plugin, or a new node plugin, etc., and you got your new format.
There is a huge difference between code that is designed for reuse, and code that is not. That is the difference between V3 and V4. We were looking at our V3 balancing code, and it special cased each different kind of item (an item is a piece of something, which the balancing code chops objects into as it squeezes them into nodes). It looked like the complexity of the balancing code was going to be N squared, where N was the number of different kinds of items.
So, we created item handlers, and wrote balancing code that could slice and dice and merge any item that implemented all of the operations required of an item handler. From there it grew, and we made everything pluggable.
Adding features to the new code is far less than the time cost of adding features to the old code. I've seen Nikita complain that something would take 6 weeks to do (changing key assignment algorithms), and then it takes him 3 afternoons, and it was because of the plugins, because when we did something similar in V3 it took 3 man months.
Quoted 3 lines
And what are the licensing implications of plugins? Are they derived works?
Yes.
Hans
Quoted 2 lines · Hans Reiser
Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem for Apple, and that means we need to put search engine and database
Dou you know a nice thing? We (as in the Linux Community) don't have to compete with anyone. Sure, we're usually trying to be better than anyone else, but unlike companies under maret pressure we can wait until something is ready.
Again, if you think you want your work in Linux cooperate with the Linux Developers, and as you have seen in this thread there's a pretty broad consensus that we want things working at the VFS (and actually working of course)
Quoted 2 lines · Hans Reiser
Did Christoph read the www.namesys.com URL that has for several years been hiding from him? ;-)
Yes, and I'm sick of your marketing bullshit Hans.
Quoted 5 lines · Christoph Hellwig
Dou you know a nice thing? We (as in the Linux Community) don't have to compete with anyone. Sure, we're usually trying to be better than anyone else, but unlike companies under maret pressure we can wait until something is ready.
It won't get ready unless it's made ready, yes?
And what about the time when people will move away from Linux because Linux has nothing but crappy file systems to offer?
Sure, it's not that we'd have to compete, but...
Quoted 4 lines
Again, if you think you want your work in Linux cooperate with the Linux Developers, and as you have seen in this thread there's a pretty broad consensus that we want things working at the VFS (and actually working of course)
So move it into VFS. Neext! ;)
Signature · 1 line
mjt
Quoted 9 lines
Quoted 3 lines · Previously Jeremy Allison
Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.
Aside from samba, is there any other application that has a use for them?
Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.
Also, just because we do not see all the benefits right now, doesn't mean there won't be any.
~S
Quoted 1 line
Wichert.
Quoted 3 lines · Rik van Riel
If your backup program reads them as a file and restores them as a file, you might lose your directory-inside-the-file magic.
Encode the magic in the names, by stealing a bit of the existing filename space to encode it.
Such works pretty well as part of the magic to map long filenames into DOS 8.3 names on my FAT partitions.
Apps linked with the appropriate Windows library see nice fancy long names.
The rest of the world, including DOS apps and my Unix backup scripts, see the primitive 8.3 names, including one or a few extra files per directory, which are nothing special to them.
So long as these other apps don't presume to know that they can keep some of the files in an apps directory, and drop others, then it works well enough. And no self-respecting general purpose backup program is going to presume such knowledge anyway.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373Quoted 17 lines · Nicholas Miell
Quoted 6 lines · Wichert Akkerman
Quoted 5 lines
Quoted 3 lines · Previously Jeremy Allison
Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.
Quoted 7 lines
Quoted 3 lines
Aside from samba, is there any other application that has a use for them?
Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.
Quoted 5 lines
That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already.
In Windows, the extra file streams are not lost or removed if you use a program that doesn't support them. They are only lost if you move the file to a file system that doesn't support the streams.
Even RAR support the NTFS file streams.
Quoted 6 lines
While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE.
A file is a stream of bytes.
Quoted 5 lines · Spam
Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.
All of which can be handled in userspace library code.
What compelling reason is there for doing this in the kernel?
Quoted 7 lines
cp, grep, cat, and tar will continue to work just fine on files with multiple streams.
tar and cp will lose the extra streams until somebody fixes them, but they lose ACLs and xattrs right now, and I don't hear anybody suggesting that ACLs and xattrs be removed from the kernel because of this.
I thought that the underlaying FS/VFS would make sure the extra
streams would be preserved. Normal applications that doesn't have support for them should not destroy the extra streams (and meta
data,attributes etc).
Of course tar is different, but any probgram that uses the normal OS API to copy, write and read files should not affect the extra info.
~S
Quoted 9 lines
Fixing programs that do recursive filesystem traversals is a matter of a glibc patch to nftw(3) and a modification to their option processing.
Holding back a useful feature because you don't want to upgrade coreutils is just plain dumb.
(BTW, for email, multipart/parallel is a start, but a specific multipart content type for multi-stream file attachments would probably be more appropriate.)
Quoted 8 lines
Quoted 2 lines
cp, grep, cat, and tar will continue to work just fine on files with multiple streams.
Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?
It should work on the main file, the extra streams and meta data won't be copied. Just as it is now with special attributes etc. Cat reads the content of a file stream, not the file container.
Quoted 1 line
I like cat < a > b. You can keep your progress.
Quoted 2 lines · Jamie Lokier
One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way.
Arguably this belongs in userspace --- and people have put it there.
--cw
Quoted 4 lines
Quoted 1 line · Spam
Quoted 8 lines
Quoted 4 lines
Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.
All of which can be handled in userspace library code.
What compelling reason is there for doing this in the kernel?
Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.
Applications that support the new features will benefit, all others will continue to work without destroying data.
Quoted 15 lines · Spam
Quoted 5 lines
Quoted 1 line · Spam
Quoted 8 lines
Quoted 4 lines
Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.
All of which can be handled in userspace library code.
What compelling reason is there for doing this in the kernel?
Quoted 6 lines
Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.
Applications that support the new features will benefit, all others will continue to work without destroying data.
Sorry, but that all sounds a bit fluffy. Please provide some examples.
(Generally, getting all of userspace to agree on a particular library is socially hard [*], but I don't see that as a reason for putting the functionality into the kernel)
[*] Example: where's the library to manipulate /etc/whatever.conf? [**]
[**] yes, I know about gconf.
Quoted 11 lines · Anton Altaparmakov
Please don't forget that if the reiser4 features are merged as they are now, then we will likely be stuck with the API reiser4 chooses. There will be tools that will rely on it springing up no doubt.
Moving the reiser4 features to VFS later is fine and good, but what if the VFS doesn't want the same API for those features? Either we would have to allow reiser4 to continue providing the old API even though the VFS now provides a new, shiny API or we would have to break all existing API users on reiser4. Things like "I rebooted into the latest kernel and my computer failed to boot because essential app FOO failed to access the reiser4 API - Help!" spring to mind.
Quoted 3 lines · Andrew Morton
b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or
If option b) is chosen Reiser4 can become a playground.
There is the reiser4() syscall which you surely don't want to implement for other filesystems.
Once there is some experience with this new fancy stuff the dust what is useful/insecure, etc. and what is not will settle and can be condensed into a vfs api.
Apps like samba and user scripts will have to be adapted once this is the case, but this should not be to big a problem if this stuff is marked experimental.
People which want something stable can continue to use xattrs and a magnitude of filesystems for now.
Signature · 1 line
lg, Chris
Quoted 2 lines · Paul Jackson
I find the idea that most backup tools and scripts will silently stop working correctly to be pretty scary.
Too late. We have xattrs already; many programs don't store them.
-- Jamie
Quoted 18 lines
Quoted 15 lines · Spam
Quoted 13 lines
Quoted 5 lines · Spam
Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.
All of which can be handled in userspace library code.
What compelling reason is there for doing this in the kernel?
Quoted 8 lines
Quoted 6 lines
Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.
Applications that support the new features will benefit, all others will continue to work without destroying data.
Sorry, but that all sounds a bit fluffy. Please provide some examples.
We already had the examples with cp and mv. Both should continue to work and the files will still be copied. The same with Konqueror and Nautilus. Files and their meta-files/streams/attributes will be retained as long as applications are using the OS API.
However, if things are to implemented in user-space, then old applications will not work correctly and there is risk that all the extra information will be lost or corrupted.
You said it would be socially hard. I think it would be very much close to impossible to get it right. Imagine that Gnome and Nautilus would implement support for these. I doubt that cp, mv, KDE, mc, app-xyz would implement this anytime soon and in the meantime the data is at risk.
If we want things to work from the start, including with old applications then these features should be at below-application level. If it is in VFS or FS doesn't matter to me, as long as it doesn't hinder or delay progress to long.
Quoted 7 lines
(Generally, getting all of userspace to agree on a particular library is socially hard [*], but I don't see that as a reason for putting the functionality into the kernel)
[*] Example: where's the library to manipulate
/etc/whatever.conf? [**][**] yes, I know about gconf.
Quoted 2 lines · Christian Mayrhuber
There is the
reiser4()syscall which you surely don't want to implement for other filesystems.
Actually a syscall that goes into filesystem code is the last thing we want. But it's not in the submission, so let's keep that flame^H^H^H^Hdiscussion for later.
Quoted 9 lines
Once there is some experience with this new fancy stuff the dust what is useful/insecure, etc. and what is not will settle and can be condensed into a vfs api.
Apps like samba and user scripts will have to be adapted once this is the case, but this should not be to big a problem if this stuff is marked experimental.People which want something stable can continue to use xattrs and a magnitude of filesystems for now.
Sure, no one stops you from playing around with new semantics. But please don't add them to the linux kernel stable series until we have semantics we
a) want to stick to for a while and b) actually work.
Quoted 3 lines · Hans Reiser
being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice
Yes. Being able to cd into filename.tar.gz and filename.iso is also nice, but all of these features should be supported by the VFS generically, not in any specific filesystem, and there should be a hook to invoke the various fun filesystem-independent handlers by name.
-- Jamie
Quoted 3 lines · Previously Spam
Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.
But having it in the kernel has the same problem. If you read the Solaris documentation you will see that a bunch of utilties had to get a new commandline option to be able to access the metadata and a special utility was added for other applications. If you look at windows you will see that you need to use a filename like <realfilename>:<streamname>:$DATA (which obviously does not work for single-character filenames).
Ignoring samba for a bit which just needs streams to stay compatible with windows I see few reasons for using streams:
* files are more complex these days and tend to include multiple different things: images with thumbnails and exif data, 'office' documents containing both text and images
* standard way to add common metadata to a file which can be used for searching tools (author, copyright, keywords, etc.)
But both can already be done in userland (modern image formats can store thumbnails and exif data internally, applications use tar or zip-like files for documents, etc.). The metadata part is a lot more complicated as well since the behaviour of attributes might need to be complex:
if I change an image using an application that is not stream-aware, what should happen to its thumbnail?
The only common benefits I can see are standardisation and optimization:
instead of every file format or application defining a way to specify metadata for a file you get a common API defined by the OS (but you'll still need to standardize on attribute names and formats, so plenty of room that will still not help), and instead of parsing different files or XML streams you can directly access a bit of metadata.
So far I'm not convinced that streams are worth the effort. Not that my opinion is all that relevant here, but still :)
Wichert.
Signature · 2 lines
Wichert Akkerman <[email protected]> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Quoted 37 lines · Spam
Quoted 19 lines
Quoted 15 lines · Spam
Quoted 13 lines
Quoted 5 lines · Spam
Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.
All of which can be handled in userspace library code.
What compelling reason is there for doing this in the kernel?
Quoted 8 lines
Quoted 6 lines
Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.
Applications that support the new features will benefit, all others will continue to work without destroying data.
Sorry, but that all sounds a bit fluffy. Please provide some examples.
We already had the examples with cp and mv. Both should continue to work and the files will still be copied. The same with Konqueror and Nautilus. Files and their meta-files/streams/attributes will be retained as long as applications are using the OS API.
However, if things are to implemented in user-space, then old applications will not work correctly and there is risk that all the extra information will be lost or corrupted.
Quoted 6 lines
You said it would be socially hard. I think it would be very much close to impossible to get it right. Imagine that Gnome and Nautilus would implement support for these. I doubt that cp, mv, KDE, mc, app-xyz would implement this anytime soon and in the meantime the data is at risk.
No. All of the applications which you initially identified can be implemented by putting the various bits of data into a single file and getting applications to agree on the format of that file.
For example, some image file formats already support embedded metadata, do they not?
Quoted 9 lines · Jamie Lokier
Quoted 3 lines · Hans Reiser
being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice
Yes. Being able to cd into filename.tar.gz and filename.iso is also nice, but all of these features should be supported by the VFS generically, not in any specific filesystem, and there should be a hook to invoke the various fun filesystem-independent handlers by name.
It doesn't belong into the kernel at all. If at all it belongs into a userspace filesystems, but even in that case the magic detection of which one to use is kinda hard. You absoutely don't want to hardcode file formats in the kernel.
Quoted 1 line · Previously Jeremy Allison
Which is exactly how Windows apps have started to use streams.
Interesting I can not find a way using the .NET framework, which is supposed to be The Future, to use streams.
Wichert.
Signature · 2 lines
Wichert Akkerman <[email protected]> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Quoted 6 lines · Chris Wedgwood
Quoted 2 lines · Jamie Lokier
One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way.
Arguably this belongs in userspace --- and people have put it there.
I agree that these belong in userspace, and that there's plenty* of userspace code doing a similar thing already. I don't think there's any argument over it.
However, as far as I know it's not accessible in a file-as-directory form as yet. In my opinion that is the most natural form and it would be very intuitive to use. I hope we can pick a useful semantics for them, and also provide filesystem-independent plugins with GNU Hurd-like per-user extensibility.
-- Jamie
* plenty == too much.
Gnome, KDE, Emacs and Bash all see different virtual filesystems. (All but Bash implement their own virtual filesystem extensions). That makes them much less useful than they could be.
Quoted 9 lines · Christoph Hellwig
Quoted 8 lines
Quoted 3 lines
being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice
Yes. Being able to cd into filename.tar.gz and filename.iso is also nice, but all of these features should be supported by the VFS generically, not in any specific filesystem, and there should be a hook to invoke the various fun filesystem-independent handlers by name.
Quoted 4 lines
It doesn't belong into the kernel at all. If at all it belongs into a userspace filesystems, but even in that case the magic detection of which one to use is kinda hard. You absoutely don't want to hardcode file formats in the kernel.
Oh, I agree userspace should be involved.
-- Jamie
[ shortened Cc: of reiser- and other folks since it is no longer reiser-specific - sorry, if it was too much ]
[ fixed missing attribution ]
Quoted 2 lines · Spam
Quoted 1 line · Andrew Morton
Spam <[email protected]> wrote:
[...]
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
Applications that support the new features will benefit, all others will continue to work without destroying data.
Sorry, but that all sounds a bit fluffy. Please provide some examples.
It is too fluffy.
Quoted 2 lines
We already had the examples with cp and mv. Both should continue to work and the files will still be copied. The same with Konqueror and
... after they are patched to support streams. As stated each stream has an own fd, so - from the user-space perspective - one has to open *all* streams of this file, read them and write them into the destination.
Quoted 2 lines
Nautilus. Files and their meta-files/streams/attributes will be retained as long as applications are using the OS API.
The OS-API is *one* file descriptor where you can read, write, mmap, ... on. Therefore stream selection must happen with open() since open() returns a fd (which uniquely identify a stream). Voila.
And actually this makes transition actually possible: Old apps simply ignore streams (yes, there will be a default stream, which is used id nothing else is specified) and throw them away (think of e.g. patch which opens a new file, writes into it, unlink's the old one and mv's the new one on the old place) until they are modified to support them.
Bernd
Signature · 3 lines
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and ServicesQuoted 47 lines
Quoted 37 lines · Spam
Quoted 19 lines
Quoted 15 lines · Spam
Quoted 13 lines
Quoted 5 lines · Spam
Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.
All of which can be handled in userspace library code.
What compelling reason is there for doing this in the kernel?
Quoted 8 lines
Quoted 6 lines
Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.
Applications that support the new features will benefit, all others will continue to work without destroying data.
Sorry, but that all sounds a bit fluffy. Please provide some examples.
We already had the examples with cp and mv. Both should continue to work and the files will still be copied. The same with Konqueror and Nautilus. Files and their meta-files/streams/attributes will be retained as long as applications are using the OS API.
However, if things are to implemented in user-space, then old applications will not work correctly and there is risk that all the extra information will be lost or corrupted.
Quoted 6 lines
You said it would be socially hard. I think it would be very much close to impossible to get it right. Imagine that Gnome and Nautilus would implement support for these. I doubt that cp, mv, KDE, mc, app-xyz would implement this anytime soon and in the meantime the data is at risk.
Quoted 6 lines
No. All of the applications which you initially identified can be implemented by putting the various bits of data into a single file and getting applications to agree on the format of that file.
For example, some image file formats already support embedded metadata, do they not?
Yes, JPEG, TIFF and PNG files for example. But, if you modify any of these with an application that doesn't support the extensions then you will loose them.
Also, you are thinking _very_ narrowly now. There are thousands of
file formats. Implementing the support for meta-data/ streams/ attributes in the kernel will make it possible to use this
generically for all files.
I use this in Windows quite much. I put information description fields for lots of different files. These descriptions are then searchable etc. I could use the command prompt to copy the files and the descriptions would still be there.
Secondly, do you expect file managers like Nautilus and Konqueror to support every piece of file format on the planet so they could read information directly from the documents?
~S
Quoted 2 lines · Spam
Keeping stuff in the kernel should make the new features transparent to the applications.
No, it will make just one special case, rename within the same filesystem, work. (Well, two special cases, rm will also delete the other forks).
Unless we add a new copy(2) syscall (which would be nice for other reasons) all applicatons that copy files will fail to copy the streams. So no working cp command, no nautilus nor konqueror without changes to the application. And if you have to change the applications anyway, isn't it much better to agree on a shared library in userspace that everyone uses? Which has the added bonus that it can be made to work on top of Linux, Windows, Ultrix and VMS?
/Christer
Signature · 4 lines
"Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <[email protected]> http://www.weinigel.se
Quoted 13 lines · Jamie Lokier
Quoted 9 lines
Quoted 3 lines · Hans Reiser
being able to cat dirname/pseudos/cat and get a concatenation of all of the files is nice, and being able to cat dirname/pseudos/tar and get an archive of the directory is nice
Yes. Being able to cd into filename.tar.gz and filename.iso is also nice, but all of these features should be supported by the VFS generically, not in any specific filesystem, and there should be a hook to invoke the various fun filesystem-independent handlers by name.
Quoted 4 lines
It doesn't belong into the kernel at all. If at all it belongs into a userspace filesystems, but even in that case the magic detection of which one to use is kinda hard. You absoutely don't want to hardcode file formats in the kernel.
Do you mean user-level file system as a VFS handled by user applications, or a intermediate file system layer between any application and the real file system? The latter would be good enough as it would still be transparent to the applications.
~S
Quoted 20 lines
Quoted 6 lines · Chris Wedgwood
Quoted 2 lines · Jamie Lokier
One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way.
Arguably this belongs in userspace --- and people have put it there.
I agree that these belong in userspace, and that there's plenty* of userspace code doing a similar thing already. I don't think there's any argument over it.
However, as far as I know it's not accessible in a file-as-directory form as yet. In my opinion that is the most natural form and it would be very intuitive to use. I hope we can pick a useful semantics for them, and also provide filesystem-independent plugins with GNU Hurd-like per-user extensibility.
Quoted 6 lines
-- Jamie
* plenty == too much.
Gnome, KDE, Emacs and Bash all see different virtual filesystems. (All but Bash implement their own virtual filesystem extensions). That makes them much less useful than they could be.
Exactly, and I doubt they have the ability to join together and create a common uniform standard out of it either. They have just far to different views on Linux and how they want things to be done.
Quoted 1 line
Too late. We have xattrs already; many programs don't store them.
If by your "too late" you mean we can stop worrying about any more breakage of file system utilities, because there exists an example in which some were already broken, then you are absolutely wrong.
Just because we caused some breakage doesn't give us license to cause even more.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373Quoted 4 lines · Spam
Secondly, do you expect file managers like Nautilus and Konqueror to support every piece of file format on the planet so they could read information directly from the documents?
Sure. You're proposing that we implement a single, golden multi-stream file format in the kernel.
We could just as well do that in libmultistreamfileformat.so.
But I'll grant that one cannot go adding new metadata to, say, C files this way. I don't know how useful such a thing is though.
Remember that my main point is that there's a lack of coordination in userspace. Hell, there's none. Putting it in-kernel forces that coordination, and may be the way to go, but it's pretty sad.
Quoted 4 lines · Previously Spam
Also, you are thinking _very_ narrowly now. There are thousands of
file formats. Implementing the support for meta-data/ streams/ attributes in the kernel will make it possible to use thisgenerically for all files.
And how many different formats will be used for thumbnails? How many different names to indicate a summary for a document?
Having the ability to add attributes standardizes a way to access that data, but not the format and naming of those attributes. It could be a step closer to a more flexible way of handling storage, but you have to realize that a lot (if not most) of the work still needs to be done in userspace.
Wichert.
Signature · 2 lines
Wichert Akkerman <[email protected]> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Quoted 3 lines · Hans Reiser
Reiser4 plugins are not for end users to download from amazon.com, they are for weekend hackers to send me a cool plugin for me to review, assign a plugin id to, and send to Linus in the next release. Sometimes
then what's the difference in having the plugin fixed in stone into reiserfs? That's my whole point. Get the patch from the weekend hacker, check it, send the patch to Linus to add the new feature to reiser4, just call it "feature" not plugin. That's how it works normally for everything. Many fs have many features, many of them optional. you wouldn't need to build any hook infrastructure either that way. Hooks would be needed if this wasn't open source me thinks. Or if you want people to fetch the module from amazon without your review.
Another reason I could see the modularization/hooking useful is if those feature would take lots of kernel space, but this sure isn't the case for reiserfs, infact having modules would waste _more_ ram due the half wasted page allocation for the module text. The only single reason we use modules is to avoid wasting tons of ram by loading every possible device driver on earth, it's not that we use modules because they're more flexible, if something they're more fragile. I never use modules in my test kernel just to go fast (because I self compile them). The last good thing of the modules is during development you don't need to reboot the machine to test a kernel change in a driver, but you don't need that with reiserfs since you can make the fs itself a module (just change the name of the fs, AFIK you do that all the time already).
Quoted 2 lines · Previously Andrew Morton
But I'll grant that one cannot go adding new metadata to, say, C files this way. I don't know how useful such a thing is though.
That is actually one of the few places where a bit of metadata would be very useful. Right now there is no way to indicate in what encoding a source is written: ascii, utf-8, ucs16, etc. are all possible. But a compiler or interpreter has no good way to figure that out.
Wichert.
Signature · 2 lines
Wichert Akkerman <[email protected]> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Quoted 4 lines · Previously Spam
How so? The whole idea is that the underlaying OS that handles the
copying should also know to copy everything, otherwise you can implement everything into applications and just skip the wholefilesystem part.
UNIX doesn't have a copy systemcall, applications copy the data manually.
Wichert.
Signature · 2 lines
Wichert Akkerman <[email protected]> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Quoted 7 lines
Quoted 2 lines · Spam
Keeping stuff in the kernel should make the new features transparent to the applications.
Quoted 7 lines
No, it will make just one special case, rename within the same filesystem, work. (Well, two special cases, rm will also delete the other forks).
Unless we add a new copy(2) syscall (which would be nice for other reasons) all applicatons that copy files will fail to copy the streams.
How so? The whole idea is that the underlaying OS that handles the
copying should also know to copy everything, otherwise you can implement everything into applications and just skip the whole
filesystem part.
Of course, it won't work if applications themselves are actually transferring the data and create the new files directly onto the filesystem. This just seem so wrong.
~S
Quoted 7 lines
So no working cp command, no nautilus nor konqueror without
changes to the application. And if you have to change the applications anyway, isn't it much better to agree on a shared library in userspace that everyone uses? Which has the added bonus that it can be made to work on top of Linux, Windows, Ultrix and VMS?/Christer
Quoted 4 lines · Denis Vlasenko
Quoted 1 line
I like cat < a > b. You can keep your progress.
cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions.
Losing permissions is one thing. Annoying, mostly.
However, actual losing file data during such a copy is nothing short of a disaster, IMHO.
In my opinion we shouldn't merge file-as-a-directory semantics into the kernel until we figure out how to fix the backup/restore problem and keep standard unix utilities work.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 10 lines
Quoted 4 lines · Previously Spam
How so? The whole idea is that the underlaying OS that handles the
copying should also know to copy everything, otherwise you can implement everything into applications and just skip the wholefilesystem part.
UNIX doesn't have a copy systemcall, applications copy the data manually.
Oh, this is very unfortunate and should be a bigger issue to fix.
Quoted 1 line
Wichert.
Am Donnerstag, den 26.08.2004, 13:28 +0200 schrieb Andrea Arcangeli:
Quoted 2 lines
then what's the difference in having the plugin fixed in stone into reiserfs? That's my whole point.
I think you're not exactly understanding what a reiser4 "plugin" is.
Currently plugins are not modules. It just means that there's a defined interface between the reiser4 core and the plugins. Just like you could see filesystems as "VFS plugins".
In fact, reiser4 plugins are
- users of the reiser4 core API
- some of them are implementing Linux VFS methods (thus being some sort of glue code between the Reiser4 storage tree and the Linux VFS)
Quoted 6 lines · Andrew Morton
No. All of the applications which you initially identified can be implemented by putting the various bits of data into a single file and getting applications to agree on the format of that file.
For example, some image file formats already support embedded metadata, do they not?
Same for MP3s and their metadata tags. _That_ kind of data should be part of the contents of the file: it is intended to be transported.
It is useful to auto-extract it from the file's data stream, using format-specific programs, and offer that metadata in separate files where it is easy to use e.g. by shell scripts and indexing programs, which can then work independently of file formats.
That kind of metadata should be _derived_ from the file's data stream, and ideally the extraction should be automatic, with userspace assistance. It may be useful for the filesystem to record what's extracted, for fast access and indexing.
The same applies to thumbnails. The only difference is that these may take longer to compute. But they're the same thing: something calculated from the file's data stream, which may (or may not) be stored in the filesystem so it doesn't have to be recomputed each time. It might even be worth expiring the metadata from disk if it hasn't been used for a while and can be recomputed.
These are the same as looking into archives or structured files. The only difference is that the unpacked views might not be stored on the filesystem (but they might be, if the filesystem knows how to cache views for faster future access).[*]
Those are all examples of data derived from the data stream of a flat file. They are useful applications, and all work fine with FTP, cp, cat <a >b, and email attachments, _no application changes required_.
There's another kind of metadata which isn't intended for transport. That would be things like the classic attributes: permissions, timestamps. It would also include things like author information when the file format cannot carry it.
-- Jamie
[*] - A fancier variant, which is hard without good fs support, it writing to structured files through a view. I.e. modifying files inside a .tar.gz, or (this is useful) inside an {MS,Open,Gnome,K}Office container file or *SQL database table. To do this efficiently, the file's main data stream is regenerated on demand from the view's data. As I said, it requires good, special fs support to do it efficiently (fancy caching). With appropriate userspace libraries you can make apps work with container files on all other OSes (just like .tar.gz and *Office containers at the moment) but are significantly faster to work with on Linux (no need to serialise and (de-)compress except on demand) and more visible to the standard unix scripting tools.
Quoted 3 lines · Christer Weinigel
all applicatons that copy files will fail to copy the streams. So no working cp command, no nautilus nor konqueror without changes to the application.
No true. A lot of the fancy stuff depends on metadata which is merely another view of the serialised contents in the flat file. Nautilus & Konqueror thumbnails are an example of that (I gave a few more in other mails). cp works fine with that, and so does ftp. The metadata is just recalculated when it's needed from the target file.
Quoted 4 lines
And if you have to change the applications anyway, isn't it much better to agree on a shared library in userspace that everyone uses? Which has the added bonus that it can be made to work on top of Linux, Windows, Ultrix and VMS?
Ideally yes, a shared library _or_ at least an agreed representation.
However, with filesystem support you can improve performance by avoiding unneed serialisation (write to your huge *Office presentation, read it from another program, no need to wait for the slow tar+compress stage yet it's _as if_ those were done), retain computed metadata with coherency guarantees (e.g. real time search indexes, crypto digests and such), and let the filesystem decide when best to prune computed metadata or convert representations.
All of these things can work with a combination of userspace plugins (not the same as reiser4 plugins), and filesystem support.
Without the filesystem support, you can still use the userspace plugins -- so apps would still be portable -- but you don't get the extras mentioned above.
-- Jamie
Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
I like cat < a > b. You can keep your progress.
cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions.
Losing permissions is one thing. Annoying, mostly.
Quoted 7 lines
However, actual losing file data during such a copy is nothing short of a disaster, IMHO.
In my opinion we shouldn't merge file-as-a-directory semantics into the kernel until we figure out how to fix the backup/restore problem and keep standard unix utilities work.
Well, again, what about xattrs and ACLs...?
Actually, reiser4 doesn't currently implement storage of arbitrary user data under a file.
It's just some sysfs-like information that can be retrieved, some properties can be changed. If you copy a file the worst thing that can happen right now is that you lose the information whether that file should be encrypted or compressed. You don't lose data.
In case some people are wondering, this is what you can find in the reiser4 metas:
leto:/home/chtephan/.muttrc/metas > la
insgesamt 1
dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . -rwx------ 1 chtephan users 290 12. Jan 2004 .. -r--r--r-- 1 chtephan users 0 26. Aug 13:52 bmap -rw-r--r-- 1 chtephan users 0 26. Aug 13:52 gid -r--r--r-- 1 chtephan users 0 26. Aug 13:52 items -r--r--r-- 1 chtephan users 0 26. Aug 13:52 key -r--r--r-- 1 chtephan users 0 26. Aug 13:52 locality --w------- 1 chtephan users 0 26. Aug 13:52 new -r--r--r-- 1 chtephan users 0 26. Aug 13:52 nlink -r--r--r-- 1 chtephan users 0 26. Aug 13:52 oid dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 plugin -r--r--r-- 1 chtephan users 0 26. Aug 13:52 pseudo -r--r--r-- 1 chtephan users 0 26. Aug 13:52 readdir -rw-r--r-- 1 chtephan users 0 26. Aug 13:52 rwx -r--r--r-- 1 chtephan users 0 26. Aug 13:52 size -rw-r--r-- 1 chtephan users 0 26. Aug 13:52 uid
Some of them are reiser4 specific things (items, key, locality, oid) which can only be queried anyway.
The rest is what you can also get using other VFS calls.
leto:/home/chtephan/.muttrc/metas/plugin > la
insgesamt 0
dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 . dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .. -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 compression -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 crypto -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 digest -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 dir_item -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 fibration -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 file -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 formatting -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 hash -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 perm -rwxr-xr-x 1 chtephan users 0 26. Aug 13:54 sd
Here you can change reiser4 specific things. Here you can change some properties how the file is stored in the reiser4 tree. For example you can activate compression, encryption or authentication or set some hints to optimize speed.
You can't create or remove these pseudo-files, just like in sysfs.
I don't know, but if someone wants to store user data I think you would have to implement a directory "userdata" or something where you can store it.
Not all of the information does make a lot sense inside a modified tar. Who is interested in the bmap information of the file or some other read-only information?
Quoted 4 lines · Rik van Riel
Quoted 2 lines
cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions.
Losing permissions is one thing. Annoying, mostly.
I find losing mtime annoying. That's why I don't use Mozilla's download manager (and Navigator's before it) because they presumed that mtime isn't worth copying. ncftp and wget do the right thing, but it's annoying that I can't use the fancy tools and get the metadata _I_ want transferred by them.
Quoted 2 lines
However, actual losing file data during such a copy is nothing short of a disaster, IMHO.
Note that file-as-directory doesn't imply that you can store just anything into those directories.
Is it a problem to decree that "file data MUST NOT be stored in a metadata directory; only non-essential metadata and data computed from the file data may be stored"?
Quoted 4 lines
In my opinion we shouldn't merge file-as-a-directory semantics into the kernel until we figure out how to fix the backup/restore problem and keep standard unix utilities work.
Same as xattrs, for data that should be stored.
But, unlike xattrs, while permissions and C-source-encoding should be stored, some of this metadata should _not_ be stored when making an archive and perhaps not when making a backup either.
Personally I wouldn't want an archive containing a file to save and restore recomputable metadata like cached, extracted-on-demand MP3 id tags and email headers-as-metadata. I'd rather they were recalculated when needed, implying a stronger guarantee about their relationship to the data from which they're extracted - and implying that a user program might not be allowed to set them to inconsistent values.
-- Jamie
Quoted 5 lines · Christophe Saout
I think you're not exactly understanding what a reiser4 "plugin" is.
Currently plugins are not modules. It just means that there's a defined interface between the reiser4 core and the plugins. Just like you could see filesystems as "VFS plugins".
but filesystems are exactly always modules (in precompiled kernels at least).
Anyways I don't see any visible EXPORT_SYMBOL in reiser4.
Quoted 5 lines
In fact, reiser4 plugins are
- users of the reiser4 core API
- some of them are implementing Linux VFS methods (thus being some sort of glue code between the Reiser4 storage tree and the Linux VFS)
then it seems a bit misleading to call those things plugins.
As you wrote those are "users of the reiser4 core API", with plugins I always meant dynamically loadable things, like plugins for web browers (hence kernel modules in kernel space). While here apparently you can't plugin anything at runtime, it's just code that uses the reiser4 core API that can be modified with a kernel patch as usual.
Am Donnerstag, den 26.08.2004, 01:45 -0700 schrieb Andrew Morton:
Quoted 2 lines
And describe the "plugin" system. Why does the filesystem need such a thing (other filesystems get their features via `patch -p1')?
Ok, let me try. (Hans, please don't shoot me if I don't get every detail right, I'm trying to simplify and translate)
The reiser4 core is just a big and fast storage layer using a single tree. It is bound to a mount point and uses the page cache to manage its memory. The only thing it can do on its own is to flush dirty data to the disk when the VM wants to (memory pressure) or the VFS wants to (unmount, sync) or some "plugin" wants to.
Additionally it's completely atomic (with respect to crashes, no isolation like in databases), comparable to data=journal but without the overhead of using a journal for everything.
Up to this point there are no users of this "database" and it does not implement any of the VFS methods for a filesystem (except mount and unmount perhaps).
All the functionality is provided to the plugins. You can insert, remove, lookup or modify key:object pairs (with an index into the object). The object is a sequence of units. For files a unit would be 1 byte and the index would be the byte offset. For directories the unit would be 1 entry and the index would be the filename.
Now there are some plugins that define how the storage layout on the disk is (some kind of "backend" plugins).
*And* there are plugins which are users of the "reiser4 client API" and implement the actual VFS methods.
There's a UNIX directory plugin and a UNIX file plugin.
Directories, inodes and file content are just key:object pairs and the plugins know how to operate on these.
There's a new plugin in work that also implements UNIX file semantics but stores the data for that file encrypted and/or compressed.
These plugins live between the VFS and the storage layer.
Just like the filesystems live between the VFS and the block layer (Hans would say that filesystems are VFS plugins ;-)).
Quoted 2 lines
And what are the licensing implications of plugins? Are they derived works? Must they be GPL'ed?
I suppose yes, at least currently, since they can only be linked with reiser4, there's no module infrastructure.
Am Donnerstag, den 26.08.2004, 14:16 +0200 schrieb Andrea Arcangeli:
Quoted 6 lines
Quoted 3 lines
Currently plugins are not modules. It just means that there's a defined interface between the reiser4 core and the plugins. Just like you could see filesystems as "VFS plugins".
but filesystems are exactly always modules (in precompiled kernels at least).
Yes, of cousre.
Quoted 1 line
Anyways I don't see any visible
EXPORT_SYMBOLin reiser4.
Yes, there's no module infrastructure there. These "plugins" need to be linked at compile time. Some plugins need to be anyway since the the filesystem is useless without them. Others whould be optional features.
Quoted 5 lines
Quoted 4 lines
In fact, reiser4 plugins are
- users of the reiser4 core API
- some of them are implementing Linux VFS methods (thus being some sort of glue code between the Reiser4 storage tree and the Linux VFS)
Quoted 7 lines
then it seems a bit misleading to call those things plugins.
As you wrote those are "users of the reiser4 core API", with plugins I always meant dynamically loadable things, like plugins for web browers (hence kernel modules in kernel space). While here apparently you can't plugin anything at runtime, it's just code that uses the reiser4 core API that can be modified with a kernel patch as usual.
Right. The name is probably at bit misleading. But assuming the compression "plugin" whould be dynamically loadable at some later point it would be a plugin then?
Quoted 3 lines · Previously Jamie Lokier
I find losing mtime annoying. That's why I don't use Mozilla's download manager (and Navigator's before it) because they presumed that mtime isn't worth copying.
It's free software, you could fix that :)
Quoted 3 lines
Is it a problem to decree that "file data MUST NOT be stored in a metadata directory; only non-essential metadata and data computed from the file data may be stored"?
That would put us right back at the samba-and-other-apps-want-streams point in the discussion, but it certainly would not be a bad starting point. From there we could always add streams later on if the API is right.
Wichert.
Signature · 2 lines
Wichert Akkerman <[email protected]> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Quoted 64 lines · Hans Reiser
Quoted 12 lines · Andrew Morton
Quoted 5 lines · Hans Reiser
I had not intended to respond to this because I have nothing positive to say, but Andrew said I needed to respond and suggested I should copy Linus.
Yes, but I didn't say "flame Christoph and ignore the issues" ;)
Oh....;-)
Quoted 6 lines
There are lots of little things to do with implementation, coding style, module exports, deadlocks, what code goes where, etc. These are all normal daily kernel business and we should set them aside for now and concentrate on the bigger issues.
Yes, you are right, but I am not sure Viro will go along with that..... ;-)
Quoted 12 lines
And as I see it, there are two big issues:
a) reiser4 extends the Linux API in ways which POSIX/Unix/etc do not anticipate and
b) it does this within the context of just a single filesystem.
I see three possible responses:
a) accept the reiser4-only extensions as-is (possibly with post-review modifications, of course) or
Quoted 9 lines
b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or
c) reject the extensions.
My own order of preference is b) c) a).
I don't object to b), though I think b) should wait for 2.7 and reiser4 should not.
Quoted 5 lines
The fact that one filesystem will
offer features which other filesystems do not and cannot offer makes me queasy for some reason.Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem for Apple, and that means we need to put search engine and database functionality into the filesystem. It takes 11 years of serious research to build a clean storage layer able to handle doing that. Reiser4 has done that, finally. None of the other Linux filesystems have. The next major release of ReiserFS is going to be bursting with semantic enhancements, because the prerequisites for them are in place now. None of the other Linux filesystems have those prerequisites. They won't be able to keep up with the semantic enhancements. This metafiles and file-directories stuff is actually fairly trivial stuff.
Quoted 9 lines
Look guys, in 1993 I anticipated the battle would be here, and I build the foundation for a defensive tower right at the spot MS and Apple are now maneuvering towards. Help me get the next level on the tower before they get here. It is one hell of a foundation, they won't be able to shake it, their trees are not as powerful. Don't move reiser4 into vfs, use reiser4 as the vfs. Don't write filesystems, write file plugins and disk format plugins and all the other kinds of plugins, and you won't be missing any expressive power that you really want....
couldnt be more true
Quoted 48 lines
Give Saveliev and I some credit. 10 years of hard work at an ivory tower nobody thought mattered. Now the battle leaves the browser and swings our way. Don't duplicate that infrastructure, use it.
There is so much we could use help with if talented people like you chose to contribute.
Quoted 9 lines
b) means that at some time in the future we need to hoist the reiser4 extensions (at a conceptual level) (and probably with restrictions) up into the VFS. This will involve much thought, argument and work.
To get us started on this route it would really help me (and, probably, others) if you could describe what these API extensions are in a very simple way, without referring to incomprehehsible web pages,
what is not comprehensible....?
Quoted 4 lines
and without
using terms which non-reiser4 people don't understand.Well, I agree that there is value in defining things in more detail than we have.
Quoted 21 lines
It would be best if each extension was addressed in a separate email thread.
We also need to discuss what a reiser4 "module" is, what its capabilities are, and what licensing implications they have.
Then, we can look at each one and say "yup, that makes sense - we want Linux to do that" and we can also think about how we would implement it at the VFS level.
If we follow the above route I believe we can make progress in a technical direction and not get deadlocked on personal/political stuff.
Now, an alternative to all the above is to just merge reiser4 as-is, after addressing all the lower-level coding issues. And see what happens. That may be a thing which Linus wishes to do. I'm easy.
Signature · 1 line
Redeeman <[email protected]>
Quoted 4 lines · Hans Reiser
Quoted 3 lines
Again,
O_DIRECTORYwas added to solve a real-world race, not just for the sake of it.Can you supply more details and we will try to reply concretely? Thanks.
It's really userland programming 101 :)
you need to fstat after open to really check that no one swtiched the path below you to a fifo or device, O_DIRECTORY gets that directly in the open call, with the additional benefit of making sure you don't get any of the sideeffects at all that would happen if you're opening a device file.
the current reiser4 semantics break that and as soon as you're having a world-writeable (e.g. /tmp) dir on it and someone is doing an opendir on it he's lost.
Quoted 15 lines · Jamie Lokier
Quoted 8 lines · Christoph Hellwig
Quoted 3 lines
Reiser4 may have a mount option for whoose who like or have to use traditional
O_DIRECTORYsemantics. There would be no /metas under non-directories, if user wants that.Again,
O_DIRECTORYwas added to solve a real-world race, not just for the sake of it. If you really want to add that option I'll research the CAN number for you so you can named it after that - or just call it -o insecure directly.man open(2) explains that
O_DIRECTORYis used byopendir()to prevent blocking when opening pipes and certain devices*, and should only by used by opendir (of course it isn't only used by opendir, as it's a handy optimisation).
Quoted 4 lines
In fact
O_DIRECTORYis also used by Glibc to optimise awaystat()before andfstat()after calls.An
O_NODEVICEflag would be equally secure, and more generally useful.
It would prevent the open device issue but you'd need another syscall. Anyways, this is an existing API and chaging it now would suddenly making everything that relies on it for that purpose insecure, so it's completely out of option.
Quoted 3 lines · Christoph Hellwig
Sure, no one stops you from playing around with new semantics. But please don't add them to the linux kernel stable series until we have semantics we
a) want to stick to for a while and b) actually work.
He's not proposing to add it to 2.4, is he?
OG.
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Christoph Hellwig
Sure, no one stops you from playing around with new semantics. But please don't add them to the linux kernel stable series until we have semantics we
a) want to stick to for a while and b) actually work.He's not proposing to add it to 2.4, is he?
We're talking about 2.6 here. 2.4 is the obsolete kernel series these days.
Quoted 3 lines · Christoph Hellwig
Quoted 1 line · Olivier Galibert
He's not proposing to add it to 2.4, is he?
We're talking about 2.6 here. 2.4 is the obsolete kernel series these days.
So run it in -mm or fork it to 2.7.
2.7 might not be a bad idea for trying such bleeding edge revolution stuff as this.
Signature · 1 line
mjt
Quoted 9 lines · Christophe Saout
The reiser4 core is just a big and fast storage layer using a single tree. It is bound to a mount point and uses the page cache to manage its memory. The only thing it can do on its own is to flush dirty data to the disk when the VM wants to (memory pressure) or the VFS wants to (unmount, sync) or some "plugin" wants to.
Additionally it's completely atomic (with respect to crashes, no isolation like in databases), comparable to data=journal but without the overhead of using a journal for everything.
Quoted 3 lines
Up to this point there are no users of this "database" and it does not implement any of the VFS methods for a filesystem (except mount and unmount perhaps).
Now that part is interesting from the how to sell it to non-Linux users POV, but not for the linux kernel.
Quoted 11 lines
All the functionality is provided to the plugins. You can insert, remove, lookup or modify key:object pairs (with an index into the object). The object is a sequence of units. For files a unit would be 1 byte and the index would be the byte offset. For directories the unit would be 1 entry and the index would be the filename.
Now there are some plugins that define how the storage layout on the disk is (some kind of "backend" plugins).
*And* there are plugins which are users of the "reiser4 client API" and implement the actual VFS methods.
Here comes the problem. All the access checking/race avoidance/loop creation avoiced, in short the posix+extension semantics are implemented in the Linux VFS layer. If you want to allow a second access method (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity inside reiser4. But this is absolutely not what we want, because we moved to this common as all previous version in the different filesystems were buggy, and in fact there are problems that are simply not solveable at the level at all (rename on directories comes to mind).
So if you want to provide an additional API you'll have to go through the VFS to get it right - ergo a plugin architecture for upper plugins is worthless.
Plugins for the disk-format level are an interesting and workable idea on the other hand.
Quoted 2 lines · Markus Törnqvist
2.7 might not be a bad idea for trying such bleeding edge revolution stuff as this.
I already suggested to Linus that 2.7 is probably the right place to experiment with that.
Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig:
Quoted 2 lines
Now that part is interesting from the how to sell it to non-Linux users POV, but not for the linux kernel.
Why? The question was what these plugins are exactly. This is the answer.
Quoted 8 lines
Quoted 2 lines
*And* there are plugins which are users of the "reiser4 client API" and implement the actual VFS methods.
Here comes the problem. All the access checking/race avoidance/loop creation avoiced, in short the posix+extension semantics are implemented in the Linux VFS layer. If you want to allow a second access method (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity inside reiser4.
Are you actually listening? If you implement a filesystem there's a place where you have to implement the Linux VFS methods. I'm talking about inode_operations and these things. This has nothing to do with doing anything outside the Linux VFS. And I'm not talking about these metas either. These really don't belong inside the filesystem.
Quoted 3 lines
So if you want to provide an additional API you'll have to go through the VFS to get it right - ergo a plugin architecture for upper plugins is worthless.
See, you didn't listen. ;-)
The reiser4 core doesn't know about inodes, directories or files. It's the glue code between the VFS and the storage layer that does. It's implemented as a plugin. This has nothing to do with semantic enhancements yet. These should be removed for now and made a 2.7 topic.
Maybe Namesys could provide them as an additional patch from their webpage for those who need it. The plugins that would benefit from this (encrypted and compressed file storage) aren't ready yet anyway.
Quoted 11 lines · Matt Mackall
What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was.
Progress? No, this has been done before. Various dead operating systems have done it or similar and regretted it. Most recently MacOS, which jumped through major hurdles to begin purging themselves of resource forks when they went to OS X. They're still there, but heavily deprecated.
Quoted 8 lines
Quoted 2 lines
cp, grep, cat, and tar will continue to work just fine on files with multiple streams.
Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?
I've been known to be silly now and then, for sure. Apple says
this about the next version of OS X, due next year:
"Tiger provides a standard, Darwin-level API for managing resource forks, filesystem metadata, security information, properties and other attributes in a consistent, cross-platform manner. For example, common UNIX utilities such as cp, tar and rsync can properly handle HFS+ resource forks."
That's marketing copy, but I think it's worth noting. It seems to
be that resource forks aren't deprecated across the board, just... changed.
Apple has found a way to make this work, at least partially. I
don't know if it's a good way, but there it is.
I'd like to be able to keep my Macintosh stuff on my Linux
fileserver, without losing the resource forks. That would require that the network fs support it of course, and whether I could use tar to back up my linux box would depend on how streams might be implemented there.
Quoted 1 line
I like cat < a > b. You can keep your progress.
Can't we have both?
Signature · 5 lines
Joseph Fannin [email protected] "Bull in pure form is rare; there is usually some contamination by data." -- William Graves Perry Jr.
Quoted 7 lines · Christophe Saout
Am Donnerstag, den 26.08.2004, 14:49 +0200 schrieb Christoph Hellwig:
Quoted 2 lines
Now that part is interesting from the how to sell it to non-Linux users POV, but not for the linux kernel.
Why? The question was what these plugins are exactly. This is the answer.
I explained it below..
Quoted 9 lines
Quoted 8 lines
Quoted 2 lines
*And* there are plugins which are users of the "reiser4 client API" and implement the actual VFS methods.
Here comes the problem. All the access checking/race avoidance/loop creation avoiced, in short the posix+extension semantics are implemented in the Linux VFS layer. If you want to allow a second access method (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity inside reiser4.
Quoted 5 lines
Are you actually listening? If you implement a filesystem there's a place where you have to implement the Linux VFS methods. I'm talking about
inode_operationsand these things. This has nothing to do with doing anything outside the Linux VFS. And I'm not talking about these metas either. These really don't belong inside the filesystem.
as I wrote in this mail this absolutely _does_ belong in the filesystem, it's a major part of it and isn't easily separatable.
Quoted 10 lines
Quoted 3 lines
So if you want to provide an additional API you'll have to go through the VFS to get it right - ergo a plugin architecture for upper plugins is worthless.
See, you didn't listen. ;-)
The reiser4 core doesn't know about inodes, directories or files. It's the glue code between the VFS and the storage layer that does. It's implemented as a plugin. This has nothing to do with semantic enhancements yet. These should be removed for now and made a 2.7 topic.
Oh yes, it is. As soon as you use different access methods on an overlapping set of objects you see exactly the problems I described.
If they don't overlap there's no point for the plugins to start with, you'll better turn the core into a library that can be used by different projects then.
Re: reiser4 plugins (was: silent semantic changes with reiser4)
Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig:
Quoted 4 lines
Quoted 2 lines
Why? The question was what these plugins are exactly. This is the answer.
I explained it below..
It's hard to talk to you if you're not even trying to understand what I'm trying to say. Andrew wanted to know what these plugins are. I tried to give an answer and then you come along to tell me that it's not relevant. Sure, Linux doesn't care about what the filesystem does internally but Andrew wanted to know that.
Quoted 8 lines
Quoted 5 lines
Are you actually listening? If you implement a filesystem there's a place where you have to implement the Linux VFS methods. I'm talking about
inode_operationsand these things. This has nothing to do with doing anything outside the Linux VFS. And I'm not talking about these metas either. These really don't belong inside the filesystem.as I wrote in this mail this absolutely _does_ belong in the filesystem, it's a major part of it and isn't easily separatable.
Huh?
Now you're completely confusing me.
First you say that that file-as-a-directory is crap then you say that it does belong into the filesystem?
I'll try again:
inode_operations, etc... are implemented as a "reiser4 plugin". This means that reiser4 is usable as filesystem, you can store files in a directory hierarchy. There's absolute nothing special here. Absolutely nothing. And yes, reiser4 plugins are invisible from the VFS. It looks like a normal filesystem because it behaves like one.
The "metas" I was referring in the second part are this file-as-a- directory thing. These don't belong in the filesystem and should therefore be removed.
The latter doesn't necessarily have anything to do with plugins. The plugins are plugins for the storage layer, not for some semantic enhancement.
Note that I changed the topic to avoid further confusion.
Re: reiser4 plugins (was: silent semantic changes with reiser4)
Quoted 5 lines · Christophe Saout
It's hard to talk to you if you're not even trying to understand what I'm trying to say. Andrew wanted to know what these plugins are. I tried to give an answer and then you come along to tell me that it's not relevant. Sure, Linux doesn't care about what the filesystem does internally but Andrew wanted to know that.
Huh? I don understand what reiser4 plugins are, and I tell you that a subset of these are _wrong_ for use in the kernel.
Quoted 9 lines
Quoted 8 lines
Quoted 5 lines
Are you actually listening? If you implement a filesystem there's a place where you have to implement the Linux VFS methods. I'm talking about
inode_operationsand these things. This has nothing to do with doing anything outside the Linux VFS. And I'm not talking about these metas either. These really don't belong inside the filesystem.as I wrote in this mail this absolutely _does_ belong in the filesystem, it's a major part of it and isn't easily separatable.
Quoted 6 lines
Huh?
Now you're completely confusing me.
First you say that that file-as-a-directory is crap then you say that it does belong into the filesystem?
I think you're talking about something different then me, I'm not talking about the magic meta files but the VFS interface in general.
This VFS interface is an integral part of ævery filesystem, and it doesn't make a whole lot to put it into a plugin. If you want your filesystem core portable it does to a certain extent make sense to abstract them out, but as someone who's worked on a few such 'portable' filesystems I can tell you that it doesn't work out as expected.
Quoted 7 lines
I'll try again:
inode_operations, etc... are implemented as a "reiser4 plugin". This means that reiser4 is usable as filesystem, you can store files in a directory hierarchy. There's absolute nothing special here. Absolutely nothing. And yes, reiser4 plugins are invisible from the VFS. It looks like a normal filesystem because it behaves like one.
Now kill the whole plugin mess and let them talk directly to another. My first mail explained to you why it doesn't make sense to have multiple such plugins to work on a common subset of data.
Quoted 3 lines
The latter doesn't necessarily have anything to do with plugins. The plugins are plugins for the storage layer, not for some semantic enhancement.
Again plugins below the pagecache (if that's what you call the "storage layer") _do_ make sense. plugins at the semantic layer don't.
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Am Donnerstag, den 26.08.2004, 15:07 +0200 schrieb Christoph Hellwig:
Quoted 7 lines
Quoted 4 lines
The reiser4 core doesn't know about inodes, directories or files. It's the glue code between the VFS and the storage layer that does. It's implemented as a plugin. This has nothing to do with semantic enhancements yet. These should be removed for now and made a 2.7 topic.
Oh yes, it is. As soon as you use different access methods on an overlapping set of objects you see exactly the problems I described.
Right. That shouldn't happen. Since an object is usally tied to exactly one plugin this can't happen I think. At least as long the programmer doesn't mess anything up.
Quoted 3 lines
If they don't overlap there's no point for the plugins to start with, you'll better turn the core into a library that can be used by different projects then.
I think that's what Hans is trying to suggest. The core is theoretically usable for anything. He made a filesystem of it.
You can see reiser4 as a library and the UNIX directory and file plugins as the filesystem. That's why they are linked in uncoditionally because it doesn't make sense otherwise. It's mainly the naming that seems to confuse people.
Quoted 5 lines · Hans Reiser
I agree that your work is important without agreeing that MS client domination will last.;-) It is indeed my desire to give you every single feature you need to emulate MS streams within files, but doing it using directories that are files. I would like to support you in emulating windows faster than windows.
1) how do you back up and restore files with streams inside ?
2) how do standard unix utilities handle them ?
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 11 lines · Markus Törnqvist
Quoted 5 lines · Christoph Hellwig
Quoted 1 line · Olivier Galibert
He's not proposing to add it to 2.4, is he?
We're talking about 2.6 here. 2.4 is the obsolete kernel series these days.
So run it in -mm or fork it to 2.7.
2.7 might not be a bad idea for trying such bleeding edge revolution stuff as this.
If I remember right, there are now some reasons for a 2.7:
1) udev / dynamic device numbers
2) enhanced filesystem semantics
maybe resulting in
3) visible automounter mountpoints
Signature · 1 line
lg, Chris
Re: reiser4 plugins (was: silent semantic changes with reiser4)
In reply to Christoph HellwigAm Donnerstag, den 26.08.2004, 15:24 +0200 schrieb Christoph Hellwig:
Quoted 8 lines
Quoted 2 lines
First you say that that file-as-a-directory is crap then you say that it does belong into the filesystem?
I think you're talking about something different then me, I'm not talking about the magic meta files but the VFS interface in general.
This VFS interface is an integral part of very filesystem, and it doesn't make a whole lot to put it into a plugin.
Right. That's why these plugins are linked in uncoditionally. It doesn't work without them. Hence "plugins" is not a very good name.
Quoted 4 lines
If you want your
filesystem core portable it does to a certain extent make sense to abstract them out, but as someone who's worked on a few such 'portable' filesystems I can tell you that it doesn't work out as expected.
That's your opinion. reiser4 seems to work very well.
Quoted 1 line
Now kill the whole plugin mess and let them talk directly to another.
This is how reiserfs v3 did it. And Hans says that it actually complicated things.
Quoted 2 lines
My first mail explained to you why it doesn't make sense to have multiple such plugins to work on a common subset of data.
What I understood is that you can select exactly one plugin that e.g. handles the file data. The default plugin is optimized for normal files, another one could implement transparent compression or encryption. Some of these plugins also give the storage layer hints how to group files together to optimized performance. Neither of these things mess with the VFS.
Quoted 6 lines
Quoted 3 lines
The latter doesn't necessarily have anything to do with plugins. The plugins are plugins for the storage layer, not for some semantic enhancement.
Again plugins below the pagecache (if that's what you call the "storage layer") _do_ make sense. plugins at the semantic layer don't.
Yes, that's what I was trying to say.
Quoted 3 lines · Andrew Morton
All of which can be handled in userspace library code.
What compelling reason is there for doing this in the kernel?
There's a compelling reason to do it in userspace. If an unaware program copies or moves such a file with streams inside, it doesn't break the streams and aware programs will continue to see them.
OTOH, if we had the streams in the kernel, unaware applications would continuously break the metadata and streams that the streams aware programs expect !
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 11 lines · Hans Reiser
Quoted 6 lines · Christoph Hellwig
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem.
VFS is the common filesystem layer. The only reason you think semantics belong in the common filesystem layer is that you are not innovating in your semantics, and feel content with stasis.
This misunderstanding is the heart of the problem really. In many ways, the VFS is the semantic layer, it defines the picture of the directory tree and the locking rules the applications see.
There's no rule that says you can't innovate in the VFS, in fact it is where the semantic innovations belong. Putting them there is what allows other filesystems to implement your semantics, and this is exactly what linux needs if these new semantics are going to be standardized.
In other words, writing a reiser4 plugin is one way to get to the new semantics, but it must not be the only way. Keeping them entirely within reiser4 is good for the research stage, but for actual deployment they need to be finalized and pushed up.
If the early linux filesystems had taken the same attitude you have (don't write new filesystems, only write plugins), there would be no framework allowing the wealth of filesystems we do have, including reiser4.
-chris
Quoted 3 lines · Spam
Yes, JPEG, TIFF and PNG files for example. But, if you modify any of these with an application that doesn't support the extensions then you will loose them.
OK, so we've got a choice.
Either we will lose the extensions when modifying the file with an unaware program, or you lose the extensions when copying (or restoring from backup) using an unaware program.
Personally I'd prefer to keep the file intact when not modifying it...
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Re: reiser4 plugins (was: silent semantic changes with reiser4)
In reply to Christophe SaoutQuoted 6 lines · Christophe Saout
Quoted 4 lines
If you want your
filesystem core portable it does to a certain extent make sense to abstract them out, but as someone who's worked on a few such 'portable' filesystems I can tell you that it doesn't work out as expected.That's your opinion. reiser4 seems to work very well.
So how many OSes is it ported to currently? The problem is not that it doesn't work but that it's really hard to maintain. And remember that this maintaince overhead is not just for Hans but for everyone that does VFS-level work. Unless of course we get a blanko permission to break it as soon as fixing the mess becomes non-trivial.
Quoted 6 lines
What I understood is that you can select exactly one plugin that e.g. handles the file data. The default plugin is optimized for normal files, another one could implement transparent compression or encryption. Some of these plugins also give the storage layer hints how to group files together to optimized performance. Neither of these things mess with the VFS.
compression or encryption must sit below the pagecache to work nicely, and this hint things that usually sit at the pagecache level. But let's assume you have a valid use for different file_operations, why don't you simply add in different file_operations instead of adding another internal dispatch layer?
Quoted 9 lines · Christoph Hellwig
you need to fstat after open to really check that no one swtiched the path below you to a fifo or device,
O_DIRECTORYgets that directly in the open call, with the additional benefit of making sure you don't get any of the sideeffects at all that would happen if you're opening a device file.the current reiser4 semantics break that and as soon as you're having a world-writeable (e.g. /tmp) dir on it and someone is doing an opendir on it he's lost.
How does the current reiser4 semantics break that?
In a reiser4 filesystem, a file _is_ a directory.opendir() is supposed to succeed on it.
There's bound to be some security issue, but I'm not sure what you're getting at with /tmp. What sort of sort of security problem arises with a world-writeable directory such as /tmp, that cannot arise with the standard fs semantics?
-- Jamie
Re: reiser4 plugins (was: silent semantic changes with reiser4)
In reply to Christophe SaoutQuoted 1 line · Christophe Saout
That's your opinion. reiser4 seems to work very well.
Have you tried /bin/cp, or a backup+restore ?
What happened to your file streams ?
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
If I may chime in here...
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Christoph Hellwig
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Quoted 6 lines
Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.
It's the UNIX way.
This is an issue that directly affects work I am doing in extended cryptfs:
http://www.linuxsymposium.org/2004/view_abstract.php?content_key=55
http://halcrow.us/~mhalcrow/ols2004.pdf
http://halcrow.us/~mhalcrow/ols_cryptfs.sxi
The basic idea is that the cryptographic context for every file is correlated with the individual file via xattr's. A file is a unit of data that should, as it stands, contain all the information requisite for the encrypting filesystem layer to transparently decrypt (and encrypt, when the file is written to). This allows for a key->file granularity, as opposed to a key->block device (dm-crypt) or a key->mount point (CFS) granularity.
My grand vision is to have a policy that determines whether or not the encrypted version of the file or the decrypted version of the file is read, dependent on whether or not the file is leaving the security domain (the storage device under the control of the currently running kernel). For example, if the ``cp'' command is copying a file from a filesystem mounted from /dev/hda1 to a filesystem mounted from /dev/fd0, then the policy would indicate that (unless otherwise noted in the .cryptfsrc file in the root of the filesystem mounted from /dev/fd0, which might also contain the default security context for that filesystem or directory - like whose public keys should be used to encrypt the symmetric key for data) the file is leaving the security domain, and the encrypted contents of the file should be given to cp. Same with mutt reading an email attachment (as opposed to, say, .muttrc, where, more likely than not, the unencrypted version is wanted).
The goal is to enable an ``encrypted by default'' policy, in which files on the storage devices are independent encrypted units that remain encrypted until an application that actually needs to see the decrypted contents opens them. Then the encryption and decryption is done transparently by the fs layer, as long as the user has the right keys. Extended attributes seem like a natural way to store this context.
Once you consider that you can have a crypto context for each file, you can start doing other neat tricks, like keyed hashes over extents within the file, to allow for dynamic integrity verification during the read. If an offset of 1.5 gigabytes into a 2-gigabyte has been tampered with, then that tampering will be caught when that portion of the file is read; you don't have to verify the hash of the entire 2-gigabyte file at the time of the open. Of course, this would very rapidly overrun the available xattr storage size. And so to realistically implement something like this, some new underlying file format is in order.
In any case, the issue of userspace applications supporting extended attributes is key to the viability of this approach. If cp, uuencode, tar, or what not do not preserve the extended attributes, then the crypto context is lost, and the file is unreadable. So the $64,000 question is, just how committed is the community to this whole concept of extended attributes? From this point, should I assume that good xattr support is forthcoming, or should I abandon the idea of using xattr's for this altogether?
One solution I've been kicking around is to make cryptfs GnuPG-compatible. Not only would this eliminate the need to store some of the crypto context in the xattr set, but it would also preserve the crypto context with apps that don't know about xattr's, and it would be possible for users who are not running cryptfs to read the files with gpg. Keyed hashes over extents would be doable if GnuPG allowed for opaque data blobs in the file that gpg would just ignore when decrypting the file (gnupg-dev list had technical issues last time I tried to post these ideas to it - any gpg guys around that can comment on this?).
Quoted 3 lines
I never liked the xattr stuff. It makes little sense, and is totally useless for 99.9999% of everything. I still don't see the point of it, except for samba. Ugly.
If xattr's wind up getting supported by a certain critical mass of applications, then they are somewhat useful for me, although, as currently implemented, are insufficient for what I really need (keyed hashes over extents require too much space).
BTW, early this week I migrated cryptfs over to use David Howell's new keyring, which is working out nicely.
Mike
.___________________________________________________________________.
Michael A. Halcrow
Security Software Engineer, IBM Linux Technology Center
GnuPG Fingerprint: 05B5 08A8 713A 64C1 D35D 2371 2D3C FDDA 3EB6 601D
Quoted 2 lines · Rik van Riel
1) how do you back up and restore files with streams inside ?
With a program that supports files with streams inside.
Quoted 1 line
2) how do standard unix utilities handle them ?
Most likely they don't ;) That is, until they are fixed or replaced. I've heard of people who want xattrs to be backed up so they use star, not gnu tar, already.
Signature · 1 line
mjt
Quoted 8 lines · Spam
Quoted 4 lines
Yes, JPEG, TIFF and PNG files for example. But, if you modify any of these with an application that doesn't support the extensions then you will loose them.
Quoted 8 lines
OK, so we've got a choice.
Either we will lose the extensions when modifying the file with an unaware program, or you lose the extensions when copying (or restoring from backup) using an unaware program.
Personally I'd prefer to keep the file intact when not modifying it...
Backup to me is a special case, not simply a copy of files, but also retaining all the extra data, info, attributes, etc, that comes with the file.
Enabling support to have all of this extra stuff below the application level it will be possible to retain everything even though applications do not support them.
Backup tools, however, must know about these new features to be able to backup them. And so will every tool that directly accesses the filesystem instead of using the OS API for it.
And from I learned in an earlier message, tools like cp actually work on the fs level and themselves move the data to the new file
instead of letting the OS handle it. This seem to me as it could
be dangerous and could also prevent any kind of enhancements to the FS unless every tool like cp is patched.
Re: reiser4 plugins (was: silent semantic changes with reiser4)
In reply to Rik van RielQuoted 7 lines · Rik van Riel
Quoted 1 line · Christophe Saout
That's your opinion. reiser4 seems to work very well.
Have you tried
/bin/cp, or a backup+restore ?What happened to your file streams ?
Shrug, backup programs can be fixed. The fact that we couldn't backup 16GB files before O_LARGEFILE wasn't a reason not to do it.
IMHO it's more important to ask the question 'is this important for linux' and then we can figure out how to make it work.
-chris
Re: reiser4 plugins (was: silent semantic changes with reiser4)
In reply to Christoph HellwigAm Donnerstag, den 26.08.2004, 15:40 +0200 schrieb Christoph Hellwig:
Quoted 6 lines
Quoted 1 line
That's your opinion. reiser4 seems to work very well.
So how many OSes is it ported to currently? The problem is not that it doesn't work but that it's really hard to maintain. And remember that this maintaince overhead is not just for Hans but for everyone that does VFS-level work.
Where does it add overhead? It's just the interface between the reiser4 storage layer and the rest abstracted out.
Ok, there's the code that dispatches the VFS method calls to the plugin call but I would say that's about it. I didn't develop reiser4 so I don't know.
/* ->mkdir() VFS method in reiser4 inode_operations */ static int
reiser4_mkdir(struct inode *parent /* inode of parent
* directory */ ,
struct dentry *dentry /* dentry of new object to
* create */ ,
int mode /* new object's mode */ )
{
reiser4_object_create_data data;
reiser4_stat_inc_at(parent->i_sb, vfs_calls.mkdir);
data.mode = S_IFDIR | mode;
data.id = DIRECTORY_FILE_PLUGIN_ID;
return invoke_create_method(parent, dentry, &data);
}The invoke_create_method then just calls the create method of the plugin after setting up a context and lookup up the plugin.
xfs_iops.c seems to do something similar.
Quoted 2 lines
Unless of course we get a blanko permission to break it
as soon as fixing the mess becomes non-trivial.
Well, I think so. It's done relatively often.
Quoted 7 lines
Quoted 6 lines
What I understood is that you can select exactly one plugin that e.g. handles the file data. The default plugin is optimized for normal files, another one could implement transparent compression or encryption. Some of these plugins also give the storage layer hints how to group files together to optimized performance. Neither of these things mess with the VFS.
Quoted 5 lines
compression or encryption must sit below the pagecache to work nicely, and this hint things that usually sit at the pagecache level. But let's assume you have a valid use for different
file_operations, why don't you simply add in differentfile_operationsinstead of adding another internal dispatch layer?
I don't know, ask Hans. How could the VFS know it a filesystem wants to do something specific with a file that is completely transparent to the VFS?
Quoted 4 lines · Rik van Riel
Quoted 3 lines
All of which can be handled in userspace library code.
What compelling reason is there for doing this in the kernel?
Quoted 8 lines
There's a compelling reason to do it in userspace. If an unaware program copies or moves such a file with streams inside, it doesn't break the streams and aware programs will continue to see them.
OTOH, if we had the streams in the kernel, unaware applications would continuously break the metadata and streams that the streams aware programs expect !
You appear not to have read any of my mails on this topic.
Properly implemented metadata can:
(1) operate in both modes simultaneously;
(2) work with unaware applications;
(3) provide performance enhancements to aware applications;
(4) provide storage enhancements to both;
(5) provide useful features that work with standard unmodified unix tools,
all at once. That includes program copies.
-- Jamie
Quoted 7 lines · Christer Weinigel
Quoted 2 lines · Spam
Keeping stuff in the kernel should make the new features transparent to the applications.
No, it will make just one special case, rename within the same filesystem, work. (Well, two special cases, rm will also delete the other forks).
In traditional UNIX filesystems, copy loses some info too (owner,group,mode bits). cp and tar was taught about this and they now can preserve that info.
However, I do not think that hacking on cp, tar, etc everytime we need to add something to our filesystems (xattrs, ACLs, whatever) is a terribly attractive idea.
I think reiserfs is going to handle that by giving uniform access to such ancillary data. It will be visible as file/meta/* filelets ('small files') or something like that and can be saved/restored by e.g. tar *without* needing to know what data is stored there, and in what format.
I think Hans is not planning turning old "file is a stream of bytes" into eight-stream octopus. One stream will remain as a 'main' one, which contains actual data. Others will keep metadata, etc...
Note that even today's files aren't simple "streams of bytes" because they also have names,mode bits etc.
--
vda
Re: reiser4 plugins (was: silent semantic changes with reiser4)
In reply to Rik van RielAm Donnerstag, den 26.08.2004, 09:52 -0400 schrieb Rik van Riel:
Quoted 5 lines
Quoted 1 line
That's your opinion. reiser4 seems to work very well.
Have you tried
/bin/cp, or a backup+restore ?What happened to your file streams ?
1. I wasn't talking about reiser4 plugins, not about the file-as-a-directory thing or file streams
2. reiser4 doesn't currently have those
3. I don't really like file streams
:-)
But I'm using ACLs on a server and had to forward-port an rsync patch to make full backups using rsync possible. That's not really better.
Quoted 7 lines · Hans Reiser
Quoted 4 lines · Jamie Lokier
One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way.
Yes, this was part of the plan, tar file-directory plugins would be cute.
Silly question:
GNU Midnight Commander allows for ages to go into e.g. tar files, so I know the benefits of this. Additionally, in GNU Midnight Commander, this works no matter which file system I use (e.g. it works on iso9660), and it even works the same way on other OS's like e.g. Solaris and NetBSD.
What is the technical reason why a tar plugin should be reiser4 specific, instead of a generic VFS or userspace solution that would allow the same also on other fs like e.g. iso9660?
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 1 line · Jamie Lokier
Properly implemented metadata can:
[snip world peace]
Quoted 1 line
all at once. That includes program copies.
Show me the design for such a properly implemented thing.
None of the designs I've seen proposed so far are able to do what you claim.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 3 lines · Denis Vlasenko
I think Hans is not planning turning old "file is a stream of bytes" into eight-stream octopus. One stream will remain as a 'main' one, which contains actual data. Others will keep metadata, etc...
This is exactly what the Samba people want, though.
Office suites can store a document with embedded images and spread sheets "easily" by putting the text, the images and spread sheets all in different file streams.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Am Donnerstag, den 26.08.2004, 10:12 -0400 schrieb Rik van Riel:
Quoted 9 lines
Quoted 3 lines
I think Hans is not planning turning old "file is a stream of bytes" into eight-stream octopus. One stream will remain as a 'main' one, which contains actual data. Others will keep metadata, etc...
This is exactly what the Samba people want, though.
Office suites can store a document with embedded images and spread sheets "easily" by putting the text, the images and spread sheets all in different file streams.
Ouch.
I think Hans' idea is (I don't know if it is a good idea nor if it is doable, but at least it sounds interesting) to have special compound files where you can do something like this:
cp text.txt test.compound/test.txt
cp image.jpg test.compound/image.jpg
And if you read test.compound (the main stream) you get a special format that contains all the components. You can copy that single stream of bytes to another (reiser4) fs and then access test.compound/test.txt again.
The only thing I'm worrying about with this approach is what happens if someone tries to simultaneously open test.compound and test.compound/test.txt.
Hans has an example somewhere where he does something like that with /etc/passwd.
Am Donnerstag, den 26.08.2004, 16:05 +0200 schrieb Adrian Bunk:
Quoted 4 lines
Quoted 1 line
Yes, this was part of the plan, tar file-directory plugins would be cute.
Silly question:
Quoted 8 lines
GNU Midnight Commander allows for ages to go into e.g. tar files, so I know the benefits of this. Additionally, in GNU Midnight Commander, this works no matter which file system I use (e.g. it works on iso9660), and it even works the same way on other OS's like e.g. Solaris and NetBSD.
What is the technical reason why a tar plugin should be reiser4 specific, instead of a generic VFS or userspace solution that would allow the same also on other fs like e.g. iso9660?
I don't think that this should be done inside the kernel. tar is a perfectly valid userspace application.
What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough).
But below page-cache and thus invisible from the VFS or the applications it is extremely useful for the compression plugin. When changing some data in the middle of the file the compressed data might grow in size. reiser4 can handle this inside the storage layer and doesn't mess with the rest of the filesystem.
Quoted 11 lines · Spam
Backup to me is a special case, not simply a copy of files, but also retaining all the extra data, info, attributes, etc, that comes with the file.
Enabling support to have all of this extra stuff below the application level it will be possible to retain everything even though applications do not support them.
Backup tools, however, must know about these new features to be able to backup them. And so will every tool that directly accesses the filesystem instead of using the OS API for it.
Thus, this kind of metadata API must be made accessible through the VFS layer, if we want to make it FS-independent (not tied to Reiser4). If the real FS does not support metadata operations, just ignore them (much like in ISO9660 ignoring write operations).
Quoted 5 lines
And from I learned in an earlier message, tools like cp actually work on the fs level and themselves move the data to the new file
instead of letting the OS handle it. This seem to me as it could
be dangerous and could also prevent any kind of enhancements to the FS unless every tool like cp is patched.
I still don't understand why a copy_file() syscall is still missing.
Quoted 7 lines · Denis Vlasenko
I think Hans is not planning turning old "file is a stream of bytes" into eight-stream octopus. One stream will remain as a 'main' one, which contains actual data. Others will keep metadata, etc...
Note that even today's files aren't simple "streams of bytes" because they also have names,mode bits etc.
This is just exactly how NTFS does work: there's an unnamed stream (which is the one that the majority of userspace programs do manipulate) which contains the user data (the pixels from a PMG image, or text from a text file, etc.), and there are named streams which not all kind of applications do know how to handle.
Christoph Hellwig and Christophe Saout, I think you two are miscommunicating a little.
I've never looked at reiser4 code, but I think I understand what you are both saying, so I will explain.
Quoted 5 lines · Christoph Hellwig
Here comes the problem. All the access checking/race avoidance/loop creation avoiced, in short the posix+extension semantics are implemented in the Linux VFS layer. If you want to allow a second access method (e.g. Hans' pet syscall) you'd have to duplicate all VFS functioanlity inside reiser4.
That would be good. Even better would be if VFS implemented the best semantics that we agree is nice, after some experimenting with reiser4 lets us discover what works well.
Quoted 11 lines
Quoted 4 lines
The reiser4 core doesn't know about inodes, directories or files. It's the glue code between the VFS and the storage layer that does. It's implemented as a plugin. This has nothing to do with semantic enhancements yet. These should be removed for now and made a 2.7 topic.
Oh yes, it is. As soon as you use different access methods on an overlapping set of objects you see exactly the problems I described.
If they don't overlap there's no point for the plugins to start with, you'll better turn the core into a library that can be used by different projects then.
Firstly, the core library _is_ used by different projects. That's the commercial aim of reiser4: to be used by projects other than just as Linux filesystem. For example it might be used by a database, or a portable application which needs to store structured data in a flat file (with random access). A good abstraction makes it more useful than "just" a filesystem.
Secondly, the reiser4 design makes the "high-level plugin" interface a good clean abstraction. It's conceptually simple and versatile, so simplifies the code and makes it easier to try new ideas as well.
I think the disagreement between you two comes down to the idea that a generally useful API abstraction to file-like objects really should be somewhere in the VFS, and not specific to a single filesystem.
Unfortunately, the problem is that reiser4 is the only filesystem which is _technically capable_ of implementing that abstraction in a practical way, apparently. (I'm not sure if this is really true. reiser4's object model is not the same as paths and inodes, but the impedance mismatch doesn't seem huge.)
Perhaps in future some other filesystems could do it. Then it would make sense for the API to be common between them.
At the same time, some of the features which Hans Reiser would like to offer users -- especially things like metadata indexing -- don't seem to be very practical on top of the VFS abstraction.
These are conflicting goals. Hence the argument.
VFS is very POSIX, which rules out some interesting things like multiple views of an ordinary file (i.e. viewing a flat file and it's structured form together), coherent files (i.e. metadata which is calculated from the file contents and always guaranteed to match it), and real time indexing (i.e. "grep", "locate" and "local-google" but fancier, always up to date, and still fast).
Those features can be implemented poorly on top of POSIX, and hence poorly on top of VFS. We do that today. It works, it's simple, but it sucks. Sometimes it goes wrong (index doesn't match files), it's slow (locate isn't real-time, and if grep kept content indexes it would still be too slow to verify them), and it's not really simple as you have to use special tools for every file format these days, each with their own unique interface.
I think we can tidy that up a lot, still using the VFS, but with some extra hooks. But it's not clear what hooks those would be, and how the interesting features would be implemented by other POSIX-only filesystems. It'd be more useful: I want the fancy stuff to work over NFS to a bog standard server, or over an old ext2 filesystem that mustn't be upgraded, not just reiser4, even if I don't have the same performance and it has to create lots of articial "metadata" files to simulate the fancy stuff.
-- Jamie
Quoted 9 lines · Christophe Saout
What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough).
But below page-cache and thus invisible from the VFS or the applications it is extremely useful for the compression plugin. When changing some data in the middle of the file the compressed data might grow in size. reiser4 can handle this inside the storage layer and doesn't mess with the rest of the filesystem.
Never say never ... regular readers of -fsdevel already know what I'm going to say so can skip this post.
When compressing data on an ext2-like filesystem, you can use the property of "holes" to deallocate disc blocks in the middle of a file. It works like this:
Split the file into 64k chunks
Compress each chunk individually, then write them to disc on 64k boundaries, leaving gaps.
Now you can quickly seek to the appropriate 64k block for the data you're looking for and decompress just that 64k block.
This is how RISCiX managed to fit an 80MB distro onto a 50MB disc. Someone has written an equivalent filesystem for Linux called SquashFS.
Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain
Quoted 5 lines · Hans Reiser
Quoted 3 lines
Obviously this is something that needs to be sorted out at the VFS layer.
It needs to be sorted out, whether it is sorted out at the VFS layer is unimportant.
So what exactly is wrong with sorting it out at the VFS layer, and why do you _insist_ on sorting it in the reiserfs4 core? I'm missing something, please fill me in on the details.
Sincerely,
Lars Marowsky-Brée <[email protected]>
Signature · 3 lines
High Availability & Clustering \\\ /// SUSE Labs, Research and Development \honk/ SUSE LINUX AG - A Novell company \\//
Quoted 3 lines · Adrian Bunk
What is the technical reason why a tar plugin should be reiser4 specific, instead of a generic VFS or userspace solution that would allow the same also on other fs like e.g. iso9660?
It should be a generic VFS plugin, not reiser4 or userspace. The VFS plugin should call out to userspace for most actions (except handling cached data), and it should take advantage of special reiser4 features for storage and performance optimisations. But it should still work over a standard filesystem, when those special features aren't available. I guess FUSE and many earlier projects are heading in this direction.
A generic userspace solution doesn't let you "cd" into a tar file from all programs like you can inside Midnight Commander.
Gnome and KDE take the approach that every userspace system call should be intercepted and filtered, to create the illusion of virtual data. As a result, different programs see different virtual data: you can't just cut and paste a path from Gnome or KDE into any other program. It's not just a "social problem of libraries" thing:
sometimes I have programs which don't link to libc. Sometimes I have programs which mustn't link with anything that calls malloc(). It'd be silly for them to have a different view of the filesystem just because they can't link with some userspace library.
The Gnome/KDE/Midnight Commander pure userspace solution is silly: if _every_ program in the system should get the same view, it makes much more sense for the kernel to filter the system calls and redirect the virtual accesses to a userspace daemon, while keeping the real accesses at full speed.
Furthermore is makes much more sense for the kernel's page cache to hold those uncompressed pages, than for every userspace application to try and cooperatively manage a cache of uncompressed fragments in the most inefficient way.
There's another problem with the Midnight Commander approach. If I "cd" into a tar file, and then a program writes to the tar file, I don't always see the changes straight away. The two views aren't coherent. This isn't an easy problem to solve, but it should be solved.
When a simple "cd" into .tar.gz or .iso is implemented properly, it will have _no_ performance penalty after you have first looked in the file, so long as it remains in the on-disk cache. And, the filesystem will manage that cache intelligently.
Imagine: for looking at source files and such, you probably won't bother untarring in future, and you won't bother keeping untarred source trees in your home directory for easy access to things you look at often. Why waste the space? You could install whole applications as a .tar and run them from within it, with no performance penalty.
Similarly, the filesystem will be able to archive directories automatically that haven't been touched in a long time, with no visible change except increased storage space. "grep" will be a bit slower, but you'll have a useful search tool by then (using coherent indexes) which will be more useful than grep, and much faster.
-- Jamie
Quoted 3 lines · Christophe Saout
What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough).
That would be one of those "special features" that a VFS-plus-userspace implementation of archive views could take advantage of on reiser4, while using a slower method (sometimes much slower) on all other filesystems.
By the way, can reiser4 share parts of files between different files?
-- Jamie
Am Donnerstag, den 26.08.2004, 16:04 +0100 schrieb Jamie Lokier:
Quoted 9 lines
Quoted 3 lines · Christophe Saout
What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough).
That would be one of those "special features" that a VFS-plus-userspace implementation of archive views could take advantage of on reiser4, while using a slower method (sometimes much slower) on all other filesystems.
I'm just thinking about something. While you can't cut bytes out of unix iles a lot of filesystems can do this (holes). Most of them only on a block boundary, reiser4 on a byte boundary. If the filesystems could export this functionality using an enhanced API we could implement a compression plugin and other things on the VFS level that works with every filesystem supporting the required mechanisms, not only reiser4. And those features would take advantage of reiser4's storage mechanisms. I think Hans made the plugins reiser4-only because only reiser4 has a similar API at that time (and obviously because he didn't even think about doing it otherwise).
Quoted 1 line
By the way, can reiser4 share parts of files between different files?
At the moment a file is exactly one object. But someone could write another file plugin that spans a file across multiple objects, then yes, multiple files could share parts.
Quoted 6 lines · Olivier Galibert
Quoted 3 lines · Christoph Hellwig
Sure, no one stops you from playing around with new semantics. But please don't add them to the linux kernel stable series until we have semantics we
a) want to stick to for a while and b) actually work.He's not proposing to add it to 2.4, is he?
2.6 is a stable series...
Quoted 1 line
OG.
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedQuoted 10 lines · Rik van Riel
Quoted 3 lines · Denis Vlasenko
I think Hans is not planning turning old "file is a stream of bytes" into eight-stream octopus. One stream will remain as a 'main' one, which contains actual data. Others will keep metadata, etc...
This is exactly what the Samba people want, though.
Office suites can store a document with embedded images and spread sheets "easily" by putting the text, the images and spread sheets all in different file streams.
Kinda far reaching. It's hard to assess whether that is good or bad idea. Can we start small, like putting metadata (e.g. ACLs) into these streams? --
vda
Re: reiser4 plugins (was: silent semantic changes with reiser4)
In reply to Christophe SaoutQuoted 9 lines
Quoted 5 lines
compression or encryption must sit below the pagecache to work nicely, and this hint things that usually sit at the pagecache level. But let's assume you have a valid use for different
file_operations, why don't you simply add in differentfile_operationsinstead of adding another internal dispatch layer?I don't know, ask Hans. How could the VFS know it a filesystem wants to do something specific with a file that is completely transparent to the VFS?
The VFS shouldn't, that the whole point. That's why it allows the filesystem to register different method tables for each object.
Quoted 11 lines · Jamie Lokier
Quoted 3 lines · Christophe Saout
What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough).
That would be one of those "special features" that a VFS-plus-userspace implementation of archive views could take advantage of on reiser4, while using a slower method (sometimes much slower) on all other filesystems.
By the way, can reiser4 share parts of files between different files?
It can, but fsck is supposed to cope with this. :)
Quoted 2 lines
-- Jamie
Nikita.
Quoted 11 lines · Christophe Saout
I think Hans' idea is (I don't know if it is a good idea nor if it is doable, but at least it sounds interesting) to have special compound files where you can do something like this:
cp text.txt
test.compound/test.txt
cp image.jpg test.compound/image.jpgAnd if you read test.compound (the main stream) you get a special format that contains all the components. You can copy that single stream of bytes to another (reiser4) fs and then access
test.compound/test.txtagain.
(To Rik especially), this is the design which more or less satisfies lots of different goals at once.
Quoted 7 lines · I wrote:
Properly implemented metadata can:
(1) operate in both modes simultaneously;
(2) work with unaware applications;
(3) provide performance enhancements to aware applications;
(4) provide storage enhancements to both;
(5) provide useful features that work with standard unmodified unix tools,
(1) is by design.
You can copy these "files" safely using cp or cat or ftp. You can email them. They're flat files. That's (2).
You can poke around inside them.
Applications which do poke around inside get performance gains, because they don't have to serialise the flat format every time they "load" or "save" a file. For example, if you have a huge OpenOffice document containing lots of sub-documents and pictures. We can imagine a version of OpenOffice which just opens the sub-documents, and writes the ones which are changed after editing. Also, another program such as a viewer, post-processor or search indexing program only needs to read the parts which have changed; it doesn't have to decode the whole thing after a change. That's (3).
We already know how useful this is with big mailboxes, hence people switching to maildir. Now imagine the attachments in each message are also viewable, cat'able etc., and that the filesystem stores each message compressed automatically, or in aggregate (your choice) and that you can use both mbox and maildir tools on the same mailbox. That's (4).
You can ls, cp, grep, less, vi, gqview etc. the pieces of files, and every supported file format needs just one tool to provide the interface to its contents. That's (5).
While I'm here I might as well add:
(6) improve support for real time search indexing.
The "view coherency" mechanism which is needed for efficient userspace VFS support has just the right properties to keep informing an indexer of changes. They could even enforce operation orderings if we cared about _perfect_ real-timeness, although we probably don't.
(7) yet another way of doing distros.
Why not just install .tgz or .zip files of entire binary packages, and let the filesystem unpack the bits that are needed when they're needed. Some platforms (i.e. Java) do this already with .jar files and it's quite popular. This would extend it in a versatile way.
Rik, I hope I've explained how those 5 properties are satisfied by Hans Reiser's brilliant design (ahem, great minds think alike, and remember Linus likes files-as-directories too :). But if I haven't please do pick holes.
Quoted 3 lines
The only thing I'm worrying about with this approach is what happens if someone tries to simultaneously open test.compound and
test.compound/test.txt.
It is possible to implement it so that every change to one file immediately affects the other in the appropriate way, _as if_ the other file is regenerated every time before reading it.
This is what I mean by coherent views. It can be done even when userspace is doing the work, and the obvious mechanisms just happen to have the property of being useful in other ways (like a better dnotify, but I digress...). They're just an ordinary cache coherency protocols.
However, if you are writing a few bytes to test.compound, then the coherent view in test.compound/test.txt is probably a corrupt or non-existent file, because test.compound's intermediate state is a truncated and/or corrupt archive file, until you've written some more.
That doesn't look nice, but it's exactly the same problem as you get when you're writing to a file that another program is reading. If you do that on a live http server, for example editing a .html file, very occasionally someone will see a truncated file. Even if your editor uses renames to update atomically, sometimes you need to update more than one file atomically. It can be done, but rarely is.
Is there a solution to this? The obvious one is when you're writing to test.compound, don't read from test.compound/test.txt. That's _no different_ from the advice which says when you're saving a file from OpenOffice, don't load it into a document viewer during the time you're writing it, and don't serve it.
The other advice is to save to a different file and call rename(). Again, you get pretty much the same results as now: it atomically changes, and currently open references to the old contents will continue to reference the old contents.
There's another solution: transactions. reiser4 offers some kind of transaction facility, doesn't it? This is a fine example of when might want to declare that a sequence of writes needs to be a single transaction, visible only when the whole sequence is complete. Perfect!
I don't know reiser4's interface to that, but it would make a lot of sense to have a simple shell tool called "atomic", to wrap a sequence of operations. That would be useful for writing files _and_ useful for those pesky "update several files on the web server at once" moments. (Yes I know you can build a hard link tree and rename the root directory, although even that doesn't always work).
-- Jamie
Re: reiser4 plugins (was: silent semantic changes with reiser4)
In reply to Christoph HellwigAm Donnerstag, den 26.08.2004, 17:37 +0200 schrieb Christoph Hellwig:
Quoted 6 lines
Quoted 6 lines
Quoted 5 lines
compression or encryption must sit below the pagecache to work nicely, and this hint things that usually sit at the pagecache level. But let's assume you have a valid use for different
file_operations, why don't you simply add in differentfile_operationsinstead of adding another internal dispatch layer?
Quoted 6 lines
Quoted 3 lines
I don't know, ask Hans. How could the VFS know it a filesystem wants to do something specific with a file that is completely transparent to the VFS?
The VFS shouldn't, that the whole point. That's why it allows the filesystem to register different method tables for each object.
Only the objects it can distinguish.
Quoted 5 lines
ops->file = reiser4_file_operations;
ops->symlink = reiser4_symlink_inode_operations;
ops->special = reiser4_special_inode_operations;
ops->dentry = reiser4_dentry_operations;
ops->as = reiser4_as_operations;How could reiser4 register other operations for files that should be stored encrypted or compressed? It's all under reiser4_file_operations.
Quoted 6 lines · Jamie Lokier
However, as far as I know it's not accessible in a file-as-directory form as yet. In my opinion that is the most natural form and it would be very intuitive to use. I hope we can pick a useful semantics for them, and also provide filesystem-independent plugins with GNU Hurd-like per-user extensibility.
Quoted 6 lines
-- Jamie
* plenty == too much.
Gnome, KDE, Emacs and Bash all see different virtual filesystems. (All but Bash implement their own virtual filesystem extensions). That makes them much less useful than they could be.
It has always bugged me that Gnome and KDE implement their own VFS layers.
It seems to me that having a standard userspace filesystem layer like FUSE (http://sourceforge.net/projects/avf) would provide a lot of the benefit that HURD users (all 5 of them) see from filesystem "translators". Now, safely allowing unprivileged users to run arbitrary userspace filesystems would be a real trick. But, if it were possible, it could be combined with files-as-directories in some rather neat ways.
Signature · 2 lines
Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman
Quoted 9 lines · Jamie Lokier
Quoted 4 lines · Christophe Saout
And if you read test.compound (the main stream) you get a special format that contains all the components. You can copy that single stream of bytes to another (reiser4) fs and then access
test.compound/test.txtagain.(To Rik especially), this is the design which more or less satisfies lots of different goals at once.
And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?
Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 6 lines · Jamie Lokier
Quoted 3 lines
the current reiser4 semantics break that and as soon as you're having a world-writeable (e.g. /tmp) dir on it and someone is doing an opendir on it he's lost.
How does the current reiser4 semantics break that?
Quoted 7 lines
In a reiser4 filesystem, a file _is_ a directory.
opendir()is supposed to succeed on it.There's bound to be some security issue, but I'm not sure what you're getting at with /tmp. What sort of sort of security problem arises with a world-writeable directory such as /tmp, that cannot arise with the standard fs semantics?
Actually you are right on that issue because it would open the device/fifo as directory and not device/fifo (in fact I'd had to look at the code again to see whether they actually do this only for files or also for special files)
Quoted 10 lines · Christophe Saout
I'm just thinking about something. While you can't cut bytes out of unix iles a lot of filesystems can do this (holes). Most of them only on a block boundary, reiser4 on a byte boundary. If the filesystems could export this functionality using an enhanced API we could implement a compression plugin and other things on the VFS level that works with every filesystem supporting the required mechanisms, not only reiser4. And those features would take advantage of reiser4's storage mechanisms. I think Hans made the plugins reiser4-only because only reiser4 has a similar API at that time (and obviously because he didn't even think about doing it otherwise).
Yes. In some cases the compression or re-archiving algorithm would choose different parameters, depending on the alignment capabilities of the underlying filesystem. In some cases, that alignment is "offset 0 is the only one you can have", which is a severe special case where truncate and rewrite is required for changes.
Quoted 5 lines
Quoted 1 line
By the way, can reiser4 share parts of files between different files?
At the moment a file is exactly one object. But someone could write another file plugin that spans a file across multiple objects, then yes, multiple files could share parts.
Nice. That's useful for files which appear as themselves and also as part of a container.
And also for revision managed filesystems :)
-- Jamie
Quoted 6 lines
Firstly, the core library _is_ used by different projects. That's the commercial aim of reiser4: to be used by projects other than just as Linux filesystem. For example it might be used by a database, or a portable application which needs to store structured data in a flat file (with random access). A good abstraction makes it more useful than "just" a filesystem.
Which is okay as long as it still fits in the kernel. Remember that lots of projects had to adjust so far despite the commerical aims of their creators.
Quoted 3 lines
I think the disagreement between you two comes down to the idea that a generally useful API abstraction to file-like objects really should be somewhere in the VFS, and not specific to a single filesystem.
IT should, and in fact it actually is most of the part. We can have differnt operation vectors per object, but reiser4 is the only filesystem in the tree not making use of that but provinding only a single one and then dispatching internally. Even XFS that carries around and horrible internal dispatching layer as IRIX legacy gets this right.
Quoted 5 lines
Unfortunately, the problem is that reiser4 is the only filesystem which is
_technicallycapable_of implementing that abstraction in a practical way, apparently. (I'm not sure if this is really true. reiser4's object model is not the same as paths and inodes, but the impedance mismatch doesn't seem huge.)
Umm, no. In fact every filesystem does this. There's not too many objects with different namespace semantic: regular files and special files vs directories vs symlinks basically, but as soon as you go to the data patch you can have hundrets of different file_operations on a single filesystems (for special files).
Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel:
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
And if you read test.compound (the main stream) you get a special format that contains all the components. You can copy that single stream of bytes to another (reiser4) fs and then access
test.compound/test.txtagain.(To Rik especially), this is the design which more or less satisfies lots of different goals at once.
Quoted 8 lines
And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?
Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work.
Yes, the main compound stream unaware applications would see would just be a writable view of its contents. Probably not trivial to implement but doable.
It should be a simple format. Something simliar to tar. The worst thing that can happen is people start writing plugins for every existing compound format out there. It should be the other way around, agree on a simple compound format and encourage applications to use this one if the want to use this advantage.
Re: reiser4 plugins (was: silent semantic changes with reiser4)
In reply to Christophe SaoutQuoted 8 lines · Christophe Saout
Quoted 6 lines
Quoted 3 lines
I don't know, ask Hans. How could the VFS know it a filesystem wants to do something specific with a file that is completely transparent to the VFS?
The VFS shouldn't, that the whole point. That's why it allows the filesystem to register different method tables for each object.
Only the objects it can distinguish.
Yes, every inode can have different operation vectors. Which is the smallest possible object the VFS knows about.
Quoted 8 lines
Quoted 5 lines
ops->file = reiser4_file_operations; ops->symlink = reiser4_symlink_inode_operations; ops->special = reiser4_special_inode_operations; ops->dentry = reiser4_dentry_operations; ops->as = reiser4_as_operations;How could reiser4 register other operations for files that should be stored encrypted or compressed? It's all under
reiser4_file_operations.
Again, your confusing upper and lower plugins. For things happening below the pagecache you could register different address_space operations which sometimes makes sense. But you want e.g. different inode_operations for directories vs symlinks vs files.
Please read through some linux filesystem code, okay :)
Quoted 9 lines · Christoph Hellwig
Quoted 4 lines
There's bound to be some security issue, but I'm not sure what you're getting at with /tmp. What sort of sort of security problem arises with a world-writeable directory such as /tmp, that cannot arise with the standard fs semantics?
Actually you are right on that issue because it would open the device/fifo as directory and not device/fifo (in fact I'd had to look at the code again to see whether they actually do this only for files or also for special files)
Are you saying that with reiser4, you can open a device or fifo with O_DIRECTORY?
-- Jamie
Quoted 5 lines · Andrew Morton
No. All of the applications which you initially identified can be implemented by putting the various bits of data into a single file and getting applications to agree on the format of that file.
So in order to avoid breaking backup and file utilities, we'd instead break every application that reads files? That way surely lies madness :) This /could/ possibly work with lots of cooperation from glibc, but we'd also need a way to insert and delete bytes in a file... moving 1MB of attributes/streams just to change a 20 character author attribute seems a bit silly. In addition, there are little things, such as how adding a thumbnail image shouldn't change the modification time of the original picture IMO, but it does if you embed it in the file.
If it should be in userspace, we could take the OS-X approach of using directories for everything, and the "data" that would be in a hybrid directory-file is a specially named file under that directory. Programs for instance are actually directories, with the elf file underneath it. When you click on the "directory-program" in the gui it runs the associated elf file rather than opening the directory. I think that approach is promising, but so few unix programs have any indirection for file access it'd be hell to teach them all how it works. Of course with emminent breakage of some sort, maybe now is the time...
- Jim Bruce
Quoted 5 lines · Christophe Saout
It should be a simple format. Something simliar to tar. The worst thing that can happen is people start writing plugins for every existing compound format out there. It should be the other way around, agree on a simple compound format and encourage applications to use this one if the want to use this advantage.
or placing this into a userspace helper that the kernel can invoke transparently. If you're doing funny thing like that for your swap device deadlocks are the last of your worries :)
Quoted 6 lines · Jamie Lokier
Quoted 5 lines · Christoph Hellwig
Quoted 4 lines
There's bound to be some security issue, but I'm not sure what you're getting at with /tmp. What sort of sort of security problem arises with a world-writeable directory such as /tmp, that cannot arise with the standard fs semantics?
Quoted 7 lines
Quoted 4 lines
Actually you are right on that issue because it would open the device/fifo as directory and not device/fifo (in fact I'd had to look at the code again to see whether they actually do this only for files or also for special files)
Are you saying that with reiser4, you can open a device or fifo with
O_DIRECTORY?
That's what I thought, but as far as I can follow the code this is not actually true.
Re: reiser4 plugins (was: silent semantic changes with reiser4)
In reply to Christoph HellwigAm Donnerstag, den 26.08.2004, 18:06 +0200 schrieb Christoph Hellwig:
Quoted 4 lines
Again, your confusing upper and lower plugins. For things happening below the pagecache you could register different
address_spaceoperations which sometimes makes sense. But you want e.g. differentinode_operationsfor directories vs symlinks vs files.
Ok, I see your point. aops. Sorry.
Looking at the code this could be done. The wrappers that dispatch the operations are really small and call the plugin that is registered with the inode of the mapping. Instead it could have directly set the corresponding operations. Right. The wrappers are doing a few things before calling the plugin. That could be done the other way round too. But that's more of an implementation issue and could still be changed.
Quoted 1 line
Please read through some linux filesystem code, okay :)
I'm partly familiar with that, I just either misunderstood you or wasn't really thinking (probably the latter...).
Quoted 11 lines · Christoph Hellwig
Quoted 5 lines
Unfortunately, the problem is that reiser4 is the only filesystem which is
_technicallycapable_of implementing that abstraction in a practical way, apparently. (I'm not sure if this is really true. reiser4's object model is not the same as paths and inodes, but the impedance mismatch doesn't seem huge.)Umm, no. In fact every filesystem does this. There's not too many objects with different namespace semantic: regular files and special files vs directories vs symlinks basically, but as soon as you go to the data patch you can have hundrets of different
file_operationson a single filesystems (for special files).
Remember that reiser4 allows you to operate on little pieces of data, glueing and rearranging them inside files (something like that).
No other filesystem has that capability, and it's a data model which the fancy features (when they exist) will use.
You can map those pieces to underlying directories and files and renames and unlinks, so that the fancy stuff works on other filesystems, but it would be a useless model because those other filesystems wouldn't be recognisably "ordinary" files any more.
For reiser4 to expose that model through a VFS interface, and the fancy stuff to use it through the VFS interface, and for the fancy stuff to work (even imperfectly) on other filesystems which don't offer those operations, some kind of fall-back "store metadata and fragment rearrangements in auxiliary files with special names" layer would be requied. That's a big job.
I think it's a good job to do (funding, anyone? :), and the right place for that (imperfect but useful) fall-back layer is userspace with perhaps minimal VFS support. Another reason to put the fall-back layer in userspace is that applications which depend on the fancy stuff can still be portable to other OSes, working, but a bit smoother, faster and more "integrated" on reiser4.
-- Jamie
Quoted 11 lines · Jamie Lokier
Remember that reiser4 allows you to operate on little pieces of data, glueing and rearranging them inside files (something like that).
No other filesystem has that capability, and it's a data model which the fancy features (when they exist) will use.
You can map those pieces to underlying directories and files and renames and unlinks, so that the fancy stuff works on other filesystems, but it would be a useless model because those other filesystems wouldn't be recognisably "ordinary" files any more.
Quoted 6 lines
For reiser4 to expose that model through a VFS interface, and the fancy stuff to use it through the VFS interface, and for the fancy stuff to work (even imperfectly) on other filesystems which don't offer those operations, some kind of fall-back "store metadata and fragment rearrangements in auxiliary files with special names" layer would be requied. That's a big job.
I seem to miss those files in the current implementation. But again the big question is how will the interface look? If the smaller objects interact with normal files (link/rename, contained in normal directories), they will have to obey the same locking protocols. If not a filesystem could experiment with it of course.
Re: reiser4 plugins (was: silent semantic changes with reiser4)
In reply to Christophe SaoutQuoted 7 lines · Christophe Saout
Ok, I see your point. aops. Sorry.
Looking at the code this could be done. The wrappers that dispatch the operations are really small and call the plugin that is registered with the inode of the mapping. Instead it could have directly set the corresponding operations. Right. The wrappers are doing a few things before calling the plugin. That could be done the other way round too. But that's more of an implementation issue and could still be changed.
I agree that it's an implementation issue. But it's also a good proof for how Hans tries to ignore all the existing infrastructure for various reasons.
Quoted 10 lines · Markus Törnqvist
Quoted 4 lines · Matt Mackall
What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was.
It breaks the concept of a file. In ways that offer more versatility, challenge the imagination to make even better progress and keeps Linux competing with competitors who are implementing this stuff as we speak.
Sigh, it really isn't all that groundbreaking. The only special feature is the fact that it introduces directory hardlinks.
Just turn the phrase 'file-as-dir' around and you might see the point. As far as userspace is concerned the implementation would behave 99% identical if we provided a 'dir-as-file' semantics. i.e. opening a directory without 'O_DIRECTORY' would return a filehandle to a default file within the directory. Something like open("/path/to/dir", O_RDONLY) would be translated to open('/path/to/dir/_contents', O_RDONLY).
So I could simply replace all my files with directories and move the original file contents into a '_contents' file. Applications can still open them by the same name, although the attributes would look a bit different. Poof, we have the same file-as-dir concept, but now we play fair with the existing VFS and don't introduced races/deadlocks and other issues. Also existing backup tools will be able to back up and restore these 'streams'.
(btw. the same could be implemented completely in userspace, for instance in glibc. Whenever an open call gets an EISDIR error, simply retry the open, but this time with /_contents appended).
Quoted 3 lines
I for one would truly welcome the coming of thumbnails and descriptions in picture files, because I have a real-life project going on where that would be extremely handy to have in the actual file.
Ehhh, that already exists and doesn't require meta-data streams,
http://www.exif.org/
Quoted 7 lines
Quoted 5 lines
Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?
Then I guess OS X ships a broken implementation of cp, yes?
No cp is working just fine, there is no copy syscall in the VFS, so any file system specific features such meta data streams will have to be handled by the application.
So at some point 'cp' will have to be taught about macos resource forks, posix acls, xattrs, afs acls, reiserfs file-as-dir, freebsd attributes, samba/ntfs streams. And possibly even learn the semantical differences between these so that it can copy metadata between the various file systems types.
Jan
Quoted 1 line · Hans Reiser
Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem for Apple, and that means we need to put search engine and database functionality into the filesystem. It takes 11 years of serious
Hans,
I'm very curious about your ideas on how to put search engine and database functionality into the filesystem. Although I've let the Befs filesystem driver largely stagnate over the past two years, from time to time I think about the problem of exporting to userspace the attribute indexes that BeFS keeps on disk.
In the original BeOS, they solved the problem by having the filesystem driver itself take a text query string and parse it, returning a list of inodes that match. The whole business of parsing a query string in the kernel (let alone in the filesystem driver) has always seemed ugly to me. However, the best alternative I've come up with is to simply export the index data as a special file (perhaps in sysfs?) and have userspace responsible for searching the index. That would probably work, but it wouldn't help other filesystems that implement even a different index format, much less a different form of extra searchability.
Signature · 2 lines
Will Dyson "Back off man, I'm a scientist!" -Dr. Peter Venkman
Quoted 6 lines · Christoph Hellwig
Quoted 3 lines
Are you saying that with reiser4, you can open a device or fifo with
O_DIRECTORY?That's what I thought, but as far as I can follow the code this is not actually true.
It should be possible to do, though. There's nothing really different in making the "default" (unnamed) fork be a special device or a fifo.
And it would be perfectly ok for O_DIRECTORY to open such a file, as long as it opens the directory branch, not the special device.
I advocated (long ago) something like this for /dev handling, just because I think it would make sense to have
/dev/hda <- special file /dev/hda/part1 <- partition 1 (aka /dev/hda1)
which just seems like a very obvious and intuitive interface to me. Of course, we have so much legacy in /dev that there's no real point to doing this, but it's still an appealing approach, I think.
But I do take Al's concerns seriously. I like the notion of supporting "containers", but there are undoubtedly serious issues in the notion. I don't strictly know exactly _how_ to implement it sanely (I can talk about using the vfsmnt structure all I like, but the fact is, it's a different thing from a normal mount, and there may be serious problems indeed there).
Still, I really do like the idea of merging the notion of file and directory into one notion of "container". I absolutely _detest_ files with internal structure that tools have to know about (ie I hate seeing all those embedded formats that I can't use "grep" on - MIME being one case). I'd much rather see a "group of files" and a "file with a grouping of information".
(Now, flattening that "group of files" is obviously needed for serial protocols, so I think MIME/tar/xxxx are fine for _transporting_ data, but I'm saying that outside of transport I really prefer a "collection of files" approach).
Linus
Re: reiser4 plugins (was: silent semantic changes with reiser4)
In reply to Christoph HellwigAm Donnerstag, den 26.08.2004, 17:25 +0100 schrieb Christoph Hellwig:
Quoted 10 lines
Quoted 6 lines
Looking at the code this could be done. The wrappers that dispatch the operations are really small and call the plugin that is registered with the inode of the mapping. Instead it could have directly set the corresponding operations. Right. The wrappers are doing a few things before calling the plugin. That could be done the other way round too. But that's more of an implementation issue and could still be changed.
I agree that it's an implementation issue. But it's also a good proof for how Hans tries to ignore all the existing infrastructure for various reasons.
I definitely agree with you on that one.
It bugged me from the beginning because things like these are very likely to suffer from unnecessary problems that have been solved in the common code or might be solved when correctly done in the common code in the future.
Am Donnerstag, den 26.08.2004, 12:32 -0400 schrieb Will Dyson:
Quoted 6 lines
Quoted 3 lines
Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem for Apple, and that means we need to put search engine and database functionality into the filesystem. It takes 11 years of serious
I'm very curious about your ideas on how to put search engine and database functionality into the filesystem.
I could imagine something like this.
There are some special folders or files or whatever under every file where some userspace can put some metadata. A keyword perhaps.
echo linux > file.bla/keywords/topic
The filesystem might then automatically put these keywords into an index and then provide a search mechanism elsewhere where it could ask "find me all dentries with the keyword 'linux'." and it would return a list like locate does. Only that it's in realtime and also works when moving the file around (but not when copying with an unaware program for obvious reasons).
Quoted 5 lines · Wichert Akkerman
Quoted 1 line · Previously Jeremy Allison
Which is exactly how Windows apps have started to use streams.
Interesting I can not find a way using the .NET framework, which is supposed to be The Future, to use streams.
That's because you don't even need .NET...
http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/q105/7/63.asp&NoWebContent=1&NoWebContent=1
Bryan
Quoted 2 lines · Christoph Hellwig
or placing this into a userspace helper that the kernel can invoke transparently.
Yes, exactly this for most file formats.
Ideally, with _some_ user control over the handlers for their own files a bit like Hurd.
Not just archive files: metadata pulled from audio files, images, distro packages, etc. In other words much the same as Gnome and KDE's things, but with more emphasis on metadata extractors and efficient on-disk cache interaction, and a sane page-cache coherency interface (possibly FUSE).
Metadata extraction is useful for search and indexing purposes. (That, and a way to hook change notifications up with indexers -- again comes with any proper coherency interface).
For new applications using a container, it's tempting to want one format which is optimised for this. But really many applications want to write XML or compressed XML, or something else that isn't especially efficient but is easy to work with on many OSes. We should support applications whatever format they need to work with, if providing an interior view as a directory is useful for them. (This implies applications can introduce their own formats for files they need to operate on.) By the way, lazy updating of containers is an even bigger advantage if you have to use XML as the flat format, because the savings from not serialising when you don't need to are more pronounced.
For some applications the "content" isn't structured in a way which maps naturally to a plain archive file. Hans' /etc/passwd is a simple
example: at least one of the "flat" representations, when there are enough entries, probably ought to be a DB file for faster lookup. (Though, it would be great if /etc/passwd could automagically get fast lookup due to an automagic underlying representation, while still appearing to be a flat file when you cat it).
Also, imagine "cd" into a MySQL table file. It might just work.
By the way, we don't need containers to waste huge amounts of space like the Microsoft Word "fast save" documents did. Because our containers are created on demand, there is no performance advantage in them having unused holes, unlike the case were Word was saving changes periodically.
-- Jamie
Quoted 4 lines · Jan Harkes
(btw. the same could be implemented completely in userspace, for instance in glibc. Whenever an open call gets an EISDIR error, simply retry the open, but this time with /_contents appended).
Yes, and no - just to make it obvious before people jump on this, a lot of things can be prototyped in user space with things like this, but once you have to deal with races and mixed tool environments, user space suddenly doesn't work so well any more.
I think Jan understands this distinction, I just wanted to make sure everybody else is aware of the _one_ thing that kernel land does well:
- safely synchronize globally visible data structures
That's quite fundamental. 99% of what a kernel does is exactly that. TCP would be in user space too, if it wasn't for _exactly_ this issue. A lot of people think that kernels are about hardware access, and yes, that's the other 99% of the picture (I see the _big_ picture, remember?), but the "safe access to common data" is really very fundamental.
The kernel is literally the thing that makes sure that you don't have - and _cannot_ have - user programs that confuse each other by modifying data unsynchronized.
For example, a filesystem is really nothing but a way to access a disk in a controlled manner - it's not so much about hardware access, as it is about maintaining a coherent view of how some shared data (disk or whatever) is maintained.
Same goes for caches. We could cache things in user space, but if you want to _share_ your caches (so that you don't have to re-load them for every new application), you need some entity that manages those shared data structures in a secure manner. In other words, you need the kernel.
The same goes for something like a "container file". Whether you see it as "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are totally equivalent), the point of having the capability in the kernel is not that the operations cannot be done in user space - the point is that they cannot be done in user space _safely_. The kernel is kind of the thing that guarantees that everybody follows the rules.
Imagine the security problems if a set-uid program were to (unwittingly) depend on a user-space library that implements what Jan's prototype library would do? Races galore, since a user-space implementation wouldn't have _any_ way to do tests like the above atomically.
Linus
Quoted 3 lines · Rik van Riel
And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?
Yes, exactly that. The streams are created on demand of course, and by userspace helpers when that's appropriate which I suspect it almost always is.
Quoted 4 lines
Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work.
Yes. Hence the idea of coherent views between two files: writing to one affects the content of the other, although the calcalation is only done on demand (or when the fs wants to migrate the representation -- for example, creating the flat container prior to deleting the regeneratable pieces in order to save space).
I haven't seen anything from Namesys that says they'll do that. I have the impression the streams are just generated in memory on the fly, not stored on disk with a cacheing policy, but that's just an impression. (We've all seen the Namesys white papers, they're not _that_ revealing). :)
I'm just pointing out how to do it _right_. I think it will turn out like this eventually, either next year or some time over the next decade after some iterations. Inevitable. Mark my words, etc. :)
-- Jamie
Quoted 2 lines · James Bruce
moving 1MB of attributes/streams just to change a 20 character author attribute seems a bit silly.
That's what we do now. Try editing an MP3's id tags.
Quoted 8 lines
If it should be in userspace, we could take the OS-X approach of using directories for everything, and the "data" that would be in a hybrid directory-file is a specially named file under that directory. Programs for instance are actually directories, with the elf file underneath it. When you click on the "directory-program" in the gui it runs the associated elf file rather than opening the directory. I think that approach is promising, but so few unix programs have any indirection for file access it'd be hell to teach them all how it works.
The interfaces we've been talking about are very close to that. The indirection is automatic: instead of a document (or program) looking like a directory, it'll look like a file, and reading (or executing) the file will do the indirection, with kernel control (and so consistency and performance) via an appropriate flat<->structured relation helper.
-- Jamie
Quoted 3 lines · El Thu, 26 Aug 2004 11:54:51 -0400 (EDT) Rik van Riel <[email protected]>…
And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?
If the old utils doesn't support streams there's no way to make them create streams. There's not a real solution for that except fixing the old tools.
Why instead of being "compound" you just make test.compound to behave as a real directory? ie:
# echo "foo" > streamfile # echo "fake metadata" > streamfile/somemetadata # cat streamfile foo # cat streamfile/somemetadata fake metadata
Now the old backup utils could see "streamfile" as a directory, ie:
old apps could see that "streamfile" is a directory and then make their backups with the following contents:
streamfile/somemetadata
streamfile/defaultstream
The defaultstream would be what its name means. There could be more default streams like "streamfile/compoundstream"
All this looks like reinventing the file/directory concept wheel. Instead of adding support for streams and "use files as directories", why not orientate it to "use directories as files? Streams could very well be provided by directories containing files, with the addon of being able to cat a directory (which would cat the output of a designed file inside every directory like "defaultstream", then there could be other "special files" like "compoundstream", although the idea of "fixed name- special files" doesn't sounds very cool) All this could be very well provided done userspace, but then couldn't userspace also supply the whole VFS functionality and still it lives in the kernel...?
Quoted 4 lines · Christoph Hellwig
If the smaller objects interact with normal files (link/rename, contained in normal directories), they will have to obey the same locking protocols. If not a filesystem could experiment with it of course.
I think the smaller objects will consist of special files, which are magic attributes like the ones you currently see in reiser4, and ordinary files (which might themselves be containers).
Link/rename/locking etc. should work as usual on the ordinary smaller files, but probably aren't allowed at all on the special attribute files, except where there's an appropriate meaning. (For example locking certain attributes might have an appropriate meaning).
It would make sense for the special attributes file to not identify as "regular files" (S_IFREG), but there aren't many mode bits to play with. They should also be harder to find than ordinary smaller files.
-- Jamie
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Jan" == Jan Harkes <[email protected]> writes:
Jan> On Thu, Aug 26, 2004 at 10:53:48AM +0300, Markus Törnqvist wrote:
[...]
Quoted 4 lines
Quoted 4 lines
I for one would truly welcome the coming of thumbnails and descriptions in picture files, because I have a real-life project going on where that would be extremely handy to have in the actual file.
Jan> Ehhh, that already exists and doesn't require meta-data streams,
Jan> http://www.exif.org/
but then you need an extra library to access the exif data. You can't just do something like "display foo.jpg/exif/thumbnail". Or even better, "echo 'this is a comment' > foo.jpg/comment" to change the jpeg comment without an external program.
And I don't think exif tags work with all image formats.
Quoted 6 lines
Quoted 6 lines · Matt Mackall
Quoted 5 lines
Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?
Under file-as-dir, it would not look the same under the shell if cp didn't copy everything -- the directory contents would be different. To copy the metadata as well, you would do something like "cp -r". Similarly with "cat < a > b".
Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Quoted 4 lines · Diego Calleja
All this looks like reinventing the file/directory concept wheel. Instead of adding support for streams and "use files as directories", why not orientate it to "use directories as files? Streams could very well be provided by directories containing files,
So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
This is getting surreal.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 2 lines · Linus Torvalds
And it would be perfectly ok for
O_DIRECTORYto open such a file, as long as it opens the directory branch, not the special device.
What about:
cd /dev/into_directory_branch/
run_setuid_program
-> calls pwd
pwd opens("."), (".."), ("../..") etc.
-> the setuid program thus ends up opening a device or fifo,
when it does pwd's path walk. Yes it could use the getcwd
syscall, but some programs do their own path walk.Quoted 9 lines
I advocated (long ago) something like this for /dev handling, just because I think it would make sense to have
/dev/hda <- special file /dev/hda/part1 <- partition 1 (aka /dev/hda1)which just seems like a very obvious and intuitive interface to me. Of course, we have so much legacy in /dev that there's no real point to doing this, but it's still an appealing approach, I think.
It also fits the container idea very well:
/dev/hda/part1/ <- the filesystem inside partition 1
That's not a contrivance, it's what I'd expect to happen if /dev/hda is an ordinary file containing a disk image:
image.bin <- hard disk image (a regular file)
image.bin/part1 <- partition 1
image.bin/part1/ <- the filesystem inside partition 1
That's assuming there's a format handler which recognises that image.
-- Jamie
Quoted 9 lines · Rik van Riel
Quoted 4 lines · Diego Calleja
All this looks like reinventing the file/directory concept wheel. Instead of adding support for streams and "use files as directories", why not orientate it to "use directories as files? Streams could very well be provided by directories containing files,
So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
No no. The stream you get from a directory is totally _independent_ of the contents of the directory. Anything else would be pointless.
Linus
Quoted 8 lines · Christophe Saout
echo linux > file.bla/keywords/topic
The filesystem might then automatically put these keywords into an index and then provide a search mechanism elsewhere where it could ask "find me all dentries with the keyword 'linux'." and it would return a list like locate does. Only that it's in realtime and also works when moving the file around (but not when copying with an unaware program for obvious reasons).
It could work even when copying with an unaware program - provided the're a tool which knows to extract metadata from files where that hasn't been done already, or where the file's been modified.
-- Jamie
Quoted 4 lines · Linus Torvalds
Quoted 1 line
So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
No no. The stream you get from a directory is totally _independent_ of the contents of the directory. Anything else would be pointless.
Surely it depends on the directory?
The stream from a "classic directory", i.e. something made with mkdir(), mode drwxr-xr-x filled with files, might well be a reproducible representation of its contents.
-- Jamie
Quoted 5 lines · Jamie Lokier
run_setuid_program
-> calls pwd
pwd opens("."), (".."), ("../..") etc.Ehh.. Not only does pwd not do that..
(hint: there's a getcwd() system call)
Quoted 3 lines
-> the setuid program thus ends up opening a device or fifo,
when it does pwd's path walk. Yes it could use the getcwd syscall, but some programs do their own path walk.
.. but even if it did that, it should use O_DIRECTORY when it did so. If it doesn't, it's broken.
So no, it would _not_ open the device or fifo when it did so.
The fact is, anything that expects to open a directory should already be opening it with O_DIRECTORY.
That said, ".." and "." are special already inside the kernel, and it migth be worth making them automatically imply O_DIRECTORY, since nothing else makes sense anyway. That would fix the case where somebody uses ".." _without_ using O_DIRECTORY.
Quoted 3 lines
It also fits the container idea very well:
/dev/hda/part1/<- the filesystem inside partition 1
I don't think you can do that. The kernel has no idea how to mount the filesystem.
If it's already mounted somewhere else, that's a different issue. Although it might be mounted in several places (as a bind mount) with different writability, I guess, so even then it might be "interesting".
Linus
Quoted 10 lines · Andrew Morton
Quoted 5 lines · Spam
Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.
All of which can be handled in userspace library code.
What compelling reason is there for doing this in the kernel?
Because without kernel support there is no way someone can publish a new metadata type and have it automatically supported by all application data files (ie. most apps ignore it, and only apps that are aware of it can see it). Without kernel support you have to have all apps agree on a data format. And that's harder to do than getting linux kernel VFS engineers to agree on things :-).
Jeremy.
Quoted 6 lines · Jamie Lokier
Quoted 3 lines · Linus Torvalds
No no. The stream you get from a directory is totally _independent_ of the contents of the directory. Anything else would be pointless.
Surely it depends on the directory?
Well, you _could_ make it do a "tar", but what's the point? That really doesn't add any value as far as user space is concerned.
Think of it this way: if the directory and the "default stream" associated with it aren't independent, then user space already has the information, and the stream is pointless. In fact, the stream is _worse_ than pointless, because it hides the fact that there is no independent information.
See my point?
Linus
Quoted 1 line · El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <[email protected]>…
So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
/bin could be separated (like linus said) but cat /bin/.compound could do it. This is the /etc/passwd Hans' example, I think:
"If /new_syntax_access_path/big_directory_of_small_files/.glued is a plugin that aggregates every file in big_directory_of_small_files with a delimiter separating every file within the aggregation, then one can simply type emacs /new_syntax_access_path/big_directory_of_small_files/.glued, and the filesystem has done all the work emacs needs to be effective at this. Not a line of emacs needs to be changed."
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines · Linus Torvalds
Quoted 1 line
/dev/hda/part1/<- the filesystem inside partition 1I don't think you can do that. The kernel has no idea how to mount the filesystem.
... and what's more, fundamental thing for mount is fs type, what to do with device(s) (if any) and any other arguments is the filesystem's business.
Quoted 4 lines · Diego Calleja
Quoted 1 line
So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
/bin could be separated (like linus said) but cat
/bin/.compoundcould do it. This is the/etc/passwdHans' example, I think:
Yes, but what about locking? If we have compound files with individually accessible components a lock on the compound files could lock its components at the same time.
But what with /usr/.compound? Is it read-only? What if you truncate it? Does it atomically delete the whole /usr directory tree? What about reading? It has to go through all these independent files, what if they change while you are reading the compound file? Does it lock everything? Point-in-time snapshot? If reiser4 can't do this. ;-)
I don't say that it's impossible to do but I think it's a lot harder than doing it the other way round.
Quoted 4 lines · Linus Torvalds
Quoted 1 line
So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
No no. The stream you get from a directory is totally _independent_ of the contents of the directory. Anything else would be pointless.
It's a relief to know that nobody's taking my humorous suggestion seriously, but now we still have the "standard Unix tools can't manipulate files" problem...
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 6 lines · Diego Calleja
Quoted 1 line · El Thu, 26 Aug 2004 13:16:22 -0400 (EDT) Rik van Riel <[email protected]>…
So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
/bin could be separated (like linus said) but cat
/bin/.compoundcould do it. This is the/etc/passwdHans' example, I think:
Arghhhh. I wrote it down to ridicule the idea and now people are taking it seriously ;(
It should be obvious enough that anything depending on the kernel parsing file contents will lead to problems.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 5 lines · --- Nicholas Miell
Anything that currently stores a file's metadata in another file really
wants this right now. Things like image thumbnails, document summaries,
digital signatures, etc.
As well as cases where the metadata is currently _in_ the file, right? Like MP3ID tags, EXIF tags, etc. And so many MP3 jukebox-server programs currently use a database to store metadata, just because they don't trust the MP3ID tags or need to store supplemental info. Being able to put this stuff in the filesystem along with the files would be so much cleaner. So I think there are plenty of uses for it.
IMO the Mac resource fork was always a good idea too, and I haven't a clue why it got left out of OSX (maybe Woz "got it" but Jobs doesn't).
Somebody said "we don't have to teach tar about it"; does that mean that tar in its present form would already successfully find the alternate streams?
=====
. _______ Shawn T. Rutledge / KB7PWD [email protected]
(_ | |_) http://ecloud.org/ [email protected] __) | | \______________________________________________
I trimmed/adjusted the CC list somewhat in the hope of getting a more productive discussion going.
Quoted 10 lines · Linus Torvalds
Quoted 4 lines · Jan Harkes
(btw. the same could be implemented completely in userspace, for instance in glibc. Whenever an open call gets an EISDIR error, simply retry the open, but this time with /_contents appended).
Yes, and no - just to make it obvious before people jump on this, a lot of things can be prototyped in user space with things like this, but once you have to deal with races and mixed tool environments, user space suddenly doesn't work so well any more.
Agreed.
Quoted 3 lines
The same goes for something like a "container file". Whether you see it as "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are totally equivalent), the point of having the capability in the kernel is
Only to some point. I was mostly trying to make the point that similar functionality is already achievable within the current VFS framework.
However, files and directories still have some semantical differences, and these are probably the points that we would have to address to get Hans's efforts work well in cooperation with the Linux VFS.
We cannot atomically unlink a dir-as-file, but can unlink a file-as-dir. In this case, the dcache seems to have no problems unlinking/unhashing a non-empty directory, so I'm assuming that the unlink problem is mostly enforced to avoid unreachable, but still allocated files in a file system. (I'm sure that Viro can probably come up with other reasons why unlinking a non-empty directory can be evil).
Also, we cannot create hardlinks to a directory (-as-file), but I guess reiserfs4 allows hardlinks to files (-as-directory). Luckily as far as hardlinked directories are concerned the file-as-dir implementation should be a relatively benign subset. I assume that we are not allowed to rename or link arbitrary objects from the regular fs into it, Coda returns EXDEV in such situations. As a result there will be no cycles and this simple premise should get rid of at least a couple of the possible deadlock scenarios that I've seen mentioned.
Jan
Quoted 1 line · Hans Reiser
Andrew Morton wrote:
[...]
Quoted 2 lines
Quoted 2 lines
The fact that one filesystem will offer features which other filesystems do not and cannot offer makes me queasy for some reason.
Not me. As long as it is clearly experimental stuff that can be removed at the head hacker's whim, that is. 2.7 material.
Quoted 2 lines
Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem for Apple,
Says who?
Quoted 2 lines
and that means we need to put search engine and database
functionality into the filesystem.
Please don't. Unix works and is extremely popular because it _doesn't_ try to be smart (policy vs mechanism distinction, simple abstractions that can be combined endlessly). If you add this to the filesystems, you either redo _all_ userland to understand _one_ particular way of doing "smart things" (which will turn out to be almost exactly the dumbest possible for some uses, and then you are stuck), or get lots of shards from broken apps (and users, and sysadmins).
Quoted 2 lines
It takes 11 years of serious
research to build a clean storage layer able to handle doing that.
Great! Build an experimental OS showing how to use it, and through that see if the ideas hold any water _in real, day-to-day, down-to-earth, practice_.
Quoted 2 lines
Reiser4 has done that, finally. None of the other Linux filesystems have.
How come nobody before you in the almost 30 years of Unix has ever seen the need for this indispensable feature?
Quoted 3 lines
The next major release of ReiserFS is going to be bursting with
semantic enhancements, because the prerequisites for them are in place now. None of the other Linux filesystems have those prerequisites.
To me that would mean that ReiserFS has no place in Linux.
Quoted 2 lines
They won't be able to keep up with the semantic enhancements. This metafiles and file-directories stuff is actually fairly trivial stuff.
Maybe. But the question of it being of any use aren't trivially answered "yes". My gut feeling is that the answer is a resounding "no", but that's just me. Direcories are directories, if you want to ship directory-like stuff around, use directories (or tarfiles, or whatever). Just look what happened to the much, much easier stuff of splitting SUID/SGID into
capabilities: Nothing at all whatsoever, because they have no decent place to stay in the hallowed Unix APIs. Sure, Linux is now large enough to be able to _define_ APIs to follow, but even so it isn't at all easy to do so.
Quoted 3 lines
Look guys, in 1993 I anticipated the battle would be here, and I build the foundation for a defensive tower right at the spot MS and Apple are now maneuvering towards.
Why place a protective tower around the pit into which they are trying desperately to throw themselves? ;-)
Quoted 6 lines
Help me get the next level on the tower before
they get here. It is one hell of a foundation, they won't be able to shake it, their trees are not as powerful. Don't move reiser4 into vfs, use reiser4 as the vfs. Don't write filesystems, write file plugins and disk format plugins and all the other kinds of plugins, and you won't be missing any expressive power that you really want....
Better write libraries handling whatever weird format you have in mind for each use. Even applications. I don't expect to be able to use emacs to edit a JPEG or PostgreSQL database. Neither do I expect gimp to be able to edit a poem. The current trend (which I heartily agree with) is to take stuff _out_ of the kernel (for flexibility, access, development ease, even performance).
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Quoted 4 lines · Rik van Riel
It's a relief to know that nobody's taking my humorous suggestion seriously, but now we still have the "standard Unix tools can't manipulate files" problem...
I disagree. They can manipulate the files a whole lot better than they can manipulate xattr's.
For example, you _could_ probably (but hey, maybe "tar" tries to strip slashes off the end of filenames, so this might not work due to silly reasons like that) back up a compound file with
tar cvf file.tar file file/
although unpacking it would require that tar be taught about the thing. And you definitely could write a script to do the thing, ie even with an unmodified tar you could do
tar cvf file-archive.tar file
cd file
tar cvf ../attribute-archive.tar .
which is a hell of a lot better than what you can do with the fsattr interfaces and unmodified legacy applications.
So one of the advantages of "dir-as-file/file-as-dir" is exactly that you _can_ manipulate the data with legacy tools. Sure, things that traverse a directory tree might need some (likely fairly trivial) modifications if they really want to take advantage of the subfiles, but that's still likely to be _much_ less of an issue than with fsattr's that have a totally different model entirely.
Linus
Quoted 6 lines · Linus Torvalds
Quoted 3 lines
-> the setuid program thus ends up opening a device or fifo,
when it does pwd's path walk. Yes it could use the getcwd syscall, but some programs do their own path walk... but even if it did that, it should use
O_DIRECTORYwhen it did so. If it doesn't, it's broken.
Didn't someone just say that O_DIRECTORY will succeed on a device, precisely because opendir() is supposed to succeed on the device?
I'm not sure. Let me refresh my memory.
Quoted 2 lines · Linus Torvalds
And it would be perfectly ok for
O_DIRECTORYto open such a file, as long as it opens the directory branch, not the special device.
Oh, I see. You're right. ".." sees the path of directory branches.
Quoted 6 lines
Quoted 3 lines
It also fits the container idea very well:
/dev/hda/part1/<- the filesystem inside partition 1I don't think you can do that. The kernel has no idea how to mount the filesystem.
It is not the kernel which decides. The filesystem containing /dev/hda/part1 opens "the directory branch". If the filesystem is suitably configured, it's exactly the same as opening a .zip or ELF
file: it asks userspace for help. Userspace helps by either mounting it, or creating a container virtual directory -- in exactly the same way it helps with .zip and ELF files by creating a container virtual directory for those. It would mount it if the kernel supports the fs, and create a virtual directory if the kernel doesn't support the fs and userspace does.
This is exactly the same as when you cd into a filesystem image: the helper mounts (this time with a lookback file mount) if the kernel supports the image fs, otherwise the helper may be able to decode it in userspace.
By the way, these mechanisms could allow some of the old filesystems that nobody uses any more to be removed from the kernel altogether and kept in an archive of userspace-only supported filesystems. What do you think of that idea?
Quoted 3 lines
If it's already mounted somewhere else, that's a different issue. Although it might be mounted in several places (as a bind mount) with different writability, I guess, so even then it might be "interesting".
The obvious implementation has the userspace helper just mounting it, end of story. If the mount command fails, it fails. Much like autofs.
-- Jamie
Quoted 1 line · Helge Hafting
Matt Mackall wrote:
[...]
Quoted 6 lines
Quoted 4 lines
Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_?
It is what I'd expect.
Great. That means that all the tools of the trade stop working. Sounds like show-killer feature to me.
Quoted 2 lines
Now, use cp -R to copy the file
_withitsdirectory_,
Either it is a file or a directory. Make up your mind. If you have no clear distinction, you'll only get messed up. Badly.
Quoted 5 lines
and see if that fares better. If not - bad
implementation of fs and/or cp. The way I see file-as -directory is that _file_ operations (like the reads issued by cat) only work on the file part. You want the directory part? Use directory operations such as those "cp -R" use.
Excuse me, I must grab my sickness bag here.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Quoted 4 lines · Jeremy Allison
Because without kernel support there is no way someone can publish a new metadata type and have it automatically supported by all application data files (ie. most apps ignore it, and only apps that are aware of it can see it).
So your backup software ignores it, and after a restore you've lost your new metadata ?
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 3 lines · Shawn Rutledge
Somebody said "we don't have to teach tar about it"; does that mean that tar in its present form would already successfully find the alternate streams?
Look at history - did anybody teach tar about xattrs or acls? If nobody cared about those, why will reiser4 metadata be any different?
Quoted 8 lines · Rik van Riel
Quoted 2 lines
/bin could be separated (like linus said) but cat
/bin/.compoundcould do it. This is the/etc/passwdHans' example, I think:Arghhhh. I wrote it down to ridicule the idea and now people are taking it seriously ;(
It should be obvious enough that anything depending on the kernel parsing file contents will lead to problems.
This is one case where the kernel _isn't_ parsing file contents, but I agree it's ridiculous :)
-- Jamie
Quoted 6 lines · Jamie Lokier
Quoted 3 lines
.. but even if it did that, it should use
O_DIRECTORYwhen it did so. If it doesn't, it's broken.Didn't someone just say that
O_DIRECTORYwill succeed on a device, precisely becauseopendir()is supposed to succeed on the device?
It will succeed on the _name_ of the device, but it won't open the device node. It would open the _directory_ node associated with that name.
Think of it this way: a pathname always points to one "container". That container is either a directory or a single node - and O_DIRECTORY ends up being the thing that chooses between them.
Quoted 5 lines
Quoted 2 lines
I don't think you can do that. The kernel has no idea how to mount the filesystem.
It is not the kernel which decides. The filesystem containing
/dev/hda/part1opens "the directory branch".
But that filesystem cannot know what the _other_ filesystem configurations are. And that's what you'd have to have to mount.
Quoted 2 lines
The obvious implementation has the userspace helper just mounting it, end of story. If the mount command fails, it fails. Much like autofs.
Yes, that would work, but it's of questionable use. If you want autofs, then just _use_ autofs.
Linus
Quoted 9 lines · Rik van Riel
Quoted 4 lines · Linus Torvalds
Quoted 1 line
So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?
No no. The stream you get from a directory is totally _independent_ of the contents of the directory. Anything else would be pointless.
It's a relief to know that nobody's taking my humorous suggestion seriously, but now we still have the "standard Unix tools can't manipulate files" problem...
Is it possible to sufficiently hide "dirs inside files" so that old tools will be unable to see them?
I just checked:
ls -d /foo does lstat64("/foo", ...)
ls -d /foo/ does lstat64("/foo", ...)
but
ls -d /foo/. does lstat64("/foo/.", ...)
Will it work out if "dir inside file" will only be visible when referred as "file/."? --
vda
Quoted 1 line · Hans Reiser
Yes, this was part of the plan, tar file-directory plugins would be cute.
Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute
directory or a directory in the tarball named "metas"?
Joel
Signature · 10 lines
"I'm so tired of being tired, Sure as night will follow day. Most things I worry about Never happen anyway." Joel Becker Senior Member of Technical Staff Oracle Corporation E-mail: [email protected] Phone: (650) 506-8127
Quoted 3 lines · Hans Reiser
How old are you? I thought you were the guy at Linux Tag with fashion oriented hair who gave a talk on his XFS work? Did I confuse you with someone else?
CREDITS +8 -0
@@ -2677,6 +2677,14 @@ S: 79098 Freiburg S: Germany +N: Hans Reiser +E: [email protected] +W: http://www.namesys.com/ +D: official Linux kernel hairstyle critic +S: 6979 Exeter Dr +S: Oakland, CA 94611 +S: USA + N: Joerg Reuter E: [email protected] W: http://yaina.de/jreuter/
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Christophe Saout wrote:
| Yes, the main compound stream unaware applications would see would just | be a writable view of its contents. Probably not trivial to implement | but doable.
Not the main stream, I think, but we do need a way to make "eight-stream octopus" files portable. But there needs to be a copy(2) syscall, and apps need to start supporting it. After all, copy-on-write was planned for Reiser4, now 4.1*. And don't anyone dare suggest "userland library".
| It should be a simple format. Something simliar to tar. The worst thing
Simple, yes. Similar to tar, no.
How about a serializer plugin? foo.mp3/serialize (say) should return some single stream which contains all of foo.mp3. It could work for directories as well, recursively, at least on the same filesystem -- you'd back up by doing
cp /serialize /backup/20040826
This has two potential problems. One is security -- I want to be able to serialize foo.mp3 without being root, but root doesn't want me to be able to serialize /etc and thus get access to shadow.
The second problem is that in order to actually be used for backup, we need snapshots, which are not done yet*. Currently, it'd have to lock the file/directory (recursively), which you don't want to do to entire live filesystems... So, no massive backups of live systems yet, but definitely useful for sending around mp3s and such.
| that can happen is people start writing plugins for every existing | compound format out there. It should be the other way around, agree on a | simple compound format and encourage applications to use this one if the | want to use this advantage.
For the format, I vote for whatever format we're using at the storage layer. It'd be a lot faster that way, both in execution time (I think) and in development time (I'm sure). Plus, imagine the restore:
cat /backup/20040826 > /dev/hda5resizefs_reiser4 /dev/hda5
* Disclaimer -- all this is "I think, last I checked".
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQS4uhngHNmZLgCUhAQKTaw//WCHt0Umgaf0GaMUpEr0PLhiXhFTYQqKV MKgpVt6SjP7lC9k3RbuCnPCLyOJGb9lIqo+Wd+B3b4TcQAQd4OYsSI6WiMPQjxXY OdNnB7/xYkkd/ULw7/7IGcokyc4FRDjhTyOMe4KscQS/6uhdX1dCriCCzVq7iELx u3HLulunHv1bwHbnZ6CyEZdQEr3j8XUu62+IXPTc6hWnfRRCvskV8Y4qyyVd8wFY Kp44TxbJJSzDeWmuI/YYkxVJsPmSoxRLxmw9/B8D72KGNg1SYLnM4ueIc5IV8RH0 kG/o8tzwTRxE/eqfcHP+UBt6yXZR/i84DDogAmJFKP73g9X+U+J5o/kONOBQODhY qY5LUwQu4UjlzXttMxC4NmZPESdUw85DgC/GhTQlsBMYBcmQHmpLAK67qt83+Kg0 1QqGPqJm4HL4weHLNknJ0DukLNKH7Bw5CVg335IwruaoiKUG7HDgthUhA319FCab VvYa+bEfeBEw9RorSE/gWM9Hg73luM7eoSzKc9zHgx++cACTi8SBsTFWqI2FU6DX UyWG5sNPybOiczpPsFu5S5FJBBNM3fLjkfeYgfqxUIjxGTNeUn/g/9I5ShUrCF1o T+3nd1jFxfoehYGkfGB2YLpdeNVBv4EdKK16j39+iducfJrXnRlL1yjdo/Kte2uz tTQZw3uTnhU=
=7bBF
-----END PGP SIGNATURE-----
Quoted 8 lines · Christoph Hellwig
Quoted 2 lines · Hans Reiser
Andrew, we need to compete with WinFS and Dominic Giampaolo's filesystem for Apple, and that means we need to put search engine and database
Dou you know a nice thing? We (as in the Linux Community) don't have to compete with anyone. Sure, we're usually trying to be better than anyone else, but unlike companies under maret pressure we can wait until something is ready.
By the way, this kind of features can be achieved easily in userspace with, for example, ext3, beagle (a userspace fulltext indexing daemon) and some (d|i)notify magic.
Quoted 5 lines · Linus Torvalds
For example, you _could_ probably (but hey, maybe "tar" tries to strip slashes off the end of filenames, so this might not work due to silly reasons like that) back up a compound file with
tar cvf file.tar file file/
So you'd have both a file and a directory that just happen to have the same name ? How would this work in the dcache?
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 9 lines · Christoph Hellwig
How do you for example suggestion exporting your semantics over the network if they're not done at the VFS level? How do you want some clusterfilesystem support them or tmpfs?
by accesses to filename/metas.....
That's the beauty of simplicity, not much is needed. xattrs on the other hand would be a lot of work to integrate into a cluster filesystem.
Hans
Quoted 3 lines · Denis Vlasenko
Is it possible to sufficiently hide "dirs inside files" so that old tools will be unable to see them?
Certainly possible.
Quoted 8 lines
I just checked:
ls -d /foo does
lstat64("/foo", ...)
ls -d /foo/ doeslstat64("/foo", ...)
but
ls -d /foo/. doeslstat64("/foo/.", ...)Will it work out if "dir inside file" will only be visible when referred as "file/."?
That would likely be the _easiest_ approach for the kernel, and does solve the problem with some apps knowingly removing the trailing '/'.
Note that we could try this out with existing filesystems with very minimal changes:
- make directory bind mounts work on top of files ("graft_tree()")
- make open_namei() and friend _not_ do the mount-point following for the last component if it's a non-directory.
- probably some trivial fixups I haven't thought about. There might be some places that use "S_ISDIR()" to check for whether something can be looked up, but the main path walking already just checks whether there is a ".lookup" operation or not.
This would already allow people to "try out" how different applications would react to a file that can show up both as a directory and a file. The patch might end up being less than 25 lines or so, the difficulty is in finding all the right places.
Al, anything I missed?
(And yes, it's a quick hack, but it's a quick hack that would probably mimic a good part of what we would have to do internally in the VFS layer to support this notion anyway).
Linus
CREDITS line 2687
Quoted 9 lines · William Lee Irwin III
CREDITS +8
-0+N: Hans Reiser +E: [email protected] +W: http://www.namesys.com/ +D: official Linux kernel hairstyle critic +S: 6979 Exeter Dr +S: Oakland, CA 94611 +S: USA +
LOL.
In all fairness, how _would_ you have described it?
Besides, I don't think this should go in the CREDITS file, since hair styling criticism is clearly an ongoing MAINTAINERS issue, no?
Linus
Quoted 3 lines · Rik van Riel
So you'd have both a file and a directory that just happen to have the same name ? How would this work in the dcache?
There would be only one entry in the dcache. The lookup will select whether it opens the file or the directory based on O_DIRECTORY (and usage, of course - if it's in the middle of a path, it obviously needs to be opened as a directory regardless).
That's not the problem. The problem from a dcache standpoint ends up being when the file has a link, and you have two paths to the same sub-file through two different ways:
.. create file 'x' with named stream 'y' ... ln x z
ls -l x/y z/y /* it's the same attribute!! */
but this is actually exactly the same thing that we already have with mounts, ie it is equivalent (from a dentry standpoint) to
.. create directory 'x' with file 'y' .. mkdir z mount --bind x z ls -l x/y z/y /* It's the same file!! */
so none of this is really anything "new" from a dcache standpoint.
Except for all the details, of course ;)
Linus
Quoted 7 lines · William Lee Irwin III
Quoted 7 lines
(quoted diff) +7
-0+N: Hans Reiser +E: [email protected] +W: http://www.namesys.com/ +D: official Linux kernel hairstyle critic +S: 6979 Exeter Dr +S: Oakland, CA 94611 +S: USA
Quoted 4 lines · Linus Torvalds
LOL.
In all fairness, how _would_ you have described it? Besides, I don't think this should go in the CREDITS file, since hair styling criticism is clearly an ongoingMAINTAINERSissue, no?
As far as maintenance goes, absolutely. As far as describing it goes, I probably would have said it was short, red, and spiky.
MAINTAINERS +6 -0
@@ -1352,6 +1352,12 @@ L: [email protected] S: Supported +LINUX KERNEL HAIRSTYLE CRITICISM +P: Hans Reiser +M: [email protected] +W: http://www.namesys.com/ +S: Maintained + LINUX SECURITY MODULE (LSM) FRAMEWORK P: Chris Wright M: [email protected]
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Rik van Riel
So you'd have both a file and a directory that just happen to have the same name ? How would this work in the dcache?
There would be only one entry in the dcache. The lookup will select whether it opens the file or the directory based on
O_DIRECTORY(and usage, of course - if it's in the middle of a path, it obviously needs to be opened as a directory regardless).
Hmmm, I just straced "cp /bin/bash /tmp".
One line stood out as a potential problem:
open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4
What do we do with O_CREAT ?
Do we always allow both a directory and a file to be created with the same name ?
Does this create a new class of "symlink attack" style security holes ?
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Linus Torvalds wrote:
[...]
| tar cvf file.tar file file/
I don't like it. Imagine this:
$ cat file/perm 040700 drwx------ $ cat file/perm/user/writable 1 $ cat file/perm/uid 0
Do that tar, and you lose the uid. This is an example, and has plenty of flaws. The point is that often there may be files/directories within files/directories.
In order to make this work, you either have to patch tar or split every file/directory into a file/directory pair, recursively, using some sort of mangling. "file" becomes file.f and file.d. Then tar works.
To be honest, I like my "serializer" better. It just feels faster and cleaner.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQS4z1XgHNmZLgCUhAQKXAQ//eJ9tPFFEf1kpFY7gYUhMSUAAs1wIvVrN pOVyLPJwGTv4oIaq8/8Op+XmQt1BWfnFdgwSal5R7vuLlrj0/Lp714I4sTOb35+6 q4hB1Q6X1oKbS1QIVWM9zw8PaebTY1GZDYITqWEN8xFN2cx41m4HZZ9P2TXNGPWx 1TqeBAaR07N1LaDzDHWvXu2v8pP6Wjy8mmWbtHbjN7U8oLEAUVXmT/ZJ5MpVK6z6 FTTA6NnARpEDtWebTJit14IQYG88syl31+lNL/N87iTveYI3+6gAQniajkMda565 PnPbVCd/aLHpx4hNLYDiNZwGkhmCgA/V3/vHJGT1vbTWN56zFLF7XQqE0BMfR5k6 ozwzBZ368+suEs0pDplCJq7ru0BF3j49r2Ad7aILCJOENvN0uVzLNxmu+LfvARTm gG7UNkJYcAjwmUaE1vXEACnwKhtyyxIpRbsb9ALqbDS9KXB7eAaI2FwLPXCU/L61 qHWHN47KllWdfT4fttwPd/Lag7UpKu9McBPnp9ZOTuYGY6QTgflBG1C7kallcJUw 7cZh831PRqmnj2hWcAlSvinD3ClxaF9swewMPfQJ9I10n7p6w9Q9EJEXUX6+gUhv 5MD6q2WxqNVM/i5Y/2s3GQ+3lJ2VvnWHkJgc7JkmP0H1N0vocYWhKaofMOQXAy57 /ocTFfcwzUc=
=YrfX
-----END PGP SIGNATURE-----
Quoted 8 lines · Jamie Lokier
Quoted 3 lines · Nicholas Miell
Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.
Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed.
In principle, not all of them. For example, a document summary for a text document or a long (textual) description of a video clip might remain the same when the user is only working on the finishing details.
Maybe the metadata needs an attribute to determine how 'immutable' it should be wrt changes on the file? (Can you spell meta-meta-data <g>)
Signature · 7 lines
Giuseppe "Oblomov" Bilotta
Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
(Roger Waters)Quoted 10 lines · Linus Torvalds
Well, you _could_ make it do a "tar", but what's the point? That really doesn't add any value as far as user space is concerned.
Think of it this way: if the directory and the "default stream" associated with it aren't independent, then user space already has the information, and the stream is pointless. In fact, the stream is _worse_ than pointless, because it hides the fact that there is no independent information.
See my point?
I see your point, but I agree 50%.
(1) One of the uses of "streams" is to have extra information which accompanies a file, and is separate from the flat file's information.
For example, metadata like "this C source file is in UTF-8", "the author is Jamie (nowhere to put it in the data file)" and "Jamie's signature".
(2) Filesystem settings like "compress this", and perhaps "permissions".
(3) Filesystem information like "this file has been compressed by 59%, uses 5 1k blocks".
(4) Another use, which is more akin to containers, is to _present_ the information of the flat file in a different way.
For example, the id tags and comments of any media file, the component parts of any media file, license (of course :), and expanded or decoded views such as inside an archive file, filesystem image, or other structured storage file (.doc :).
Those alternate views are useful just because they can be accessed using ordinary unix tools, no matter what the obscure codec format provided it's supported. With a bit of effort a standard presentation for certain kinds of data may arise.
(5) There is also data which is _calculated_ from the flat file or its extra metadata, and which the kernel guarantees is either consistent with the file, not set at all, or at least is invalidated if the file is changed. It may be calculated on demand through the kernel interface, or applications may simply have to calculate it themselves when they don't have an up to date value.
For example MD5 or SHA1 digests of the flat file, or of its parts. The key thing is that these are always up to date, not dependent on weak tests like mtime.
Partial checksums for speeding up rsync and other delta algorithms. Computed thumbnails, MP3 average power levels, character encoding extracted from parsing old HTML META tags. Anything really that can be calculated and usefully cached.
This is similar to (4), but I'm emphasising things that may take a while to compute, may be cached with the filesystem itself helping to decide when to evict, and writing them won't affect the data they're calculated from, if writing is allowed at all.
The only really important of these are MD5 or SHA1 digests, or some other way of tracking modifications like a strong serial number.
All other computed cached data can be maintained by applications elsewhere, if they know when the data has been modified since the last time. (Though there are advantages to making the data disappear if the file is deleted, or move if it's renamed).
The thing with these different categories is that we want different behaviour when serialising the data for a backup or to copy it.
We probably always (1) to be copied.
(2) is questionable: it's like unix file permissions, you might want to copy it; you might not.
(3) Shouldn't be copied or stored except for special reasons. It's a bit like /proc: it's not meant to be stored on your backup CD and it's definitely not meant to be written during a restore. :)
(4) Should never be copied if you're copying the flat file, because it's exactly the same information, just accessed differently. You might want to copy it by itself for some reason though, and you certainly would like to work with it from scripts and applications. These streams are special in another way: they might themselves decompose into further streams. Just think of a AVI file containing an OGG file containing a FLAC containing metadata and data streams, or something like that. They happen.
(5) shouldn't be copied if you're transmitting a file, as a matter of data integrity among other things. You might want it on the backup, then again maybe only some of the values. Probably never the crypto digests.
To summarise: most data inside a directory-as-file should _not_ usually be copied and transported. A little of it, (1), should be copied if the serialised form will have it, and some of the other metadata should be copied under special circumstances.
I've suggested that we could simply not have (1), and recommend that only reproducible, non-essential data appears inside the directory. Everything essential would be part of the serialised, flat file.
However, you seem to like (1), metadata which should be serialised and copied around, and which is not part of the flat file.
Note that (1) will only be copied around on systems which handle "extra forks" anyway -- things like Samba and MacOS compatibility. FTP and HTTP obviously won't copy the metadata of (1), although HTTP WebDAV might allow it to be visible as properties.
All that suggests a common structure for the directory inside a "container" -- one where the metadata to be copied by default is clearly distinguished from the metadata which should never by copied, or which is cached, or which is a view. Possibly it should be distinguished by name, although permission bits or (not sure I like this) meta-metadata could serve the same role without being name dependent.
-- Jamie
[reposted in thread]
Quoted 11 lines · Linus Torvalds
Note that we could try this out with existing filesystems with very minimal changes:
- make directory bind mounts work on top of files ("
graft_tree()")
- makeopen_namei()and friend _not_ do the mount-point following for the last component if it's a non-directory.
- probably some trivial fixups I haven't thought about. There might be some places that use "S_ISDIR()" to check for whether something can be looked up, but the main path walking already just checks whether there is a ".lookup" operation or not.
Quoted 4 lines
This would already allow people to "try out" how different applications would react to a file that can show up both as a directory and a file. The patch might end up being less than 25 lines or so, the difficulty is in finding all the right places.
The real issue is what to do with unlink() et.al. on these guys. Note that "unlink is OK if all we have there is a bunch of directory mounts" won't work well - we have no good way to check that condition.
Even funnier one is what we do if we have directory mounted there *and* have something mounted on stuff in that directory.
Yes, that's one of the probable directions for such stuff, but there's a lot of fun semantics questions and answers to them will matter a lot.
Hey, if we lose the "can't unlink/rmdir/rename over something that is a mountpoint in other life" - I'm happy and we can get a lot of much more interesting stuff to work. It will take some work (e.g. making sure we can find all vfsmounts over given mountpoint and sorting out the locking issues, which won't be trivial), but the main obstacle in that direction is not in architecture - it's in SuS and tradition; as the matter of fact, our life would be much easier if we stopped trying to give -EBUSY here and just dissolved all subtrees mounted on anything that has that dentry.
Am Donnerstag, den 26.08.2004, 14:59 -0400 schrieb Rik van Riel:
Quoted 6 lines
open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755)= 4What do we do with
O_CREAT?Do we always allow both a directory and a file to be created with the same name ?
I would say that the directory under a file is implicit. There are still the pseudo-files a filesystem might want to provide or the VFS if there should be an agreement to generally provide file/pseudo/uid or possibly some security hooks. Some filesystem might want to provide those pseudo files but doesn't have the technical possibility to store non-pseudeo files from the user under such a directory.
Quoted 10 lines · Rik van Riel
Hmmm, I just straced "cp
/bin/bash/tmp".
One line stood out as a potential problem:
open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755)= 4What do we do with
O_CREAT?Do we always allow both a directory and a file to be created with the same name ?
Either I am confused, or you are.
To me, a filesystem that allows this thing doesn't really _have_ the concept of "directory vs file". It's just a "filesystem object", and it can act as _both_ a directory and a file.
So when you create "/tmp/bash" - assuming /tmp supports the file-as-dir semantics at all, you're just creating a new inode - the same you always have. When you write to that inode, it writes to the default stream. There's no special cases here.
Now, after you have your regular /tmp/bash, you can then start adding named streams to it, ie you can do
open("/tmp/bash/icon", O_WRONLY|O_CREAT|O_LARGEFILE, 0755);and that will create the "icon" named stream. See?
So "/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part).
Hey, think of it as a wave-particle duality. Both "modes" exist at the same time, and cannot be separated from each other. Which one you see depends entirely on your "experiment", ie how you open the file.
Quoted 2 lines
Does this create a new class of "symlink attack" style security holes ?
I don't believe so.
Linus
Quoted 12 lines · Hans Reiser
Quoted 5 lines · Andrew Morton
And describe the "plugin" system. Why does the filesystem need such a thing (other filesystems get their features via `patch -p1')?
It takes 6 months or more to become competent to change a usual filesystem. Creating a new reiser4 plugin is a weekend programmer fun hack to do. Weekend programmers matter, because they tend to have
clever ideas based on understanding a need they have. How many people
can easily add new features to ext3 or reiserfs V3? Very few.
Quoted 8 lines
What happens if you need a disk format change?
Well, in V4, you can easily compose a plugin from plugin methods of other plugins, write a little piece of code with the one thing you want different, and add it in. Disk format changes, no big deal, add a new disk format plugin, or a new item plugin, or a new node plugin, etc., and you got your new format.
OK, real world example. My roommate has an AKAI MPC-2000, a very popular hardware sampler from the 90's. The disk format is known,there are a few utilities to edit the disks on a PC and extract the PCM samples, but there are no tools to mount it on a modern PC. Are you saying that, since I know the MPC disk format, I could write a reiser4 plugin to mount an MPC drive?
If so, then Hans has an excellent point. Users do want this kind of thing, and it is worth having to fix tar et al.
Lee
Quoted 4 lines · Rik van Riel
What do we do with
O_CREAT?Do we always allow both a directory and a file to be created with the same name ?
I haven't analysed it thoroughly. I suspect it's fine as long as:
(1) O_CREAT creates something with "file-like attributes", meaning stat() says it's a regular file.
(2) File-like means it can be unlinked, linked and renamed, even if someone has something inside it open. Nothing that can be created inside it will prevent it from being unlinked (unlike a directory-like object, where rmdir() will refuse if it's not empty).
File-like also means that programs like "cp from to_my_new_file" won't do anything so silly as to write _inside_, the way that "cp from to_my_new_file/from" would.
Basic utilities will need to be checked to make sure they don't try appending "/" as their method of deciding if the target object is a directory and should be entered.
Quoted 2 lines
Does this create a new class of "symlink attack" style security holes ?
Yes, but they don't need O_CREAT.
An adversary creates a symlink to metadata inside your file. You write to it: it has interesting effects that weren't anticipated, such as either modifying another (virtual) file, or altering permissions or other parameters which writing doesn't normally do.
This is very difficult to prevent.
We are creating a way for scripts and classic unix tools to have easy access to fancy attributes which may affect the security: things like being able to change a file's permissions just by writing to the appropriate path, or reading characteristics of files that shouldn't be visible. In Hans Reiser's example of expanded /etc/passwd, atimes and mtimes of individual passwd entries is security information that perhaps shouldn't be exposed.
In a way, these holes are similar to /proc, in that just writing has side effects which might not be acceptable.
The solution is the same as for /proc (I hope): make sure the read permissions on all metadata inside a directory branch are restricted to the permissions of the file branch, and write permissions even more restricted at least by default.
For the paranoid, one way is to make new files "not readable or executable as directories". It's possible for the file and directory branches to have different permission bits, though I could see that getting a bit annoying. Other possibilities abound.
-- Jamie
Quoted 3 lines · Jamie> Rik van Riel wrote:
Quoted 3 lines
And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?
Jamie> Yes, exactly that. The streams are created on demand of Jamie> course, and by userspace helpers when that's appropriate which Jamie> I suspect it almost always is.
So how would a program that converts between a JPEG file (with exif data) and a PNG work, such as ImageMagick? Are we proposing to teach the VFS (or worse yet each filesystem) how to do this?
I've been following this discussion a bit and I'm not sure that I've actually seen any concrete examples of where this is a *good* thing to have. People talk about only having to modify 20 bytes at a time instead of reading and writing 1mb of data. Isn't that what mmap() does?
Now I can sorta understand the idea that having a directory look like a file is neat, and certainly simplifies some aspects, but I think that going all the way down to the logical conclusion here is a bit silly.
To use the principle of blowing things up to make them very large or very small, what happens if I decide that the best idea is to make all files just be directories which contain single byte files? Isn't that the logical extreme here? So my 1mb JPEG file is not just some image data and header info in multiple files, but it's really just 1 million (ok 1024 * 1024) individual files that the VFS knows how to put together. Seems like the logical extreme. Oh wait, maybe we should be exposing a single file per bit instead!
Quoted 1 line
From my point of view, there lies madness. As Rik pointed out, how do
backup and restore tools work with this stuff? Most people could care less about how their data is organized, but they certainly care when they can't restore it from backups.
I'd really like to see a concrete example from Hans or other proponents about why this makes things easier/faster/better to do. Mostly, I've just seen "proof by vigorous handwaving" that it's a good thing.
In alot of ways, I think people are going in the wrong direction, you want to excapsulate and hide the details more, not expose them. That's what a good API does, it hides the details while giving you a rich set of semantics to manage your data.
God knows I'm not smart enough or driven enough to actually come up with my own ideas, but I certainly haven't seen anyone else (even Linus) come up with an earth shattering arguement to say why this is the right move to make.
As Linus says, most of the OS's job is to mediate access to objects/data. Why do we want to expose such low level data then?
John
John Stoffel - Senior Unix Systems Administrator - Lucent Technologies
[email protected] - http://www.lucent.com - 978-952-7548
Am Donnerstag, den 26.08.2004, 15:26 -0400 schrieb Lee Revell:
Quoted 6 lines
Quoted 5 lines
Well, in V4, you can easily compose a plugin from plugin methods of other plugins, write a little piece of code with the one thing you want different, and add it in. Disk format changes, no big deal, add a new disk format plugin, or a new item plugin, or a new node plugin, etc., and you got your new format.
Quoted 6 lines
OK, real world example. My roommate has an AKAI MPC-2000, a very popular hardware sampler from the 90's. The disk format is known,there are a few utilities to edit the disks on a PC and extract the PCM samples, but there are no tools to mount it on a modern PC. Are you saying that, since I know the MPC disk format, I could write a reiser4 plugin to mount an MPC drive?
No, the underlying storage must be a reiser4-like tree. If you want to mount an MPC drive, write an MPC filesystem.
Quoted 9 lines · Lee Revell
OK, real world example. My roommate has an AKAI MPC-2000, a very popular hardware sampler from the 90's. The disk format is known,there are a few utilities to edit the disks on a PC and extract the PCM samples, but there are no tools to mount it on a modern PC. Are you saying that, since I know the MPC disk format, I could write a reiser4 plugin to mount an MPC drive?
If so, then Hans has an excellent point. Users do want this kind of thing, and it is worth having to fix tar et al.
You don't need reiser4 for that, writing read-only linux filesystems is trivial as soon as you have a specification of the ondisk format.
On Thu, Aug 26, 2004 at 09:28:41PM +0300, Denis Vlasenko wrote:
|
| Is it possible to sufficiently hide "dirs inside files"
| so that old tools will be unable to see them?
|
| I just checked:
|
| ls -d /foo does lstat64("/foo", ...)
| ls -d /foo/ does lstat64("/foo", ...)
| but
| ls -d /foo/. does lstat64("/foo/.", ...)
|
| Will it work out if "dir inside file" will only be visible when referred as "file/."?I'm used to using ls symlink/. to get ls to show me the directory on the far side of a symbolic link. That's a pretty analagous case to the one we're discussing here, I think?
Jon
Signature · 4 lines
------------------------------------------------------------------------------- Jonathan Abbey [email protected] Applied Research Laboratories The University of Texas at Austin GPG Key: 71767586 at keyserver pgp.mit.edu, http://www.ganymeta.org/workkey.gpg
Quoted 5 lines · Linus Torvalds
Quoted 2 lines
It is not the kernel which decides. The filesystem containing
/dev/hda/part1opens "the directory branch".But that filesystem cannot know what the _other_ filesystem configurations are. And that's what you'd have to have to mount.
At which point, userspace comes in. Who knows what userspace chooses.
Quoted 5 lines
Quoted 2 lines
The obvious implementation has the userspace helper just mounting it, end of story. If the mount command fails, it fails. Much like autofs.
Yes, that would work, but it's of questionable use. If you want autofs, then just _use_ autofs.
I don't mean to imply it's useful behaviour: only that it seems to happen by default if you don't do anything special to prevent it, if you see what I mean.
It does seem useful to have _files_ mounted automagically (using loopback devices): for example, cd into a .iso file and it would be nice to see the contents. At this point, deciding on the preferred -EBUSY or not behaviour becomes relevant, as does deciding whether the .iso in this example is viewed using the kernel's isofs code, or using a userspace helper in the same way as we'd use one to browse a .zip file.
-- Jamie
Am Donnerstag, den 26.08.2004, 20:29 +0100 schrieb Jamie Lokier:
Quoted 7 lines
(1)
O_CREATcreates something with "file-like attributes", meaningstat()says it's a regular file.(2) File-like means it can be unlinked, linked and renamed, even if someone has something inside it open. Nothing that can be created inside it will prevent it from being unlinked (unlike a directory-like object, where
rmdir()will refuse if it's not empty).
I would say hat the file and all of its directory gets invalidated and deleted as soon as the last opener of the file's main stream or some of its directory content disappears. How would that be?
Quoted 9 lines
Quoted 2 lines
Does this create a new class of "symlink attack" style security holes ?
Yes, but they don't need
O_CREAT.An adversary creates a symlink to metadata inside your file. You write to it: it has interesting effects that weren't anticipated, such as either modifying another (virtual) file, or altering permissions or other parameters which writing doesn't normally do.
Hmm.
Quoted 1 line
This is very difficult to prevent.
Right. But to change some properties you usually need to be the owner of that file anyway, or root.
Quoted 3 lines
In Hans Reiser's example of expanded
/etc/passwd, atimes and mtimes of individual passwd entries is security information that perhaps shouldn't be exposed.
Only assuming mtime/ctime/atime/rwx/uid/gid of file contents can actually be independant of those from the main stream. I would say that the rights (read and write) for non-pseudo sub-files should the same as the file, uid and gid 644 (only owner can change these), something like that. mtime, atime and ctime always show the same as the file itself. Hmm.
Quoted 4 lines
The solution is the same as for /proc (I hope): make sure the read permissions on all metadata inside a directory branch are restricted to the permissions of the file branch, and write permissions even more restricted at least by default.
That's what I was thinking.
Quoted 3 lines · Linus Torvalds
So "
/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part).
OK, that makes sense.
Quoted 3 lines
Hey, think of it as a wave-particle duality. Both "modes" exist at the same time, and cannot be separated from each other. Which one you see depends entirely on your "experiment", ie how you open the file.
Guess I'm scared again now. We need to make sure that backup programs don't fall victim to the uncertainty principle ;)
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 8 lines · Christoph Hellwig
Quoted 7 lines · Lee Revell
OK, real world example. My roommate has an AKAI MPC-2000, a very popular hardware sampler from the 90's. The disk format is known,there are a few utilities to edit the disks on a PC and extract the PCM samples, but there are no tools to mount it on a modern PC. Are you saying that, since I know the MPC disk format, I could write a reiser4 plugin to mount an MPC drive?
Quoted 6 lines
Quoted 2 lines
If so, then Hans has an excellent point. Users do want this kind of thing, and it is worth having to fix tar et al.
You don't need reiser4 for that, writing read-only linux filesystems is trivial as soon as you have a specification of the ondisk format.
Of course I could just write an MPC filesystem driver. And, like Andrew said, a filesystem normally gets new features with 'patch -p1'. My question, which was answered by a previous post, was whether the same could be done in a reiser4 plugins.
Lee
Am Donnerstag, den 26.08.2004, 15:44 -0400 schrieb Rik van Riel:
Quoted 7 lines
Quoted 3 lines
Hey, think of it as a wave-particle duality. Both "modes" exist at the same time, and cannot be separated from each other. Which one you see depends entirely on your "experiment", ie how you open the file.
Guess I'm scared again now. We need to make sure that backup programs don't fall victim to the uncertainty principle ;)
At least you can be sure that the backup will go into a defined state as soon as you observe it.
Quoted 24 lines · John Stoffel
Quoted 3 lines · Jamie> Rik van Riel wrote:
Quoted 3 lines
And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?
Jamie> Yes, exactly that. The streams are created on demand of Jamie> course, and by userspace helpers when that's appropriate which Jamie> I suspect it almost always is.
So how would a program that converts between a JPEG file (with exif data) and a PNG work, such as ImageMagick? Are we proposing to teach the VFS (or worse yet each filesystem) how to do this?
I've been following this discussion a bit and I'm not sure that I've actually seen any concrete examples of where this is a *good* thing to have. People talk about only having to modify 20 bytes at a time instead of reading and writing 1mb of data. Isn't that what
mmap()does?Now I can sorta understand the idea that having a directory look like a file is neat, and certainly simplifies some aspects, but I think that going all the way down to the logical conclusion here is a bit silly.
Quoted 8 lines
To use the principle of blowing things up to make them very large or very small, what happens if I decide that the best idea is to make all files just be directories which contain single byte files? Isn't that the logical extreme here? So my 1mb JPEG file is not just some image data and header info in multiple files, but it's really just 1 million (ok 1024 * 1024) individual files that the VFS knows how to put together. Seems like the logical extreme. Oh wait, maybe we should be exposing a single file per bit instead!
It is doable. It doesn't mean we shall do this.
I think some reiser4 supporters try to do too much too soon. It is more sensible to do it in small steps.
I think we can start small and make all file metadata to be accessible via file/meta/{uid,gid,mode,mtime,atime,...}.
Normal tools will be unable to see file/meta/ because file is not a directory and file/ is not a directory (i.e. open(O_DIRECTORY) will fail on them). Hardlinking to file/meta/* is not allowed.
However, file/. _is_ a directory. We can make a tar-like
tool which can do its work and save/restore metadata along with data by just tarring/untarring file/meta/*. Notice how this 'new tar' does not need to know about exact kind of metadata supported by fs. You can add ACLs to the fs, and this 'new tar' will be able to save/restore ACLs. Without any modufications.
Contrast this with existing tar which has hardcoded knowlendge about uid,gid,mode,etc...
--
vda
Quoted 5 lines · Joel Becker
Quoted 1 line · Hans Reiser
Yes, this was part of the plan, tar file-directory plugins would be cute.
Question: Is "cat /foo/bar/baz.tar.gz/metas" the attribute
directory or a directory in the tarball named "metas"?
This needs to be designed.
Perhaps /foo/bar/baz.tar.gz/tar/metas is the directory in the tarball named "metas".
Or perhaps /foo/bar/baz.tar.gz/x/metas is: it's independent of archive format, and I personally tend to extract things into a directory called "x". [*]
Or perhaps /foo/bar/baz.tar.gz/metas is, and the attribute directory is /foo/bar/baz.tar.gz/../metas, to be perverse ;)
I prefer the second one, ("x/metas"), but not with any conviction.
-- Jamie
[*] Actually I prefer:
/foo/bar/baz.tar.gz/content/metas /foo/bar/baz-0.01.tar.gz/content/baz-0.01/metas
Archives always in "content". One layer of decompression always tried for .tar files and other uncompressed archive formats.
/foo/bar/baz.tar.gz/x -> content/ /foo/bar/baz-0.01.tar.gz/x -> content/baz-0.01/
If the root of the archive contains a single directory, "x" is a symlink to it. Otherwise "x" is a symlink to the root directory of the archive. This is comfortable with the common practice by which archives are distributed, without making a mess when someone forgets to put everything in a top-level directory.
Quoted 3 lines · Markus
Quoted 1 line
2) how do standard unix utilities handle them ?
Most likely they don't ;) That is, until they are fixed or replaced.
Not good, in my view.
Signature · 3 lines
I won't rest till it's the best ...
Programmer, Linux Scalability
Paul Jackson <[email protected]> 1.650.933.1373Quoted 1 line
Don't move reiser4 into vfs, use reiser4 as the vfs.
Ah ... that seems to be a good insight into what you're driving for. That would seem to be good for you ... and bad for other filesystems. So whilst I can see why you'd want it - but I don't think it's a good idea. Competition is good.
Quoted 2 lines · Andrew
And what are the licensing implications of plugins? Are they derived works? Must they be GPL'ed?
You replied:
Quoted 4 lines
Quoted 2 lines
And what are the licensing implications of plugins? Are they derived works?
Yes.
I find it odd that you cut out the last part of his question. So must they be GPL'ed or not?
M.
Quoted 3 lines · Linus Torvalds
So "
/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part).
Thinking about it some more, how would file managers and file chosers handle this situation ?
Currently the user browses the directory tree and when the user clicks on something, one of the following happens:
1) if it is a directory, the file manager/choser changes into that directory
2) if it is a file, the file is opened
Now how do we present things to users ?
How will users know when an object can only be chdired into, or only be opened ?
For objects that do both, how does the user choose ?
Do we really want to have a file paradigm that's different from the other OSes out there ?
What happens when users want to transfer data from Linux to another system ?
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 7 lines · John Stoffel
Jamie> Yes, exactly that. The streams are created on demand of Jamie> course, and by userspace helpers when that's appropriate which Jamie> I suspect it almost always is.
So how would a program that converts between a JPEG file (with exif data) and a PNG work, such as ImageMagick? Are we proposing to teach the VFS (or worse yet each filesystem) how to do this?
No. (1) That's what "userspace helpers" are for. (2) What does image format conversion have to do with viewing files in their component parts? (3) I suspect someone will write a plugin that does indeed convert virtually every known image format to a common format (probably PNG and something "raw") at some point. Why not? It's just a small script (it would run ImageMagick!).
Quoted 5 lines
I've been following this discussion a bit and I'm not sure that I've actually seen any concrete examples of where this is a *good* thing to have. People talk about only having to modify 20 bytes at a time instead of reading and writing 1mb of data. Isn't that what
mmap()does?
Sure, if you think mmap() is an easy substitute for "parse the author & license tags from this [unknown format] audio, video or font file".
Quoted 5 lines
Now I can sorta understand the idea that having a directory look like a file is neat, and certainly simplifies some aspects, but I think that going all the way down to the logical conclusion here is a bit silly.
Quoted 8 lines
To use the principle of blowing things up to make them very large or very small, what happens if I decide that the best idea is to make all files just be directories which contain single byte files? Isn't that the logical extreme here? So my 1mb JPEG file is not just some image data and header info in multiple files, but it's really just 1 million (ok 1024 * 1024) individual files that the VFS knows how to put together. Seems like the logical extreme. Oh wait, maybe we should be exposing a single file per bit instead!
If you decide to do that, you are welcome to write the userspace helper which creates that view in your directory.
Why not? It's very silly, but it's no sillier then writing a program to read your 1mb JPEG and write it out one file per bit, which you're welcome to write right now.
Quoted 4 lines
Quoted 1 line
From my point of view, there lies madness. As Rik pointed out, how do
backup and restore tools work with this stuff? Most people could care less about how their data is organized, but they certainly care when they can't restore it from backups.
Generated views are something which should _not_ be backed up and restored. That's not what they are for.
Auxiliary metadata, such as author info and signatures which cannot be stored in the main file for some reason -- that should be backed up.
Permissions, that should be backed up.
But not views which are computed from the main file. You don't need to back them up, and they don't need to take any real space. They're virtual, just like an infinitely deep directory on a web server can be virtual. You can make those very silly too, if you want to.
(Some help from the filesystem with storing temporarily cached values is fine, for performance, but we shouldn't pretend that generated views are anything other than virtual).
Quoted 9 lines
I'd really like to see a concrete example from Hans or other proponents about why this makes things easier/faster/better to do. Mostly, I've just seen "proof by vigorous handwaving" that it's a good thing.
In alot of ways, I think people are going in the wrong direction, you want to excapsulate and hide the details more, not expose them. That's what a good API does, it hides the details while giving you a rich set of semantics to manage your data.
Would you like to propose an alternate API? I was under the impression that we were discussing one, right here in this thread. It's called POSIX with an extension where you can cd into a file. It has a rich set of semantics to manage your data, and works with a lot of familiar programs.
A better one would be welcome, if you have an idea.
Quoted 4 lines
God knows I'm not smart enough or driven enough to actually come up with my own ideas, but I certainly haven't seen anyone else (even Linus) come up with an earth shattering arguement to say why this is the right move to make.
Come up with a better one otherwise it's right ;)
Quoted 2 lines
As Linus says, most of the OS's job is to mediate access to objects/data. Why do we want to expose such low level data then?
Eh? In this case the OS is mediating by providing a uniform interface between programs that access the data and programs that handle file formats.
What is this "low level" data you speak of? I would say something like the "designer" of a font, or the "from address" and "subject" of an email are very high level abstractions, and that's among the sorts of things which are to be exposed by these interfaces.
-- Jamie
Quoted 6 lines · Jonathan Abbey
| Will it work out if "dir inside file" will only be visible when referred as "file/."?
I'm used to using ls symlink/. to get ls to show me the directory on the far side of a symbolic link. That's a pretty analagous case to the one we're discussing here, I think?
By the way, do symlinks have metadata? Where do you find it? :)
-- Jamie
Am Donnerstag, den 26.08.2004, 16:10 -0400 schrieb Rik van Riel:
Quoted 6 lines
Quoted 3 lines
So "
/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part).Thinking about it some more, how would file managers and file chosers handle this situation ?
I would say they don't. That's where metadata and pseudo-files sit. It's up to the applications to do something useful with the data.
Quoted 2 lines
Do we really want to have a file paradigm that's different from the other OSes out there ?
Some other OSes have similar things.
Quoted 2 lines
What happens when users want to transfer data from Linux to another system ?
No one should store really important data inside file/ that makes the file completely useless. I just love these postscript fonts on the Mac. The actual main file stream is 0 bytes in size. The actual font is inside the resource fork. When I want to convert it the Mac user needs to pack it into a compound file using a special tool and then I can decode it using another tool to access the PFB file. Crap.
It's useful for metadata like keywords, ACLs and these things. If you can pack it using an aware backup program, fine. If you don't, you still have the data.
Quoted 5 lines · Christophe Saout
I just love these postscript fonts on the Mac. The actual main file stream is 0 bytes in size. The actual font is inside the resource fork. When I want to convert it the Mac user needs to pack it into a compound file using a special tool and then I can decode it using another tool to access the PFB file. Crap.
I thought it was a cunning way to stop me from copying those nice fonts onto my Linux box. The original DRM. Thanks for the tip :)
-- Jamie
Am Donnerstag, den 26.08.2004, 21:16 +0100 schrieb Jamie Lokier:
Quoted 8 lines
Quoted 6 lines
| Will it work out if "dir inside file" will only be visible when referred as "file/."?
I'm used to using ls symlink/. to get ls to show me the directory on the far side of a symbolic link. That's a pretty analagous case to the one we're discussing here, I think?
By the way, do symlinks have metadata? Where do you find it? :)
Oops. :)
At least in reiser4 they don't have, or at least you can't access them. ln -s foo bar; cd bar/metas shows me the content of foo/metas.
symlinks are special anyway. Their rights are 777. The only thing they can have is an owner. No chance to do a symlink/metas/uid then. Hmm.
Hello,
Quoted 9 lines · Lee Revell
Quoted 7 lines · Hans Reiser
Well, in V4, you can easily compose a plugin from plugin methods of other plugins, write a little piece of code with the one thing you want different, and add it in. Disk format changes, no big deal, add a new disk format plugin, or a new item plugin, or a new node plugin, etc., and you got your new format.
Quoted 7 lines
OK, real world example. My roommate has an AKAI MPC-2000, a very popular hardware sampler from the 90's. The disk format is known,there are a few utilities to edit the disks on a PC and extract the PCM samples, but there are no tools to mount it on a modern PC. Are you saying that, since I know the MPC disk format, I could write a reiser4 plugin to mount an MPC drive?
Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4 plugins? From Hans' words it seems so. If this is correct, then maybe reiser4 core should be updated to completely replace current VFS layer? Then it's a good point to create a branch (in old development model it would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and rewrite all (or at least most used) FS as reiser4 plugins. Then everybody will be happy.
But this looks too good to be true. Perhaps I misunderstood Hans' words aboud 'new disk format', did I?
Signature · 2 lines
With best wishes Dmitry Baryshkov
Quoted 9 lines · On Thu, Aug 26, 2004 at 08:13:23PM +0100…
Hey, if we lose the "can't unlink/rmdir/rename over something that is a mountpoint in other life" - I'm happy and we can get a lot of much more interesting stuff to work. It will take some work (e.g. making sure we can find all vfsmounts over given mountpoint and sorting out the locking issues, which won't be trivial), but the main obstacle in that direction is not in architecture - it's in SuS and tradition; as the matter of fact, our life would be much easier if we stopped trying to give
-EBUSYhere and just dissolved all subtrees mounted on anything that has that dentry.
Argh... OK, now I remember why I went for -EBUSY for unlink() (we obviously are not bound by SuS on that one). Consider the following scenario:
* local file foo got something else bound on it for a while
* we are tight on space - time to clean up
* oh, look - contents of foo is junk
* rm foo
* ... oh, fuck, there goes the underlying file.
Quoted 9 lines · Rik van Riel
Quoted 4 lines · Denis Vlasenko
Quoted 1 line
I like cat < a > b. You can keep your progress.
cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions.
Losing permissions is one thing. Annoying, mostly.
cat <a >b does preserve permissions - the permissions of b. I always considered this a feature.
Lee
Quoted 8 lines · Rik van Riel
Quoted 5 lines · Linus Torvalds
So "
/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part).
Quoted 10 lines
Thinking about it some more, how would file managers and file chosers handle this situation ?
Currently the user browses the directory tree and when the user clicks on something, one of the following happens:
1) if it is a directory, the file manager/choser changes into that directory
How does the file manager / chooser decide whether you're trying to move into a directory, or the meta-data-directory for a directory?
It's not just files that should have metadata - directories need* them too. Making it possible to see attributes as a directory under a file is great, but you'd still need an O_META flag for accessing directory metadata (since there are already files under a directory).
Quoted 4 lines
2) if it is a file, the file is opened
Now how do we present things to users ?
Quoted 9 lines
How will users know when an object can only be chdired into, or only be opened ?
For objects that do both, how does the user choose ?
Do we really want to have a file paradigm that's different from the other OSes out there ?
MacOS does, Be did (sort of). I'm not sure it would be the end of the world, as long as the data can be extracted.
Quoted 4 lines
What happens when users want to transfer data from Linux to another system ?
That would depend on the other system. Data is easy, metadata is hard.
It would be possible to create an XML schema for metadata, and if requested (O_EVERYTHING?), the file data is returned with all metadata in XML tags. (not advocating this, just an idea :)
- Steve
* I say need in the same way as one *needs* to upgrade their 2GHz computer - it would be nice :)
Am Donnerstag, den 26.08.2004, 21:32 +0100 schrieb :
Quoted 7 lines
Argh... OK, now I remember why I went for
-EBUSYforunlink()(we obviously are not bound by SuS on that one). Consider the following scenario:
* local file foo got something else bound on it for a while
* we are tight on space - time to clean up
* oh, look - contents of foo is junk
* rm foo
* ... oh, fuck, there goes the underlying file.
Right. Good thinking. You can't delete a directory while there regular files in it either.
Another question: /mnt/test is a mountpoint. Why can I rename mnt' but can't rename test'?
Am Freitag, den 27.08.2004, 00:30 +0400 schrieb Dmitry Baryshkov:
Quoted 10 lines
Another example: Can ext2/etx3/reiserfsv3/xfs be implemented as reiser4 plugins? From Hans' words it seems so. If this is correct, then maybe reiser4 core should be updated to completely replace current VFS layer? Then it's a good point to create a branch (in old development model it would be 2.7, dunno for new :), replace VFS layer with reiser4 core, and rewrite all (or at least most used) FS as reiser4 plugins. Then everybody will be happy.
But this looks too good to be true. Perhaps I misunderstood Hans' words aboud 'new disk format', did I?
No. You can change the format the reiser4 storage tree is stored in. As long as other filesystems don't use the same underlying storage tree this is not possible.
Quoted 8 lines · Will Dyson
Quoted 6 lines · Jamie Lokier
However, as far as I know it's not accessible in a file-as-directory form as yet. In my opinion that is the most natural form and it would be very intuitive to use. I hope we can pick a useful semantics for them, and also provide filesystem-independent plugins with GNU Hurd-like per-user extensibility.
Quoted 9 lines
Quoted 6 lines
-- Jamie
* plenty == too much.
Gnome, KDE, Emacs and Bash all see different virtual filesystems. (All but Bash implement their own virtual filesystem extensions). That makes them much less useful than they could be.It has always bugged me that Gnome and KDE implement their own VFS layers.
Same here. This always seemed like something the kernel should be able to handle. It seems to me that if reiser4 had been available at the time the Gnome and KDE developers would not have needed to do this.
Lee
Quoted 3 lines · Rik van Riel
Thinking about it some more, how would file managers and file chosers handle this situation ?
The same way they already handle it on other platforms that support it? By taking advantage of it..
People literally use this for icon and preview information, so some of the stuff shows up very much in file managers. And I'd assume that a normal double-click would just open the main file, and you can right-click for management information, including opening the file.
If you want an entity that acts as a directory, you create a directory.
Directories don't go away - you still have S_ISDIR() and S_ISREG(), and they still return information. A file that has associated information is still a _file_, and people should treat it that way, it's just that it also has a list of named sub-streams. You can open it as a directory, but the stat information clearly says it is a file, and the "directory" view is very much associated with _that_ file.
I definitely don't think you want the file manager to act as if a named stream is exactly the same as a stand-alone file. They have all the same operations, but there's no question that there are differences too. Especially on a conceptual level - but for most filesystems there are likely real technical differences too.
For example, it's likely that most filesystems would _not_ allow linking of a named stream anywhere else. And you might not be able to change the permissions or date on the named stream either, since it may or may not have a separate date/permission thing from the container.
So don't believe that just because the named streams are _named_ like real files, that they suddenly have any existence beyond the container that they are part of.
There may be other limitations too - again depending on how the filesystem actually implements named streams. It might not support more than one level of naming, for example - so you might not be able to create a directory structure within the named streams, for example.
In short: the fact that the VFS layer exposes the _capability_ to see the named streams as a full POSIX filesystem of its own does _not_ mean that the low-level filesystem necessarily allows the full possible semantics. So you shouldn't design your apps that know about named streams to think they are normal directories.
The directory thing is just a very powerful naming scheme, and one that fits into the existing UNIX model.
Quoted 2 lines
Do we really want to have a file paradigm that's different from the other OSes out there ?
Different from what other OSes?
Last I looked, Windows, Solaris, and OSX all supported named streams. What other OS's are out there that you care about?
In other words, this is _not_ a different paradigm from what others do. The discussion is whether we want to implement it at all, and more importantly about syntactic issues (ie we clearly already implement extended attributes, just with a much more limited syntactic power).
We don't have to go all the way. Solaris has "openat()", which is kind of a half-way there - not really directly available in the same namespace, but at least the result is available as a real file interface (as opposed to the Linux "xattr" interfaces that are _totally_ special-cased system calls).
In other words - the paradigm is already there. It's just that currently it's pretty much unusable under Linux, because it requires so much specialized knowledge that it's not worth it to modify existing apps. And the interfaces are really very limited, so even if you _do_ end up using the specialized Linux interfaces, you can't actually do a lot of what you might want to do.
Linus
Quoted 26 lines · --On Thursday, August 26, 2004 11:15:19 -0700 Linus Torvalds…
Quoted 4 lines · Rik van Riel
It's a relief to know that nobody's taking my humorous suggestion seriously, but now we still have the "standard Unix tools can't manipulate files" problem...
I disagree. They can manipulate the files a whole lot better than they can manipulate xattr's.
For example, you _could_ probably (but hey, maybe "tar" tries to strip slashes off the end of filenames, so this might not work due to silly reasons like that) back up a compound file with
tar cvf file.tar file file/
although unpacking it would require that tar be taught about the thing. And you definitely could write a script to do the thing, ie even with an unmodified tar you could do
tar cvf file-archive.tar file
cd file
tar cvf ../attribute-archive.tar .which is a hell of a lot better than what you can do with the fsattr interfaces and unmodified legacy applications.
Quoted 6 lines
So one of the advantages of "dir-as-file/file-as-dir" is exactly that you _can_ manipulate the data with legacy tools. Sure, things that traverse a directory tree might need some (likely fairly trivial) modifications if they really want to take advantage of the subfiles, but that's still likely to be _much_ less of an issue than with fsattr's that have a totally different model entirely.
What would "test -d" and "test -f" return on these magic beasties? I can't think of any combinations that wouldn't confuse the crap out of userspace.
M.
Quoted 8 lines · On Thu, 26 Aug 2004 [email protected] wrote:
Argh... OK, now I remember why I went for
-EBUSYforunlink()(we obviously are not bound by SuS on that one). Consider the following scenario:
* local file foo got something else bound on it for a while
* we are tight on space - time to clean up
* oh, look - contents of foo is junk
* rm foo
* ... oh, fuck, there goes the underlying file.
Hey, that's a valid reason for doing -EBUSY for normal bind-mounts, but it actually _is_ what we want for an "implied-by-way-of-container-mount". After all, when you do a "rm foo", you do mean "remove the container foo".
I replied to your earlier off-list mail in private, so let's re-iterate for the list: the easiest way to handle this is to just have a "mount option", and have "MNT_ALLOWUNLINK" that gets set for containers, and that users could possibly choose to set for regular mounts too (as a mount option) if they really want to (and if we want them to).
So there's no reason we'd have to drop existing mount behaviour only because we also have special files that look like mountpoints.
Linus
Quoted 4 lines · Rik van Riel
For objects that do both, how does the user choose ?
Do we really want to have a file paradigm that's different from the other OSes out there ?
What does MacOS X do? Someone said that documents are directories in it; it must know how to handle that.
What does Windows do when you click on a .zip file and WinZIP is installed? It opens the .zip file and lets you explore inside. When you click on a .doc file, though, it opens a viewer or editor -- you don't get the option to look inside.
Quoted 2 lines
What happens when users want to transfer data from Linux to another system ?
This is why I favour storing all essential metadata (about the file's content) inside the file's contents, the primary stream.
We have another problem: what happens when users want to transfer data from Windows (with secondary streams) and MacOS (with resource forks)?
-- Jamie
Hello,
Quoted 5 lines · Christophe Saout
No. You can change the format the reiser4 storage tree is stored in. As long as other filesystems don't use the same underlying storage tree this is not possible.
Thank you for clarification. As I wrote, it's too good to be true.
Signature · 2 lines
With best wishes Dmitry Baryshkov
Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell:
Quoted 5 lines
Quoted 1 line
It has always bugged me that Gnome and KDE implement their own VFS layers.
Same here. This always seemed like something the kernel should be able to handle. It seems to me that if reiser4 had been available at the time the Gnome and KDE developers would not have needed to do this.
Well, the kernel doesn't have a filesystem that speaks http, scp and those things. GnomeVFS is URL-based. It has some pseudo-protocols that extract a pseudo directory-tree for all installed applications + the changes the used made, created on the fly from a set of XML files that are read-only and system-wide and the user-overridden changes. I don't know if all of these things would really make sense inside the kernel.
Quoted 6 lines · Jamie Lokier
This is why I favour storing all essential metadata (about the file's content) inside the file's contents, the primary stream.
We have another problem: what happens when users want to transfer data from Windows (with secondary streams) and MacOS (with resource forks)?
I can tell you that - right now we (Samba) has to drop all but the primary stream, because the Linux filesystems don't have data stream support. Oh look, we're losing user data (that's a *bad* thing :-).
Jeremy.
Quoted 3 lines · Martin J. Bligh
What would "test -d" and "test -f" return on these magic beasties? I can't think of any combinations that wouldn't confuse the crap out of userspace.
"It's a feature".
The S_ISDIR/S_ISREG tests show real information: it shows not only user intent ("you should consider this a file, even if it has attributes"), but also whether it is a directory or a container.
And there's a real technical difference there: the streams contained within a file are bound to that file. The files contained within a directory are _independent_ of that directory. Big difference. HUGE difference.
So it's not confusing. If it tests as a file, you think of it as a file. It may have attributes aka named streams associated with it, and you may be able to open those attributes by treating the file as a directory, but that doesn't really change the fact that it's a file.
The _big_ difference is that when you can make the compound object _look_ like a directory, that means that you can now manage the attributes with standard tools. They are still attributes, though.
Linus
Quoted 8 lines · Christophe Saout
Am Donnerstag, den 26.08.2004, 16:38 -0400 schrieb Lee Revell:
Quoted 5 lines
Quoted 1 line
It has always bugged me that Gnome and KDE implement their own VFS layers.
Same here. This always seemed like something the kernel should be able to handle. It seems to me that if reiser4 had been available at the time the Gnome and KDE developers would not have needed to do this.
Quoted 7 lines
Well, the kernel doesn't have a filesystem that speaks http, scp and those things. GnomeVFS is URL-based. It has some pseudo-protocols that extract a pseudo directory-tree for all installed applications + the changes the used made, created on the fly from a set of XML files that are read-only and system-wide and the user-overridden changes. I don't know if all of these things would really make sense inside the kernel.
True. FWIW, I never use most of those features. It's just too damn slow. Windows seems to implement all of the useful features of GnomeVFS, and they are 10x faster.
Lee
Quoted 3 lines · Lee Revell
Same here. This always seemed like something the kernel should be able to handle. It seems to me that if reiser4 had been available at the time the Gnome and KDE developers would not have needed to do this.
reiser4 has prompted the discussion again, but it doesn't provide the virtual filesystem capabilities that we're talking about, which Gnome and KDE implement in userspace. The FUSE project is much more relevant for that.
We had file-as-directory discussions years ago, and 5 years ago I was experimenting with doing virtual filesystems with that capability over NFS.
podfuk offered something close to Gnome and KDE but in kernel space:
virtual filesytems accessible through the kernel, but it didn't get too popular. It was based on Midnight Commander's VFS (which has evolved into Gnome's VFS), using the CODA filesystem interface to provide the kernel hooks. It had some problems, and was a bit fiddly to use. We still don't have something that's particularly nice to use for ordinary users.
-- Jamie
Quoted 16 lines · Martin J. Bligh
Quoted 3 lines
What would "test -d" and "test -f" return on these magic beasties? I can't think of any combinations that wouldn't confuse the crap out of userspace.
"It's a feature".
The S_ISDIR/S_ISREG tests show real information: it shows not only user intent ("you should consider this a file, even if it has attributes"), but also whether it is a directory or a container.
And there's a real technical difference there: the streams contained within a file are bound to that file. The files contained within a directory are _independent_ of that directory. Big difference. HUGE difference.
Quoted 8 lines
So it's not confusing. If it tests as a file, you think of it as a file. It may have attributes aka named streams associated with it, and you may be able to open those attributes by treating the file as a directory, but that doesn't really change the fact that it's a file.
The _big_ difference is that when you can make the compound object _look_ like a directory, that means that you can now manage the attributes with standard tools. They are still attributes, though.
I think what you're saying is that they'd both return positive, right?
Unfortunately I think lots of code in userspace assumes files / directories are mutually exclusive conditions, and acts accordingly ... eg:
if [ -f file ]
do file stuff
else
do directory stuff
fi.
Now I'm not saying they're not broken ... but seems like a dangerous thing to change in the midst of a stable kernel series.
M.
Quoted 23 lines · Christophe Saout
Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel:
Quoted 8 lines
Quoted 7 lines
Quoted 4 lines
And if you read test.compound (the main stream) you get a special format that contains all the components. You can copy that single stream of bytes to another (reiser4) fs and then access
test.compound/test.txtagain.(To Rik especially), this is the design which more or less satisfies lots of different goals at once.
Quoted 8 lines
And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?
Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work.
Yes, the main compound stream unaware applications would see would just be a writable view of its contents. Probably not trivial to implement but doable.
Quoted 6 lines
It should be a simple format. Something simliar to tar. The worst thing that can happen is people start writing plugins for every existing compound format out there. It should be the other way around, agree on a simple compound format and encourage applications to use this one if the want to use this advantage.
I also don't see why the VFS/Filesystem can't decide that (for example) this tar.gz is so active that instead of storing it as a tar.gz and providing a virtual directory of the contents that it instead stores the directory of the contents and makes the tar.gz virtual (regenerating it as needed or as extra system resources are available)
implementation wise I see headaches in doing this, but conceptually this is just an optimization that could take place in the future if we fine that it's needed.
David Lang
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Am Donnerstag, den 26.08.2004, 14:00 -0700 schrieb Martin J. Bligh:
Quoted 16 lines
Quoted 16 lines
"It's a feature".
The S_ISDIR/S_ISREG tests show real information: it shows not only user intent ("you should consider this a file, even if it has attributes"), but also whether it is a directory or a container.
And there's a real technical difference there: the streams contained within a file are bound to that file. The files contained within a directory are _independent_ of that directory. Big difference. HUGE difference.
So it's not confusing. If it tests as a file, you think of it as a file. It may have attributes aka named streams associated with it, and you may be able to open those attributes by treating the file as a directory, but that doesn't really change the fact that it's a file.
Quoted 5 lines
Quoted 3 lines
The _big_ difference is that when you can make the compound object _look_ like a directory, that means that you can now manage the attributes with standard tools. They are still attributes, though.
I think what you're saying is that they'd both return positive, right?
No. A file is still a file and S_ISREG will thus return false. The application should just try to open it as a directory if it wants to access the assiociated embedded attribute files. If the open fails, well, it knows that the filesystem (or Linux version, whatever) doesn't support this feature.
Quoted 2 lines · Martin J. Bligh
I think what you're saying is that they'd both return positive, right?
No. I'd say that a file would look like a file, even if it has attributes.
It wouldn't show as a directory at all - unless you start looking at attributes. Because it really _is_ a file, and it's "directory aspect" is really nothing but a way to make its named streams visible.
So you really should consider it a perfectly regular file, and so only S_ISREG() will return true, and S_ISDIR() will return false.
Think of it this way: when you add a named attribute to current files using the xattr interfaces, do you start thinking of the file as a directory? No. Even though it actually now is a starting point for finding more information.
It's just that thanks to it's directory aspect, people and apps that _care_ about the attributes suddenly can trivially access them. You can access them in shell scripts, you can access them in programs, you can access them in perl. With no special knowledge necessary.
Remember: in the file-as-directory model, we're always talking about just _one_ object. That one object just happens to have namd streams or attributes or whatever you want to call them associated with it, and you can _see_ them through the normal directory interfaces. But that doesn't really make the object a "directory". It's still a file.
In other words, the "directory" part is just a _view_ into the file. A view that potentially exposes a lot _more_ of the file, but we're still talking about the same file.
In contrast, a S_IFDIR-like _directory_ is something else entirely. When you view the things in that, you aren't looking at data "inside" the directory. You're looking at somethign totally independent.
Linus
Quoted 37 lines · David Masover
Christophe Saout wrote:
| Yes, the main compound stream unaware applications would see would just | be a writable view of its contents. Probably not trivial to implement | but doable.Not the main stream, I think, but we do need a way to make "eight-stream octopus" files portable. But there needs to be a copy(2) syscall, and apps need to start supporting it. After all, copy-on-write was planned for Reiser4, now 4.1*. And don't anyone dare suggest "userland library".
| It should be a simple format. Something simliar to tar. The worst thing
Simple, yes. Similar to tar, no.
How about a serializer plugin? foo.mp3/serialize (say) should return some single stream which contains all of foo.mp3. It could work for directories as well, recursively, at least on the same filesystem -- you'd back up by doing
cp /serialize /backup/20040826
This has two potential problems. One is security -- I want to be able to serialize foo.mp3 without being root, but root doesn't want me to be able to serialize /etc and thus get access to shadow.
The second problem is that in order to actually be used for backup, we need snapshots, which are not done yet*. Currently, it'd have to lock the file/directory (recursively), which you don't want to do to entire live filesystems... So, no massive backups of live systems yet, but definitely useful for sending around mp3s and such.
| that can happen is people start writing plugins for every existing | compound format out there. It should be the other way around, agree on a | simple compound format and encourage applications to use this one if the | want to use this advantage.
Quoted 7 lines
For the format, I vote for whatever format we're using at the storage layer. It'd be a lot faster that way, both in execution time (I think) and in development time (I'm sure). Plus, imagine the restore:
cat /backup/20040826 >
/dev/hda5resizefs_reiser4/dev/hda5
one of the virtual metatags should be something about the method to serialize the data
if you are viewing a tarfile as a directory the VFS/Filesystem shoud know that the link between the whole thing and the contents is tar, and it should be possible for userspace programs to find this out by reading teh appropriate thing inside the virtual directory (and eventually change it as well, which would trigger regeneration of the base object, or at least noting that the current version is wrong and needs to be regenerated)
David Lang
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Martin" == Martin J Bligh <[email protected]> writes:
Quoted 2 lines · Martin> Andrew wrote:
And what are the licensing implications of plugins? Are they derived
Quoted 1 line
works? Must they be GPL'ed?
Martin> [Hans] replied:
Quoted 4 lines
Quoted 4 lines
Quoted 2 lines
And what are the licensing implications of plugins? Are they derived works?
Yes.
Martin> I find it odd that you cut out the last part of his question. So Martin> must they be GPL'ed or not?
If they are derived works of GPL'ed software, then they obviously must be GPL'ed (or else they are illegal). But if the authors obtain an alternate license from Hans/Namesys, then they are bound by the terms of that license. (Of course, this case would most likely apply only to Reiser4 running under a different operating system.)
Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Quoted 5 lines · David Lang
I also don't see why the VFS/Filesystem can't decide that (for example) this tar.gz is so active that instead of storing it as a tar.gz and providing a virtual directory of the contents that it instead stores the directory of the contents and makes the tar.gz virtual (regenerating it as needed or as extra system resources are available)
Absolutely. It could keep both views, if they're both actively being used. Or more than both, if there are more. (You could think of compression being an alternate view, and both compressed and uncompressed may as well remain on disk if there's space and it's being actively accessed.
Quoted 3 lines
implementation wise I see headaches in doing this, but conceptually this is just an optimization that could take place in the future if we fine that it's needed.
untarring a big source tree takes ages. I wouldn't like to do it after every reboot, if there was a .tar.bz2 tree I looked at often. That's why I have things like glibc untarred in my home directory.
With good cacheing, I could cd into the .tar.bz2 files and _effectively_ have the performance of untarred source trees for the ones I look at often on my disk -- automatically cleaned if the space if needed for something else, too. It would be quite nice.
-- Jamie
Quoted 7 lines · Linus Torvalds
In other words, the "directory" part is just a _view_ into the file. A view that potentially exposes a lot _more_ of the file, but we're still talking about the same file.
In contrast, a S_IFDIR-like _directory_ is something else entirely. When you view the things in that, you aren't looking at data "inside" the directory. You're looking at somethign totally independent.
One of the constraints is that you can create any name in an S_IFDIR-like directory, but you probably can't create any name "in" an S_IFREG-like file, because the names have meaning -- except in designated subtrees of the view intended for holding arbitrary names.
-- Jamie