silent semantic changes with reiser4
After looking trough the code and mailinglists I'm quite unhappy with a bunch of user-visible changes that Hans sneaked in and make reiser4 incompatible with other filesystems and have a slight potential to break even in the kernel.
o files as directories
- O_DIRECTORY opens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely useless
- meaning of the -x permission. This one has different meanings on directories vs files on UNIX systems. If we want to support directories as files we'll probably have to find a way to work around this.
- dentry aliasing. I can't find a formal guarantee in the code this can't happen
o metafiles - ..metas as a magic name that's just taken out of the namespace doesn't sound like a good idea. If we want this it should be a VFS-level option and there should be a translation-layer to xattrs. Not doing this will again confuse applications greatly that expect uniform filesystem behaviour.
Given these problems I request that these interfaces are removed from reiser4 for the kernel merge, and if added later at the proper VFS level after discussion on linux-kernel and linux-fsdevel, like we did for xattrs.
Quoted 4 lines · Christoph Hellwig
-
O_DIRECTORYopens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely useless
So `find -type d' would list every file on the system?
Lee
Quoted 7 lines · Lee Revell
Quoted 4 lines · Christoph Hellwig
-
O_DIRECTORYopens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely uselessSo `find -type d' would list every file on the system?
the find I have here is using lstat and not open with O_DIRECTORY, so no.
Quoted 7 lines · Christoph Hellwig
Quoted 6 lines · Lee Revell
Quoted 4 lines · Christoph Hellwig
-
O_DIRECTORYopens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely useless
Quoted 5 lines
Quoted 1 line
So `find -type d' would list every file on the system?
the find I have here is using lstat and not open with
O_DIRECTORY, so no.
Ugh, how embarrassing, I completely forgot about stat().
Lee
Quoted 9 lines · Christoph Hellwig
Quoted 6 lines
Quoted 4 lines
-
O_DIRECTORYopens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely uselessSo `find -type d' would list every file on the system?
the find I have here is using lstat and not open with
O_DIRECTORY, so no.
The find-like program I use (called treescan) uses O_DIRECTORY as an optimisation. It assumes that O_DIRECTORY will only open objects which are directories and can be read using readdir().
However, if reiser4 returns d_type values, then it won't even attempt an open on non-DT_DIR objects, and that's a better optimisation. (reiserfs doesn't return d_type values, unfortunately).
So the list of files that treescan finds depends on whether reiser4 implements d_type.
This is nothing like a POSIX filesystem. You untar a tree, and then listing it recursively shows extra things created by reiser4.
I quite like the principle, but because it's not like POSIX and doesn't match some program's expectations, it's a problem in its present form.
xattrs aren't a complete solution as you can't store structured data in an xattr. For example, with reiser4's model, you can cd into a .tar, .zip, .mp3 or .xml file and list the internal structure along with the file's metadata. You can't do that with xattrs.
Programs exist which quite reasonably assume that when you create a file, you can't opendir() the file, and recursive listings (like find, ls -R et al.) won't automatically traverse into every file.
On the other hand, being able to enter a file in a directory-like way allows structured representations of the contents to be accessed in the very useful "everything's a file" way -- i.e. ordinary tools.
So here's a semantic proposal:
1. O_DIRECTORY won't open an ordinary file.
Corollary: opendir("file") won't open an ordinary file.
2. An ordinary file path followed by "/" won't open an ordinary file.
Corollary: opendir("file/") won't open an ordinary file.
This is because appending a trailing slash is an alternate way for userspace to get the same results as O_DIRECTORY.
3. An ordinary file path followed by "/" _and_ one or more path components will open the file as a directory and enter it.
Corollary: opendir("file/.") will open an ordinary file.
4. The type of "file/." shall be S_IFDIR, _not_ S_IFREG.
Corollary: stat("file/.") will return that it's a directory.
The intention here is that explicit requests to examine the metadata or alternate structure representations of a file will create such a view, but the view is only available if requested explicitly.
When such a view is created, the results of stat(), O_DIRECTORY and opendir() are absolutely consistent. This will minimise confusion. Programs which recurse over a directory tree won't look inside any of the files. However they can be explicitly asked to recurse starting from a path inside a file: then they'll recurse over a single file's metadata and structured data.
Regarding the problems of safe locking in the VFS. The VFS assumes that directories are not hard linked: i.e. that they cannot appear at more than one path in a filesystem. Files-as-directories breaks that.
However, VFS does support directories on multiple paths, using bind mounts.
So it wouldn't be out of the question if entering a file (as described above) effectively auto-mounted a bind mount at that point.
-- Jamie
Quoted 5 lines · Christoph Hellwig
o files as directories
-O_DIRECTORYopens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely useless
Ouch.
I would definitely classify this as a security hole, since userland definitely uses O_DIRECTORY to avoid races.
Jeff
Quoted 8 lines · Jeff Garzik
Quoted 5 lines · Christoph Hellwig
o files as directories
-O_DIRECTORYopens succeed on all files on reiser4. Besides breaking .htaccess handling in apache and glibc compilation this also renders this flag entirely useless and opens up the races it tries to prevent against cmpletely useless
Quoted 4 lines
Ouch.
I would definitely classify this as a security hole, since userland definitely uses
O_DIRECTORYto avoid races.
Feh. That's far from the worst parts of the mess introduced by "hybrid" crap - trivial sys_link(2) deadlocks triggerable by any user rate a bit higher on the suckitude scale, IMO.
Quoted 3 lines · On Tue, Aug 24, 2004 at 09:53:44PM +0100…
Feh. That's far from the worst parts of the mess introduced by "hybrid" crap - trivial
sys_link(2)deadlocks triggerable by any user rate a bit higher on the suckitude scale, IMO.
While we are at it - consider these hybrids vetoed until
a) sys_link()/sys_link() deadlock is fixed
b) sys_link()/sys_rename() deadlock is fixed
c) correctness proof of the locking scheme (inDocumentation/filesystems/directory-locking) is updated to match the presense of the file/directory hybrids.
Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince us that there's no more crap of that kind". IMO a reasonable request, seeing that the first look at the patches in -mm4 had turned up two exploits in that area, despite the *YEARS* of warnings about potential trouble and need to be careful there (actually, I've given Hans too much credit and assumed that link/link never happens since nobody would be dumb enough to provide ->link() method for non-directory inodes; turns out that somebody is dumb enough and link/link is as exploitable as link/rename).
I allowed myself to get talked out of a final top to bottom code audit, and obviously that was a mistake.
It will probably take about 6 weeks. Apologies for wasting your time before that was done.
Hans
Quoted 34 lines · [email protected] wrote:
Quoted 5 lines · On Tue, Aug 24, 2004 at 09:53:44PM +0100…
Feh. That's far from the worst parts of the mess introduced by "hybrid" crap - trivial
sys_link(2)deadlocks triggerable by any user rate a bit higher on the suckitude scale, IMO.While we are at it - consider these hybrids vetoed until
a) sys_link()/sys_link() deadlock is fixed
b) sys_link()/sys_rename() deadlock is fixed
c) correctness proof of the locking scheme (inDocumentation/filesystems/directory-locking) is updated to match the presense of the file/directory hybrids.Rationale: (a) and (b) - immediately exploitable by any user, (c) - "convince us that there's no more crap of that kind". IMO a reasonable request, seeing that the first look at the patches in -mm4 had turned up two exploits in that area, despite the *YEARS* of warnings about potential trouble and need to be careful there (actually, I've given Hans too much credit and assumed that link/link never happens since nobody would be dumb enough to provide ->link() method for non-directory inodes; turns out that somebody is dumb enough and link/link is as exploitable as link/rename).
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Quoted 5 lines · Hans Reiser
I allowed myself to get talked out of a final top to bottom code audit, and obviously that was a mistake.
It will probably take about 6 weeks. Apologies for wasting your time before that was done.
I don't think you'll get anywhere with auditing. We need to write down the semantics you want, define them at the VFS level and make sure they're not conflicting with defined userspace semantics or kernel assumptions.
I think you need to learn the basic distinction between the VFS layer and a lowlevel filesystem driver.
Quoted 14 lines · Christoph Hellwig
I don't think you'll get anywhere with auditing. We need to write down the semantics you want, define them at the VFS level and make sure they're not conflicting with defined userspace semantics or kernel assumptions.
I think you need to learn the basic distinction between the VFS layer and a lowlevel filesystem driver.
How old are you? I thought you were the guy at Linux Tag with fashion oriented hair who gave a talk on his XFS work? Did I confuse you with someone else?
Hans
Quoted 3 lines · Hans Reiser
How old are you? I thought you were the guy at Linux Tag with fashion oriented hair who gave a talk on his XFS work? Did I confuse you with someone else?
CREDITS +8 -0
@@ -2677,6 +2677,14 @@ S: 79098 Freiburg S: Germany +N: Hans Reiser +E: [email protected] +W: http://www.namesys.com/ +D: official Linux kernel hairstyle critic +S: 6979 Exeter Dr +S: Oakland, CA 94611 +S: USA + N: Joerg Reuter E: [email protected] W: http://yaina.de/jreuter/
CREDITS line 2687
Quoted 9 lines · William Lee Irwin III
CREDITS +8
-0+N: Hans Reiser +E: [email protected] +W: http://www.namesys.com/ +D: official Linux kernel hairstyle critic +S: 6979 Exeter Dr +S: Oakland, CA 94611 +S: USA +
LOL.
In all fairness, how _would_ you have described it?
Besides, I don't think this should go in the CREDITS file, since hair styling criticism is clearly an ongoing MAINTAINERS issue, no?
Linus
Quoted 7 lines · William Lee Irwin III
Quoted 7 lines
(quoted diff) +7
-0+N: Hans Reiser +E: [email protected] +W: http://www.namesys.com/ +D: official Linux kernel hairstyle critic +S: 6979 Exeter Dr +S: Oakland, CA 94611 +S: USA
Quoted 4 lines · Linus Torvalds
LOL.
In all fairness, how _would_ you have described it? Besides, I don't think this should go in the CREDITS file, since hair styling criticism is clearly an ongoingMAINTAINERSissue, no?
As far as maintenance goes, absolutely. As far as describing it goes, I probably would have said it was short, red, and spiky.
MAINTAINERS +6 -0
@@ -1352,6 +1352,12 @@ L: [email protected] S: Supported +LINUX KERNEL HAIRSTYLE CRITICISM +P: Hans Reiser +M: [email protected] +W: http://www.namesys.com/ +S: Maintained + LINUX SECURITY MODULE (LSM) FRAMEWORK P: Chris Wright M: [email protected]
Quoted 6 lines · William Lee Irwin III
As far as maintenance goes, absolutely. As far as describing it goes, I probably would have said it was short, red, and spiky.
Ah, so it was him, is he disowning XFS these days or?
Quoted 24 lines · William Lee Irwin III
Quoted 7 lines · William Lee Irwin III
Quoted 7 lines
(quoted diff) +7
-0+N: Hans Reiser +E: [email protected] +W: http://www.namesys.com/ +D: official Linux kernel hairstyle critic +S: 6979 Exeter Dr +S: Oakland, CA 94611 +S: USAQuoted 4 lines · Linus Torvalds
LOL.
In all fairness, how _would_ you have described it? Besides, I don't think this should go in the CREDITS file, since hair styling criticism is clearly an ongoingMAINTAINERSissue, no?As far as maintenance goes, absolutely. As far as describing it goes, I probably would have said it was short, red, and spiky.
Index: mm1-2.6.9-rc1/MAINTAINERS
===================================================================
MAINTAINERS line 1355
Quoted 4 lines
MAINTAINERS +1
-0@@ -1352,6 +1352,12 @@ L: [email protected] S: Supported +LINUX KERNEL HAIRSTYLE CRITICISM
Hm. He doesn't critic the Linux Kernel hairstyle, but the Linux Kernel Developers' ...
Signature · 7 lines
Giuseppe "Oblomov" Bilotta
Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
(Roger Waters)Quoted 1 line · William Lee Irwin III
Quoted 1 line
+LINUX KERNEL HAIRSTYLE CRITICISM
Quoted 2 lines · Giuseppe Bilotta
Hm. He doesn't critic the Linux Kernel hairstyle, but the Linux Kernel Developers' ...
Absolutely right.
MAINTAINERS +6 -0
@@ -1352,6 +1352,12 @@ L: [email protected] S: Supported +LINUX KERNEL DEVELOPER'S HAIRSTYLE CRITICISM +P: Hans Reiser +M: [email protected] +W: http://www.namesys.com/ +S: Maintained + LINUX SECURITY MODULE (LSM) FRAMEWORK P: Chris Wright M: [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 4 lines · William Lee Irwin III
Quoted 1 line
+LINUX KERNEL DEVELOPER'S HAIRSTYLE CRITICISM
Aren't there more than one Linux Kernel Developer? Or is this to indicate that he indeed only criticised the hairstyle of one developer?
Best I repair that, then.
MAINTAINERS +6 -0
@@ -1352,6 +1352,12 @@ L: [email protected] S: Supported +LINUX KERNEL DEVELOPERS' HAIRSTYLE CRITICISM +P: Hans Reiser +M: [email protected] +W: http://www.namesys.com/ +S: Maintained + LINUX SECURITY MODULE (LSM) FRAMEWORK P: Chris Wright M: [email protected]
I had not intended to respond to this because I have nothing positive to say, but Andrew said I needed to respond and suggested I should copy Linus. Sigh.
Dear Christoph,
Let me see if I can summarize what you and your contingent are saying, and if I misconstrue anything, let me know.;-)
You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles. When I said we should just add some nice optional features to files and directories so that they can do everything that attributes can do if they are used that way, you just didn't get it. You instead went for the quick ugly hack called xattrs. You then got that ugly hack done first, because quick hacks are, well, quick. I then went about doing it the right way for Reiser4, and got DARPA to fund doing it. I was never silent about it.
Making files into directories caused only two applications out of the entire OS to notice the change, and that was because of a bug in what error code we returned that we are going to fix. You think that was a disaster; I think it was a triumph.
Now a cleanly architected filesystem with no attributes and just files and directories that can do everything attributes are used for exists. You don't want it to have the competitive advantage. Instead, you want it to have its clean design excised until you have something that duplicates it ready to go, and only then should it be allowed that users will use the features of your competitor's filesystem which you disdained implementing for so long.
Since you never studied or understood namespace design principles (or you would not have created and supported xattrs), you want to rename it to be called VFS, rewrite what we have done, and take over as the maintainer, mangling its design in a committee clusterfuck as you go.
We have just implemented very trivial semantic enhancements of the FS namespace, nothing like as ambitious as www.namesys.com/whitepaper.html or WinFS, and you are already pissing your pants.
Is that a fair summary?
Eat my dust.
Hans
PS
I should of course qualify what I have said. The use of files and directories in place of attributes is not a finished work. It has bugs, sys_reiser4() does not yet work, and there are little features still missing like having files readdir ignores.
Still, except for the bugs, what we have is usable, and there are a lot of happy reiser4 users right now even with the bugs. It will need a little bit more time, and then all the pieces will be in place.
PPS
If you implement your filesystems as reiser4 plugins, and rename reiser4's plugin code to be called "vfs", your filesystems will go faster. Not as fast as reiser4 though, because it has a better layout and that affects performance a lot, but faster is faster.... See www.namesys.com/benchmarks.html for details.
PPPS
Since we have such a performance lead, Namesys is about to change its focus from the storage layer to semantics, look at www.namesys.com/whitepaper.html for details. Semantic enhancements are the important stuff, and finally Namesys is where we have all the storage layer prerequisites done right, and the real work can begin. The gap between us is about to widen further.
Quoted 4 lines · Christoph Hellwig
After looking trough the code and mailinglists I'm quite unhappy with a bunch of user-visible changes that Hans sneaked in and make reiser4 incompatible with other filesystems
if we leave you in the dust, run faster.... not my problem....
Quoted 8 lines
Given these problems I request that these interfaces are removed from reiser4 for the kernel merge, and if added later at the proper VFS level after discussion on linux-kernel and linux-fsdevel, like we did for xattrs.
If you can't help fight WinFS, then get out of the way. Namesys is on the march. Read www.namesys.com/whitepaper.html.
Or, be smart, recognize that reiser4 is faster and more flexible than your storage layers because we are older and wiser and worked harder at it, join the team, and start contributing plugins that tap into the higher performance it offers.
Microsoft tried to build a storage layer that could handle small objects without losing performance, failed, and gave up at considerable cost to their architecture and pocketbook.
We just broke a hole in the enemy line. You could come swarming through it with us, but it sounds like you prefer complaining to HQ that we are getting too far in front of you.
That's a nice marketing talk. Get back to us when you have some technical contribution to make.
Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain
Quoted 5 lines · Matthew Wilcox
That's a nice marketing talk. Get back to us when you have some technical contribution to make.
www.namesys.com/download.html
Quoted 4 lines · Hans Reiser
You ignored everything I said during the discussion of xattrs about how there is no need to have attributes when you can just have files and directories, and that xattrs reflected a complete ignorance of name space design principles.
Actually in most of the discussion you simply didn't participate. While xattrs might not be the nicest interface they have the advantag of not breaking the SuS assumption of what directories vs files are, and they do not break the Linux O_DIRECTORY semantics that are defined and need to solve real-world races either.
Quoted 7 lines
When I said we should just add some nice
optional features to files and directories so that they can do everything that attributes can do if they are used that way, you just didn't get it. You instead went for the quick ugly hack called xattrs. You then got that ugly hack done first, because quick hacks are, well, quick. I then went about doing it the right way for Reiser4, and got DARPA to fund doing it. I was never silent about it.
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Quoted 7 lines
Now a cleanly architected filesystem with no attributes and just files and directories that can do everything attributes are used for exists. You don't want it to have the competitive advantage. Instead, you want it to have its clean design excised until you have something that duplicates it ready to go, and only then should it be allowed that users will use the features of your competitor's filesystem which you disdained implementing for so long.
My competitors filesystem? If you look at MAINTAINERS I maintain only vxfs and sysvfs, neither of which I'd suggest anyone to run their system on.
Quoted 4 lines
Since you never studied or understood namespace design principles (or you would not have created and supported xattrs), you want to rename it to be called VFS, rewrite what we have done, and take over as the maintainer, mangling its design in a committee clusterfuck as you go.
Hans, please stop the personal crap or the black helicopters will kidnap
you. When was the last time you actually worked on kernel namespace
code instead of talking marketing bullshit and ignoring all real world problems.
Quoted 5 lines
If you implement your filesystems as reiser4 plugins, and rename reiser4's plugin code to be called "vfs", your filesystems will go faster. Not as fast as reiser4 though, because it has a better layout and that affects performance a lot, but faster is faster.... See www.namesys.com/benchmarks.html for details.
Could you pass on that crack pipe please?
Btw, I just got reminded you might take my saying as an "piss of you're idea stinks" or similar things. So let me clarify again the actual technical and project managment issues another time before we start getting really personal :)
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem. Your attributes are absoultely a VFS thing and as such should not happen at the filesystem layer, and no, that doesn't mean they're bad per se, I just think they are a rather bad fit for Linux.
So now go on and try to work together with the other peope doing VFS level work instead of hiding, or if you think you can't work together with us search a nice research OS where you can take over the VFS layer, if your ideas prove to be good I'm sure Linux will pick them up sooner or later.
Christoph
Quoted 7 lines · Christoph Hellwig
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem. Your attributes are absoultely a VFS thing and as such should not happen at the filesystem layer, and no, that doesn't mean they're bad per se, I just think they are a rather bad fit for Linux.
Now this I agree with, in the sense that I think that if we want to support this, it should be supported at a VFS layer.
On the other hand, I think doing it inside the filesystem with ugly hacks is an acceptable way to prototype the idea before it's been proven to really be workable. Maybe it has more problems with legacy apps than we'd expect..
Linus
Quoted 7 lines · Linus Torvalds
Now this I agree with, in the sense that I think that if we want to support this, it should be supported at a VFS layer.
On the other hand, I think doing it inside the filesystem with ugly hacks is an acceptable way to prototype the idea before it's been proven to really be workable. Maybe it has more problems with legacy apps than we'd expect..
Oh, I'm the last person to tell anyone how to prototype things. I just don't want such inconsistancies in the mainline kernel.
Quoted 11 lines · Linus Torvalds
Quoted 7 lines · Christoph Hellwig
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem. Your attributes are absoultely a VFS thing and as such should not happen at the filesystem layer, and no, that doesn't mean they're bad per se, I just think they are a rather bad fit for Linux.
Quoted 2 lines
Now this I agree with, in the sense that I think that if we want to support this, it should be supported at a VFS layer.
ACK. However, I'm still not seeing *ANYTHING* that would look like a workable scheme in presense of hardlinks. Show me how to make that deadlock- and race-free and we might very well do it in VFS.
_That_ is what's missing and it's needed no matter where it's implemented. You want hybrid objects - you want to solve that one. So far I've seen nothing workable.
Quoted 12 lines · Linus Torvalds
Quoted 8 lines · Christoph Hellwig
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem. Your attributes are absoultely a VFS thing and as such should not happen at the filesystem layer, and no, that doesn't mean they're bad per se, I just think they are a rather bad fit for Linux.
Quoted 5 lines
Now this I agree with, in the sense that I think that if we want to support this, it should be supported at a VFS layer.
On the other hand, I think doing it inside the filesystem with ugly hacks
what is ugly? ;-/
Quoted 7 lines
is an acceptable way to prototype the idea before it's been proven to really be workable. Maybe it has more problems with legacy apps than we'd expect..
Linus
Quoted 6 lines · Christoph Hellwig
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem.
VFS is the common filesystem layer. The only reason you think semantics belong in the common filesystem layer is that you are not innovating in your semantics, and feel content with stasis.
I don't. I expect that semantics will get radically changed over the next few years as we compete with Giampaolo and whatever lesser lights are working at Microsoft.
Quoted 4 lines
Your attributes are absoultely a VFS thing and as such
should not happen at the filesystem layer, and no, that doesn't mean they're bad per se, I just think they are a rather bad fit for Linux.
Quoted 11 lines
So now go on and try to work together with the other peope doing VFS level work instead of hiding, or if you think you can't work together with us search a nice research OS where you can take over the VFS layer, if your ideas prove to be good I'm sure Linux will pick them up sooner or later.
Christoph
I tell you what, use xattrs for all the half speed filesystems, and the users and I will use metafiles.
Quoted 8 lines · Hans Reiser
Quoted 6 lines · Christoph Hellwig
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem.
Quoted 7 lines
VFS is the common filesystem layer. The only reason you think semantics belong in the common filesystem layer is that you are not innovating in your semantics, and feel content with stasis.
I don't. I expect that semantics will get radically changed over the next few years as we compete with Giampaolo and whatever lesser lights are working at Microsoft.
Hans, please stop the gooddamn personal attack bullshit.
How do you for example suggestion exporting your semantics over the network if they're not done at the VFS level? How do you want some clusterfilesystem support them or tmpfs?
Quoted 9 lines · Christoph Hellwig
How do you for example suggestion exporting your semantics over the network if they're not done at the VFS level? How do you want some clusterfilesystem support them or tmpfs?
by accesses to filename/metas.....
That's the beauty of simplicity, not much is needed. xattrs on the other hand would be a lot of work to integrate into a cluster filesystem.
Hans
Quoted 11 lines · Hans Reiser
Quoted 6 lines · Christoph Hellwig
Over the last at least five years we've taken as much as possible semantics out of the filesystems and into the VFS layer, thus having a separation between the semantical layer (VFS) and the low level filesystem.
VFS is the common filesystem layer. The only reason you think semantics belong in the common filesystem layer is that you are not innovating in your semantics, and feel content with stasis.
This misunderstanding is the heart of the problem really. In many ways, the VFS is the semantic layer, it defines the picture of the directory tree and the locking rules the applications see.
There's no rule that says you can't innovate in the VFS, in fact it is where the semantic innovations belong. Putting them there is what allows other filesystems to implement your semantics, and this is exactly what linux needs if these new semantics are going to be standardized.
In other words, writing a reiser4 plugin is one way to get to the new semantics, but it must not be the only way. Keeping them entirely within reiser4 is good for the research stage, but for actual deployment they need to be finalized and pushed up.
If the early linux filesystems had taken the same attitude you have (don't write new filesystems, only write plugins), there would be no framework allowing the wealth of filesystems we do have, including reiser4.
-chris
Quoted 9 lines · Chris Mason
If the early linux filesystems had taken the same attitude you have (don't write new filesystems, only write plugins), there would be no framework allowing the wealth of filesystems we do have, including reiser4.
Au contraire, with my approach, there would be 10x as many filesystems, because people would be easily able to mix-and-match different methods and plugins from different filesystems together, and if you wanted the MySQL semantics, you would mount mysql, and if you wanted the zope semantics you would mount zope.
Quoted 7 lines · Hans Reiser
Quoted 7 lines
If the early linux filesystems had taken the same attitude you have (don't write new filesystems, only write plugins), there would be no framework allowing the wealth of filesystems we do have, including reiser4.
Quoted 5 lines
Au contraire, with my approach, there would be 10x as many filesystems, because people would be easily able to mix-and-match different methods and plugins from different filesystems together, and if you wanted the MySQL semantics, you would mount mysql, and if you wanted the zope semantics you would mount zope.
And if you wanted to make free love you'd make free love you'd make free love and there would be piece on all earth..
You seem to have lost contact to reality a little.
Quoted 4 lines · Christoph Hellwig
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.
It's the UNIX way.
And yes, the semantics can _easily_ be solved in very unixy ways.
One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is:
- without the slash, a file-as-dir won't open with O_DIRECTORY (ENOTDIR)
- with the slash, it won't open _without_ O_DIRECTORY (EISDIR)
Problem solved. Very user-friendly, and very intuitive.
Will it potentially break something? Sure. Do we care? Me, I'll take that kind of extension _any_ day over xattrs, that are fundamentally flawed in my opinion and totally useless. The argument that applications like "tar" won't understand the file-as-directory thing is _flawed_, since legacy apps won't understand xattrs either.
Oh, add a O_NOXATTRS flag to force a path lookup to only use regular directories, the same way we have O_NOFOLLOW and friends. That allows people to see the difference, if they care (ie a file server might decide that it doesn't want to expose things like this).
I never liked the xattr stuff. It makes little sense, and is totally useless for 99.9999% of everything. I still don't see the point of it, except for samba. Ugly.
Linus
Quoted 10 lines · Linus Torvalds
And yes, the semantics can _easily_ be solved in very unixy ways.
One way to solve it is to just realize that a final slash at the end implies pretty strongly that you want to treat it as a directory. So what you do is:
- without the slash, a file-as-dir won't open with
O_DIRECTORY(ENOTDIR)
- with the slash, it won't open _without_O_DIRECTORY(EISDIR)Problem solved. Very user-friendly, and very intuitive.
That would solve the O_DIRECTORY issue, the dentry aliasing still needs work though with the semantics for link/unlink/rename.
Maybe Hans & you should start 2.7 to work this out? :)
I just want to add that I AM capable of working with the other filesystem developers in a team-player way, and I am happy to cooperate with making portions more reusable where there is serious interest from other filesystems in that, but Christoph is a puppy who has never written or designed a major filesystem from scratch, and Nikita is a big dog who has written stuff very few projects are lucky enough to see the likes of, and when Christoph insults Nikita's code, or my design guidance for that code, it is not going to bring out my good side. The plugin and metafiles code needs many improvements, but Christoph does not have the expertise to understand what those needed improvements are because he hasn't invested the work into understanding the code.
Christoph is a bright and clever young fellow, who just hasn't had the years of study of the field yet. I wish him well, and away.;-)
Hans
Quoted 4 lines · Hans Reiser
I just want to add that I AM capable of working with the other filesystem developers in a team-player way, and I am happy to cooperate with making portions more reusable where there is serious interest from other filesystems in that,
Prove it. Stop replying for today and come back tomorrow with some useful discussions. Christoph suggested that some of the v4 semantics belong in the VFS and therefore linux as a whole. He's helping you to make sure the semantics and fit nicely with the rest of kernel interfaces and are race free.
Take him up on the offer.
-chris
Quoted 2 lines · Hans Reiser
Christoph is a bright and clever young fellow, who just hasn't had the years of study of the field yet. I wish him well, and away.;-)
I see this is as an opportunity where you can share some of your experience to Cristoph and many others and work to get the semantics into VFS.
Please make this work :)
Signature · 1 line
mjt
Quoted 1 line · Hans Reiser
other filesystems in that, but Christoph is a puppy who has never
It's not Christoph who's shown more bark than bite in this thread.
Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan
Quoted 4 lines · Rik van Riel
On Wed, 25 Aug 2004, Hans Reiser wrote:
Quoted 8 lines
Quoted 3 lines
other filesystems in that, but Christoph is a puppy who has never
It's not Christoph who's shown more bark than bite in this thread.
The bite is at www.namesys.com/download.html
Hans
Quoted 8 lines · Linus Torvalds
Quoted 4 lines · Christoph Hellwig
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Quoted 6 lines
Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.
It's the UNIX way.
Not if you allow link(2) on them. And not if you design and market your stuff as a general-purpose backdoor into kernel. Note how *EVERY* *DAMN* *OPERATION* is made possible to override by "plugins". Which is the reason for deadlocks in question, BTW.
Don't fool yourself - that's what Hans is selling. Target market: ISV. Marketed product: a set of hooks, the wider the better, no matter how little sense it makes. The reason for doing that outside of core kernel:
bypassing any review and being able to control the product being sold (see above).
Shame that it got an actual filesystem mixed in with the marketing plans and general insanity...
Am Mittwoch, den 25.08.2004, 21:42 +0100 schrieb :
Quoted 4 lines
Quoted 4 lines
Quoted 3 lines
For one thing _I_ didn't decide about xattrs anyway. And I still haven't seen a design from you on -fsdevel how you try to solve the problems with files as directories.
Quoted 8 lines
Quoted 6 lines
Hey, files-as-directories are one of my pet things, so I have to side with Hans on this one. I think it just makes sense. A hell of a lot more sense than xattrs, anyway, since it allows scripts etc standard tools to touch the attributes.
It's the UNIX way.
Not if you allow link(2) on them.
That doesn't make sense anyway. (actually, I tried what happens and the result was an Oops ;))
It should be completely forbidden to link into a meta-directory or out of such a directory. You could think of those meta-directory as a sysfs for that inode. Of course it's not an own filesystem and that means that there need to be a lot of security precautions in the VFS layer. Where something like that belongs anyway, if done correctly.
Quoted 4 lines
And not if you design and market your
stuff as a general-purpose backdoor into kernel. Note how *EVERY* *DAMN* *OPERATION* is made possible to override by "plugins". Which is the reason for deadlocks in question, BTW.
What do you mean? If you tell that file that you want it to be compressed or encrypted or modify some attributes (like ACLs) this isn't necessarily a backdoor.
Quoted 5 lines
Don't fool yourself - that's what Hans is selling. Target market: ISV. Marketed product: a set of hooks, the wider the better, no matter how little sense it makes. The reason for doing that outside of core kernel:
bypassing any review and being able to control the product being sold (see above).
Yes, I don't think it was a good idea either. Probably someone should remove these features and make it a "normal" filesystem. The people who need it now can turn it on again and a real solution could be worked out in Linux 2.7.
I wouldn't use it on a public server anyway now because I'm not convinced some malicious guy could find a way to exploit that. What if you changed into a meta directory using ftp and some manage to break things? This might be very dangerous.
I personally think that the idea of doing something like this (I'm not speaking of the current implementation which I think is really bad) is the right way to go in the long term.
Quoted 2 lines · Christophe Saout
It should be completely forbidden to link into a meta-directory or out of such a directory. [..]
agreed.
Quoted 1 line
Yes, I don't think it was a good idea either. Probably someone should
I personally would like plugins only if the API they use wouldn't allow them to corrupt the underlying fs, I'm not sure if this is the case with reiserfs4.
About the backwards compatibility, another option is to add a O_FILEDIR and have bash learn about it when you cd into a file. No magic with the slashes then.
Quoted 12 lines · Andrea Arcangeli
Quoted 4 lines · Christophe Saout
It should be completely forbidden to link into a meta-directory or out of such a directory. [..]
agreed.
Quoted 9 lines
Quoted 3 lines
Yes, I don't think it was a good idea either. Probably someone should
I personally would like plugins only if the API they use wouldn't allow them to corrupt the underlying fs, I'm not sure if this is the case with reiserfs4.
You compile crap into the kernel, you are screwed. We have not and will not change that.
Reiser4 plugins are not for end users to download from amazon.com, they are for weekend hackers to send me a cool plugin for me to review, assign a plugin id to, and send to Linus in the next release. Sometimes being less ambitious works better, and dynamically loaded plugins would have made the infrastructure too bulky to be fun and fast.
Quoted 3 lines · Hans Reiser
Reiser4 plugins are not for end users to download from amazon.com, they are for weekend hackers to send me a cool plugin for me to review, assign a plugin id to, and send to Linus in the next release. Sometimes
then what's the difference in having the plugin fixed in stone into reiserfs? That's my whole point. Get the patch from the weekend hacker, check it, send the patch to Linus to add the new feature to reiser4, just call it "feature" not plugin. That's how it works normally for everything. Many fs have many features, many of them optional. you wouldn't need to build any hook infrastructure either that way. Hooks would be needed if this wasn't open source me thinks. Or if you want people to fetch the module from amazon without your review.
Another reason I could see the modularization/hooking useful is if those feature would take lots of kernel space, but this sure isn't the case for reiserfs, infact having modules would waste _more_ ram due the half wasted page allocation for the module text. The only single reason we use modules is to avoid wasting tons of ram by loading every possible device driver on earth, it's not that we use modules because they're more flexible, if something they're more fragile. I never use modules in my test kernel just to go fast (because I self compile them). The last good thing of the modules is during development you don't need to reboot the machine to test a kernel change in a driver, but you don't need that with reiserfs since you can make the fs itself a module (just change the name of the fs, AFIK you do that all the time already).
Am Donnerstag, den 26.08.2004, 13:28 +0200 schrieb Andrea Arcangeli:
Quoted 2 lines
then what's the difference in having the plugin fixed in stone into reiserfs? That's my whole point.
I think you're not exactly understanding what a reiser4 "plugin" is.
Currently plugins are not modules. It just means that there's a defined interface between the reiser4 core and the plugins. Just like you could see filesystems as "VFS plugins".
In fact, reiser4 plugins are
- users of the reiser4 core API
- some of them are implementing Linux VFS methods (thus being some sort of glue code between the Reiser4 storage tree and the Linux VFS)
Quoted 5 lines · Christophe Saout
I think you're not exactly understanding what a reiser4 "plugin" is.
Currently plugins are not modules. It just means that there's a defined interface between the reiser4 core and the plugins. Just like you could see filesystems as "VFS plugins".
but filesystems are exactly always modules (in precompiled kernels at least).
Anyways I don't see any visible EXPORT_SYMBOL in reiser4.
Quoted 5 lines
In fact, reiser4 plugins are
- users of the reiser4 core API
- some of them are implementing Linux VFS methods (thus being some sort of glue code between the Reiser4 storage tree and the Linux VFS)
then it seems a bit misleading to call those things plugins.
As you wrote those are "users of the reiser4 core API", with plugins I always meant dynamically loadable things, like plugins for web browers (hence kernel modules in kernel space). While here apparently you can't plugin anything at runtime, it's just code that uses the reiser4 core API that can be modified with a kernel patch as usual.
Am Donnerstag, den 26.08.2004, 14:16 +0200 schrieb Andrea Arcangeli:
Quoted 6 lines
Quoted 3 lines
Currently plugins are not modules. It just means that there's a defined interface between the reiser4 core and the plugins. Just like you could see filesystems as "VFS plugins".
but filesystems are exactly always modules (in precompiled kernels at least).
Yes, of cousre.
Quoted 1 line
Anyways I don't see any visible
EXPORT_SYMBOLin reiser4.
Yes, there's no module infrastructure there. These "plugins" need to be linked at compile time. Some plugins need to be anyway since the the filesystem is useless without them. Others whould be optional features.
Quoted 5 lines
Quoted 4 lines
In fact, reiser4 plugins are
- users of the reiser4 core API
- some of them are implementing Linux VFS methods (thus being some sort of glue code between the Reiser4 storage tree and the Linux VFS)
Quoted 7 lines
then it seems a bit misleading to call those things plugins.
As you wrote those are "users of the reiser4 core API", with plugins I always meant dynamically loadable things, like plugins for web browers (hence kernel modules in kernel space). While here apparently you can't plugin anything at runtime, it's just code that uses the reiser4 core API that can be modified with a kernel patch as usual.
Right. The name is probably at bit misleading. But assuming the compression "plugin" whould be dynamically loadable at some later point it would be a plugin then?
Quoted 11 lines · Andrea Arcangeli
Quoted 5 lines · Christophe Saout
I think you're not exactly understanding what a reiser4 "plugin" is.
Currently plugins are not modules. It just means that there's a defined interface between the reiser4 core and the plugins. Just like you could see filesystems as "VFS plugins".
but filesystems are exactly always modules (in precompiled kernels at least).
Anyways I don't see any visible
EXPORT_SYMBOLin reiser4.
they would be there, I think.
Quoted 8 lines
Quoted 6 lines
In fact, reiser4 plugins are
- users of the reiser4 core API
- some of them are implementing Linux VFS methods (thus being some sort of glue code between the Reiser4 storage tree and the Linux VFS)then it seems a bit misleading to call those things plugins.
There should be infracture to change plugin on fly, and choosing object's plugins at object's creation time. Currently only certain object's plugins can be changed through <object>/metas/plugin interface.
Quoted 6 lines
As you wrote those are "users of the reiser4 core API", with plugins I always meant dynamically loadable things, like plugins for web browers (hence kernel modules in kernel space). While here apparently you can't plugin anything at runtime, it's just code that uses the reiser4 core API that can be modified with a kernel patch as usual.
Signature · 1 line
Alex.
Quoted 7 lines · Alex Zarochentsev
There should be infracture to change plugin on fly, and choosing object's plugins at object's creation time. Currently only certain object's plugins can be changed through <object>/metas/plugin interface.
To be more clear for those not familiar with our project, there is much more work to be done, and one of the areas that (extremely) needs more work is changing plugins on the fly and clean interfaces for specifying them at creation time.
Quoted 12 lines · Andrea Arcangeli
Quoted 3 lines · Hans Reiser
Reiser4 plugins are not for end users to download from amazon.com, they are for weekend hackers to send me a cool plugin for me to review, assign a plugin id to, and send to Linus in the next release. Sometimes
then what's the difference in having the plugin fixed in stone into reiserfs? That's my whole point. Get the patch from the weekend hacker, check it, send the patch to Linus to add the new feature to reiser4, just call it "feature" not plugin.
Think of it as being like VFS, where you can plugin new filesystems. Only in this case, you can plugin new kinds of files, and everything you need to implement those new kinds of files (items, nodes, keys, processing before flushing to disk, etc.) also gets implemented as a plugin. It is an Uber-VFS.
Quoted 4 lines
The only single reason we
use modules is to avoid wasting tons of ram by loading every possible device driver on earth,
I am not concerned about ram in this design, I want nifty new kinds of files easy crafted over a weekend by sysadmins working in Canada and Guatemala. Software engineering cost is what matters to me ( I turned 40, so I think different now....;-) )
Quoted 5 lines · Hans Reiser
Think of it as being like VFS, where you can plugin new filesystems. Only in this case, you can plugin new kinds of files, and everything you need to implement those new kinds of files (items, nodes, keys, processing before flushing to disk, etc.) also gets implemented as a plugin. It is an Uber-VFS.
I'm just confused by the naming, following your comparison with it being like VFS, I don't call reiser4 a "plugin" of the VFS.
The vfs offers an interface for the fs to register into it. Like the scsi highlevel code also offers an interface for the lowlevel driver to register into it, but we don't call scsi lowlevel drivers "plugins".
the ext3 directory code calling ext3_getblk would be also a plugin in your vocabulary, and the ext3_getblk call would be plugin-hook.
Quoted 4 lines
I am not concerned about ram in this design, I want nifty new kinds of files easy crafted over a weekend by sysadmins working in Canada and Guatemala. Software engineering cost is what matters to me ( I turned 40, so I think different now....;-) )
Adding proper layering and abstractions is sure a good thing, if you can turn the street into a crowd of reiser4 kernel hackers because your API is so clean that any sysadmin can craft new kind of files over the weekend you're welcome to go ahead ;). (I just doubt the "plugin" terminology can make a difference by the time people start scratching their head into the code, the design and the clean abstraction of the code will make a difference instead)
Quoted 16 lines · Andrea Arcangeli
Quoted 3 lines · Hans Reiser
Reiser4 plugins are not for end users to download from amazon.com, they are for weekend hackers to send me a cool plugin for me to review, assign a plugin id to, and send to Linus in the next release. Sometimes
then what's the difference in having the plugin fixed in stone into reiserfs? That's my whole point. Get the patch from the weekend hacker, check it, send the patch to Linus to add the new feature to reiser4, just call it "feature" not plugin. That's how it works normally for everything. Many fs have many features, many of them optional. you wouldn't need to build any hook infrastructure either that way. Hooks would be needed if this wasn't open source me thinks. Or if you want people to fetch the module from amazon without your review.
Quoted 14 lines
Another reason I could see the modularization/hooking useful is if those feature would take lots of kernel space, but this sure isn't the case for reiserfs, infact having modules would waste _more_ ram due the half wasted page allocation for the module text. The only single reason we use modules is to avoid wasting tons of ram by loading every possible device driver on earth, it's not that we use modules because they're more flexible, if something they're more fragile. I never use modules in my test kernel just to go fast (because I self compile them). The last good thing of the modules is during development you don't need to reboot the machine to test a kernel change in a driver, but you don't need that with reiserfs since you can make the fs itself a module (just change the name of the fs, AFIK you do that all the time already).
There lots of good things about plugins as they are in reiser4. And probably the best one (as for me) is that, reiser4 core code (quite complex thing) does not need to be changed after adding new plugins (read new features).
It is written once and provides some basic functionality (balancing, atoms, etc.), which works well with all kinds of plugins if they implement required functions from plugin interface.
As for me, this looks like Linux's VFS. Sure, if I want to add some very new filesystem with features which were not foreseen, and VFS does not provide needed interface, it will need some changes. But in simple case, when I just want to develop once more disk filesystem, which say will be different from another disk filesystem in only that respect it manages its metadata in different manner and aims to achieve better performance this way. In this case I will not need to chnage VFS at all. The same about reiser4 core and new plugins.
Quoted 9 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
umka
Quoted 16 lines · Andrea Arcangeli
Quoted 3 lines · Hans Reiser
Reiser4 plugins are not for end users to download from amazon.com, they are for weekend hackers to send me a cool plugin for me to review, assign a plugin id to, and send to Linus in the next release. Sometimes
then what's the difference in having the plugin fixed in stone into reiserfs? That's my whole point. Get the patch from the weekend hacker, check it, send the patch to Linus to add the new feature to reiser4, just call it "feature" not plugin. That's how it works normally for everything. Many fs have many features, many of them optional. you wouldn't need to build any hook infrastructure either that way. Hooks would be needed if this wasn't open source me thinks. Or if you want people to fetch the module from amazon without your review.
Quoted 14 lines
Another reason I could see the modularization/hooking useful is if those feature would take lots of kernel space, but this sure isn't the case for reiserfs, infact having modules would waste _more_ ram due the half wasted page allocation for the module text. The only single reason we use modules is to avoid wasting tons of ram by loading every possible device driver on earth, it's not that we use modules because they're more flexible, if something they're more fragile. I never use modules in my test kernel just to go fast (because I self compile them). The last good thing of the modules is during development you don't need to reboot the machine to test a kernel change in a driver, but you don't need that with reiserfs since you can make the fs itself a module (just change the name of the fs, AFIK you do that all the time already).
There lots of good things about plugins as they are in reiser4. And probably the best one (as for me) is that, reiser4 core code (quite complex thing) does not need to be changed after adding new plugins (read new features).
It is written once and provides some basic functionality (balancing, atoms, etc.), which works well with all kinds of plugins if they implement required functions from plugin interface.
As for me, this looks like Linux's VFS. Sure, if I want to add some very new filesystem with features which were not foreseen, and VFS does not provide needed interface, it will need some changes. But in simple case, when I just want to develop once more disk filesystem, which say will be different from another disk filesystem in only that respect it manages its metadata in different manner and aims to achieve better performance this way. In this case I will not need to chnage VFS at all. The same about reiser4 core and new plugins.
Quoted 9 lines
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Signature · 1 line
umka
Quoted 6 lines · On Wed, 25 Aug 2004 [email protected] wrote:
Quoted 2 lines · Linus Torvalds
It's the UNIX way.
Not if you allow link(2) on them.
Heh. I don't think that's a very strong argument against being "unixy", considering how traditional unix _used_ to handle directories.
mkdir/rmdir/rename only came later. Now, obviously they did come later for a good reason, but still..
The interesting part is that thanks to the dcache, we should be perfectly able to actually _see_ circular links etc, so some of the problems with linking directories should actually be quite solvable - something that is _not_ true for a traditional UNIX VFS layer.
Of course, the dcache introduces some new problems of its own wrt directory aliasing, but I don't actually think that should be fundamental either. Treating them more as a "static mountpoint" from an FS angle and less as a traditional Unix hardlink should be doable, I'd have thought.
(Also, it's entirely possible that the filesystem may not support some of the more esoteric linking/renaming operations. For example, in a traditional xattrs setup where the xattr is linked on-disk with the file it is associated with, you simply _can't_ link it somewhere else, or rename it to any other directory. That's not a VFS layer issue, obviously, but I thought I'd bring up the point that file-as-dir cases may have limitations that normal files don't have).
Quoted 2 lines
And not if you design and market your stuff as a general-purpose
backdoor into kernel.
Now that's a separate argument, and not one I'm personally interested in arguing at least right now. I haven't actually looked at the reiser4 code, so I'm really _only_ arguing against special-case attributes.
Linus
Quoted 4 lines · Linus Torvalds
Of course, the dcache introduces some new problems of its own wrt directory aliasing, but I don't actually think that should be fundamental either. Treating them more as a "static mountpoint" from an FS angle and less as a traditional Unix hardlink should be doable, I'd have thought.
Yeah, if we ditch the "mountpoints are busy and untouchable" stuff. Which I'd love to, but it's a hell of a visible (and admin-visible) change.
FWIW, current deadlocks are unrelated to actual operation succeeding.
Look: we have sys_link() making sure that parent of target is a directory (PATH_LOOKUP, in a "it has ->lookup()" sense), then locking target's parent, then checking that it has ->link() (everyone on reiser4 does) and then checking that source (old link to file) is *not* a directory (in S_ISDIR sense). Then we lock source.
Note that currently it's OK - we get "all non-directories are always locked after all directories". With filesystem that provides hybrid objects with non-NULL ->link() it's not true and we are in deadlock country. Before we get anywhere near fs code.
I'm not saying that this particular instance is hard to fix, but it wasn't even looked at. All it would take is checking the description of current locking scheme and looking through the proof of correctness (present in the tree). That's the first point where said proof breaks if we have hybrids. And it's what, about 4 screenfuls of text?
I have no problems with discussing such stuff and no problems with having it merged if it actually works. But let's start with something better than "let's hope nothing breaks if we just add such objects and do nothing else, 'cause hybridi files/directories are good, mmmkay?"
This message suggests a way to extend the VFS safe locking rules to include files-as-directories.
Quoted 4 lines · [email protected] wrote:
Note that currently it's OK - we get "all non-directories are always locked after all directories". With filesystem that provides hybrid objects with non-NULL ->link() it's not true and we are in deadlock country. Before we get anywhere near fs code.
Is this a problem if we treat entering a file-as-directory as crossing a mount point (i.e. like auto-mounting)?
Simply doing a path walk would lock the file and then cross the mount point to a directory.
A way to ensure that preserves the lock order is to require that the metadata is in a different filesystem to its file (i.e. not crossing a bind mount to the same filesystem).
That has the side effect of preventing hard links between metadata files and non-metadata, which in my opinion is fine.
Path walking will lock the file, and then lock the directory on a different filesystem. Lock order is still safe, provided a strict order is maintained between the two filesystems.
The strict order is ensured by preventing bind mounts which create a path cycle containing a file->metadata edge. One way to ensure that is to prevent mounts on the metadata filesystems, but the rule doesn't have to be that strict. This condition only needs to be checked in the mount() syscall.
-- Jamie
Quoted 2 lines · Jamie Lokier
Is this a problem if we treat entering a file-as-directory as crossing a mount point (i.e. like auto-mounting)?
Yes - mountpoints can't be e.g. unlinked. Moreover, having directory mounted on non-directory is also an interesting situation.
Quoted 2 lines
Simply doing a path walk would lock the file and then cross the mount point to a directory.
*Ugh*
What would happen if you open that directory or chdir there? If it's "underlying file stays locked" - we are in even more obvious deadlocks.
Quoted 6 lines
A way to ensure that preserves the lock order is to require that the metadata is in a different filesystem to its file (i.e. not crossing a bind mount to the same filesystem).
That has the side effect of preventing hard links between metadata files and non-metadata, which in my opinion is fine.
We don't actually need a different fs - different vfsmount will do just fine.
Quoted 5 lines
The strict order is ensured by preventing bind mounts which create a path cycle containing a
file->metadataedge. One way to ensure that is to prevent mounts on the metadata filesystems, but the rule doesn't have to be that strict. This condition only needs to be checked in themount()syscall.
You really don't want to lock mountpoint on path lookup, so I don't see how that would be relevant - it's a hell to clean up, for one thing (I've crossed ten mountpoints on the way, when do I unlock them and how do I prevent deadlocks from that?) Besides, different namespaces can have completely different mount trees, so tracking down all that stuff would be hell in its own right.
The main issue I see with all schemes in that direction (and something like that could be made workable) is the semantics of unlink() on mountpoints. *Especially* with users being able to see attributes of files they do not own (e.g. reiser4 mode/uid/gid stuff). Ability to pin down any damn file on the system and make it impossible to replace is not something you want to give to any user.
Quoted 5 lines · [email protected] wrote:
Quoted 2 lines
Is this a problem if we treat entering a file-as-directory as crossing a mount point (i.e. like auto-mounting)?
Yes - mountpoints can't be e.g. unlinked. Moreover, having directory mounted on non-directory is also an interesting situation.
Ok, so can we make it so mountpoints can be unlinked? :)
The mount would continue to exist, but with no name, until its last user disappears.
Quoted 7 lines
Quoted 2 lines
Simply doing a path walk would lock the file and then cross the mount point to a directory.
*Ugh*
What would happen if you open that directory or chdir there? If it's "underlying file stays locked" - we are in even more obvious deadlocks.
I think the underlying file does not stay locked, and once you've entered it as a directory, it can be unlinked.
If you have the directory open or chdir into it, then it _may_ have the effect of keeping the file's storage allocated when you unlink it -- just like when a file is unlinked while opened. As that is not a user-visible property, it's a filesystem-specific implementation detail as to whether it keeps the file's data in existence while the metadata directories and/or files are open.
Quoted 15 lines
Quoted 6 lines
A way to ensure that preserves the lock order is to require that the metadata is in a different filesystem to its file (i.e. not crossing a bind mount to the same filesystem).
That has the side effect of preventing hard links between metadata files and non-metadata, which in my opinion is fine.
We don't actually need a different fs - different vfsmount will do just fine.
Quoted 5 lines
The strict order is ensured by preventing bind mounts which create a path cycle containing a
file->metadataedge. One way to ensure that is to prevent mounts on the metadata filesystems, but the rule doesn't have to be that strict. This condition only needs to be checked in themount()syscall.
Quoted 6 lines
You really don't want to lock mountpoint on path lookup, so I don't see how that would be relevant - it's a hell to clean up, for one thing (I've crossed ten mountpoints on the way, when do I unlock them and how do I prevent deadlocks from that?) Besides, different namespaces can have completely different mount trees, so tracking down all that stuff would be hell in its own right.
I didn't mean locking a chain of mountpoints, I meant the temporary state where two dentries and/or inodes are locked, parent and child, during a path walk. However I'm not very familiar with that part of the VFS and I see that the current RCU dcache might not lock that much during a path walk.
Quoted 6 lines
The main issue I see with all schemes in that direction (and something like that could be made workable) is the semantics of
unlink()on mountpoints. *Especially* with users being able to see attributes of files they do not own (e.g. reiser4 mode/uid/gid stuff). Ability to pin down any damn file on the system and make it impossible to replace is not something you want to give to any user.
I agree, users shouldn't be able to pin down a file.
I think unlink() should succeed on a file while something is visiting inside its metadata directory.
It's a filesystem quality-of-implementation feature whether that actually releases the file's data. It's a desirable feature because one user shouldn't be able to pin another user's quota'd data if they don't have permission to open the file, but if it's not implemented by a filesystem then it doesn't break anything fundamental.
It's a semantics question whether unlinking a file makes the metadata (i.e. "uid", "mode", "content-type" etc.) disappear at the same time, or if the metadata stays around until the last visitor leaves it. A filesystem might be able to keep the metadata in existence even if it deletes the file's storage on unlink(), but it would be nice for the VFS to declare which semantic is preferred.
One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way. In those cases, if you open one of the virtually generated "archive content" files, then you might expect the data to continue to exist after the underlying file is unlinked. I think that's reasonable:
being inside an archive is very similar to having it open. There is no quota pinning problem with this, because "archive content" files should inherit permission restrictions from the underlying file. If you can't read the file, then you can't read it's unpacked contents.
(reiser4 doesn't offer that last feature, but any of the myriad userspace filesystem hooks could offer it if the VFS has approprate auto-mounting file-as-directory hooks).
-- Jamie
Quoted 8 lines · Jamie Lokier
Quoted 5 lines · [email protected] wrote:
Quoted 2 lines
Is this a problem if we treat entering a file-as-directory as crossing a mount point (i.e. like auto-mounting)?
Yes - mountpoints can't be e.g. unlinked. Moreover, having directory mounted on non-directory is also an interesting situation.
Ok, so can we make it so mountpoints can be unlinked? :)
User-visible change of behaviour and IIRC a SuS violation on top of that.
Quoted 2 lines
I think the underlying file does not stay locked, and once you've entered it as a directory, it can be unlinked.
So why lock it at all in that case?
Quoted 5 lines
I didn't mean locking a chain of mountpoints, I meant the temporary state where two dentries and/or inodes are locked, parent and child, during a path walk. However I'm not very familiar with that part of the VFS and I see that the current RCU dcache might not lock that much during a path walk.
Never had been needed on crossing mountpoints, actually.
Quoted 4 lines
I agree, users shouldn't be able to pin down a file.
I think
unlink()should succeed on a file while something is visiting inside its metadata directory.
See above. Again, the fundamental problem with that is allowing unlink and friends on a mountpoint. I would love to do that, but it always generated -EBUSY on all Unices. Linux got a bit more users and userland code than Plan 9 - they can afford such changes, but...
And yes, from the kernel POV it's trivial to do - witness the MNT_DETACH codepath in umount - it's much simpler than "normal" umount exactly because it doesn't try to emulate old "it's busy, can't umount" behaviour.
With umount we could introduce "don't bother with that shit" flag. With unlink() we would have to make that default behaviour to be useful.
Quoted 6 lines · Jamie Lokier
One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way.
Yes, this was part of the plan, tar file-directory plugins would be cute.
Quoted 7 lines · Hans Reiser
Quoted 4 lines · Jamie Lokier
One of the big potential uses for file-as-directory is to go inside archive files, ELF files, .iso files and so on in a convenient way.
Yes, this was part of the plan, tar file-directory plugins would be cute.
Silly question:
GNU Midnight Commander allows for ages to go into e.g. tar files, so I know the benefits of this. Additionally, in GNU Midnight Commander, this works no matter which file system I use (e.g. it works on iso9660), and it even works the same way on other OS's like e.g. Solaris and NetBSD.
What is the technical reason why a tar plugin should be reiser4 specific, instead of a generic VFS or userspace solution that would allow the same also on other fs like e.g. iso9660?
cu
Adrian
Signature · 4 lines
"Is there not promise of rain?" Ling Tan asked suddenly out
of the darkness. There had been need of rain for many days.
"Only a promise," Lao Er said.
Pearl S. Buck - Dragon SeedAm Donnerstag, den 26.08.2004, 16:05 +0200 schrieb Adrian Bunk:
Quoted 4 lines
Quoted 1 line
Yes, this was part of the plan, tar file-directory plugins would be cute.
Silly question:
Quoted 8 lines
GNU Midnight Commander allows for ages to go into e.g. tar files, so I know the benefits of this. Additionally, in GNU Midnight Commander, this works no matter which file system I use (e.g. it works on iso9660), and it even works the same way on other OS's like e.g. Solaris and NetBSD.
What is the technical reason why a tar plugin should be reiser4 specific, instead of a generic VFS or userspace solution that would allow the same also on other fs like e.g. iso9660?
I don't think that this should be done inside the kernel. tar is a perfectly valid userspace application.
What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough).
But below page-cache and thus invisible from the VFS or the applications it is extremely useful for the compression plugin. When changing some data in the middle of the file the compressed data might grow in size. reiser4 can handle this inside the storage layer and doesn't mess with the rest of the filesystem.
Quoted 9 lines · Christophe Saout
What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough).
But below page-cache and thus invisible from the VFS or the applications it is extremely useful for the compression plugin. When changing some data in the middle of the file the compressed data might grow in size. reiser4 can handle this inside the storage layer and doesn't mess with the rest of the filesystem.
Never say never ... regular readers of -fsdevel already know what I'm going to say so can skip this post.
When compressing data on an ext2-like filesystem, you can use the property of "holes" to deallocate disc blocks in the middle of a file. It works like this:
Split the file into 64k chunks
Compress each chunk individually, then write them to disc on 64k boundaries, leaving gaps.
Now you can quickly seek to the appropriate 64k block for the data you're looking for and decompress just that 64k block.
This is how RISCiX managed to fit an 80MB distro onto a 50MB disc. Someone has written an equivalent filesystem for Linux called SquashFS.
Signature · 6 lines
"Next the statesmen will invent cheap lies, putting the blame upon the nation that is attacked, and every man will be glad of those conscience-soothing falsities, and will diligently study them, and refuse to examine any refutations of them; and thus he will by and by convince himself that the war is just, and will thank God for the better sleep he enjoys after this process of grotesque self-deception." -- Mark Twain
Quoted 3 lines · Christophe Saout
What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough).
That would be one of those "special features" that a VFS-plus-userspace implementation of archive views could take advantage of on reiser4, while using a slower method (sometimes much slower) on all other filesystems.
By the way, can reiser4 share parts of files between different files?
-- Jamie
Am Donnerstag, den 26.08.2004, 16:04 +0100 schrieb Jamie Lokier:
Quoted 9 lines
Quoted 3 lines · Christophe Saout
What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough).
That would be one of those "special features" that a VFS-plus-userspace implementation of archive views could take advantage of on reiser4, while using a slower method (sometimes much slower) on all other filesystems.
I'm just thinking about something. While you can't cut bytes out of unix iles a lot of filesystems can do this (holes). Most of them only on a block boundary, reiser4 on a byte boundary. If the filesystems could export this functionality using an enhanced API we could implement a compression plugin and other things on the VFS level that works with every filesystem supporting the required mechanisms, not only reiser4. And those features would take advantage of reiser4's storage mechanisms. I think Hans made the plugins reiser4-only because only reiser4 has a similar API at that time (and obviously because he didn't even think about doing it otherwise).
Quoted 1 line
By the way, can reiser4 share parts of files between different files?
At the moment a file is exactly one object. But someone could write another file plugin that spans a file across multiple objects, then yes, multiple files could share parts.
Quoted 10 lines · Christophe Saout
I'm just thinking about something. While you can't cut bytes out of unix iles a lot of filesystems can do this (holes). Most of them only on a block boundary, reiser4 on a byte boundary. If the filesystems could export this functionality using an enhanced API we could implement a compression plugin and other things on the VFS level that works with every filesystem supporting the required mechanisms, not only reiser4. And those features would take advantage of reiser4's storage mechanisms. I think Hans made the plugins reiser4-only because only reiser4 has a similar API at that time (and obviously because he didn't even think about doing it otherwise).
Yes. In some cases the compression or re-archiving algorithm would choose different parameters, depending on the alignment capabilities of the underlying filesystem. In some cases, that alignment is "offset 0 is the only one you can have", which is a severe special case where truncate and rewrite is required for changes.
Quoted 5 lines
Quoted 1 line
By the way, can reiser4 share parts of files between different files?
At the moment a file is exactly one object. But someone could write another file plugin that spans a file across multiple objects, then yes, multiple files could share parts.
Nice. That's useful for files which appear as themselves and also as part of a container.
And also for revision managed filesystems :)
-- Jamie
Quoted 13 lines · Jamie Lokier
Quoted 4 lines
Quoted 8 lines
Quoted 3 lines
By the way, can reiser4 share parts of files between different files?
At the moment a file is exactly one object. But someone could write another file plugin that spans a file across multiple objects, then yes, multiple files could share parts.
Quoted 11 lines
Nice. That's useful for files which appear as themselves and also as part of a container.
And also for revision managed filesystems :)
-- Jamie
Inheritance is one of the unimplemented reiser4 features that Vitaly is working on.
Quoted 11 lines · Jamie Lokier
Quoted 3 lines · Christophe Saout
What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough).
That would be one of those "special features" that a VFS-plus-userspace implementation of archive views could take advantage of on reiser4, while using a slower method (sometimes much slower) on all other filesystems.
By the way, can reiser4 share parts of files between different files?
It can, but fsck is supposed to cope with this. :)
Quoted 2 lines
-- Jamie
Nikita.
Quoted 11 lines · Jamie Lokier
Quoted 3 lines · Christophe Saout
What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough).
That would be one of those "special features" that a VFS-plus-userspace implementation of archive views could take advantage of on reiser4, while using a slower method (sometimes much slower) on all other filesystems.
By the way, can reiser4 share parts of files between different files?
no, those file plugins are not written yet :)
Do you mean COW files or some another thing? For COW files, the reiser4 support is not enough, we need to teach cp(1) or another utility to inform the fs layer that copying can be done by creation of COW files.
Quoted 1 line
-- Jamie
Signature · 1 line
Alex.
Quoted 3 lines · Alex Zarochentsev
Do you mean COW files or some another thing? For COW files, the reiser4 support is not enough, we need to teach cp(1) or another utility to inform the fs layer that copying can be done by creation of COW files.
it's easy enough to add that via an fs ioctl(), I did the equivalent last semester for some research I'm doing while using ext3cow (which unfortunately was put on hold for the summer), it's just a take off of sys_link(), a sys_cowpy() of sorts.
Quoted 9 lines · Alex Zarochentsev
Quoted 9 lines
Quoted 3 lines
What reiser4 can do, but the VFS can't is to insert or remove data in the middle of a file. Adding this above the page cache would probably be almost impossible (truncate seems already complicated enough).
That would be one of those "special features" that a VFS-plus-userspace implementation of archive views could take advantage of on reiser4, while using a slower method (sometimes much slower) on all other filesystems.
Quoted 5 lines
Quoted 1 line
By the way, can reiser4 share parts of files between different files?
no, those file plugins are not written yet :)
Do you mean COW files or some another thing?
COW files would be nice as well, but I meant another thing: for files which are parts of an uncompressed archive, it makes sense, if the fs offers it efficiently, to share the storage.
Doing it properly requires that memory is also shared, for example when COW files are used to isolate trees in virtual server jails, you still want the mapped executables to share memory.
You can't do that just by writing a plugin.
Quoted 3 lines
For COW files, the reiser4 support is not enough, we need to teach cp(1) or another utility to inform the fs layer that copying can be done by creation of COW files.
Actually we don't. The VM and VFS layers could deduce it with no changes to cp(1), by tracking which pages are identical due to read(3) and write(3). However, changing cp(1) is easier.
reiser4 support is obviously the first prerequisite...
-- Jamie
Quoted 3 lines · Adrian Bunk
What is the technical reason why a tar plugin should be reiser4 specific, instead of a generic VFS or userspace solution that would allow the same also on other fs like e.g. iso9660?
It should be a generic VFS plugin, not reiser4 or userspace. The VFS plugin should call out to userspace for most actions (except handling cached data), and it should take advantage of special reiser4 features for storage and performance optimisations. But it should still work over a standard filesystem, when those special features aren't available. I guess FUSE and many earlier projects are heading in this direction.
A generic userspace solution doesn't let you "cd" into a tar file from all programs like you can inside Midnight Commander.
Gnome and KDE take the approach that every userspace system call should be intercepted and filtered, to create the illusion of virtual data. As a result, different programs see different virtual data: you can't just cut and paste a path from Gnome or KDE into any other program. It's not just a "social problem of libraries" thing:
sometimes I have programs which don't link to libc. Sometimes I have programs which mustn't link with anything that calls malloc(). It'd be silly for them to have a different view of the filesystem just because they can't link with some userspace library.
The Gnome/KDE/Midnight Commander pure userspace solution is silly: if _every_ program in the system should get the same view, it makes much more sense for the kernel to filter the system calls and redirect the virtual accesses to a userspace daemon, while keeping the real accesses at full speed.
Furthermore is makes much more sense for the kernel's page cache to hold those uncompressed pages, than for every userspace application to try and cooperatively manage a cache of uncompressed fragments in the most inefficient way.
There's another problem with the Midnight Commander approach. If I "cd" into a tar file, and then a program writes to the tar file, I don't always see the changes straight away. The two views aren't coherent. This isn't an easy problem to solve, but it should be solved.
When a simple "cd" into .tar.gz or .iso is implemented properly, it will have _no_ performance penalty after you have first looked in the file, so long as it remains in the on-disk cache. And, the filesystem will manage that cache intelligently.
Imagine: for looking at source files and such, you probably won't bother untarring in future, and you won't bother keeping untarred source trees in your home directory for easy access to things you look at often. Why waste the space? You could install whole applications as a .tar and run them from within it, with no performance penalty.
Similarly, the filesystem will be able to archive directories automatically that haven't been touched in a long time, with no visible change except increased storage space. "grep" will be a bit slower, but you'll have a useful search tool by then (using coherent indexes) which will be more useful than grep, and much faster.
-- Jamie
Jamie Lokier <[email protected]> said:
[...]
Quoted 4 lines
When a simple "cd" into .tar.gz or .iso is implemented properly, it will have _no_ performance penalty after you have first looked in the file, so long as it remains in the on-disk cache. And, the filesystem will manage that cache intelligently.
Nonsense. The .iso or .tar or whatever would have to be kept un-isoed or un-tarred in memory (or on disk cache) for this to be true, and that takes quite a long time. Each time you want to peek anew at linux/Makefile, the whole tarfile will have to be read and stored somewhere, and that is just too slow for my taste. The .tar format is optimized for compact storage, the on-disk format of a filesystem is optimized for fast access and modifiability. Now go ahead and enlarge a file on your .iso/.tar a bit...it will take ages to rebuild the whole thing. There is a _reason_ why there are filesystems and archives, and they use different formats. If it weren't so, everybody and Aunt Tillie would just carry .ext3's around, and would wonder what the heck all this fuss is about.
Quoted 5 lines
Imagine: for looking at source files and such, you probably won't bother untarring in future, and you won't bother keeping untarred source trees in your home directory for easy access to things you look at often. Why waste the space? You could install whole applications as a .tar and run them from within it, with no performance penalty.
Perpetuum mobile is a nice idea too.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
I'm going to explain why filesystem support for .tar.gz or other "document container"-like formats is useful. This does _not_ mean tar in the kernel (I know someone who can't read will think that if I don't say it isn't); it does mean hooks in the kernel for maintaining coherency between different views and filesystem support for cacheing.
The vision I'm going for here is:
1. OpenOffice and similar programs store compound documents in standard file formats, such as .tar.gz, compressed XML and such.
Fs support can reduce CPU time handling these sorts of files, as I explain below, while still working with the standard file formats.
With appropriate userspace support, programs can be written which have access to the capabilities on all platforms, but reduced CPU time occurs only on platforms with the fs support.
2. Real-time indexing and local search engine tools. This isn't just things like local Google; it's also your MP3 player scanning for titles & artists, your email program scanning for subject lines to display the summary fast, your blog server caching built pages, your programming environment scanning for tags, and your file transfer program scanning for shared deltas to reduce bandwidth.
I won't explain how these work as it would make this mail too long. It should be clear to anyone who thinks about it why the coherency mechanism is essential for real-time, and a consistent interface to container internals helps with performance.
Quoted 10 lines · Horst von Brand
Quoted 4 lines · Jamie Lokier
When a simple "cd" into .tar.gz or .iso is implemented properly, it will have _no_ performance penalty after you have first looked in the file, so long as it remains in the on-disk cache. And, the filesystem will manage that cache intelligently.
Nonsense. The .iso or .tar or whatever would have to be kept un-isoed or un-tarred in memory (or on disk cache) for this to be true, and that takes quite a long time. Each time you want to peek anew at linux/Makefile, the whole tarfile will have to be read and stored somewhere,
Wrong. "So long as it remains in the on-disk cache" means each time you peek at linux/Makefile, the tarfile is _not_ read.
For a tarfile it's slow the first time, and when it falls out of the on-disk cache, otherwise, for component files you are using regularly (even over a long time) it's as fast as reading a plain file.
You obviously know this, as you mentioned on-disk cache in the reply, so I infer from the rest of your mail that what you're trying to say is more about modifications than reading archives. That it would be silly to keep working data in .tar.gz files, because working inside them regularly would be slow.
Which means you must be assuming, incorrectly, that these .tar.gz files are really kept up to date on disk with every component file modification.
Which is silly. .tar.gz files are suitable for *transport* and *archival*, not regular random access; it's almost rude of you to suggest I didn't know that.
The proposal is that .tar.gz files (and others) are analysed on demand and content cached on disk as it is read. Then subsequent reads will be as fast as if you had unpacked the archives by hand, manually using the tar command. This is obviously exactly the same as you do now, with a small bit of added convenience.
The other part of the proposal is that when you modify a component, the modifications are stored on disk in the same way as ordinary files, using the regular high performance random access disk structures. Nothing is done to recreate the archive at this point; I think this is where you misunderstood and thus flamed.
_If_ after modifying components, you then read the .tar.gz as a file, then (and only then) is it recreated, taking in the worst case the same time as running the tar command.
The _only_ times when that occurs are precisely those times when you would have run the tar command manually: because you only read the .tar.gz file when you need the flat file for some purpose, such as attaching it to an email, transferring by FTP or HTTP, or reading it into a program that needs it in that format.
If there is anything about that strategy that doesn't make sense, then I suggest I have failed to explain it properly, and you're welcome to demand a clearer explanation.
Quoted 2 lines
the .tar format is optimized for compact storage, the on-disk format of a filesystem is optimized for fast access and modifiability.
Actually no, .tar is not compact at all. It's also not optimised for random read access, but after an index is build it is very fast for that.
.tar.gz is compact. Although that is not especially fast for random read access, you can build a "compression dictionary index" which optimises random read access even in a .tar.gz, without ever unpacking the whole thing.
Some formats like .iso, .zip and .jar are optimised for compact storage _and_ fast random access. They come with an index, and don't need one to be built and cached.
These are not filesystem-like formats, obviously, but they are the formats you need to pack and unpack when exchanging data with other people. That's the _only_ reason they're on your disk (virtually; they may not really exist some of the time).
Quoted 5 lines
Now go ahead and enlarge a file on your .iso/.tar a bit...it will take ages to rebuild the whole thing. There is a _reason_ why there are filesystems and archives, and they use different formats. If it weren't so, everybody and Aunt Tillie would just carry .ext3's around, and would wonder what the heck all this fuss is about.
If you enlarge a file in your .iso/.tar subdirectory a bit... nothing happens. Why would a smart programmer do anything so silly as rebuild the archive at that point?
_If_ you subsequently read the .iso/.tar _file_, then and only then does it rebuild. Once, after lots of component writes. The only time you would ever do that is if you are specifically reading the archive file, which means you actually want to use the repacked file at that point, for example to FTP it somewhere or use as an email attachment.
If the filesystem does not do that on demand, then you would have run the tar command manually at that point, precisely because that's a point where you need the repacked archive. So in case where the filesystem repacks the archive, it takes exactly the same time as you would have taken anyway; it's just automatic instead of manual. (As a handy side effect, the automatic method offers lower latency for transmissions).
Now, why would we bother with all this?
I see three reasons: convenience, time efficiency, and storage efficiency.
Convenience is simply that it is handy to be able to look inside archive files, in those situations where we _currently_ use them, without having to manually untar when we need to, and without having to remember to clean up old directories when we discover we aren't using those often any more. This is _not_ an argument for using .tar.gz files in place of ordinary directories! Convenience applies to doing the things you do now.
Time efficiency has two angles. A simple one is that accessing .tar.gz contents through any kind of filesystem interface, even pure userspace, can be faster than unpacking whole files, simply because there are ways to decode parts without unpacking it all.
However, the main time efficiency that I see comes from the increasing number of applications where the "Open" and "Save" operations store data in *.gz files (e.g. OpenOffice compressed XML documents), or *.tar.gz files (some compound document formats), or other things like that. (If you think about it, quite a lot of things are like that).
With these, every "Open" currently has to decompress and maybe unpack an archive format. Every "Save" currently has to pack and then compress. This is done so the user sees a single flat file containing a complex document, but it is a waste of CPU time until the user actually transports the flat file.
The lazy proposal, as described earlier in this mail, _removes_ these decompression, unpacking, packing and compression CPU-intensive steps when they are unnecessary. The experience of a single file containing a complex document is maintained, but the CPU time is reduced in many typical operations. "Open" gets faster after you first look at a file, "Save" gets a lot faster for large documents, and the equivalent of grep (or later, real-time local search engine) gets a lot faster too. There is no operation where CPU time is overall increased.
This is what I've meant throughout this thread, when I say containers:
document files of the kind used to hold text, figures, etc. that are typically transported as a unit, and edited as a unit, but nonetheless at the moment they're stored in somewhat CPU intensive formats, for compactness. That's fine for a 1 page letter, but think of the OpenOffice 500-page book containing a large number of diagrams.
However, even simple programs that read & write compressed XML benefit.
The proposal allows that sort of thing to be handled more time efficiently that it is today, and in a way that is very practical to use.
(It's unthinkable that OpenOffice and similar programs would have a lot of code which stored data in a special way just for Linux, just for these performance benefits which are otherwise user-invisible, but it's thinkable that a general purpose userspace library which is portable to all platforms could be written, which takes advantage of the facility when it's available and does the equivalent of today's "compress on save" when the filesystem facility isn't available).
Finally, storage efficiency comes from simply allowing the filesystem and supporting tools to decide when it is best to store data in unpacked, packed & compressed, both at the same time, or another archival form. The filesystem has comparativaly good knowledge of which data to archive and when, but it can only maintain the illusion if there's a mechanism to make archived forms and unpacked forms coherent.
Now, I'm sure there is a way to implement this on top of a neat and simple kernel feature involving weird bind mounts, leases, dnotifies and FUSE. But those kernel offering are quite a mess at the moment and don't fit together in a way which can usefully create this effect.
Auto-mounting uservfs directories over file-as-directory, using moveable bind mounts _nearly_ offers the kernel primitives we need to build this in userspace and get the all the efficiencies. But not quite.
(We could obviously do it all in userspace by putting _everything_ in a userspace filesystem, but that would be silly as it would throw away all of the performance of having a threaded filesystem in the kernel. It might do as a proof of concept though).
-- Jamie
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 6 lines · Jamie Lokier
I'm going to explain why filesystem support for .tar.gz or other "document container"-like formats is useful. This does _not_ mean tar in the kernel (I know someone who can't read will think that if I don't say it isn't); it does mean hooks in the kernel for maintaining coherency between different views and filesystem support for cacheing.
I think that's a valid thing, but there are some fundamental problems with it if you expect it to work on a normal filesystem (ie something that isn't fundamentally designed as a database).
For example, _what_ kind of coherency do you think is acceptable? Quite frankly, using standard UNIX interfaces, absolute coherency just isn't an option, because it's just not possible to try to atomically update a view at the same time somebody else is writing to the "main file". "mmap()" is the most obvious example of this, but even the _basic_ notion of multiple "read" calls is not atomic without locking that is _way_ too expensive.
A "read()" on a file is not atomic even on the _plain_ file: if somebody does a concurrent "write()", the reader may see a partial update. This becomes a million times more confusing if the reader is seeing a structured view of the file the writer is modifying.
Also, it's likely impossible to write() to the view-file, again unless you expect all the underlying filesystems to be something really special.
So from a _practical_ standpoint, I suspect that the best you can really do pretty cheaply (and which gets you 90% of what you probably want) is:
- open-close consistency: the "validity" of the cache is checked at _open_ time, and no guarantees are given about the cache being updated afterwards.
- read-only access to the cache (ie you can only read the view, not write to it).
and quite frankly, I think you can do the above pretty much totally in user space with a small library and a daemon (in fact, ignoring security issues you probably don't even need the daemon). And if you can prototype it like that, and people actually find it useful, I suspect kernel support for better performance might be possible.
Suggested interface:
int open_cached_view(int base_fd, char *type, char *subname);
where "type" would be the type of the view (ie "tar" for a tar-file view, "idtag" for a mp3 ID tag, or NULL for "autodetect default view") and "subname" would be the cache entry name (ie the tar-file filename, or the tag type to open).
I bet you could write a small library to test this out for a few types. See if it's useful to you. And only if it's useful (and would make a huge performance difference) would it be worth putting in the kernel.
Implementation of the _user_space_ library would be something like this:
#define MAXNAME 1024
int open_cached_view(int base_fd, char *type, char *subname)
{
struct stat st;
char filename[PATH_MAX];
char name[MAXNAME];
int len, cachefd;
if (fstat(base_fd, &st) < 0)
return -1;
sprintf(name, "/proc/self/fd/%d", base_fd);
len = readlink(name, filename, sizeof(filename)-1);
if (len < 0)
return -1;
filename[len] = 0;
/* FIXME! Replace '/' with '#' in "type" and "subname" */
len = snprintf(name, sizeof(name),
"%04llx/%04llx/%s/%s/%s",
(unsigned long long) st.st_dev,
(unsigned long long) st.st_ino,
type ? : "default",
subname,
filename);
errno = ENAMETOOLONG;
if (len >= sizeof(name))
return -1;
cachefd = open(name, O_RDONLY);
if (cachefd >= 0) {
/* Check mtime here - maybe we could have kernel support */
return cachefd;
}
if (errno != ENOENT)
return -1;
/*
.. try to generate cache file here ..
*/see what I'm aiming at? You start out with a generic "attribute cache" library that does some hacky things (like depending on "mtime" for coherency) and then if that works out you can see if it's useful.
Linus
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 5 lines · Linus Torvalds
Quoted 5 lines
I'm going to explain why filesystem support for .tar.gz or other "document container"-like formats is useful. This does _not_ mean tar in the kernel (I know someone who can't read will think that if I don't say it isn't); it does mean hooks in the kernel for maintaining coherency between different views and filesystem support for cacheing.
Thanks for your reply.
Quoted 11 lines
I think that's a valid thing, but there are some fundamental problems with it if you expect it to work on a normal filesystem (ie something that isn't fundamentally designed as a database).
For example, _what_ kind of coherency do you think is acceptable? Quite frankly, using standard UNIX interfaces, absolute coherency just isn't an option, because it's just not possible to try to atomically update a view at the same time somebody else is writing to the "main file". "
mmap()" is the most obvious example of this, but even the _basic_ notion of multiple "read" calls is not atomic without locking that is _way_ too expensive.
Quoted 4 lines
A "
read()" on a file is not atomic even on the _plain_ file: if somebody does a concurrent "write()", the reader may see a partial update. This becomes a million times more confusing if the reader is seeing a structured view of the file the writer is modifying.
I agree. Coherency should be about equivalent to what writing flat files offers.
(For now. Eventually I think we'll see transactional I/O in Linux, to solve the sort of updater-to-server glitches which are currently solved by putting even trivial data behind clunky database servers).
The coherency I'd like would also ensure that you'll never see an invalid file structure in the long term, even if during writing operations you might see temporarily invalid structure.
That is easy if you only have read-only views.
Quoted 2 lines
Also, it's likely impossible to
write()to the view-file, again unless you expect all the underlying filesystems to be something really special.
Right. I wonder if you read the part of my message which deals with lazy updates of container files.
The idea is that write() to a view-file doesn't repack the container file until the container file is read.
Practically, that means the view-file's write handler, which is probably in userspace, grabs a kind of mandatory lock (similar to F_SETLEASE) on the container file and then truncates it. After a time, or when a program tries to read the container (whichever comes first), the view-file's handler is notified, it regenerates the container file, and releases the lock.
This is the form of coherency needed to make writes work properly.
(You can take it more fine grained, locking at the page level, but that's just to improve performance some more. It's not fundamental.)
Notice that it doesn't need a special filesystem. View-file writes will work with any ordinary filesystem. A special filesystem would make it perform better (much better), by allowing the truncated state to persist across reboots, with an xattr to say what's needed to recreate the data -- but it's not fundamentally necessary.
It does need kernel support for that kind of lock, which F_SETLEASE isn't, and notifications of mount events, and notifications of file writes.
Quoted 8 lines
So from a _practical_ standpoint, I suspect that the best you can really do pretty cheaply (and which gets you 90% of what you probably want) is:
- open-close consistency: the "validity" of the cache is checked at _open_ time, and no guarantees are given about the cache being updated afterwards.
- read-only access to the cache (ie you can only read the view, not write to it).
I think that gets 90% of the useful features, but leaves out some applications which would raise eyebrows when they actually work -- i.e. the component editing applications.
Quoted 5 lines
and quite frankly, I think you can do the above pretty much totally in user space with a small library and a daemon (in fact, ignoring security issues you probably don't even need the daemon). And if you can prototype it like that, and people actually find it useful, I suspect kernel support for better performance might be possible.
Right now it's stupidly impossible for a daemon to monitor a file for changes and not be tricked by mtime faults, because someone can link to the file and modify it on another path. That's obviously a bug in dnotify which I keep meaning to fix. There are other dnotify problems, some of which are fundamental when it's over a network.
I do care about security issues, for things like an "md5" or "compiled" attribute especially, but for a prototype that's just something to live with.
Quoted 3 lines
Suggested interface:
int open_cached_view(int base_fd, char *type, char *subname);
Something userspace is useful anyway, so that a fully userspace alternative is available, so that people writing apps will take up the approach and take advantage of fs-level support where available while still being portable elsewhere.
Quoted 3 lines
see what I'm aiming at? You start out with a generic "attribute cache" library that does some hacky things (like depending on "mtime" for coherency) and then if that works out you can see if it's useful.
Ok but I don't think that form of it is useful! It's the sort of interface that specific attribute-using programs will have to use if they're to be portable, but it doesn't provide any special advantages to any other programs.
Gnome VFS, KDE etc. provide much of that kind of interface already. Evidently some people find that useful. But I don't, precisely because it's so tied into one set of programs or another.
This is where the file-as-directory metadata stuff is so potentially interesting. It's actually a nice interface that every program can see.
It's the nice interface when running on Linux that'll make it deliciously worthwhile for portable applications, such as MP3 retaggers and PDF image extractors, to be written in the form of a tool plus a plugin for a common view-file library, instead of all in the tool as they are now.
Without the feature of a nice interface on Linux, there's no reason why application writers would bother to learn and use a view-file library.
To be fair, the whole thing could be prototyped (with glitches, but enough to demonstrate) in userspace, running everything through something like uservfs or even NFS, especially with file-as-directory being blessed at the VFS level.
That is exactly the right way to go about it. I started some work on that back in '99, around the same time Pavel was exerimenting with podfuk (which become uservfs), but haven't and don't have the time or money(*) to take it further.
-- Jamie
(*) That's a hint to potential employers who want to support this work, by the way.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 3 lines · Jamie Lokier
This is where the file-as-directory metadata stuff is so potentially interesting. It's actually a nice interface that every program can see.
.. but you can _still_ do exactly this in a user-space library and prototype it. Just take over "open" (and opendir/readdir), and you can do so.
Quoted 4 lines
To be fair, the whole thing could be prototyped (with glitches, but enough to demonstrate) in userspace, running everything through something like uservfs or even NFS, especially with file-as-directory being blessed at the VFS level.
I do believe that you can prototype it even without any kernel support.
This is why I want to have stronger reasons for real VFS support. I see the Samba argument, and I buy that one, if only because it has such a high consistency requirement (let's face it, people expect more guarantees of their file servers than of most other things). But on the other hand, the Samba guys apparently _are_ happy with "openat()", so they don't necessarily need any normal namespace stuff.
And I suspect that a user-level prototype would show that what you _really_ want may be something else altogether..
Linus
Why do we need openat? (was: The argument for fs assistance in handling archives)
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Linus" == Linus Torvalds <[email protected]> writes:
[...]
Linus> This is why I want to have stronger reasons for real VFS Linus> support. I see the Samba argument, and I buy that one, if only Linus> because it has such a high consistency requirement (let's face Linus> it, people expect more guarantees of their file servers than of Linus> most other things). But on the other hand, the Samba guys Linus> apparently _are_ happy with "openat()", so they don't necessarily Linus> need any normal namespace stuff.
Maybe things have got lost in this thread (or these many threads), so bear with me, but what exactly does openat() gain us that we can't get from exporting things from the normal file namespace? The only issue that I've seen raised is that you need to be able to distinguish between the contents of a directory and its substreams. And I've responded that streams should be stuck in a specially-named subdirectory (like ..streams) (which should be done even if only files could contain substreams), to which I have seen no objections raised (not even as much as a "you're stupid"), and so can only assume nobody (who cares) has any objections.
So as far as I can see, openat() has no advantages, and exporting things through the normal namespace has several advantages, such as:
- can manipulate substreams using legacy programs -- no modifications needed. And don't tell me "runat" until someone shows me how to deal with my GIMP/emacs problem.
- useful for exporting different things (streams, acl's or other access control schemes, xattrs, version control, automatic untar, ...)
- only need to modify tar (and backup programs) once -- any new filesystem extensions, no matter how complex, can be exported through the filesystem interface, and tar will be able to catch them.
- referring to the substreams is simpler ("foo.txt/streams/bar" instead of "the bar stream of foo.txt"), which leads to simpler user interface. This is even more marked if we, for some strange reason, discover that it's a good idea for streams to be allowed to have substreams of their own ("foo.txt/streams/bar/streams/baz" vs. "the baz stream of the bar stream of foo.txt")
- I think I'm forgetting some...
If this is indeed the case, then it seems to me that the logical choice is to export streams through the normal file namespace.
Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Re: The argument for fs assistance in handling archives
In reply to Linus Torvalds-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Linus Torvalds wrote:
[...]
| This is why I want to have stronger reasons for real VFS support. I see | the Samba argument, and I buy that one, if only because it has such a high | consistency requirement (let's face it, people expect more guarantees of | their file servers than of most other things). But on the other hand, the | Samba guys apparently _are_ happy with "openat()", so they don't | necessarily need any normal namespace stuff.
I see your Samba and I raise you an NFS.
For that matter, how about every distributed FS to come.
All it needs to do is support transactions (which most try to anyway) and files-as-directories to support _all_ future FS features transparently.
It might ask the fs for a non-recursive serialization/dump of a file rather than just reading it directly -- this makes things like named pipes and device nodes sane. This would be optional and only supported when the client and server both have support for the same fs -- other, more hack-ish things would be needed to support Samba.
It would also use the best caching mechanism discussed earlier. (I'm not yet arrogant enough to insist that mine is the best.) -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTaXs3gHNmZLgCUhAQIG5g/+MEHPsNK41goKI8yJPwKdigbRbrfGf3sj xpFGR15/DAW4b4qxr1BIt5hyAQoFLq13POM/lCV12hKp1RSo9atuKim6DyUhOX1c 0xUK6efY0ew+mvYHKaJUgwrD8BV7XYaTS+5y/4wbpgdOlPHsx06lHVh90KtAUAE7 vdsFWRAgHT/g34rSRBJfHAaom/7VtDce25roLrB3CNI2v+9tERYee+xkBPoy2+Y1 5tLgoInDi05z7bUu0lCJ+Rq2LCHuKGmgInI86gul5BfHlO3j2zYnJ5VYT8QR6onM iThiPzFrrpoXEjQRRo7E4WHQCD/2BGXzQdS78JzwTt1B8oNrIkZR9/tcv6keNzr6 fyUhtQef5TZJtwpON1gZPqraBUiTDUdWHJSXGLB5+78vMy+bS7gHqXN5DR049+7b urQFtnZBhi3zY8cpT0hTPnW2PcGnDuGMt9qUiFBn8Hx9Tkti6y+7Jl42a40jgBAW z4hSHPKVzu8d/qeBapVHtqIkaLTdim1JnmFY+SCiiuCPEMOELDxw+o/rcwTIls9s 3TBJ5lqL/+84W8JU6t1CpzEgF+dRhPSBNyrG5msVixT7cmB2GO87rLBfchM1cotq lBMyuDwmwO6RanwkEvOV//3wsG8POMw5f2Gx8eljqU9uzSpYcKry+Is2ZVmHKJ/1 24eqEXIB24Y=
=GSMf
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
In reply to Jamie Lokier-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Jamie Lokier wrote:
[...]
|>A "read()" on a file is not atomic even on the _plain_ file: if somebody |>does a concurrent "write()", the reader may see a partial update. This |>becomes a million times more confusing if the reader is seeing a |>structured view of the file the writer is modifying. | | | I agree. Coherency should be about equivalent to what writing flat | files offers. | | (For now. Eventually I think we'll see transactional I/O in Linux, to
Eventually? This was supposed to be in reiser4, maybe it's been put off until 4.1...
And that is the right solution. Not the only one, but the right one. Caching isn't the only thing sorely in need of transaction support right now. Actually, I find it hard to think of anything on Linux which shouldn't have transactions -- why should /etc/fstab or /home/david/homework be more fragile than /var/lib/mysql?
|>Also, it's likely impossible to write() to the view-file, again unless you
|>expect all the underlying filesystems to be something really special. | | | Right. I wonder if you read the part of my message which deals with | lazy updates of container files. | | The idea is that write() to a view-file doesn't repack the container | file until the container file is read. | | Practically, that means the view-file's write handler, which is | probably in userspace, grabs a kind of mandatory lock (similar to
In userspace? I don't buy that -- this is exactly what a reiser4 plugin is for, just in OOP technology. Instead of "write handler", it's probably called "write method".
| F_SETLEASE) on the container file and then truncates it. After a | time, or when a program tries to read the container (whichever comes | first), the view-file's handler is notified, it regenerates the | container file, and releases the lock.
Why "after a time"? Should be "when a program tries to read the container" -- and with transactions or even some clever locking, it can be just as consistant as the flatfile write.
[...]
| (You can take it more fine grained, locking at the page level, but | that's just to improve performance some more. It's not fundamental.)
Actually, "locking at the page level" rarely makes more sense than "use two files". And with transactions, I think it works like this:
Process A opens file.tar.gz/contents/foo.c and starts writing. When it's maybe halfway through, process B opens file.tar.gz for reading. So it gets the version of foo.c that existed before A started writing. B finishes before A does, and then process C opens file.tar.gz for writing
- -- it has to wait for A to complete.
| Notice that it doesn't need a special filesystem. View-file writes | will work with any ordinary filesystem. A special filesystem would | make it perform better (much better), by allowing the truncated state | to persist across reboots, with an xattr to say what's needed to | recreate the data -- but it's not fundamentally necessary.
I am hoping that someone will implement a VFS-based storage layer plugin for reiser4. This means that one can use reiser4 interface features (such as reading file.tar.gz as a directory) without needing to reformat. It also means that one could conceivably port reiser4 to a userland library, even. I think.
I wonder if it's as much of a performance hit as uservfs.
[...]
|>Suggested interface: |> |> int open_cached_view(int base_fd, char *type, char *subname);
Oh, God no. Alright, it's more useful than no caching at all, but still...
Alright, let's say I have /usr/src/linux as a hardlink farm to /usr/src/linux.tar.bz2/contents/. With kernel support, I can actually do a make from inside /usr/src/linux and have it behave sanely.
Do you want to patch make? If yes, you may as well patch everything. In fact, you may as well patch glibc to replace open with your open_cached_file. Not to mention handling things like readdir, stat, and so on...
Not to mention the overhead of checking if a file is part of a cache for every single open, ever...
Putting it in the filesystem simplifies things. There's already some sort of read/write method for the file -- overload that for files which need to be fetched and cached. Files which are already in the cache, or have nothing to do with the cache, remain exactly as fast as they are now.
I'm not sure about the best way to make this portable between filesystems. I'm not sure if it's really that useful to do so. I'm beginning to see why Hans would like reiser4 to be the new VFS.
| Something userspace is useful anyway, so that a fully userspace | alternative is available, so that people writing apps will take up the | approach and take advantage of fs-level support where available while | still being portable elsewhere.
Maybe. But here, I'd suggest that the kernel implementation is fine, too. AFAIK, Reiser4 was designed to be portable -- specifically because certain apps -- not OSes, but apps -- might embed it for a fee.
|>see what I'm aiming at? You start out with a generic "attribute cache" |>library that does some hacky things (like depending on "mtime" for |>coherency) and then if that works out you can see if it's useful. | | | Ok but I don't think that form of it is useful! It's the sort of | interface that specific attribute-using programs will have to use if | they're to be portable, but it doesn't provide any special advantages | to any other programs. | | Gnome VFS, KDE etc. provide much of that kind of interface already. | Evidently some people find that useful. But I don't, precisely | because it's so tied into one set of programs or another. | | This is where the file-as-directory metadata stuff is so potentially | interesting. It's actually a nice interface that every program can see.
Exactly! Every program should be able to see the interface. The interface should also be transparent sometimes. Example -- reimplementing InterMezzo/Lustre. Files that are cached are exactly as fast to write as local files. To invalidate the cache, you delete the file.
| Without the feature of a nice interface on Linux, there's no reason | why application writers would bother to learn and use a view-file library.
That's not as compelling a reason. Application writers don't care at all whether it's in the kernel or userland. All putting it in the kernel does for adoption is make everyone sit up and notice.
[...]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTaU5XgHNmZLgCUhAQJsBQ//WBWU28IrQc8mFokLyeFtIAIErqUE65oj TDcx0/Dg+2JD4zAMht6idhMDfYU5kvBj5C9srMvdklprgPJ/dIQSS/MS8o6qnZl4 kRvjUwI/vBV9a2gwxzt32NrLK+ZFuWwOiRJ1ia6zOyxqOE/jx8z1VDR7R17lbakh EbFG/2taoHyRke4O+rxig93OxTUBJ72KBKZ+LGrDMwzm7uNP069UupWO0ARpBlMM 6T0o8bm4QK7dgqXgowDGx22hFo5ZPLc3undsHepnu+i8cZMPpOMigbgwpiG/VNnu t00WOrJGigNkeOrCB4y4LO6LQFL7DjLRRI3T9Y3t3YUFegYDdG6A1L2ZQcrxqtlj alSx6YYYigH2+MBaN+ONIPlZVe9ybdTpAWl+hkS1nvKaNVc5xYUTFBhBAzIr/4Ol Zuu5sE8/Q7+5oOCqp9zW51g4oBIlDE0CY3EoZ1P5UrQXT5KphVWr8AUXcP+9CL5c vsIbYWTmbU/XgxIb+lyB2anTGnMzy1TxUHVxAR4ijXPSDuAbojJhtg1VuDMGhahq NfW34jRbDJSPDoar/eN2jxF8a4bG7xt8w10LO0lnnjznNJdsurk4HmsHFhPZ0DJj OJTaChsEr+KUldLyNila8A+6Qg4LvSvNSPArXxCTkGizN+jeyFlNSgqFOQajS/Pk fptJFhA1x3Q=
=dUCH
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 6 lines · David Masover
And that is the right solution. Not the only one, but the right one. Caching isn't the only thing sorely in need of transaction support right now. Actually, I find it hard to think of anything on Linux which shouldn't have transactions -- why should
/etc/fstabor/home/david/homeworkbe more fragile than/var/lib/mysql?
It's easy to talk big.
It's damn hard to _implement_ a complex system, and make it stable and bug-free, and support legacy applications.
There is a reason why we do only what _must_ be done in kernel space.
Linus
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Linus Torvalds wrote:
| | On Wed, 1 Sep 2004, David Masover wrote: | |>And that is the right solution. Not the only one, but the right one. |>Caching isn't the only thing sorely in need of transaction support right |>now. Actually, I find it hard to think of anything on Linux which |>shouldn't have transactions -- why should /etc/fstab or |>/home/david/homework be more fragile than /var/lib/mysql? | | | It's easy to talk big. | | It's damn hard to _implement_ a complex system, and make it stable and | bug-free, and support legacy applications. | | There is a reason why we do only what _must_ be done in kernel space. | | Linus
I think transactions should be done in kernel space. It's the only way to "enforce" them sanely.
Besides, I'm not the one who first suggested this transaction idea. That was Hans, and it was in the whitepaper. I don't know if it's done yet -- I think it got put off for v4.1.
But if it is going to be done anyway, doesn't it solve the consistency issue?
I can understand wanting caching to work now, but which is the priority
- -- caching or transactions? I vote transactions. The consistancy of fstab is far more important to me than the consistancy of foo.tgz/bar.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTahTXgHNmZLgCUhAQIumQ//SqEViCUbBcFJ3xsiQfEEUSe2gU+6dyJZ 7i3OeexnLRKrnxIsij5SmYTmGdYXhpaQWMrrjie8uFX0ocA0HTMsA1BzB4ZhtKnO sYc/jAZusGpAjnyy0t0Q/hwjoMRIsCRVwqnD17Qw8rcuwYT5yaABMboXPiOPjQqu JTkCeb+ATq5hF9VYL9TbcqaOycV7BMh+N7VXGPS/PIgeyYXyNwfIBWHrN8sjVQDA 3h9a9IEjEyReMdqlNbv73/wggGNqsmprDLIMFhMSPP4LXS5/xt1AhXW1R+mNDXjA ckNq1lLjD1iUIQ7gIPHjR1Vi5op5GaXqVtdJLXtTyvNuc3nM/yFzM0T7+hgO4rSj F0VkFFV8FbKRSVv1MsPv1O8EpFNpAs1uX+1g/vZ4ZHwPuOaMeXKT6rVTsr9WesfU NdPMi0bXF6fAHbpjw3k6v+Jg95Mg+w8bgYOEDptz6zRuex8xC+LPrXE9ZDrYX+nm +xG2wtMZyMr2y8wMp71RLz6Sj+Vq0cp4HNqccsMLfZ1OEWu2IzneBAmCSGQjLrxL 9coj0G3WpFwspTClh1DGBfuBJPS5O33iEsJ2w80bMhy/wKK5kZG/NyMAGQKazxAr +7Se6t2RGHIxX5/EEoPI/e0AKLxFuvPi9ilq4bJNNDBMUO2wOkB8V0GLGzFA/0F2 9ALDAaO/dg8=
=pvxM
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 3 lines · David Masover
I think transactions should be done in kernel space. It's the only way to "enforce" them sanely.
Well, they clearly can't, since it depends on the filesystem. Also, transactions have a tendency to be expensive, and people have a tendency to ask for more than you give them. Doing transactions on one file is only the beginning - you'll find people who want transactions across file boundaries etc.
That's a basic fact when it comes to pretty much anything: you can always find people who want something better. You can't be all things to all people, so what you actually _want_ to do is to expose the _minimal_ set of capabilities that people can build on.
So I definitely don't want this discussion to degenerate (any more than it lng since has ;) into what people _wish_ for. No, it should be a "what is the _least_ we can absolutely do" that solves real problems.
Linus
Re: The argument for fs assistance in handling archives
Quoted 4 lines · Linus Torvalds
. Doing transactions on one file is only the beginning - you'll find people who want transactions across file boundaries etc.
Yup, that's what reiser4 is aiming at and where things get exciting for version control systems and the like. We are just missing the api code for it at the moment, all the infrastructure is there, and the api is in progress in sys_reiser4.
Re: The argument for fs assistance in handling archives
Quoted 10 lines · Hans Reiser
Quoted 4 lines · Linus Torvalds
. Doing transactions on one file is
only the beginning - you'll find people who want transactions across file boundaries etc.Yup, that's what reiser4 is aiming at and where things get exciting for version control systems and the like. We are just missing the api code for it at the moment, all the infrastructure is there, and the api is in progress in
sys_reiser4.
But _my_ point is, no user program is going to take _advantage_ of anything that only one filesystem on one system offers.
So there's no point.
It's much saner (from _any_ app standpoint) to roll their own database in user space - that way it just works.
In other words, nobody is really ever going to take advantage of this. This is _not_ how technical advanncement happens. The way you get people to take advantage of something is to have a nice gradual ramp-up, not a sudden new feature that they can't realistically use.
So before you do transactions in the filesystem, you have to be able to do them in user space - and then make the filesystem version be _compatible_ with that user space library. That way you can get people to use the library ("hey, it works anywhere") and then you can get them to use your filesystem ("hey, if you use our super-duper filesystem, you can do what you are already doing five times faster").
See? You're starting at the wrong end. Give me a portable interface to use _first_. Then do transactions in the filesystem.
(Now, as an academic exercise and as a learnign experience and prototyping work it migt make sense to do the filesystem first. But then you have to really _consider_ it a prototype, and realize that to make it useful you'd still have to do the user thing before you can "sell" people on the idea).
Linus
Re: The argument for fs assistance in handling archives
Quoted 5 lines · Linus Torvalds
But _my_ point is, no user program is going to take _advantage_ of
anything that only one filesystem on one system offers.
Apple does not have this problem....
and yes, the apps will take advantage of it, which is different from depending on it. If you use the wrong fs you will lose some of the features of the app.
For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice (excepting Clearcase, which priced itself into a niche market). It is 25 years past time for someone to change things. That someone will have first mover advantage, and the more little semantic features possessed the more lure there will be to use it which will increase market share which will lure more apps into depending on it and in a few years the other filesystems will (deservedly) have only a small market share because the apps won't all work on them.
Besides, there are enhancements which are simply compelling. You can write a dramatically better performance version control system with a much simpler design if the FS is atomic. Our transaction manager first draft was written by a version control guy, and he would probably be happy to tell you how lack of atomicity other than rename makes version control software design hideous.
We have the performance lead. By next year we will be stable enough for mission critical servers, and then we start the serious semantic enhancements.
If you don't embrace progress, then you doom Linux to following behind, because the guys at Apple are pretty aggressive now that Jobs is back, and they WILL change the semantics, and they will do so in compelling ways, and Linux will be reduced to aping them when it should be leading them.
Hans
Re: The argument for fs assistance in handling archives
How would we go about finding out how many data forks were in a file? Because in order to be able to retrieve data from a fork we would need to know that the fork were there. Currently this would imply that we go looking through mtab or some such to find out what fs we're running on, which seems ugly.
Alternatively we go through the _exciting_ task of making every other fs (with the exceptions of ntfs, and whatever it is that macs use, which would need there own custom code) and add code that effectively goes
getNumForks(fileref){ return 1;}
or add a new open call that can take a fork number...
either way we have to add a new syscall, that doesn't conform to any real standard(though i suppose it would be possible to use macOS/windows style fork iopening interface)
I personally like the concept of having multiple forks in a file, but in this case I'm inclined towards usermode first, then if it takes off add kernel level support.
--Oliver Hunt
Quoted 49 lines · Hans Reiser
Quoted 5 lines · Linus Torvalds
But _my_ point is, no user program is going to take _advantage_ of
anything that only one filesystem on one system offers.
Apple does not have this problem....
and yes, the apps will take advantage of it, which is different from depending on it. If you use the wrong fs you will lose some of the features of the app.
For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice (excepting Clearcase, which priced itself
into a niche market). It is 25 years past time for someone to change
things. That someone will have first mover advantage, and the more little semantic features possessed the more lure there will be to use it which will increase market share which will lure more apps into depending on it and in a few years the other filesystems will (deservedly) have only a small market share because the apps won't all work on them.Besides, there are enhancements which are simply compelling. You can write a dramatically better performance version control system with a
much simpler design if the FS is atomic. Our transaction manager
first draft was written by a version control guy, and he would probably be happy to tell you how lack of atomicity other than rename makes version control software design hideous.We have the performance lead. By next year we will be stable enough for mission critical servers, and then we start the serious semantic enhancements.
If you don't embrace progress, then you doom Linux to following behind, because the guys at Apple are pretty aggressive now that Jobs is back, and they WILL change the semantics, and they will do so in compelling ways, and Linux will be reduced to aping them when it should be leading them.
Hans
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Re: The argument for fs assistance in handling archives
Quoted 7 lines · Oliver Hunt
How would we go about finding out how many data forks were in a file? Because in order to be able to retrieve data from a fork we would need to know that the fork were there. Currently this would imply that we
go looking through mtab or some such to find out what fs we're running
on, which seems ugly.
Depends on how the forks eventually get implemented.
With the file-as-directory concept, all you need is to look at the file's directory part to see what is there. (The forks, implemented as files in a subdirectory.) It is done the same way as for an ordinary directory, so nothing "new".
Quoted 9 lines
Alternatively we go through the _exciting_ task of making every other fs (with the exceptions of ntfs, and whatever it is that macs use, which would need there own custom code) and add code that effectively goes
getNumForks(fileref){ return 1;}
Necessary if some other mechanism is used, sure.
Helge Hafting
Re: The argument for fs assistance in handling archives
Quoted 6 lines
Depends on how the forks eventually get implemented.
With the file-as-directory concept, all you need is to look at the file's directory part to see what is there. (The forks, implemented as files in a subdirectory.) It is done the same way as for an ordinary directory, so nothing "new".
This still doesn't solve the problem of how we distinguish between a multi-fork file
and a directory, for those old programs(ie. almost all that currently exist), to be able to access meaningful data we would need to either return just the primary fork, a serialized version of all forks in the file, or make the file look more or less identical to a directory.
The first option could cause problems when transfering files between different filesystems,
the second results in programs getting metadata they can't handle, and the third option results in few of the advantages over, well, directories... And even those applications that could handle the fork information nicely would need to fs type to find out whether they were handling a directory or a multi-forked file...
As I say I like the idea, but I can't see anyway of implementing it in a way that is useful without first putting considerable effort into at least the VFS if not all the actual fs drivers.
--Oliver Hunt
Re: The argument for fs assistance in handling archives
Quoted 9 lines
Quoted 9 lines
Depends on how the forks eventually get implemented.
With the file-as-directory concept, all you need is to look at the file's directory part to see what is there. (The forks, implemented as files in a subdirectory.) It is done the same way as for an ordinary directory, so nothing "new".
Quoted 6 lines
This still doesn't solve the problem of how we distinguish between a multi-fork file
and a directory, for those old programs(ie. almost all that currently exist), to be able to access meaningful data we would need to either return just the primary fork, a serialized version of all forks in the file, or make the file look more or less identical to a directory.
Well. wasn't the idea that unless programs specifically tried to open the file-as-dir as a directory it would look like a file?
ls -F would show it as file. Or have I understood wrong?
Quoted 2 lines
The first option could cause problems when transfering files between different filesystems,
The meta-data should be deleted if it the file is copied or moved to a medium that doesn't support it. However a _warning_ may be shown to the user if there is risk to loose data.
I do not think we should not implement something like this because most other filesystems in Linux doesn't support it. Think of other limitations there is in various ones. Large file support. Limited amount of files, length of file names, etc.
Quoted 5 lines
the second results in programs getting metadata they can't handle, and the third option results in few of the advantages over, well, directories... And even those applications that could handle the fork information nicely would need to fs type to find out whether they were handling a directory or a multi-forked file...
No, if an application supported the file-as-dir then it could simply try to open the file as a directory. It would fail on systems that didn't support it.
The meta-data and file streams would be seen as ordinary files. If applications support the contents or not isn't really relevant, is it?.
Quoted 3 lines
As I say I like the idea, but I can't see anyway of implementing it in a way that is useful without first putting considerable effort into at least the VFS if not all the actual fs drivers.
Indeed. It is important that something like this gets implemented as a transparent way as possible. If it could be done in a general way so other filesystems like ext3/4 can eventually support it then that would be wonderful. I do not, however, think that we should block it in reiser4 because no other filesystems support it.
~S
Quoted 1 line
--Oliver Hunt
Re: The argument for fs assistance in handling archives
Quoted 4 lines · Spam
Well. wasn't the idea that unless programs specifically tried to open the file-as-dir as a directory it would look like a file?
ls -F would show it as file. Or have I understood wrong?
Yes, otherwise apps that do
if (S_ISDIR()) {
..
} else if (S_ISREG()) {
..
}would behave differently from apps that check the file in different order, and they would probably break because in a regular fs that order is not important. The goal is to keep the semantic changes as hidden as possible for apps that don't know about the new features.
--
Giuliano.
Re: The argument for fs assistance in handling archives
In reply to SpamQuoted 9 lines · Spam
Quoted 3 lines
As I say I like the idea, but I can't see anyway of implementing it in a way that is useful without first putting considerable effort into at least the VFS if not all the actual fs drivers.
Indeed. It is important that something like this gets implemented as a transparent way as possible. If it could be done in a general way so other filesystems like ext3/4 can eventually support it then that would be wonderful. I do not, however, think that we should block it in reiser4 because no other filesystems support it.
What about extending the namespace with leading ".." and "...".
In Unix names starting with a "." already have the meaning of being a hidden/config entry.
A name starting ".." means streams/metainformation, etc. something belonging to the file/directory that should get backed up, copied, etc. There could be a "..streams", "..metas" or a "..acl" entry.
If it starts with "..." it means some system specific information, like the name of the hash algorithm used for the current directory. It's basically information that is not portable and not required by applications, like "...fsplugins".
There is still the big problem of how to copy files with their associated streams or meta information onto a standard unix filesystem as the file/directory duality cannot be expressed. (It's forbidden to have a directory + file with the same name in the same directory) Maybe copy could create something like a ".#filename" directory for this kind of information if the advanced features are not supported on the target filesystem. This is neither nice nor clean, but at least you don't loose information. I do not suggest that the kernel should simulate the advanced features with ".#filename" directories, it's more a backup/restore thing that could work if it's written down properly.
Signature · 1 line
lg, Chris
Re: The argument for fs assistance in handling archives
In reply to SpamQuoted 3 lines · Spam
The meta-data should be deleted if it the file is copied or moved to a medium that doesn't support it. However a _warning_ may be shown to the user if there is risk to loose data.
OK... I'll bite. How do you report the warning to the user if you're using an unenhanced utility to copy a file to a file system that may be lossy?
Re: The argument for fs assistance in handling archives
Quoted 9 lines · Spam
Quoted 4 lines
The meta-data should be deleted if it the file is copied or moved to a medium that doesn't support it. However a _warning_ may be shown to the user if there is risk to loose data.
OK... I'll bite. How do you report the warning to the user if you're using an unenhanced utility to copy a file to a file system that may be lossy?
That I do not really know. But I think this is somewhat the discussion here - how to make the support for metas and streams as good as possible.
I suppose it would be difficult to actually query the user about something like this unless the applications themselves do. Perhaps warnings in the syslog is good for now?
Already now things like advanced attributes get lost when copying data from a supported fs to a non-supported one.
Perhaps documentation around the ways to use streams and meta data should mention that only the supported fs will keep the extra info.
As long as the user is aware that whatever information he puts in the meta or streams will be lost then it is ok, isn't it?
~S
Re: The argument for fs assistance in handling archives
In reply to [email protected]Quoted 4 lines · [email protected] wrote:
On Thu, 02 Sep 2004 12:02:59 +0200, Spam said:
Quoted 10 lines
Quoted 5 lines
The meta-data should be deleted if it the file is copied or moved to a medium that doesn't support it. However a _warning_ may be shown to the user if there is risk to loose data.
OK... I'll bite. How do you report the warning to the user if you're using an unenhanced utility to copy a file to a file system that may be lossy?
Well, _if_ you're using my sort of file-as-dir then you'd expect a plain "cp" to only copy the file. You'd use cp -a to copy the file's subdirectory too. If you're copying to anther filesystem that doesn't support file-as-directory then "cp -a" won't be able to create a directory with the same name as the file, and will do whatever cp does in such a situation. I.e. something like "NAME exists but is not a directory".
Helge Hafting
Re: The argument for fs assistance in handling archives
In reply to Oliver HuntQuoted 8 lines · Oliver Hunt
Quoted 8 lines
Depends on how the forks eventually get implemented.
With the file-as-directory concept, all you need is to look at the file's directory part to see what is there. (The forks, implemented as files in a subdirectory.) It is done the same way as for an ordinary directory, so nothing "new".
Quoted 8 lines
This still doesn't solve the problem of how we distinguish between a multi-fork file
and a directory, for those old programs(ie. almost all that currently exist), to be able to access meaningful data we would need to either return just the primary fork, a serialized version of all forks in the file, or make the file look more or less identical to a directory.
If they open "filename" then they get what you calls the primary fork. If they open "filename/someforkname" then they get some other fork, using exactly the same mechanism as when they opens "directoryname/somefilename" And if you want to use the old tools to look at some forks, just do a
$ cd filename It will work if "filename" happens to be a file-as-dir thing, then do $ ls And you'll see the forks by name. Because the "forks" are merely plain files in a subdirectory.
The only new thing needed is the ability for something to be both file and directory at the same time. Some tools will need
a update - usually only because they blindly assume that a directory isn't a file too, or that a file can't be a directory too. Remove the mistaken assumption and things will work because the underlying system calls (chdir or open) _will_ work.
Quoted 6 lines
The first option could cause problems when transfering files between different filesystems,
the second results in programs getting metadata they can't handle, and the third option results in few of the advantages over, well, directories...
Few advantages over directories perhaps, but other implementatins
don't have more advantages over directories than this one. Requiring another syscall to open forks other than the primary breaks _all_ existing software because it obviously don't use that
syscall yet. And then it doesn't provide any advantages over the
file-as-plain-directory way. . .
Quoted 6 lines
And even those applications that could handle the fork information nicely would need to fs type to find out whether they were handling a directory or a multi-forked file...
The distinction might not be important. A directory can be seen as a multi-stream file without the main stream. And there's always "stat" which should be able to tell what you're looking at.
Quoted 5 lines
As I say I like the idea, but I can't see anyway of implementing it in a way that is useful without first putting considerable effort into at least the VFS if not all the actual fs drivers.
Sure, implementing it well will take effort, wich is why it is important to find a good way of doing it so the effort won't be wasted.
Helge Hafting
Re: The argument for fs assistance in handling archives
Helge Hafting <[email protected]> said:
[...]
Quoted 2 lines
The only new thing needed is the ability for something to be both file and directory at the same time.
Then why have files and directories in the first place?
Quoted 5 lines
Some tools will need
a update - usually only because they blindly assume that a directory isn't a file too, or that a file can't be a directory too. Remove the mistaken assumption and things will work because the underlying system calls (chdir or open) _will_ work.
But with some weird restrictions: No moving stuff around between files, no linking, some "files" can't be deleted (how would you handle removing the principal stream of a file?). Some stuff you'd love to do (is, in fact, the reason for this all) just can't be allowed (i.e., J. Random Luser setting his own icon for system-wide emacs). So the tools/scripts/users/sysadmins will have to be painfully aware that some of the files aren't, and some of the directories aren't either. Major pain in the neck to use, if you look closer. Add extra kernel complexity. For little (if any) gain.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Re: The argument for fs assistance in handling archives
Quoted 11 lines
Helge Hafting <[email protected]> said:
[...]
Quoted 2 lines
The only new thing needed is the ability for something to be both file and directory at the same time.
Then why have files and directories in the first place?
Good point, we don't need them :) Directories are just a visible grouping of files to make it easier for the user to manage. But some things aren't really that intuitive with todays layout - especially for non-unix users.
Just an example where the user needs to edit config file for some program. Where should he look?
/etc/app.conf ?
/etc/app/app.conf ?
/etc/conf.d/app.conf ?
/var/lib/app/app.conf ?
and so on....
Using file-as-dir isn't really that much of a change. It isn't those that will confuse people anyway.
Quoted 9 lines
Quoted 5 lines
Some tools will need
a update - usually only because they blindly assume that a directory isn't a file too, or that a file can't be a directory too. Remove the mistaken assumption and things will work because the underlying system calls (chdir or open) _will_ work.But with some weird restrictions: No moving stuff around between files, no linking, some "files" can't be deleted (how would you handle removing the principal stream of a file?).
Well. there are read-only files. And if you remove the main stream (which is the file, really) then it will all be gone =)
Quoted 3 lines
Some stuff you'd love to do (is, in fact, the
reason for this all) just can't be allowed (i.e., J. Random Luser setting his own icon for system-wide emacs).
Users do not normally have write permissions to system-wide applications. Why would it be any different now?
Quoted 4 lines
So the
tools/scripts/users/sysadmins
will have to be painfully aware that some of the files aren't, and some of the directories aren't either. Major pain in the neck to use, if you look closer. Add extra kernel complexity. For little (if any) gain.
Not sure what you mean "aren't". Things shouldn't be that much different to administer. Normal permissions should still apply. Sure extra complexity comes _if_ you want to use extended features that are using meta-info. But there is where we need some patches to tar and other backup tools.
One other way would be to enter a specific mode (chroot, a bash flag --show-metas, etc) that would allow all streams and metas to be seen as files in directories. Then tar and other tools would back it all up. Restoring will be a little trickier as we don't know if stuff was files or folders before. But I am sure that would be solvable. Perhaps a tool to convert them back to normal files with meta-data and streams.
~S
Re: The argument for fs assistance in handling archives
Quoted 25 lines · Spam
Quoted 11 lines
Helge Hafting <[email protected]> said:
[...]
Quoted 2 lines
The only new thing needed is the ability for something to be both file and directory at the same time.
Then why have files and directories in the first place?
Good point, we don't need them :) Directories are just a visible grouping of files to make it easier for the user to manage. But some things aren't really that intuitive with todays layout - especially for non-unix users.
Just an example where the user needs to edit config file for some program. Where should he look?
/etc/app.conf?
/etc/app/app.conf?
/etc/conf.d/app.conf?
/var/lib/app/app.conf?
and so on....
Quoted 3 lines
Using file-as-dir isn't really that much of a change. It isn't those that will confuse people anyway.
If the non-computer literate user is expected to edit the config file, a pretty interface that hides the internals from the user should be provided. For the sake of power users, users that believe themselves to be power users, and configuration changes on multiple systems.
1) Track the location of the config file with the native package management system.
2) Keep a human readable/editable format.
3) Provide APIs or schema so that the files may be parsed for correctness, or edited or created programatically.
4) Provide a tool that not only checks for a parseable file, but does sanity checks such as (Does file exist).
I'm at a loss as to how any of this has anything to do with filesystems. The closest I see is one implementation of the above (gconf2) happens to use the filesystem for it's default hierarchical database backend.
As for developers that are too stupid/lazy to provide the GUI and meet the other criteria... I suggest extensive use of cattle prods on them as they attend an automata class.
Signature · 4 lines
Chris Dukes Warning: Do not use the reflow toaster oven to prepare foods after it has been used for solder paste reflow. http://www.stencilsunlimited.com/stencil_article_page5.htm
Re: The argument for fs assistance in handling archives
Quoted 45 lines · Spam
Quoted 25 lines
Quoted 11 lines
Helge Hafting <[email protected]> said:
[...]
Quoted 2 lines
The only new thing needed is the ability for something to be both file and directory at the same time.
Then why have files and directories in the first place?
Good point, we don't need them :) Directories are just a visible grouping of files to make it easier for the user to manage. But some things aren't really that intuitive with todays layout - especially for non-unix users.
Just an example where the user needs to edit config file for some program. Where should he look?
/etc/app.conf?
/etc/app/app.conf?
/etc/conf.d/app.conf?
/var/lib/app/app.conf?
and so on....Quoted 3 lines
Using file-as-dir isn't really that much of a change. It isn't those that will confuse people anyway.
If the non-computer literate user is expected to edit the config file, a pretty interface that hides the internals from the user should be provided. For the sake of power users, users that believe themselves to be power users, and configuration changes on multiple systems.
1) Track the location of the config file with the native package management system.
2) Keep a human readable/editable format.
3) Provide APIs or schema so that the files may be parsed for correctness, or edited or created programatically.
4) Provide a tool that not only checks for a parseable file, but does sanity checks such as (Does file exist).
Quoted 3 lines
I'm at a loss as to how any of this has anything to do with filesystems. The closest I see is one implementation of the above (gconf2) happens to use the filesystem for it's default hierarchical database backend.
The relevance was as to the presentation of meta-data, streams and such and their implication on users.
I just used the example of config files because there are not so much of a standard among them. Administrators still need to learn and remember where and what to backup. This wouldn't be so much of a change if they decide to introduce some usage for meta-data or named streams.
Remember, even if the functionality exist you do not have to use everything of it.
Quoted 3 lines
As for developers that are too stupid/lazy to provide the GUI and meet the other criteria... I suggest extensive use of cattle prods on them as they attend an automata class.
Lets all start with the bash(ing) ;).
But really. What you say is true. The users do not care so much about what is going on below the curtains. Here is where the clash between different groups of Linux users and developers appear most vivid.
Sure KDE and Gnome are on their way, but their efforts still have a long way to go. They still don't talk to eachother to strive for a comon goal - but competes instead (which may even be a good thing).
~S
Re: The argument for fs assistance in handling archives
In reply to SpamQuoted 11 lines · Spam
Quoted 8 lines · Horst von Brand
Helge Hafting <[email protected]> said:
[...]
Quoted 2 lines
The only new thing needed is the ability for something to be both file and directory at the same time.
Then why have files and directories in the first place?
Good point, we don't need them :)
Great. Then everything is a firectory (or dile?). And a firectory points at other firectories and contains data. I just don't see how you are supposed to distinguish the data from further firectories...
Quoted 4 lines
Directories are just a visible
grouping of files to make it easier for the user to manage. But some things aren't really that intuitive with todays layout - especially for non-unix users.
No operating system is "intuitive". A person is able to glance over some 7 items, a filesystem handles tens of thousands of objects. Nothing "intuitive" possible in the layout, everybody has to _learn_ how to handle this.
Quoted 2 lines
Just an example where the user needs to edit config file for some program. Where should he look?
Another example: User looks for the sysadmin. Where does she look? Answers to both questions depend on extra data that can't possibly be deduced just from the statement.
[...]
Quoted 2 lines
Using file-as-dir isn't really that much of a change. It isn't those that will confuse people anyway.
See above.
Quoted 6 lines
Quoted 5 lines
Quoted 5 lines
Some tools will need
a update - usually only because they blindly assume that a directory isn't a file too, or that a file can't be a directory too. Remove the mistaken assumption and things will work because the underlying system calls (chdir or open) _will_ work.
Quoted 6 lines
Quoted 3 lines
But with some weird restrictions: No moving stuff around between files, no linking, some "files" can't be deleted (how would you handle removing the principal stream of a file?).
Well. there are read-only files. And if you remove the main stream (which is the file, really) then it will all be gone =)
While some other process is messing with another stream. Right, all goes away when the last reference goes away. But it stays iff some other process recreates the data stream. Now what if another process goes ahead and creates a new file with the same name in the meanwhile... but this is very diferent from "real" directories, if you delete some particular member it doesn't go away. Explain that to the poor user that can't be trusted to keep the distinction between directories and files straight...
Quoted 6 lines
Quoted 3 lines
Some stuff you'd love to do (is, in fact, the
reason for this all) just can't be allowed (i.e., J. Random Luser setting his own icon for system-wide emacs).Users do not normally have write permissions to system-wide applications.
Exactly.
Quoted 1 line
Why would it be any different now?
Because the user wants to be able to associate an icon/description/comment/... with system files too. This is supposed to be one of the key reasons for the feature...
Quoted 7 lines
Quoted 4 lines
So the
tools/scripts/users/sysadmins
will have to be painfully aware that some of the files aren't, and some of the directories aren't either. Major pain in the neck to use, if you look closer. Add extra kernel complexity. For little (if any) gain.Not sure what you mean "aren't". Things shouldn't be that much different to administer.
They aren't. Some directories are files, and different rules apply to them. Some files have structure, and you have to exercise extra care not to loose that. Very different indeed in my book.
Quoted 1 line
Normal permissions should still apply.
Even streams inside a file have permissions? What does it mean if a file is "searchable" (new permission needed to handle "x" on directories vs files!) but not writeable, and the default stream is writeable? Or do they just have the permissions of the container? In that case, the file must be readable so the user can have a nice icon for it. Consider /etc/shadow... And if somebody tries to change permissions on one of the streams?
Quoted 4 lines
Sure
extra complexity comes _if_ you want to use extended features that are using meta-info. But there is where we need some patches to tar and other backup tools.
And new options (where people already get confused by current options).
Quoted 6 lines
One other way would be to enter a specific mode (chroot, a bash flag --show-metas, etc) that would allow all streams and metas to be seen as files in directories. Then tar and other tools would back it all up. Restoring will be a little trickier as we don't know if stuff was files or folders before. But I am sure that would be solvable.
"I'm sure it can be solved somehow" isn't enough.
Quoted 2 lines
Perhaps a tool to convert them back to normal files with
meta-data and streams.
Yikes!
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Re: The argument for fs assistance in handling archives
Horst von Brand wrote on Fri, 03 Sep 2004 22:27:07 -0400:
Quoted 3 lines
Great. Then everything is a firectory (or dile?). And a firectory points at other firectories and contains data. I just don't see how you are supposed to distinguish the data from further firectories...
I like to call them fildirutes (file/directory/attribute). A file typing system would tell you the intended purpose of a particular fildurute. So if a fildurute called X has X/..metas/mimefiletype containing "application/x-directory" then you know that it should be treated as being primarily a container for other fildirutes and shown to the user as a folder in a GUI view. If it said it was "application/x-text-document" then the GUI system would default to opening it in a word processor. Either way, that's only a hint about how it should be presented to the user, not something the kernel enforces.
For efficiency, the file type might just be stored in the fildirute's inode as a code number (since most things would have a file type) that the file system exposes as a file called ..metas/mimefiletype.
- Alex
Re: The argument for fs assistance in handling archives
In reply to Horst von BrandQuoted 22 lines · Horst von Brand
Quoted 16 lines · Spam
Quoted 13 lines · Horst von Brand
Helge Hafting <[email protected]> said:
[...]
Quoted 4 lines
The only new thing needed is the ability for something to be both file and directory at the same time.
Then why have files and directories in the first place?
Quoted 9 lines
Quoted 3 lines
Good point, we don't need them :)
Great. Then everything is a firectory (or dile?). And a firectory points at other firectories and contains data. I just don't see how you are supposed to distinguish the data from further firectories...
By name.fopen("somefile", "r"); //opens somefile data fopen("somefile/substream", "r"); //opens a substream of somefile.
Helge Hafting
Re: The argument for fs assistance in handling archives
In reply to Horst von BrandQuoted 23 lines · Horst von Brand
Helge Hafting <[email protected]> said:
[...]
Quoted 4 lines
The only new thing needed is the ability for something to be both file and directory at the same time.
Then why have files and directories in the first place?
Quoted 7 lines
Some tools will need
a update - usually only because they blindly assume that a directory isn't a file too, or that a file can't be a directory too. Remove the mistaken assumption and things will work because the underlying system calls (chdir or open) _will_ work.
Quoted 3 lines
But with some weird restrictions: No moving stuff around between files, no
I see no need for such a weird restriction. If it is "weird", why have it? If someone want to move stuff from one subdirectory to another, let them. I see no need to stop that just because a file-as-directory was involved.
Quoted 3 lines
linking, some "files" can't be deleted (how would you handle removing the
linking is a problem of course. We can't have hardlinks to directories, so no hardlinks to file-as-dir either unless the general problems with directory links are solved.
Quoted 2 lines
principal stream of a file?).
A design decision. Removing the principal stream may delete the entire tree. Or it may delete the principal stream only, turning that file-as-dir back into a plain directory.
Quoted 8 lines
Some stuff you'd love to do (is, in fact, the
reason for this all) just can't be allowed (i.e., J. Random Luser setting his own icon for system-wide emacs). So thetools/scripts/users/sysadminswill have to be painfully aware that some of the files aren't, and some of the directories aren't either. Major pain in the neck to use, if you look closer. Add extra kernel complexity. For little (if any) gain.
The gain is indeed the interesting question here. It isn't something I desperately need. But if file-as-directory _happens_, then I hope it'll happen in a good way. No unnecessary complications or restrictions, and allowing existing stuff to work as well as possible. There will surely be some tool problems - but we certainly don't need to break every existing tool the way a new form of "open" will. . .
Helge Hafting
Re: The argument for fs assistance in handling archives
In reply to Helge HaftingSalut,
Quoted 4 lines · Helge Hafting
Requiring another syscall to open forks other than the primary breaks _all_ existing software because it obviously don't use that syscall yet. And then it doesn't provide any advantages over the file-as-plain-directory way. . .
Actually...
We might tune the sys_open() call to take an additional argument (the stream ID), and introduce a compatibility interface into *libc which chooses strid=0 by default if the plain old open call is being used.
Maybe this can be handled transparently by cpp..
Tonnerre
Re: The argument for fs assistance in handling archives
Quoted 11 lines
Salut,
Quoted 4 lines · Helge Hafting
Requiring another syscall to open forks other than the primary breaks _all_ existing software because it obviously don't use that syscall yet. And then it doesn't provide any advantages over the file-as-plain-directory way. . .
Quoted 7 lines
Actually...
We might tune the
sys_open()call to take an additional argument (the stream ID), and introduce a compatibility interface into *libc which chooses strid=0 by default if the plain old open call is being used.Maybe this can be handled transparently by cpp..
How are things like copy handled today? Is it just some code linked in during compilation (ie, the application handles the copy itself) or is it a library function in libc? If it is in libc then it ought to be enough if libc is patched to support the new semantics with file-as-dir etc.
~S
Quoted 1 line
Tonnerre
Re: The argument for fs assistance in handling archives
In reply to TonnerreQuoted 12 lines · Tonnerre
Salut,
Quoted 6 lines · Helge Hafting
Requiring another syscall to open forks other than the primary breaks _all_ existing software because it obviously don't use that syscall yet. And then it doesn't provide any advantages over the file-as-plain-directory way. . .
Quoted 7 lines
Actually...
We might tune the
sys_open()call to take an additional argument (the stream ID), and introduce a compatibility interface into *libc which chooses strid=0 by default if the plain old open call is being used.
But this isn't necessary, as I pointed out above.
There is no _need_ for a new kind of open(), so
why make one in libc?
If you can open a fork/substream/whatever by issuing open("filename/forkname", ...
then the old-fashioned open() works with multi-fork files too. An tools based on "open() something, then work with the resulting file descriptor" will work _unchanged_ with such a multi-fork fs.
Tools that rely on "stat" or directory traversals might need some updating to work perfectly with such a fs, but not something that just opens by name.
Helge Hafting
Re: The argument for fs assistance in handling archives
Salut,
Quoted 6 lines · Helge Hafting
If you can open a fork/substream/whatever by issuing open("filename/forkname", ...
then the old-fashionedopen()works with multi-fork files too. An tools based on "open()something, then work with the resulting file descriptor" will work _unchanged_ with such a multi-fork fs.
In my version they'd run unchanged as well.
And BTW, I wasn't talking about introducing a new open at libc level. I was talking about modifying the open system call in the kernel and having libc provide compatibility for the old call.
Since I'm not sure how much breakage it takes to make a file a directory.
Tonnerre
Re: The argument for fs assistance in handling archives
Quoted 14 lines · Tonnerre
Salut,
Quoted 8 lines · Helge Hafting
If you can open a fork/substream/whatever by issuing open("filename/forkname", ...
then the old-fashionedopen()works with multi-fork files too. An tools based on "open()something, then work with the resulting file descriptor" will work _unchanged_ with such a multi-fork fs.
Quoted 9 lines
In my version they'd run unchanged as well.
And BTW, I wasn't talking about introducing a new open at libc level. I was talking about modifying the open system call in the kernel and having libc provide compatibility for the old call.
Since I'm not sure how much breakage it takes to make a file a directory.
I believe most of the changes necessary for supporting file-as-dir will be in the internal workings of filesystems, not so much in the interfaces. I don't think it is necessary to change the open system call, because opening a file in a subdirectory or a named substream is the same thing.
The _uses_ people have suggested for file-as-dir (complex file attributes and different views of the data) tends to be different from what a directory usually does, but the implementation doesn't have to be. There's no need to add extra interfaces or restrictions. File-as-dir that really is a directory, so that it may contain a whole tree is more flexible. And the underlying fs can use existing directory code instead of something extra.
So the only change needed for the open system call is that it will suddenly succeed in the open("filename/forkname") case that
used to complain about an invalid path because "filename" isn't a directory. "stat" may have to change though, so it can identify a file-as-dir.
Helge Hafting
Re: The argument for fs assistance in handling archives
In reply to Oliver HuntSalut,
Quoted 1 line · Oliver Hunt
getNumForks(fileref){ return 1;}
Actually it might be even cooler to have a generic function which you can apply to any file on a file system (or whatever) and which returns you the file system's capabilities (can write, has xattrs, has acl, has multiple streams, *not* has btree lookups etc, as that is *not* interesting to userland anyway.)
Tonnerre
Re: The argument for fs assistance in handling archives
In reply to Hans ReiserQuoted 26 lines
Quoted 5 lines · Linus Torvalds
But _my_ point is, no user program is going to take _advantage_ of
anything that only one filesystem on one system offers.
Apple does not have this problem....
and yes, the apps will take advantage of it, which is different from depending on it. If you use the wrong fs you will lose some of the features of the app.
For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice (excepting Clearcase, which priced itself
into a niche market). It is 25 years past time for someone to change
things. That someone will have first mover advantage, and the more little semantic features possessed the more lure there will be to use it which will increase market share which will lure more apps into depending on it and in a few years the other filesystems will (deservedly) have only a small market share because the apps won't all work on them.
Quoted 6 lines
Besides, there are enhancements which are simply compelling. You can write a dramatically better performance version control system with a
much simpler design if the FS is atomic. Our transaction manager
first draft was written by a version control guy, and he would probably be happy to tell you how lack of atomicity other than rename makes version control software design hideous.
Btw, version control for ordinary files would be a great feature. I think something like it is available through Windows 2000/3 server. Isn't it called "Shadow Copies". It works over network shares. :)
It allows you to restore previous versions of the file even if you delete or overwrite it.
Features like this do make a good point and helps protecting data - something that is important IMHO.
Quoted 11 lines
We have the performance lead. By next year we will be stable enough for mission critical servers, and then we start the serious semantic enhancements.
If you don't embrace progress, then you doom Linux to following behind, because the guys at Apple are pretty aggressive now that Jobs is back, and they WILL change the semantics, and they will do so in compelling ways, and Linux will be reduced to aping them when it should be leading them.
Hans
Re: The argument for fs assistance in handling archives
Quoted 3 lines · Spam
Btw, version control for ordinary files would be a great feature. I think something like it is available through Windows 2000/3 server. Isn't it called "Shadow Copies". It works over network shares. :)
Netapps have done this for years with NFS, Subversion does it with DAV, clearcase does it with NFS.
Re: The argument for fs assistance in handling archives
In reply to SpamQuoted 12 lines · Spam
Quoted 7 lines
Linus Torvalds wrote:
Besides, there are enhancements which are simply compelling. You can write a dramatically better performance version control system with a
much simpler design if the FS is atomic. Our transaction manager
first draft was written by a version control guy, and he would probably be happy to tell you how lack of atomicity other than rename makes version control software design hideous.Btw, version control for ordinary files would be a great feature. I think something like it is available through Windows 2000/3 server. Isn't it called "Shadow Copies". It works over network shares. :)
Quoted 5 lines
It allows you to restore previous versions of the file even if you delete or overwrite it.
Features like this do make a good point and helps protecting data - something that is important IMHO.
I believe you mean something simillar to:
file1.txt;1
file1.txt;2
file1.txt;3 (yeap, it's VMS)
where you'll have to cleanup old versions when you don't need them any more... AFAIK that this is older than HDDs
<<V13>>
Re: The argument for fs assistance in handling archives
V13> I believe you mean something simillar to:
V13> file1.txt;1
V13> file1.txt;2
V13> file1.txt;3 (yeap, it's VMS)
Or TOPS-20, a precursor to VMS in some ways. It was a nice feature.
V13> where you'll have to cleanup old versions when you don't need V13> them any more... AFAIK that this is older than HDDs
It was usually an automatic cleanup past a certain point or if you went over disk quota.
In any case, while I do like this feature, I'm not sure how we would cleanly implement this inside the unix namespace, or if inside a new namespace, how that new namespace would be joined with a standard Unix one.
John
Re: The argument for fs assistance in handling archives
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"John" == John Stoffel <[email protected]> writes:
V13> I believe you mean something simillar to:
V13> file1.txt;1
V13> file1.txt;2
V13> file1.txt;3 (yeap, it's VMS)
[...]
John> In any case, while I do like this feature, I'm not sure how we John> would cleanly implement this inside the unix namespace, or if John> inside a new namespace, how that new namespace would be joined John> with a standard Unix one.
file1.txt/..version/1
or, for those version control systems like svn and arch where the version number refers to the version of the whole project tree:
project/..version/1/file1.txt
You could even export strange things like:
project/..by-date/2004-08-24/file1.txt
to see what the file looked like on August 24. Or
project/..diff/3-1/file1.txt
to see a diff between version 3 and 1.
Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Re: The argument for fs assistance in handling archives
In reply to SpamAm Donnerstag, 2. September 2004 11:52 schrieb Spam:
Quoted 6 lines
Btw, version control for ordinary files would be a great feature. I think something like it is available through Windows 2000/3 server. Isn't it called "Shadow Copies". It works over network shares. :)
It allows you to restore previous versions of the file even if you delete or overwrite it.
There's no need to do that in kernel, unless you want to be able to force it unto users.
Regards
Oliver
Re: The argument for fs assistance in handling archives
Quoted 8 lines
Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
Quoted 4 lines
Btw, version control for ordinary files would be a great feature. I think something like it is available through Windows 2000/3 server. Isn't it called "Shadow Copies". It works over network shares. :)
Quoted 5 lines
Quoted 2 lines
It allows you to restore previous versions of the file even if you delete or overwrite it.
There's no need to do that in kernel, unless you want to be able to force it unto users.
Exactly ;)
Difference with having it in just certain applications like Gnome-VFS etc is that it would work with all applications, over SMB shares etc. This is a great advantage and benefit for many desktop users but also in a corporate environment. Even though there are backups of users documents it is a pain to restore them every time they loose the original file. Besides, the backup is never as fresh either.
Quoted 2 lines
Regards
Oliver
Re: The argument for fs assistance in handling archives
In reply to Oliver Neukum-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Oliver Neukum wrote:
| Am Donnerstag, 2. September 2004 11:52 schrieb Spam: | |> Btw, version control for ordinary files would be a great feature. I |> think something like it is available through Windows 2000/3 server. |> Isn't it called "Shadow Copies". It works over network shares. :) |> |> It allows you to restore previous versions of the file even if you |> delete or overwrite it. | | | There's no need to do that in kernel, unless you want to be able | to force it unto users.
And on apps. Should I teach OpenOffice.org to do version control? Seems a lot easier to just do it in the kernel, and teach everything to do version control in one fell swoop.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTe+NXgHNmZLgCUhAQLW+RAAnoQIQLXSd2vQudOKAaLgXXCxoa+gEWIq zcNd+bIWPuQxwcZYlqbs/pDFiDky3XqLRaZv8PZx++rs9mMnv36eAtMX7uUp3egz MxymeDDI796wtsDDBGcZtO+WwmqJ0fPgQMN/Q237s2yccPGJoDIVle9mzZAfNIdO RDdLq142D+0t1iqfiVqaA+NgkArrGhcr0Hs/3TdviJX30gPL/Jm9eOvc3H3NgIQu XToDQB1UnLoyiIwc16IY4ZxYoxf1YnFtrivZb+YgC61mIQLeqJMjixCAtBVzAfKN 2GmlsnfWcMCgXSEX8imkoNSLcmkv886+esGXZVzBbY1/Qg/f5MCXZilbb8e6+4I9 U4ReEUg5fGJW8JyKAXqKGyvbC/lim55Vgcjp/J8mrhUa3Q3cwPKoi2GKNRcbdpT/ pPB2SADA1eNrmFyGB7kLcqt0F9i4fIyTrItpHq9+m72OdFEze2OnWn+eIY8dRut7 SaGTq6ZEYtkH8oWT9xSXp1d2TUxeSVTdx8EffIIdwXAVOO0Z9xd71MJPLMp3VOMH Qld4gjYjKFzwj3QlcWcsiVhk6Zq2fmqoOCQNfxhBzH1Le3v0F0ugammkQUXtRAub dKEWxGfkDGOwviJXKeckDNn98x88vh7C/7gxpAOIcVSRDhEV+MDWIfMMvotE/lxe 393hENsXOgQ=
=g+3J
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 9 lines
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Oliver Neukum wrote:
| Am Donnerstag, 2. September 2004 11:52 schrieb Spam:
|
|>> Btw, version control for ordinary files would be a great feature. I |>> think something like it is available through Windows 2000/3 server. |>> Isn't it called "Shadow Copies". It works over network shares. :) |>> |>> It allows you to restore previous versions of the file even if you |>> delete or overwrite it.
Quoted 8 lines
| | | There's no need to do that in kernel, unless you want to be able | to force it unto users.And on apps. Should I teach OpenOffice.org to do version control? Seems a lot easier to just do it in the kernel, and teach everything to do version control in one fell swoop.
Do you mean in the kernel or as a filesystem/VFS plugin that would extend the functionality to include version control?
~S
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Spam wrote:
[...]
| Do you mean in the kernel or as a filesystem/VFS plugin that would | extend the functionality to include version control?
The filesystem plugin. But this needs some kernel support, which scares people. Read the archives if you don't believe me.
Besides, we'd want per-block copy-on-write support anyway to make it sane. That has to go in the filesystem.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTfKhXgHNmZLgCUhAQIR5w//Z6hg58uVvIJuwNCb764lFj22iHwLvg0T JvjQeOSCz7p2/uv+dHN2MRNDeezmvGfH49Y7RE78Q9e1xrfZFGZKyybHVisfea1t f9Bk8M6ZtykKnUdPhq5MsDH9Oo98dXMNEWhDw5WUYMzvglitMUcd4np3gGEtr539 yKHuxZhqjc/lQ9rd6PFKJYab1m9RAabQnGMHRIvx0T98iW3EI9pyOd0u1oUVCiRi RxllyP0KCg7wxFv93b4zYHsv5BO8E7MWVI4aKuBUCqrc8ObJnDLDbO/8FfMvDt/1 Do9nlikFh4IaRvwZtmQA0nVv80uxtjnSOCs7F2r+qqNyprcHIdj2m+3x5zE8Tbfz YCkI5BopHub94yBvhlcFGiKBtigLDsH3zHc66dVDl4SreJPKo+yirjjZ+RrU3HN1 iK+aFp5GxapcYDkMNOS0ab9TPejqLVyJ+Ci5mSNckcMARC9v+2mT7JTCBupyXZar LSLFYef+W7ipKq5ZMurDB17I5y7VXfwOTd+oSwQdMwlsQlJs/AFOOGdJywtLGLy1 6jhejQlziPLWB9hz9wFAjWyHPrS0XsCNbtGbBESIVmqvLsZmpcAULpMn7TGc8aJg 0r79ODKlGK8lYJRIquRhZzd51oqiBfFJcX969XGPMPUBX4x3GZjaOlj79OQ7Ih3s Qd8PSfUVzGU=
=Cw9B
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
In reply to David MasoverQuoted 3 lines · David Masover
And on apps. Should I teach OpenOffice.org to do version control? Seems a lot easier to just do it in the kernel, and teach everything to do version control in one fell swoop.
Including files you didn't really want to keep version control of?
How many temp files does gcc create and unlink in the course of a kernel build? (And remember, you can't say "don't enable that on /tmp" - gcc respects the setting of $TMPDIR - so an 'export TMPDIR=~/tmp' confuses things quite nicely...)
And it's hard for the kernel to know that an unlink() done by gcc should be treated differently than the "recover the last version" you *want* it do be able to do after you work on a source file for a long while, save it, and then fumble-finger a 'rm * .o' - you can't even use a heuristic like "don't version control it unless it's N seconds or more old"
(Note that the "obvious" solution of creating a chattr flag has its own complexity issues - should versioning be turned on by default for some types and not others, etc...)
There be dragons here - it's not as simple as "drop in a plugin and be happy".
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
[email protected] wrote:
| On Thu, 02 Sep 2004 19:43:34 CDT, David Masover said: | | |>And on apps. Should I teach OpenOffice.org to do version control? |>Seems a lot easier to just do it in the kernel, and teach everything to |>do version control in one fell swoop. | | | Including files you didn't really want to keep version control of?
No one said you have to apply it to everything. Only that you can. Right now vim cannot do version control, AFAIK. But you could do something like:
echo 1 > foo/version_control
vim foo
Default would be off, of course.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTfLR3gHNmZLgCUhAQKybRAAkF4sqLUCJEfpDNvKEyv8MJGcy0w+qV12 3Rf50OQeeurZMZrnli0vNIuSIWwIsGI+j7qbj1mlfi+Ps3dAjOlKXav3tGoGshLl HH/Wl9XvGPOAWlIQOqqBGAyno8sDIvHAjh5lH3C7m+9/2Ao7s1G/bQ+hm/deYPJK WPZxExd/1BdchrbjNjbfWHKrD6LdF/GBa/Vbj1F8m95WEvxsfbDnSTwb5HhvcUxD TtqwkINmx7Tle2p9q7PcgR7y76dmcyoWw82ST3BfF/AbGauMQAr0h6jNjvzclfgJ yrIOwGXdR3pn8ZK8dHVct79N0f+PYFdeAzLYxMZR2vefB2enQolbbs6Lf7X60g2g 5hrH/ezenFJIZMPhQtlJqBaFuA5GHw9B2An6VbOmDm7sSOFyHSlKoCUFlJ/I5nZd Bcn46lhNxvTJhu3tvnvDqTB8f0/yVYt5QfoBFA1mzHKig2iNN6vr2xLGdWSLU6XD vRjy+KyVC+PvjL13E0JrjXy9UuqY7CH2xAunSgLJ2cvfRmNRoVoV1hkp3McQcUqF GiD3s0REcztlvsrP+DxKLRtPpnOJi5By5NmZpJkiPdVB9zpIsL/Ia7KL/A7IGGh3 IF3t/xSUmjgrek01SeadHqsIngeWQ+F8fiars7NtzNhrvRgPVfDVheey3vhQjFxM 3kDKH+I9rGk=
=3+nq
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
In reply to David MasoverQuoted 17 lines · David Masover
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Oliver Neukum wrote:
| Am Donnerstag, 2. September 2004 11:52 schrieb Spam: | |> Btw, version control for ordinary files would be a great feature. I |> think something like it is available through Windows 2000/3 server. |> Isn't it called "Shadow Copies". It works over network shares. :) |> |> It allows you to restore previous versions of the file even if you |> delete or overwrite it. | | | There's no need to do that in kernel, unless you want to be able | to force it unto users.
Quoted 3 lines
And on apps. Should I teach OpenOffice.org to do version control? Seems a lot easier to just do it in the kernel, and teach everything to do version control in one fell swoop.
Just add a post-save trigger that can check it into any SCM you want.
Re: The argument for fs assistance in handling archives
Am Freitag, 3. September 2004 10:13 schrieb Christoph Hellwig:
Quoted 10 lines · David Masover
Quoted 7 lines
| There's no need to do that in kernel, unless you want to be able | to force it unto users.
And on apps. Should I teach OpenOffice.org to do version control? Seems a lot easier to just do it in the kernel, and teach everything to do version control in one fell swoop.
Just add a post-save trigger that can check it into any SCM you want.
That depends on whom you refer to. If you want to impose the RCS on the users as an administrative measure and force compliance, then you'll need to do it in kernel.
I see some nasty issues with disk quotas there.
Regards
Oliver
Re: The argument for fs assistance in handling archives
In reply to SpamSpam <[email protected]> said:
[...]
Quoted 3 lines
Btw, version control for ordinary files would be a great feature. I think something like it is available through Windows 2000/3 server. Isn't it called "Shadow Copies". It works over network shares. :)
Right. And when space runs low, the old copies get the axe. And then you need the old version... and are royally screwed because you trusted the system to save it for you while willy-nilly fooling around.
I never liked the MS-DOS "undelete" feature, because I saw too many people burned. It is _less_ data security than "deleted file is gone for good", in the end sum.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Re: The argument for fs assistance in handling archives
In reply to SpamQuoted 4 lines · Spam
Btw, version control for ordinary files would be a great feature. I think something like it is available through Windows 2000/3 server. Isn't it called "Shadow Copies". It works over network shares. :)
Quoted 8 lines
It allows you to restore previous versions of the file even if you delete or overwrite it.
Features like this do make a good point and helps protecting data - something that is important IMHO.
Netapp's WAFL has had this for ages, they are called snapshots, and they are one of the most useful file system features ever. But Microsoft certainly did not invent them.
Lee
Re: The argument for fs assistance in handling archives
In reply to SpamQuoted 9 lines · Spam
Btw, version control for ordinary files would be a great feature. I think something like it is available through Windows 2000/3 server. Isn't it called "Shadow Copies". It works over network shares. :)
It allows you to restore previous versions of the file even if you delete or overwrite it.
Feel free to use cvs on your home directory. :-)
Helge Hafting
Re: The argument for fs assistance in handling archives
In reply to Hans ReiserI think Hans' idea about further evolution is right.
(2.7 which will lead into 3.0 maybe?)
Regards,
Bostjan
Quoted 39 lines · Hans Reiser
Quoted 3 lines · Linus Torvalds
But _my_ point is, no user program is going to take _advantage_ of
anything that only one filesystem on one system offers.
Apple does not have this problem....
and yes, the apps will take advantage of it, which is different from depending on it. If you use the wrong fs you will lose some of the features of the app.
For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice (excepting Clearcase, which priced itself
into a niche market). It is 25 years past time for someone to change
things. That someone will have first mover advantage, and the more little semantic features possessed the more lure there will be to use it which will increase market share which will lure more apps into depending on it and in a few years the other filesystems will (deservedly) have only a small market share because the apps won't all work on them.Besides, there are enhancements which are simply compelling. You can write a dramatically better performance version control system with a
much simpler design if the FS is atomic. Our transaction manager
first draft was written by a version control guy, and he would probably be happy to tell you how lack of atomicity other than rename makes version control software design hideous.We have the performance lead. By next year we will be stable enough for mission critical servers, and then we start the serious semantic enhancements.
If you don't embrace progress, then you doom Linux to following behind, because the guys at Apple are pretty aggressive now that Jobs is back, and they WILL change the semantics, and they will do so in compelling ways, and Linux will be reduced to aping them when it should be leading them.
Hans
Re: The argument for fs assistance in handling archives
In reply to Hans ReiserHans> For 30 years nothing much has happened in Unix filesystem Hans> semantics because of sheer cowardice (excepting Clearcase, which Hans> priced itself into a niche market).
Keeping a consistent namespace is cowardice? And ClearCase is an example of an application which sets up it's own namespace and grafts it onto the standard Unix one. Most standard Unix tools work just fine inside clearcase Views, but to manage the metadata (xattrs, or whatever you want to call them), you need to use the 'cleartool' command. Hmmm... sounds like 'runat' to me.
Hans> It is 25 years past time for someone to change things. That Hans> someone will have first mover advantage, and the more little Hans> semantic features possessed the more lure there will be to use Hans> it which will increase market share which will lure more apps Hans> into depending on it and in a few years the other filesystems Hans> will (deservedly) have only a small market share because the Hans> apps won't all work on them.
This is all pure marketing speak and economic theory. Show us the *technical* advantages, not just wishful thinking.
Hans> Besides, there are enhancements which are simply compelling. Hans> You can write a dramatically better performance version control Hans> system with a much simpler design if the FS is atomic.
Define atomic please, with state diagrams and clear examples.
Hans> We have the performance lead. By next year we will be stable Hans> enough for mission critical servers, and then we start the Hans> serious semantic enhancements.
I think you've got it backwards. Make your serious semantic enhancements first, then make it stable, then make it fast. Because when you change the semantics, you break all kinds of things and then it doesn't matter how fast you are.
Hans> If you don't embrace progress, then you doom Linux to following Hans> behind, because the guys at Apple are pretty aggressive now that Hans> Jobs is back, and they WILL change the semantics, and they will Hans> do so in compelling ways, and Linux will be reduced to aping Hans> them when it should be leading them.
Monkey see, Monkey do then.
I'd like to point out another successful company which has extended the standard Unix namespace and that's 'Network Appliance' with it's .snapshot directory structure. It's a great idea and allows my users to restore files from snapshots without me having to think about it.
But it still causes problems since when I use rsync to move data, I need to put in stuff like:
rsync -az --exclude ".snapshot" --delete --delete-excluded <src> <dest>
to make sure it doesn't descend into that directory of previous versions and try to copy them over as well. And of course now now other apps can use the .snapshot name. But what if other vendors aped (ook ook) this decided to use their own names? Who decides which gets priority?
I think you need to go back and re-read Pike's paper on namespaces that you pointed to before and mull it over. And look at how simplicity is inherently powerful. If the design is too complicated, you're probably doing it wrong.
John
Re: The argument for fs assistance in handling archives
Quoted 1 line · John Stoffel
This is all pure marketing speak and economic theory.
I wouldn't grace it with the label economic theory. Lets be honest about this.. its pure -research-. Research is good but don't screw around with Linux semantics until you've done the research, figured out how to also make it useful to real users and proved its a good idea despite the past evidence from things like MacOS
Alan
Re: The argument for fs assistance in handling archives
In reply to Hans ReiserQuoted 6 lines · Hans Reiser
Quoted 2 lines · Linus Torvalds
But _my_ point is, no user program is going to take _advantage_ of anything that only one filesystem on one system offers.
Apple does not have this problem....
... because Apple customers are a captive lot...
Quoted 3 lines
and yes, the apps will take advantage of it, which is different from depending on it. If you use the wrong fs you will lose some of the features of the app.
... which is undesirable to say the very least for cross-platform Unix (even worse, Unix + Windows) applications, so applications won't ever use it (until Linux + super-duper filesystem is more than, say, 80% of the market). I.e., it won't happen soon enough for this to make sense right now. Perhaps in 13 more years...
Quoted 2 lines
For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice
Careful threading there... calling respected people names without any proof isn't exactly great PR.
Quoted 3 lines
(excepting Clearcase, which priced itself
into a niche market). It is 25 years past time for someone to change things.
Your opinion. Some people disagree.
Quoted 5 lines
That someone will have first mover advantage, and the more
little semantic features possessed the more lure there will be to use it which will increase market share which will lure more apps into depending on it and in a few years the other filesystems will (deservedly) have only a small market share because the apps won't all work on them.
You seem to be blinded by the "success effect": You only ever see the ones who succeed, you don't see the masses who tried to be "first movers" and failed utterly.
Quoted 3 lines
Besides, there are enhancements which are simply compelling. You can write a dramatically better performance version control system with a much simpler design if the FS is atomic.
Iff the performance of the FS is better _at doing what the system wants than the system itself_. That the CVS is simpler/faster is no use whatsoever if the underlying OS is more complex/slower. Besides, what is ideal for CVS isn't necesarily good for word processing or databases, and probably irrelevant or at least overkill for a lot of other applications, which all end up paying the costs.
Concentrate on giving what _most_ applications want, make it possible for applications with weird requirements to do their job.
Quoted 4 lines
Our transaction manager
first draft was written by a version control guy, and he would probably be happy to tell you how lack of atomicity other than rename makes version control software design hideous.
Right. But it doesn't make the design of text editing, or word processing, or mail handling, or... hideous, does it? Having a strong background in an area does put blinds on one's eyes.
Quoted 1 line
We have the performance lead.
I recently saw (preliminary) numbers directly contradicting this somewhere.
Quoted 3 lines
By next year we will be stable enough for
mission critical servers, and then we start the serious semantic enhancements.
By which the performance lead (if any) will do down the drain. There ain't no free lunch.
Quoted 5 lines
If you don't embrace progress, then you doom Linux to following behind, because the guys at Apple are pretty aggressive now that Jobs is back, and they WILL change the semantics, and they will do so in compelling ways, and Linux will be reduced to aping them when it should be leading them.
Why on earth should Linux copy a niche system? By now Linux's marketshare is _much_ larger than Apple's... I'd say it is the other way around: Apple tries desperately to differentiate itself from MSFT (and traditional Unix + Linux) in order not to dissapear (and keep their grip on their users). If they are doing it right, only time will tell. But that is no reason to screw up Linux' strengths.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Re: The argument for fs assistance in handling archives
Salut,
Quoted 9 lines · Horst von Brand
Quoted 6 lines · Hans Reiser
Quoted 2 lines · Linus Torvalds
But _my_ point is, no user program is going to take _advantage_ of anything that only one filesystem on one system offers.
Apple does not have this problem....
... because Apple customers are a captive lot...
�a suffit! Stop that damn evangelization. I've seen lots of content MacOS/X users. Personally I use MacOS/X, NetBSD and Linux to develop my software, and I quite like all the three. I'm actively developing Linux and MacOS/X. Why I do that? Because I'm a technician.
Linux users are no better people whatsoever. Restricting yourself to one OS is bad. I think most of the FS developers here will agree on that.
Tonnerre
(who's a technician, not an evangelist)
Re: The argument for fs assistance in handling archives
In reply to Hans ReiserQuoted 2 lines · Hans Reiser
Apple does not have this problem....
Deep, Hans. Deep.
You're ignoring the issue. You're flaming about not "embracing progress", when I was talking about how progress happens. You don't seem to understand _how_ people embrace progress, Hans. Railing at windmills isn't real smart.
Linus
Re: The argument for fs assistance in handling archives
In reply to Hans ReiserQuoted 2 lines
For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice
Or because it works fine, and isn't broken.
M.
Re: The argument for fs assistance in handling archives
Quoted 4 lines · Martin J. Bligh
Quoted 2 lines
For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice
Or because it works fine, and isn't broken.
I've heard the same argument a lot of times. People complaining that Unix is so seventies because it sticks to the old boring philosophy of everything is a file and that a file is a stream of bytes, nothing more. Modern operating systems such as VMS with basic database handling in the OS itself, or MacOS or NT with named streams is so much more modern. Why don't we get with the times?
It may be because just because of the simplicity it's fairly easy to use, harder to break och does one thing well. If you want structured storage, use a database, on top of the low level primitives, or use multiple files in a directory. Why complicate things?
/Christer
Signature · 4 lines
"Just how much can I get away with and still go to heaven?" Freelance consultant specializing in device driver programming for Linux Christer Weinigel <[email protected]> http://www.weinigel.se
Re: The argument for fs assistance in handling archives
Quoted 13 lines · Christer Weinigel
Quoted 4 lines · Martin J. Bligh
Quoted 2 lines
For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice
Or because it works fine, and isn't broken.
I've heard the same argument a lot of times. People complaining that Unix is so seventies because it sticks to the old boring philosophy of everything is a file and that a file is a stream of bytes, nothing more. Modern operating systems such as VMS with basic database handling in the OS itself, or MacOS or NT with named streams is so much more modern. Why don't we get with the times?
Quoted 4 lines
It may be because just because of the simplicity it's fairly easy to use, harder to break och does one thing well. If you want structured storage, use a database, on top of the low level primitives, or use multiple files in a directory. Why complicate things?
Sounds very much like the reason for using only triangle primitives on modern 3D-graphics hardware. And that works :)
Signature · 2 lines
Regards, Christian Iversen
Re: The argument for fs assistance in handling archives
In reply to Christer WeinigelQuoted 5 lines · Christer Weinigel
Quoted 3 lines · Martin J. Bligh
Quoted 2 lines
For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice
Quoted 8 lines
Quoted 1 line
Or because it works fine, and isn't broken.
I've heard the same argument a lot of times. People complaining that Unix is so seventies because it sticks to the old boring philosophy of everything is a file and that a file is a stream of bytes, nothing more. Modern operating systems such as VMS with basic database handling in the OS itself, or MacOS or NT with named streams is so much more modern. Why don't we get with the times?
Actually the handleing of special filesystem features in the OS is an OLD idea. VMS is actually not more modern than Unix, it is older than Unix owing much to earlier OS's such as RSX-11. It was a great insight on the part of Ritche and Thompson that the kernel should present a file as a byte stream and that the interpretation of the contents of a file is the province of the application, not the operating system.
Now it maybe, that there are features that can only be provided by adding support in the kernel, but we should be very careful not to be distracted by what we see as new wizbang features. The filesystem as a database has been tried before, ISAM is a very old concept.
Quoted 7 lines
It may be because just because of the simplicity it's fairly easy to use, harder to break och does one thing well. If you want structured storage, use a database, on top of the low level primitives, or use multiple files in a directory. Why complicate things?
/Christer
Signature · 3 lines
Brian Beattie LFS12947 | "Honor isn't about making the right choices. [email protected] | It's about dealing with the consequences." www.beattie-home.net | -- Midori Koto
Re: The argument for fs assistance in handling archives
In reply to Martin J. BlighQuoted 4 lines · Martin J. Bligh
Quoted 2 lines
For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice
Or because it works fine, and isn't broken.
OK. I'm not a kernel hacker. I'm not a crack C programmer. Nor am I a designer of filesystems. I'm just a guy that puts together and supports Linux systems for my customers.
In following this thread, I may be missing huge chunks of concept.
However, a few things are becoming clear to me:
1. The file as directory thing adds complexity that the administrator has to deal with. Symlinks are useful, but it's still aggravating to tar off a directory structure, take it somewhere, and then realize that all you have is links to something not in the archive because you didn't get your tar switches just right. Now we're talking about adding another set of "files which are not really files" to the semantics. More complexity. I'll take simplicity over some ivory tower ideal of "unified name space" any day.
2. The use of multiple streams within files by Linux apps would make Linux as cross-platform unfriendly as MS is trying to be. Say these features start getting used and you copy an OO.org document from a Linux box to a BSD box. It's broken. Of course, OO.org wouldn't use the streams in the first place because it would destroy their cross platform portability. So what's the point? No one who cares about cross platform portability can use it. Everyone who doesn't care about cross platform portability please raise your hand.
3. MS does require attributes and multiple streams, which makes these features important (even essential) to Samba, and Samba alone. Samba is important to Linux, so this can't be ignored. (Here I am implicitly assuming that Samba will need kernel support for this to do it right.)
So it seems to me that the only real consideration is giving Samba what it needs without making the semantics one bit more complex than absolutely necessary. It might even be wise to discourage use of these ambiguous new objects by the casual application programmer.
Then again, maybe I just have tunnel vision...
-Steve Bergman
Re: The argument for fs assistance in handling archives
Quoted 17 lines · Martin J. Bligh
Quoted 4 lines
Quoted 2 lines
For 30 years nothing much has happened in Unix filesystem semantics because of sheer cowardice
Or because it works fine, and isn't broken.
OK. I'm not a kernel hacker. I'm not a crack C programmer. Nor am I a designer of filesystems. I'm just a guy that puts together and supports Linux systems for my customers.
In following this thread, I may be missing huge chunks of concept.
However, a few things are becoming clear to me:
Quoted 8 lines
1. The file as directory thing adds complexity that the administrator has to deal with. Symlinks are useful, but it's still aggravating to tar off a directory structure, take it somewhere, and then realize that all you have is links to something not in the archive because you didn't get your tar switches just right. Now we're talking about adding another set of "files which are not really files" to the semantics. More complexity. I'll take simplicity over some ivory tower ideal of "unified name space" any day.
Are you afraid to learn something new? ;) Just joking. But really, it doesn't have to be very difficult. The extra streams etc would just be saved as files. If tar is patched then it would be no problem and no extra stuff but perhaps a switch --save-metas.
Quoted 8 lines
2. The use of multiple streams within files by Linux apps would make Linux as cross-platform unfriendly as MS is trying to be. Say these features start getting used and you copy an OO.org document from a Linux box to a BSD box. It's broken. Of course, OO.org wouldn't use the streams in the first place because it would destroy their cross platform portability. So what's the point? No one who cares about cross platform portability can use it. Everyone who doesn't care about cross platform portability please raise your hand.
Actually MacOS and Windows support multiple streams, only Linux doesn't. But of course there are BSD's etc too. I'd say to leave them behind.
The file streams would make my day a lot easier. The idea to split up contents of OO.org files into streams is bad. But that doesn't make the file streams bad. I see many uses that would make my every day life easier.
It isn't about cross plat form compatibility, but to add features that are useful and meta-data and file streams are.
Also. No one forces you to use either meta-data or streams, just as no one forces you to use ACLs or other things.
Quoted 4 lines
3. MS does require attributes and multiple streams, which makes these features important (even essential) to Samba, and Samba alone. Samba is important to Linux, so this can't be ignored. (Here I am implicitly assuming that Samba will need kernel support for this to do it right.)
I do not think the Samba would really require the streams support but it would certainly make life easier for Samba. Not to mention that these files would also be natively viewed on the Linux host.
Quoted 6 lines
So it seems to me that the only real consideration is giving Samba what it needs without making the semantics one bit more complex than absolutely necessary. It might even be wise to discourage use of these ambiguous new objects by the casual application programmer.
Then again, maybe I just have tunnel vision...
I would agree with tunnel vision. The kernel should provide the tools and options. Users and developers can then invent new things to use them. :)
~S
Quoted 1 line
-Steve Bergman
Re: The argument for fs assistance in handling archives
Quoted 9 lines
Quoted 8 lines
1. The file as directory thing adds complexity that the administrator has to deal with. Symlinks are useful, but it's still aggravating to tar off a directory structure, take it somewhere, and then realize that all you have is links to something not in the archive because you didn't get your tar switches just right. Now we're talking about adding another set of "files which are not really files" to the semantics. More complexity. I'll take simplicity over some ivory tower ideal of "unified name space" any day.
Quoted 4 lines
Are you afraid to learn something new? ;) Just joking. But really, it doesn't have to be very difficult. The extra streams etc would just be saved as files. If tar is patched then it would be no problem and no extra stuff but perhaps a switch --save-metas.
If they're saved as files, and the app has to be changed to use them anyway, then what's the point? Just change the app to use new files instead.
Quoted 3 lines
I would agree with tunnel vision. The kernel should provide the tools and options. Users and developers can then invent new things to use them. :)
Ugh. Change for changes sake is not a good thing. There's enough real problems in the world without inventing random features. More complexity without gain is a Bad Thing (tm).
I'm not saying streams is bad ... just that there don't seem to have been very many convincing (to me) arguments raised for it yet. The versioning stuff would be nice, IMHO, because the stuff mainly using it wouldn't need to be modified in many instances ... "vi /etc/configfile" would keep old copies for you (only the recovery tool would need to understand it). Saving icons seems easy enough to do with "foo.icon" as another file.
M.
Re: The argument for fs assistance in handling archives
In reply to SpamQuoted 2 lines
If tar is patched then it would be no
problem and no extra stuff but perhaps a switch --save-metas.
Aaargh. How about a tar that recognizes when the filesystem has metas and saves/restores them by default, with a warning when it's losings data because it's restoring to a simpler filesystem?
And maybe a switch --forget-metas.
Thousands of sys-admins will thank you to not make it easier to lose data with the scripts they have already written.
Signature · 2 lines
You are in a maze of twisty passages, all alike. Again. http://www.hacksaw.org -- http://www.privatecircus.com -- KB1FVD
Re: The argument for fs assistance in handling archives
In reply to SpamOn Thu, 2004-09-02 at 22:06 +0200, Spam wrote:
[...]
Quoted 4 lines
The file streams would make my day a lot easier. The idea to split up contents of OO.org files into streams is bad. But that doesn't make the file streams bad. I see many uses that would make my every day life easier.
When can we expect patches?
Quoted 2 lines
It isn't about cross plat form compatibility, but to add features that are useful and meta-data and file streams are.
Putting everything (including web server and browser) in the kernel is not a good (or even desireable) target - perhaps that's the reason for Win* to be where there are.
And it *is* done in user-space (ok, there more than one apprach and implementation but this may change. And you want to invent the next version.).
Quoted 2 lines
Also. No one forces you to use either meta-data or streams, just as no one forces you to use ACLs or other things.
Which is probably an argument against such features. How long do ACLs exist - as well as standard and in implementation?
How many people and/or apps are using it?
No more questions.
Quoted 8 lines
Quoted 4 lines
3. MS does require attributes and multiple streams, which makes these features important (even essential) to Samba, and Samba alone. Samba is important to Linux, so this can't be ignored. (Here I am implicitly assuming that Samba will need kernel support for this to do it right.)
I do not think the Samba would really require the streams support but it would certainly make life easier for Samba. Not to mention that these files would also be natively viewed on the Linux host.
It is not in the interest of MSFT that their aplication's files are viewable on other OSes. Expect incompatible changes to formats, streams and metadata contents (and surely patents on them - even if the idea is not new) if it starts to be really useful. It happended already in the past with the SMB protocol IIRC ....
Bernd
Signature · 3 lines
Firmix Software GmbH http://www.firmix.at/
mobil: +43 664 4416156 fax: +43 1 7890849-55
Embedded Linux Development and ServicesRe: The argument for fs assistance in handling archives
Quoted 15 lines · Spam
[...]
Quoted 4 lines
The file streams would make my day a lot easier. The idea to split up contents of OO.org files into streams is bad. But that doesn't make the file streams bad. I see many uses that would make my every day life easier.
When can we expect patches?
Quoted 2 lines
It isn't about cross plat form compatibility, but to add features that are useful and meta-data and file streams are.
Quoted 6 lines
Putting everything (including web server and browser) in the kernel is not a good (or even desireable) target - perhaps that's the reason for Win* to be where there are.
And it *is* done in user-space (ok, there more than one apprach and implementation but this may change. And you want to invent the next version.).
As I said before. It doesn't matter if it is in the kernel or user space. Not to me at least - as long as I could use extend the usage of my filesystem with meta-data, my own descriptions, perhaps streams, etc in a transparent way. I do not want to be confined to only use future Gnome apps.
Quoted 17 lines
Quoted 2 lines
Also. No one forces you to use either meta-data or streams, just as no one forces you to use ACLs or other things.
Which is probably an argument against such features. How long do ACLs exist - as well as standard and in implementation?
How many people and/or apps are using it?
No more questions.Quoted 8 lines
Quoted 4 lines
3. MS does require attributes and multiple streams, which makes these features important (even essential) to Samba, and Samba alone. Samba is important to Linux, so this can't be ignored. (Here I am implicitly assuming that Samba will need kernel support for this to do it right.)
I do not think the Samba would really require the streams support but it would certainly make life easier for Samba. Not to mention that these files would also be natively viewed on the Linux host.
Quoted 5 lines
It is not in the interest of MSFT that their aplication's files are viewable on other OSes. Expect incompatible changes to formats, streams and metadata contents (and surely patents on them - even if the idea is not new) if it starts to be really useful. It happended already in the past with the SMB protocol IIRC ....
No perhaps. But it doesn't make a difference to the ideas of having plugins for reiser4 that can give such features.
~S
Quoted 1 line
Bernd
Re: The argument for fs assistance in handling archives
In reply to Steve Bergman-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Steve Bergman wrote:
[...]
| In following this thread, I may be missing huge chunks of concept.
You are, but don't worry. You aren't the only one. (Before I get assaulted for this, let me say that there are people on this list smarter than me and I miss concept chunks quite frequently.)
| 1. The file as directory thing adds complexity that the administrator | has to deal with. Symlinks are useful, but it's still aggravating to | tar off a directory structure, take it somewhere, and then realize that | all you have is links to something not in the archive because you didn't | get your tar switches just right. Now we're talking about adding | another set of "files which are not really files" to the semantics. | More complexity. I'll take simplicity over some ivory tower ideal of | "unified name space" any day.
You want simplicity? How does this strike you:
scp file/serialize daemon@bsd:~/file.img
You don't have to worry about "file is not a file". You don't have to worry about fifos or device nodes. You don't have to worry about files in files. In fact, you don't have to worry about any new kind of file added, ever.
| 2. The use of multiple streams within files by Linux apps would make | Linux as cross-platform unfriendly as MS is trying to be. Say these
The use of ext3 as a filesystem isn't cross-platform. Every disk-write is platform-specific! We should all be using captive-ntfs instead!
Seriously, this isn't hard -- it's called a "library". And anything as bloated as OO.org has got to already be modular enough to add this sort of thing.
| features start getting used and you copy an OO.org document from a Linux | box to a BSD box. It's broken. Of course, OO.org wouldn't use the | streams in the first place because it would destroy their cross platform
If that was so, OO.org wouldn't use a GUI, because that isn't cross-platform. They'd use a purely commandline interface, because that is universal.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTe8PHgHNmZLgCUhAQKiNQ//Zo6BLeGK6/GBbYzcgyp51Ra4fRf2xoVo 0aOve2vinVl+RRzFGWSa66p+/rHd339BhxTPYhlLkm9SWbU9Et80J0R/bySYtc17 RmUHhmukSEXhJx+py7Vc99jXr03zr2fNaA3UBIfpMiAlj1UB2yOls5iwJgqfPVnX +YwYcVRfZXc3KWqH9SGcY9q+NX38wXoQh3rbeJ0OEzaIxnvPWlMvUFEN4OF92cub on+n0h+JKptwtXOhLh9HaKAzG9fDhy4tJwmN3GIemSzKpEZ0ZGS4+Pz4dcnRXP+S C6cMxpsmCnm5WVzmeoyBYBY28Fmnji00ShvxDGRFYOp48RPFhNaoMbVhLkQFoYfX a5k3iedhbjaWOLmpfNGaDNdpKFtqD3LJO1xOSkXz3sndAiLW9ArKhuu2+63ASAQy wlFknP09k13UHzD2iL5kPUPzy0A/BLqZPSpX4tXnF0xcSp4X8RoLnyu8BmrqaTjG fi7FkEsoVEB7BWslNNtKrH596sDhWCP+BQdkAYqOBh0fzSIJ+0uyo2nJRw3d8FEi /ynNVUqZEN8Xj08WHI+ucwUmRpTnh6spwXZC42K2tokAQa+95nSTegx3uNkypcGU l12s+BtIHFB25BKDaBReyGoBqUj0sdFvoHVkRNc/3DNY6MfRHV3rdgjz5NieT4gH uwMwLNstv8c=
=O3r8
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 3 lines · David Masover
The use of ext3 as a filesystem isn't cross-platform. Every disk-write is platform-specific! We should all be using captive-ntfs instead!
;-)
All this stuff about how no filesystem should be allowed to have semantic features others don't, it seems very Bolshevist to me.
Let Linux have an ecosystem with a diverse ecology of filesystems, and the features that work will reproduce to other filesystems. I thought that was the Linus way?
If not, why did I spend 10 years laying the storage layer groundwork for semantic enhancements when I could have taken that job at Sun as filesystems architect and made a lot more money?
I want to tinker. Let me play in my sandbox, and if you don't like what I do, don't imitate it..... I think there are plenty of users who like reiser4 though....
Linus, trying to outguess someone who has spent 2 decades studying namespace design as to what will be useful to users is risky. Look at reiser4's performance, see if it obsoletes V3, and if it does then let me play a bit.
Objecting on the grounds that it causes VFS bugs is reasonable, but I answered those questions and you did not respond (I can resend if asked). If you really really don't like what we do to VFS, well, we can confine ourselves to sys_reiser4(), but that is only a last resort from my view.
Hans
Re: The argument for fs assistance in handling archives
Quoted 7 lines · Hans Reiser
All this stuff about how no filesystem should be allowed to have semantic features others don't, it seems very Bolshevist to me.
Let Linux have an ecosystem with a diverse ecology of filesystems, and the features that work will reproduce to other filesystems. I thought that was the Linus way?
There's no reason you can't have semantics the other filesystems don't. There's also no reason you shouldn't continue researching in any direction you choose.
When you submit to the kernel (even just -mm), call the FS stable and suggest people start broadly using it, you're telling them the parts you have submitted are in fact fixed and reliable.
This implies you are done changing the semantics you have submitted, which also implies you would like applications to start seriously using it. Those application writers will want other filesystems to be able to support these new semantics, which means they should have a race free framework vetted by the kernel community and available for reuse by the other filesystems. In other words, the semantics should be in the VFS, without any requirement other FS's implement them, but the possibility for every FS to implement them.
If you don't want the framework to be agreed on by the community and available for other implementations, you either haven't stabilized the semantics, or you don't want anyone else using them. Either way, it precludes inclusion in the kernel. There are simply higher expectations for a filesystem in these cases, and a higher burden for stable semantics and interfaces.
This doesn't mean new semantics are impossible, it just means that constantly changing semantics of the filesystem core are bad for users and applications. They belong in patches that live outside the kernel until they are sufficiently researched to justify their inclusion.
The biggest victim of this discussion is the non-semantic part of reiser4. You've done a huge amount of research that is being held back because you haven't separated the semantic research from the storage layer areas. If the storage layers are ready now, please please just submit them.
-chris
Re: The argument for fs assistance in handling archives
In reply to Hans ReiserQuoted 8 lines · Hans Reiser
Quoted 2 lines · David Masover
The use of ext3 as a filesystem isn't cross-platform. Every disk-write is platform-specific! We should all be using captive-ntfs instead!
;-)
All this stuff about how no filesystem should be allowed to have semantic features others don't, it seems very Bolshevist to me.
Haven seen that. Just requests that _experimental_ features be prototyped in ReiserFS, and _if_ they show they are worth the hassle, _then_ design a nice VFS interface on them.
Quoted 3 lines
Let Linux have an ecosystem with a diverse ecology of filesystems, and the features that work will reproduce to other filesystems. I thought that was the Linus way?
It is.
Quoted 3 lines
If not, why did I spend 10 years laying the storage layer groundwork for semantic enhancements when I could have taken that job at Sun as filesystems architect and made a lot more money?
Dunno. That _you_ choose to do this is no reason _others_ will have to take the results over lock, stock and barrel.
Linux' development works because there is a lot of "wasted" effort on alternatives that don't pan out. Many don't ever leave the drawing board, others (like the (now defunct) Xiafs, devfs, and volume management code) went far, even got fully implemented, only to be scrapped later. It might just be ReiserFS' fate too..
Quoted 3 lines
I want to tinker. Let me play in my sandbox, and if you don't like what I do, don't imitate it..... I think there are plenty of users who like reiser4 though....
Go tinker and play. But don't aggravate non-ReiserFS-users, please. Come up with solutions to the issues Al Viro and Linus raised.
Quoted 2 lines
Linus, trying to outguess someone who has spent 2 decades studying namespace design as to what will be useful to users is risky.
I'd guess LKML adds up to a few thousand years mulling over this, so this isn't a great point.
Quoted 3 lines
Look at
reiser4's performance, see if it obsoletes V3, and if it does then let me play a bit.
I'd say it is a free world.
Quoted 5 lines
Objecting on the grounds that it causes VFS bugs is reasonable, but I answered those questions and you did not respond (I can resend if asked). If you really really don't like what we do to VFS, well, we can confine ourselves to
sys_reiser4(), but that is only a last resort from my view.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Re: The argument for fs assistance in handling archives
In reply to Steve BergmanQuoted 10 lines · Steve Bergman
However, a few things are becoming clear to me:
1. The file as directory thing adds complexity that the administrator has to deal with. Symlinks are useful, but it's still aggravating to tar off a directory structure, take it somewhere, and then realize that all you have is links to something not in the archive because you didn't get your tar switches just right. Now we're talking about adding another set of "files which are not really files" to the semantics. More complexity. I'll take simplicity over some ivory tower ideal of "unified name space" any day.
As with any addition of complexity we have to ask ourselves: do the advantages outweigh the disadvantages?
Adding complexity into the kernel may simplify userspace. Adding the complexity of an IP layer into the kernel simplifies networking applications, for example (they won't have to talk IP to the kernel).
In this case it's hard to imagine what advantages a multistream file will bring, simply because we aren't used to working with them. Or maybe there aren't any advantages at all. Point is: we don't know yet. Maybe we have to toy with the idea for some years before we really learn why multistream files are so great.
I do realize that this isn't how Linux development is done. First we need a problem, then somebody sits down and codes a solution. With reiser4 we've got a solution but we haven't figured out what the problem is yet.
Quoted 8 lines
2. The use of multiple streams within files by Linux apps would make Linux as cross-platform unfriendly as MS is trying to be. Say these features start getting used and you copy an OO.org document from a Linux box to a BSD box. It's broken. Of course, OO.org wouldn't use the streams in the first place because it would destroy their cross platform portability. So what's the point? No one who cares about cross platform portability can use it. Everyone who doesn't care about cross platform portability please raise your hand.
But do we really want to be the lowest common denominator? If you want to lead, you can't be compatible (in some cases). Also, if the reiser4 ideas catch on, they may spark the development of other, similar, filesystems. Maybe we'll end up with a ext4 in the 2.8 kernel...
Quoted 11 lines
3. MS does require attributes and multiple streams, which makes these features important (even essential) to Samba, and Samba alone. Samba is important to Linux, so this can't be ignored. (Here I am implicitly assuming that Samba will need kernel support for this to do it right.)
So it seems to me that the only real consideration is giving Samba what it needs without making the semantics one bit more complex than absolutely necessary. It might even be wise to discourage use of these ambiguous new objects by the casual application programmer.
Then again, maybe I just have tunnel vision...
Not tunnel vision. You're just looking at what we need *now*, and that happens to be exactly what we *have* now.
One possible use for multistream files that I see, is storage of 'not so important' data alongside of the real data of a file.
Think: the mimetype of a file; some metadata like author, keywords, etc.
When you copy the file and lose all metadata except the file itself, no real harm is done.
This could be used by the new mime system Gnome uses, for example. On regular filesystems, the mimetype of a file is determined by its extension. On reiser4(-like filesystems), the mimetype is determined by the 'mimetype' stream.
This would clearly simplify the gnome userspace, as there is no need anymore for a seperate extension<->mimetype conversion anymore. It would also be compatible with other tools or desktop environements, we only have to agree on what stream we're going to use for what metadata.
IMHO reiser4 is a toy we should give the Linux users to play with. Let's see what they can come up with. With the talented hackers in the community, it should be great.
Signature · 2 lines
Erik Hensema <[email protected]> (attempting to write english while drinking beer)
Re: The argument for fs assistance in handling archives
Salut,
Quoted 5 lines · Erik Hensema
In this case it's hard to imagine what advantages a multistream file will bring, simply because we aren't used to working with them. Or maybe there aren't any advantages at all. Point is: we don't know yet. Maybe we have to toy with the idea for some years before we really learn why multistream files are so great.
Thus I recommend doing the same thing everything does to get his stuff implemented.
How did User Mode Linux get accepted into the kernel?
In the beginning, people couldn't imagine any advantage of it. Then,
someone started coding it and provided patches. People tried it, improved it (important! It got to work on many platforms while being
out of the main tree!) and some day we noticed that it was really cool for kernel hacking and virtualization.
So when it approached a mature level where we finally saw it was cool, it went into the mainline kernel, where it is now.
For multistream files, what is that development cycle going to be? Get the idea, flame, flame, flame... ?
Code talks, bullshit walks.
Tonnerre
Re: The argument for fs assistance in handling archives
In reply to Martin J. Bligh-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Martin J. Bligh wrote:
|>For 30 years nothing much has happened in Unix filesystem semantics |>because of sheer cowardice | | | Or because it works fine, and isn't broken.
Ok, maybe it wasn't cowardice. Maybe it was laziness.
Because cowardice tells me that data corruption is bad. Data corruption kills kittens. End data corruption now!
Unfortunately, the only way to allow files to survive a crash is to implement transactions. The simplest way is to do it in the filesystem and to export an interface to userland. This is what Hans is doing, though AFAIK it's not done yet.
Laziness tells me that I should just use Windows anyway, because "it works". Switching to Linux requires training, porting apps requires work, and using Linux instead of Windows is a hell of a lot more work than just patching programs to do something like
+ start_transaction()
~ write() ~ blah() ~ write()
+ end_transaction()
How'd you like to be able to tell users "If a file gets corrupted, it's either a bug in OpenOffice or your hardware eats data." -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTe9pngHNmZLgCUhAQLTZQ/5AZq31sPKOWB/eI4WtVhM76V2+pmQdfNk aZkPtwhZlVrGDJscp45OVNthwMiSskD07om4obUMac79BCmA5805clZ7i0X1uzQB SecyX/7XITWz3iwF4VIaxJgDoo7YPFfbpWS0NTmhDJPt6h5Z+goEccLdTABj5kVk gbex4xNgXzPrnopvjnWx+K2K3ydvXiX+8bWV200F9S+j0uCObcfSZ357uQPqlReP gKQhEs0pCcoQQua81rXJfUhP8hCbegCpUmhJNI0MwoTUvHTnBU/+99+cCsTLR4gN 1XlbS9SIaaBXG7/AV9L7iQO5GF1t8mTMSe163EOzZ7ypLDb2km5e7zZ7t2DrA6SE Jgd0xT3cHjDgp1b3qhdeWYvt0QVfz5CGaiHukkT4pEOpsrM09BOhDhY72B74swuV byDIJ+y4X3J08i/a0zt15hBlVz1FUq7ac7doZZDt3orj93t3B/y4Xpb7tgzVGoWP T5AkR+uS5dGRUoMSI/btnPIK4ERlhbwFzCB5lgkas5kjp4dnawIKAk1wxPdhvEle g3SsNqSfBsA9hs6tzVw341GaWj6AWV7oOfg3j0c9I+7nt8SX5UeuTCwBWtCuwd/H +nnBulusiMnAm3tzQxq8aMpSVAqsZDywjgXSler1mNik4Ow5I5JdXl+Bf2ZWHF+G VHk4JVxMt/c=
=3z3Z
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 7 lines
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Martin J. Bligh wrote:
|>>For 30 years nothing much has happened in Unix filesystem semantics |>>because of sheer cowardice
Quoted 6 lines
| | | Or because it works fine, and isn't broken.Ok, maybe it wasn't cowardice. Maybe it was laziness.
Quoted 7 lines
Because cowardice tells me that data corruption is bad. Data corruption kills kittens. End data corruption now!
Unfortunately, the only way to allow files to survive a crash is to implement transactions. The simplest way is to do it in the filesystem and to export an interface to userland. This is what Hans is doing, though AFAIK it's not done yet.
I thought reiser4 had its journaling and atomic commits. Am I mistaken? I run reiser4 as primary fs on my test systems and it seem to work as expected.
Quoted 31 lines
Laziness tells me that I should just use Windows anyway, because "it works". Switching to Linux requires training, porting apps requires work, and using Linux instead of Windows is a hell of a lot more work than just patching programs to do something like
+
start_transaction()~ write() ~ blah() ~ write()+
end_transaction()How'd you like to be able to tell users "If a file gets corrupted, it's either a bug in OpenOffice or your hardware eats data." -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.orgiQIVAwUBQTe9pngHNmZLgCUhAQLTZQ/5AZq31sPKOWB/eI4WtVhM76V2+pmQdfNk aZkPtwhZlVrGDJscp45OVNthwMiSskD07om4obUMac79BCmA5805clZ7i0X1uzQB SecyX/7XITWz3iwF4VIaxJgDoo7YPFfbpWS0NTmhDJPt6h5Z+goEccLdTABj5kVk gbex4xNgXzPrnopvjnWx+K2K3ydvXiX+8bWV200F9S+j0uCObcfSZ357uQPqlReP gKQhEs0pCcoQQua81rXJfUhP8hCbegCpUmhJNI0MwoTUvHTnBU/+99+cCsTLR4gN 1XlbS9SIaaBXG7/AV9L7iQO5GF1t8mTMSe163EOzZ7ypLDb2km5e7zZ7t2DrA6SE Jgd0xT3cHjDgp1b3qhdeWYvt0QVfz5CGaiHukkT4pEOpsrM09BOhDhY72B74swuV byDIJ+y4X3J08i/a0zt15hBlVz1FUq7ac7doZZDt3orj93t3B/y4Xpb7tgzVGoWP T5AkR+uS5dGRUoMSI/btnPIK4ERlhbwFzCB5lgkas5kjp4dnawIKAk1wxPdhvEle g3SsNqSfBsA9hs6tzVw341GaWj6AWV7oOfg3j0c9I+7nt8SX5UeuTCwBWtCuwd/H +nnBulusiMnAm3tzQxq8aMpSVAqsZDywjgXSler1mNik4Ow5I5JdXl+Bf2ZWHF+G VHk4JVxMt/c=
=3z3Z
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Spam wrote:
[...]
|> I thought reiser4 had its journaling and atomic commits. Am I |> mistaken? I run reiser4 as primary fs on my test systems and it seem |> to work as expected.
consider this:
save_file () {
write() /* what if the write flushes halfway through
* then crashes?
*/
blah() /* what if "blah" crashes? */
write()
}Some apps need consistency across multiple files, but we don't even have it on a single file. You need a new interface to do that. As you can see, reiser4 has absolutely no way of knowing, anywhere in the above code, when you're done writing -- and when the file is consistent.
AFAIK, all that has to be done now for this to work is for them to finish the userland interface to the journalling and atomic commits that already exist for kernel space. But so far, all that is truly atomic is metadata operations -- chmod, mv, mkdir, touch, and rm/rmdir are all atomic, so long as you only use them on a single file/dir. But this has been true in reiserfs3, xfs, ext3, and others.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTfKF3gHNmZLgCUhAQKp3Q//WcijAx8VVdKjJFHENZvrA2P3/wjdCLIn OeIGi9VxbcQhS1cBM+BU6nBN6wz8S59o9hEM/F8L5oDxto8YW6sfyceoanbtVkLo r7k4i39A4Ao1UQFzYEzxWzgCA607WvGBboNN1J/5pYe+bWATWBw+oQgc6XikQofQ QwFX+0KxEDUva5Xkei/wBNhvpSveFBjAm/t79nVkANOVLji2KYALtewps+7QDiGc PNwfjSUcAbNgXrvuhwpocFgAGEhgS/Y7ANR4DqFAiUseDwY6WgkZqhTaxC8t9ZzF /Hd82sEb/J+hGpMfrTIe+D5VYhnjX+bPPs/U0OcfoH4CfkgOFgGOeuzlWg3xeTyX vgtgYGwuGQuWux7WNte0GOCNxfaFDE8lLGexdQYLkdVCSsLmzACA17H1bQdeswgH UnWL3oBoBx5BcdpkW1y1ZeoJv4xInazkzeU3GtEOd50tlJmOXmdjRNfj5LTrTh+1 CUpEKVvSjQ8MY70uQW6XKEUAkCYyBmSVeMay5SSQS/1Q0ISLEG3gMqPdsL2BOBO/ LkkQKjLe3LQS913wUtibL1CCfJCS/BrDswNCMsboivcg1SnwY+xJb9ffFMW0blBL kQ0B1AGZCjWHDfxXKIlC31PJYKzgE5vaKZ6+INj8mG8t5PzqgikBDQStvsXqczJq VgvvOVCjvho=
=I+Xw
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 11 lines · David Masover
Spam wrote:
[...]|> I thought reiser4 had its journaling and atomic commits. Am I |> mistaken? I run reiser4 as primary fs on my test systems and it seem |> to work as expected.consider this:
save_file() {
write()/* what if the write flushes halfway through
* then crashes?
reiser4 does protect from this. reiserfs v3 does not.
Quoted 17 lines
*/ blah() /* what if "blah" crashes? */ write() }Some apps need consistency across multiple files, but we don't even have it on a single file. You need a new interface to do that. As you can see, reiser4 has absolutely no way of knowing, anywhere in the above code, when you're done writing -- and when the file is consistent.
AFAIK, all that has to be done now for this to work is for them to finish the userland interface to the journalling and atomic commits that already exist for kernel space. But so far, all that is truly atomic is metadata operations -- chmod, mv, mkdir, touch, and rm/rmdir are all atomic, so long as you only use them on a single file/dir. But this has been true in reiserfs3, xfs, ext3, and others.
Re: The argument for fs assistance in handling archives
In reply to Hans ReiserHey, you know how device nodes have a bit set, indicating that they're device nodes and not regular files? Can a set of such properties be defined for reiser4 metadata properties?
Like a "metadata" bit so you can distinguish (if you wish) between regular files and metadata objects, and in addition an "archivable metadata" bit which indicates that the given piece of metadata is not automatically generated and should be archived during backup (some manually-generated metadata which does not need to be backed up will not have this bit set -- perhaps add another flag indicating that it's not automatic but unnecessary to archive).
Re: The argument for fs assistance in handling archives
Quoted 4 lines · Timothy Miller
Hey, you know how device nodes have a bit set, indicating that they're device nodes and not regular files? Can a set of such properties be defined for reiser4 metadata properties?
Like a "metadata" bit so you can distinguish (if you wish) between regular files and metadata objects, and in addition an "archivable metadata" bit which indicates that the given piece of metadata is not automatically generated and should be archived during backup (some manually-generated metadata which does not need to be backed up will not have this bit set -- perhaps add another flag indicating that it's not automatic but unnecessary to archive).
Interesting idea, particularly for autogenerated metadata. Some metadata, like thumbnails and icons might as well be ordinary files though. (Like some icon and want to use it as background bitmap? Just copy it, because it is a plain file. Or want to use it as clipart - fine, the word processor supports bitmap files. . .)
This is a case where something is "metadata" only because it is used as such, not because it has to be. File-as-dir is an interesting concept for such cases, because it is convenient to group the "metadata" with the file, have it move with the file and so on. But still have this metadata accessible as a file because it isn't all that special.
Helge Hafting
Re: The argument for fs assistance in handling archives
In reply to Linus TorvaldsQuoted 8 lines · Linus Torvalds
But _my_ point is, no user program is going to take _advantage_ of anything that only one filesystem on one system offers.
So there's no point.
I
Filesystem diversity will only be crippling for Linux if you choose for it to be so.
Re: The argument for fs assistance in handling archives
In reply to Linus TorvaldsQuoted 11 lines · Linus Torvalds
...
But _my_ point is, no user program is going to take _advantage_ of anything that only one filesystem on one system offers.So there's no point.
It's much saner (from _any_ app standpoint) to roll their own database in user space - that way it just works.
In other words, nobody is really ever going to take advantage of this. This is _not_ how technical advanncement happens. The way you get people to take advantage of something is to have a nice gradual ramp-up, not a sudden new feature that they can't realistically use.
...
Sure, but there are plenty of existing interfaces that you could emulate. One could make a small library to use a transactional filesystem to implement the Berkely DB interface (libdb) for example. So on filesystems without such support your app could use the regular userspace database, but on a transactional filesystem it'd just use regular files, which would simplify database management and likely increase performance over the userspace-only version (libdb is pretty slow). In terms of functionality it'd just be a drop-in replacement, just like math libraries that use MMX/SSE when available.
- Jim Bruce
Re: The argument for fs assistance in handling archives
Quoted 9 lines · James Bruce
Quoted 4 lines · Linus Torvalds
In other words, nobody is really ever going to take advantage of this. This is _not_ how technical advanncement happens. The way you get people to take advantage of something is to have a nice gradual ramp-up, not a sudden new feature that they can't realistically use.
Sure, but there are plenty of existing interfaces that you could emulate.
Absolutely. We should look at them, and whether they solve any issues.
I'm not saying that Hans would have to make up a new interface. Quite the reverse. One interface I suggested to use for attributes (and everybody hates ;) was the existing one from Solaris. Similarly, there are probably perfectly fine interfaces for the issue of transactions..
Linus
Re: The argument for fs assistance in handling archives
In reply to James BruceSalut,
Quoted 1 line · James Bruce
filesystem to implement the Berkely DB interface (libdb) for example.
Problem is that BerkeleyDB changes its API all the time.
Tonnerre
Re: The argument for fs assistance in handling archives
In reply to Linus Torvalds-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Linus Torvalds wrote:
[...]
| It's much saner (from _any_ app standpoint) to roll their own database in | user space - that way it just works.
Then why do we have apps which use Windows' Internet Options where it can, and otherwise ask the user for proxy settings?
|
| In other words, nobody is really ever going to take advantage of this.
| This is _not_ how technical advanncement happens. The way you get people
| to take advantage of something is to have a nice gradual ramp-up, not a
| sudden new feature that they can't realistically use.
|
| So before you do transactions in the filesystem, you have to be able
to do
| them in user space - and then make the filesystem version be _compatible_
| with that user space library. That way you can get people to use the
| library ("hey, it works anywhere") and then you can get them to use your
| filesystem ("hey, if you use our super-duper filesystem, you can do what
| you are already doing five times faster").Why not:
do transactions in the filesystem write a library to talk to that filesystem have the library do emulation on other systems
in that order.
| See? You're starting at the wrong end. Give me a portable interface to use
| _first_. Then do transactions in the filesystem.
If you're going to have to do both the portable interface and the fs support, does it really matter which order you do them in?
Again, this is curiosity, not sarcasm. Here my reasons for wanting the fs support first:
You don't always know exactly how the filesystem transactions will work. ~ You don't know until it's done whether you'll deviate from your design, cut certain features, replace them with others...
And if you write the portable library first, and it ends up supporting features a, b, and c, while the kernel supports features b, c, and d, which obsoletes a, you have to add some features and remove some features from the portable library to make it sane.
It'd be like writing OpenGL entirely in software, before hardware accelerators work, and at the last minute have to change the library to use triangles instead of splines. What's more, every single app now has to be rewritten or use the library's emulation -- that's assuming many apps would consider using a 3D library that doesn't have hardware accel support yet.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTfEsHgHNmZLgCUhAQK1ORAAiSQ6UHSYc/PPMMOv2VjTkSuK2MgPylKP d2jEitIuvWx01xAa++CLcUS6ncaiLuFYDA7TTHjhKYKAj+zuNN9OTI9EMGc6D+Ua lFcbcRFapqbT9HPrUnGPM8VQIR8UrBt54hRAGm0/jOCc8DKLa3d6xnRfqmKQhnkv w3ut4FiNo0GCd19hF82NxEA1YtwbOIgOIUxVlbADGI9QT/d+I3ZMA8QUNCrgrudn e9g7o0yTa2c16t/rrqkiYCqSHIhCgY8ZH8VQzG4uYSjU+qYS+QbpJakD82EyuF/k 4T4jyJOqqSv6tLwE1j2y0gteXDYOdmfda/TpDeBSaRvGf+SmNxfSDFL448B2QJgA e3V0HZHVVums1Vl6kX4pBaWM01V3n9BGeLaF2S9XSP7WCSWgW7t4ONRSLE3Bf2mm 1e5cMJ63mrtImb8Jyd8GhyCQ8/tkGnEYOIAHL3laGOjuc2ARBeyqmWrGKJsQ8iwu i9NsyukQC3NP2FY1nBSmKs9J2bWtixqk5OXpzupkqBkbpeKL1H/b9Mxpnn5+Aomi PjI/5i4QcaHMNBggxCG3B5zGJIWmvMkmZmHrHxvbLeENGvS5fiZ0Kscq7PL0FnE0 /2etDgjg65I66SqguXMzvtZdQ0hYgdwvoR83nZymHMSLYrFJ8FOxuB7kYQBeKZnu QpUz0UDTUZA=
=jVvF
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 3 lines · David Masover
It'd be like writing OpenGL entirely in software, before hardware accelerators work, and at the last minute have to change the library to use triangles instead of splines.
I expect that SGI did a software-only version of IrisGL first, so they could figure out what the hardware accelerators needed to support. And even then, the API for IrisGL got modified when it became OpenGL.....
Re: The argument for fs assistance in handling archives
In reply to David MasoverThe reason for supporting transactions in the kernel is that there is a whole slew of security bugs that have their origins in fs semantics being non-transactional. rename does not cut it as a transactions API. rename results in horribly performing apps that engage in contortions that cost a lot of programmer time to code them to stuff a transaction into a file that can be renamed. Most of the time programmers who should use a transaction/rename are just too busy to do it, and they take the risk that the user will lose data and be less secure rather than write the code to do it right. When they do do it, because it is complex code, they often screw it up.
That said, there is a whole slew of reasons for not supporting transactions in the kernel:
* the history of the literature stars transactional filesystems that paid too high a cost in performance
* isolation is very tricky to do without killing performance
* transactions which are kept open too long are a problem
* avoiding problems with locking shared data structures long enough to cause painful exclusion of things that should be able to run in parallel just fine
Reiser4, by being very modest in its ambitions, manages to not pay a performance cost. We say, isolation and rollback are for user space to manage until we can find a clever implementation, if such an implementation can even be found. Even if such an implementation can be found, there are plenty of cases where special case knowledge of what the app is doing can improve performance a lot, and thus for that app it will belong in user space.
All reiser4 does, is allow you to guarantee (using sys_reiser4) that a limited size set of fs operations will be fused into one atom together, and therefor will either all survive a crash or none will survive. This does however meet a lot of apps needs.
Hans
Quoted 54 lines · David Masover
Linus Torvalds wrote:
[...]
| It's much saner (from _any_ app standpoint) to roll their own database in | user space - that way it just works.Then why do we have apps which use Windows' Internet Options where it can, and otherwise ask the user for proxy settings?
| | In other words, nobody is really ever going to take advantage of this. | This is _not_ how technical advanncement happens. The way you get people | to take advantage of something is to have a nice gradual ramp-up, not a | sudden new feature that they can't realistically use. | | So before you do transactions in the filesystem, you have to be able to do | them in user space - and then make the filesystem version be _compatible_ | with that user space library. That way you can get people to use the | library ("hey, it works anywhere") and then you can get them to use your | filesystem ("hey, if you use our super-duper filesystem, you can do what | you are already doing five times faster").Why not:
do transactions in the filesystem write a library to talk to that filesystem have the library do emulation on other systemsin that order.
| See? You're starting at the wrong end. Give me a portable interface to use
| _first_. Then do transactions in the filesystem.If you're going to have to do both the portable interface and the fs support, does it really matter which order you do them in?
Again, this is curiosity, not sarcasm. Here my reasons for wanting the fs support first:
You don't always know exactly how the filesystem transactions will work. ~ You don't know until it's done whether you'll deviate from your design, cut certain features, replace them with others...
And if you write the portable library first, and it ends up supporting features a, b, and c, while the kernel supports features b, c, and d, which obsoletes a, you have to add some features and remove some features from the portable library to make it sane.
It'd be like writing OpenGL entirely in software, before hardware accelerators work, and at the last minute have to change the library to use triangles instead of splines. What's more, every single app now has to be rewritten or use the library's emulation -- that's assuming many apps would consider using a 3D library that doesn't have hardware accel support yet.
Re: The argument for fs assistance in handling archives
In reply to Hans ReiserQuoted 9 lines · Hans Reiser
Quoted 3 lines · Linus Torvalds
. Doing transactions on one file is
only the beginning - you'll find people who want transactions across file boundaries etc.Yup, that's what reiser4 is aiming at and where things get exciting for version control systems and the like. We are just missing the api code for it at the moment, all the infrastructure is there, and the api is in progress in
sys_reiser4.
Hope that is the same as or implied XA-support. Having to syncrhonized file updates with database transactions is not an uncommon problem.
-- robin
Re: The argument for fs assistance in handling archives
In reply to Linus TorvaldsQuoted 11 lines · Linus Torvalds
Quoted 7 lines · David Masover
And that is the right solution. Not the only one, but the right one. Caching isn't the only thing sorely in need of transaction support right now. Actually, I find it hard to think of anything on Linux which shouldn't have transactions -- why should
/etc/fstabor/home/david/homeworkbe more fragile than/var/lib/mysql?
Quoted 11 lines
It's easy to talk big.
It's damn hard to _implement_ a complex system, and make it stable and bug-free, and support legacy applications.
There is a reason why we do only what _must_ be done in kernel space.
Linus
Filesystems need transactions. Given that they need it, it was very tempting to build an infrastructure offering some of the functionality to user space, and so we did in reiser4. What we have is very rudimentary, but I think I am prepared to say that important aspects of high performance transactions are better done in the kernel than in user space. It will be interesting to see what aspects of transaction functionality belong in user space as our transactions API comes into existence. Clearly there are some things about isolation that are better implemented by user space because user space can make optimizations the kernel doesn't know enough to make. Still, it will be interesting to see how low we can make the overhead of isolation. Right now reiser4 supports atomicity but not isolation, which is to say we offer to guarantee that a set of operations will either all survive a crash or none of them will, but we don't let you roll them back without a crash. (Originally I called this a transcrash rather than atomicity, but nobody liked the term.) Being modest in our ambition was important to performance, but maybe in time we will find a high performance way to be more ambitious.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Linus TorvaldsQuoted 5 lines · Linus Torvalds
and quite frankly, I think you can do the above pretty much totally in user space with a small library and a daemon (in fact, ignoring security issues you probably don't even need the daemon). And if you can prototype it like that, and people actually find it useful, I suspect kernel support for better performance might be possible.
Gnome already supports this in the gnome-vfs2 layer. "MC" has supported it since the late 1990's.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 9 lines · Alan Cox
Quoted 5 lines · Linus Torvalds
and quite frankly, I think you can do the above pretty much totally in user space with a small library and a daemon (in fact, ignoring security issues you probably don't even need the daemon). And if you can prototype it like that, and people actually find it useful, I suspect kernel support for better performance might be possible.
Gnome already supports this in the gnome-vfs2 layer. "MC" has supported it since the late 1990's.
Firstly, if I have to do it from a Gnome program, about the only program where looking in a tar file is visibly useful is Nautilus. Ironically, clicking on a tar file in Nautilus doesn't work, despite having a dependency on gnome-vfs2. :/
Secondly, no, Gnome and MC don't support entering a container file, letting you make changes in it, and remembering those changes to _lazily_ regenerate the container file when you need it linearized, possibly months later or never, by some unrelated program.
Thirdly, you must be referring to the Gnome versions of Bash, Make, GCC, coreutils and Perl which I haven't found. Perhaps we have a different idea of what "supports this" means :)
uservfs, which is based on gnome-vfs and getting a bit rusty due to disuse, does try to solve the last problem. Unfortunately it needs further work to have a nicer interface, and the second problem is still not solved by gnome-vfs.
-- Jamie
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 4 lines · Jamie Lokier
Firstly, if I have to do it from a Gnome program, about the only program where looking in a tar file is visibly useful is Nautilus. Ironically, clicking on a tar file in Nautilus doesn't work, despite having a dependency on gnome-vfs2. :/
This should be fixed in Nautilus, not the kernel.
Quoted 4 lines
Secondly, no, Gnome and MC don't support entering a container file, letting you make changes in it, and remembering those changes to _lazily_ regenerate the container file when you need it linearized, possibly months later or never, by some unrelated program.
Why do this in a tar file? tar = "tape archive". It isn't designed to be a file system. Sure, it's nice to have tools that make it easier to access files in a tar file, but to this isn't a job for the kernel.
Quoted 3 lines
Thirdly, you must be referring to the Gnome versions of Bash, Make, GCC, coreutils and Perl which I haven't found. Perhaps we have a different idea of what "supports this" means :)
Please don't tell me that we have expectations to run make from within a tar file. This is getting silly. tar does a pretty good job of extracting files into real directories, and putting them back into an archive. I don't see a need to teach the kernel how to deal with compound files when user space can do it very easily.
Shaggy
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Dave Kleikamp wrote:
[...]
| Please don't tell me that we have expectations to run make from within a | tar file. This is getting silly. tar does a pretty good job of | extracting files into real directories, and putting them back into an | archive. I don't see a need to teach the kernel how to deal with | compound files when user space can do it very easily.
Suppose I've got a tar file with an index attached. Suppose it's something like /usr/src/linux. Am I expected to extract all code for all architectures, with all drivers, all docs, etc? Now, yes -- or I have to figure out exactly which ones I need before I extract them manually, one by one.
But with tar support for make (and so on), files can be extracted on demand. It's possible to do this in userspace, with named pipes, but that's much slower and insanely clumsy.
This has further implications -- imagine a desktop, binary distro shipped with all files except the very most basic stuff as package archives. They can all be extracted, on demand -- the first time I run OpenOffice.org, it's installed. If there needs to be post-installation, that's handled by the .deb plugin (or whatever).
I don't know offhand how big OOo is. I think it's something like this:
~ The installer is at most half (and maybe only a third) of the full installation. That's a HUGE optimization!
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTe5/HgHNmZLgCUhAQI5vQ/+NyU/tbW1Dyaf/OlDUEScH8jHghdcMPQQ qcyBbzid9hMT0pm4fRX4CQJ/vm+VLhfvYzEmgRUCyNY3JybCKeS/EynRt/ybdblu aB+hO8meFitBmAa7kYrj1UhWvoSvDSZgAwC9k50DYPuQO1kVZFjFYcPee1P54iwJ UMn9RE01aeufCt1+jWFxhsEZKfNWvXDCaQtqa483A2AWWzklwF25ZW2kSfp6G+i0 g1jND8pPDkQcP8ujGTuDxEI8LsN62glNzVZ8MhPa65lZI1vO5Ll2dDL2QKgNwziK MqtMMJD1d3HWa7QBHwMegJ0teR/hiqJ62SgQr3QpW4Xy9Ss0VUVH1HNuhxwPB2rl YYomqw2yO/GGSDs5XuXm/cRM5E9d+nvu1V8bsrSa5LK/64Vlp6huLkLNvOZ3y6vK 38ELPBxbmIA3iWTgaYDPANX/vrpnA0K8JQU9M4LMveaHhxfEcDbH+iZHtpjsYqF3 allfHH2SEZRFlXGxKBNZsXTcrudAHjoyEOQ+UiI9QLCM83G4bFGr1WEGOEHmD0ry hBETe8GkwuQK1CfxFm5obgFUmE4TwVRIWVD71EvoJFuBS+dlezO6GOZ5mDf91tSe goPS2f/9XKwyYfOnEnnfXT17k5SYjTB9m0upi6q7dJpxvg1535E6N1nCqdLZzTcJ mVkdhfFsUqI=
=YjJH
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 8 lines · David Masover
Dave Kleikamp wrote:
[...]| Please don't tell me that we have expectations to run make from within a | tar file. This is getting silly. tar does a pretty good job of | extracting files into real directories, and putting them back into an | archive. I don't see a need to teach the kernel how to deal with | compound files when user space can do it very easily.
Quoted 5 lines
Suppose I've got a tar file with an index attached. Suppose it's something like
/usr/src/linux. Am I expected to extract all code for all architectures, with all drivers, all docs, etc? Now, yes -- or I have to figure out exactly which ones I need before I extract them manually, one by one.
I don't think it's unreasonable to expect someone to either extract the whole tar file, or identify what files they want from it. If you think there is too much in the tar file, roll your own with only the files you need.
Quoted 3 lines
But with tar support for make (and so on), files can be extracted on demand. It's possible to do this in userspace, with named pipes, but that's much slower and insanely clumsy.
This doesn't justify bloating the kernel. untar the darn thing and user space does fine.
Quoted 5 lines
This has further implications -- imagine a desktop, binary distro shipped with all files except the very most basic stuff as package archives. They can all be extracted, on demand -- the first time I run OpenOffice.org, it's installed. If there needs to be post-installation, that's handled by the .deb plugin (or whatever).
Are you saying install it on demand the first time it's run? This doesn't take any new kernel function.
Or are you saying that the files are never installed on the filesystem, but always accessed from the package archives? In this case, why not ship each package as a compressed iso, and have the system mount the iso to run the app. I really don't see the point though, in that disk space is very cheap these days.
Quoted 3 lines
I don't know offhand how big OOo is. I think it's something like this:
~ The installer is at most half (and maybe only a third) of the full installation. That's a HUGE optimization!
Optimization? How much performance are you willing to give up to save a little bit of disk space?
Signature · 2 lines
David Kleikamp IBM Linux Technology Center
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Dave Kleikamp wrote:
| On Thu, 2004-09-02 at 19:25, David Masover wrote: | |>Dave Kleikamp wrote: |>[...] |>| Please don't tell me that we have expectations to run make from within a |>| tar file. This is getting silly. tar does a pretty good job of |>| extracting files into real directories, and putting them back into an |>| archive. I don't see a need to teach the kernel how to deal with |>| compound files when user space can do it very easily. |> |>Suppose I've got a tar file with an index attached. Suppose it's |>something like /usr/src/linux. Am I expected to extract all code for |>all architectures, with all drivers, all docs, etc? Now, yes -- or I |>have to figure out exactly which ones I need before I extract them |>manually, one by one. | | | I don't think it's unreasonable to expect someone to either extract the | whole tar file, or identify what files they want from it. If you think | there is too much in the tar file, roll your own with only the files you | need.
That is not a solution, that is exactly the opposite of a solution. "Roll my own" implies that I must download the tar file, extract the whole thing, grab what I need, and then tar it up again. Then I extract it again. Doh!
More seriously, suppose it's a format like zip, where I don't have to decompress the whole file to get a listing. And suppose I want to compile some little example from it, but the file itself is huge -- half a gig, say, and I only need ten megabytes.
But the only way I'm going to find out _which_ ten megabytes I need is to extract the Makefile, read it, then go find all the other little Makefiles, not to mention the configure script, the header files, and so on...
But suppose that make can _transparently_ extract _only_ the files I need for this?
|>But with tar support for make (and so on), files can be extracted on |>demand. It's possible to do this in userspace, with named pipes, but |>that's much slower and insanely clumsy. | | | This doesn't justify bloating the kernel. untar the darn thing and user | space does fine.
Does it really bloat the kernel? My kernel doesn't feel bloated, and it's got reiser4 and much of what's needed for this.
Remember, saying "I can access foo.zip/contents" doesn't mean "zip is in the kernel".
|>This has further implications -- imagine a desktop, binary distro |>shipped with all files except the very most basic stuff as package |>archives. They can all be extracted, on demand -- the first time I run |>OpenOffice.org, it's installed. If there needs to be post-installation, |>that's handled by the .deb plugin (or whatever). | | | Are you saying install it on demand the first time it's run? This | doesn't take any new kernel function.
It does for it to appear installed. And to only install the pieces that are needed at that time.
| Or are you saying that the files are never installed on the filesystem, | but always accessed from the package archives? In this case, why not
No, only that if I only wanted OpenOffice writer, why should I install all of OpenOffice? Yes, it could be broken into smaller packages. What I'm talking about is exactly the pieces of an installation that you need
- -- insanely fine granularity -- done automatically, on demand. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTkLiHgHNmZLgCUhAQK7+A//TATQvQ3U61VA/mdVqylnrkWC7tNOewwq MqF+KKU3Cc/n54mOlGTpph2qpPJTzv1y4KlVgcDM/d0bn1cDPx41n//xEe9QXlqu vOywb8g11HSlAhKmbl4APwCHFHr1HibHgYqM7PmeVSD+Xfy5gJvIW5Oc44f16+q9 agEXWAk0EgM0WCAKEQFxN56i8e7qHq28PPGzcpGcn08xmBmD9Ik71jjpLY88csYy kjH32ExEy+uABq+Tglfr0EBZR4RDuqkxsei7cL3Rn58O8twJtn8UP3VcukTLciZw jb7If3ekuO7BXYJbwB/foFEESFql68jNKH7c7+Bzeb5pREloreVine/2rRM1iekD FUeTv78kn+6G/INl9XwUB2ER0KZOy8n2wZut35T5w94GtZgmdHpm+3mCOscS6BdG JNx/HRGJJXfm0P/7tKbgZ/3wjQlFzbC5HcByn9Ocfm8qrNsoLxwtF/8aId/9ctD9 lEmDMUHYuVC51/m5ka+i/XUQeuzgbtY5QKoNsxWXYZfeBNQfMqfMOsVWP1wFMlpB mPmf6w+4idp1aIYwgvPyQee1BZiXmkbncglcnY+J4y46AZ+tDEO5eqjJrEU9kA7v NNVfCehCZ2IIo1TMHcQizsHQ53NEmZ5gwCYXvymGS1+6fyE0SqcP9EolVuUfnRN9 yNdulMJ0gDg=
=TeOM
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
In reply to David MasoverSalut,
Quoted 5 lines · David Masover
This has further implications -- imagine a desktop, binary distro shipped with all files except the very most basic stuff as package archives. They can all be extracted, on demand -- the first time I run OpenOffice.org, it's installed. If there needs to be post-installation, that's handled by the .deb plugin (or whatever).
zsh using apt extensions does that.
Tonnerre
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Dave KleikampHi!
Quoted 9 lines
Quoted 3 lines
Thirdly, you must be referring to the Gnome versions of Bash, Make, GCC, coreutils and Perl which I haven't found. Perhaps we have a different idea of what "supports this" means :)
Please don't tell me that we have expectations to run make from within a tar file. This is getting silly. tar does a pretty good job of extracting files into real directories, and putting them back into an archive. I don't see a need to teach the kernel how to deal with compound files when user space can do it very easily.
Actually its not easy. User has to manually extract it and manually delete it when he's done. Not nice.
Pavel
Signature · 1 line
When do you have heart between your knees?
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Jamie LokierQuoted 1 line · Jamie Lokier
Firstly, if I have to do it from a Gnome program, about the only program where looking in a tar file is visibly useful is Nautilus. Ironically, clicking on a tar file in Nautilus doesn't work, despite having a dependency on gnome-vfs2. :/
Do you have file-roller installed?
I can open tar/zip/rar/etc.. files from anywhere in gnome2, eg in Galeon I can click on a tar.gz URL (http or whatever) and have it open it in file-roller, from where i can browse the files to my hearts content.
regards,
Signature · 4 lines
Paul Jakma [email protected] [email protected] Key ID: 64A2FF6A Fortune: Why isn't there a special name for the tops of your feet? -- Lily Tomlin
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 10 lines · Jamie Lokier
Quoted 5 lines
Firstly, if I have to do it from a Gnome program, about the only program where looking in a tar file is visibly useful is Nautilus. Ironically, clicking on a tar file in Nautilus doesn't work, despite having a dependency on gnome-vfs2. :/
Quoted 6 lines
Do you have file-roller installed?
I can open tar/zip/rar/etc.. files from anywhere in gnome2, eg in Galeon I can click on a tar.gz URL (http or whatever) and have it open it in file-roller, from where i can browse the files to my hearts content.
But can you actually do things with these files? Can you run applications or edit files directly, or is there need for temporary unzip first?
~S
Quoted 1 line
regards,
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 3 lines · Spam
But can you actually do things with these files? Can you run applications or edit files directly, or is there need for temporary unzip first?
You always need that for zip files. Firstly because executables are paged so you need an accessible random access copy of the bits. Secondly because data may be paged, and also for seek performance.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 11 lines · Spam
Quoted 3 lines
But can you actually do things with these files? Can you run applications or edit files directly, or is there need for temporary unzip first?
You always need that for zip files. Firstly because executables are paged so you need an accessible random access copy of the bits. Secondly because data may be paged, and also for seek performance.
Yes, some archive types can't be partially unzipped either. But my point is that it wouldn't be transparent to the application/user in the same way.
~S
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 3 lines · Spam
Yes, some archive types can't be partially unzipped either. But my point is that it wouldn't be transparent to the application/user in the same way.
It doesnt matter whether it is transparent to the application. It can be the application which implements the required level of transparency.
User doesnt care what provides the transparency or how it's implemented.
Quoted 1 line
~S
regards,
Signature · 4 lines
Paul Jakma [email protected] [email protected] Key ID: 64A2FF6A Fortune: A committee is a group that keeps the minutes and loses hours. -- Milton Berle
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 9 lines · Spam
Quoted 4 lines
Yes, some archive types can't be partially unzipped either. But my point is that it wouldn't be transparent to the application/user in the same way.
Quoted 6 lines
It doesnt matter whether it is transparent to the application. It can be the application which implements the required level of transparency.
User doesnt care what provides the transparency or how it's implemented.
Indeed. I hope I didn't say otherwise :). Just that I think it will be very difficult to have this transparency in all apps. Just thinking of "nano file.jpg/description.txt" or "ls file.tar/untar/*.doc". Sure in some environments like Gnome it could work, but it still doesn't for the rest of the flora of Linux programs.
~S
Quoted 1 line
regards,
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 12 lines · Spam
Quoted 11 lines · Spam
Quoted 4 lines
Yes, some archive types can't be partially unzipped either. But my point is that it wouldn't be transparent to the application/user in the same way.
It doesnt matter whether it is transparent to the application. It can be the application which implements the required level of transparency.
Quoted 5 lines
Quoted 2 lines
User doesnt care what provides the transparency or how it's implemented.
Indeed. I hope I didn't say otherwise :). Just that I think it will be very difficult to have this transparency in all apps.
You're missing the point. We don't need transparency in all apps. You can write an application to be as transparent as you want, but you don't need every app to to understand every file format.
Quoted 3 lines
Just
thinking of "nanofile.jpg/description.txt" or "ls file.tar/untar/*.doc".
I don't do much image editting, but I'm sure there are applications that let you edit the description in a text file. You can even create a script that extracts it, runs nano, and puts it back into the jpeg.
This works for me:
tar -tf file.tar | grep '\.doc'
There are userland tools that deal with hundreds of file formats. Use the tool you need, rather than try to have the kernel do everything.
Quoted 3 lines
Sure in some environments like Gnome it could
work, but it still doesn't for the rest of the flora of Linux programs.
Just choose the right program. tar groks tar files, not ls.
Signature · 2 lines
David Kleikamp IBM Linux Technology Center
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 19 lines · Spam
Quoted 15 lines
Quoted 14 lines · Spam
Quoted 4 lines
Yes, some archive types can't be partially unzipped either. But my point is that it wouldn't be transparent to the application/user in the same way.
It doesnt matter whether it is transparent to the application. It can be the application which implements the required level of transparency.
User doesnt care what provides the transparency or how it's implemented.
Quoted 6 lines
Quoted 2 lines
Indeed. I hope I didn't say otherwise :). Just that I think it will be very difficult to have this transparency in all apps.
You're missing the point. We don't need transparency in all apps. You can write an application to be as transparent as you want, but you don't need every app to to understand every file format.
No, but not every user "can write an application" either, or even have the skills to apply patches. What I was talking about wasn't just tar, which itself isn't the best example anyway, but the idea that users can load plugins that will extend the functionality of their filesystems. That idea seem to be to be _much_ better than trying to teach every user how to write applications or patch existing ones.
Quoted 8 lines
Quoted 3 lines
Just
thinking of "nanofile.jpg/description.txt" or "ls file.tar/untar/*.doc".I don't do much image editting, but I'm sure there are applications that let you edit the description in a text file. You can even create a script that extracts it, runs nano, and puts it back into the jpeg.
Quoted 5 lines
This works for me:
tar -tf file.tar | grep '\.doc'There are userland tools that deal with hundreds of file formats. Use the tool you need, rather than try to have the kernel do everything.
No, but if I wanted to have an encryption plugin active for some of my files or directories then why should I not be able to? I still want to edit, view and save my encrypted files.
Again, this was just an example of what could be done with plugins. It is not said that every conceivable plugin will be written, nor loaded per default. Even though plugins cannot today be dynamically used, they will be eventually. Reiser4 is still very young.
Please separate your thoughts for specific plugins from those of the idea to have plugins at all.
~S
Quoted 5 lines
Quoted 3 lines
Sure in some environments like Gnome it could
work, but it still doesn't for the rest of the flora of Linux programs.Just choose the right program. tar groks tar files, not ls.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 7 lines · Spam
Quoted 3 lines
You're missing the point. We don't need transparency in all apps. You can write an application to be as transparent as you want, but you don't need every app to to understand every file format.
No, but not every user "can write an application" either, or even have the skills to apply patches. What I was talking about wasn't just tar, which itself isn't the best example anyway,
That was one of the examples you gave, that and .jpg. I believe they are both ridiculous.
Quoted 5 lines
but the idea
that users can load plugins that will extend the functionality of their filesystems. That idea seem to be to be _much_ better than trying to teach every user how to write applications or patch existing ones.
If I understand Hans' plugins, they are not user-loadable, but rather a statically built part of the kernel.
Quoted 4 lines
No, but if I wanted to have an encryption plugin active for some of my files or directories then why should I not be able to? I still want to edit, view and save my encrypted files.
I would not argue against an encryption plugin.
Quoted 3 lines
Again, this was just an example of what could be done with plugins. It is not said that every conceivable plugin will be written, nor loaded per default.
This I agree with.
Quoted 2 lines
Even though plugins cannot today be dynamically
used, they will be eventually. Reiser4 is still very young.
As kernel modules, this would make sense. I don't see just any unprivileged user being able to add code into the file system, though.
Quoted 2 lines
Please separate your thoughts for specific plugins from those of the idea to have plugins at all.
I'm not against reiser4 plugins. I don't think file system code should care about the type of data in a file, and do any interpretation based on it.
Shaggy
Signature · 2 lines
David Kleikamp IBM Linux Technology Center
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to SpamQuoted 2 lines · Spam
Quoted 1 line · Dave Kleikamp
Spam <[email protected]> said:
[...]
Quoted 11 lines
Quoted 3 lines
You're missing the point. We don't need transparency in all apps. You can write an application to be as transparent as you want, but you don't need every app to to understand every file format.
No, but not every user "can write an application" either, or even have the skills to apply patches. What I was talking about wasn't just tar, which itself isn't the best example anyway, but the idea that users can load plugins that will extend the functionality of their filesystems. That idea seem to be to be _much_ better than trying to teach every user how to write applications or patch existing ones.
Why compare "write application or apply patches" to "load plugin"? It would be locical to compare running applications with loading plugins (and even so, loading plugins is presumably root-only).
[...]
Quoted 6 lines
Quoted 2 lines
There are userland tools that deal with hundreds of file formats. Use the tool you need, rather than try to have the kernel do everything.
No, but if I wanted to have an encryption plugin active for some of my files or directories then why should I not be able to? I still want to edit, view and save my encrypted files.
Use an editor that knows about encrypted files. Decrypt/edit/encrypt if no other option (I'm sure emacs can be coerced to do that transparently ;-). I for one would be _way_ more confortable with my users doing that than them loading random modules into the kernel. Besides, if one of my users doesn't trust the system encryption programs, and prefers hers, she can be happy.
Quoted 4 lines
Again, this was just an example of what could be done with plugins. It is not said that every conceivable plugin will be written, nor loaded per default. Even though plugins cannot today be dynamically used, they will be eventually. Reiser4 is still very young.
Modules of the kernel were supposed to have all those magic properties too, until there were nasty races... and it was _seriously_ considered to take them out. They stayed because they are root-only business, and (un)loading is rare. FS plugins are kernel modules, AFAIU, and are subject to the same problems.
Quoted 2 lines
Please separate your thoughts for specific plugins from those of the idea to have plugins at all.
If you can't find concrete uses for specific plugins, they are the proverbial solution searching for a problem.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Horst von Brand wrote:
[...]
| Use an editor that knows about encrypted files. Decrypt/edit/encrypt if no
You use emacs. I use vim. My brother uses gedit. My parents use abiword. Perhaps I should patch them all? If that was so easy, why is there cryptoloop/dm-crypt?
| is rare. FS plugins are kernel modules, AFAIU, and are subject to the same | problems.
Actually, FS plugins currently cannot be modules. They are currently called "plugins" because they share some concepts with browser plugins, and it sounds great in marketing.
| If you can't find concrete uses for specific plugins, they are the | proverbial solution searching for a problem.
Fine, let them be. They are just very well structured code. If you use reiser4 and don't look in the source or the "metas" dir, they are completely invisible to you. I used the betas for months, and "metas" never burned me.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTlUt3gHNmZLgCUhAQJVGhAAki5RLZckm5jiZnw7MXQLqURBvm+cz7wU eFPfGyOE0SpFKJLFntWr0zrAgyK5ClgwqB7wsJEldWUfGBPdYpH5lroOQEHVEGBs 4X+ze/xyUOL6z3S07a85jNibYamDeoCDc5P0Vc6GWrdpsU7FQGXrHykNyglDxFJ1 MiYEQkB8NYDzQukl+7HPR3qPhQpAl5hx3XtmOcC5w0/88ATMqXg81DoVzPAPlsrL IPu4ai7KjXRaY1sKo8SU4orj7iQHmmkiFJJg+QwVU9sO2GMBGpXZRSr3KcUL3ux5 nr+++ceVXyLADZaJRYp5LoTxL0KPJUKhaa9ABLmN2zQ5hT/v6AlQmKKD3s6ca02a A8MQxy69hG50RVSeJm9yjRYQQBvATEXslCQXPXSAlLJGrPZ1FZgQdYyo2wNboD23 ep+JP2qTPdyTFFl2TOtoeR7fIsjg6DF5Bq0uh0maqC0UXXIo1GRO/OQGsNMfCN88 pevDg0GvE+bdeL8CEZYfDzu4aaUs+ltzZSPEKlXHCGFORL9iSuYhqdUCRPbcKYOy uyhE7fgZoPYsOZLfChmXllEF69Cs5Vm5R0ymIgHqprfAjfqYf4ypbU0fukFDQ5dS GGFTfxketTHdhYr7ATAfZg08ZMP819UnvcwISflLlDBwvpL4BDAWOrnDFnbHSugk lPAxyPnfECM=
=h24Y
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 2 lines · David Masover
I used the betas for months, and "metas"
never burned me.
metas is much less likely than what clearcase uses to be hit accidentally (After all these years I forget what exactly clearcase special cases, maybe it was "@@"). When people using clearcase suffer a namespace collision, life goes on, no big deal, they structure a filename slightly differently and so what? I mean, just how much do we suffer from not being able to use '/' in filenames? Every once in a while it is annoying, but not that much....
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Horst von BrandQuoted 21 lines
Quoted 2 lines · Spam
Quoted 1 line · Dave Kleikamp
Spam <[email protected]> said:
[...]
Quoted 11 lines
Quoted 3 lines
You're missing the point. We don't need transparency in all apps. You can write an application to be as transparent as you want, but you don't need every app to to understand every file format.
No, but not every user "can write an application" either, or even have the skills to apply patches. What I was talking about wasn't just tar, which itself isn't the best example anyway, but the idea that users can load plugins that will extend the functionality of their filesystems. That idea seem to be to be _much_ better than trying to teach every user how to write applications or patch existing ones.
Quoted 3 lines
Why compare "write application or apply patches" to "load plugin"? It would be locical to compare running applications with loading plugins (and even so, loading plugins is presumably root-only).
Doesn't matter if it is root-only. Ask the admin about the specific plugin the user wants. For most other users they can su to root and do this on their own computers. Not everyone is using Linux is a corporate world.
Quoted 9 lines
[...]
Quoted 6 lines
Quoted 2 lines
There are userland tools that deal with hundreds of file formats. Use the tool you need, rather than try to have the kernel do everything.
No, but if I wanted to have an encryption plugin active for some of my files or directories then why should I not be able to? I still want to edit, view and save my encrypted files.
Quoted 5 lines
Use an editor that knows about encrypted files. Decrypt/edit/encrypt if no other option (I'm sure emacs can be coerced to do that transparently ;-). I for one would be _way_ more confortable with my users doing that than them loading random modules into the kernel. Besides, if one of my users doesn't trust the system encryption programs, and prefers hers, she can be happy.
That just doesn't do it. I doubt there will be an option to save encrypted Word and Excel files and be able to open them in Abiword, StarOffice or OpenOffice unless the decryption/encryption is done on a lower level.
Also, eventually there may be a userland interface for loading certain modules without root access.?
Quoted 11 lines
Quoted 4 lines
Again, this was just an example of what could be done with plugins. It is not said that every conceivable plugin will be written, nor loaded per default. Even though plugins cannot today be dynamically used, they will be eventually. Reiser4 is still very young.
Modules of the kernel were supposed to have all those magic properties too, until there were nasty races... and it was _seriously_ considered to take them out. They stayed because they are root-only business, and (un)loading is rare. FS plugins are kernel modules, AFAIU, and are subject to the same problems.
Quoted 5 lines
Quoted 2 lines
Please separate your thoughts for specific plugins from those of the idea to have plugins at all.
If you can't find concrete uses for specific plugins, they are the proverbial solution searching for a problem.
To me there are fine and good uses for plugins. Everyone seem to think different though.
Sure many things can perhaps be done by linking and piping lots of programs, just to access some data. But that certainly is far from user friendly.
~S
Re: The argument for fs assistance in handling archives
In reply to Dave Kleikamp-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Dave Kleikamp wrote:
| On Thu, 2004-09-02 at 19:39, Spam wrote: | |> |> |> |>>On Fri, 3 Sep 2004, Spam wrote: |> |>>> Yes, some archive types can't be partially unzipped either. But my |>>> point is that it wouldn't be transparent to the application/user in |>>> the same way. |> |>>It doesnt matter whether it is transparent to the application. It can |>>be the application which implements the required level of |>>transparency. |> |>>User doesnt care what provides the transparency or how it's |>>implemented. |> |> Indeed. I hope I didn't say otherwise :). Just that I think it will |> be very difficult to have this transparency in all apps. | | | You're missing the point. We don't need transparency in all apps. You | can write an application to be as transparent as you want, but you don't | need every app to to understand every file format.
You don't need every app to understand every filesystem, either. Come to think of it, why would I want anything other than CD burning software and file browsers to understand a CD drive?
Let's all just wait -- quietly -- for the powers that be to work out whether it's feasable and sane to support it in all apps. Because wouldn't that be better?
|> Just |> thinking of "nano file.jpg/description.txt" or "ls |> file.tar/untar/*.doc". | | | I don't do much image editting, but I'm sure there are applications that | let you edit the description in a text file. You can even create a | script that extracts it, runs nano, and puts it back into the jpeg.
That is a PITA. Because you'll need to make more scripts. And still more.
Say you want to grep for a jpeg with a particular description. Say you want to copy the description from one jpeg into another.
Maybe you could make a general-purpose command, like "jpeg_run", which runs a command on the jpeg description, but it'd still be hackish, slow, and more to type and remember. Consider that you might have no idea what "jpeg_run" is called, but you can always do "ls file.jpeg/metas" to find out how to edit it.
| | This works for me: | tar -tf file.tar | grep '\.doc'
And then you need to run "tar -tf" a minute later, this time looking for *.xls. Maybe file.tar is actually a several gigabyte file.tar.bz2.
|> Sure in some environments like Gnome it could |> work, but it still doesn't for the rest of the flora of Linux |> programs. | | | Just choose the right program. tar groks tar files, not ls.
tar groks tar. bzip2 groks bz2. gzip groks gz. "mount -o loop" groks images. zip groks zip. rar groks rar. openoffice groks .sxw, unless staroffice does. xmms can read id3 info from mp3s. gcc compiles C.
If the file is an object, it's easier. You can ask the file what it is, and what you can do with it. And you can tell it to do certain standard things. You can do all of this while reading almost no documentation on what the file is. At the end of the day, it won't matter to you whether a file is zip, rar, or tar -- it's an archive, and you can extract it by copying files out of its /contents directory.
Your way (the traditional way) means you have to learn to use which program is the right program and how to use that program, and you'll have to remember it constantly.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTkIq3gHNmZLgCUhAQJaNRAAnp/o0Wr+SJK5tkgYFCuc+CKIP+eALJo/ 9wHnqK0nLdnji+vG0Czd9TUj1vWtoMrUichAwFoguMHHg3VeZGu61YwZoZ4idLNM QbZ+CuQdUygNmyT0byGMFemP+cSbyvff1PRMy2BlSHKW3gUhvnQyggLGVKxpMRWf VNnEHvkvJeA9PpEm6QGi1VRNp5bc0+Ocl4kO4CJk5ZYZ9D+BV6NwN/MZwqwlsu+Y RZsYYEa6mLiCnU4rEo0tEAvvwMdC0e/9s9TQMcmJbT6JnybkWIRFMrTS4pmSabKg uYGG9p1WrX8/V8WRNnaodlvx35gRPQj5S5SWDBoSWr999nmq31Y5RZ9QwbVj0d+U yB4yNu0NpvFBJfwg8nVIKUa+bhPLCkdY5w+GnlEYGweSN20FYOaLiqiPtXBqJm8a PP+8OCL35zy+1X7t/tq+JG/K91fYbECPR/qrAyHDXzNSuxdqidvBjfsEPBHOGwu6 RpsPoKyEvkzdXgauotWbVzWLt1ijGmYd/8Uk19OnmiloggViQhUWAJTIVGcvMNt8 Tnk8ESQMzGHbbVOIu3gB6DZD6P0IEEN4L6+gOZWoYqe2nvaYKvIL1My/i4BMU5Ml cxDkzzeq58iVK07Has8lwgFp0U9iD1LrbIT5p0ZJn/52EXqlX4dJCYSSNaHzUqE8 hcLG4aDELEI=
=Wcyv
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 13 lines · On Fri, 03 Sep 2004 19:13:31 -0500 David Masover <[email protected]> wrote:
| Just choose the right program. tar groks tar files, not ls.
tar groks tar. bzip2 groks bz2. gzip groks gz. "mount -o loop" groks images. zip groks zip. rar groks rar. openoffice groks .sxw, unless staroffice does. xmms can read id3 info from mp3s. gcc compiles C.
If the file is an object, it's easier. You can ask the file what it is, and what you can do with it. And you can tell it to do certain standard things. You can do all of this while reading almost no documentation on what the file is. At the end of the day, it won't matter to you whether a file is zip, rar, or tar -- it's an archive, and you can extract it by copying files out of its /contents directory.
Quoted 3 lines
Your way (the traditional way) means you have to learn to use which program is the right program and how to use that program, and you'll have to remember it constantly.
Just a short input from someone listening quite a while to this thread:
I think your approach to the problem leads actually nowhere. The reason for this is implicit in your own explanation. _Currently_ you talk of archives, but as soon as you got that, what's next?
Your idea needs abstraction, then you see the problem more clearly:
An archive is only some sort of file type, just as (you already named) mp3 or .sxw is another type. If you really want to do something _generally_ useful you should think of a method to parse and use all kinds of filetypes and create an interface for that.
And one thing is clear: as there are numerous different types you cannot pull all that code inside the kernel. Obviously there should be a way some application can install a hook, a helper, a plugin or whatever to provide extended functionality on its special filetypes. If you don't want to use tar, you don't need the plugin either.
If you want tar, you should (as a user) be able to install the "fs-plugin" (just a name, do not shoot me for it) together with tar as an application. You get the idea? Obviously this must be possible during runtime, everything else is senseless.
So, please, if you go on investigating this whole stuff do it in a more general way, because "tar inside kernel" is not really what your idea is all about, no? And one last obvious thing: if one doesn't want this kind of file-handling, he should not need it. It has to be an add-on functionality, not a replacement.
Regards,
Stephan
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Stephan von Krawczynski wrote:
| On Fri, 03 Sep 2004 19:13:31 -0500 | David Masover <[email protected]> wrote: | | |>| Just choose the right program. tar groks tar files, not ls. |> |>tar groks tar. bzip2 groks bz2. gzip groks gz. "mount -o loop" groks |>images. zip groks zip. rar groks rar. openoffice groks .sxw, unless |>staroffice does. xmms can read id3 info from mp3s. gcc compiles C. |> |>If the file is an object, it's easier. You can ask the file what it is, |>and what you can do with it. And you can tell it to do certain standard |>things. You can do all of this while reading almost no documentation on |>what the file is. At the end of the day, it won't matter to you whether |>a file is zip, rar, or tar -- it's an archive, and you can extract it by |>copying files out of its /contents directory. |> |>Your way (the traditional way) means you have to learn to use which |>program is the right program and how to use that program, and you'll |>have to remember it constantly. | | | Just a short input from someone listening quite a while to this thread: | I think your approach to the problem leads actually nowhere. The
reason for
| this is implicit in your own explanation. _Currently_ you talk of archives, but
| as soon as you got that, what's next?
Anything sane. Currently I talk of archives, because I need examples, and archives have been a good example for awhile of something that can gain things from plugins which they cannot get anywhere else, which have implications immediately after the archive plugin is wrote, without requiring any application support.
| Your idea needs abstraction, then you see the problem more clearly:
| An archive is only some sort of file type, just as (you already named) mp3 or
| .sxw is another type. If you really want to do something _generally_ useful you
| should think of a method to parse and use all kinds of filetypes and create an
| interface for that.
Fine. It'd certainly be nice to use vim to edit an id3 tag, or a text portion of a .sxw document (openoffice is HUGE compared to vim).
| And one thing is clear: as there are numerous different types you cannot pull
| all that code inside the kernel. Obviously there should be a way some
I never said it should all go in the kernel. I repeatedly said that most of it should not -- all that goes in the kernel is the most generic interface that is sane. Obviously an archive plugin is different than a compression plugin, but once you've got the archive plugin, the rest (individual format support) can go in userland.
| application can install a hook, a helper, a plugin or whatever to provide | extended functionality on its special filetypes. If you don't want to use tar,
| you don't need the plugin either.
So you disable it at compile time. I don't want to use XFS, and I certainly don't want to use ATI's video drivers. The latter I choose not to download, and the former I choose not to activate.
| If you want tar, you should (as a user) be able to install the "fs-plugin"
If by "as a user" you mean "not root", I have to disagree. I think it'd have security implications. It'd be great if I'm wrong.
| (just a name, do not shoot me for it) together with tar as an application. You
| get the idea? Obviously this must be possible during runtime, everything else
| is senseless.
I would love for this same attitude of "runtime or bust" to be more common. I would love for there to be a generic way to write a userland fs plugin, just as there's a generic way to write a userland filesystem (lufs). Just understand that some plugins will be in userland, and some will be in the kernel.
And, in fact, the part of the archive plugin that knows about tar/zip/rar/whatever goes in userspace. But the part that does caching probably goes in kernel space, even if it becomes a more generic "caching plugin".
| So, please, if you go on investigating this whole stuff do it in a more general
| way, because "tar inside kernel" is not really what your idea is all about, no?
Absolutely not.
| And one last obvious thing: if one doesn't want this kind of file-handling, he
| should not need it. It has to be an add-on functionality, not a replacement.
For awhile, yes. But I can imagine that sometime in the future, an app would need this kind of functionality. And why not? Is it so much worse for an app to insist on "mp3 metadata plugin" than to insist on "libid3"?
In any case, look at the current implementation. I can do
echo 750 > file/metas/rwx
but I can also use chmod, which is currently much more efficient. I think Hans wants to eventually create an efficient interface to access lots of small files at once (or maybe this is done?) -- and his dream is to see things like chmod replaced with a call to this interface. But at best, chmod will become a library call sometime years from now. It will never go away entirely.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQToI+3gHNmZLgCUhAQKEWg/9F0FZGxB2yXoo9l5UKI7Fvh19RlnpJvZE D3DajNqxXZ5smYIba7QxATEwMLWNMwOdiJTDIcQ7jylbD3hTkp0uFQLXcS5FeWt5 yxA0oDYEHsW14aQov4pHGplHVvCHSkm6vJn/zU91QdJAZVYnH4na5NFuWsAIoUWi 5vigYTefavabFv0/RnyPCN6zFfn3hrjOnKqSELUXXleG/fV+XJVw3Nyg4DSe1hK+ xmXHFWkGC832tH052EH/CFlE17/K05QdpfBKYNzCFA5dvTtCs11OQZSggtGVipmn cSycyDHtJraMdF78Qa8LZzjGiZbT/3befM6wrnekcwhjxCqok8CSbOvbVh2ENtiu klNtHeQEEtm+4MhbEEkdiiuWaU7EHIFJFGYC6LJE0B4DAKDdi4TM9QbqxLW8HghQ WUJfMh8H2FtqotjMsdDW0OCmSuzh0B4ZZ9EXvSMIttXBJ7T2+q442hm4PAFQWXSO FPCAiUgGZ+ZCbyZI1/pjM+CGblMTCfCBOm8UI3MSjTTI7cVzWFrQHJSAUhCPIsxA 9RZlEvmg+ElbnGrJkpDggpToDFN5uHXj41r4CZBnXgjfUdA0jmHnWpzO29Smfu7i cJPtN+LzR1Se+fz4IvYamWd1p/Mi3vQACPt4r3IGVKa27tYXzDw+CNa+CliHnczt lAhISMnhoVo=
=7rV6
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to SpamQuoted 1 line · Spam
Indeed. I hope I didn't say otherwise :).
Sure.
Quoted 6 lines
Just that I think it will
be very difficult to have this transparency in all apps. Just thinking of "nanofile.jpg/description.txt" or "ls file.tar/untar/*.doc". Sure in some environments like Gnome it could work, but it still doesn't for the rest of the flora of Linux programs.
"will it be transparent for all apps?", whether that's worth doing depends on the technical implications. Thankfully we have Al and Linus to make the judgement call on that ;)
Personally, I think that if GNOME can provide transparency for GNOME users, I think that's probably enough - unless there are literally no issues in adding some kind of VFS support.
The nano / ls /tar user is likely a very different user to the GNOME user. That user is also likely to appreciate the problems with backups and such more.
Anyway, userspace transparency is sufficient for most classes of users. Only reason to provide some kernel support is if it makes sense ("but not all apps can use GNOME transparency" not being one of those reasons).
regards,
Signature · 3 lines
Paul Jakma [email protected] [email protected] Key ID: 64A2FF6A Fortune: Beat your son every day; you may not know why, but he will.
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Paul Jakma wrote:
| On Fri, 3 Sep 2004, Spam wrote: | |> Indeed. I hope I didn't say otherwise :). | | | Sure. | |> Just that I think it will |> be very difficult to have this transparency in all apps. Just |> thinking of "nano file.jpg/description.txt" or "ls |> file.tar/untar/*.doc". Sure in some environments like Gnome it could |> work, but it still doesn't for the rest of the flora of Linux |> programs. | | | "will it be transparent for all apps?", whether that's worth doing | depends on the technical implications. Thankfully we have Al and Linus | to make the judgement call on that ;)
So far, the technical implications are mainly "does it create a serious issue in the distant future" or "does some exotic new feature that doesn't exist yet cause problems" and not "it's currently broken".
Right now, I can edit a file's permissions, transparently, from any app, and I haven't noticed any stability issues at all.
| Personally, I think that if GNOME can provide transparency for GNOME | users, I think that's probably enough - unless there are literally no | issues in adding some kind of VFS support.
Only it can't. Especially if it's a typical GNOME user, who gets used to having their files encrypted in foo.tar.pgp, say. Works in abiword, works in gedit, breaks in OpenOffice. Not good.
| The nano / ls /tar user is likely a very different user to the GNOME | user. That user is also likely to appreciate the problems with backups | and such more.
I'm a vim/ls/tar user. And I believe the problems with backups to be very minor. There are other issues that I don't understand as thoroughly, so I'll leave them to Al and Linus.
| Anyway, userspace transparency is sufficient for most classes of users.
| Only reason to provide some kernel support is if it makes sense ("but
| not all apps can use GNOME transparency" not being one of those reasons).Can you justify why "not all apps can use GNOME transparency" is not a valid reason? Would you still think so if GNOME transparency was the only way to read isofs?
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTkCL3gHNmZLgCUhAQJVSxAAiFd+e/Q0VjAvPaQL1IDwA56SP7LF2M1v jnk5wehjvicPEfSn2HpWik7Pr55K2/7dUn6mEJO4N43x/zmfyiA2wXLibfWJtD3X VgtFMgbOeKu+WobfdnvlBPAKYoZ+MYsC2jyDYwOHguJbR9epmFtHv0mMhmD60s+B 3JDLaYXwPjmjs7zezy2B/Xc3mgm060VgDZRVOnrYZLzHAi3dLvZLzGjORf5kBZtd NtsiPUZw3oIqORBHqfWDWUUi4irfzklZkiuYp44RNFZs0z0xvPrUJ1klXiJpbQ9t HTkz9qRe0sxjRYfLcJzWxfahaNN+2SXyR5FFqkpzfUfyNuoOUMZDtin+ZM83qyVc J/zRCZvWQOaZUtgas9KCCYnbwgCFLcDEE0xLLDRMGAMKeHOwCImU55ChTC/0HTas Q7Bd0df456dAv+ktbdMRaznWeNOpW5rJWXtxNeWnncTYKG0ogXct9mD0+BEIssaK qaEYUousp6ZRvOSlCCNCF5hOvfEuZLuAZe2Q8zKH39B4Hg7BMnmIOw6VPLmlLKds pGNuIKfKjyzkxcdHwqGp36e27wizyKtCbZ58Zd0x1wnf47gkm8kgkY38tb0ct7Xf Ch+s3+zLsOtVnnipnlGhIPEtxYv78DiqhGUc+pdZisi5VlfA4PTXZAAjoICeXA/H EqfVU6yYsu8=
=ggxZ
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to SpamSalut,
Quoted 1 line · Spam
"nano
file.jpg/description.txt"
Is that supposed to do steganography?
Tonnerre
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 9 lines
Salut,
Quoted 1 line · Spam
"nano
file.jpg/description.txt"Is that supposed to do steganography?
No, not at all. I already use similar things in Windows this is why I suggested it here. Having a separate file for descriptions are not as convenient. Especially if you have a few hundred JPEGs and I want to quickly view some properties of the one you have just looked at.
~S
Quoted 1 line
Tonnerre
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Alan CoxQuoted 10 lines · Alan Cox
Quoted 5 lines · Linus Torvalds
and quite frankly, I think you can do the above pretty much totally in user space with a small library and a daemon (in fact, ignoring security issues you probably don't even need the daemon). And if you can prototype it like that, and people actually find it useful, I suspect kernel support for better performance might be possible.
Gnome already supports this in the gnome-vfs2 layer. "MC" has supported it since the late 1990's.
And nobody has asked for kernel support that I know of.
So either "it just works" in user space, or people haven't figured out the kernel could help them. Or decided it's not worth it, exactly because they'd still have to support systems/filesystems that can't be converted.
Linus
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 7 lines · Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 5 lines
and quite frankly, I think you can do the above pretty much totally in user space with a small library and a daemon (in fact, ignoring security issues you probably don't even need the daemon). And if you can prototype it like that, and people actually find it useful, I suspect kernel support for better performance might be possible.
Quoted 8 lines
Quoted 2 lines
Gnome already supports this in the gnome-vfs2 layer. "MC" has supported it since the late 1990's.
And nobody has asked for kernel support that I know of.
So either "it just works" in user space, or people haven't figured out the kernel could help them. Or decided it's not worth it, exactly because they'd still have to support systems/filesystems that can't be converted.
http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto a kernel filesystem. The code is horrible, but it shows that it can be done.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 4 lines · Christoph Hellwig
http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto a kernel filesystem. The code is horrible, but it shows that it can be done.
I do like the setup where the extended features are done as a "view" on top of some other filesystem, so that you can choose to _either_ access the raw (and supposedly stable, simply by virtue of simplicity) or the "fancy" interface. Without having to reformat the disk to a filesystem you don't trust, or you have other reasons you can't use (disk sharing with other systems, whatever).
It doesn't have to be "user", btw, in the sense that a lot of the normal code could be in kernel mode. Same way as Tux handling all the regular static requests entirely in kernel mode, but having the ability for calling down to apache..
Linus
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 13 lines · Linus Torvalds
Quoted 4 lines · Christoph Hellwig
http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto a kernel filesystem. The code is horrible, but it shows that it can be done.
I do like the setup where the extended features are done as a "view" on top of some other filesystem, so that you can choose to _either_ access the raw (and supposedly stable, simply by virtue of simplicity) or the "fancy" interface. Without having to reformat the disk to a filesystem you don't trust, or you have other reasons you can't use (disk sharing with other systems, whatever).
Quoted 4 lines
It doesn't have to be "user", btw, in the sense that a lot of the normal code could be in kernel mode. Same way as Tux handling all the regular static requests entirely in kernel mode, but having the ability for calling down to apache..
Stackable file systems are a good fit for doing things like this. They layer on top of an existing directory and intercept all of the vfs_* calls. You can do lots of neat things like encryption versioning, compression, and more.
At Stony Brook's Filesystems and Storage Laboratory
(http://www.fsl.cs.sunysb.edu), we maintain a toolkit (FiST) that makes developing these file systems easier by providing stackable file system templates for Linux and FreeBSD.
There are papers and the FiST toolkit available at
http://www.filesystems.org/
Charles
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 5 lines · Charles P. Wright
Stackable file systems are a good fit for doing things like this. They layer on top of an existing directory and intercept all of the vfs_* calls. You can do lots of neat things like encryption versioning, compression, and more.
Stackable filesystems are really hard, and ambiguous on where things go, and what happens when you remove an entry that hides another one. Sure, you can make up any amount of semantics, but the fact is, all the semantics _are_ going to be pretty arbitrary, and fairly complex.
I'd frankly _much_ prefer not having stackable filesystems supported by the VFS layer.
That said, I think you can make stackable filesystems as a _separate_ filesystem (aka "view" or whatever), which is slightly different from supporting them in the VFS layer itself: make it a filesystem of its own, rather than supporting it on a VFS layer as a merge of two different filesystems. That way you still have totally arbitrary semantics for how the stacking works, but at least it's not codified by the VFS layer - it's codified by nothing more special than one more random filesystem implementation, and you could in fact have _different_ semantics depending on what the user wants.
Same end result, different "mindset". I think this is what FiST actually does, but I could be wrong.
What's the difference? If the VFS layer supports it (like the VFS supports regular "opaque" overmounting), you'd just do
mount -o stack /dir /otherdir
the same way you do "bind" mounts. In contrast, if you do it as a third-party "view" filesystem, it would be more like
mount -t stackfs -o under=/usr,over=/usr/local /mnt
where /usr and /usr/local would be the same old underlying filesystems, and you'd see a totally new filesystem at /mnt.
Btw, this kind of non-VFS-supported filesystem would not necessarily have coherency guarantees: if you changed /usr or /usr/local directly (rather than through the stackable /mnt), you could confuse stackfs. Kind of the same way you can confuse a NFSv2 client by operating directly on the server (ie I'm not saying "confuse" in a "blow up the kernel" kind of way, but in a "would return -ESTALE or similar").
Linus
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 11 lines · Linus Torvalds
Quoted 5 lines · Charles P. Wright
Stackable file systems are a good fit for doing things like this. They layer on top of an existing directory and intercept all of the vfs_* calls. You can do lots of neat things like encryption versioning, compression, and more.
Stackable filesystems are really hard, and ambiguous on where things go, and what happens when you remove an entry that hides another one. Sure, you can make up any amount of semantics, but the fact is, all the semantics _are_ going to be pretty arbitrary, and fairly complex.
I think by stackable you are thinking what we call a Union.
By stackable, I mean that you just layer some functionality on an existing file system, without having to modify it.
I agree that Unioning semantics are difficult. We've actually implemented a stackable unioning file system, and dealing with an arbitrary number of branches, especially in the face of some operations failing in the middle, is quite complex. Rename is especially hard.
I think for the type of problem, where you have the file system jump down into archives or do other like-minded operations, you will also end up with having to make some arbitrary decisions with complex semantics.
Quoted 13 lines
I'd frankly _much_ prefer not having stackable filesystems supported by the VFS layer.
That said, I think you can make stackable filesystems as a _separate_ filesystem (aka "view" or whatever), which is slightly different from supporting them in the VFS layer itself: make it a filesystem of its own, rather than supporting it on a VFS layer as a merge of two different filesystems. That way you still have totally arbitrary semantics for how the stacking works, but at least it's not codified by the VFS layer - it's codified by nothing more special than one more random filesystem implementation, and you could in fact have _different_ semantics depending on what the user wants.
Quoted 2 lines
Same end result, different "mindset". I think this is what FiST actually does, but I could be wrong.
Yes, to the VFS FiST behaves as a file system. To the file system it is stacked on, it behaves like the VFS.
Quoted 5 lines
What's the difference? If the VFS layer supports it (like the VFS supports regular "opaque" overmounting), you'd just do
mount -o stack /dir /otherdir
Quoted 7 lines
the same way you do "bind" mounts. In contrast, if you do it as a third-party "view" filesystem, it would be more like
mount -t stackfs -o under=/usr,over=/usr/local /mnt
where /usr and
/usr/localwould be the same old underlying filesystems, and you'd see a totally new filesystem at /mnt.
That is very close to how a FiST file system works. However, I think you are more closely describing a Union mount that would provide a merged view of "/user" and "/usr/local" (with /usr/local taking precedence over /user).
More concretely, something like:
mount -t gzipfs -o dir=/compressed none /mnt
Would transparently compress data written to /mnt/a, and physically store it in /compressed/a. When reading /mnt/a, the data is decompressed and returned to the user-space.
This a real example that is actually included as part of FiST.
Quoted 6 lines
Btw, this kind of non-VFS-supported filesystem would not necessarily have coherency guarantees: if you changed /usr or
/usr/localdirectly (rather than through the stackable /mnt), you could confuse stackfs. Kind of the same way you can confuse a NFSv2 client by operating directly on the server (ie I'm not saying "confuse" in a "blow up the kernel" kind of way, but in a "would return-ESTALEor similar").
With a simple stacking file system, one way to avoid this is an overlay mount.
mount -t cryptfs -o dir=/home/cpwright none /home/cpwright
There are some disadvantages to hiding the lower level (e.g., with Cryptfs you want to backup the encrypted data, not the decrypted data).
Charles
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Linus TorvaldsIn message <[email protected]>, Linus Torvalds writes:
[...]
Quoted 18 lines
That said, I think you can make stackable filesystems as a _separate_ filesystem (aka "view" or whatever), which is slightly different from supporting them in the VFS layer itself: make it a filesystem of its own, rather than supporting it on a VFS layer as a merge of two different filesystems. That way you still have totally arbitrary semantics for how the stacking works, but at least it's not codified by the VFS layer - it's codified by nothing more special than one more random filesystem implementation, and you could in fact have _different_ semantics depending on what the user wants.
Same end result, different "mindset". I think this is what FiST actually does, but I could be wrong.
What's the difference? If the VFS layer supports it (like the VFS supports regular "opaque" overmounting), you'd just do
mount -o stack /dir /otherdir
Quoted 7 lines
the same way you do "bind" mounts. In contrast, if you do it as a third-party "view" filesystem, it would be more like
mount -t stackfs -o under=/usr,over=/usr/local /mnt
where /usr and
/usr/localwould be the same old underlying filesystems, and you'd see a totally new filesystem at /mnt.
[...]
Yes, that's exactly how our stacked f/s's work: not touching or modifying VFS or lower f/s's, but of course we use the VFS the way other f/s's use it (calling vfs_* functions and such). Each of our stackable f/s is a separate f/s in its own.
And they're a nice way to augment the functionality of any other f/s w/o the hassles of changing a lower-level f/s format or the VFS. We recently did an example f/s that adds EA+ACL functionality on top of anything else, using the lower file system's EA/ACL support if it's there, and for f/s's that don't support EA/ACL natively, we support the ea/acl f/s calls and store the actual ea/acl data in a small database. We've even demonstrated that it worked on v/fat (yes, we know, vfat+ACLs is an an oxymoron :-)
We also found out over the years that our stackable file systems are an interesting way to prototype features that might eventually move up to the VFS itself; this is b/c each stacked f/s has to appear to the lower f/s as the VFS itself, essentially emulating most of the VFS's functionality while at the same time appearing to the VFS as a lower-level f/s. It's not easy maintaining this split personality. :-)
Erez.
Re: The argument for fs assistance in handling archives
In reply to Linus TorvaldsQuoted 8 lines · Linus Torvalds
Quoted 4 lines · Christoph Hellwig
http://oss.oracle.com/projects/userfs/ has code that clues gnomevfs onto a kernel filesystem. The code is horrible, but it shows that it can be done.
Quoted 6 lines
I do like the setup where the extended features are done as a "view" on top of some other filesystem, so that you can choose to _either_ access the raw (and supposedly stable, simply by virtue of simplicity) or the "fancy" interface. Without having to reformat the disk to a filesystem you don't trust, or you have other reasons you can't use (disk sharing with other systems, whatever).
It'd be something similar to what clearcase does (not that I like clearcase, I hate it with a passion for other reasons!)
On such a system, one would have multiple virtual views mounted (by root) under:
/view/tar, /view/dpkg, /view/rpm, etc.
for every regular file /home/joe/blah.tar
the path /view/tar/home/joe/blah.tar/ is a directory where member of the archives directly accessible.
old tools continue work as is. new tools can take a look on virtual views for virtual access.
Not sure how such a system would work with the dentry cache.
Signature · 1 line
Linh Dang
Re: The argument for fs assistance in handling archives
Quoted 7 lines · Linh Dang
On such a system, one would have multiple virtual views mounted (by root) under:
/view/tar, /view/dpkg, /view/rpm, etc.for every regular file
/home/joe/blah.tar
Quoted 7 lines
the path /view/tar/home/joe/blah.tar/ is a directory where member of the archives directly accessible.
old tools continue work as is. new tools can take a look on virtual views for virtual access.
Not sure how such a system would work with the dentry cache.
How does it cope with 'view withint view'? Say you have a .zipped ISO. Can you do
cd /view/iso/view/zip/home/joe/mycd.zip/mycd.iso/
?
Signature · 7 lines
Giuseppe "Oblomov" Bilotta
Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
(Roger Waters)Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Linus TorvaldsQuoted 4 lines · Linus Torvalds
Quoted 2 lines
Gnome already supports this in the gnome-vfs2 layer. "MC" has supported it since the late 1990's.
And nobody has asked for kernel support that I know of.
I asked our desktop people. They want something like inotify because dontify doesn't cut it. They have zero interest in the multiple streams and hiding icons in streams type stuff.
Alan
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 3 lines · Alan Cox
I asked our desktop people. They want something like inotify because dontify doesn't cut it.
Well, dnotify() really _is_ inotify(), since it does actually work on inodes, not dentries.
I think what they are really complaining about is that dnotify() only notifies the _directory_ when a file is changed, and they'd like it to notify the file itself too. Which is a one-liner, really.
Does the following make sense? (Totally untested, use-at-your-own-risk, I've-never-actually-used-dnotify-in-user-space, whatever).
Linus
===== fs/dnotify.c 1.17 vs edited =====
fs/dnotify.c +2 -0
@@ -160,6 +160,8 @@if (!dir_notify_enable)return ; +__inode_dir_notify (dentry->d_inode, event); +spin_lock (&dentry->d_lock); parent = dentry->d_parent;if (parent->d_inode->i_dnotify_mask & event) {
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 3 lines · On Thu, 2 Sep 2004 13:22:41 -0700 (PDT) Linus Torvalds <[email protected]>…
Well,
dnotify()really _is_inotify(), since it does actually work on inodes, not dentries.
The "d" stands for directory not dentry :-)
Quoted 3 lines
I think what they are really complaining about is that
dnotify()only notifies the _directory_ when a file is changed, and they'd like it to notify the file itself too. Which is a one-liner, really.
I don't think so, since this notify will only happen if the process has registered for the notification and there is no way to register unless the file is a directory ...
Quoted 2 lines
Does the following make sense? (Totally untested, use-at-your-own-risk, I've-never-actually-used-dnotify-in-user-space, whatever).
I had intended to extend dnotify to do file notifies, but I think the real killer is needing the keep the file open that you want to be notified about when you want to be notified about lots of files ...
I think that is what inotify was trying to fix (but I haven't had a chance to look at it recently). It reminds me of omirr that we had many years ago - I wonder what happened to it?
Signature · 3 lines
Cheers, Stephen Rothwell [email protected] http://www.canb.auug.org.au/~sfr/
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Alan CoxQuoted 9 lines · Alan Cox
Quoted 4 lines · Linus Torvalds
Quoted 2 lines
Gnome already supports this in the gnome-vfs2 layer. "MC" has supported it since the late 1990's.
And nobody has asked for kernel support that I know of.
I asked our desktop people. They want something like inotify because dontify doesn't cut it. They have zero interest in the multiple streams and hiding icons in streams type stuff.
It also depends on who you ask. I can't take a lot of the mainstream X folks serious since they are still using integer math as parameters to half broken drawing primitives and barely discovered things like OpenGL. Their attitude doesn't treat these things as first class citizens in what ever software system they create. They also haven't create a modern and highly dynamic structured document system that's in wide use yet, so this problem space hasn't really been pushed as hard as other much more dynamic systems. And the advent of XML (basically a primitive and flat model of what Hans is doing) for .NET style systems are going to push these systems into those areas in new and unique ways. (Actually retro Smalltalk-ish)
It seems that many of the original ideas about "why" GUI systems exist have been lost to older commericial interests (Microsoft Win32) and that has wiped out the fundamental classic computer science backing this from history. This simple "MP3 metadata" stuff is a very superficial example of how something like this is used.
The problems are fundamentally about data representation in a manner so simple that its "expressive power" (Hans here) can extend itself to even the dorkiest of shell scripts. To have that power immediately available as network/local objects and to have their relationships clearly defined is a very powerful manner to build software systems.
Unix folks tend to forget that since they either have never done this kind of programming or never understood why this existed in the first place. It's about a top-down methodology effecting the entire design of the software system, not just purity Unix. If it can be integrate smoothly into the system, then it should IMO.
The folks against this system forget about how important the context of all this is work is set in... The mindset is fundamentally different and I'm quite sick of hearing "It's not Unix" over and over again. And notion of Linux being marginalize to a minority OS over this stuff is just plain crazy.
bill
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 2 lines · Bill Huey
It also depends on who you ask. I can't take a lot of the mainstream X folks serious since they are still using integer math as parameters
The X folks know what they are doing. Modern X has a complete compositing model. Modern X has a superb font handling system. Nobody broke anything along the way. The new API's can be mixed with the old, there are good fallbacks for old servers.
In fact they are so good at it that most people don't notice beyond the fact their UI looks better than before.
That is how you do change *right*
Quoted 2 lines
more dynamic systems. And the advent of XML (basically a primitive and flat model of what Hans is doing) for .NET style systems are going to
I see you don't really get XML either. XML is just an encoding. Its larger and prettier than ASN.1 and easier to hack about with perl. You can do the same thing with lisp lists for that matter.
Quoted 4 lines
have been lost to older commericial interests (Microsoft Win32) and that has wiped out the fundamental classic computer science backing this from history. This simple "MP3 metadata" stuff is a very superficial example of how something like this is used.
The trouble with computer science is that most of it sucks in the real world. We don't write our OS's in Standard ML, we don't implement some of the provably secure capability computing models. At the end of the day they are neat, elegant and useless to real people.
Quoted 5 lines
Unix folks tend to forget that since they either have never done this kind of programming or never understood why this existed in the first place. It's about a top-down methodology effecting the entire design of the software system, not just purity Unix. If it can be integrate smoothly into the system, then it should IMO.
The Unix world succeeded because Unix (at least in v7 days) was the other way around to every other grungy OS on the planet. It had only thing things it needed. I've used some of the grungy crawly horrors that were its rivals and there is a reason they don't exist any more.
I would sum up the essence of the unix kernel side as
- Does only what it must do
- "Makes the usual easy makes the unusual possible"
- Has an API that is small enough for developers to learn easily (an API so good every other OS promptly ripped it off) People forget the worlds of SYS$QIO, RMS, FCB's and the like
Its worked remarkably well for a very very long time, and most of the nasties have come from people trying to break that model or not understanding it.
Alan
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 8 lines · Alan Cox
Quoted 2 lines · Bill Huey
It also depends on who you ask. I can't take a lot of the mainstream X folks serious since they are still using integer math as parameters
The X folks know what they are doing. Modern X has a complete compositing model. Modern X has a superb font handling system. Nobody broke anything along the way. The new API's can be mixed with the old, there are good fallbacks for old servers.
Keith Packard knows what he's doing. The rest that don't follow his lead on these issues are useless or detracting and redirecting resources away from critical problems that have been solved under other, more mature, operating systems, specifically Apple's OS X.
Notice, that you didn't deflect the claim that folks are still using integer math to draw things. This is critical...
Quoted 4 lines
In fact they are so good at it that most people don't notice beyond the fact their UI looks better than before.
That is how you do change *right*
It's one way, but it's not top-level redesign heavily enough that you still have a substandard rendering system. Notice that were wasn't a mention of device independent rendering, WYSIWYG (what you get is what you see) in their rasterization model, nor was there any mention of how a scalable unified rasterization model used in an application. OpenStep ruled during it's time and rendered and generate better PostScript than Macintosh applications because of this unified rasterization model. Split development and the lack of "connecting the dots" across systems is one of the reasons why X has such poor app development. It takes way more time to do things in this environment than in other system.
What folks X land are missing here is a notion of design verses piecewise hacks to half emulate a critical function. The fault in this matter is really spread out to multipule camps not doing their homework regarding these matters.
Also, there's no mention of making OpenGL a first class citizen in this system so that all compositing is hardware accelerate. What OS has this ? You guess it Apple's OS X has this with full transparency and bitmap backstore for region invalidating and redrawing. X currently has none of this stuff. You could say that this is planned in the future, but there has been no push for doing this in this aggressive a manner which is a leadership failure of that community as a whole.
Also, the notion of color space, not just RGB, but CMYK is a first class citizen in Display Postscript. X is also missing this. These aren't things that you can gradually add in. It's about looking at the system from the top-down and solving these problems using other systems that.
X is at best piecewise and substandard to just about any modern rendering system that I've used.
Quoted 6 lines
Quoted 2 lines
more dynamic systems. And the advent of XML (basically a primitive and flat model of what Hans is doing) for .NET style systems are going to
I see you don't really get XML either. XML is just an encoding. Its larger and prettier than ASN.1 and easier to hack about with perl. You can do the same thing with lisp lists for that matter.
I get it fine. What you don't get is something called Model View Controller in that .NET applications have the ability to take that serialized structure, apart of their entire object store/serialization system and transform it into many views. The same widget data can be used not only to build HTML for a web page, but also a standard component driven GUI system a native windowing kit. This exploits polymorphism and how OOP should have been done in the first place. This use of it is very important and obviously useful in this case.
In MVC, all data has relationships and XML s the kind of DB format that these systems use for object serialization. XML is the storage layer in these systems. It's used for every object, remote or local.
This is why something like Reiser FS 4 with it's persistent object store capability may have a critical influence on how these systems are built. These systems are returning nothing but structured storage in the form of XML analougs.
Your Perl/Python, what ever example here doesn't cut it and explicate how XML based system are being generically used in this sense. It's too technically shallow for this context.
Quoted 9 lines
Quoted 4 lines
have been lost to older commericial interests (Microsoft Win32) and that has wiped out the fundamental classic computer science backing this from history. This simple "MP3 metadata" stuff is a very superficial example of how something like this is used.
The trouble with computer science is that most of it sucks in the real world. We don't write our OS's in Standard ML, we don't implement some of the provably secure capability computing models. At the end of the day they are neat, elegant and useless to real people.
Java and .NET are just those things that you have been talking about. As these systems become more and more prevalent (they will) they're influence is just the beginning. These system have pretty much happened overnight and can't be ignored by kernel folks.
Quoted 10 lines
Quoted 5 lines
Unix folks tend to forget that since they either have never done this kind of programming or never understood why this existed in the first place. It's about a top-down methodology effecting the entire design of the software system, not just purity Unix. If it can be integrate smoothly into the system, then it should IMO.
The Unix world succeeded because Unix (at least in v7 days) was the other way around to every other grungy OS on the planet. It had only thing things it needed. I've used some of the grungy crawly horrors that were its rivals and there is a reason they don't exist any more.
Look, I agree with you for the most part about Unix, but these issues have to be reexamined again after some period of time has elapsed.
I can only guess that some of these system are horribly implemented minimally, but I can't comment about their original intentions since I've personally never used them.
Quoted 10 lines
I would sum up the essence of the unix kernel side as
- Does only what it must do
- "Makes the usual easy makes the unusual possible"
- Has an API that is small enough for developers to learn easily (an API so good every other OS promptly ripped it off) People forget the worlds of SYS$QIO, RMS, FCB's and the likeIts worked remarkably well for a very very long time, and most of the nasties have come from people trying to break that model or not understanding it.
This is a difficult topic, but I don't how folks can dimiss something like Reiser FS semantics and the importance of systems like this for future software design across all application boundaries.
There has to be a way of getting this stuff in and keeping the traditional semantics in place.
I was thinking about have a dual mounted file system, one with traditional Unix semantics with read-only permissions, but then having it also being mounted in a special namespace directory structure (something like /proc) that would reveal other streams for "stream aware" applications with read/write permissions.
It might solve both problems in this case. Don't know.
I've been on vacation for the last couple of days at Burning Man, which is why this response has been delayed. :)
bill
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Linus TorvaldsQuoted 14 lines · Alan Cox
Quoted 8 lines · Linus Torvalds
Quoted 5 lines
and quite frankly, I think you can do the above pretty much totally in user space with a small library and a daemon (in fact, ignoring security issues you probably don't even need the daemon). And if you can prototype it like that, and people actually find it useful, I suspect kernel support for better performance might be possible.
Quoted 4 lines
Quoted 2 lines
Gnome already supports this in the gnome-vfs2 layer. "MC" has supported it since the late 1990's.
And nobody has asked for kernel support that I know of.
Actually. Doesn't matter if it is in kernel or not for the users as long as it works.
The problem is that I do not see either Gnome or KDE to ever get along to form one standard that everyone will use. Their libraries are huge and memory hogging which so many Linux users just do not like. What if a user doesn't want KDE or Gnome? Would all files created with either be broken?
I doubt that something like file streams and meta-data can successfully be implemented purely in user-space and get the same support (ie be used by many programs) if this change doesn't come from the kernel. I just do not see it happen.
Quoted 5 lines
So either "it just works" in user space, or people haven't figured out the kernel could help them. Or decided it's not worth it, exactly because they'd still have to support systems/filesystems that can't be converted.
Linus
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Spam wrote:
[...]
| I doubt that something like file streams and meta-data can | successfully be implemented purely in user-space and get the same | support (ie be used by many programs) if this change doesn't come | from the kernel. I just do not see it happen.
The issue is not "many programs". The issue is "all programs".
Even if the political issues were solved -- even if Linus said to us all "Thou shalt use this library or suffer my wrath!" -- it'd have to be everywhere. Bash. Perl. Make. Gcc. Vim.
And btw, if it was political, you'd get no sympathy here. ("Oh no, everyone's using their own game engine! We need to put the doom3 engine in the kernel, now!") No, _people_ solve political problems. Kernels don't.
Kernel support automatically adds support for a lot of features without patching a thing.
There should be an interface in the filesystem. And for certain things, uservfs will be incredibly slower than reiser4 as that interface. (Remember Linus' point about TUX and Apache.)
I'll say this again: most of it -- all but the bare interface stuff -- should be in userspace. In fact, let's all add this to our signature so no one brings it up again. ("Oh no, they want to put tar support in the kernel!" no, we don't.)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTe4BXgHNmZLgCUhAQKH1g//WXaL5xdLpX37TdgFhXidiRJGe/ojehj3 CZ7kseI9GkOBSxHt/yb5/xC6r+XT7JLlvJZybT7HLIRIxGp+WQHHBOD/xezWC+eX OyRaOlkZ7o9HRQKhKNRIAwI4jgftpLhFUhePgibubS4UdxtzN2FWuULfKvMKIGHn L4Zv4Dpje5ld7l7ce8jhfcURJ7AgAPwja3Tc7C38pmG+dSo2mj0I+YlCUED7mx3R ZSv6WtdAUCZjnKv9hSQVruk3fjYZc4dLEGzGH1ZJsD1ZkH5wNmWds5gHGEvQrc4Z 9reNanTxy+0ECxndk2H/ukw5Wv011rJWubLy/CnaPakPrSvrsmmoEs8ZcVZavlg9 ABJX/NtyBVl/y8+6Eh6/BdAhQr30U+c/UZLNbOflmcPGPiJCiXfBuaX1OF+qffQ1 QQvAGPgO2R9egHJWqFhBaLHtBAmiXSRWUU4+4nPBYZ/X5dCmGuV46knQGHdqoAQc l/qILh+spY09q9g118QbdnXBseiuVh/a+vf2GrbxbEMuWQu1kAI0DJbN0KKgUdtE ZkmIqXYULO6QZsYk3L41ZyKyE7oFMUqbT0uxSQZUCjOcnuBpMn/PzwM0yMJeDUKx 295Yzq5lkqkGmHJHi7XGOOI5XVIPb++DWXXv9E6Bfgoj4TSZscfwM69PQRoSq0hu iL9VOiyLRBU=
=zrvZ
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to SpamSalut,
Quoted 2 lines · Spam
Their libraries are huge and memory hogging which so many Linux users just do not like.
This is rather a fud argument: both the gnome VFS code and the KIOserver/KIOslave code aren't really large. You don't want to use
them for a busybox/tinylogin system, however.
Quoted 2 lines
What if a user doesn't want KDE or Gnome? Would all files created with either be broken?
The files still work well, just that you can't access them over the old fancy URL schemes.
Quoted 4 lines
I doubt that something like file streams and meta-data can
successfully be implemented purely in user-space and get the same support (ie be used by many programs) if this change doesn't come from the kernel. I just do not see it happen.
Actually, practical discordianism. If you develop a common API, there'll always be people disagreeing.
GTK+ with all its features is just cool. Desktop warping is a really nice thing. But there are people out there who don't want to use it. They use QT, or even plain old Athena Widgets. So what? Will we be implementing the X toolkits into the kernel?
In case of marketing it's up to the distributions to provide something concise so everyone can use their programs through a coherent namespace. (I.e. port all the apps they ship to gnome-vfs or kio).
Tonnerre
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 5 lines
Salut,
Quoted 7 lines
Quoted 2 lines · Spam
Their libraries are huge and memory hogging which so many Linux users just do not like.
This is rather a fud argument: both the gnome VFS code and the KIOserver/KIOslave code aren't really large. You don't want to usethem for a busybox/tinylogin system, however.
Then it is good. Just I see no programs other than Gnome or KDE apps that are using them.
Quoted 5 lines
Quoted 2 lines
What if a user doesn't want KDE or Gnome? Would all files created with either be broken?
The files still work well, just that you can't access them over the old fancy URL schemes.
No only, but if you cp then with a non KDE/Gnome app then you will loose the meta-data and extra info too. That was my point.
Quoted 5 lines
Quoted 4 lines
I doubt that something like file streams and meta-data can
successfully be implemented purely in user-space and get the same support (ie be used by many programs) if this change doesn't come from the kernel. I just do not see it happen.
Quoted 7 lines
Actually, practical discordianism. If you develop a common API, there'll always be people disagreeing.
GTK+ with all its features is just cool. Desktop warping is a really nice thing. But there are people out there who don't want to use it. They use QT, or even plain old Athena Widgets. So what? Will we be implementing the X toolkits into the kernel?
This is certainly not what I said or wanted.
Quoted 3 lines
In case of marketing it's up to the distributions to provide something concise so everyone can use their programs through a coherent namespace. (I.e. port all the apps they ship to gnome-vfs or kio).
Do you really believe this will happen? Good if it did. I do not believe it. And I certainly do not see the thousands of man hours needed to actually provide all the patches as a benefit.
~S
Quoted 1 line
Tonnerre
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Salut,
Quoted 2 lines · Spam
Then it is good. Just I see no programs other than Gnome or KDE apps that are using them.
Because KDE people hate Gnome people and vice versa, and because the rest of the world just neglects the two races for political reasons.
Maybe the Freedesktop project should provide some convenient
specification/code to do it. Like they do for HAL and DBUS (Please note that this is something interesting because it does clever things on hardware without requiring to patch the kernel.)
Quoted 6 lines
Quoted 4 lines
In case of marketing it's up to the distributions to provide something concise so everyone can use their programs through a coherent namespace. (I.e. port all the apps they ship to gnome-vfs or kio).
Do you really believe this will happen?
If the distributors really want to be able to gain money, and if the Free Unix community wants to gain a significant market share, this is supposed to happen. It's the question of whether we can ignore our childish concept wars, or if we're always going to stay at that low level we're at now.
Actually, this can't be fixed by putting everything into the kernel.
Tonnerre
Re: The argument for fs assistance in handling archives
Quoted 5 lines · Tonnerre
Salut,
On Mon, Sep 06, 2004 at 10:05:34AM +0200, Spam wrote:
Quoted 9 lines
Quoted 4 lines
Then it is good. Just I see no programs other than Gnome or KDE apps that are using them.
Because KDE people hate Gnome people and vice versa, and because the rest of the world just neglects the two races for political reasons.
Hi,
This is completely wrong. Neither KDE people hate GNOME people nor viceversa.
Time ago, I repeat in a conference the GNOME people need to jump on the KDE wagon, so the train will move to a single direction, and people does not approve this comment.
More than on the license, QPL, there are a developer problem.
Develop in C (Gtk, GNOME) is a lot more easy than develop in C++
(Qt, KDE)
But, for the GUI, the C++ approach is a lot superior ... a lot.
The problem is this. They start to develop on Gtk to create GNOME, in C for a license problem.
Now, they said why I need to change?
But they need to change, because it is supeiror. Everyone may of
course wants to do what prefer.
Quoted 7 lines
Maybe the Freedesktop project should provide some convenient
specification/code to do it. Like they do for HAL and DBUS (Please note that this is something interesting because it does clever things on hardware without requiring to patch the kernel.)
I don't agree FreeDesktop.org, because it is handled by a RedHat employee and the code is made on Gtk2, generally.
... They are not neutral.
Thanks,
Giovanni
Quoted 21 lines
Quoted 9 lines
Quoted 6 lines
In case of marketing it's up to the distributions to provide something concise so everyone can use their programs through a coherent namespace. (I.e. port all the apps they ship to gnome-vfs or kio).
Do you really believe this will happen?
If the distributors really want to be able to gain money, and if the Free Unix community wants to gain a significant market share, this is supposed to happen. It's the question of whether we can ignore our childish concept wars, or if we're always going to stay at that low level we're at now.
Actually, this can't be fixed by putting everything into the kernel.
Tonnerre
Signature · 11 lines
-- -- Check FT Websites ... http://www.futuretg.com - ftp://ftp.futuretg.com http://www.FTLinuxCourse.com http://www.FTLinuxCourse.com/Certification http://www.rpmparadaise.org http://GNULinuxUtilities.com http://www.YourPersonalOperatingSystem.com --
Re: The argument for fs assistance in handling archives
Quoted 11 lines · Dr. Giovanni A. Orlando
Quoted 2 lines · Tonnerre
Because KDE people hate Gnome people and vice versa, and because the rest of the world just neglects the two races for political reasons.
This is completely wrong. Neither KDE people hate GNOME people nor viceversa.
Time ago, I repeat in a conference the GNOME people need to jump on the KDE wagon, so the train will move to a single direction, and people does not approve this comment.
More than on the license, QPL, there are a developer problem.
KDE is covered with LGPL/GPL licence, so is QT, so I don't see a problem here.
Quoted 2 lines
Develop in C (Gtk, GNOME) is a lot more easy than develop in C++
(Qt, KDE)
Well, some ppl knowing well C++ will say something completly different. Since I've started QT/KDE development, my life as developer has changed. I am developing everything in QT, as far as desktop stuff goes. And it's much more clear and easier.
You just need to lear C++ well ;)
Quoted 8 lines
But, for the GUI, the C++ approach is a lot superior ... a lot.
The problem is this. They start to develop on Gtk to create GNOME, in C for a license problem.
I don't agree FreeDesktop.org, because it is handled by a RedHat employee and the code is made on Gtk2, generally.... They are not neutral.
Agreed.
Signature · 1 line
GJ
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Alan CoxQuoted 9 lines · Alan Cox
Quoted 5 lines · Linus Torvalds
and quite frankly, I think you can do the above pretty much totally in user space with a small library and a daemon (in fact, ignoring security issues you probably don't even need the daemon). And if you can prototype it like that, and people actually find it useful, I suspect kernel support for better performance might be possible.
Gnome already supports this in the gnome-vfs2 layer. "MC" has supported it since the late 1990's.
Can it do this:
cd FC2-i386-disc1.iso ls
or this:
cd /dev/sda1
ls
cd /dev/floppy
ls
cd /dev/cdrom
ls
?
Signature · 1 line
Frank
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 7 lines · Frank van Maarseveen
Can it do this:
cd FC2-i386-disc1.iso lsor this:
Quoted 8 lines
cd
/dev/sda1
ls
cd/dev/floppy
ls
cd/dev/cdrom
ls?
We have the mount command for that. :^)
Signature · 2 lines
David Kleikamp IBM Linux Technology Center
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 15 lines · Dave Kleikamp
Quoted 14 lines · Frank van Maarseveen
Can it do this:
cd FC2-i386-disc1.iso lsor this:
cd
/dev/sda1
ls
cd/dev/floppy
ls
cd/dev/cdrom
ls
Quoted 3 lines
Quoted 1 line
?
We have the mount command for that. :^)
mount is nice for root, clumsy for user. And a rather complicated way of accessing data the kernel has knowledge about in the first place. For filesystem images, cd'ing into the file is the most obvious concept for file-as-a-dir IMHO.
Signature · 1 line
Frank
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 4 lines · Frank van Maarseveen
mount is nice for root, clumsy for user. And a rather complicated way of accessing data the kernel has knowledge about in the first place. For filesystem images, cd'ing into the file is the most obvious concept for file-as-a-dir IMHO.
The hell it is.
a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
b) kernel has no way to guess which options to use
c) fs _type_ is a fundamental part of mount - device(s) (if any) involved are arguments to be interpreted by that particular fs driver.
d) permissions required for that lovely operation (and questions like whether we force nosuid/noexec, etc.) are nightmare to define.
Frankly, the longer that thread grows, the more obvious it becomes that file-as-a-dir is a solution in search of problem. Desperate search, at that.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 4 lines · On Thu, Sep 02, 2004 at 11:00:27PM +0100…
The hell it is.
a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
excuse me, but how does the kernel mount the root fs?
Signature · 1 line
Frank
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 7 lines · Frank van Maarseveen
Quoted 4 lines · On Thu, Sep 02, 2004 at 11:00:27PM +0100…
The hell it is.
a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
excuse me, but how does the kernel mount the root fs?
trial and error. That's why you see all thos ext3 mounted as ext2 problems, or $RANDOMFS as fat.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 11 lines · Christoph Hellwig
Quoted 7 lines · Frank van Maarseveen
Quoted 4 lines · On Thu, Sep 02, 2004 at 11:00:27PM +0100…
The hell it is.
a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
excuse me, but how does the kernel mount the root fs?
trial and error. That's why you see all thos ext3 mounted as ext2 problems, or $RANDOMFS as fat.
Andb btw, for an lkml discussion RTFS wouldn't hurt.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Frank van MaarseveenQuoted 7 lines · Frank van Maarseveen
Quoted 4 lines · On Thu, Sep 02, 2004 at 11:00:27PM +0100…
The hell it is.
a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
excuse me, but how does the kernel mount the root fs?
By trying all fs types it has registered in a more or less random (OK, defined by order of fs type registration, which is kinda-sorta deterministic at boot time) order. With no flags, unless you pass them explicitly in kernel command line. Fs types list can also be set explicitly in the command line.
Next question?
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 9 lines · On Thu, Sep 02, 2004 at 11:06:40PM +0100…
Quoted 3 lines · Frank van Maarseveen
Quoted 1 line
a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
excuse me, but how does the kernel mount the root fs?
By trying all fs types it has registered in a more or less random (OK, defined by order of fs type registration, which is kinda-sorta deterministic at boot time) order. With no flags, unless you pass them explicitly in kernel command line. Fs types list can also be set explicitly in the command line.
Of course I know that: the point is, the kernel _has_ knowlegde contrary to what you blatantly said.
Signature · 1 line
Frank
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 6 lines · Frank van Maarseveen
Quoted 5 lines · On Thu, Sep 02, 2004 at 11:06:40PM +0100…
Quoted 3 lines · Frank van Maarseveen
Quoted 1 line
a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
excuse me, but how does the kernel mount the root fs?
Quoted 7 lines
Quoted 4 lines
By trying all fs types it has registered in a more or less random (OK, defined by order of fs type registration, which is kinda-sorta deterministic at boot time) order. With no flags, unless you pass them explicitly in kernel command line. Fs types list can also be set explicitly in the command line.
Of course I know that: the point is, the kernel _has_ knowlegde contrary to what you blatantly said.
What knowledge does the kernel have about fs type that could deal with the contents of given device? Details, please.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 3 lines · On Thu, Sep 02, 2004 at 11:17:22PM +0100…
What knowledge does the kernel have about fs type that could deal with the contents of given device? Details, please.
Try a "make tags;grep SUPER_MAGIC tags".
Or is it there for a different purpose?
Signature · 1 line
Frank
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 7 lines · Frank van Maarseveen
Quoted 3 lines · On Thu, Sep 02, 2004 at 11:17:22PM +0100…
What knowledge does the kernel have about fs type that could deal with the contents of given device? Details, please.
Try a "make tags;grep
SUPER_MAGICtags".
Or is it there for a different purpose?
RTFS and you'll see. Individual fs generally knows how to check if it would be immediately unhappy with given image (not all types do, BTW). Exact form of checks depends on fs type; for crying out loud, there's not even a promise that they are mutually exclusive!
Read the fucking source. Read through the code that "chooses" fs type of root fs. Look at it. Then use whatever you have between your ears.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 5 lines · On Thu, Sep 02, 2004 at 11:33:24PM +0100…
RTFS and you'll see. Individual fs generally knows how to check if it would be immediately unhappy with given image (not all types do, BTW). Exact form of checks depends on fs type; for crying out loud, there's not even a promise that they are mutually exclusive!
so?
A user can stick an USB memory card with _any_ malformed fs data and make troubles via the automounter or user mounts. Yes, mount might do some more checks but it sure won't do an fsck.
The user gets what he deserves when sticking crap in an USB port.
And that doesn't mean that the kernel should accept any fs image when a user tries to cd into the file.
Signature · 1 line
Frank
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 2 lines · Frank van Maarseveen
And that doesn't mean that the kernel should accept any fs image when a user tries to cd into the file.
No, it shouldn't. It should say "it's not a directory, bugger off". Which it does.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Frank van MaarseveenQuoted 12 lines · Frank van Maarseveen
Quoted 6 lines · On Thu, Sep 02, 2004 at 11:33:24PM +0100…
RTFS and you'll see. Individual fs generally knows how to check if it would be immediately unhappy with given image (not all types do, BTW). Exact form of checks depends on fs type; for crying out loud, there's not even a promise that they are mutually exclusive!
so?
Quoted 10 lines
A user can stick an USB memory card with _any_ malformed fs data and make troubles via the automounter or user mounts. Yes, mount might do some more checks but it sure won't do an fsck.
The user gets what he deserves when sticking crap in an USB port.
And that doesn't mean that the kernel should accept any fs image when a user tries to cd into the file.
You don't need kernel support for cd'ing into fs images. You need a shell (or GUI app) that:
1. notices that user tries to CD into a file, not a directory
2. Attempts fs type detection and do a loop mount.
3. Give error message if it wasn't a supported fs image.
Helge Hafting
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 6 lines · Helge Hafting
Quoted 1 line
You don't need kernel support for cd'ing into fs images. You need a shell (or GUI app) that:
1. notices that user tries to CD into a file, not a directory
2. Attempts fs type detection and do a loop mount.
3. Give error message if it wasn't a supported fs image.
Ok, and right now I'm in vim typing this message and want to ":new /tmp/backup.iso/etc/fstab"
modifying 1000+ applications is not an option IMO. Putting it in a preloaded library might be doable except maybe for permission problems but this is incredibly clumsy and lacks the possibility to implement sane caching behavior because it is process bound.
Signature · 1 line
Frank
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 2 lines · On Fri, Sep 03, 2004 at 10:50:18AM +0200, I wrote:
and lacks the possibility
to implement sane caching behavior because it is process bound.
sorry, this part is crap.
Signature · 1 line
Frank
Re: The argument for fs assistance in handling archives
In reply to Helge Hafting-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Helge Hafting wrote:
[...]
| You don't need kernel support for cd'ing into fs images. | You need a shell (or GUI app) that: | 1. notices that user tries to CD into a file, not a directory | 2. Attempts fs type detection and do a loop mount. | 3. Give error message if it wasn't a supported fs image.
You can argue this about FS plugins, too. You don't need kernel support for cryptocompress, for example. Just have a fifo farm. Every file that has compression turned on is actually a named pipe, the original file is hidden (starts with a dot), and you have a script running to each fifo which runs zcat.
The kernel support (read: interface) is just cleaner in some ways.
In the loop mount example, it isn't universal -- the user can't CD into a file if they use the wrong shell, and two different shells might make two different mounts. How many mounts before we run out of loop devices?
In the cryptocompress example, there's no caching or compress-on-flush optimization, and it isn't entirely transparent to anything. It doesn't look like a file, it looks like a fifo. ls will make it look yellow. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTkEWngHNmZLgCUhAQKmMhAAkECgB9hUCxM2vNn0NgzQmkuz028QsAEN 6B1ej6tCbVyT8whsVD2YA7rQH5UavUeUKjuFNJNwelXeEYqG8tbv4vcE1Z65U5tR GUJkDxWwgjfx4JtE137VmWXcMWW/6FeIBbHuKFohDeBz2nBYmc3w/b4NjgT//JOG 0iUQGRDQcGxDuCYqMqCO8+m/LOSL9wbYDG+bXa4NxVAcghmxAhAJFiDpRDQHcYxX CeUI7ZkFnNX/4A52YTAFyHQmeoQRX1SLxJsn0GH9mcf2Gsxs8AtoTQgtedDsz9I2 Ct6Dil6NfI94VfOx0EWa1y4I0p7UnFjAsye9zeElhumhocFvEcd5Lzn6SnCY/rQh o4VukNKUtDE92noL62BZGgCc9ek3/YZivMMFfmQOrjKjfQxEMwSvyjKWIj9XPATT FWkN7mZn7JSxluB1n1BtHTYvdieD65RGWF1aW1atJZ1r0z0xzy95LF/kqm9Rkigr zv6+1j9ND79pBikYuq4ol916R9bW93TkriSYbENBP5dtMx4nXUYI6kRU53Y1UxZH +mstouIpGr9MB7kHLLDYFFFV8upRJi2EQHA2sAtHOaMmjZUoJrbsIdA9n1r1LsKt 6Wjq8MofdyvZWXbzh9aGD4QkovNf6tw4UHAGzVf9Uh3n6mwh7yw4qyDC5h8dqMEe P4wWkRawtHc=
=GC9p
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 11 lines · David Masover
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Helge Hafting wrote:
[...]| You don't need kernel support for cd'ing into fs images. | You need a shell (or GUI app) that: | 1. notices that user tries to CD into a file, not a directory | 2. Attempts fs type detection and do a loop mount. | 3. Give error message if it wasn't a supported fs image.
Quoted 5 lines
You can argue this about FS plugins, too. You don't need kernel support for cryptocompress, for example. Just have a fifo farm. Every file that has compression turned on is actually a named pipe, the original file is hidden (starts with a dot), and you have a script running to each fifo which runs zcat.
Eww. A shell/filemanager supporting fs images is one running program. The fifo farm is one per file. Ouch.
Quoted 6 lines
The kernel support (read: interface) is just cleaner in some ways.
In the loop mount example, it isn't universal -- the user can't CD into a file if they use the wrong shell, and two different shells might make two different mounts. How many mounts before we run out of loop devices?
This limitation apply to the kernel interface too. But you can configure for an arbitrary number of loop mounts anyway. The user will have to mount using the correct tool - after that the door is opened for anything.
Quoted 4 lines
In the cryptocompress example, there's no caching or compress-on-flush optimization, and it isn't entirely transparent to anything. It doesn't look like a file, it looks like a fifo. ls will make it look yellow.
Even the kernel supported "cd into fs images" has problems. How should the kernel know which files you want to support with mounting, and which
ones are the wast majority of plain files? A file manager doesn't
have to do this fully automatic, it can have a button/hotkey for "loop mount this". So the user will have to tell it what to do, but in a much easier way than typing "mount -o loop . . . " The file manager can also ask for encryption keys - something the kernel cannot do because it doesn't know what user interface(s) is in use by the process that attempts the "cd". There may be one interface (stdin, X) there may be both, i.e. something started from an xterm can use both, there may be none, (the web/ftp/file-server got a request for something inside a encrypted fs image.)
And even if the kernel can figure out the correct user interface, do we want it to contain an X client for asking for keys? :-/
Helge Hafting
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Frank van MaarseveenSalut,
Quoted 2 lines · Frank van Maarseveen
Try a "make tags;grep
SUPER_MAGICtags".
Or is it there for a different purpose?
Problem is:
There are cool superblock magics for reiserfs. And for ext[23]. And even for good old Minix. Cool.
However, there are also ugly file systems, such as fat for
example. Fat has been defined as "something that can be read by a fat driver", which can be pretty much anything. You can't really detect whether it is fat. Even Microsoft only guess whether some FS is fat or not.
And don't say detect it via a partition table. There are a lot of cases where you get absolutely no or wrong information out of it, and there are also lots of cases where you plainly have none.
Tonnerre
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 8 lines · Tonnerre
Problem is:
There are cool superblock magics for reiserfs. And for ext[23]. And even for good old Minix. Cool.
However, there are also ugly file systems, such as fat for
example. Fat has been defined as "something that can be read by a fat
This problem is not new. Kernel probes for filesystems in a particular order for mounting the root fs. And mount understands the fstype=auto in /etc/fstab. There is no perfect solution but it's sure possible to come up with something acceptable and workable with not too much effort, for a configuratble subset of file-systems. This is not that much different from an automounter/usermount mounting a USB storage device or cdrom:
ext3, ext2, udf, iso9660, vfat, read-only or not, just to name a few things. This should work anyway.
Signature · 1 line
Frank
Re: The argument for fs assistance in handling archives
In reply to [email protected]-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
[email protected] wrote:
| On Thu, Sep 02, 2004 at 11:48:06PM +0200, Frank van Maarseveen wrote: | |>mount is nice for root, clumsy for user. And a rather complicated |>way of accessing data the kernel has knowledge about in the first |>place. For filesystem images, cd'ing into the file is the most |>obvious concept for file-as-a-dir IMHO. | | | The hell it is. | | a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
reiser4 kernel will contain knowledge of fs type contained in a file.
"file/..metas/type" might contain a mime type. Mime type might have to be guessed, but at least if it's made by a local "mkisofs" then we're fine.
Indeed, that's not the only interface that's been discussed. "file/..metas/is_isofs" might be consulted.
| b) kernel has no way to guess which options to use | c) fs _type_ is a fundamental part of mount - device(s) (if any) involved | are arguments to be interpreted by that particular fs driver.
Unless there's some severe security issues with "mount this iso as fat and you get root access", this should work fine.
I see no reason why there can't be a global setting of the mount commandline to use.
And it doesn't all have to be in the kernel. Only it'd be nice to have some of it there because the kernel knows how to deal with an isofs, even if it won't know what it looks like.
| d) permissions required for that lovely operation (and questions like | whether we force nosuid/noexec, etc.) are nightmare to define.
They are quite simple, actually. Just set them globally -- some admins would force nosetuid/noexec, some wouldn't. And the operation happens transparently -- you need no "permissions" other than to read the directory which would contain the mount.
| Frankly, the longer that thread grows, the more obvious it becomes that | file-as-a-dir is a solution in search of problem. Desperate search, at | that.
Actually, the longer this thread grows, the more obvious it is how when there's a hot issue, everyone has an opinion, even if the same opinion has been expressed ten or twenty times already.
File-as-a-dir has numerous advantages, but enough have been discussed. Short list is image mounts, tarballs, streams, metas, and namespace unification. Longer list and explanations can be found if you RTFA. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTe19XgHNmZLgCUhAQJdow/+Knsw1GgpauqDUcg8sKtxzgXZ18OxMQ3Q By8sRrSTuKAzI5A3BtYIzndsj1veP+7wndG7nYPz8NS1fU2+xWSIhoGq/YMaQsu4 70uMLu448PFXZua4hZMk2w4mkULXbGyYHJ1Bf+2Z7QkQ/8W08hozC8QQynxMXIkX SrcWCS5hK8Nh7Ol691sDpPqexH7F1GwUyoslNGj63U5r6ViLAawt2ZKDYdT7ZPo8 0a/pWUHoHMPbv/KwqZZxRr1/qncA9QYQo6JqQBPPCr+tWNJs/ei3nAKGi58iOt1M DK1TEKd2lpbmwiK5pWDwGz+nwWmaFTAyfTEEEcP4gZedSJtRXaxyNh0jRl1iLATB SCO5Eb4jkQs8hdjHqQcQ1q7XKFX9eSXWeDdrGrtWaYC/QYOHxT+ci3lnKBKCG99Y YTqg3sNEZlV1N0jIcNvFSDEYbbX12v1Y6xbwvUx48+sMyUj3suT76niTRbwEydfO MA9y+wE2k4wF+h+sJCbTjimCNFvvuFTTJuBCbQTpfY4eOYBAFalxnWmrpTEfVzka 4iAqAYygWObGDkFFy/rp1HEVZPIKM0NwGLOsRwJsgyUMOsccBrEc0bg8sgMECVfs 5qNIb27tLokh8NBR6RodAv2NZYKC+foM0T+PC5bZMFD/Q7f6yDklqK4C4RCIYaXj xO9z1C6FPcM=
=/gmK
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 2 lines · David Masover
reiser4 kernel will contain knowledge of fs type contained in a file.
That's a disaster, btw.
There is no one "fs type" of a file. Files have at _least_ one type (bytestream), but most have more. Which is why automatically doing the right thing (in the sense you seem to want) in kernel space is simply not possible.
Linus
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Linus Torvalds wrote:
| | On Thu, 2 Sep 2004, David Masover wrote: | |>reiser4 kernel will contain knowledge of fs type contained in a file. | | | That's a disaster, btw. | | There is no one "fs type" of a file. Files have at _least_ one type | (bytestream), but most have more. Which is why automatically doing the | right thing (in the sense you seem to want) in kernel space is simply not | possible.
Oops, my bad. The interface is kernel, the file type database is user. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTfH7XgHNmZLgCUhAQL0aw//SG/RDCy30xcVh36FVVAOG7GjP6UkHBeo 14/O3hSELxwm9r8z0YnGnbpoJj9atbOCu6VJhiumtjeyikdH5OaOrcMo7DQYp47V holyEKvjd+YUkCBJSjFBmG279Ac7wuk2orFlG76gCkFjNk11W/FQsjEM3gi8yKa0 ZODFmh0XJa2p0mKRnApwynO1ma7HgyqNRKrjmxMUK2VHpOsbqjVe1+Gc2lk0E9gv xc2PIdCgi3yDEpDpxSY8LEBXad7GTwa2VEID5G7C6Z0wOH38YyMEpoHQz03se9zO aCnVg8LPdFkZveWXPiiyJrwDERCyKt/yrRxrVznWyNVaWmhoxXmw7TmQexvGiDD2 E6+XT8uWfdeMZRPhv284qcBegIDw5c1wHgz+GRso61T6x04hfJyY/onbF5lHz661 lfys4JRT0zmYbrG1b3GVgmfKDv7t3V6DkY+Pi1E5raeZr2F0idiZo+7uKh2NZAML PLhb8LLr/lbHOSRG3Rq0YY/l+Q6wZXy770qF/51jp/c3UXXJyVusK+bCLrVZ3VPa 0Uf3KkuFLcoD4jDTNjt79QVJXIvEHmNOxp/g80nJUFM0WDj7u4a7pFpLCHZzFeBl 5uoS9c7dYjuV+/EkQ2S8YwryeyCHQNt381icb8HtbdlQOCVuK0v9OspA3IZmphcB oD1RBCkVTYQ=
=l8Vz
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
In reply to Linus TorvaldsQuoted 6 lines · Linus Torvalds
Quoted 3 lines · David Masover
reiser4 kernel will contain knowledge of fs type contained in a file.
Whoa! No it won't. It will allow you to create a metafile named type if you choose. Or maybe a metadirectory named types, I don't really care much about this stuff yet, we need to do other things to the semantics more urgently than this, Typing requires a lot of cooperation from user space apps. I don't really expect to have that much social pull on the issue, but if someone chooses to design their app to use a types metadirectory, I don't mind accomodating on it. I apologize if from my saying that I gave the impression that I think files should be strongly typed.
I am quite well aware of the disadvantages of OS/400 hindering usability with type information, as well the advantages of having types available for those that want to look at them.....
Quoted 4 lines
That's a disaster, btw.
In every implementation so far it has been a net disaster, because OS designers who type things have been willing to make users pay attention to type when they don't want to. Sometimes trying to make too much out of something makes it a mistake when being a bit more laid back would make it ok. Me, I am so laid back, I prefer to work on other features for a few years first.....;-)
Quoted 10 lines
There is no one "fs type" of a file. Files have at _least_ one type (bytestream), but most have more. Which is why automatically doing the right thing (in the sense you seem to want) in kernel space is simply not possible.
Linus
Re: The argument for fs assistance in handling archives
In reply to David MasoverQuoted 3 lines · David Masover
| a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
reiser4 kernel will contain knowledge of fs type contained in a file.
Right. And when I decide to talk to delusional nutcases I will go to talk.origin and find a creationist to chat with.
Time to extend the killfile...
*plonk*
Re: The argument for fs assistance in handling archives
In reply to David MasoverQuoted 9 lines
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1[email protected] wrote:
| On Thu, Sep 02, 2004 at 11:48:06PM +0200, Frank van Maarseveen wrote:
|
|>>mount is nice for root, clumsy for user. And a rather complicated |>>way of accessing data the kernel has knowledge about in the first |>>place. For filesystem images, cd'ing into the file is the most |>>obvious concept for file-as-a-dir IMHO.
Quoted 6 lines
| | | The hell it is. | | a) kernel has *NO* *FUCKING* *KNOWLEDGE* of fs type contained on a device.
Quoted 7 lines
reiser4 kernel will contain knowledge of fs type contained in a file.
"file/..metas/type" might contain a mime type. Mime type might have to be guessed, but at least if it's made by a local "mkisofs" then we're fine.
Indeed, that's not the only interface that's been discussed. "file/..metas/is_isofs" might be consulted.
What you are talking about isn't the kernel or such, but plugins that could extend the filesystem. Plugins could store information about contents, encodings, formatting, filesystems, etc, as meta-info. If you have a plugin that would allow you to traverse files as disk images then it could read those meta-data. But before those plugins exist then there is no such standard for info stored as meta-data and the kernel wouldn't know anything about this to begin with.
Quoted 51 lines
| b) kernel has no way to guess which options to use | c) fs _type_ is a fundamental part of mount - device(s) (if any) involved | are arguments to be interpreted by that particular fs driver.Unless there's some severe security issues with "mount this iso as fat and you get root access", this should work fine.
I see no reason why there can't be a global setting of the mount commandline to use.
And it doesn't all have to be in the kernel. Only it'd be nice to have some of it there because the kernel knows how to deal with an isofs, even if it won't know what it looks like.
| d) permissions required for that lovely operation (and questions like | whether we force nosuid/noexec, etc.) are nightmare to define.
They are quite simple, actually. Just set them globally -- some admins would force nosetuid/noexec, some wouldn't. And the operation happens transparently -- you need no "permissions" other than to read the directory which would contain the mount.
| Frankly, the longer that thread grows, the more obvious it becomes that | file-as-a-dir is a solution in search of problem. Desperate search, at | that.Actually, the longer this thread grows, the more obvious it is how when there's a hot issue, everyone has an opinion, even if the same opinion has been expressed ten or twenty times already.
File-as-a-dir has numerous advantages, but enough have been discussed. Short list is image mounts, tarballs, streams, metas, and namespace unification. Longer list and explanations can be found if you RTFA. -----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.orgiQIVAwUBQTe19XgHNmZLgCUhAQJdow/+Knsw1GgpauqDUcg8sKtxzgXZ18OxMQ3Q By8sRrSTuKAzI5A3BtYIzndsj1veP+7wndG7nYPz8NS1fU2+xWSIhoGq/YMaQsu4 70uMLu448PFXZua4hZMk2w4mkULXbGyYHJ1Bf+2Z7QkQ/8W08hozC8QQynxMXIkX SrcWCS5hK8Nh7Ol691sDpPqexH7F1GwUyoslNGj63U5r6ViLAawt2ZKDYdT7ZPo8 0a/pWUHoHMPbv/KwqZZxRr1/qncA9QYQo6JqQBPPCr+tWNJs/ei3nAKGi58iOt1M DK1TEKd2lpbmwiK5pWDwGz+nwWmaFTAyfTEEEcP4gZedSJtRXaxyNh0jRl1iLATB SCO5Eb4jkQs8hdjHqQcQ1q7XKFX9eSXWeDdrGrtWaYC/QYOHxT+ci3lnKBKCG99Y YTqg3sNEZlV1N0jIcNvFSDEYbbX12v1Y6xbwvUx48+sMyUj3suT76niTRbwEydfO MA9y+wE2k4wF+h+sJCbTjimCNFvvuFTTJuBCbQTpfY4eOYBAFalxnWmrpTEfVzka 4iAqAYygWObGDkFFy/rp1HEVZPIKM0NwGLOsRwJsgyUMOsccBrEc0bg8sgMECVfs 5qNIb27tLokh8NBR6RodAv2NZYKC+foM0T+PC5bZMFD/Q7f6yDklqK4C4RCIYaXj xO9z1C6FPcM=
=/gmK
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Spam wrote:
[...]
| Indeed, that's not the only interface that's been discussed. | "file/..metas/is_isofs" might be consulted. | | |> What you are talking about isn't the kernel or such, but plugins that
Plugins are kernel-space. As Linus points out, dealing with mime-types in the kernel is uncool.
|> could extend the filesystem. Plugins could store information about |> contents, encodings, formatting, filesystems, etc, as meta-info. If |> you have a plugin that would allow you to traverse files as disk |> images then it could read those meta-data. But before those plugins |> exist then there is no such standard for info stored as meta-data and |> the kernel wouldn't know anything about this to begin with.
So implement a plugin which knows how to talk to a userland program which knows about metadata. The plugin controls access to file-type.
Maybe there ought to be a general-purpose userland plugin interface? So that the only things left in the kernel are things that have to be there for speed and/or sanity reasons? (Things like cryptocompress and standard file/directory plugins.)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTfIs3gHNmZLgCUhAQJCqA//cPGI/TPfgtwovk5a8HdcY9TbjJELGtIb CHcNa2LKGUB8uVvoL9M0Pe9ei4iVK2QOL1QjUAbIE0Dx7t18KR/5qEIfCHTSw0sJ 8u/r3aaJhFjFwMcVLQOZWJYuTeodgMwkV96GgViGoHoiqDiV7BzZgjd43qwiH8rW FUTKlPn2VmijyTTbf5VfX4hvmsU/He+5W0t08/xe3vpCa+ihNFLJQAwGioo/wzFq aHl9jBT1esFLxONd7OxQpgVl/2uHx+rSAY6F5RyBqL/Tpm1ZKlrMdAzmdDWcAJA9 KnOLN8ltcPmjP0eCzgCO/iq8yczcwcagfmbD+WcYOmQbTXMTjxktrZqRuLrUHU+7 tl8JISKch5epHfOnQ/RTMEgotlcQ0SCoE7K5lIUuyheMYRWoVDJSvy3okET6ZxQL NP3PVHguQbu1Bo2X8LNsrnU0KFT7XrejpXzKalPWVbQxayEEWUwLXBdNIpgBoYwX FZXQKEpS4MmB/8kEC9xuQ077PfAclcjcHSj4B7phbSVMioFy4/HojOLOLA8gjHyr 7p+wimfDpsuioDLlccwc1b+fLB2eJM5G+zfzh//uy7LdfFxuI+074FJYxebRI2eL N/rpjT+S0S7L0/k71Rwp/5y4YgDgdQqU0wyGaBLRjP0qD7k2S41g+yrua6qArhzP baR9dCu1zAQ=
=Nrye
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 8 lines · David Masover
So implement a plugin which knows how to talk to a userland program which knows about metadata. The plugin controls access to file-type.
Maybe there ought to be a general-purpose userland plugin interface? So that the only things left in the kernel are things that have to be there for speed and/or sanity reasons? (Things like cryptocompress and standard file/directory plugins.)
Ahem,
Wasn't this the goal of GNU HURD?
I really think you should ask them why they haven't delivered something useful, then come back to this thread.
Thanks.
Signature · 4 lines
Chris Dukes Warning: Do not use the reflow toaster oven to prepare foods after it has been used for solder paste reflow. http://www.stencilsunlimited.com/stencil_article_page5.htm
Re: The argument for fs assistance in handling archives
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"Chris" == Chris Dukes <[email protected]> writes:
Chris> Ahem, Wasn't this the goal of GNU HURD?
Chris> I really think you should ask them why they haven't delivered Chris> something useful, then come back to this thread.
Note: this is just an outsider's view of HURD. I mostly just lurk on some HURD lists.
AFAIK, their concept of translators seems to be working fine. The ext2 translator had problems because it wanted to mmap the whole partition, and so could only load partitions up to 2 GB or so, but I haven't heard any problems with the translators concept. That particular problem with the ext2 translator, I think, has been fixed.
They are also porting the HURD to another microkernel (L4), since GNU Mach is outdated (I forget exactly what problems they were having, but it might have been that IPC is too slow on Mach). (Unfortunately, L4 itself seems to still be under development...)
They also have a lot less developers than Linux...
Again, I haven't heard of any problems with their concept of translators. That seems to be working fine. The fact that they haven't produced anything "useful" (for some values of useful) seems to be mostly due to other things.
Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/ PGP/GnuPG key: 1024D/124B61FA Fingerprint: 96C5 012F 5F74 A5F7 1FF7 5291 AF29 C719 124B 61FA Key available at wwwkeys.pgp.net. Encrypted e-mail preferred.
Re: The argument for fs assistance in handling archives
In reply to Chris Dukes-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Chris Dukes wrote:
| On Thu, Sep 02, 2004 at 08:28:20PM -0500, David Masover wrote: | |>So implement a plugin which knows how to talk to a userland program |>which knows about metadata. The plugin controls access to file-type. |> |>Maybe there ought to be a general-purpose userland plugin interface? So |>that the only things left in the kernel are things that have to be there |>for speed and/or sanity reasons? (Things like cryptocompress and |>standard file/directory plugins.) | | | Ahem, | Wasn't this the goal of GNU HURD?
The goal of GNU HURD was to take everything out of the kernel and make it entirely daemons. That's a far cry from keeping a file-type database (historically the realm of file managers) out of the kernel.
Most people hate putting things in the kernel that don't belong there. Most of what's in the kernel is for speed and/or sanity -- filesystems are there for speed, device drivers (and scheduling, and vm, and so on) are there for sanity.
Correct me if I'm wrong on this.
| I really think you should ask them why they haven't delivered | something useful, then come back to this thread.
Honestly? I think it's mostly got nothing to do with architecture. I think it's mostly got to do with politics. Most people would rather hack on Linux, which is already done, than try to develop HURD, which is something new. Most people also enjoy working with Linus (or prefer Linus to the FSF).
I do not like how Linux is monolithic. I do not like having to reboot to upgrade the kernel, and I do not like having to run closed software (the nvidia drivers) in the kernel (as in, full privelages, can crash entire system, yadda yadda). But Linux is the best we have.
The HURD people have delivered at least something. I think there's even a Debian/HURD distro. Whether it's useful probably has to do with whether it's stable/fast, which isn't likely. You hear of Linux news every day, you hear of HURD maybe once in a lifetime -- "Hey, HURD exists!"
But you are right -- I need to take a break from this thread. It's becoming addictive, and it's hard keeping up with so many people who are so much smarter than I am.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iQIVAwUBQTf0/3gHNmZLgCUhAQKGrw//XMBsWBo7uqlocjdvSati4GkxsB+f2GPW hI3JboYhWVMB7Ya+mS9VbrruxVn7uCCUwLwNDDnMncHTf09Em+WhzwJCQXZ14Y2r AxZ8iaVR9Qv8hIHGw59uJYWvtTSHgSmWiFFVa4UB4D3NneSFr2vgXXDbU8LjhHQB 46qs8IM6s+RCb0L608uGml6AmyrxFvDgWz9p+++tWicLjQPVhBnLQQnqYy92we7x qUwxCyGQywmdcSsJlkcjKzLQ/dCDwQRq7LXmSCx3qbSRr27D6zpE6l9He62ZuoBR I9NmO12S6VgYifO5fnq6pxSu2+GuJVW469tm7EiXmKZ/rjfMJKp0cqg+mtGTU91m gk7cZTBbe4id9+IQPJPJJ08IMH5XTI4DI6dXCkKIB0TZ8teFnU2+DsUdZ50ZXwiL etFUjortzjPjEqZoKfvK+4n7HOMu1w1og/7aE6P5UNTbj5ViwLQB9KjwLBDyBMJr dTmcLRyxzHtDZJMQuLwvU/SsdF86CdlR4spkEI6CitRMiiWvYghOeOkasOO8qxVj ckzZzUCRHNF/PTPSAEiIaw2HyjVzf0Nru96Y80/hik3hRYO0XuTP/3K/twHoebkz Fo1NLtGU3g631pDYcpqnkkAcQN+3fiyZ3VEiNoYYLhZpkQiEH8goRv5lof9XJPAC krhwM8lNHoM=
=JJEe
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 15 lines · David Masover
Chris Dukes wrote:
| On Thu, Sep 02, 2004 at 08:28:20PM -0500, David Masover wrote: | |>So implement a plugin which knows how to talk to a userland program |>which knows about metadata. The plugin controls access to file-type. |> |>Maybe there ought to be a general-purpose userland plugin interface? So |>that the only things left in the kernel are things that have to be there |>for speed and/or sanity reasons? (Things like cryptocompress and |>standard file/directory plugins.) | | | Ahem, | Wasn't this the goal of GNU HURD?
Quoted 3 lines
The goal of GNU HURD was to take everything out of the kernel and make it entirely daemons. That's a far cry from keeping a file-type database (historically the realm of file managers) out of the kernel.
Yep. Traditional microkernel.
[...]
Quoted 8 lines
| I really think you should ask them why they haven't delivered | something useful, then come back to this thread.
Honestly? I think it's mostly got nothing to do with architecture. I think it's mostly got to do with politics. Most people would rather hack on Linux, which is already done, than try to develop HURD, which is something new. Most people also enjoy working with Linus (or prefer Linus to the FSF).
Partly right. Hurd is _much_ older than Linux, and didn't start from scratch (Mach was around, and worked). Yet I remember hearing "any day now" for Hurd from 86 or 87... and as of now, AFAIU even the people hacking on Hurd prefer Linux (because Linux does work, Hurd doesn't).
Quoted 2 lines
I do not like how Linux is monolithic. I do not like having to reboot to upgrade the kernel,
Tough luck. An upgrade of the core will _always_ mean rebooting. Plus you can replace modules in Linux without a reboot.
Quoted 3 lines
and I do not like having to run closed software
(the nvidia drivers) in the kernel (as in, full privelages, can crash entire system, yadda yadda).
Me neither. But that is not exactly Linux' fault, binary drivers are quite strongly discouraged.
Quoted 1 line
But Linux is the best we have.
By far.
Quoted 5 lines
The HURD people have delivered at least something. I think there's even a Debian/HURD distro. Whether it's useful probably has to do with whether it's stable/fast, which isn't likely. You hear of Linux news every day, you hear of HURD maybe once in a lifetime -- "Hey, HURD exists!"
It is still vaporware, some 20 years later.
Signature · 4 lines
Dr. Horst H. von Brand User #22616 counter.li.org Departamento de Informatica Fono: +56 32 654431 Universidad Tecnica Federico Santa Maria +56 32 654239 Casilla 110-V, Valparaiso, Chile Fax: +56 32 797513
Re: The argument for fs assistance in handling archives
In reply to David MasoverI don't think streams are a good thing, I think that all of the different pieces of additional functionality necessary to emulate them with files and directories are a good thing.
Keeping streams out of linux was one of the (less important) ideas behind reiser4. Streams are a rigid hack. The toolkit that can emulate them, is useful, and the perceived importance of that emulation will fade as people start to use the toolkit for things that are much more fun than streams.
I agree with most of the rest of what you say though David.
Hans
Quoted 3 lines · David Masover
File-as-a-dir has numerous advantages, but enough have been discussed. Short list is image mounts, tarballs, streams, metas, and namespace unification. Longer list and explanations can be found if you RTFA.
Re: The argument for fs assistance in handling archives
In reply to David MasoverQuoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"David" == David Masover <[email protected]> writes:
David> File-as-a-dir has numerous advantages, but enough have been David> discussed. Short list is image mounts, tarballs, streams, David> metas, and namespace unification. Longer list and explanations David> can be found if you RTFA.
And it has numerous dis-advantages as well, since it doesn't have a good set of semantics and syntax defined yet, nor does it explain except by vigorous handwaving the performance and security impacts it can have.
My personal feeling is that the mount(8) command should be the tool used to extract and expose the internal namespace of files like this and to then graft it onto the standard Unix namespace with gross Unix semantics, but it's own wacky internal semantics. This way, standard tools don't care, but special tools which know how to handle it can do what they want.
Quoted 3 lines
mount -t tarfs /some/place/on/disk/foo.tar.gz
/mnt/tarcp/var/tmp/img.gif.
umount/mnt/tar
Oops! Someone did a rm /some/place/on/disk/foo.tar.gz between steps one and two. Now what happens? Please define those semantics...
John
Re: The argument for fs assistance in handling archives
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
"David" == David Masover <[email protected]> writes:
David>> File-as-a-dir has numerous advantages, but enough have been David>> discussed. Short list is image mounts, tarballs, streams, David>> metas, and namespace unification. Longer list and explanations David>> can be found if you RTFA.
Quoted 13 lines
And it has numerous dis-advantages as well, since it doesn't have a good set of semantics and syntax defined yet, nor does it explain except by vigorous handwaving the performance and security impacts it can have.
My personal feeling is that the mount(8) command should be the tool used to extract and expose the internal namespace of files like this and to then graft it onto the standard Unix namespace with gross Unix semantics, but it's own wacky internal semantics. This way, standard tools don't care, but special tools which know how to handle it can do what they want.
Quoted 6 lines
Quoted 3 lines
mount -t tarfs /some/place/on/disk/foo.tar.gz
/mnt/tarcp/var/tmp/img.gif.
umount/mnt/tarOops! Someone did a rm /some/place/on/disk/foo.tar.gz between steps one and two. Now what happens? Please define those semantics...
Uhm, can you delete a file (loop) that is mounted?
Quoted 1 line
John
Re: The argument for fs assistance in handling archives
Am 2004-09-03 16:01:17, schrieb Spam:
Quoted 10 lines
Quoted 8 lines
Quoted 5 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
"David" == David Masover <[email protected]> writes:
mount -t tarfs /some/place/on/disk/foo.tar.gz
/mnt/tarcp/var/tmp/img.gif.
umount/mnt/tarOops! Someone did a rm /some/place/on/disk/foo.tar.gz between steps one and two. Now what happens? Please define those semantics...
Uhm, can you delete a file (loop) that is mounted?
Not as $USER but root... :-/
Quoted 1 line
Quoted 1 line
John
Greetings
Michelle
Signature · 4 lines
Linux-User #280138 with the Linux Counter, http://counter.li.org/
Michelle Konzack Apt. 917 ICQ #328449886
50, rue de Soultz MSM LinuxMichi
0033/3/88452356 67100 Strasbourg/France IRC #Debian (irc.icq.com)Re: The argument for fs assistance in handling archives
In reply to SpamSalut,
Quoted 8 lines · Spam
Quoted 6 lines
Quoted 3 lines
mount -t tarfs /some/place/on/disk/foo.tar.gz
/mnt/tarcp/var/tmp/img.gif.
umount/mnt/tarOops! Someone did a rm /some/place/on/disk/foo.tar.gz between steps one and two. Now what happens? Please define those semantics...
Uhm, can you delete a file (loop) that is mounted?
"Text file busy"
But for files this isn't true.
Currently, we have another method assuring data coherency: we remove the inode only when the last reference goes away.
Tonnerre
Re: The argument for fs assistance in handling archives
In reply to Frank van Maarseveen-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Frank van Maarseveen wrote:
|> |>We have the mount command for that. :^) | | | mount is nice for root, clumsy for user. And a rather complicated | way of accessing data the kernel has knowledge about in the first | place. For filesystem images, cd'ing into the file is the most | obvious concept for file-as-a-dir IMHO.
thats why we have automount.
lg, clemens
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFBPF3djBz/yQjBxz8RAmVhAKCODKNKOV9V0I59SUfQ1pp+qk88xwCg6+4p Q+JcbO+W0+6cUfSEf3z/Iwk=
=IwZQ
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 7 lines
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Frank van Maarseveen wrote:
|>>
|>>We have the mount command for that. :^)
Quoted 8 lines
| | | mount is nice for root, clumsy for user. And a rather complicated | way of accessing data the kernel has knowledge about in the first | place. For filesystem images, cd'ing into the file is the most | obvious concept for file-as-a-dir IMHO.thats why we have automount.
Which still needs to be setup in fstab, right?
Quoted 9 lines
lg, clemens
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.orgiD8DBQFBPF3djBz/yQjBxz8RAmVhAKCODKNKOV9V0I59SUfQ1pp+qk88xwCg6+4p Q+JcbO+W0+6cUfSEf3z/Iwk=
=IwZQ
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Spam wrote:
| thats why we have automount. | | |> Which still needs to be setup in fstab, right?
no, in /etc/automount* actually. I have no fstab entry here and still I can mount various samba shares, cdroms, etc ...
lg, clemens
even my usbhd is automount :)
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFBPF8tjBz/yQjBxz8RApB5AKDOjvCM2foy0lTXAy3IOVXXxdBcUACgxUyn 1WlAMXdnleNZOTOwvVRQCz0=
=LX+8
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
Quoted 5 lines · Clemens Schwaighofer
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Spam wrote:
Quoted 7 lines
| thats why we have automount. | | |> Which still needs to be setup in fstab, right?no, in
/etc/automount*actually. I have no fstab entry here and still I can mount various samba shares, cdroms, etc ...
but can you access images that way?
Signature · 1 line
Frank
Re: The argument for fs assistance in handling archives
Quoted 14 lines · Clemens Schwaighofer
Quoted 10 lines
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Spam wrote:
| thats why we have automount. | | |> Which still needs to be setup in fstab, right?
Quoted 4 lines
Quoted 2 lines
no, in
/etc/automount*actually. I have no fstab entry here and still I can mount various samba shares, cdroms, etc ...but can you access images that way?
Probably, but only predefined ones.. IE, no random automount?
~S
Re: The argument for fs assistance in handling archives
In reply to Frank van Maarseveen-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Frank van Maarseveen wrote:
| On Mon, Sep 06, 2004 at 09:59:25PM +0900, Clemens Schwaighofer wrote: | |>-----BEGIN PGP SIGNED MESSAGE----- |>Hash: SHA1 |> |>Spam wrote: |> |>| thats why we have automount. |>| |>| |>|> Which still needs to be setup in fstab, right? |> |>no, in /etc/automount* actually. I have no fstab entry here and still I |>can mount various samba shares, cdroms, etc ... | | | but can you access images that way? |
in theory yes, but only predefined ones. But I don't see automount for this. more for cdroms and other removable medias.
I haven't had the need to mount and iso for quite some time, so I don't know if I need root access for that, but I think yes like for all mount things.
lg, clemens
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFBPGNgjBz/yQjBxz8RAsm7AJsEwa18ymMBWKf1rJ2PAwPOfCMIJgCglaNS d8OqXmD0jsYezcE8u7BGrkM=
=ctEE
-----END PGP SIGNATURE-----
Re: The argument for fs assistance in handling archives
In reply to Clemens SchwaighoferSalut,
Quoted 1 line · Clemens Schwaighofer
thats why we have automount.
Plus the HAL automounting code which dynamically generates mountpoints for HAL detected devices...
Tonnerre
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Frank van MaarseveenQuoted 4 lines · Frank van Maarseveen
Can it do this:
cd FC2-i386-disc1.iso ls
That one's at least theoretically doable, assuming that it really *IS* the Fedora Core disk and an ISO9660 format...
Quoted 2 lines
cd
/dev/cdrom
ls
And the CD in the drive at the moment is AC/DC "Back in Black". What should this produce as output?
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 6 lines · On Thu, Sep 02, 2004 at 07:19:47PM -0400, [email protected] wrote:
Quoted 3 lines
cd FC2-i386-disc1.iso lsThat one's at least theoretically doable, assuming that it really *IS* the Fedora Core disk and an ISO9660 format...
Impressive.
But when it comes to file-systems like ext[23] I think an in-kernel solution might be preferable to get the exact semantics wrt locking, atomicity, synchronization, coherency, whatever a kernel is good at for filesystems.
Quoted 5 lines
Quoted 2 lines
cd
/dev/cdrom
lsAnd the CD in the drive at the moment is AC/DC "Back in Black". What should this produce as output?
bash: cd: /dev/cdrom: Not a directory
When it doesn't match a (sub)set of known file system types I think. Because that is the area the kernel has knowledge about. Nothing else, no tarballs for example unless of course the kernel has "tarfs" :-)
Signature · 1 line
Frank
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to [email protected]Quoted 10 lines · Frank van Maarseveen
Quoted 5 lines
Can it do this:
cd FC2-i386-disc1.iso ls
Quoted 8 lines
That one's at least theoretically doable, assuming that it really *IS* the Fedora Core disk and an ISO9660 format...
Quoted 2 lines
cd
/dev/cdrom
lsAnd the CD in the drive at the moment is AC/DC "Back in Black". What should this produce as output?
Yes why not? If there was any filesystem drivers for the AudioCD format then it could.
I had such a driver for Windows 9x which would display several folders and files for inserted AudioCD's:
D: (cdrom)
Stereo
22050
Track01.wav
Track02.wav
...
44100
Track01.wav
...
Mono
22050
Track01.wav
...
44100
Track01.wav
...Normal AudioCD players would also work even though this driver was installed. These files were also visible for legacy applications in the command prompt (inside Windows).
I do not see why this would not be possible in Linux. Of course, it would perhaps require a filesystem driver/module to be present when you mount.
If you just want to do a cd file.iso then it may be a totally different thing. Either you would have a automount feature or a filesystem/vfs plugin that could load secondary modules to support this kind of thing.
~S
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 16 lines · Spam
Yes why not? If there was any filesystem drivers for the AudioCD format then it could.
I had such a driver for Windows 9x which would display several folders and files for inserted AudioCD's:
D: (cdrom)
Stereo 22050 Track01.wav Track02.wav ... 44100 Track01.wav ...
...
Quoted 5 lines
If you just want to do a cd file.iso then it may be a totally different thing. Either you would have a automount feature or a filesystem/vfs plugin that could load secondary modules to support this kind of thing.
Why is this so different, compared to your example? They are both filesystem drivers.
Signature · 1 line
Frank
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to SpamSalut,
Quoted 7 lines · Spam
Yes why not? If there was any filesystem drivers for the AudioCD format then it could.
I had such a driver for Windows 9x which would display several folders and files for inserted AudioCD's:
D: (cdrom)
Quoted 15 lines
Stereo
22050
Track01.wav
Track02.wav
...
44100
Track01.wav
...
Mono
22050
Track01.wav
...
44100
Track01.wav
...So you'd like the kernel to know about raw CD PCM and RIFF PCM format and conversion? Great.. That's really Solarisish!
Tonnerre
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 29 lines
Salut,
Quoted 7 lines · Spam
Yes why not? If there was any filesystem drivers for the AudioCD format then it could.
I had such a driver for Windows 9x which would display several folders and files for inserted AudioCD's:
D: (cdrom)
Quoted 15 lines
Stereo 22050 Track01.wav Track02.wav ... 44100 Track01.wav ... Mono 22050 Track01.wav ... 44100 Track01.wav ...
Quoted 2 lines
So you'd like the kernel to know about raw CD PCM and RIFF PCM format and conversion? Great.. That's really Solarisish!
If it could be done in userland, and still be accessible for most applications then that is great.
If the user things that such a plugin is of value for him then I do not see why he should not be able to use it? Again these are example of stuff that could be done if there was a plugin/extensible interface.
You make it sound like everything on the planet should be predefined and included in the kernel. This is not what I was saying or wanted.
From what I understood from Hans, there will be a way to load plugins without having to recompile reiser4 module (or the kernel). This is what I'd like to see.
Some things/plugins may even be partially user-space.
~S
Quoted 1 line
Tonnerre
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to [email protected]Am 03.09.2004 01:19:47 schrieb(en) [email protected]:
Quoted 11 lines · Frank van Maarseveen
Quoted 5 lines
Can it do this:
cd FC2-i386-disc1.iso lsThat one's at least theoretically doable, assuming that it really *IS* the
Fedora Core disk and an ISO9660 format...
Quoted 6 lines
Quoted 2 lines
cd
/dev/cdrom
lsAnd the CD in the drive at the moment is AC/DC "Back in Black". What should this produce as output?
With cdfs (http://www.elis.ugent.be/~ronsse/cdfs/)
you get something like:
track-01.wav track-03.wav track-05.wav track-02.wav track-04.wav track-06.wav
This could easily be coupled with cddb for the real titles.
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to [email protected]Quoted 6 lines · On Thu, Sep 02, 2004 at 07:19:47PM -0400, [email protected] wrote:
Quoted 2 lines · Frank van Maarseveen
cd
/dev/cdrom
lsAnd the CD in the drive at the moment is AC/DC "Back in Black". What should this produce as output?
Hehe, cdfs already figured that one out. Ofcourse you show the individual tracks as .wav files.
Jan
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 10 lines · Jan Harkes
Quoted 6 lines · On Thu, Sep 02, 2004 at 07:19:47PM -0400, [email protected] wrote:
Quoted 2 lines · Frank van Maarseveen
cd
/dev/cdrom
lsAnd the CD in the drive at the moment is AC/DC "Back in Black". What should this produce as output?
Hehe, cdfs already figured that one out. Ofcourse you show the individual tracks as .wav files.
That's sidestepping the *real* issue - which is that you need reasonable semantics even if you don't have cdfs or whatever special gee-wizz-bang driver handy.
Consider an embedded system - it may have iso9660 support, and boot off a CD like Knoppix, but not have cdfs. What do you do then?
(For bonus points, figure out the security issues involved in dealing with an intentionall corrupted image on a CD......)
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
In reply to Frank van MaarseveenSalut,
Quoted 7 lines · Frank van Maarseveen
Can it do this:
cd FC2-i386-disc1.iso lsor this:
Quoted 8 lines
cd
/dev/sda1
ls
cd/dev/floppy
ls
cd/dev/cdrom
ls?
Actually I see some
small-security-hole-you-can-drive-a-big-yellow-truck-with-flashes-on-through problem.
$ cat fs_header owner_root flags_with_suid evil_program > evil.iso $ ls -l evil.iso/evil_program -rwsr-xr-x 1 root root 24 2003-11-09 16:41 evil.iso/evil_program $ evil.iso/evil_program Mwahaha, I got root! Copy-right violating IP... done. Hijacking administrators wife... done. Overwriting OS with Parachute 2010... done.
etc. pp.
Tonnerre
Re: The argument for fs assistance in handling archives (was: silent semantic changes with reiser4)
Quoted 3 lines · Tonnerre
$ cat fs_header owner_root flags_with_suid evil_program > evil.iso $ ls -l evil.iso/evil_program
It should of course be equivalent to a user mount: nodev nosuid etc.
Signature · 1 line
Frank