Skip to contentSkip to conversation

silent semantic changes with reiser4

DISCUSSION Christoph Hellwig 141 participants 1,073 messages Aug 24, 2004 – Sep 17, 2004 7 messages not in the archive VFS Page 3 of 5

lore
Share & tools
lore
#533 lore raw
In reply to Spam
Quoted 8 lines · Spam
Quoted 4 lines

My own order of preference is b) c) a). The fact that one filesystem will offer features which other filesystems do not and cannot offer makes me queasy for some reason.

This last sentence makes me wonder. Where is Linux heading? The idea
that a FS cannot contain features that no other FS has is very
scary.

That's not what Andrew said or meant.

Note the "cannot offer". As in "there is no way to offer them even if the filesystem could support it otherwise".

We have tons of filesystems that do things other filesystems cannot do. Most filesystems support writing to a file - despite the fact that some filesystems (iso9600 being an obvious one) cannot. The infrastructure is there in the VFS layer, and it becomes a _choice_ for the filesystem whether it offers certain capabilities.

So look at what Andrew said, again: his top choice would be (b). Let's see what that was again, shall we?

Quoted 3 lines

b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or

In other words, if reiserfs does something special, we should make standard interfaces for doing that special thing, so that everybody can do it without stepping on other peoples toes.

That doesn't mean that we'd _force_ everybody to do it. The same way we don't force iso9660 to write to a CD-ROM.

Linus

#534 lore raw
Quoted 28 lines · Spam
Quoted 8 lines
Quoted 4 lines

My own order of preference is b) c) a). The fact that one filesystem will offer features which other filesystems do not and cannot offer makes me queasy for some reason.

This last sentence makes me wonder. Where is Linux heading? The idea
that a FS cannot contain features that no other FS has is very
scary.

That's not what Andrew said or meant.

Note the "cannot offer". As in "there is no way to offer them even if the filesystem could support it otherwise".

We have tons of filesystems that do things other filesystems cannot do. Most filesystems support writing to a file - despite the fact that some filesystems (iso9600 being an obvious one) cannot. The infrastructure is there in the VFS layer, and it becomes a _choice_ for the filesystem whether it offers certain capabilities.

So look at what Andrew said, again: his top choice would be (b). Let's see what that was again, shall we?

Quoted 7 lines
Quoted 3 lines

b) accept the reiser4-only extensions with a view to turning them into kernel-wide extensions at some time in the future, so all filesystems will offer the extensions (as much as poss) or

In other words, if reiserfs does something special, we should make standard interfaces for doing that special thing, so that everybody can do it without stepping on other peoples toes.

Agreed that would be the best. But how much time and effort will it be, and how much of the original ideas would be lost on the way to implement them in the VFS? Especially with new and very advanced FS's like Reiser4.

Isn't the line between the actual file system and the virtual one very hair thin? Where would the separation lay in Reiser4?

Quoted 4 lines

That doesn't mean that we'd _force_ everybody to do it. The same way we don't force iso9660 to write to a CD-ROM.

Linus

I got caught in the moment of flame war. My appologies.

~S

#535 lore raw
Quoted 7 lines · Spam
Quoted 4 lines

In other words, if reiserfs does something special, we should make standard interfaces for doing that special thing, so that everybody can do it without stepping on other peoples toes.

Agreed that would be the best. But how much time and effort will it be

Zero.

We can add these new features tomorrow, as reiser4-only features, with a plan in hand to generalise them later.

-->>__if__<<-- we think these are features which Linux should offer.

#536 lore raw
Quoted 5 lines · Andrew Morton

We can add these new features tomorrow, as reiser4-only features, with a plan in hand to generalise them later.

-->>__if__<<-- we think these are features which Linux should offer.

Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.

Jeremy.

#537 lore raw
Quoted 3 lines · Previously Jeremy Allison

Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.

Aside from samba, is there any other application that has a use for them?

Wichert.

Signature · 2 lines
Wichert Akkerman <[email protected]>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.
#538 lore raw
Quoted 5 lines · Wichert Akkerman
Quoted 3 lines · Previously Jeremy Allison

Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.

Quoted 9 lines

Aside from samba, is there any other application that has a use for them?

Wichert.

--
Wichert Akkerman <[email protected]> It is simple to make things.
http://www.wiggy.net/ It is hard to make things simple.

It's not so much about samba having a use for them as it is about other applications using them between platforms in the same way. I'm glad to see Jeremy is on top of this because it's not something I had even thought about having available on the linux platform in the sense that my work deals with data streams on NTFS, HFS, and whatever other filesystems join the venture.

NTFS had very poor support for them for a long time. At one point I was convinced it was going to be taken out eventually, but SP2 just showed with even better support for them, and more MS software has started using them. They can't turn around at this point and take them out, so they must be planning on using them to a wider extent.

Bryan

#539 lore raw
Quoted 2 lines · Previously Tierra

It's not so much about samba having a use for them as it is about other applications using them between platforms in the same way.

But what for? Personally I still haven't seen a good use for them. Apple's resource fork comes close though.

Quoted 4 lines

NTFS had very poor support for them for a long time. At one point I was convinced it was going to be taken out eventually, but SP2 just showed with even better support for them, and more MS software has started using them.

I wonder how they intend to shoehorn that into distribution across other media such as emails or zip files.

Wichert.

Signature · 2 lines
Wichert Akkerman <[email protected]>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#541 lore raw
Quoted 2 lines · Wichert Akkerman

But what for? Personally I still haven't seen a good use for them. Apple's resource fork comes close though.

There's one way where it can really come in handy, and that's with specific application file types. Certain media could be saved in a better format and read more efficiently. The reason you haven't seen this done yet though is because multiple data streams isn't a standard and isn't available in most enviroments. There also isn't any file transfer protocols that will handle transfer of multiple data streams except through Windows File Sharing.

Take into account though that I have absolutely no experience or knowledge of how it all works in the backend, and don't know the implications involved in the implementation.

As for applications that already use data streams, you mentioned Apple's resource fork already, so I assume you know about file types there. IIS currently uses ADS for image thumbnailing and it's also possible to hide an alternate website right inline with a public one. I've heard of a couple other applications on Windows that uses them. But you are right, none of them _appropriately_ use data streams for any specific advantage over regular files except to keep system controlled information hidden.

Bryan

#542 lore raw
In reply to Wichert Akkerman
Quoted 8 lines · Wichert Akkerman
Quoted 3 lines · Previously Jeremy Allison

Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.

Aside from samba, is there any other application that has a use for them?

Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.

As to how to do it, I think the Solaris interface is reasonably decent. The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view

(An important detail for those who want to access their multiple-data-streams from non-MDS aware apps is the runat shell command, which basically does a chdir into the specified file's attribute directory and then runs a command. i.e. 'runat ~/blah ls' will list the ~/blah's attributes.)

The only real problem I have with their design is the calling them attributes and using "at" everywhere.

"Attributes", because it will get confused with the current Linux xattr implementation (which is still useful for things that actually are file attributes, like security labels, ACLs, weird attributess that FAT/NTFS/whatever have, etc.).

I don't like "at" because the API changes don't have anything to do with the actual attributes. It's a general set of changes to allow paths relative to a fd instead of the cwd, and doesn't really have anything specifically to do with attributes (with the exception of the O_XATTR flag).

Replace "at" with "rel" and O_XATTR with O_FORK or O_MULTI or something, and it's all good.

#543 lore raw
Quoted 3 lines · Nicholas Miell

Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.

Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed.

Whereas there are some other kinds of metadata which should not be deleted if the file is modified.

-- Jamie

#544 lore raw
Quoted 5 lines · Jamie Lokier
Quoted 3 lines · Nicholas Miell

Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.

Quoted 8 lines

Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed.

Whereas there are some other kinds of metadata which should not be deleted if the file is modified.

-- Jamie

Presumably the app which uses the metadata will be smart enough to compare the st_mtime of the MDS/stream/attribute/whatever (can we choose a name for these things now?) to the st_mtime of the file and do the right thing.

thumbnail - regenerate it
summary - keep it, it's relatively independent of the file's exact contents
signature - always verify it
etc.

#545 lore raw
Quoted 7 lines · Nicholas Miell
Quoted 7 lines

Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed.

Whereas there are some other kinds of metadata which should not be deleted if the file is modified.

Quoted 6 lines
Quoted 1 line

-- Jamie

Presumably the app which uses the metadata will be smart enough to compare the st_mtime of the MDS/stream/attribute/whatever (can we choose a name for these things now?) to the st_mtime of the file and do the right thing.

[This is straying off the topic of files-as-directories].

st_mtime tests are weak. They break sometimes, and are not suitable for strong data models such as transparently caching generated data from a file's contents.

They're especially breakable if you change a file and then read it within a second. Sometimes, more than a second.

As has been raised before, nanosecond timestamps (a) don't solve this unless they're stored in the filesystem; (b) even then they will fail one day, on a sufficiently fast box; (c) don't work when there are changes to wall clock time.

They're fine where you don't care if the generated data is wrong sometimes.

If you want the _equivalent_ behaviour to opening and parsing the file, but fast, then they're no good.

Modification serial numbers would work, though.

-- Jamie

#546 lore raw
Quoted 29 lines · Jamie Lokier
Quoted 7 lines · Nicholas Miell
Quoted 7 lines

Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed.

Whereas there are some other kinds of metadata which should not be deleted if the file is modified.

Quoted 6 lines
Quoted 1 line

-- Jamie

Presumably the app which uses the metadata will be smart enough to compare the st_mtime of the MDS/stream/attribute/whatever (can we choose a name for these things now?) to the st_mtime of the file and do the right thing.

[This is straying off the topic of files-as-directories].

st_mtime tests are weak. They break sometimes, and are not suitable for strong data models such as transparently caching generated data from a file's contents.

They're especially breakable if you change a file and then read it within a second. Sometimes, more than a second.

As has been raised before, nanosecond timestamps (a) don't solve this unless they're stored in the filesystem; (b) even then they will fail one day, on a sufficiently fast box; (c) don't work when there are changes to wall clock time.

Quoted 8 lines

They're fine where you don't care if the generated data is wrong sometimes.

If you want the _equivalent_ behaviour to opening and parsing the file, but fast, then they're no good.

Modification serial numbers would work, though.

-- Jamie

Remember that the Solaris attribute model is just another filesystem tree rooted in a hidden directory associated with a regular file. If you can come up with semantics that work in general for all files, that's fine.

inodes that delete themselves when other inodes are changed creep me out.

#547 lore raw
In reply to Jamie Lokier
Quoted 9 lines · Jamie Lokier
Quoted 5 lines · Nicholas Miell

Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.

Quoted 12 lines

Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed.

Whereas there are some other kinds of metadata which should not be deleted if the file is modified.

-- Jamie

Yes, I agree.

Actually we plan to have a whole link taxonomy, and one expected feature is that some links don't count towards the refcount needed to keep an object in existence (For instance, the links between key words and text documents, you don't want to have to explicitly unlink every keyword in order to delete a document).

#548 lore raw
In reply to Jamie Lokier
Quoted 8 lines · Jamie Lokier
Quoted 3 lines · Nicholas Miell

Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.

Additionally, all of those things you describe should be deleted if the file is modified -- to indicate that they're no longer valid and should be regenerated if needed.

In principle, not all of them. For example, a document summary for a text document or a long (textual) description of a video clip might remain the same when the user is only working on the finishing details.

Maybe the metadata needs an attribute to determine how 'immutable' it should be wrt changes on the file? (Can you spell meta-meta-data <g>)

Signature · 7 lines
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)
#549 lore raw
In reply to Nicholas Miell
Quoted 4 lines · Nicholas Miell

Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.

Which is exactly how Windows apps have started to use streams.

Quoted 2 lines

As to how to do it, I think the Solaris interface is reasonably decent. The overview is at http://docs.sun.com/db/doc/816-0220/6m6nkorp9?a=view

I agree. This is an interface Samba can live with I think. I was thinking of implementing it anyway, just so I could piss off the Linux kernel developers by saying "oh if you need proper Windows semantics on Samba you have to use an advanced OS like Solaris, Linux doesn't cut it" :-) :-).

Quoted 2 lines

The only real problem I have with their design is the calling them attributes and using "at" everywhere.

Yep - they're different from xattrs. The easiest way to remember this is that file streams are seekable and get a fd, xattrs aren't and don't :-).

Jeremy.

#550 lore raw
Quoted 1 line · Previously Jeremy Allison

Which is exactly how Windows apps have started to use streams.

Interesting I can not find a way using the .NET framework, which is supposed to be The Future, to use streams.

Wichert.

Signature · 2 lines
Wichert Akkerman <[email protected]>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#552 lore raw
Quoted 5 lines · Wichert Akkerman
Quoted 1 line · Previously Jeremy Allison

Which is exactly how Windows apps have started to use streams.

Interesting I can not find a way using the .NET framework, which is supposed to be The Future, to use streams.

That's because you don't even need .NET...
http://support.microsoft.com/default.aspx?scid=http://support.microsoft.com:80/support/kb/articles/q105/7/63.asp&NoWebContent=1&NoWebContent=1

http://tinyurl.com/45mfa

Bryan

#553 lore raw
In reply to Nicholas Miell
Quoted 6 lines · Nicholas Miell
Quoted 5 lines · Wichert Akkerman
Quoted 3 lines · Previously Jeremy Allison

Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.

Quoted 7 lines
Quoted 3 lines

Aside from samba, is there any other application that has a use for them?

Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.

That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already.

While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE.

A file is a stream of bytes.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#554 lore raw
Quoted 21 lines · Matt Mackall
Quoted 6 lines · Nicholas Miell
Quoted 5 lines · Wichert Akkerman
Quoted 3 lines · Previously Jeremy Allison

Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.

Quoted 7 lines
Quoted 3 lines

Aside from samba, is there any other application that has a use for them?

Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.

That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already.

Quoted 6 lines

While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE.

A file is a stream of bytes.

"OMG! It breaks tar and email!!!" argument doesn't fly. Things break all the time and are fixed. It's called progress.

cp, grep, cat, and tar will continue to work just fine on files with multiple streams.

tar and cp will lose the extra streams until somebody fixes them, but they lose ACLs and xattrs right now, and I don't hear anybody suggesting that ACLs and xattrs be removed from the kernel because of this.

Fixing programs that do recursive filesystem traversals is a matter of a glibc patch to nftw(3) and a modification to their option processing.

Holding back a useful feature because you don't want to upgrade coreutils is just plain dumb.

(BTW, for email, multipart/parallel is a start, but a specific multipart content type for multi-stream file attachments would probably be more appropriate.)

#555 lore raw
Quoted 3 lines · Nicholas Miell

(BTW, for email, multipart/parallel is a start, but a specific multipart content type for multi-stream file attachments would probably be more appropriate.)

I don't understand why something like this is so important that it needs underlying VFS support. Multipart messages seem to have survived fine up until now. Where is the justification?

- James

Signature · 2 lines
James Morris
<[email protected]>
#556 lore raw
In reply to Nicholas Miell
Quoted 27 lines · Nicholas Miell
Quoted 26 lines · Matt Mackall
Quoted 6 lines · Nicholas Miell
Quoted 5 lines · Wichert Akkerman
Quoted 3 lines · Previously Jeremy Allison

Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.

Quoted 7 lines
Quoted 3 lines

Aside from samba, is there any other application that has a use for them?

Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.

That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already.

While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE.

Quoted 4 lines
Quoted 1 line

A file is a stream of bytes.

"OMG! It breaks tar and email!!!" argument doesn't fly. Things break all the time and are fixed. It's called progress.

What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was.

Progress? No, this has been done before. Various dead operating systems have done it or similar and regretted it. Most recently MacOS, which jumped through major hurdles to begin purging themselves of resource forks when they went to OS X. They're still there, but heavily deprecated.

Quoted 2 lines

cp, grep, cat, and tar will continue to work just fine on files with multiple streams.

Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?

I like cat < a > b. You can keep your progress.

Signature · 1 line
Mathematics is the supreme nostalgia of our time.
#557 lore raw
Quoted 19 lines
Quoted 4 lines
Quoted 1 line

A file is a stream of bytes.

"OMG! It breaks tar and email!!!" argument doesn't fly. Things break all the time and are fixed. It's called progress.

What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was.

Progress? No, this has been done before. Various dead operating systems have done it or similar and regretted it. Most recently MacOS, which jumped through major hurdles to begin purging themselves of resource forks when they went to OS X. They're still there, but heavily deprecated.

Quoted 2 lines

cp, grep, cat, and tar will continue to work just fine on files with multiple streams.

Quoted 7 lines

Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?

I like cat < a > b. You can keep your progress.

cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions.

Signature · 1 line
vda
#558 lore raw
Quoted 4 lines · Denis Vlasenko
Quoted 1 line

I like cat < a > b. You can keep your progress.

cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions.

Losing permissions is one thing. Annoying, mostly.

However, actual losing file data during such a copy is nothing short of a disaster, IMHO.

In my opinion we shouldn't merge file-as-a-directory semantics into the kernel until we figure out how to fix the backup/restore problem and keep standard unix utilities work.

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#559 lore raw

Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:

Quoted 7 lines
Quoted 4 lines
Quoted 1 line

I like cat < a > b. You can keep your progress.

cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions.

Losing permissions is one thing. Annoying, mostly.

Quoted 7 lines

However, actual losing file data during such a copy is nothing short of a disaster, IMHO.

In my opinion we shouldn't merge file-as-a-directory semantics into the kernel until we figure out how to fix the backup/restore problem and keep standard unix utilities work.

Well, again, what about xattrs and ACLs...?

Actually, reiser4 doesn't currently implement storage of arbitrary user data under a file.

It's just some sysfs-like information that can be retrieved, some properties can be changed. If you copy a file the worst thing that can happen right now is that you lose the information whether that file should be encrypted or compressed. You don't lose data.

In case some people are wondering, this is what you can find in the reiser4 metas:

leto:/home/chtephan/.muttrc/metas > la
insgesamt 1

dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 .
-rwx------  1 chtephan users 290 12. Jan 2004  ..
-r--r--r--  1 chtephan users   0 26. Aug 13:52 bmap
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 gid
-r--r--r--  1 chtephan users   0 26. Aug 13:52 items
-r--r--r--  1 chtephan users   0 26. Aug 13:52 key
-r--r--r--  1 chtephan users   0 26. Aug 13:52 locality
--w-------  1 chtephan users   0 26. Aug 13:52 new
-r--r--r--  1 chtephan users   0 26. Aug 13:52 nlink
-r--r--r--  1 chtephan users   0 26. Aug 13:52 oid
dr-xr-xr-x  1 chtephan users   0 26. Aug 13:52 plugin
-r--r--r--  1 chtephan users   0 26. Aug 13:52 pseudo
-r--r--r--  1 chtephan users   0 26. Aug 13:52 readdir
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 rwx
-r--r--r--  1 chtephan users   0 26. Aug 13:52 size
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 uid

Some of them are reiser4 specific things (items, key, locality, oid) which can only be queried anyway.

The rest is what you can also get using other VFS calls.

leto:/home/chtephan/.muttrc/metas/plugin > la
insgesamt 0

dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 .
dr-xr-xr-x  1 chtephan users 0 26. Aug 13:52 ..
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 compression
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 crypto
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 digest
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir_item
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 fibration
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 file
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 formatting
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 hash
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 perm
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 sd

Here you can change reiser4 specific things. Here you can change some properties how the file is stored in the reiser4 tree. For example you can activate compression, encryption or authentication or set some hints to optimize speed.

You can't create or remove these pseudo-files, just like in sysfs.

I don't know, but if someone wants to store user data I think you would have to implement a directory "userdata" or something where you can store it.

Not all of the information does make a lot sense inside a modified tar. Who is interested in the bmap information of the file or some other read-only information?

#560 lore raw
Quoted 76 lines · Christophe Saout

Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

I like cat < a > b. You can keep your progress.

cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions.

Losing permissions is one thing. Annoying, mostly.

Quoted 9 lines

However, actual losing file data during such a copy is nothing short of a disaster, IMHO.

In my opinion we shouldn't merge file-as-a-directory semantics into the kernel until we figure out how to fix the backup/restore problem and keep standard unix utilities work.

Well, again, what about xattrs and ACLs...?

Actually, reiser4 doesn't currently implement storage of arbitrary user data under a file.

It's just some sysfs-like information that can be retrieved, some properties can be changed. If you copy a file the worst thing that can happen right now is that you lose the information whether that file should be encrypted or compressed. You don't lose data.

In case some people are wondering, this is what you can find in the reiser4 metas:

leto:/home/chtephan/.muttrc/metas > la
insgesamt 1
dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .

(quoted diff) +0 -9
-rwx------  1 chtephan users 290 12. Jan 2004  ..
-r--r--r--  1 chtephan users   0 26. Aug 13:52 bmap
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 gid
-r--r--r--  1 chtephan users   0 26. Aug 13:52 items
-r--r--r--  1 chtephan users   0 26. Aug 13:52 key
-r--r--r--  1 chtephan users   0 26. Aug 13:52 locality
--w-------  1 chtephan users   0 26. Aug 13:52 new
-r--r--r--  1 chtephan users   0 26. Aug 13:52 nlink
-r--r--r--  1 chtephan users   0 26. Aug 13:52 oid

dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 plugin

(quoted diff) +0 -5
-r--r--r--  1 chtephan users   0 26. Aug 13:52 pseudo
-r--r--r--  1 chtephan users   0 26. Aug 13:52 readdir
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 rwx
-r--r--r--  1 chtephan users   0 26. Aug 13:52 size
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 uid

Some of them are reiser4 specific things (items, key, locality, oid) which can only be queried anyway.

The rest is what you can also get using other VFS calls.

leto:/home/chtephan/.muttrc/metas/plugin > la
insgesamt 0
dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .
dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 ..

(quoted diff) +0 -11
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 compression
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 crypto
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 digest
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir_item
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 fibration
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 file
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 formatting
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 hash
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 perm
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 sd
Quoted 6 lines

Here you can change reiser4 specific things. Here you can change some properties how the file is stored in the reiser4 tree. For example you can activate compression, encryption or authentication or set some hints to optimize speed.

I suppose that you are talking about the prinsiple ability to change these things. Because in real life it will also need some kind of convertion from one look (say with old plugin) to another one. Just like tail conversion rebuilds file's body after tail policy plugin gets changed.

Signature · 1 line
umka
#561 lore raw
In reply to Christophe Saout
Quoted 76 lines · Christophe Saout

Am Donnerstag, den 26.08.2004, 07:42 -0400 schrieb Rik van Riel:

Quoted 9 lines
Quoted 7 lines
Quoted 3 lines

I like cat < a > b. You can keep your progress.

cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions.

Losing permissions is one thing. Annoying, mostly.

Quoted 9 lines

However, actual losing file data during such a copy is nothing short of a disaster, IMHO.

In my opinion we shouldn't merge file-as-a-directory semantics into the kernel until we figure out how to fix the backup/restore problem and keep standard unix utilities work.

Well, again, what about xattrs and ACLs...?

Actually, reiser4 doesn't currently implement storage of arbitrary user data under a file.

It's just some sysfs-like information that can be retrieved, some properties can be changed. If you copy a file the worst thing that can happen right now is that you lose the information whether that file should be encrypted or compressed. You don't lose data.

In case some people are wondering, this is what you can find in the reiser4 metas:

leto:/home/chtephan/.muttrc/metas > la
insgesamt 1
dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .

(quoted diff) +0 -9
-rwx------  1 chtephan users 290 12. Jan 2004  ..
-r--r--r--  1 chtephan users   0 26. Aug 13:52 bmap
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 gid
-r--r--r--  1 chtephan users   0 26. Aug 13:52 items
-r--r--r--  1 chtephan users   0 26. Aug 13:52 key
-r--r--r--  1 chtephan users   0 26. Aug 13:52 locality
--w-------  1 chtephan users   0 26. Aug 13:52 new
-r--r--r--  1 chtephan users   0 26. Aug 13:52 nlink
-r--r--r--  1 chtephan users   0 26. Aug 13:52 oid

dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 plugin

(quoted diff) +0 -5
-r--r--r--  1 chtephan users   0 26. Aug 13:52 pseudo
-r--r--r--  1 chtephan users   0 26. Aug 13:52 readdir
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 rwx
-r--r--r--  1 chtephan users   0 26. Aug 13:52 size
-rw-r--r--  1 chtephan users   0 26. Aug 13:52 uid

Some of them are reiser4 specific things (items, key, locality, oid) which can only be queried anyway.

The rest is what you can also get using other VFS calls.

leto:/home/chtephan/.muttrc/metas/plugin > la
insgesamt 0
dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 .
dr-xr-xr-x 1 chtephan users 0 26. Aug 13:52 ..

(quoted diff) +0 -11
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 compression
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 crypto
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 digest
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 dir_item
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 fibration
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 file
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 formatting
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 hash
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 perm
-rwxr-xr-x  1 chtephan users 0 26. Aug 13:54 sd
Quoted 6 lines

Here you can change reiser4 specific things. Here you can change some properties how the file is stored in the reiser4 tree. For example you can activate compression, encryption or authentication or set some hints to optimize speed.

I suppose that you are talking about the principle ability to change these things. Because in real life it will also need some kind of conversion from one look (say with old plugin) to another one. Just like tail conversion rebuilds file's body after tail policy plugin gets changed.

Signature · 1 line
umka
#562 lore raw
In reply to Rik van Riel
Quoted 4 lines · Rik van Riel
Quoted 2 lines

cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions.

Losing permissions is one thing. Annoying, mostly.

I find losing mtime annoying. That's why I don't use Mozilla's download manager (and Navigator's before it) because they presumed that mtime isn't worth copying. ncftp and wget do the right thing, but it's annoying that I can't use the fancy tools and get the metadata _I_ want transferred by them.

Quoted 2 lines

However, actual losing file data during such a copy is nothing short of a disaster, IMHO.

Note that file-as-directory doesn't imply that you can store just anything into those directories.

Is it a problem to decree that "file data MUST NOT be stored in a metadata directory; only non-essential metadata and data computed from the file data may be stored"?

Quoted 4 lines

In my opinion we shouldn't merge file-as-a-directory semantics into the kernel until we figure out how to fix the backup/restore problem and keep standard unix utilities work.

Same as xattrs, for data that should be stored.

But, unlike xattrs, while permissions and C-source-encoding should be stored, some of this metadata should _not_ be stored when making an archive and perhaps not when making a backup either.

Personally I wouldn't want an archive containing a file to save and restore recomputable metadata like cached, extracted-on-demand MP3 id tags and email headers-as-metadata. I'd rather they were recalculated when needed, implying a stronger guarantee about their relationship to the data from which they're extracted - and implying that a user program might not be allowed to set them to inconsistent values.

-- Jamie

#563 lore raw
Quoted 3 lines · Previously Jamie Lokier

I find losing mtime annoying. That's why I don't use Mozilla's download manager (and Navigator's before it) because they presumed that mtime isn't worth copying.

It's free software, you could fix that :)

Quoted 3 lines

Is it a problem to decree that "file data MUST NOT be stored in a metadata directory; only non-essential metadata and data computed from the file data may be stored"?

That would put us right back at the samba-and-other-apps-want-streams point in the discussion, but it certainly would not be a bad starting point. From there we could always add streams later on if the API is right.

Wichert.

Signature · 2 lines
Wichert Akkerman <[email protected]>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.
#564 lore raw
In reply to Jamie Lokier
Quoted 8 lines · Jamie Lokier

Note that file-as-directory doesn't imply that you can store just anything into those directories.

Is it a problem to decree that "file data MUST NOT be stored in a metadata directory; only non-essential metadata and data computed from the file data may be stored"?

That's exactly what folks seem to lost in this debate.
We (developers) want to have files as dirs to:
* be able to extract/modify file part on fly. For instance, simple .iso file, without need to mount it over loop (btw, if someone is going to redesign VFS, can we handle that kinda case too please?), .tar, or unzipped (kernel can handle zips) "streams". File is a stream, why shouldn't I have a chance to grab a stream out of archive/image than ?
* be able to store metadata, that doesn't matter on copy, and even should not be copied sometimes. Things like thumbnails, metainformation (used for search). All of these extracted from acctual file (either by VFS 'plugins', or by userspace application), and saved there so that it can be used later on. So we save some time on extracting it.

--
GJ

#565 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Grzegorz" == Grzegorz Jaśkiewicz <[email protected]> writes:

Quoted 6 lines · Grzegorz> Jamie Lokier wrote:
Quoted 6 lines

Note that file-as-directory doesn't imply that you can store just anything into those directories.

Is it a problem to decree that "file data MUST NOT be stored in a metadata directory; only non-essential metadata and data computed from the file data may be stored"?

Grzegorz> That's exactly what folks seem to lost in this debate. Grzegorz> We (developers) want to have files as dirs to:
Grzegorz> * be able to extract/modify file part on fly. For instance, Grzegorz> simple .iso file, without need to mount it over loop (btw, if Grzegorz> someone is going to redesign VFS, can we handle that kinda Grzegorz> case too please?), .tar, or unzipped (kernel can handle zips) Grzegorz> "streams". File is a stream, why shouldn't I have a chance to Grzegorz> grab a stream out of archive/image than ?

I'm not entirely convinced that it's a good idea. At least, I agree with those who say that they don't want a tar parser in the kernel. I would only want this to happen if the parsing is handled in a user-space helper that somehow connects to the kernel.

Grzegorz> * be able to store metadata, that doesn't matter on copy, and Grzegorz> even should not be copied sometimes. Things like thumbnails, Grzegorz> metainformation (used for search). All of these extracted from Grzegorz> acctual file (either by VFS 'plugins', or by userspace Grzegorz> application), and saved there so that it can be used later Grzegorz> on. So we save some time on extracting it.

Yes. I think that file-as-dir is useful for things like when Nautilus wants to store a thumbnail, or annotations (their "emblems", notes, etc.). In the situation right now, if you copy something using cp, you lose all that information already. If file-as-dir is implemented, then when you cp on a filesystem that supports it, you don't lose your notes. If you cp to a filesystem that doesn't support it, well, you would have lost that information under the old scheme anyways.

I also think that file-as-dir is also useful for exporting new metadata, like acl's or other access control schemes. Again, if you copy to a filesystem that doesn't implement these, then you would have lost that information anyways. But it presents a nice consistent interface. But in this way, you get an easy way to access this stuff that programmers won't have to spend (much) time to learn about. (Only time spent should be learning the naming convention.)

Of course, you could do all this with openat(), but people who are reading this thread should already know my opinions on openat(), so we'll leave that out.

Signature · 9 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#566 lore raw
In reply to Grzegorz Jaśkiewicz
Quoted 7 lines · Grzegorz Ja??kiewicz
Quoted 3 lines · Jamie Lokier

Is it a problem to decree that "file data MUST NOT be stored in a metadata directory; only non-essential metadata and data computed from the file data may be stored"?

That's exactly what folks seem to lost in this debate.

And that's not even the most important part: How would you educate all the developers _not_ on this list: application developers? I think this is just a disaster waiting for us to occur.

Quoted 2 lines

* be able to extract/modify file part on fly.
* be able to store metadata, that doesn't matter on copy

those may be orthogonal.

Signature · 1 line
Frank
#567 lore raw
In reply to Grzegorz Jaśkiewicz
Quoted 8 lines · =?UTF-8?B?R3J6ZWdvcnogSmHFm2tpZXdpY3o=?= <[email protected]> said:
Quoted 6 lines · Jamie Lokier

Note that file-as-directory doesn't imply that you can store just anything into those directories.

Is it a problem to decree that "file data MUST NOT be stored in a metadata directory; only non-essential metadata and data computed from the file data may be stored"?

Quoted 7 lines

That's exactly what folks seem to lost in this debate.
We (developers) want to have files as dirs to:
* be able to extract/modify file part on fly. For instance, simple .iso file, without need to mount it over loop (btw, if someone is going to redesign VFS, can we handle that kinda case too please?), .tar, or unzipped (kernel can handle zips) "streams". File is a stream, why shouldn't I have a chance to grab a stream out of archive/image than ?

ISO9660 is designed for read-only, i.e., modifying something in the middle is probably as hard to do as just unpack/modify/repack. Same for tar, cpio, and compressed versions of same. No gain, for a major hassle implementing this.

Quoted 5 lines

* be able to store metadata, that doesn't matter on copy, and even should not be copied sometimes. Things like thumbnails, metainformation (used for search). All of these extracted from acctual file (either by VFS 'plugins', or by userspace application), and saved there so that it can be used later on. So we save some time on extracting it.

You save iff the resources spent updating this stuff is less than recomputing on the fly each time. Also, placing random stuff in the kernel is _not_ "zero cost", it is _very_ expensive (kernel memory is always resident, kernel code is _extremely_ critical, once it is in the kernel it is very hard to change interfaces). Besides, this kind of "metadata" will most probably be different for each user, and a user should also be able to associate it to files she has no write right on. I.e., keep it in a per-user directory, managed by whatever GUI tools she uses.

What you (developer) wish for and what kernel hackers are able to deliver aren't the same. As it stands this is a badly-thought-through proposal, with very complex ramifications that few here are aware of (see Al Viro's and Linus' posts about locking; it breaks the fundamental principle that something is a file or a directory, so this would break many applications), and no clean solutions as of now.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#568 lore raw
In reply to Jamie Lokier

Oh, and if I may notice something more. Just my opinion.
It's going to be 'samba uses only' thing, as long as it's not in VFS.
xattrs are not so much needed as file streams there. So argument that xattrs exists already in VFS, and are not used, so why do we need streams there, and that it wouldn't change anything is just silly.

Btw, reiserfs4 on my laptop is providing very nice base for KDE source tree (XFS sucked so much, I had to replace it). Saving me some time on recompilation, and cvs up.

Thanks.

--
GJ

#569 lore raw
In reply to Rik van Riel
Quoted 9 lines · Rik van Riel
Quoted 4 lines · Denis Vlasenko
Quoted 1 line

I like cat < a > b. You can keep your progress.

cat <a >b does not preserve following file properties even on standard UNIX filesystems: name,owner,group,permissions.

Losing permissions is one thing. Annoying, mostly.

cat <a >b does preserve permissions - the permissions of b. I always considered this a feature.

Lee

#570 lore raw
In reply to Rik van Riel
Quoted 9 lines · Rik van Riel

In my opinion we shouldn't merge file-as-a-directory semantics into the kernel until we figure out how to fix the backup/restore problem and keep standard unix utilities work.

One incremental piece at a time our solution will fall into place. Pieces should go in when they are individually ready. Warning users that some pieces might be changed out from under them up until when 2.8 ships is ok with me though.

#571 lore raw
In reply to Matt Mackall
Quoted 5 lines · Matt Mackall

What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was.

It breaks the concept of a file. In ways that offer more versatility, challenge the imagination to make even better progress and keeps Linux competing with competitors who are implementing this stuff as we speak.

I for one would truly welcome the coming of thumbnails and descriptions in picture files, because I have a real-life project going on where that would be extremely handy to have in the actual file. Were I any richer, I'd pay Namesys to have this work for me :)

Quoted 5 lines

Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?

Then I guess OS X ships a broken implementation of cp, yes?

On the cat example, what if cat < a > b simply copies the "main stream" and not the metadata, as a feature. The key being, "as a feature"

The metadata streams could get file descriptors of their own OR another program, streamcat or something, could be written to compensate.

Quoted 1 line

I like cat < a > b. You can keep your progress.

With all due respect, I hope not too many people agree with you :)

Signature · 1 line
mjt
#572 lore raw
Quoted 10 lines · Markus Törnqvist
Quoted 4 lines · Matt Mackall

What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was.

It breaks the concept of a file. In ways that offer more versatility, challenge the imagination to make even better progress and keeps Linux competing with competitors who are implementing this stuff as we speak.

Sigh, it really isn't all that groundbreaking. The only special feature is the fact that it introduces directory hardlinks.

Just turn the phrase 'file-as-dir' around and you might see the point. As far as userspace is concerned the implementation would behave 99% identical if we provided a 'dir-as-file' semantics. i.e. opening a directory without 'O_DIRECTORY' would return a filehandle to a default file within the directory. Something like open("/path/to/dir", O_RDONLY) would be translated to open('/path/to/dir/_contents', O_RDONLY).

So I could simply replace all my files with directories and move the original file contents into a '_contents' file. Applications can still open them by the same name, although the attributes would look a bit different. Poof, we have the same file-as-dir concept, but now we play fair with the existing VFS and don't introduced races/deadlocks and other issues. Also existing backup tools will be able to back up and restore these 'streams'.

(btw. the same could be implemented completely in userspace, for instance in glibc. Whenever an open call gets an EISDIR error, simply retry the open, but this time with /_contents appended).

Quoted 3 lines

I for one would truly welcome the coming of thumbnails and descriptions in picture files, because I have a real-life project going on where that would be extremely handy to have in the actual file.

Ehhh, that already exists and doesn't require meta-data streams,

    http://www.exif.org/
Quoted 7 lines
Quoted 5 lines

Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?

Then I guess OS X ships a broken implementation of cp, yes?

No cp is working just fine, there is no copy syscall in the VFS, so any file system specific features such meta data streams will have to be handled by the application.

So at some point 'cp' will have to be taught about macos resource forks, posix acls, xattrs, afs acls, reiserfs file-as-dir, freebsd attributes, samba/ntfs streams. And possibly even learn the semantical differences between these so that it can copy metadata between the various file systems types.

Jan

#573 lore raw
Quoted 4 lines · Jan Harkes

(btw. the same could be implemented completely in userspace, for instance in glibc. Whenever an open call gets an EISDIR error, simply retry the open, but this time with /_contents appended).

Yes, and no - just to make it obvious before people jump on this, a lot of things can be prototyped in user space with things like this, but once you have to deal with races and mixed tool environments, user space suddenly doesn't work so well any more.

I think Jan understands this distinction, I just wanted to make sure everybody else is aware of the _one_ thing that kernel land does well:

- safely synchronize globally visible data structures

That's quite fundamental. 99% of what a kernel does is exactly that. TCP would be in user space too, if it wasn't for _exactly_ this issue. A lot of people think that kernels are about hardware access, and yes, that's the other 99% of the picture (I see the _big_ picture, remember?), but the "safe access to common data" is really very fundamental.

The kernel is literally the thing that makes sure that you don't have - and _cannot_ have - user programs that confuse each other by modifying data unsynchronized.

For example, a filesystem is really nothing but a way to access a disk in a controlled manner - it's not so much about hardware access, as it is about maintaining a coherent view of how some shared data (disk or whatever) is maintained.

Same goes for caches. We could cache things in user space, but if you want to _share_ your caches (so that you don't have to re-load them for every new application), you need some entity that manages those shared data structures in a secure manner. In other words, you need the kernel.

The same goes for something like a "container file". Whether you see it as "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are totally equivalent), the point of having the capability in the kernel is not that the operations cannot be done in user space - the point is that they cannot be done in user space _safely_. The kernel is kind of the thing that guarantees that everybody follows the rules.

Imagine the security problems if a set-uid program were to (unwittingly) depend on a user-space library that implements what Jan's prototype library would do? Races galore, since a user-space implementation wouldn't have _any_ way to do tests like the above atomically.

Linus

#574 lore raw

I trimmed/adjusted the CC list somewhat in the hope of getting a more productive discussion going.

Quoted 10 lines · Linus Torvalds
Quoted 4 lines · Jan Harkes

(btw. the same could be implemented completely in userspace, for instance in glibc. Whenever an open call gets an EISDIR error, simply retry the open, but this time with /_contents appended).

Yes, and no - just to make it obvious before people jump on this, a lot of things can be prototyped in user space with things like this, but once you have to deal with races and mixed tool environments, user space suddenly doesn't work so well any more.

Agreed.

Quoted 3 lines

The same goes for something like a "container file". Whether you see it as "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are totally equivalent), the point of having the capability in the kernel is

Only to some point. I was mostly trying to make the point that similar functionality is already achievable within the current VFS framework.

However, files and directories still have some semantical differences, and these are probably the points that we would have to address to get Hans's efforts work well in cooperation with the Linux VFS.

We cannot atomically unlink a dir-as-file, but can unlink a file-as-dir. In this case, the dcache seems to have no problems unlinking/unhashing a non-empty directory, so I'm assuming that the unlink problem is mostly enforced to avoid unreachable, but still allocated files in a file system. (I'm sure that Viro can probably come up with other reasons why unlinking a non-empty directory can be evil).

Also, we cannot create hardlinks to a directory (-as-file), but I guess reiserfs4 allows hardlinks to files (-as-directory). Luckily as far as hardlinked directories are concerned the file-as-dir implementation should be a relatively benign subset. I assume that we are not allowed to rename or link arbitrary objects from the regular fs into it, Coda returns EXDEV in such situations. As a result there will be no cycles and this simple premise should get rid of at least a couple of the possible deadlock scenarios that I've seen mentioned.

Jan

#575 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

- safely synchronize globally visible data structures
That's quite fundamental. 99% of what a kernel does is exactly that. TCP would be in user space too, if it wasn't for _exactly_ this issue. A lot

What about microkernels? They do tcp in userspace. So did winsock, iirc. As long as a trusted process keeps data such as free ports, what's the problem? Perhaps a process could even simply hand off proto:port handles and let a library do the rest, but i know little about exokernels. (There's probably some catches.)

Signature · 2 lines
Tom Vier <[email protected]>
DSA Key ID 0x15741ECE
#576 lore raw
Quoted 6 lines · Tom Vier
Quoted 3 lines · Linus Torvalds

- safely synchronize globally visible data structures
That's quite fundamental. 99% of what a kernel does is exactly that. TCP would be in user space too, if it wasn't for _exactly_ this issue. A lot

What about microkernels? They do tcp in userspace.

No they don't. They do TCP in a separate address space from user space, that just also happens to be separate from the "microkernel address space".

So a microkernel will have _more_ address spaces, and they won't be "user space". They'll be "server deamon space" or something. Now, that's also why they tend to have performance problems - because you need to copy the data between different address spaces, and switch the CPU context etc around.

Not user space. They may be "ring 3" from a CPU standpoint, but they aren't user space from a _user_ standpoint - it's still very much a separate address space, with domain protection.

Quoted 1 line

So did winsock, iirc.

Now that is a different case. Things like the PalmOS TCP stack (and, I believe, Winsock) are true "user space" TCP stacks, in that they really do live as libraries in the same address space as the user app.

It sucks. Exactly because now the data structures are _not_ protected, and they are _not_ shared. So your library basically ends up being a "single client" library, with no protection between clients (or no sharing: you can have "protected" single-stream TCP, but then you won't share the TCP state that needs to be shared like listen queues etc).

This works in an environment like Palm or Win-3.0, which are really just single-client _anyway_, without any protection. But notice how Windows started doing TCP in the kernel, and notice how you can actually use it as a server these days?

In short: you _need_ to have a separate address space (either kernel, or "TCP server" or whatever) if you want to have reliable, secure and generally usable TCP.

Quoted 2 lines

As long as a trusted process keeps data such as free ports, what's the problem?

None - because it's not user space any more.

Well, performance might still suck, of course. And it does.

Linus

#577 lore raw
Quoted 13 lines · Linus Torvalds
Quoted 7 lines · Tom Vier
Quoted 3 lines · Linus Torvalds

- safely synchronize globally visible data structures
That's quite fundamental. 99% of what a kernel does is exactly that. TCP would be in user space too, if it wasn't for _exactly_ this issue. A lot

What about microkernels? They do tcp in userspace.

Quoted 7 lines

No they don't. They do TCP in a separate address space from user space, that just also happens to be separate from the "microkernel address space".

So a microkernel will have _more_ address spaces, and they won't be "user space". They'll be "server deamon space" or something. Now, that's also why they tend to have performance problems - because you need to copy the data between different address spaces, and switch the CPU context etc around.

Man, this thread came a long way. I peek in and see a pet topic...

_My_ own definition of microkernel, which differs from traditional CS, is where I would love to see Linux go: move as much as humanly possible to userspace, such that, the kernel only contains pagecache operations, driver fast paths, and security checks. Move slow paths, including ACPI probing, PCI bus walking, driver init/reset paths, some of the ioctl handling, to userspace. Be willing to accept extra context switches as a cost in all but the fast paths.

What you have left after you move all the slow paths to userspace is my version of a microkernel. The kernel is still monolithic and a single address space, but a lot smaller.

Now this (a) is likely just a pipe dream and (b) will increase complexity because each driver will _require_ a userspace component, but hey... you gotta have goals in life.

Jeff

#578 lore raw

Userspace file systems & MKs (Re: silent semantic changes with reiser4)

Quoted 7 lines · Jeff Garzik

_My_ own definition of microkernel, which differs from traditional CS, is where I would love to see Linux go: move as much as humanly possible to userspace, such that, the kernel only contains pagecache operations, driver fast paths, and security checks. Move slow paths, including ACPI probing, PCI bus walking, driver init/reset paths, some of the ioctl handling, to userspace. Be willing to accept extra context switches as a cost in all but the fast paths.

http://www.dragonflybsd.org

DragonFly BSD, the only remotely functional open source BSD project on this planet, has plans in place to push certain kernel components like their VFS layer into userspace for easier debugging, testing and other things that go with developing file systems easily. If they back it with something like C++ for doing constructor style type conversion on top of overloaded operators to back VFS data structure translation, could easily import stuff like most Linux file systems without major restructuring, say, if they had their translation library written. In this case, userspace kernel systems have some serious programming advantages over traditional kernels.

They're also pushing an async syscall model to support a non-1:1 threading system for userspace unlike what Linux has done with futexes. It'll allow for them to plug in various userspace schedulers at will for whatever scheduling policy you want dynamically, even while the program is running.

This isn't mentioning their single system image clustering goals that have abstracted various kernel systems like their VFS stuff to a messaging model which should get them proccess migration as an intrinsic part of their OS design. They've made a lot of progress over the last year and it's largely because of Matt Dillon's ability to create a non-overly political group that creates trust. He's a true leader, not some random loud mouth(s) unlike other BSD efforts. It's much like the Linux community in this regard with Linus.

bill

#579 lore raw

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

Quoted 3 lines · Bill Huey (hui)

DragonFly BSD, the only remotely functional open source BSD project on this planet, has plans in place to push certain kernel components like their VFS layer into userspace for easier debugging, testing and other things that go

That violates Jeff's First Rule, put the fast path stuff in the kernel.

Quoted 6 lines

with developing file systems easily. If they back it with something like C++ for doing constructor style type conversion on top of overloaded operators to back VFS data structure translation, could easily import stuff like most Linux file systems without major restructuring, say, if they had their translation library written. In this case, userspace kernel systems have some serious programming advantages over traditional kernels.

Sounds like security would be a pain in the ass :)

Quoted 2 lines

They're also pushing an async syscall model to support a non-1:1 threading system for userspace unlike what Linux has done with futexes. It'll allow

messaging passing is also known as "really slow C function calls"

My PCI bus passes messages all the time. A message is in the eye of the beholder.

Jeff

#580 lore raw

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

Quoted 6 lines · Jeff Garzik
Quoted 3 lines · Bill Huey (hui)

DragonFly BSD, the only remotely functional open source BSD project on this planet, has plans in place to push certain kernel components like their VFS layer into userspace for easier debugging, testing and other things that go

That violates Jeff's First Rule, put the fast path stuff in the kernel.

It's not for native file system, but one that are coming around. I kind of misspoke when I said that the VFS layer is going to be pushed into userspace. It's not. Access to the VFS layer is going to be exported to userspace. The system is interesting because of its potential programming flexibility.

Quoted 9 lines
Quoted 7 lines

with developing file systems easily. If they back it with something like C++
for doing constructor style type conversion on top of overloaded operators to back VFS data structure translation, could easily import stuff like most Linux file systems without major restructuring, say, if they had their translation library written. In this case, userspace kernel systems have some serious programming advantages over traditional kernels.

Sounds like security would be a pain in the ass :)

Don't know. But if you can get away from the typical C programming errors, then it could be a win.

Quoted 7 lines
Quoted 2 lines

They're also pushing an async syscall model to support a non-1:1 threading system for userspace unlike what Linux has done with futexes. It'll allow

messaging passing is also known as "really slow C function calls"

My PCI bus passes messages all the time. A message is in the eye of the beholder.

Message passing is used as a method of concurrency in that system outside of the typical use of function calles. It makes access to kernel facilities, remote or local, transparent. They also don't use a mutex for their locking, but something called a token which is per-CPU kind of locking primitive. It's definitely an interesting system with a lot of potential.

bill

A message that is not in the archive (sent off-list or not yet received): [email protected]

#582 lore raw

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

Quoted 9 lines · Diego Calleja

El Tue, 31 Aug 2004 01:56:45 -0400 Jeff Garzik <[email protected]> escribi?:

Quoted 4 lines

messaging passing is also known as "really slow C function calls"

My PCI bus passes messages all the time. A message is in the eye of the beholder.

There're some numbers for that:

http://lists.freebsd.org/pipermail/freebsd-hackers/2003-August/002426.html

As you can see the numbers are very fast for a general purpose system like that. Add that with their XIO framework for data propagation and you have a really amazingly fast system that combines both aggressive design and optimized low level facilities for both remote and local commuication.

Matt's one of the very few open source engineers on this planet that can do both well.

bill

#583 lore raw

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

Quoted 2 lines · El Tue, 31 Aug 2004 13:52:11 -0700 Bill Huey (hui) <[email protected]> escribió:

As you can see the numbers are very fast for a general purpose system like that. Add that with their XIO framework for data propagation and

For the syscall case, I remember that they would be able to "batch" syscalls, so this framework could have even better performance in some cases. The good thing is that they didn't do it just for the sake of doing it (like microkernel people) but to achieve their goals (SSI, etc)

#584 lore raw

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

Quoted 9 lines · Diego Calleja

El Tue, 31 Aug 2004 13:52:11 -0700 Bill Huey (hui) <[email protected]> escribi?:

Quoted 2 lines

As you can see the numbers are very fast for a general purpose system like that. Add that with their XIO framework for data propagation and

For the syscall case, I remember that they would be able to "batch" syscalls, so this framework could have even better performance in some cases. The good thing is that they didn't do it just for the sake of doing it (like microkernel people) but to achieve their goals (SSI, etc)

For what they're doing, they have minimize the overhead cross boundaries as much as possible for usage in both user and kernel spaces. If dfBSD achieves what they intend to do, they'll have on of the best SSI systems on the planet and it'll be able to use the VFS layer in ways that other systems can't even touch with iSCSI drivers, etc...

They've got process suspend and restore already in order to support serialization across machines. Having it VFS driven, VM objects, etc... makes the concept of migration a function of the VFS layer. It's quite clever and I'm sure that it's driven by Matt's experience with distributed databases, specifically Backplane.

All locking in his system is CPU local, no mutexes of any sort, so migration as a function of messaging, scheduling and other things are done in per-CPU chunks. It's quite clever make handling in per-process resources chunks.

	http://twiki.im.ufba.br/pub/MAT154/WebHome/duality78.pdf

Talks about part of this and how "procedural" is isomorphic to "message-passing" concurrency system. One can be transformed to another.

It's a project that's move much faster in the area of SMP than FreeBSD 5.x (Solaris style SMP) and really the only BSD project on this planet that has the talent and social chemistry to make it happen.

bill

#586 lore raw

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

In reply to Bill Huey (hui)

Salut,

Quoted 9 lines · Bill Huey

DragonFly BSD, the only remotely functional open source BSD project on this planet, has plans in place to push certain kernel components like their VFS layer into userspace for easier debugging, testing and other things that go with developing file systems easily. If they back it with something like C++ for doing constructor style type conversion on top of overloaded operators to back VFS data structure translation, could easily import stuff like most Linux file systems without major restructuring, say, if they had their translation library written. In this case, userspace kernel systems have some serious programming advantages over traditional kernels.

Uh, what about enhancing lufs code in this regard and porting all fs'es to it?

Tonnerre

#587 lore raw

Re: Userspace file systems

Il Tue, 31 Aug 2004 11:01:58 +0200, Tonnerre <[email protected]> ha scritto:

Quoted 2 lines

Uh, what about enhancing lufs code in this regard and porting all fs'es to it?

I think that porting all file systems to LUFS is a mad idea :), but providing a facility in the kernel for user-space file system implementations could be great.

I am working on PackageFS (http://packagefs.sourceforge.net)
which is an user-space file system to manage packages:
it relies on LUFS, but I plan to port it soon to FUSE ( http://sourceforge.net/projects/avf)
that, at present, is maintained by Miklos Szeredi and works with latest kernels.

PackageFS relies also on existent package managers, it will be transparent to all package formats. I. e.: you can install a package by copying,
and see package informations by a simple cat.
Now it works only in Debian distro, and it is far from complete (it is 0.09 version :) ).

So, what do you think about including a kernel module to make a user able to develop, install and use an user-space file system ?

Maybe Miklos Szeredi will write more details about FUSE which is the interesting part for kernel inclusion.

Bye
Luca

Signature · 6 lines
-----------------------------------------------
They'll never stole us our.... FREEDOM!!!
Luca Ferroni
ICQ #317977679
www.cs.unibo.it/~fferroni/
-----------------------------------------------
#588 lore raw

Re: Userspace file systems & MKs (Re: silent semantic changes with reiser4)

In reply to Tonnerre
Quoted 3 lines · Tonnerre

Salut,

On Mon, Aug 30, 2004 at 10:30:55PM -0700, Bill Huey wrote:

...

Quoted 4 lines
Quoted 2 lines

translation library written. In this case, userspace kernel systems have some serious programming advantages over traditional kernels.

Uh, what about enhancing lufs code in this regard and porting all fs'es to it?

I wasn't aware that your project has gotten this far. I'll tell Hiten about this to see if some of this can be used in their project.

Thanks

bill

#589 lore raw
In reply to Jeff Garzik
Quoted 3 lines · Jeff Garzik

Man, this thread came a long way.

You said it -- from Reiser to microkernel. If we can only figure out a way to get a BitKeeper discussion going, we'll have the Grand Unified Flamewar.

#590 lore raw
Quoted 9 lines · Ryan Breen
Quoted 2 lines · Jeff Garzik

Man, this thread came a long way.

You said it -- from Reiser to microkernel. If we can only figure out a way to get a BitKeeper discussion going, we'll have the Grand Unified Flamewar.

If you can find a link between the reiserfs thread and the webcam driver thread, you've gone all the way from filesystems to Nazis, thus invoking Godwin's Law ;-)

Jeff

[1] http://en.wikipedia.org/wiki/Godwin's_law

#591 lore raw
In reply to Ryan Breen

On Tue, 31 Aug 2004, Ryan Breen verbalised:

Quoted 8 lines · Jeff Garzik
Quoted 3 lines

Man, this thread came a long way.

You said it -- from Reiser to microkernel. If we can only figure out a way to get a BitKeeper discussion going, we'll have the Grand Unified Flamewar.

OK, I'll byte: consider file versioning of an SCM implemented as a ReiserFS 4 plug-in, as in:

diff -u ultrastore.c/delta/3.2.3 ultrastore.c/delta/3.2.4

(Brrr! Just had a VMS flash-back shiver.)

Signature · 5 lines
Clay Haapala ([email protected]) Cisco Systems Storage +1 763-398-1056
   6450 Wedgwood Rd, Suite 130 Maple Grove MN 55311 PGP: C89240AD
 The mysterious blob seen in satellite photos off the east coast of Florida
 during the passing of Hurricane Charley has now been positively identified
 as a large cloud of Democrat absentee-ballot applications and voter lists.
#592 lore raw
Quoted 16 lines · Clay Haapala

On Tue, 31 Aug 2004, Ryan Breen verbalised:

Quoted 12 lines · Jeff Garzik
Quoted 6 lines

Man, this thread came a long way.

You said it -- from Reiser to microkernel. If we can only figure out a way to get a BitKeeper discussion going, we'll have the Grand Unified Flamewar.

Quoted 8 lines

OK, I'll byte: consider file versioning of an SCM implemented as a ReiserFS 4 plug-in, as in:

diff -u ultrastore.c/delta/3.2.3 ultrastore.c/delta/3.2.4

(Brrr! Just had a VMS flash-back shiver.)

hehe ;) Seems to develop this extention NAMESYS should stop use BitKeeper. Or should buy it.

Signature · 1 line
umka
#593 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds
Quoted 1 line

What about microkernels? They do tcp in userspace.

No they don't. They do TCP in a separate address space from user space, that just also happens to be separate from the "microkernel address space".

Several do TCP in user space. The only thing you need in kernel for TCP/IP is enough decode to decide who gets the packet. Even some non microkernel embedded OS's do this in order to keep kernel size down.

Alan

#594 lore raw
Quoted 3 lines · Alan Cox

Several do TCP in user space. The only thing you need in kernel for TCP/IP is enough decode to decide who gets the packet.

Only thing? I don't think so.

You also want to make sure that regular users cannot send "impossible" packets. Think about the old "ping of death" kind of thing, where a normal mis-behaving (and I'm not saying intentionally so: it might be a small bug that just overwrites some data) program should _not_ be able to cause problems on the network.

Admins absolutely _hate_ that. They will ban an OS if it sends out packets that cause troublem. You should remember that - we used to do strange things on the net (long long time ago), and we brought down servers by mistake, and nobody ever considered it a server bug: it was a Linux bug that it wouldn't do the right thing.

Things like not sending FIN-packets when a program suddenly goes away is NOT acceptable behaviour! Neither is it acceptable behaviour to allow user programs to make up their own packets.

Quoted 2 lines

Even some non microkernel embedded OS's do this in order to keep kernel size down.

..and I'm not disagreeing that it doesn't happen. I explicitly mentioned PalmOS, I bet it happens in other cases too. But I'd strongly argue that it's a bug, not a feature.

It's a bug that people tend to accept in a "single-client" environment.

NOTE! This is totally ignoring the fact that you can't be called "UNIX" any more. You _need_ to have sequence numbers etc be shared between multiple programs that all write to the stream. Again, that _does_ mean that you have another protection domain (aka "kernel" or "TCP deamon") that keeps track of the sequence number.

Linus

#595 lore raw
Quoted 10 lines · Linus Torvalds

Admins absolutely _hate_ that. They will ban an OS if it sends out packets that cause troublem. You should remember that - we used to do strange things on the net (long long time ago), and we brought down servers by mistake, and nobody ever considered it a server bug: it was a Linux bug that it wouldn't do the right thing.

Things like not sending FIN-packets when a program suddenly goes away is NOT acceptable behaviour! Neither is it acceptable behaviour to allow user programs to make up their own packets.

The user/kernel distinction is not always (heck, maybe almost never) present in the embedded world (a large world). The notion of a "regular user" does not apply at all in such a case. This is not a bug but merely the state of technology. It will slowly go away I think because embedded software becomes more complex and hardware becomes cheaper.

To implement multiple TCP clients _and_ the TCP/IP stack in one space is perfectly possible and it's actually done in practice. It has advantages (speed, when done properly) and disadvantages (complexity/bug-prone).

Quoted 6 lines

NOTE! This is totally ignoring the fact that you can't be called "UNIX" any more. You _need_ to have sequence numbers etc be shared between multiple programs that all write to the stream. Again, that _does_ mean that you have another protection domain (aka "kernel" or "TCP deamon") that keeps track of the sequence number.

There is nothing in the networking or UNIX standards that prescibe another protection domain for this. Would be insane to leave that out in a hosted environment but it _can_ be done without.

Signature · 1 line
Frank
#596 lore raw
Quoted 4 lines · Frank van Maarseveen

There is nothing in the networking or UNIX standards that prescibe another protection domain for this. Would be insane to leave that out in a hosted environment but it _can_ be done without.

My point is that TCP _does_ have a lot of state that needs to be handled in a safe manner by a proper operating system.

The fact that there are OS's out there that are crap doesn't change that matter. There are lots of embedded OS's out there that still do multitasking in a purely cooperative way. I don't think it's a valid model for anything but toys. Same goes for putting TCP in user space. It's doable, but it's not an "OS". It's a program loader.

Linus

#597 lore raw
In reply to Linus Torvalds
Quoted 10 lines · Linus Torvalds
Quoted 6 lines · Tom Vier
Quoted 4 lines · Linus Torvalds

- safely synchronize globally visible data structures
That's quite fundamental. 99% of what a kernel does is exactly that. TCP would be in user space too, if it wasn't for _exactly_ this issue. A lot

Quoted 5 lines
Quoted 1 line

What about microkernels? They do tcp in userspace.

No they don't. They do TCP in a separate address space from user space, that just also happens to be separate from the "microkernel address space".

The HP NonStop Server (formerly called Tandem) has an MPP architecture and a message passing OS (the NonStop Kernel). On this OS the TCP/IP stack is implemented as a library against which the application software is linked. The ethernet controllers/drivers maintain a routing table for routing incoming traffic to the right process. So the TCP/IP stack is implemented in userspace on the NonStop Server, I would say.

Regards,
Toon.

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#598 lore raw
In reply to Linus Torvalds
Quoted 5 lines · Linus Torvalds
Quoted 1 line

What about microkernels? They do tcp in userspace.

No they don't. They do TCP in a separate address space from user space, that just also happens to be separate from the "microkernel address space".

Well, I'd call that userspace (if it has a pid and it's own addr space, like every other proc).

Quoted 5 lines

So a microkernel will have _more_ address spaces, and they won't be "user space". They'll be "server deamon space" or something. Now, that's also why they tend to have performance problems - because you need to copy the data between different address spaces, and switch the CPU context etc around.

Just out of curiousity, what do you think of L4? I don't remember the numbers, but it wasn't much slower than linux (iirc), even on x86. I think k42 has msg passing close to the speed of a syscall. (Bulk data could use shared mem, maybe.)

Anyway, it would be neat to have a tcp/ip daemon. You could run experimental code w/o fear of it clobbering other things. Somethings, of course, a microkernel can't help (if the rootfs's fs daemon dies, you're screwed). A little bit more containment from bad kernel code would be nice.

(BTW, for those who don't know, mach optimized away msg passing and used a single addr space, once code was "trusted". You could choose speed over protection. That wasn't a hard choice since Mach msg passing sucks. Mach gave microkernels a bad name.)

Quoted 3 lines

Not user space. They may be "ring 3" from a CPU standpoint, but they aren't user space from a _user_ standpoint - it's still very much a separate address space, with domain protection.

How are they different from regular user procs, other then being trusted to manage certain resources?

Quoted 3 lines

In short: you _need_ to have a separate address space (either kernel, or "TCP server" or whatever) if you want to have reliable, secure and generally usable TCP.

Exokernels are another topic. 8)

Quoted 4 lines
Quoted 2 lines

As long as a trusted process keeps data such as free ports, what's the problem?

None - because it's not user space any more.

Yes it is, it's a user process.

Quoted 1 line

Well, performance might still suck, of course. And it does.

Signature · 2 lines
Tom Vier <[email protected]>
DSA Key ID 0x15741ECE
#599 lore raw
Quoted 7 lines · Tom Vier
Quoted 4 lines

Not user space. They may be "ring 3" from a CPU standpoint, but they aren't user space from a _user_ standpoint - it's still very much a separate address space, with domain protection.

How are they different from regular user procs, other then being trusted to manage certain resources?

Ehh, they are separate the same way "inetd" is separate. It's not a _user_ proc, it's a system proc. The user can't actually do anything about it.

In many ways UNIX _is_ a microkernel. It does nonessential stuff in "user space". Anything that is critical for performance or the working of the machine is in kernel space.

The big difference between UNIX and what people _call_ "microkernels" is that UNIX has a very functional and sane partitioning of what is a critical thing.

But from a kernel _protection_ angle, the only part that is important is that the services be in some protected domain. That was what started this
discussion: 99% of what the kernel does is protecting shared data. Whether it does so by passing it on to some trusted third party or not is an implementation issue, and is totally pointless from a user standpoint, since the user won't see it anyway.

Linus

#600 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds

The same goes for something like a "container file". Whether you see it as "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are totally equivalent), the point of having the capability in the kernel is not that the operations cannot be done in user space - the point is that they cannot be done in user space _safely_. The kernel is kind of the thing that guarantees that everybody follows the rules.

Well, it CAN be done safely if every client has to go through the kernel which does all the appropriate locks/semaphors and then passes the request down to the daemon. (Isn't NFS implemented this way?)

This is very micro-kernel-ish, but it's a reasonable idea to do it this way in cases where things are non-critical.

Say there's a way to cd into a tgz file to look around. If the access methods through the kernel get routed back to a user-space process (which probably does some amount of caching in memory and on disk of uncompressed bits of the archive), it could be a bit slower than if it were all in-kernel. The thing is that the processing time in the daemon is probably quite high compared to the overhead of the context switches, so it doesn't cost much. (And if it CAN be done in userspace without being horribly convoluted or unsafe, then it SHOULD be done in userspace.) Besides, even if it were a LOT slower to access a tgz file without extracting it first, I would STILL think it was wonderful AND use it a LOT.

Heh... now, the next thing is to be able to do some sort of union-mount like thing where the tgz is really read-only but if you WRITE to it, it gets stored somewhere else. This way, you can compile a kernel tree without extracting it first.

Taking this a bit further, the union-mount could be done away with if the daemon were to lazily rebuild the tgz file from its cache when the tgz is modified. That is, writes have little or no penalty because they just get stored in the daemon's cache, and the daemon rebuilds the archive in the background.

The VFS layer would have to recognize that a particular file has been accessed in such a way that it must go through the daemon for ANY access to it, which means that if someone tries to access the original archive as a file, requests have to go through the daemon. This makes it coherent, although with an expected performance penalty.

One issue to consider is that of shutdowns, both intentional and crashes. This is a non-issue with an underlying journaling file system.
As long as the tgz daemon comes up early enough, it would see its
cache (which is on-disk anyway), recognize which archive it belongs to, and continue where it left off. This way, the daemon doesn't have to go through a long process of rebuilding every archive you've ever modified on the disk on shut-down. If, however, you needed to access the archive without the daemon, that could be a problem (but what you'd get is probably the archive before it was modified). Also, some sort of sync command would be needed to cause the daemon to rebuild all archives immediately.

I really dig the idea of being able to access archives without extracting them first, regardless of any performance penalty.

#601 lore raw
Quoted 17 lines · Timothy Miller
Quoted 6 lines · Linus Torvalds

The same goes for something like a "container file". Whether you see it as "dir-as-file" or "file-as-dir" (and I agree with Jan that the two are totally equivalent), the point of having the capability in the kernel is not that the operations cannot be done in user space - the point is that they cannot be done in user space _safely_. The kernel is kind of the thing that guarantees that everybody follows the rules.

Well, it CAN be done safely if every client has to go through the kernel which does all the appropriate locks/semaphors and then passes the request down to the daemon. (Isn't NFS implemented this way?)

This is very micro-kernel-ish, but it's a reasonable idea to do it this way in cases where things are non-critical.

Quoted 11 lines

Say there's a way to cd into a tgz file to look around. If the access methods through the kernel get routed back to a user-space process (which probably does some amount of caching in memory and on disk of uncompressed bits of the archive), it could be a bit slower than if it were all in-kernel. The thing is that the processing time in the daemon is probably quite high compared to the overhead of the context switches, so it doesn't cost much. (And if it CAN be done in userspace without being horribly convoluted or unsafe, then it SHOULD be done in userspace.) Besides, even if it were a LOT slower to access a tgz file without extracting it first, I would STILL think it was wonderful AND use it a LOT.

This is only safe for some limited definitions of safety.

Leaving aside the obvious complexities of making sure the user space daemon doesn't just do something crazy, you have a number of problems:

* Which user does the daemon run as? If it runs as root, it needs to enforce strict security requirements in terms of VFS operations coming in, and also, it has all the problems of a SUID application running on an arbitrary user file. I don't know about you, but I don't trust joebob's tarfsd to be suid when running on some script kiddie tarball.

If it runs as the user making the request, then what happens when two users try to open the same file at the same time. Do they see two different views of the file? Is one of them locked out? How can they possibly synchronize if the views are writable?

* If the daemon runs as a user, then what happens if you try to run a suid program inside of the view? What if root tries to walk into a tar file? The conservative view is that the kernel would need to always return ENOENT for any process with elevated capabilities, but even this is not safe.

* Consider what happens if foo.tar/blah_blah is automatically bound to enter a tarfsd view of foo.tar. What happens if I point the web server at foo.tar/blah? The web server runs as httpd or something, so presumably httpd ends up running some sort of tarfsd view on the file. But if the tarball was made by a malicious person, presumably it can obtain httpd user access now by exploiting a bug in tarfsd.

* Even if you assume the view is read-only and the kernel coerces all permissions and ownership and such, there's the possibility of tarfsd presenting unexpected syscall results - weird error codes, short reads, file data changing underneath mmap, etc. that user applications don't expect and may be susceptible to.

* Besides all these security issues, if this scheme is writable it has all the resource problems that loopback network filesystems suffer. What if the kernel is short on memory and tries to flush dirty buffers to reclaim it. If those buffers are running against the user FS daemon, then that daemon must wake up to clear dirty buffers. If it tries to allocate memory, deadlock in the kernel.

Probably the security problems can be solved to some degree by being very paranoid in the kernel (at an associated loss in utility), and the resource issues can be solved by restricting dirty buffers for loopback mounts (at an associated loss in performance), but it's hardly simple.

-J

#602 lore raw

Hi!

I'm talking about uservfs.sf.net, aka podfuk, which implements tarfs (and more).

Quoted 8 lines

Leaving aside the obvious complexities of making sure the user space daemon doesn't just do something crazy, you have a number of problems:

* Which user does the daemon run as? If it runs as root, it needs to enforce strict security requirements in terms of VFS operations coming in, and also, it has all the problems of a SUID application running on an arbitrary user file. I don't know about you, but I don't trust joebob's tarfsd to be suid when running on some script kiddie tarball.

It needs to run as root. If there's bug in tar, that script kiddie is going to get your users anyway. Audit the code.

Quoted 1 line

* If the daemon runs as a user, then what happens if you try to run a

It does not.

Quoted 6 lines

* Consider what happens if foo.tar/blah_blah is automatically bound to enter a tarfsd view of foo.tar. What happens if I point the web server at foo.tar/blah? The web server runs as httpd or something, so presumably httpd ends up running some sort of tarfsd view on the file. But if the tarball was made by a malicious person, presumably it can obtain httpd user access now by exploiting a bug in tarfsd.

If there's a bug in tar, I can do a lot of interesting things. Audit tar.

Quoted 5 lines

* Even if you assume the view is read-only and the kernel coerces all permissions and ownership and such, there's the possibility of tarfsd presenting unexpected syscall results - weird error codes, short reads, file data changing underneath mmap, etc. that user applications don't expect and may be susceptible to.

FUD. Fix tarfsd if its problem.

Quoted 1 line

* Besides all these security issues, if this scheme is writable it has

There are no security issues. Yes, uservfs is security-sensitive code.

Quoted 5 lines

all the resource problems that loopback network filesystems suffer. What if the kernel is short on memory and tries to flush dirty buffers to reclaim it. If those buffers are running against the user FS daemon, then that daemon must wake up to clear dirty buffers. If it tries to allocate memory, deadlock in the kernel.

Coda solves this: all writable things are file-backed on file in /tmp => bad performance if you only need piece of very large file, but no deadlocks.

Quoted 4 lines

Probably the security problems can be solved to some degree by being very paranoid in the kernel (at an associated loss in utility), and the resource issues can be solved by restricting dirty buffers for loopback mounts (at an associated loss in performance), but it's hardly simple.

See sources. It is <1000 lines of code.

Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         
#603 lore raw
In reply to Timothy Miller

Hi!

Quoted 5 lines

Say there's a way to cd into a tgz file to look around. If the access methods through the kernel get routed back to a user-space process (which probably does some amount of caching in memory and on disk of uncompressed bits of the archive), it could be a bit slower than if it were all in-kernel. The thing is that the processing time

Exactly. See uservfs.sf.net. It is using coda hooks into the kernel.

Quoted 3 lines

be done in userspace.) Besides, even if it were a LOT slower to access a tgz file without extracting it first, I would STILL think it was wonderful AND use it a LOT.

Go ahead :-).

Signature · 1 line
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms         
#604 lore raw
In reply to Jan Harkes
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Jan" == Jan Harkes <[email protected]> writes:

Jan> On Thu, Aug 26, 2004 at 10:53:48AM +0300, Markus Törnqvist wrote:

[...]

Quoted 4 lines
Quoted 4 lines

I for one would truly welcome the coming of thumbnails and descriptions in picture files, because I have a real-life project going on where that would be extremely handy to have in the actual file.

Jan> Ehhh, that already exists and doesn't require meta-data streams,

Jan> http://www.exif.org/

but then you need an extra library to access the exif data. You can't just do something like "display foo.jpg/exif/thumbnail". Or even better, "echo 'this is a comment' > foo.jpg/comment" to change the jpeg comment without an external program.

And I don't think exif tags work with all image formats.

Quoted 6 lines
Quoted 6 lines · Matt Mackall
Quoted 5 lines

Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?

Under file-as-dir, it would not look the same under the shell if cp didn't copy everything -- the directory contents would be different. To copy the metadata as well, you would do something like "cp -r". Similarly with "cat < a > b".

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#605 lore raw
In reply to Markus Törnqvist
Quoted 11 lines · Markus Törnqvist
Quoted 7 lines · Matt Mackall

Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?

Quoted 4 lines

Then I guess OS X ships a broken implementation of cp, yes?

Nope, GUI handles it perfectly. it's maybe 0.1% of users of MacOS that acctually care about cp being broken.

--
GJ

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#606 lore raw
Quoted 4 lines · Grzegorz Ja??kiewicz
Quoted 2 lines

Then I guess OS X ships a broken implementation of cp, yes?

Nope, GUI handles it perfectly. it's maybe 0.1% of users of MacOS that acctually care about cp being broken.

Gotta love the Mac logics: "Is $FOO broken?" - "Nope, $BAR works. Very few care about $FOO being broken".

#607 lore raw
Quoted 7 lines · [email protected] wrote:
Quoted 4 lines · Grzegorz Ja??kiewicz
Quoted 4 lines

Then I guess OS X ships a broken implementation of cp, yes?

Quoted 8 lines
Quoted 3 lines

Nope, GUI handles it perfectly. it's maybe 0.1% of users of MacOS that acctually care about cp being broken.

Gotta love the Mac logics: "Is $FOO broken?" - "Nope, $BAR works. Very few care about $FOO being broken".

I don't represent Mac logic ;) IMO it's wrong, but it's still a fact.

--
GJ

#608 lore raw
In reply to Grzegorz Jaśkiewicz

Salut,

Quoted 4 lines · Grzegorz Ja??kiewicz
Quoted 1 line

Then I guess OS X ships a broken implementation of cp, yes?

Nope, GUI handles it perfectly. it's maybe 0.1% of users of MacOS that acctually care about cp being broken.

Actually, there is a (non-GNU) free implementation of the fileutils on MacOS/X that treats and copies the .DS_Store entries (or say is metadata aware).

And even though I might agree on that the principle of having per-directory database files is somewhat hacky, it appears to work like a charm (As long as userland programs do care about it).

On Linux we have a solution that actually works just the same and is a lot cooler. It's called POSIX extended attributes...

Tonnerre

PS. If you want to talk further on the subject of how MacOS/X does it and how I would do it using extended attributes, discuss it in private mail with me, since MacOS/X isn't really Linux. And no, I don't want a single daemon do that, but I want it to be a standard implementation that libc and fileutils support.

#609 lore raw
In reply to Matt Mackall

Matt Mackall wrote:
[...]

Quoted 18 lines

What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was.

Progress? No, this has been done before. Various dead operating systems have done it or similar and regretted it. Most recently MacOS, which jumped through major hurdles to begin purging themselves of resource forks when they went to OS X. They're still there, but heavily deprecated.

Quoted 4 lines

cp, grep, cat, and tar will continue to work just fine on files with multiple streams.

Quoted 5 lines

Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_?

It is what I'd expect. Now, use cp -R to copy the file _with its directory_, and see if that fares better. If not - bad implementation of fs and/or cp. The way I see file-as -directory is that _file_ operations (like the reads issued by cat) only work on the file part. You want the directory part? Use directory operations such as those "cp -R" use.

Quoted 4 lines

Now try cat < a > b on a file with a
fork. How is that ever going to work?

It is going to copy the _file_ part, of course. I wouldn't expect anything else - "cat" doesn't deal with directories.

This also raise the question of when to use file-as-directory. A usage where you need everything to follow the file, even when using "cat" calls out for an application that puts everything into one file. Directory-as-file is the wrong tool for that job, so don't worry about such problems.

Sticking thumbnails in a file-as-dir is another story though. Move the file (with "mv a b", not "cat a>b;rm a") moves the image file _and_ the thumbnail. No time wasted on regenerating thumbnails, no disk space or cleanup time wasted on dangling thumbs. Use the image file for its intended purposes (view it, mail it off, serve it on the web, edit it, print it) and the thumbnail doesn't get in the way because it isn't embedded in the file format. Embedding it in the file might work for jpeg which support generic embedded data, it surely won't work for every image format out there.

This is my idea of how file-as-directory should work.

Helge Hafting

Helge Hafting

#610 lore raw
Quoted 1 line · Helge Hafting

Matt Mackall wrote:

[...]

Quoted 6 lines
Quoted 4 lines

Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_?

It is what I'd expect.

Great. That means that all the tools of the trade stop working. Sounds like show-killer feature to me.

Quoted 2 lines

Now, use cp -R to copy the file
_with its directory_,

Either it is a file or a directory. Make up your mind. If you have no clear distinction, you'll only get messed up. Badly.

Quoted 5 lines

and see if that fares better. If not - bad
implementation of fs and/or cp. The way I see file-as -directory is that _file_ operations (like the reads issued by cat) only work on the file part. You want the directory part? Use directory operations such as those "cp -R" use.

Excuse me, I must grab my sickness bag here.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#611 lore raw
Quoted 10 lines · Horst von Brand
Quoted 1 line · Helge Hafting

Matt Mackall wrote:

[...]

Quoted 4 lines
Quoted 4 lines

Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_?

Quoted 5 lines
Quoted 1 line

It is what I'd expect.

Great. That means that all the tools of the trade stop working. Sounds like show-killer feature to me.

All wrong. Feel free to dislike the concept all you want, but don't provide false arguments. I am sure you can think up some _good_ ones, if you feel so strongly about this.

_No_ tool "stop working" with such a fs. Of course the new feature (file-as-directory) may not be supported - which is a problem only for files that use the new feature. Stick a lot of ordinary files and directories in such a fs, and the tools works as before with no breakage. People obviously won't deploy new features before they are useable - the fs have to come first. Then the tools, and widespread use only after that. This is similiar to other extensions like ACLs which many tools don't understand. They're still useable though.

Of course file-as-directory can have pilot uses like the often mentioned thumbnails - because it doesn't matter if those aren't backed up or get lost at random during the experimental phase.

Quoted 4 lines
Quoted 2 lines

Now, use cp -R to copy the file
_with its directory_,

Either it is a file or a directory. Make up your mind.

Wrong again. The file-as-directory concept means that a filename refers to something that is a file, or a directory, or both. That's the point of it. Being both isn't possible with current filesystems (except reiser4) of course, but file-as-directory is a proposed _extension_.

Quoted 3 lines

If you have no clear
distinction, you'll only get messed up. Badly.

The distinction is clear enough - three options instead of two.

Quoted 1 line

Excuse me, I must grab my sickness bag here.

Oh, very impressive. Please take a look at what we're talking about before attacking.

Helge Hafting

#612 lore raw
In reply to Helge Hafting

What's wrong with ~/.thumbcache or a daemon that manages system wide cache?

I have no problem if a plugin wants to parse a file and split it inside the file-as-dir. However, multiple file streams are a cross-platform nighmare. Anyone who's used mac os knows this.

Signature · 2 lines
Tom Vier <[email protected]>
DSA Key ID 0x15741ECE
#613 lore raw
Quoted 4 lines · Tom Vier

What's wrong with ~/.thumbcache or a daemon that manages system wide cache?

Moving a file doen't move the associated thumbnail, and then you
notice something is missing, or don't find the file, or have to wait
for regeneration when the app notices a file without a tumb. That could take some time if you moved a directory full of postscript files, for example.

Helge Hafting

#614 lore raw

Zitat von Helge Hafting <[email protected]>:

Quoted 7 lines
Quoted 1 line · Tom Vier

What's wrong with ~/.thumbcache or a daemon that manages system wide

cache?

Quoted 3 lines
Quoted 5 lines

Moving a file doen't move the associated thumbnail, and then you notice something is missing, or don't find the file, or have to wait for regeneration when the app notices a file without a tumb. That could take some time if you moved a directory full of postscript files, for example.

Use hash + filename in ~/.thumbcache and be smart when trying to find a thumbnail. That really can all be done in userspace.

Jan

#615 lore raw
Quoted 8 lines · Jan Dittmer

Zitat von Helge Hafting <[email protected]>:

Quoted 7 lines
Quoted 1 line · Tom Vier

What's wrong with ~/.thumbcache or a daemon that manages system wide

cache?

Quoted 3 lines
Quoted 8 lines
Quoted 5 lines

Moving a file doen't move the associated thumbnail, and then you notice something is missing, or don't find the file, or have to wait for regeneration when the app notices a file without a tumb. That could take some time if you moved a directory full of postscript files, for example.

Use hash + filename in ~/.thumbcache and be smart when trying to find a thumbnail. That really can all be done in userspace.

Yes, it can be done. It's slow, ineffecient and dangerous (rename a file, lose the thumbnail... so you'd have to do some garbage collection on the thumbnail cache).

Thumbnails are actually an *excelent* example of a multistream file. Renaming/moving a file leaves the image<->thumbnail link intact. Removing the image removes the thumbnail. Copying the file to another multistream-file capable machine also copies the thumbnail, saving you the time to regenerate the thumbnail. When the target machine doesn't understand a multistream file, you simply lose the thumbnail, no harm done.

Signature · 1 line
Erik Hensema <[email protected]>
#616 lore raw
In reply to Helge Hafting

Salut,

Quoted 5 lines · Helge Hafting

Moving a file doen't move the associated thumbnail, and then you notice something is missing, or don't find the file, or have to wait for regeneration when the app notices a file without a tumb. That could take some time if you moved a directory full of postscript files, for example.

That's why the userland file utilities must be aware of the feature..

Tonnerre

#617 lore raw
Quoted 5 lines · Tonnerre

Salut,

On Fri, Sep 03, 2004 at 10:07:00AM +0200, Helge Hafting wrote:

Quoted 10 lines
Quoted 6 lines

Moving a file doen't move the associated thumbnail, and then you
notice something is missing, or don't find the file, or have to wait
for regeneration when the app notices a file without a tumb. That could take some time if you moved a directory full of postscript files, for example.

That's why the userland file utilities must be aware of the feature..

That won't ever happen. There are many ways of implementing thumbnails, and people don't want to slow down the operation of "mv" by having it search for _all_ the known thumbnails schemes. (Yes - several may be in use simultaneosly because the user uses several file managers!) The waste of time is even more interesting for the many directories that doesn't have thumbs at all.

A file-as-dir implementation of thumbnails avoids this particular problem. There are other uses, so I hope people won't see file-as-dir as a mechanism for "file attributes" _only_.

Helge Hafting

#618 lore raw
In reply to Matt Mackall
Quoted 8 lines
Quoted 2 lines

cp, grep, cat, and tar will continue to work just fine on files with multiple streams.

Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?

It should work on the main file, the extra streams and meta data won't be copied. Just as it is now with special attributes etc. Cat reads the content of a file stream, not the file container.

Quoted 1 line

I like cat < a > b. You can keep your progress.

#619 lore raw
In reply to Matt Mackall
Quoted 11 lines · Matt Mackall

What it breaks is the concept of a file. In ways that are ill-defined, not portable, hard to work with, and needlessly complex. Along the way, it breaks every single application that ever thought it knew what a file was.

Progress? No, this has been done before. Various dead operating systems have done it or similar and regretted it. Most recently MacOS, which jumped through major hurdles to begin purging themselves of resource forks when they went to OS X. They're still there, but heavily deprecated.

Quoted 8 lines
Quoted 2 lines

cp, grep, cat, and tar will continue to work just fine on files with multiple streams.

Find some silly person with an iBook and open a shell on OS X. Use cp to copy a file with a resource fork. Oh look, the Finder has no idea what the new file is, even though it looks exactly identical in the shell. Isn't that _wonderful_? Now try cat < a > b on a file with a fork. How is that ever going to work?

I've been known to be silly now and then, for sure. Apple says
this about the next version of OS X, due next year:

"Tiger provides a standard, Darwin-level API for managing resource forks, filesystem metadata, security information, properties and other attributes in a consistent, cross-platform manner. For example, common UNIX utilities such as cp, tar and rsync can properly handle HFS+ resource forks."

That's marketing copy, but I think it's worth noting. It seems to
be that resource forks aren't deprecated across the board, just... changed.

Apple has found a way to make this work, at least partially. I
don't know if it's a good way, but there it is.

I'd like to be able to keep my Macintosh stuff on my Linux
fileserver, without losing the resource forks. That would require that the network fs support it of course, and whether I could use tar to back up my linux box would depend on how streams might be implemented there.

Quoted 1 line

I like cat < a > b. You can keep your progress.

Can't we have both?

Signature · 5 lines
Joseph Fannin
[email protected]

"Bull in pure form is rare; there is usually some contamination by data."
    -- William Graves Perry Jr.
#620 lore raw
In reply to Nicholas Miell
Quoted 2 lines

"OMG! It breaks tar and email!!!" argument doesn't fly. Things break all the time and are fixed. It's called progress.

Yes - we break things all the time. But we take notice, when we are breaking things, of how long standing and deeply embedded they are.

The deeper the roots, the more respect we show it, and the harder it will be to change.

Heaping scorn on someone who is reluctant to change something as deeply embedded as "a file is a byte stream" does not further the discussion.

Signature · 3 lines
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <[email protected]> 1.650.933.1373
#621 lore raw
In reply to Nicholas Miell
Quoted 7 lines

cp, grep, cat, and tar will continue to work just fine on files with multiple streams.

tar and cp will lose the extra streams until somebody fixes them, but they lose ACLs and xattrs right now, and I don't hear anybody suggesting that ACLs and xattrs be removed from the kernel because of this.

I thought that the underlaying FS/VFS would make sure the extra

  streams  would  be  preserved. Normal applications that doesn't have
  support  for  them  should  not  destroy the extra streams (and meta

data,attributes etc).

Of course tar is different, but any probgram that uses the normal OS API to copy, write and read files should not affect the extra info.

~S

Quoted 9 lines

Fixing programs that do recursive filesystem traversals is a matter of a glibc patch to nftw(3) and a modification to their option processing.

Holding back a useful feature because you don't want to upgrade coreutils is just plain dumb.

(BTW, for email, multipart/parallel is a start, but a specific multipart content type for multi-stream file attachments would probably be more appropriate.)

#622 lore raw
In reply to Matt Mackall
Quoted 10 lines · Matt Mackall
Quoted 3 lines

Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.

That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already.

Quoted 6 lines

While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE.

A file is a stream of bytes.

I couldn't agree more. Metadata which is easily lost is a terrible place to put a "document summary".

However, it's not a bad place to put thumbnails which are generated from the file contents, if they can be regenerated after transporting. It's slightly better than a ".thumbnails" directory because the latter won't follow renames and things like that.

It's a very good place to put metadata which is semantically bound to the file in special ways not intended for transport, such as security attributes, auto-invalidated digests of the file's contents, an auto-unpacked view of an archive, a virtual tree of an XML or other structured file, or an auto-generated textual representation of a binary file.

The key concept is "not intended for transport".

Generally that means permissions metadata, and things which are re-generated from the file's contents on demand.*

It's a very bad place to put an abstract of a text document, or the names of authors, or the MIME content type, or the character encoding of the document, or the name of the shell to run the file as a script, unless these things are also deducible from the file's contents. However, sometimes the content doesn't give any clue, and then something like the character encoding of a text file is usefully stored in the metadata simply because that's better than nothing.

Of course the facility can be abused. What does Windows XP do? Does it only store the appropriate kind of metadata in the alternate streams? Even Windows XP users expect files transfered over HTTP or WebDAV to work, don't they?

-- Jamie

* - By my definition, modification time arguably shouldn't be metadata. Personally I like downloads to be stamped with their source's modification time, which is why I don't use Mozilla's download manager but cut-and-paste URLs to "wget -N" instead.

#623 lore raw
In reply to Matt Mackall
Quoted 25 lines · Matt Mackall
Quoted 11 lines · Nicholas Miell
Quoted 8 lines · Wichert Akkerman
Quoted 5 lines · Previously Jeremy Allison

Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.

Quoted 10 lines
Quoted 5 lines

Aside from samba, is there any other application that has a use for them?

Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.

Quoted 7 lines

That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already.

Utilities that works on files only, such as cat, should keep working. No problem there. If you cat a file that also is a directory, then the file contents is all you get - by design.

Utilities that _only_ traverse the directory tree, such as find, should keep working too. Perhaps with a very minor update so they don't mistake a file-directory for a file only. I.e. find must recurse into anything that support directory semantics.

Something that both recurse and operate on files (cp -a, tar, grep, ... will need minor updating. They already knows how to handle files and directories, now they will need an update so they're open for objects that are both. I.e. let grep scan the file as usual, then recurse into its directory part in the usual way too. Performance problems should be avoided by not supporting directory operations on files with no directory content, which I believe will be many of them.

The "file-as-directory" thing will not be that useful before the tools gets these relatively simple updates. Till then it'll be a toy, which shouldn't stop it from getting into the VFS. Updating the tools will be a task for the file-as-dir fans.

(I don't know wether reiser4 does things this way - it is certainly they way I would want file-as-directory though.)

Quoted 6 lines

While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE.

Stick the file in an attachment and you get the file only. No problem, it is designed that way. An app that really wants everything in a single file should use a file structured
for that, not file-as-dir. File-as-dir attach stuff to a file in a
more loose way.

If you want to attach the directory contents too, do what you usually do when you want to mail someone a directory tree. You can't stick a directory in an attachment because it is not a file. So you either attach every file in the tree, or use tar. In this case, an updated tar.

The ftp server shouldn't be a problem. It supports both files and directories already. It may need a minor update in order to not mistake directory for file or vice-versa when someone request an operation.

ftp> get filename #Will get you the contents of the file part only - by design.
ftp> cd filename #Will change into the directory (if the file indeed provides one.)

Quoted 3 lines

A file is a stream of bytes.

Sure. And a file with a directory support both directory and file operations. You can get the stream of bytes as usual - that's the file part. Or you can cd into the directory as usual. There isn't much overlap between file operations and directory operations, so there is little conflict the way I see it. Merely letting the tools know that being a file no longer rule out the possibilities of directory operations. Getting the VFS right is another matter of course, but I don't worry about userland tools.

Helge Hafting

#624 lore raw
In reply to Matt Mackall
Quoted 17 lines · Nicholas Miell
Quoted 6 lines · Wichert Akkerman
Quoted 5 lines
Quoted 3 lines · Previously Jeremy Allison

Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.

Quoted 7 lines
Quoted 3 lines

Aside from samba, is there any other application that has a use for them?

Anything that currently stores a file's metadata in another file really wants this right now. Things like image thumbnails, document summaries, digital signatures, etc.

Quoted 5 lines

That is _highly_ debatable. I would much rather have my cp and grep and cat and tar and such continue to work than have to rewrite every tool because we've thrown the file-is-a-stream-of-bytes concept out the window. Never mind that I've got thumbnails, document summaries, and digital signatures already.

In Windows, the extra file streams are not lost or removed if you use a program that doesn't support them. They are only lost if you move the file to a file system that doesn't support the streams.

Even RAR support the NTFS file streams.

Quoted 6 lines

While the number of annoying properties of files with forks is practically endless, the biggest has got to be utter lack of portability. How do you stick the thing in an attachment or on an ftp site? Well you can't because it's NOT A FILE.

A file is a stream of bytes.

#625 lore raw
In reply to Nicholas Miell
Quoted 5 lines · --- Nicholas Miell

Anything that currently stores a file's metadata in another file really
wants this right now. Things like image thumbnails, document summaries,
digital signatures, etc.

As well as cases where the metadata is currently _in_ the file, right? Like MP3ID tags, EXIF tags, etc. And so many MP3 jukebox-server programs currently use a database to store metadata, just because they don't trust the MP3ID tags or need to store supplemental info. Being able to put this stuff in the filesystem along with the files would be so much cleaner. So I think there are plenty of uses for it.

IMO the Mac resource fork was always a good idea too, and I haven't a clue why it got left out of OSX (maybe Woz "got it" but Jobs doesn't).

Somebody said "we don't have to teach tar about it"; does that mean that tar in its present form would already successfully find the alternate streams?

=====
. _______ Shawn T. Rutledge / KB7PWD [email protected]

 (_  | |_)    http://ecloud.org/  [email protected]
 __) | | \______________________________________________
#626 lore raw
Quoted 3 lines · Shawn Rutledge

Somebody said "we don't have to teach tar about it"; does that mean that tar in its present form would already successfully find the alternate streams?

Look at history - did anybody teach tar about xattrs or acls? If nobody cared about those, why will reiser4 metadata be any different?

#627 lore raw
Quoted 6 lines · [email protected] wrote:

Look at history - did anybody teach tar about xattrs or acls? If nobody

cared about those, why will reiser4 metadata be any different?

My funding proposals often contain budget for modifying the most important user space apps to know how to use the new features. Somebody should fund me....

#628 lore raw
In reply to Wichert Akkerman
Quoted 9 lines
Quoted 3 lines · Previously Jeremy Allison

Multiple-data-stream files are something we should offer, definately (IMHO). I don't care how we do it, but I know it's something we need as application developers.

Aside from samba, is there any other application that has a use for them?

Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.

Also, just because we do not see all the benefits right now, doesn't mean there won't be any.

~S

Quoted 1 line

Wichert.

#629 lore raw
Quoted 5 lines · Spam

Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.

All of which can be handled in userspace library code.

What compelling reason is there for doing this in the kernel?

#630 lore raw
Quoted 4 lines
Quoted 1 line · Spam
Quoted 8 lines
Quoted 4 lines

Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.

All of which can be handled in userspace library code.

What compelling reason is there for doing this in the kernel?

Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.

Applications that support the new features will benefit, all others will continue to work without destroying data.

#631 lore raw
Quoted 15 lines · Spam
Quoted 5 lines
Quoted 1 line · Spam
Quoted 8 lines
Quoted 4 lines

Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.

All of which can be handled in userspace library code.

What compelling reason is there for doing this in the kernel?

Quoted 6 lines

Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.

Applications that support the new features will benefit, all others will continue to work without destroying data.

Sorry, but that all sounds a bit fluffy. Please provide some examples.

(Generally, getting all of userspace to agree on a particular library is socially hard [*], but I don't see that as a reason for putting the functionality into the kernel)

[*] Example: where's the library to manipulate /etc/whatever.conf? [**]

[**] yes, I know about gconf.

#632 lore raw
Quoted 18 lines
Quoted 15 lines · Spam
Quoted 13 lines
Quoted 5 lines · Spam

Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.

All of which can be handled in userspace library code.

What compelling reason is there for doing this in the kernel?

Quoted 8 lines
Quoted 6 lines

Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.

Applications that support the new features will benefit, all others will continue to work without destroying data.

Sorry, but that all sounds a bit fluffy. Please provide some examples.

We already had the examples with cp and mv. Both should continue to work and the files will still be copied. The same with Konqueror and Nautilus. Files and their meta-files/streams/attributes will be retained as long as applications are using the OS API.

However, if things are to implemented in user-space, then old applications will not work correctly and there is risk that all the extra information will be lost or corrupted.

You said it would be socially hard. I think it would be very much close to impossible to get it right. Imagine that Gnome and Nautilus would implement support for these. I doubt that cp, mv, KDE, mc, app-xyz would implement this anytime soon and in the meantime the data is at risk.

If we want things to work from the start, including with old applications then these features should be at below-application level. If it is in VFS or FS doesn't matter to me, as long as it doesn't hinder or delay progress to long.

Quoted 7 lines

(Generally, getting all of userspace to agree on a particular library is socially hard [*], but I don't see that as a reason for putting the functionality into the kernel)

[*] Example: where's the library to manipulate /etc/whatever.conf? [**]

[**] yes, I know about gconf.

#633 lore raw
Quoted 37 lines · Spam
Quoted 19 lines
Quoted 15 lines · Spam
Quoted 13 lines
Quoted 5 lines · Spam

Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.

All of which can be handled in userspace library code.

What compelling reason is there for doing this in the kernel?

Quoted 8 lines
Quoted 6 lines

Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.

Applications that support the new features will benefit, all others will continue to work without destroying data.

Sorry, but that all sounds a bit fluffy. Please provide some examples.

We already had the examples with cp and mv. Both should continue to work and the files will still be copied. The same with Konqueror and Nautilus. Files and their meta-files/streams/attributes will be retained as long as applications are using the OS API.

However, if things are to implemented in user-space, then old applications will not work correctly and there is risk that all the extra information will be lost or corrupted.

Quoted 6 lines

You said it would be socially hard. I think it would be very much close to impossible to get it right. Imagine that Gnome and Nautilus would implement support for these. I doubt that cp, mv, KDE, mc, app-xyz would implement this anytime soon and in the meantime the data is at risk.

No. All of the applications which you initially identified can be implemented by putting the various bits of data into a single file and getting applications to agree on the format of that file.

For example, some image file formats already support embedded metadata, do they not?

#634 lore raw
Quoted 47 lines
Quoted 37 lines · Spam
Quoted 19 lines
Quoted 15 lines · Spam
Quoted 13 lines
Quoted 5 lines · Spam

Yes, for example documents, image files etc. The multiple data streams can contain thumbnails, info about who is editing the file (useful for networked files) etc. Could be used for version handling and much more.

All of which can be handled in userspace library code.

What compelling reason is there for doing this in the kernel?

Quoted 8 lines
Quoted 6 lines

Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.

Applications that support the new features will benefit, all others will continue to work without destroying data.

Sorry, but that all sounds a bit fluffy. Please provide some examples.

We already had the examples with cp and mv. Both should continue to work and the files will still be copied. The same with Konqueror and Nautilus. Files and their meta-files/streams/attributes will be retained as long as applications are using the OS API.

However, if things are to implemented in user-space, then old applications will not work correctly and there is risk that all the extra information will be lost or corrupted.

Quoted 6 lines

You said it would be socially hard. I think it would be very much close to impossible to get it right. Imagine that Gnome and Nautilus would implement support for these. I doubt that cp, mv, KDE, mc, app-xyz would implement this anytime soon and in the meantime the data is at risk.

Quoted 6 lines

No. All of the applications which you initially identified can be implemented by putting the various bits of data into a single file and getting applications to agree on the format of that file.

For example, some image file formats already support embedded metadata, do they not?

Yes, JPEG, TIFF and PNG files for example. But, if you modify any of these with an application that doesn't support the extensions then you will loose them.

Also, you are thinking _very_ narrowly now. There are thousands of

  file  formats.  Implementing  the  support  for  meta-data/ streams/
  attributes  in  the  kernel  will  make  it  possible  to  use  this

generically for all files.

I use this in Windows quite much. I put information description fields for lots of different files. These descriptions are then searchable etc. I could use the command prompt to copy the files and the descriptions would still be there.

Secondly, do you expect file managers like Nautilus and Konqueror to support every piece of file format on the planet so they could read information directly from the documents?

~S

#635 lore raw
Quoted 4 lines · Spam

Secondly, do you expect file managers like Nautilus and Konqueror to support every piece of file format on the planet so they could read information directly from the documents?

Sure. You're proposing that we implement a single, golden multi-stream file format in the kernel.

We could just as well do that in libmultistreamfileformat.so.

But I'll grant that one cannot go adding new metadata to, say, C files this way. I don't know how useful such a thing is though.

Remember that my main point is that there's a lack of coordination in userspace. Hell, there's none. Putting it in-kernel forces that coordination, and may be the way to go, but it's pretty sad.

#636 lore raw
Quoted 2 lines · Previously Andrew Morton

But I'll grant that one cannot go adding new metadata to, say, C files this way. I don't know how useful such a thing is though.

That is actually one of the few places where a bit of metadata would be very useful. Right now there is no way to indicate in what encoding a source is written: ascii, utf-8, ucs16, etc. are all possible. But a compiler or interpreter has no good way to figure that out.

Wichert.

Signature · 2 lines
Wichert Akkerman <[email protected]>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.
#637 lore raw
Quoted 4 lines · Wichert Akkerman

That is actually one of the few places where a bit of metadata would be very useful. Right now there is no way to indicate in what encoding a source is written: ascii, utf-8, ucs16, etc. are all possible. But a compiler or interpreter has no good way to figure that out.

Make it a cmd line option. The makefile can contain info on what files are what and invoke the right options.

Signature · 2 lines
Tom Vier <[email protected]>
DSA Key ID 0x15741ECE
#638 lore raw
In reply to Wichert Akkerman
Quoted 11 lines · Wichert Akkerman
Quoted 2 lines · Previously Andrew Morton

But I'll grant that one cannot go adding new metadata to, say, C files this way. I don't know how useful such a thing is though.

That is actually one of the few places where a bit of metadata would be very useful. Right now there is no way to indicate in what encoding a source is written: ascii, utf-8, ucs16, etc. are all possible. But a compiler or interpreter has no good way to figure that out.

[NOTE: I am 5000 messages behind. Please forgive any redundancy.]

This reminds me of a paper someone wrote on how HFS(+) stored the file type (actually, application that knows how to use the file) as metadata, separate from the filename. He was lamenting the fact that the Mac was being 'corrupted' by the PC's broken philosophy of including as part of the filename something which should not be. He also mentioned that Windows' feature of hiding the extension doesn't cut it. One benefit, I recall, was that you can't change the association accidentally when changing the filename. Another thing was that file name and file type are not semantically related, so they shouldn't be squished together.

I don't remember this well enough, so I can't argue the point, but having the file type as metadata separate from the filename has SOME amount of elegant appeal to me.

#639 lore raw
In reply to Andrew Morton
Quoted 8 lines · Andrew Morton
Quoted 4 lines · Spam

Secondly, do you expect file managers like Nautilus and Konqueror to support every piece of file format on the planet so they could read information directly from the documents?

Sure. You're proposing that we implement a single, golden multi-stream file format in the kernel.

Which won't be able to cater for all needs, I'm afraid. Just look at /etc, and contemplate why there are so many different configuration file formats. Sure, some are just random coder preferences set in stone, but others do make sense being different. The same will happen with file formats. Heck, it is already that way, with file formats for C source code, ELF shared libraries, various graphics and audio and video formats, ...

Quoted 1 line

We could just as well do that in libmultistreamfileformat.so.

Right.

Quoted 2 lines

But I'll grant that one cannot go adding new metadata to, say, C files this way. I don't know how useful such a thing is though.

You can never know. Perhaps annotations in swahili? Alternative versions for different architectures? In object format(s)? Copyright info? Changelog?

Quoted 3 lines

Remember that my main point is that there's a lack of coordination in userspace. Hell, there's none. Putting it in-kernel forces that coordination, and may be the way to go, but it's pretty sad.

I think that forcing coordination is wrong. For example, I do value having ar/tar/cpio/zip doing essentially the same thing (and being able to define and use my own way of doing it if it pleases me), and I'd strongly oppose the kernel forcing one way of doing it down my throat.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#640 lore raw
In reply to Spam
Quoted 4 lines · Previously Spam

Also, you are thinking _very_ narrowly now. There are thousands of

  file  formats.  Implementing  the  support  for  meta-data/ streams/
  attributes  in  the  kernel  will  make  it  possible  to  use  this

generically for all files.

And how many different formats will be used for thumbnails? How many different names to indicate a summary for a document?

Having the ability to add attributes standardizes a way to access that data, but not the format and naming of those attributes. It could be a step closer to a more flexible way of handling storage, but you have to realize that a lot (if not most) of the work still needs to be done in userspace.

Wichert.

Signature · 2 lines
Wichert Akkerman <[email protected]>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.
#641 lore raw
In reply to Spam
Quoted 3 lines · Spam

Yes, JPEG, TIFF and PNG files for example. But, if you modify any of these with an application that doesn't support the extensions then you will loose them.

OK, so we've got a choice.

Either we will lose the extensions when modifying the file with an unaware program, or you lose the extensions when copying (or restoring from backup) using an unaware program.

Personally I'd prefer to keep the file intact when not modifying it...

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#642 lore raw
Quoted 8 lines · Spam
Quoted 4 lines

Yes, JPEG, TIFF and PNG files for example. But, if you modify any of these with an application that doesn't support the extensions then you will loose them.

Quoted 8 lines

OK, so we've got a choice.

Either we will lose the extensions when modifying the file with an unaware program, or you lose the extensions when copying (or restoring from backup) using an unaware program.

Personally I'd prefer to keep the file intact when not modifying it...

Backup to me is a special case, not simply a copy of files, but also retaining all the extra data, info, attributes, etc, that comes with the file.

Enabling support to have all of this extra stuff below the application level it will be possible to retain everything even though applications do not support them.

Backup tools, however, must know about these new features to be able to backup them. And so will every tool that directly accesses the filesystem instead of using the OS API for it.

And from I learned in an earlier message, tools like cp actually work on the fs level and themselves move the data to the new file
instead of letting the OS handle it. This seem to me as it could
be dangerous and could also prevent any kind of enhancements to the FS unless every tool like cp is patched.

#643 lore raw
Quoted 11 lines · Spam

Backup to me is a special case, not simply a copy of files, but also retaining all the extra data, info, attributes, etc, that comes with the file.

Enabling support to have all of this extra stuff below the application level it will be possible to retain everything even though applications do not support them.

Backup tools, however, must know about these new features to be able to backup them. And so will every tool that directly accesses the filesystem instead of using the OS API for it.

Thus, this kind of metadata API must be made accessible through the VFS layer, if we want to make it FS-independent (not tied to Reiser4). If the real FS does not support metadata operations, just ignore them (much like in ISO9660 ignoring write operations).

Quoted 5 lines

And from I learned in an earlier message, tools like cp actually work on the fs level and themselves move the data to the new file
instead of letting the OS handle it. This seem to me as it could
be dangerous and could also prevent any kind of enhancements to the FS unless every tool like cp is patched.

I still don't understand why a copy_file() syscall is still missing.

#644 lore raw
In reply to Spam
Quoted 1 line · Spam

Andrew Morton <[email protected]> said:

[...]

Quoted 6 lines
Quoted 2 lines

For example, some image file formats already support embedded metadata, do they not?

Yes, JPEG, TIFF and PNG files for example. But, if you modify any of these with an application that doesn't support the extensions then you will loose them.

As you will each time you muck around with your files-are-directories. Nothing new there.

Quoted 4 lines

Also, you are thinking _very_ narrowly now. There are thousands of

  file  formats.  Implementing  the  support  for  meta-data/ streams/
  attributes  in  the  kernel  will  make  it  possible  to  use  this

generically for all files.

So the _kernel_ has to know about thousands of formats, just in case it some blue day it comes across a strange file? Better leave that to the applications.

You will _not_ be able to define a single format for extra data about the file, there will be differing extra data for different users (do you suggest a root-only file with special writable pouches for "graphical icon for the file" for each user on the system?!), so the idea in itself is doomed from the start.

Quoted 1 line

I use this in Windows quite much.

Then use it and be happy. No need to screw up Linux for that.

Quoted 4 lines

I put information description
fields for lots of different files. These descriptions are then searchable etc. I could use the command prompt to copy the files and the descriptions would still be there.

The descriptions might make sense to _you_, _now_. No guarantee they make any sense (or are in the least useful) for other users on your system, and I might want them in arabic or some such. The descriptions might make no sense to you in a couple of years.

Quoted 3 lines

Secondly, do you expect file managers like Nautilus and Konqueror to support every piece of file format on the planet so they could read information directly from the documents?

That's their (self-selected) job, yes.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#645 lore raw
Quoted 4 lines · Horst von Brand

So the _kernel_ has to know about thousands of formats, just in case it some blue day it comes across a strange file? Better leave that to the applications.

Or come up with a way to export some interface to userspace.

Or make a standard description format in something like XML that can be used for arbitrary files and have userspace interface with that metadata stream that is the XML.
This is, however, dangerously close to what we have now and does. not. work, so the only added bonus we might have is the fact that it's endorsed by the kernel.
I don't like this idea except as a last fallback solution...

Quoted 2 lines
Quoted 1 line

I use this in Windows quite much.

Then use it and be happy. No need to screw up Linux for that.

Linux does not need to be screwed up by that.
It can be done right.

Quoted 2 lines

The descriptions might make sense to _you_, _now_. No guarantee they make any sense (or are in the least useful) for other users on your system, and

They also make perfect sense to me, and I would like this to have had this yesterday. or last year ;)

Quoted 2 lines

I might want them in arabic or some such. The descriptions might make no sense to you in a couple of years.

So what do you do if you use someone else's system and he doesn't have Arabic man pages but you want them in Arabic?

And a description that doesn't make sense in many years is badly formed.

Quoted 4 lines
Quoted 3 lines

Secondly, do you expect file managers like Nautilus and Konqueror to support every piece of file format on the planet so they could read information directly from the documents?

That's their (self-selected) job, yes.

Sure, but they store it differently in different places. Having them in the actual file makes it universal.

I think the biggest issue is that you are arguing against this out of some principle. There is a certain want for these new features among the users, there are people who are willing to implement them, there is Linus who seems to agree and accept.

What do you, and people who agree with you, lose if this gets implemented? Drive space when you have the slightly-grown kernel sources downloaded?

Signature · 1 line
mjt
#646 lore raw
In reply to Horst von Brand
Quoted 18 lines
Quoted 1 line · Spam

Andrew Morton <[email protected]> said:

[...]

Quoted 6 lines
Quoted 2 lines

For example, some image file formats already support embedded metadata, do they not?

Yes, JPEG, TIFF and PNG files for example. But, if you modify any of these with an application that doesn't support the extensions then you will loose them.

As you will each time you muck around with your files-are-directories. Nothing new there.

Quoted 8 lines
Quoted 4 lines

Also, you are thinking _very_ narrowly now. There are thousands of

  file  formats.  Implementing  the  support  for  meta-data/ streams/
  attributes  in  the  kernel  will  make  it  possible  to  use  this

generically for all files.

So the _kernel_ has to know about thousands of formats, just in case it some blue day it comes across a strange file? Better leave that to the applications.

No, not at all. The only think the kernel would need to support is the actual meta files and streams - not the contents of the streams. That is up to the applications to know and use.

The idea with plugins to reiser4 would allow someone to create a plugin that would export the thumbnails from TIFF and JPEG images as separate streams.

Quoted 5 lines

You will _not_ be able to define a single format for extra data about the file, there will be differing extra data for different users (do you suggest a root-only file with special writable pouches for "graphical icon for the file" for each user on the system?!), so the idea in itself is doomed from the start.

We do not need to define a single format. You are locking yourself to the contents of streams and meta files.

If someone likes the idea of a plugin that would allow for icons then that may be it. It would be up to application developers to know about this and use it. Perhaps they could agree on a standard even.

Quoted 3 lines
Quoted 1 line

I use this in Windows quite much.

Then use it and be happy. No need to screw up Linux for that.

This wouldn't screw anything up besides allowing for extending the functionality. There is noone that forces you to enable and instll plugins, use extra attributes or meta info. All of this always was optional (as I see it).

Quoted 9 lines
Quoted 4 lines

I put information description
fields for lots of different files. These descriptions are then searchable etc. I could use the command prompt to copy the files and the descriptions would still be there.

The descriptions might make sense to _you_, _now_. No guarantee they make any sense (or are in the least useful) for other users on your system, and I might want them in arabic or some such. The descriptions might make no sense to you in a couple of years.

So? The same may be for your own word documents etc. It doesn't mean I do not find it useful now, or that anyone else might not. Shall we not include features because "someone" on the same system may not use it? Seems very backward to me.

If I include a filestream, description.xml, then that wouldn't
affect anyone. I would know how to read the file, and that would be enough.

Quoted 5 lines
Quoted 3 lines

Secondly, do you expect file managers like Nautilus and Konqueror to support every piece of file format on the planet so they could read information directly from the documents?

That's their (self-selected) job, yes.

Yes and no. I used them as example but there are lots of other programs that can access files. Midnight commander, cp, tar, email programs and lots of others.

besides, I think it would be easier for them to 1) write plugins and
2) just use normal file access method to provide the extra data to the users.

#647 lore raw
In reply to Horst von Brand

Salut,

Quoted 8 lines · Horst von Brand
Quoted 4 lines

Also, you are thinking _very_ narrowly now. There are thousands of

  file  formats.  Implementing  the  support  for  meta-data/ streams/
  attributes  in  the  kernel  will  make  it  possible  to  use  this

generically for all files.

So the _kernel_ has to know about thousands of formats, just in case it some blue day it comes across a strange file? Better leave that to the applications.

I'd do that quite differently: Collect metadata about files in extended attributes (which are rather universally useable), and do it from an userspace daemon/application/file manager plugin which uses
dynamically loadable plugins for the different file types
differentiated by libmagic.

This has nothing to do in the kernel.

Tonnerre

#648 lore raw

Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200:

Quoted 5 lines · Horst von Brand
Quoted 3 lines

So the _kernel_ has to know about thousands of formats, just in case it some blue day it comes across a strange file? Better leave that to the applications.

Quoted 7 lines

I'd do that quite differently: Collect metadata about files in extended attributes (which are rather universally useable), and do it from an userspace daemon/application/file manager plugin which uses
dynamically loadable plugins for the different file types
differentiated by libmagic.

This has nothing to do in the kernel.

I mostly agree, like I was saying earlier, file types are needed! The kernel doesn't have to know about all of them, just some of them. It should be possible to attach a file type to everything so you know what kind of thing it is, not just block or character device or file, but something like a MIME type. If space is a concern, add to the inode a unique 8 byte character code for each popular type and a global translation table to get back the MIME string.

The kernel (includes file systems and plug-ins) would have to know some standard file types and how to compare them for indexing purposes. For example, an integer attribute like the year a music album was released would be stored as a 4 byte attribute "file" with a type like "primitive/int32", which tells the kernel to use 32 bit integer number comparison for indexing order. Another type the kernel could be aware of are the file types used for security attributes. A third kind it might need to know about are the computed attributes, which get set whenever related attributes they are computed from change. For any unknown types, the kernel just stores the data and doesn't do any extra processing.

Optionally the kernel could also maintain the global list of all file types and their properties (such as which ones are indexed, which are computed), though that could also be done in userland if you aren't doing indexing or computed attributes or other fancy operations.

The determination of the types would be in userland (smart programs would write the type when they created the file). Like you said, a daemon could classify unknown files in the background (BeOS does this). Most of the uses of the file type would also be in userland, such as deciding which program can handle a given file, or what icon to display, or to hint that a file/directory/attribute thing should be interpreted as an old style directory to the user, or to mark a file thing as being worth backing up (don't need to back up the computed ones).

- Alex

#649 lore raw
Quoted 17 lines

Tonnerre wrote on Sun, 29 Aug 2004 21:10:44 +0200:

Quoted 5 lines · Horst von Brand
Quoted 3 lines

So the _kernel_ has to know about thousands of formats, just in case it some blue day it comes across a strange file? Better leave that to the applications.

Quoted 7 lines

I'd do that quite differently: Collect metadata about files in extended attributes (which are rather universally useable), and do it from an userspace daemon/application/file manager plugin which uses
dynamically loadable plugins for the different file types
differentiated by libmagic.

This has nothing to do in the kernel.

Quoted 8 lines

I mostly agree, like I was saying earlier, file types are needed! The kernel doesn't have to know about all of them, just some of them. It should be possible to attach a file type to everything so you know what kind of thing it is, not just block or character device or file, but something like a MIME type. If space is a concern, add to the inode a unique 8 byte character code for each popular type and a global translation table to get back the MIME string.

The kernel doesn't have anything to know about normal files.It is up to applications to determine and know the contents. Isn't it?

Exception would be of course special meta files that define attributes and permissions for the filesystem.

Quoted 29 lines

The kernel (includes file systems and plug-ins) would have to know some standard file types and how to compare them for indexing purposes. For example, an integer attribute like the year a music album was released would be stored as a 4 byte attribute "file" with a type like "primitive/int32", which tells the kernel to use 32 bit integer number comparison for indexing order. Another type the kernel could be aware of are the file types used for security attributes. A third kind it might need to know about are the computed attributes, which get set whenever related attributes they are computed from change. For any unknown types, the kernel just stores the data and doesn't do any extra processing.

Optionally the kernel could also maintain the global list of all file types and their properties (such as which ones are indexed, which are computed), though that could also be done in userland if you aren't doing indexing or computed attributes or other fancy operations.

The determination of the types would be in userland (smart programs would write the type when they created the file). Like you said, a daemon could classify unknown files in the background (BeOS does this). Most of the uses of the file type would also be in userland, such as deciding which program can handle a given file, or what icon to display, or to hint that a file/directory/attribute thing should be interpreted as an old style directory to the user, or to mark a file thing as being worth backing up (don't need to back up the computed ones).

- Alex

#650 lore raw
In reply to Alexander G. M. Smith

Salut,

Quoted 5 lines · Alexander G. M. Smith

I mostly agree, like I was saying earlier, file types are needed! The kernel doesn't have to know about all of them, just some of them. It should be possible to attach a file type to everything so you know what kind of thing it is, not just block or character device or file, but something like a MIME type.

Then store the value computed by libmagic into an extended attribute, if you like. I still don't see why the kernel should even care. It shouldn't read things from files anyway, as a kernel shall be bootable on a busybox system (and *Step) as well, so we don't rely on the fs layout in any way.

Quoted 5 lines

Optionally the kernel could also maintain the global list of all file types and their properties (such as which ones are indexed, which are computed), though that could also be done in userland if you aren't doing indexing or computed attributes or other fancy operations.

Quel horreur!

Do it in userland, really.

If I get the time, I'll write you a small daemon based on libmagic which stores the file attributes in xattrs, or if they're not supported, in some MacOS/Xish per-directory files. Even a file manager ("finder") can do that, there's not even the need for a daemon.

Tonnerre

#651 lore raw
Quoted 2 lines · Tonnerre

Quel horreur!
Do it in userland, really.

I'm amazed that after all this discussion where all the realistic implementations are done in userland with kernel support for calling out to it, there are people who think the kernel is supposed to decode MP3 files or whatever.

Nobody is advocating that!

Quoted 4 lines

If I get the time, I'll write you a small daemon based on libmagic which stores the file attributes in xattrs, or if they're not supported, in some MacOS/Xish per-directory files. Even a file manager ("finder") can do that, there's not even the need for a daemon.

How are you going to do the part where the xattr changes when the file is modified?

(For example, if I edit an HTML file which is encoded in iso-8859-1, change it to utf-8 and indicate that in a META element, and save it under the same name, the full content-type should change from "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)

I don't see how you can do that without kernel support.

Don't say dnotify or inotify, because neither would work.

-- Jamie

#652 lore raw
Quoted 6 lines · Jamie Lokier

(For example, if I edit an HTML file which is encoded in iso-8859-1, change it to utf-8 and indicate that in a META element, and save it under the same name, the full content-type should change from "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)

I don't see how you can do that without kernel support.

Your html editor should do that.

Signature · 2 lines
David Kleikamp
IBM Linux Technology Center
#653 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Dave" == Dave Kleikamp <[email protected]> writes:

Quoted 6 lines · Jamie Lokier
Quoted 6 lines

(For example, if I edit an HTML file which is encoded in iso-8859-1, change it to utf-8 and indicate that in a META element, and save it under the same name, the full content-type should change from "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)

I don't see how you can do that without kernel support.

Dave> Your html editor should do that.

And if it doesn't, you'll need to patch your editor. Every editor will need to be patched to make sure that it is always consistent. Even sed. And you'll have to do that for every dynamically-changing attribute that people want. Sure, you can go that route if you want, but I wish you luck. You'll need it.

(Just for the record, I don't particularly think that the example given by Jamie is a good one. The "meta" element to me is a hack necessitated by not being able to attach attributes to a file in a consistent manner. The "proper" way to do it would be to set the content-type as an attribute, and let the webserver read that and send it in the HTTP header, where it's supposed to be.

Another example could be thumbnails. There are probably more, but I'm not feeling very creative right now.)

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#654 lore raw
In reply to Dave Kleikamp

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dave Kleikamp wrote:

| On Wed, 2004-09-01 at 15:16, Jamie Lokier wrote:
|
|>(For example, if I edit an HTML file which is encoded in iso-8859-1,
|>change it to utf-8 and indicate that in a META element, and save it
|>under the same name, the full content-type should change from
|>"text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)
|>
|>I don't see how you can do that without kernel support.
|
|
| Your html editor should do that.

Isn't the kernel faced with the same issues here as userland programs without meta attribute support? How do you read the META element before you know the encoding?

Is there property attached to each file descriptor that defines what encoding it's using? I know there is in perl, but I doubt it's at the OS level. If it is at the OS level, the kernel should use that.

Otherwise I agree -- the html editor should do that. Although I'd hate to see Vim patched to do that. Maybe save it as file/utf8?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTaZEHgHNmZLgCUhAQLmHxAAif/DzI3HDPPixHEJURTW8LMSBYi+RhgC Rsldit8G6N9Sx1wKReSZqoTI27OYUodVcXjgtSYbzr9w0dw2ifuLaU87550nCcwl p/3ivEiPww3Pqc6PgN2PpAhYT2b/xnZrWJSU3g9siGp9D4APBZ4yxz8GO9N8pAgg a/juHtLCNCtPKD8/IdjCYRn9W7Ka7uWF7cEHp58Yyj0R70EG4ljIRbJRhqKGT2Ew 6EIFcieStmuveA2Kna4Od4gAXIE5CrKEO8whRvplOsvHAlsZMp0zkgUGiKzBzQU0 K7ICkddo3Bnos1pc7BQ/7F595tNCNREqnlvCkWGVifLPdcyujULZ/U/OAVdD7zSJ AvhZs5A1/hPFYTEqrzSNsYteDsTSc0TJsiWS8gx+qyGzOzaVMAOIuf86jQGKzBT7 WSYutU4itrE1QYvdJ9XXeess8m9aSY1A9G1EkhtWmaRDq/SuCpSf01cNjZZVu52v Q1h6faCC9JTHM1pDPMBlMjeQOnhrhv/AuhrZnOfXmrODK06l+ZhQ/LSUTRAZwzkv UV6PSc+Pbq7ijYrurH4attPI8IkelzExhIPHziGQCTMnIN3pXSr6RT2zEtDG7LNR 98MHtKkuNC8bsuYFhyqGRbMLr9GANiByzu3rh1ccHuMbB42FR8/Z7X6UCDVZtP80 Bp4g6WwVG6A=
=xNYt
-----END PGP SIGNATURE-----

A message that is not in the archive (sent off-list or not yet received): [email protected]

#656 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jamie Lokier wrote:

| David Masover wrote:
|
|>Isn't the kernel faced with the same issues here as userland programs
|>without meta attribute support?  How do you read the META element before
|>you know the encoding?
|
|
| Ideally you guess whether it's ASCII-compatible, EBCDIC-compatible, or
| UTF16 or UCS4 :) Then you scan for a META element.  I suspect everyone
| only tries for ASCII compatibilty.  There's a bit of guessing
| involved.  Every parser has its own way - it's not well defined

And though it's nice for that guessing to be in one place, I'd rather not guess at all. Especially when we don't have to -- see below.

| No.  File descriptors are pure binary.
|>Otherwise I agree -- the html editor should do that.  Although I'd hate
|>to see Vim patched to do that.  Maybe save it as file/utf8?
|
|
| file/utf8 would be wrong.

Why? No, I don't mean as a MIME type ... literally do
vim foo.txt/utf8
to open a file named "foo.txt" and save it as "utf8".

That means that if vim doesn't automatically know how to tell a file what its encoding is -- since vim already knows its encoding. But it provides a generic way for a user to tell the text editor to tell the FS, all in one easy step, how to save the file.

It has its drawback -- it can really only support encoding -- or only one save option -- unless we're truly using files to send queries to the fs -- that is, something like foo.txt/writer/encoding=utf8&eol=crlf

I think that's feasable, and I think similar things (stranger things, like SQL queries) have been planned for reiser5 and such.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTfLzHgHNmZLgCUhAQKP0w/+IWnJM6qI04xsKXQJCCOd1tArnr+NB4us aGKj1JECibZCQhZiMc9aDLSr5FDosmTuMzdkVzMERoatzeuXjEyZrFFQNmsFQmzl c6KIvRrIgNOP4L5vOYxMUhGII1WEVI2Ub91t5TOYlcERn0raO4FL3WIUJ9gq6Lhc PcXrPvPp6yzIegJhe+mpHJi8/NrLnDdmmO8kg50C6ckYTrBeEzHwtTaJycEsaJub JDKjNZlhceRreTx6wccW26lPKXVus6rp5GOZgX9WliA2ktW8K9D8/cyZvOqRzC0y aDQUfhy5ZYkmxcHxKENEFrA85Gz3LTOwd4ci1DKCWW1HFp33EkmWIvXL7iiwMI7X xpkFfwap78O1ohUk0VdvkuQuKtKp2FmxMVwJpJe/982EcyECeMXqYbD3D5WJeXN4 1fZVPiZnt5FKOMBXgQ8qPAGit5eZkfI4SDc6uMfoQ0st+gd/E02j1Bf4ArheWRse 7f7Mk1C8IaQ4j565IGTABDjkpJwWWsmkItJRdZctCv+u6A2g70DREykU9G0iG4ua HGm53G8gba/A25Nt2Rw+7jt18zdeO/rEY9PaGmRKJVDkQSufHzWUuQ79LZrWr7as z5wLW6u+3FEn5xXCDAWyv4tbJOxPcxXTAuVe12KB4nYAHDjBgin77q9BcP6gqOQJ 3z/s3Tu95Yw=
=m/jf
-----END PGP SIGNATURE-----

#657 lore raw
In reply to Dave Kleikamp
Quoted 4 lines · Tonnerre
Quoted 4 lines
Quoted 4 lines

I'll write you a small daemon based on libmagic which stores the file attributes in xattrs, or if they're not supported, in some MacOS/Xish per-directory files. Even a file manager ("finder") can do that, there's not even the need for a daemon.

Quoted 6 lines · Jamie Lokier
Quoted 6 lines

(For example, if I edit an HTML file which is encoded in iso-8859-1, change it to utf-8 and indicate that in a META element, and save it under the same name, the full content-type should change from "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)

I don't see how you can do that without kernel support.

Quoted 1 line · Dave Kleikamp

Your html editor should do that.

My html editor is "vi". If I am supposed to manually set the content-type attribute after exiting vi, doesn't that rather invalidate the idea of a "small daemon based on libmagic" which sets it for me?

That was the sole point of my statement.

-- Jamie

#658 lore raw
In reply to Jamie Lokier
Quoted 8 lines · Jamie Lokier

(For example, if I edit an HTML file which is encoded in iso-8859-1, change it to utf-8 and indicate that in a META element, and save it under the same name, the full content-type should change from "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)

I don't see how you can do that without kernel support.

Don't say dnotify or inotify, because neither would work.

inotify done right is useful here as well as in a lot of other desktop cases where dnotify doesn't really scale. Its enough to let me

- Find the new file
- Virus scan it
- Classify its possible type heirarchies
- Index it

Alan

#659 lore raw
Quoted 10 lines · Alan Cox
Quoted 8 lines · Jamie Lokier

(For example, if I edit an HTML file which is encoded in iso-8859-1, change it to utf-8 and indicate that in a META element, and save it under the same name, the full content-type should change from "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)

I don't see how you can do that without kernel support.

Don't say dnotify or inotify, because neither would work.

Quoted 7 lines

inotify done right is useful here as well as in a lot of other desktop cases where dnotify doesn't really scale. Its enough to let me

- Find the new file
- Virus scan it
- Classify its possible type heirarchies
- Index it

Indeed it does, but it fails for the example I was commenting on to which you replied..

1. The file /var/www/site/index.html's written in vi.
2. "The daemon" (that's what I objected to) receives inotify. blocks waiting for scheduler, however...
3. Seeing that vi is now finished, I phone person on other side of world and say the updated file is now available.
4. Person fetches http://site/index.html
...oh! They receive the wrong content-type (bollocks!)
5. Eventually after paging & scheduling deems it appropriate, "The daemon" gets to run, looks at file, updates content-type.

In other words I was criticising Tonnerre's idea of a daemon which updates the xattrs by looking at file contents... by definition, a daemon runs in the background, and the whole point of it updating a content-type xattr was obviously so that programs like httpd could just _use_ that xattr.

That doesn't work, even if that daemon uses inotify.

Basically, anytime where you want an ordering guarantee that something will be recalculated in the interval after a file is modified and before the calculated result is used, you have to be very careful about exactly which code is using dnotify or inotify to achieve that. Single threaded programs can use it easily, but a multi-threaded file server has to be very careful about ordering if it's to avoid glitches.

It is exactly a real problem I have faced in a multi-threaded HTTP server which uses dnotify to detect changes to prerequisite files and also changes to the path walked and permissions on it, and thus invalidate cached generated pages, thus giving strong cache guarantees. Performance is great, if you ignore the race conditions...

(One of the more popular small web servers, lighttpd (PHP people seem to like it), also uses dnotify. Apparently it provides a big performance boost, just by avoiding stat() calls. So I'm not alone in using dnotify for such things.)

-- Jamie

#660 lore raw
In reply to Jamie Lokier
Quoted 6 lines · Jamie Lokier

(For example, if I edit an HTML file which is encoded in iso-8859-1, change it to utf-8 and indicate that in a META element, and save it under the same name, the full content-type should change from "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)

I don't see how you can do that without kernel support.

charset_cache = dbm(os.getenv('HOME') + '/.charset_cache')

def get_charset(path):
file_mtime = get_mtime(path)
cache_tuple = dbm.get(path, (None, None))
if file_mtime != cache_tuple[0]:
cache_tuple = (file_mtime, figure_out_charset(path)) dbm.put(path, cache_tuple)
return cache_tuple[1]

This code is guaranteed to always give you an up to date charset for a file, provided that the mtime is guaranteed to change every time the file changes.

If the file can change during the call to get_charset you'll have to lock the file while you are working with it or handle it some other way (comparing mtime before and after and retrying if they differ), but that is true even if you do it all in the kernel. This has the added advantage that the cache is updated lazily, so the charset is never calculated unless someone really needs it.

If the disk gets full, it is time to shrink the cache and preferably stale entries are evicted first. If that's not good enough, a notifier such as dnotify or inotify can be used to invalidate the cache immediately after the file has changed.

If this gets popular, the cache management can be moved to a daemon (naturally with all the security aspects in mind).

Quoted 1 line

Don't say dnotify or inotify, because neither would work.

Why not? The approach above works on any filesystem, even without dnotify or inotify but will be more efficient with them.

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#661 lore raw
Quoted 7 lines · Christer Weinigel
Quoted 5 lines · Jamie Lokier

(For example, if I edit an HTML file which is encoded in iso-8859-1, change it to utf-8 and indicate that in a META element, and save it under the same name, the full content-type should change from "text/html; charset=iso-8859-1" to "text/html; charset=utf-8".)

Quoted 7 lines
Quoted 1 line

I don't see how you can do that without kernel support.

[code which runs in the program which needs to read the content-type]

This code is guaranteed to always give you an up to date charset for a file, provided that the mtime is guaranteed to change every time the file changes.

...which it isn't, but ignoring that.

Yes we all know that code will work, turning a blind eye to occasional mtime failures.

What I objected to isn't that, but Tonerre's idea of a daemon (i.e. in the background, that's what daemons do) updating xattrs. Your code will work. Tonerre's has a serious race condition.

Quoted 4 lines
Quoted 1 line

Don't say dnotify or inotify, because neither would work.

Why not? The approach above works on any filesystem, even without dnotify or inotify but will be more efficient with them.

The above approach is in fact better with dnotify/inotify, when they area available, because then it doesn't have mtime problems.

My objection is that dnotify & inotify don't fix the problems of a "daemon" (i.e. in the background) updating an xattr - because the daemon might not run scheduled between the write and when the file needs to be used.

That said, yours has practical problems too. The intended purpose (*) of storing content-type in an xattr was so that programs which need to read content without understanding it may just do so (e.g. mail agents, httpd), and are distinct from the programs which compute and set it (e.g. programs which understand specific content formats).

(*) I don't necessarily agree with that purpose.

-- Jamie

#662 lore raw
In reply to Andrew Morton
Quoted 6 lines · Andrew Morton

No. All of the applications which you initially identified can be implemented by putting the various bits of data into a single file and getting applications to agree on the format of that file.

For example, some image file formats already support embedded metadata, do they not?

Same for MP3s and their metadata tags. _That_ kind of data should be part of the contents of the file: it is intended to be transported.

It is useful to auto-extract it from the file's data stream, using format-specific programs, and offer that metadata in separate files where it is easy to use e.g. by shell scripts and indexing programs, which can then work independently of file formats.

That kind of metadata should be _derived_ from the file's data stream, and ideally the extraction should be automatic, with userspace assistance. It may be useful for the filesystem to record what's extracted, for fast access and indexing.

The same applies to thumbnails. The only difference is that these may take longer to compute. But they're the same thing: something calculated from the file's data stream, which may (or may not) be stored in the filesystem so it doesn't have to be recomputed each time. It might even be worth expiring the metadata from disk if it hasn't been used for a while and can be recomputed.

These are the same as looking into archives or structured files. The only difference is that the unpacked views might not be stored on the filesystem (but they might be, if the filesystem knows how to cache views for faster future access).[*]

Those are all examples of data derived from the data stream of a flat file. They are useful applications, and all work fine with FTP, cp, cat <a >b, and email attachments, _no application changes required_.

There's another kind of metadata which isn't intended for transport. That would be things like the classic attributes: permissions, timestamps. It would also include things like author information when the file format cannot carry it.

-- Jamie

[*] - A fancier variant, which is hard without good fs support, it writing to structured files through a view. I.e. modifying files inside a .tar.gz, or (this is useful) inside an {MS,Open,Gnome,K}Office container file or *SQL database table. To do this efficiently, the file's main data stream is regenerated on demand from the view's data. As I said, it requires good, special fs support to do it efficiently (fancy caching). With appropriate userspace libraries you can make apps work with container files on all other OSes (just like .tar.gz and *Office containers at the moment) but are significantly faster to work with on Linux (no need to serialise and (de-)compress except on demand) and more visible to the standard unix scripting tools.

#663 lore raw
In reply to Andrew Morton
Quoted 5 lines · Andrew Morton

No. All of the applications which you initially identified can be implemented by putting the various bits of data into a single file and getting applications to agree on the format of that file.

So in order to avoid breaking backup and file utilities, we'd instead break every application that reads files? That way surely lies madness :) This /could/ possibly work with lots of cooperation from glibc, but we'd also need a way to insert and delete bytes in a file... moving 1MB of attributes/streams just to change a 20 character author attribute seems a bit silly. In addition, there are little things, such as how adding a thumbnail image shouldn't change the modification time of the original picture IMO, but it does if you embed it in the file.

If it should be in userspace, we could take the OS-X approach of using directories for everything, and the "data" that would be in a hybrid directory-file is a specially named file under that directory. Programs for instance are actually directories, with the elf file underneath it. When you click on the "directory-program" in the gui it runs the associated elf file rather than opening the directory. I think that approach is promising, but so few unix programs have any indirection for file access it'd be hell to teach them all how it works. Of course with emminent breakage of some sort, maybe now is the time...

- Jim Bruce

#664 lore raw
Quoted 2 lines · James Bruce

moving 1MB of attributes/streams just to change a 20 character author attribute seems a bit silly.

That's what we do now. Try editing an MP3's id tags.

Quoted 8 lines

If it should be in userspace, we could take the OS-X approach of using directories for everything, and the "data" that would be in a hybrid directory-file is a specially named file under that directory. Programs for instance are actually directories, with the elf file underneath it. When you click on the "directory-program" in the gui it runs the associated elf file rather than opening the directory. I think that approach is promising, but so few unix programs have any indirection for file access it'd be hell to teach them all how it works.

The interfaces we've been talking about are very close to that. The indirection is automatic: instead of a document (or program) looking like a directory, it'll look like a file, and reading (or executing) the file will do the indirection, with kernel control (and so consistency and performance) via an appropriate flat<->structured relation helper.

-- Jamie

#665 lore raw
Quoted 5 lines · Jamie Lokier
Quoted 2 lines · James Bruce

moving 1MB of attributes/streams just to change a 20 character author attribute seems a bit silly.

That's what we do now. Try editing an MP3's id tags.

What's the problem? Let's say a file has three streams/forks/whatever: the unnamed one, and two additional named. The filesystem can treat these three streams like three independent (but related) files on disk so, if the author attribute belongs to the first named stream, only the first named stream (and its corresponding file on the backend) would need to be modified. No need to move MBs of data around.

#666 lore raw
In reply to Andrew Morton
Quoted 11 lines · Andrew Morton

No. All of the applications which you initially identified can be implemented by putting the various bits of data into a single file and getting applications to agree on the format of that file.

For example, some image file formats already support embedded metadata, do they not?

-

and that leads to vasts numbers of file formats, all of which require special code to parse them, and apps that don't duplicate that code can't access the data, and it is a software engineering mistake. Uniform interfaces greatly reduce the cost of an OS and increase its expressive power. Closure is the most important and least understood principle of OS design.

By contrast, suppose everything was stored in files and directories. Every app can afford the coding cost to learn about files and directories. Data can freely bounce from one object to another in the OS because the APIs for the objects are all the same. That's increased expressive power. The number of connections between objects determines the expressive power of the OS, not the number of objects in it. Unified namespaces do a lot for an OS.

Hans

#667 lore raw
In reply to Spam

[ shortened Cc: of reiser- and other folks since it is no longer reiser-specific - sorry, if it was too much ]
[ fixed missing attribution ]

Quoted 2 lines · Spam
Quoted 1 line · Andrew Morton

Spam <[email protected]> wrote:

[...]

Quoted 4 lines
Quoted 4 lines
Quoted 2 lines

Applications that support the new features will benefit, all others will continue to work without destroying data.

Sorry, but that all sounds a bit fluffy. Please provide some examples.

It is too fluffy.

Quoted 2 lines

We already had the examples with cp and mv. Both should continue to work and the files will still be copied. The same with Konqueror and

... after they are patched to support streams. As stated each stream has an own fd, so - from the user-space perspective - one has to open *all* streams of this file, read them and write them into the destination.

Quoted 2 lines

Nautilus. Files and their meta-files/streams/attributes will be retained as long as applications are using the OS API.

The OS-API is *one* file descriptor where you can read, write, mmap, ... on. Therefore stream selection must happen with open() since open() returns a fd (which uniquely identify a stream). Voila.
And actually this makes transition actually possible: Old apps simply ignore streams (yes, there will be a default stream, which is used id nothing else is specified) and throw them away (think of e.g. patch which opens a new file, writes into it, unlink's the old one and mv's the new one on the old place) until they are modified to support them.

Bernd

Signature · 3 lines
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services
#668 lore raw
In reply to Spam

What is important is that it all be one packaged solution from the same maintainer. That is necessary for coherency of solution.

There is also benefit to it being one coherent body of code, but this is harder to articulate. It is always hard to articulate why functionality belongs in the same program rather than split into multiple programs. Filesystems should provide primitives for ordering, structuring, and naming objects. It is hard to say why we should do all of that, but on some level I just know it is our task.

Hans

Quoted 13 lines · Spam

You said it would be socially hard. I think it would be very much close to impossible to get it right. Imagine that Gnome and Nautilus would implement support for these. I doubt that cp, mv, KDE, mc, app-xyz would implement this anytime soon and in the meantime the data is at risk.

#669 lore raw
In reply to Spam
Quoted 3 lines · Previously Spam

Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.

But having it in the kernel has the same problem. If you read the Solaris documentation you will see that a bunch of utilties had to get a new commandline option to be able to access the metadata and a special utility was added for other applications. If you look at windows you will see that you need to use a filename like <realfilename>:<streamname>:$DATA (which obviously does not work for single-character filenames).

Ignoring samba for a bit which just needs streams to stay compatible with windows I see few reasons for using streams:

* files are more complex these days and tend to include multiple different things: images with thumbnails and exif data, 'office' documents containing both text and images

* standard way to add common metadata to a file which can be used for searching tools (author, copyright, keywords, etc.)

But both can already be done in userland (modern image formats can store thumbnails and exif data internally, applications use tar or zip-like files for documents, etc.). The metadata part is a lot more complicated as well since the behaviour of attributes might need to be complex:
if I change an image using an application that is not stream-aware, what should happen to its thumbnail?

The only common benefits I can see are standardisation and optimization:
instead of every file format or application defining a way to specify metadata for a file you get a common API defined by the OS (but you'll still need to standardize on attribute names and formats, so plenty of room that will still not help), and instead of parsing different files or XML streams you can directly access a bit of metadata.

So far I'm not convinced that streams are worth the effort. Not that my opinion is all that relevant here, but still :)

Wichert.

Signature · 2 lines
Wichert Akkerman <[email protected]>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.
#670 lore raw
Quoted 9 lines · Wichert Akkerman
Quoted 5 lines · Previously Spam

Because having user space tools and code will make it not work with everything. Keeping stuff in the kernel should make the new features transparent to the applications.

Quoted 5 lines

But having it in the kernel has the same problem. If you read the Solaris documentation you will see that a bunch of utilties had to get a new commandline option to be able to access the metadata and a special utility was added for other applications.

They did it wrong.

Quoted 6 lines

If you look
at windows you will see that you need to use a filename like <realfilename>:<streamname>:$DATA (which obviously does not work for single-character filenames).

again, they did it wrong. See Rob Pike's "The Hideous Name" for why that is wrong.

Quoted 30 lines

Ignoring samba for a bit which just needs streams to stay compatible with windows I see few reasons for using streams:

* files are more complex these days and tend to include multiple different things: images with thumbnails and exif data, 'office' documents containing both text and images

* standard way to add common metadata to a file which can be used for searching tools (author, copyright, keywords, etc.)

But both can already be done in userland (modern image formats can store thumbnails and exif data internally, applications use tar or zip-like files for documents, etc.). The metadata part is a lot more complicated as well since the behaviour of attributes might need to be complex:
if I change an image using an application that is not stream-aware, what should happen to its thumbnail?

The only common benefits I can see are standardisation and optimization:
instead of every file format or application defining a way to specify metadata for a file you get a common API defined by the OS (but you'll still need to standardize on attribute names and formats, so plenty of
room that will still not help), and instead of parsing different files
or XML streams you can directly access a bit of metadata.

So far I'm not convinced that streams are worth the effort. Not that my opinion is all that relevant here, but still :)

Wichert.

#671 lore raw
In reply to Spam
Quoted 2 lines · Spam

Keeping stuff in the kernel should make the new features transparent to the applications.

No, it will make just one special case, rename within the same filesystem, work. (Well, two special cases, rm will also delete the other forks).

Unless we add a new copy(2) syscall (which would be nice for other reasons) all applicatons that copy files will fail to copy the streams. So no working cp command, no nautilus nor konqueror without changes to the application. And if you have to change the applications anyway, isn't it much better to agree on a shared library in userspace that everyone uses? Which has the added bonus that it can be made to work on top of Linux, Windows, Ultrix and VMS?

/Christer

Signature · 4 lines
"Just how much can I get away with and still go to heaven?"

Freelance consultant specializing in device driver programming for Linux 
Christer Weinigel <[email protected]>  http://www.weinigel.se
#672 lore raw
Quoted 7 lines
Quoted 2 lines · Spam

Keeping stuff in the kernel should make the new features transparent to the applications.

Quoted 7 lines

No, it will make just one special case, rename within the same filesystem, work. (Well, two special cases, rm will also delete the other forks).

Unless we add a new copy(2) syscall (which would be nice for other reasons) all applicatons that copy files will fail to copy the streams.

How so? The whole idea is that the underlaying OS that handles the

  copying  should  also  know  to  copy  everything, otherwise you can
  implement  everything  into  applications  and  just  skip the whole

filesystem part.

Of course, it won't work if applications themselves are actually transferring the data and create the new files directly onto the filesystem. This just seem so wrong.

~S

Quoted 7 lines

So no working cp command, no nautilus nor konqueror without
changes to the application. And if you have to change the applications anyway, isn't it much better to agree on a shared library in userspace that everyone uses? Which has the added bonus that it can be made to work on top of Linux, Windows, Ultrix and VMS?

/Christer

#673 lore raw
Quoted 4 lines · Previously Spam

How so? The whole idea is that the underlaying OS that handles the

  copying  should  also  know  to  copy  everything, otherwise you can
  implement  everything  into  applications  and  just  skip the whole

filesystem part.

UNIX doesn't have a copy systemcall, applications copy the data manually.

Wichert.

Signature · 2 lines
Wichert Akkerman <[email protected]>    It is simple to make things.
http://www.wiggy.net/                   It is hard to make things simple.
#674 lore raw
Quoted 10 lines
Quoted 4 lines · Previously Spam

How so? The whole idea is that the underlaying OS that handles the

  copying  should  also  know  to  copy  everything, otherwise you can
  implement  everything  into  applications  and  just  skip the whole

filesystem part.

UNIX doesn't have a copy systemcall, applications copy the data manually.

Oh, this is very unfortunate and should be a bigger issue to fix.

Quoted 1 line

Wichert.

#675 lore raw
Quoted 10 lines · Spam
Quoted 8 lines
Quoted 4 lines · Previously Spam

How so? The whole idea is that the underlaying OS that handles the

  copying  should  also  know  to  copy  everything, otherwise you can
  implement  everything  into  applications  and  just  skip the whole

filesystem part.

UNIX doesn't have a copy systemcall, applications copy the data manually.

Oh, this is very unfortunate and should be a bigger issue to fix.

Then you have to rewrite POSIX und SuSv3.

Bernd

Signature · 3 lines
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services
#676 lore raw
Quoted 16 lines · Bernd Petrovitsch
Quoted 13 lines · Spam
Quoted 13 lines
Quoted 6 lines · Previously Spam

How so? The whole idea is that the underlaying OS that handles the

 copying  should  also  know  to  copy  everything, otherwise you can
 implement  everything  into  applications  and  just  skip the whole

filesystem part.

UNIX doesn't have a copy systemcall, applications copy the data manually.

Quoted 9 lines
Quoted 3 lines

Oh, this is very unfortunate and should be a bigger issue to fix.

Then you have to rewrite POSIX und SuSv3.

Bernd

Spam is right. Posix is a standard, not a vision, and the future is always a vision not a standard.

#677 lore raw
Quoted 2 lines · Hans Reiser

Spam is right. Posix is a standard, not a vision, and the future is always a vision not a standard.

An old friend of mine always said "if someone has visions he needs to stop somking mushrooms.."

#678 lore raw
In reply to Bernd Petrovitsch
Quoted 6 lines · Bernd Petrovitsch
Quoted 4 lines
Quoted 2 lines

UNIX doesn't have a copy systemcall, applications copy the data manually.

Oh, this is very unfortunate and should be a bigger issue to fix.

Then you have to rewrite POSIX und SuSv3.

They don't say 'you must now have a copy syscall'. Having one that's actually used by system tools would be a great optimization for many network or distributed filesystems.

#679 lore raw
Quoted 11 lines · Christoph Hellwig
Quoted 6 lines · Bernd Petrovitsch
Quoted 4 lines
Quoted 2 lines

UNIX doesn't have a copy systemcall, applications copy the data manually.

Oh, this is very unfortunate and should be a bigger issue to fix.

Then you have to rewrite POSIX und SuSv3.

They don't say 'you must now have a copy syscall'. Having one that's actually used by system tools would be a great optimization for many network or distributed filesystems.

ACK. But actually spam apparently assumes that the kernel - userspace interface is at the wrong abstraction level. And this is conceptually bound IMHO to POSIX.
And it does not solve the problem, that the read() and write() sys-calls will not vanish. Even if a copy() sys-call exists.

Bernd

Signature · 3 lines
Firmix Software GmbH                   http://www.firmix.at/
mobil: +43 664 4416156                 fax: +43 1 7890849-55
          Embedded Linux Development and Services
#680 lore raw
In reply to Wichert Akkerman
Quoted 10 lines · Wichert Akkerman
Quoted 6 lines · Previously Spam

How so? The whole idea is that the underlaying OS that handles the

 copying  should  also  know  to  copy  everything, otherwise you can
 implement  everything  into  applications  and  just  skip the whole

filesystem part.

Quoted 7 lines

UNIX doesn't have a copy systemcall, applications copy the data manually.

Wichert.

See sys_reiser4()..... ;-) you can go "A<-B". I have hopes of getting drive vendors to support us in doing single disk copies without even leaving the disk drive.....

#681 lore raw
Quoted 7 lines · Hans Reiser
Quoted 2 lines · Wichert Akkerman

UNIX doesn't have a copy systemcall, applications copy the data manually.

See sys_reiser4()..... ;-) you can go "A<-B". I have hopes of getting drive vendors to support us in doing single disk copies without even leaving the disk drive.....

Yes, we've wanted to have a "copy" syscall for a while now also, so that Lustre (or other network FS for that matter) can do remote copies of large files without having to go through the VM of the client. The work on sendfile() in 2.6 is a step in that direction, I eagerly await the day when "cp" will try to do a sys_sendfile() call first, before doing the read+write looping it does now.

Consider for a network filesystem we are limited to 1/2 of the client network bandwidth (say 50-75MB/s) for copying instead of the server-local or multiple server-server bandwidth for copies (400-600MB/s or more).

Cheers, Andreas
--
Andreas Dilger
http://sourceforge.net/projects/ext2resize/
http://members.shaw.ca/adilger/ http://members.shaw.ca/golinux/

#682 lore raw
In reply to Wichert Akkerman
Quoted 8 lines · Wichert Akkerman
Quoted 4 lines · Previously Spam

How so? The whole idea is that the underlaying OS that handles the

  copying  should  also  know  to  copy  everything, otherwise you can
  implement  everything  into  applications  and  just  skip the whole

filesystem part.

UNIX doesn't have a copy systemcall, applications copy the data manually.

couldn't sendfile(2) be used for that ?
perhaps a flag parameter could be added to sendfile(2) which specifies whether to copy _all_ metadata if both fd's correspond to files, something like either O_METADATA or O_NOMETADATA

Signature · 3 lines
Stelian Ionescu aka fe[nl]ix

Quidquid latine dictum sit, altum viditur.
#683 lore raw
In reply to Christer Weinigel
Quoted 3 lines · Christer Weinigel

all applicatons that copy files will fail to copy the streams. So no working cp command, no nautilus nor konqueror without changes to the application.

No true. A lot of the fancy stuff depends on metadata which is merely another view of the serialised contents in the flat file. Nautilus & Konqueror thumbnails are an example of that (I gave a few more in other mails). cp works fine with that, and so does ftp. The metadata is just recalculated when it's needed from the target file.

Quoted 4 lines

And if you have to change the applications anyway, isn't it much better to agree on a shared library in userspace that everyone uses? Which has the added bonus that it can be made to work on top of Linux, Windows, Ultrix and VMS?

Ideally yes, a shared library _or_ at least an agreed representation.

However, with filesystem support you can improve performance by avoiding unneed serialisation (write to your huge *Office presentation, read it from another program, no need to wait for the slow tar+compress stage yet it's _as if_ those were done), retain computed metadata with coherency guarantees (e.g. real time search indexes, crypto digests and such), and let the filesystem decide when best to prune computed metadata or convert representations.

All of these things can work with a combination of userspace plugins (not the same as reiser4 plugins), and filesystem support.

Without the filesystem support, you can still use the userspace plugins -- so apps would still be portable -- but you don't get the extras mentioned above.

-- Jamie

#684 lore raw
In reply to Christer Weinigel
Quoted 7 lines · Christer Weinigel
Quoted 2 lines · Spam

Keeping stuff in the kernel should make the new features transparent to the applications.

No, it will make just one special case, rename within the same filesystem, work. (Well, two special cases, rm will also delete the other forks).

In traditional UNIX filesystems, copy loses some info too (owner,group,mode bits). cp and tar was taught about this and they now can preserve that info.

However, I do not think that hacking on cp, tar, etc everytime we need to add something to our filesystems (xattrs, ACLs, whatever) is a terribly attractive idea.

I think reiserfs is going to handle that by giving uniform access to such ancillary data. It will be visible as file/meta/* filelets ('small files') or something like that and can be saved/restored by e.g. tar *without* needing to know what data is stored there, and in what format.

I think Hans is not planning turning old "file is a stream of bytes" into eight-stream octopus. One stream will remain as a 'main' one, which contains actual data. Others will keep metadata, etc...

Note that even today's files aren't simple "streams of bytes" because they also have names,mode bits etc.
--
vda

#685 lore raw
Quoted 3 lines · Denis Vlasenko

I think Hans is not planning turning old "file is a stream of bytes" into eight-stream octopus. One stream will remain as a 'main' one, which contains actual data. Others will keep metadata, etc...

This is exactly what the Samba people want, though.

Office suites can store a document with embedded images and spread sheets "easily" by putting the text, the images and spread sheets all in different file streams.

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#686 lore raw

Am Donnerstag, den 26.08.2004, 10:12 -0400 schrieb Rik van Riel:

Quoted 9 lines
Quoted 3 lines

I think Hans is not planning turning old "file is a stream of bytes" into eight-stream octopus. One stream will remain as a 'main' one, which contains actual data. Others will keep metadata, etc...

This is exactly what the Samba people want, though.

Office suites can store a document with embedded images and spread sheets "easily" by putting the text, the images and spread sheets all in different file streams.

Ouch.

I think Hans' idea is (I don't know if it is a good idea nor if it is doable, but at least it sounds interesting) to have special compound files where you can do something like this:

cp text.txt test.compound/test.txt
cp image.jpg test.compound/image.jpg

And if you read test.compound (the main stream) you get a special format that contains all the components. You can copy that single stream of bytes to another (reiser4) fs and then access test.compound/test.txt again.

The only thing I'm worrying about with this approach is what happens if someone tries to simultaneously open test.compound and test.compound/test.txt.

Hans has an example somewhere where he does something like that with /etc/passwd.

#687 lore raw
Quoted 11 lines · Christophe Saout

I think Hans' idea is (I don't know if it is a good idea nor if it is doable, but at least it sounds interesting) to have special compound files where you can do something like this:

cp text.txt test.compound/test.txt
cp image.jpg test.compound/image.jpg

And if you read test.compound (the main stream) you get a special format that contains all the components. You can copy that single stream of bytes to another (reiser4) fs and then access test.compound/test.txt again.

(To Rik especially), this is the design which more or less satisfies lots of different goals at once.

Quoted 7 lines · I wrote:

Properly implemented metadata can:

(1) operate in both modes simultaneously;
(2) work with unaware applications;
(3) provide performance enhancements to aware applications;
(4) provide storage enhancements to both;
(5) provide useful features that work with standard unmodified unix tools,

(1) is by design.

You can copy these "files" safely using cp or cat or ftp. You can email them. They're flat files. That's (2).

You can poke around inside them.

Applications which do poke around inside get performance gains, because they don't have to serialise the flat format every time they "load" or "save" a file. For example, if you have a huge OpenOffice document containing lots of sub-documents and pictures. We can imagine a version of OpenOffice which just opens the sub-documents, and writes the ones which are changed after editing. Also, another program such as a viewer, post-processor or search indexing program only needs to read the parts which have changed; it doesn't have to decode the whole thing after a change. That's (3).

We already know how useful this is with big mailboxes, hence people switching to maildir. Now imagine the attachments in each message are also viewable, cat'able etc., and that the filesystem stores each message compressed automatically, or in aggregate (your choice) and that you can use both mbox and maildir tools on the same mailbox. That's (4).

You can ls, cp, grep, less, vi, gqview etc. the pieces of files, and every supported file format needs just one tool to provide the interface to its contents. That's (5).

While I'm here I might as well add:

(6) improve support for real time search indexing.

The "view coherency" mechanism which is needed for efficient userspace VFS support has just the right properties to keep informing an indexer of changes. They could even enforce operation orderings if we cared about _perfect_ real-timeness, although we probably don't.

(7) yet another way of doing distros.

Why not just install .tgz or .zip files of entire binary packages, and let the filesystem unpack the bits that are needed when they're needed. Some platforms (i.e. Java) do this already with .jar files and it's quite popular. This would extend it in a versatile way.

Rik, I hope I've explained how those 5 properties are satisfied by Hans Reiser's brilliant design (ahem, great minds think alike, and remember Linus likes files-as-directories too :). But if I haven't please do pick holes.

Quoted 3 lines

The only thing I'm worrying about with this approach is what happens if someone tries to simultaneously open test.compound and test.compound/test.txt.

It is possible to implement it so that every change to one file immediately affects the other in the appropriate way, _as if_ the other file is regenerated every time before reading it.

This is what I mean by coherent views. It can be done even when userspace is doing the work, and the obvious mechanisms just happen to have the property of being useful in other ways (like a better dnotify, but I digress...). They're just an ordinary cache coherency protocols.

However, if you are writing a few bytes to test.compound, then the coherent view in test.compound/test.txt is probably a corrupt or non-existent file, because test.compound's intermediate state is a truncated and/or corrupt archive file, until you've written some more.

That doesn't look nice, but it's exactly the same problem as you get when you're writing to a file that another program is reading. If you do that on a live http server, for example editing a .html file, very occasionally someone will see a truncated file. Even if your editor uses renames to update atomically, sometimes you need to update more than one file atomically. It can be done, but rarely is.

Is there a solution to this? The obvious one is when you're writing to test.compound, don't read from test.compound/test.txt. That's _no different_ from the advice which says when you're saving a file from OpenOffice, don't load it into a document viewer during the time you're writing it, and don't serve it.

The other advice is to save to a different file and call rename(). Again, you get pretty much the same results as now: it atomically changes, and currently open references to the old contents will continue to reference the old contents.

There's another solution: transactions. reiser4 offers some kind of transaction facility, doesn't it? This is a fine example of when might want to declare that a sequence of writes needs to be a single transaction, visible only when the whole sequence is complete. Perfect!

I don't know reiser4's interface to that, but it would make a lot of sense to have a simple shell tool called "atomic", to wrap a sequence of operations. That would be useful for writing files _and_ useful for those pesky "update several files on the web server at once" moments. (Yes I know you can build a hard link tree and rename the root directory, although even that doesn't always work).

-- Jamie

#688 lore raw
Quoted 9 lines · Jamie Lokier
Quoted 4 lines · Christophe Saout

And if you read test.compound (the main stream) you get a special format that contains all the components. You can copy that single stream of bytes to another (reiser4) fs and then access test.compound/test.txt again.

(To Rik especially), this is the design which more or less satisfies lots of different goals at once.

And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?

Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work.

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#689 lore raw

Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel:

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

And if you read test.compound (the main stream) you get a special format that contains all the components. You can copy that single stream of bytes to another (reiser4) fs and then access test.compound/test.txt again.

(To Rik especially), this is the design which more or less satisfies lots of different goals at once.

Quoted 8 lines

And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?

Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work.

Yes, the main compound stream unaware applications would see would just be a writable view of its contents. Probably not trivial to implement but doable.

It should be a simple format. Something simliar to tar. The worst thing that can happen is people start writing plugins for every existing compound format out there. It should be the other way around, agree on a simple compound format and encourage applications to use this one if the want to use this advantage.

#690 lore raw
Quoted 5 lines · Christophe Saout

It should be a simple format. Something simliar to tar. The worst thing that can happen is people start writing plugins for every existing compound format out there. It should be the other way around, agree on a simple compound format and encourage applications to use this one if the want to use this advantage.

or placing this into a userspace helper that the kernel can invoke transparently. If you're doing funny thing like that for your swap device deadlocks are the last of your worries :)

#691 lore raw
Quoted 2 lines · Christoph Hellwig

or placing this into a userspace helper that the kernel can invoke transparently.

Yes, exactly this for most file formats.
Ideally, with _some_ user control over the handlers for their own files a bit like Hurd.

Not just archive files: metadata pulled from audio files, images, distro packages, etc. In other words much the same as Gnome and KDE's things, but with more emphasis on metadata extractors and efficient on-disk cache interaction, and a sane page-cache coherency interface (possibly FUSE).

Metadata extraction is useful for search and indexing purposes. (That, and a way to hook change notifications up with indexers -- again comes with any proper coherency interface).

For new applications using a container, it's tempting to want one format which is optimised for this. But really many applications want to write XML or compressed XML, or something else that isn't especially efficient but is easy to work with on many OSes. We should support applications whatever format they need to work with, if providing an interior view as a directory is useful for them. (This implies applications can introduce their own formats for files they need to operate on.) By the way, lazy updating of containers is an even bigger advantage if you have to use XML as the flat format, because the savings from not serialising when you don't need to are more pronounced.

For some applications the "content" isn't structured in a way which maps naturally to a plain archive file. Hans' /etc/passwd is a simple
example: at least one of the "flat" representations, when there are enough entries, probably ought to be a DB file for faster lookup. (Though, it would be great if /etc/passwd could automagically get fast lookup due to an automagic underlying representation, while still appearing to be a flat file when you cat it).

Also, imagine "cd" into a MySQL table file. It might just work.

By the way, we don't need containers to waste huge amounts of space like the Microsoft Word "fast save" documents did. Because our containers are created on demand, there is no performance advantage in them having unused holes, unlike the case were Word was saving changes periodically.

-- Jamie

#692 lore raw
In reply to Christophe Saout

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Christophe Saout wrote:

| Yes, the main compound stream unaware applications would see would just
| be a writable view of its contents. Probably not trivial to implement
| but doable.

Not the main stream, I think, but we do need a way to make "eight-stream octopus" files portable. But there needs to be a copy(2) syscall, and apps need to start supporting it. After all, copy-on-write was planned for Reiser4, now 4.1*. And don't anyone dare suggest "userland library".

| It should be a simple format. Something simliar to tar. The worst thing

Simple, yes. Similar to tar, no.

How about a serializer plugin? foo.mp3/serialize (say) should return some single stream which contains all of foo.mp3. It could work for directories as well, recursively, at least on the same filesystem -- you'd back up by doing

cp /serialize /backup/20040826

This has two potential problems. One is security -- I want to be able to serialize foo.mp3 without being root, but root doesn't want me to be able to serialize /etc and thus get access to shadow.

The second problem is that in order to actually be used for backup, we need snapshots, which are not done yet*. Currently, it'd have to lock the file/directory (recursively), which you don't want to do to entire live filesystems... So, no massive backups of live systems yet, but definitely useful for sending around mp3s and such.

| that can happen is people start writing plugins for every existing
| compound format out there. It should be the other way around, agree on a
| simple compound format and encourage applications to use this one if the
| want to use this advantage.

For the format, I vote for whatever format we're using at the storage layer. It'd be a lot faster that way, both in execution time (I think) and in development time (I'm sure). Plus, imagine the restore:

cat /backup/20040826 > /dev/hda5
resizefs_reiser4 /dev/hda5

* Disclaimer -- all this is "I think, last I checked".

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQS4uhngHNmZLgCUhAQKTaw//WCHt0Umgaf0GaMUpEr0PLhiXhFTYQqKV MKgpVt6SjP7lC9k3RbuCnPCLyOJGb9lIqo+Wd+B3b4TcQAQd4OYsSI6WiMPQjxXY OdNnB7/xYkkd/ULw7/7IGcokyc4FRDjhTyOMe4KscQS/6uhdX1dCriCCzVq7iELx u3HLulunHv1bwHbnZ6CyEZdQEr3j8XUu62+IXPTc6hWnfRRCvskV8Y4qyyVd8wFY Kp44TxbJJSzDeWmuI/YYkxVJsPmSoxRLxmw9/B8D72KGNg1SYLnM4ueIc5IV8RH0 kG/o8tzwTRxE/eqfcHP+UBt6yXZR/i84DDogAmJFKP73g9X+U+J5o/kONOBQODhY qY5LUwQu4UjlzXttMxC4NmZPESdUw85DgC/GhTQlsBMYBcmQHmpLAK67qt83+Kg0 1QqGPqJm4HL4weHLNknJ0DukLNKH7Bw5CVg335IwruaoiKUG7HDgthUhA319FCab VvYa+bEfeBEw9RorSE/gWM9Hg73luM7eoSzKc9zHgx++cACTi8SBsTFWqI2FU6DX UyWG5sNPybOiczpPsFu5S5FJBBNM3fLjkfeYgfqxUIjxGTNeUn/g/9I5ShUrCF1o T+3nd1jFxfoehYGkfGB2YLpdeNVBv4EdKK16j39+iducfJrXnRlL1yjdo/Kte2uz tTQZw3uTnhU=
=7bBF
-----END PGP SIGNATURE-----

#693 lore raw
Quoted 37 lines · David Masover

Christophe Saout wrote:

| Yes, the main compound stream unaware applications would see would just
| be a writable view of its contents. Probably not trivial to implement
| but doable.

Not the main stream, I think, but we do need a way to make "eight-stream octopus" files portable. But there needs to be a copy(2) syscall, and apps need to start supporting it. After all, copy-on-write was planned for Reiser4, now 4.1*. And don't anyone dare suggest "userland library".

| It should be a simple format. Something simliar to tar. The worst thing

Simple, yes. Similar to tar, no.

How about a serializer plugin? foo.mp3/serialize (say) should return some single stream which contains all of foo.mp3. It could work for directories as well, recursively, at least on the same filesystem -- you'd back up by doing

cp /serialize /backup/20040826

This has two potential problems. One is security -- I want to be able to serialize foo.mp3 without being root, but root doesn't want me to be able to serialize /etc and thus get access to shadow.

The second problem is that in order to actually be used for backup, we need snapshots, which are not done yet*. Currently, it'd have to lock the file/directory (recursively), which you don't want to do to entire live filesystems... So, no massive backups of live systems yet, but definitely useful for sending around mp3s and such.

| that can happen is people start writing plugins for every existing
| compound format out there. It should be the other way around, agree on a
| simple compound format and encourage applications to use this one if the
| want to use this advantage.
Quoted 7 lines

For the format, I vote for whatever format we're using at the storage layer. It'd be a lot faster that way, both in execution time (I think) and in development time (I'm sure). Plus, imagine the restore:

cat /backup/20040826 > /dev/hda5
resizefs_reiser4 /dev/hda5

one of the virtual metatags should be something about the method to serialize the data

if you are viewing a tarfile as a directory the VFS/Filesystem shoud know that the link between the whole thing and the contents is tar, and it should be possible for userspace programs to find this out by reading teh appropriate thing inside the virtual directory (and eventually change it as well, which would trigger regeneration of the base object, or at least noting that the current version is wrong and needs to be regenerated)

David Lang

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#694 lore raw
In reply to Christophe Saout
Quoted 23 lines · Christophe Saout

Am Donnerstag, den 26.08.2004, 11:54 -0400 schrieb Rik van Riel:

Quoted 8 lines
Quoted 7 lines
Quoted 4 lines

And if you read test.compound (the main stream) you get a special format that contains all the components. You can copy that single stream of bytes to another (reiser4) fs and then access test.compound/test.txt again.

(To Rik especially), this is the design which more or less satisfies lots of different goals at once.

Quoted 8 lines

And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?

Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work.

Yes, the main compound stream unaware applications would see would just be a writable view of its contents. Probably not trivial to implement but doable.

Quoted 6 lines

It should be a simple format. Something simliar to tar. The worst thing that can happen is people start writing plugins for every existing compound format out there. It should be the other way around, agree on a simple compound format and encourage applications to use this one if the want to use this advantage.

I also don't see why the VFS/Filesystem can't decide that (for example) this tar.gz is so active that instead of storing it as a tar.gz and providing a virtual directory of the contents that it instead stores the directory of the contents and makes the tar.gz virtual (regenerating it as needed or as extra system resources are available)

implementation wise I see headaches in doing this, but conceptually this is just an optimization that could take place in the future if we fine that it's needed.

David Lang

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#695 lore raw
Quoted 5 lines · David Lang

I also don't see why the VFS/Filesystem can't decide that (for example) this tar.gz is so active that instead of storing it as a tar.gz and providing a virtual directory of the contents that it instead stores the directory of the contents and makes the tar.gz virtual (regenerating it as needed or as extra system resources are available)

Absolutely. It could keep both views, if they're both actively being used. Or more than both, if there are more. (You could think of compression being an alternate view, and both compressed and uncompressed may as well remain on disk if there's space and it's being actively accessed.

Quoted 3 lines

implementation wise I see headaches in doing this, but conceptually this is just an optimization that could take place in the future if we fine that it's needed.

untarring a big source tree takes ages. I wouldn't like to do it after every reboot, if there was a .tar.bz2 tree I looked at often. That's why I have things like glibc untarred in my home directory.

With good cacheing, I could cd into the .tar.bz2 files and _effectively_ have the performance of untarred source trees for the ones I look at often on my disk -- automatically cleaned if the space if needed for something else, too. It would be quite nice.

-- Jamie

#696 lore raw
Quoted 5 lines · Jamie Lokier

With good cacheing, I could cd into the .tar.bz2 files and _effectively_ have the performance of untarred source trees for the ones I look at often on my disk -- automatically cleaned if the space if needed for something else, too. It would be quite nice.

What you're talking about is just a caching filesystem. Then you can have any deamon do whatever it wants to fill it - whether from tar-files or over the network or anything else.

I think you should ask David Howells about it..

Linus

#697 lore raw
In reply to David Lang
Quoted 5 lines · David Lang

I also don't see why the VFS/Filesystem can't decide that (for example) this tar.gz is so active that instead of storing it as a tar.gz and providing a virtual directory of the contents that it instead stores the directory of the contents and makes the tar.gz virtual (regenerating it as needed or as extra system resources are available)

Because that would mean the kernel should "talk" the tar format, which is, IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to perform this task, instead.

#698 lore raw
Quoted 11 lines · Felipe Alfaro Solana
Quoted 5 lines · David Lang

I also don't see why the VFS/Filesystem can't decide that (for example) this tar.gz is so active that instead of storing it as a tar.gz and providing a virtual directory of the contents that it instead stores the directory of the contents and makes the tar.gz virtual (regenerating it as needed or as extra system resources are available)

Because that would mean the kernel should "talk" the tar format, which is, IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to perform this task, instead.

Wasn't HURD supposed to do this for us?

Signature · 4 lines
Chris Dukes
Warning: Do not use the reflow toaster oven to prepare foods after
it has been used for solder paste reflow. 
http://www.stencilsunlimited.com/stencil_article_page5.htm
#699 lore raw
In reply to Felipe Alfaro Solana
Quoted 8 lines · Felipe Alfaro Solana
Quoted 5 lines · David Lang

I also don't see why the VFS/Filesystem can't decide that (for example) this tar.gz is so active that instead of storing it as a tar.gz and providing a virtual directory of the contents that it instead stores the directory of the contents and makes the tar.gz virtual (regenerating it as needed or as extra system resources are available)

Quoted 4 lines

Because that would mean the kernel should "talk" the tar format, which is, IMHO, a Bad Idea (TM). Maybe the kernel could notify a user-space daemon to perform this task, instead.

the kernel will definantly need the ability to use user-space code to do the transformations from one version to the other (if nothing else think of the thumbnail version of images, we don't want the image manipulation code in the kernel and we definantly want this sort of option available)

the interesting issue is going to be defining the kernel->user-space interface for doing the extractions.

David Lang

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#700 lore raw
In reply to Rik van Riel
Quoted 3 lines · Rik van Riel

And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?

Yes, exactly that. The streams are created on demand of course, and by userspace helpers when that's appropriate which I suspect it almost always is.

Quoted 4 lines

Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work.

Yes. Hence the idea of coherent views between two files: writing to one affects the content of the other, although the calcalation is only done on demand (or when the fs wants to migrate the representation -- for example, creating the flat container prior to deleting the regeneratable pieces in order to save space).

I haven't seen anything from Namesys that says they'll do that. I have the impression the streams are just generated in memory on the fly, not stored on disk with a cacheing policy, but that's just an impression. (We've all seen the Namesys white papers, they're not _that_ revealing). :)

I'm just pointing out how to do it _right_. I think it will turn out like this eventually, either next year or some time over the next decade after some iterations. Inevitable. Mark my words, etc. :)

-- Jamie

#701 lore raw
Quoted 3 lines · Jamie> Rik van Riel wrote:
Quoted 3 lines

And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?

Jamie> Yes, exactly that. The streams are created on demand of Jamie> course, and by userspace helpers when that's appropriate which Jamie> I suspect it almost always is.

So how would a program that converts between a JPEG file (with exif data) and a PNG work, such as ImageMagick? Are we proposing to teach the VFS (or worse yet each filesystem) how to do this?

I've been following this discussion a bit and I'm not sure that I've actually seen any concrete examples of where this is a *good* thing to have. People talk about only having to modify 20 bytes at a time instead of reading and writing 1mb of data. Isn't that what mmap() does?

Now I can sorta understand the idea that having a directory look like a file is neat, and certainly simplifies some aspects, but I think that going all the way down to the logical conclusion here is a bit silly.

To use the principle of blowing things up to make them very large or very small, what happens if I decide that the best idea is to make all files just be directories which contain single byte files? Isn't that the logical extreme here? So my 1mb JPEG file is not just some image data and header info in multiple files, but it's really just 1 million (ok 1024 * 1024) individual files that the VFS knows how to put together. Seems like the logical extreme. Oh wait, maybe we should be exposing a single file per bit instead!

Quoted 1 line

From my point of view, there lies madness. As Rik pointed out, how do

backup and restore tools work with this stuff? Most people could care less about how their data is organized, but they certainly care when they can't restore it from backups.

I'd really like to see a concrete example from Hans or other proponents about why this makes things easier/faster/better to do. Mostly, I've just seen "proof by vigorous handwaving" that it's a good thing.

In alot of ways, I think people are going in the wrong direction, you want to excapsulate and hide the details more, not expose them. That's what a good API does, it hides the details while giving you a rich set of semantics to manage your data.

God knows I'm not smart enough or driven enough to actually come up with my own ideas, but I certainly haven't seen anyone else (even Linus) come up with an earth shattering arguement to say why this is the right move to make.

As Linus says, most of the OS's job is to mediate access to objects/data. Why do we want to expose such low level data then?

John

John Stoffel - Senior Unix Systems Administrator - Lucent Technologies

	 [email protected] - http://www.lucent.com - 978-952-7548
#702 lore raw
Quoted 24 lines · John Stoffel
Quoted 3 lines · Jamie> Rik van Riel wrote:
Quoted 3 lines

And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?

Jamie> Yes, exactly that. The streams are created on demand of Jamie> course, and by userspace helpers when that's appropriate which Jamie> I suspect it almost always is.

So how would a program that converts between a JPEG file (with exif data) and a PNG work, such as ImageMagick? Are we proposing to teach the VFS (or worse yet each filesystem) how to do this?

I've been following this discussion a bit and I'm not sure that I've actually seen any concrete examples of where this is a *good* thing to have. People talk about only having to modify 20 bytes at a time instead of reading and writing 1mb of data. Isn't that what mmap() does?

Now I can sorta understand the idea that having a directory look like a file is neat, and certainly simplifies some aspects, but I think that going all the way down to the logical conclusion here is a bit silly.

Quoted 8 lines

To use the principle of blowing things up to make them very large or very small, what happens if I decide that the best idea is to make all files just be directories which contain single byte files? Isn't that the logical extreme here? So my 1mb JPEG file is not just some image data and header info in multiple files, but it's really just 1 million (ok 1024 * 1024) individual files that the VFS knows how to put together. Seems like the logical extreme. Oh wait, maybe we should be exposing a single file per bit instead!

It is doable. It doesn't mean we shall do this.

I think some reiser4 supporters try to do too much too soon. It is more sensible to do it in small steps.

I think we can start small and make all file metadata to be accessible via file/meta/{uid,gid,mode,mtime,atime,...}.

Normal tools will be unable to see file/meta/ because file is not a directory and file/ is not a directory (i.e. open(O_DIRECTORY) will fail on them). Hardlinking to file/meta/* is not allowed.

However, file/. _is_ a directory. We can make a tar-like
tool which can do its work and save/restore metadata along with data by just tarring/untarring file/meta/*. Notice how this 'new tar' does not need to know about exact kind of metadata supported by fs. You can add ACLs to the fs, and this 'new tar' will be able to save/restore ACLs. Without any modufications.

Contrast this with existing tar which has hardcoded knowlendge about uid,gid,mode,etc...
--
vda

#703 lore raw
In reply to John Stoffel
Quoted 7 lines · John Stoffel

Jamie> Yes, exactly that. The streams are created on demand of Jamie> course, and by userspace helpers when that's appropriate which Jamie> I suspect it almost always is.

So how would a program that converts between a JPEG file (with exif data) and a PNG work, such as ImageMagick? Are we proposing to teach the VFS (or worse yet each filesystem) how to do this?

No. (1) That's what "userspace helpers" are for. (2) What does image format conversion have to do with viewing files in their component parts? (3) I suspect someone will write a plugin that does indeed convert virtually every known image format to a common format (probably PNG and something "raw") at some point. Why not? It's just a small script (it would run ImageMagick!).

Quoted 5 lines

I've been following this discussion a bit and I'm not sure that I've actually seen any concrete examples of where this is a *good* thing to have. People talk about only having to modify 20 bytes at a time instead of reading and writing 1mb of data. Isn't that what mmap() does?

Sure, if you think mmap() is an easy substitute for "parse the author & license tags from this [unknown format] audio, video or font file".

Quoted 5 lines

Now I can sorta understand the idea that having a directory look like a file is neat, and certainly simplifies some aspects, but I think that going all the way down to the logical conclusion here is a bit silly.

Quoted 8 lines

To use the principle of blowing things up to make them very large or very small, what happens if I decide that the best idea is to make all files just be directories which contain single byte files? Isn't that the logical extreme here? So my 1mb JPEG file is not just some image data and header info in multiple files, but it's really just 1 million (ok 1024 * 1024) individual files that the VFS knows how to put together. Seems like the logical extreme. Oh wait, maybe we should be exposing a single file per bit instead!

If you decide to do that, you are welcome to write the userspace helper which creates that view in your directory.

Why not? It's very silly, but it's no sillier then writing a program to read your 1mb JPEG and write it out one file per bit, which you're welcome to write right now.

Quoted 4 lines
Quoted 1 line

From my point of view, there lies madness. As Rik pointed out, how do

backup and restore tools work with this stuff? Most people could care less about how their data is organized, but they certainly care when they can't restore it from backups.

Generated views are something which should _not_ be backed up and restored. That's not what they are for.

Auxiliary metadata, such as author info and signatures which cannot be stored in the main file for some reason -- that should be backed up.

Permissions, that should be backed up.

But not views which are computed from the main file. You don't need to back them up, and they don't need to take any real space. They're virtual, just like an infinitely deep directory on a web server can be virtual. You can make those very silly too, if you want to.

(Some help from the filesystem with storing temporarily cached values is fine, for performance, but we shouldn't pretend that generated views are anything other than virtual).

Quoted 9 lines

I'd really like to see a concrete example from Hans or other proponents about why this makes things easier/faster/better to do. Mostly, I've just seen "proof by vigorous handwaving" that it's a good thing.

In alot of ways, I think people are going in the wrong direction, you want to excapsulate and hide the details more, not expose them. That's what a good API does, it hides the details while giving you a rich set of semantics to manage your data.

Would you like to propose an alternate API? I was under the impression that we were discussing one, right here in this thread. It's called POSIX with an extension where you can cd into a file. It has a rich set of semantics to manage your data, and works with a lot of familiar programs.

A better one would be welcome, if you have an idea.

Quoted 4 lines

God knows I'm not smart enough or driven enough to actually come up with my own ideas, but I certainly haven't seen anyone else (even Linus) come up with an earth shattering arguement to say why this is the right move to make.

Come up with a better one otherwise it's right ;)

Quoted 2 lines

As Linus says, most of the OS's job is to mediate access to objects/data. Why do we want to expose such low level data then?

Eh? In this case the OS is mediating by providing a uniform interface between programs that access the data and programs that handle file formats.

What is this "low level" data you speak of? I would say something like the "designer" of a font, or the "from address" and "subject" of an email are very high level abstractions, and that's among the sorts of things which are to be exposed by these interfaces.

-- Jamie

#704 lore raw
In reply to Jamie Lokier
Quoted 20 lines · Jamie Lokier
Quoted 3 lines · Rik van Riel

And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?

Yes, exactly that. The streams are created on demand of course, and by userspace helpers when that's appropriate which I suspect it almost always is.

Quoted 4 lines

Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work.

Yes. Hence the idea of coherent views between two files: writing to one affects the content of the other, although the calcalation is only done on demand (or when the fs wants to migrate the representation -- for example, creating the flat container prior to deleting the regeneratable pieces in order to save space).

Quoted 5 lines

I haven't seen anything from Namesys that says they'll do that. I have the impression the streams are just generated in memory on the fly, not stored on disk with a cacheing policy, but that's just an impression. (We've all seen the Namesys white papers, they're not _that_ revealing). :)

would it be a tautology if I say that the streams implementation depends on (reiser4 now, may be vfs later) plugins? All proposals including user-level helpers, shadow caching/indexing may be implemented, why not?

Quoted 5 lines

I'm just pointing out how to do it _right_. I think it will turn out like this eventually, either next year or some time over the next decade after some iterations. Inevitable. Mark my words, etc. :)

-- Jamie

Signature · 1 line
Alex.
#705 lore raw
Quoted 23 lines · Alex Zarochentsev
Quoted 20 lines · Jamie Lokier
Quoted 5 lines · Rik van Riel

And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?

Yes, exactly that. The streams are created on demand of course, and by userspace helpers when that's appropriate which I suspect it almost always is.

Quoted 6 lines

Unless I overlook something (please tell me what), the scheme just proposed requires filesystems to look at the content of files that is being written out, in order to make the streams work.

Quoted 8 lines
Quoted 8 lines

Yes. Hence the idea of coherent views between two files: writing to one affects the content of the other, although the calcalation is only done on demand (or when the fs wants to migrate the representation -- for example, creating the flat container prior to deleting the regeneratable pieces in order to save space).

I haven't seen anything from Namesys that says they'll do that.

We do that, err, it is our design to do that. Hmm, the plugins to do that seem to all be in the next to implement list and not on the implemented already list.....

However, with respect to streams, those are just files in a file-directory object in our scheme. If you want them unified, well, then you cat compound-document/pseudos/glued, or you link compound-document/pseudos/glued to compound-document/pseudos/body, where body is our term for the default stream, and then you just cat compound-document

If you have the impression I am being vaporwarish, well, you are right.... glued is not yet on the already implemented list.

Hans

#706 lore raw
In reply to Rik van Riel
Quoted 3 lines · El Thu, 26 Aug 2004 11:54:51 -0400 (EDT) Rik van Riel <[email protected]>…

And if an unaware application reads the compound file and then writes it out again, does the filesystem interpret the contents and create the other streams ?

If the old utils doesn't support streams there's no way to make them create streams. There's not a real solution for that except fixing the old tools.

Why instead of being "compound" you just make test.compound to behave as a real directory? ie:

# echo "foo" > streamfile
# echo "fake metadata" > streamfile/somemetadata
# cat streamfile
foo
# cat streamfile/somemetadata
fake metadata

Now the old backup utils could see "streamfile" as a directory, ie:
old apps could see that "streamfile" is a directory and then make their backups with the following contents:
streamfile/somemetadata
streamfile/defaultstream

The defaultstream would be what its name means. There could be more default streams like "streamfile/compoundstream"

All this looks like reinventing the file/directory concept wheel. Instead of adding support for streams and "use files as directories", why not orientate it to "use directories as files? Streams could very well be provided by directories containing files, with the addon of being able to cat a directory (which would cat the output of a designed file inside every directory like "defaultstream", then there could be other "special files" like "compoundstream", although the idea of "fixed name- special files" doesn't sounds very cool) All this could be very well provided done userspace, but then couldn't userspace also supply the whole VFS functionality and still it lives in the kernel...?

#707 lore raw
Quoted 4 lines · Diego Calleja

All this looks like reinventing the file/directory concept wheel. Instead of adding support for streams and "use files as directories", why not orientate it to "use directories as files? Streams could very well be provided by directories containing files,

So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?

This is getting surreal.

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#708 lore raw
Quoted 9 lines · Rik van Riel
Quoted 4 lines · Diego Calleja

All this looks like reinventing the file/directory concept wheel. Instead of adding support for streams and "use files as directories", why not orientate it to "use directories as files? Streams could very well be provided by directories containing files,

So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?

No no. The stream you get from a directory is totally _independent_ of the contents of the directory. Anything else would be pointless.

Linus

#709 lore raw
Quoted 4 lines · Linus Torvalds
Quoted 1 line

So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?

No no. The stream you get from a directory is totally _independent_ of the contents of the directory. Anything else would be pointless.

Surely it depends on the directory?

The stream from a "classic directory", i.e. something made with mkdir(), mode drwxr-xr-x filled with files, might well be a reproducible representation of its contents.

-- Jamie

#710 lore raw
Quoted 6 lines · Jamie Lokier
Quoted 3 lines · Linus Torvalds

No no. The stream you get from a directory is totally _independent_ of the contents of the directory. Anything else would be pointless.

Surely it depends on the directory?

Well, you _could_ make it do a "tar", but what's the point? That really doesn't add any value as far as user space is concerned.

Think of it this way: if the directory and the "default stream" associated with it aren't independent, then user space already has the information, and the stream is pointless. In fact, the stream is _worse_ than pointless, because it hides the fact that there is no independent information.

See my point?

Linus

#711 lore raw
Quoted 10 lines · Linus Torvalds

Well, you _could_ make it do a "tar", but what's the point? That really doesn't add any value as far as user space is concerned.

Think of it this way: if the directory and the "default stream" associated with it aren't independent, then user space already has the information, and the stream is pointless. In fact, the stream is _worse_ than pointless, because it hides the fact that there is no independent information.

See my point?

I see your point, but I agree 50%.

(1) One of the uses of "streams" is to have extra information which accompanies a file, and is separate from the flat file's information.

For example, metadata like "this C source file is in UTF-8", "the author is Jamie (nowhere to put it in the data file)" and "Jamie's signature".

(2) Filesystem settings like "compress this", and perhaps "permissions".

(3) Filesystem information like "this file has been compressed by 59%, uses 5 1k blocks".

(4) Another use, which is more akin to containers, is to _present_ the information of the flat file in a different way.

For example, the id tags and comments of any media file, the component parts of any media file, license (of course :), and expanded or decoded views such as inside an archive file, filesystem image, or other structured storage file (.doc :).

Those alternate views are useful just because they can be accessed using ordinary unix tools, no matter what the obscure codec format provided it's supported. With a bit of effort a standard presentation for certain kinds of data may arise.

(5) There is also data which is _calculated_ from the flat file or its extra metadata, and which the kernel guarantees is either consistent with the file, not set at all, or at least is invalidated if the file is changed. It may be calculated on demand through the kernel interface, or applications may simply have to calculate it themselves when they don't have an up to date value.

For example MD5 or SHA1 digests of the flat file, or of its parts. The key thing is that these are always up to date, not dependent on weak tests like mtime.

Partial checksums for speeding up rsync and other delta algorithms. Computed thumbnails, MP3 average power levels, character encoding extracted from parsing old HTML META tags. Anything really that can be calculated and usefully cached.

This is similar to (4), but I'm emphasising things that may take a while to compute, may be cached with the filesystem itself helping to decide when to evict, and writing them won't affect the data they're calculated from, if writing is allowed at all.

The only really important of these are MD5 or SHA1 digests, or some other way of tracking modifications like a strong serial number.

All other computed cached data can be maintained by applications elsewhere, if they know when the data has been modified since the last time. (Though there are advantages to making the data disappear if the file is deleted, or move if it's renamed).

The thing with these different categories is that we want different behaviour when serialising the data for a backup or to copy it.

We probably always (1) to be copied.

(2) is questionable: it's like unix file permissions, you might want to copy it; you might not.

(3) Shouldn't be copied or stored except for special reasons. It's a bit like /proc: it's not meant to be stored on your backup CD and it's definitely not meant to be written during a restore. :)

(4) Should never be copied if you're copying the flat file, because it's exactly the same information, just accessed differently. You might want to copy it by itself for some reason though, and you certainly would like to work with it from scripts and applications. These streams are special in another way: they might themselves decompose into further streams. Just think of a AVI file containing an OGG file containing a FLAC containing metadata and data streams, or something like that. They happen.

(5) shouldn't be copied if you're transmitting a file, as a matter of data integrity among other things. You might want it on the backup, then again maybe only some of the values. Probably never the crypto digests.

To summarise: most data inside a directory-as-file should _not_ usually be copied and transported. A little of it, (1), should be copied if the serialised form will have it, and some of the other metadata should be copied under special circumstances.

I've suggested that we could simply not have (1), and recommend that only reproducible, non-essential data appears inside the directory. Everything essential would be part of the serialised, flat file.

However, you seem to like (1), metadata which should be serialised and copied around, and which is not part of the flat file.

Note that (1) will only be copied around on systems which handle "extra forks" anyway -- things like Samba and MacOS compatibility. FTP and HTTP obviously won't copy the metadata of (1), although HTTP WebDAV might allow it to be visible as properties.

All that suggests a common structure for the directory inside a "container" -- one where the metadata to be copied by default is clearly distinguished from the metadata which should never by copied, or which is cached, or which is a view. Possibly it should be distinguished by name, although permission bits or (not sure I like this) meta-metadata could serve the same role without being name dependent.

-- Jamie

#712 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds
Quoted 1 line

So all I need to do is "cat /bin | gzip -9 > /path/to/backup.tar.gz" ?

No no. The stream you get from a directory is totally _independent_ of the contents of the directory. Anything else would be pointless.

It's a relief to know that nobody's taking my humorous suggestion seriously, but now we still have the "standard Unix tools can't manipulate files" problem...

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#713 lore raw
Quoted 4 lines · Rik van Riel

It's a relief to know that nobody's taking my humorous suggestion seriously, but now we still have the "standard Unix tools can't manipulate files" problem...

I disagree. They can manipulate the files a whole lot better than they can manipulate xattr's.

For example, you _could_ probably (but hey, maybe "tar" tries to strip slashes off the end of filenames, so this might not work due to silly reasons like that) back up a compound file with

tar cvf file.tar file file/

although unpacking it would require that tar be taught about the thing. And you definitely could write a script to do the thing, ie even with an unmodified tar you could do

tar cvf file-archive.tar file
cd file
tar cvf ../attribute-archive.tar .

which is a hell of a lot better than what you can do with the fsattr interfaces and unmodified legacy applications.

So one of the advantages of "dir-as-file/file-as-dir" is exactly that you _can_ manipulate the data with legacy tools. Sure, things that traverse a directory tree might need some (likely fairly trivial) modifications if they really want to take advantage of the subfiles, but that's still likely to be _much_ less of an issue than with fsattr's that have a totally different model entirely.

Linus

#714 lore raw
Quoted 5 lines · Linus Torvalds

For example, you _could_ probably (but hey, maybe "tar" tries to strip slashes off the end of filenames, so this might not work due to silly reasons like that) back up a compound file with

tar cvf file.tar file file/

So you'd have both a file and a directory that just happen to have the same name ? How would this work in the dcache?

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#715 lore raw
Quoted 3 lines · Rik van Riel

So you'd have both a file and a directory that just happen to have the same name ? How would this work in the dcache?

There would be only one entry in the dcache. The lookup will select whether it opens the file or the directory based on O_DIRECTORY (and usage, of course - if it's in the middle of a path, it obviously needs to be opened as a directory regardless).

That's not the problem. The problem from a dcache standpoint ends up being when the file has a link, and you have two paths to the same sub-file through two different ways:

.. create file 'x' with named stream 'y' ... ln x z
ls -l x/y z/y /* it's the same attribute!! */

but this is actually exactly the same thing that we already have with mounts, ie it is equivalent (from a dentry standpoint) to

	.. create directory 'x' with file 'y' ..
	mkdir z
	mount --bind x z
	ls -l x/y z/y	/* It's the same file!! */

so none of this is really anything "new" from a dcache standpoint.

Except for all the details, of course ;)

Linus

#716 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 3 lines · Rik van Riel

So you'd have both a file and a directory that just happen to have the same name ? How would this work in the dcache?

There would be only one entry in the dcache. The lookup will select whether it opens the file or the directory based on O_DIRECTORY (and usage, of course - if it's in the middle of a path, it obviously needs to be opened as a directory regardless).

Hmmm, I just straced "cp /bin/bash /tmp".
One line stood out as a potential problem:

open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4

What do we do with O_CREAT ?

Do we always allow both a directory and a file to be created with the same name ?

Does this create a new class of "symlink attack" style security holes ?

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#717 lore raw

Am Donnerstag, den 26.08.2004, 14:59 -0400 schrieb Rik van Riel:

Quoted 6 lines

open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4

What do we do with O_CREAT ?

Do we always allow both a directory and a file to be created with the same name ?

I would say that the directory under a file is implicit. There are still the pseudo-files a filesystem might want to provide or the VFS if there should be an agreement to generally provide file/pseudo/uid or possibly some security hooks. Some filesystem might want to provide those pseudo files but doesn't have the technical possibility to store non-pseudeo files from the user under such a directory.

#718 lore raw
In reply to Rik van Riel
Quoted 10 lines · Rik van Riel

Hmmm, I just straced "cp /bin/bash /tmp".
One line stood out as a potential problem:

open("/tmp/bash", O_WRONLY|O_CREAT|O_LARGEFILE, 0100755) = 4

What do we do with O_CREAT ?

Do we always allow both a directory and a file to be created with the same name ?

Either I am confused, or you are.

To me, a filesystem that allows this thing doesn't really _have_ the concept of "directory vs file". It's just a "filesystem object", and it can act as _both_ a directory and a file.

So when you create "/tmp/bash" - assuming /tmp supports the file-as-dir semantics at all, you're just creating a new inode - the same you always have. When you write to that inode, it writes to the default stream. There's no special cases here.

Now, after you have your regular /tmp/bash, you can then start adding named streams to it, ie you can do

	open("/tmp/bash/icon", O_WRONLY|O_CREAT|O_LARGEFILE, 0755);

and that will create the "icon" named stream. See?

So "/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part).

Hey, think of it as a wave-particle duality. Both "modes" exist at the same time, and cannot be separated from each other. Which one you see depends entirely on your "experiment", ie how you open the file.

Quoted 2 lines

Does this create a new class of "symlink attack" style security holes ?

I don't believe so.

Linus

#719 lore raw
Quoted 3 lines · Linus Torvalds

So "/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part).

OK, that makes sense.

Quoted 3 lines

Hey, think of it as a wave-particle duality. Both "modes" exist at the same time, and cannot be separated from each other. Which one you see depends entirely on your "experiment", ie how you open the file.

Guess I'm scared again now. We need to make sure that backup programs don't fall victim to the uncertainty principle ;)

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#720 lore raw

Am Donnerstag, den 26.08.2004, 15:44 -0400 schrieb Rik van Riel:

Quoted 7 lines
Quoted 3 lines

Hey, think of it as a wave-particle duality. Both "modes" exist at the same time, and cannot be separated from each other. Which one you see depends entirely on your "experiment", ie how you open the file.

Guess I'm scared again now. We need to make sure that backup programs don't fall victim to the uncertainty principle ;)

At least you can be sure that the backup will go into a defined state as soon as you observe it.

#721 lore raw
In reply to Linus Torvalds
Quoted 3 lines · Linus Torvalds

So "/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part).

Thinking about it some more, how would file managers and file chosers handle this situation ?

Currently the user browses the directory tree and when the user clicks on something, one of the following happens:

1) if it is a directory, the file manager/choser changes into that directory

2) if it is a file, the file is opened

Now how do we present things to users ?

How will users know when an object can only be chdired into, or only be opened ?

For objects that do both, how does the user choose ?

Do we really want to have a file paradigm that's different from the other OSes out there ?

What happens when users want to transfer data from Linux to another system ?

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#722 lore raw

Am Donnerstag, den 26.08.2004, 16:10 -0400 schrieb Rik van Riel:

Quoted 6 lines
Quoted 3 lines

So "/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part).

Thinking about it some more, how would file managers and file chosers handle this situation ?

I would say they don't. That's where metadata and pseudo-files sit. It's up to the applications to do something useful with the data.

Quoted 2 lines

Do we really want to have a file paradigm that's different from the other OSes out there ?

Some other OSes have similar things.

Quoted 2 lines

What happens when users want to transfer data from Linux to another system ?

No one should store really important data inside file/ that makes the file completely useless. I just love these postscript fonts on the Mac. The actual main file stream is 0 bytes in size. The actual font is inside the resource fork. When I want to convert it the Mac user needs to pack it into a compound file using a special tool and then I can decode it using another tool to access the PFB file. Crap.

It's useful for metadata like keywords, ACLs and these things. If you can pack it using an aware backup program, fine. If you don't, you still have the data.

#723 lore raw
Quoted 5 lines · Christophe Saout

I just love these postscript fonts on the Mac. The actual main file stream is 0 bytes in size. The actual font is inside the resource fork. When I want to convert it the Mac user needs to pack it into a compound file using a special tool and then I can decode it using another tool to access the PFB file. Crap.

I thought it was a cunning way to stop me from copying those nice fonts onto my Linux box. The original DRM. Thanks for the tip :)

-- Jamie

#724 lore raw
In reply to Rik van Riel
Quoted 8 lines · Rik van Riel
Quoted 5 lines · Linus Torvalds

So "/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part).

Quoted 10 lines

Thinking about it some more, how would file managers and file chosers handle this situation ?

Currently the user browses the directory tree and when the user clicks on something, one of the following happens:

1) if it is a directory, the file manager/choser changes into that directory

How does the file manager / chooser decide whether you're trying to move into a directory, or the meta-data-directory for a directory?
It's not just files that should have metadata - directories need* them too. Making it possible to see attributes as a directory under a file is great, but you'd still need an O_META flag for accessing directory metadata (since there are already files under a directory).

Quoted 4 lines

2) if it is a file, the file is opened

Now how do we present things to users ?

Quoted 9 lines

How will users know when an object can only be chdired into, or only be opened ?

For objects that do both, how does the user choose ?

Do we really want to have a file paradigm that's different from the other OSes out there ?

MacOS does, Be did (sort of). I'm not sure it would be the end of the world, as long as the data can be extracted.

Quoted 4 lines

What happens when users want to transfer data from Linux to another system ?

That would depend on the other system. Data is easy, metadata is hard.
It would be possible to create an XML schema for metadata, and if requested (O_EVERYTHING?), the file data is returned with all metadata in XML tags. (not advocating this, just an idea :)
- Steve

* I say need in the same way as one *needs* to upgrade their 2GHz computer - it would be nice :)

#725 lore raw
Quoted 6 lines · Stephen Wille Padnos <[email protected]> schrieb/wrote:

How does the file manager / chooser decide whether you're trying to move into a directory, or the meta-data-directory for a directory? It's not just files that should have metadata - directories need* them too. Making it possible to see attributes as a directory under a file is great, but you'd still need an O_META flag for accessing directory metadata (since there are already files under a directory).

A simple convention that meta data files start with, say ".$", would be enough.

Claus

Signature · 1 line
http://www.faerber.muc.de
#726 lore raw
Quoted 2 lines · Claus Färber

A simple convention that meta data files start with, say ".$", would be enough.

POSIX/SuS don't permit this. The only "free" namespace is that starting "//" (and not as some desktops seem to think foo://). Remember always send GUI desktop users files called http://read.me .. its fun 8)

The // doesn't really help because you don't want extensions at the path top.

Another interesting question btw with substreams of a file is what the semantics of fchown, fstat, fchmod, fchdir etc are, or of mounting a file over a substream.

Alan

#727 lore raw
Quoted 7 lines · Alan Cox <[email protected]> schrieb/wrote:
Quoted 2 lines · Claus Färber

A simple convention that meta data files start with, say ".$", would be enough.

POSIX/SuS don't permit this. The only "free" namespace is that starting "//" (and not as some desktops seem to think foo://). Remember always send GUI desktop users files called http://read.me .. its fun 8)

Well, the problem is that one does want the metadata to be in the ordinary namespace so that they are accessible with POSIX tools. (The current version of) POSIX does not have a "unified" namespace so a unified namespace can't be POSIX.

I wonder if it makes sense to have an environment variable (maybe POSIXLY_CORRECT) to switch between POSIX and the unified namespace model. Of course, this would mean that all of this is handled in userspace (e.g. libc), whereas the kernel and filesystems export more "traditional" (e.g. with openat) interfaces. Accessing metadata as ordinary files would then work with other filesystems, too.

Claus

Signature · 1 line
http://www.faerber.muc.de
#728 lore raw
In reply to Rik van Riel
Quoted 3 lines · Rik van Riel

Thinking about it some more, how would file managers and file chosers handle this situation ?

The same way they already handle it on other platforms that support it? By taking advantage of it..

People literally use this for icon and preview information, so some of the stuff shows up very much in file managers. And I'd assume that a normal double-click would just open the main file, and you can right-click for management information, including opening the file.

If you want an entity that acts as a directory, you create a directory.

Directories don't go away - you still have S_ISDIR() and S_ISREG(), and they still return information. A file that has associated information is still a _file_, and people should treat it that way, it's just that it also has a list of named sub-streams. You can open it as a directory, but the stat information clearly says it is a file, and the "directory" view is very much associated with _that_ file.

I definitely don't think you want the file manager to act as if a named stream is exactly the same as a stand-alone file. They have all the same operations, but there's no question that there are differences too. Especially on a conceptual level - but for most filesystems there are likely real technical differences too.

For example, it's likely that most filesystems would _not_ allow linking of a named stream anywhere else. And you might not be able to change the permissions or date on the named stream either, since it may or may not have a separate date/permission thing from the container.

So don't believe that just because the named streams are _named_ like real files, that they suddenly have any existence beyond the container that they are part of.

There may be other limitations too - again depending on how the filesystem actually implements named streams. It might not support more than one level of naming, for example - so you might not be able to create a directory structure within the named streams, for example.

In short: the fact that the VFS layer exposes the _capability_ to see the named streams as a full POSIX filesystem of its own does _not_ mean that the low-level filesystem necessarily allows the full possible semantics. So you shouldn't design your apps that know about named streams to think they are normal directories.

The directory thing is just a very powerful naming scheme, and one that fits into the existing UNIX model.

Quoted 2 lines

Do we really want to have a file paradigm that's different from the other OSes out there ?

Different from what other OSes?

Last I looked, Windows, Solaris, and OSX all supported named streams. What other OS's are out there that you care about?

In other words, this is _not_ a different paradigm from what others do. The discussion is whether we want to implement it at all, and more importantly about syntactic issues (ie we clearly already implement extended attributes, just with a much more limited syntactic power).

We don't have to go all the way. Solaris has "openat()", which is kind of a half-way there - not really directly available in the same namespace, but at least the result is available as a real file interface (as opposed to the Linux "xattr" interfaces that are _totally_ special-cased system calls).

In other words - the paradigm is already there. It's just that currently it's pretty much unusable under Linux, because it requires so much specialized knowledge that it's not worth it to modify existing apps. And the interfaces are really very limited, so even if you _do_ end up using the specialized Linux interfaces, you can't actually do a lot of what you might want to do.

Linus

#729 lore raw
Quoted 4 lines · Linus Torvalds

For example, it's likely that most filesystems would _not_ allow linking of a named stream anywhere else.

Why? It is just a file that inherits its stat data and is referenced by a concatenation method of its parent directory. The link will also inherit its stat data from the original parent directory.

Quoted 3 lines

And you might not be able to change the permissions or date on the named stream either, since it may or may not have a separate date/permission thing from the container.

You should be able to change the permission and data, but when you change it, you change it for the whole container.

#730 lore raw
Quoted 9 lines · Hans Reiser
Quoted 3 lines · Linus Torvalds

For example, it's likely that most filesystems would _not_ allow linking of a named stream anywhere else.

Why? It is just a file that inherits its stat data and is referenced by a concatenation method of its parent directory. The link will also inherit its stat data from the original parent directory.

In reiser4, maybe. Not in general. Look at what other filesystems do for attributes..

Quoted 6 lines
Quoted 3 lines

And you might not be able to change the
permissions or date on the named stream either, since it may or may not have a separate date/permission thing from the container.

You should be able to change the permission and data, but when you change it, you change it for the whole container.

Why? I'd much rather disallow it, than allow it and then have "nonlocal" effects.

I can also see a filesystem validly having _extra_ permissions (ie apart from the container) on it too.

Not all the world is reiser4.

Linus

#731 lore raw
Quoted 8 lines · Linus Torvalds
Quoted 8 lines
Quoted 6 lines

And you might not be able to change the permissions or date on the named stream either, since it may or may not have a separate date/permission thing from the container.

You should be able to change the permission and data,

^data^date

Quoted 7 lines
Quoted 3 lines

but when you change it, you change it for the whole container.

Why? I'd much rather disallow it, than allow it and then have "nonlocal" effects.

Sometimes you want the nonlocal effects and sometimes you don't, and by decomposing streams into smaller primitives we can let users choose as they want.

#732 lore raw
Quoted 3 lines · Hans Reiser

Sometimes you want the nonlocal effects and sometimes you don't, and by decomposing streams into smaller primitives we can let users choose as they want.

Right. Now, would you kindly post the detailed technical analysis of your wonderful design that handles that stuff safely? With proof of correctness, please. Since we are expected to take your code and use it as "Uber" replacement for the existing one, it surely would not be too much to expect, would it?

And no, "we have a nice presentation somewhere on namesys" does not qualify.

I apologize for the obvious posting in a thread already full of noise, but I would like to make *very* sure that lack of ripping you another one does *not* mutate into "no objections from viro" in a lovely thread on SlashDot in a week or so.

#733 lore raw
Quoted 14 lines · [email protected] wrote:
Quoted 3 lines · Hans Reiser

Sometimes you want the nonlocal effects and sometimes you don't, and by decomposing streams into smaller primitives we can let users choose as they want.

Right. Now, would you kindly post the detailed technical analysis of your wonderful design that handles that stuff safely? With proof of correctness, please. Since we are expected to take your code and use it as "Uber" replacement for the existing one, it surely would not be too much to expect, would it?

Quoted 11 lines

And no, "we have a nice presentation somewhere on namesys" does not qualify.

I apologize for the obvious posting in a thread already full of noise, but I would like to make *very* sure that lack of ripping you another one does *not* mutate into "no objections from viro" in a lovely thread on SlashDot in a week or so.

Over the next 6-18 months, one piece at a time, it will all fall into place.

Reiser4 as it is shipping today, consists of close to the minimal functionality necessary to replace reiser3, albeit done according to our plugin model, and with vastly higher performance. The full solution is not yet written, wait for it. I understand why you think deadlocks are everything, but avoiding them is only a small, albeit necessary, piece of the task.

Thanks though for pointing out issues with unlinking files that have been cd'd to.

#734 lore raw
In reply to Rik van Riel
Quoted 4 lines · Rik van Riel

For objects that do both, how does the user choose ?

Do we really want to have a file paradigm that's different from the other OSes out there ?

What does MacOS X do? Someone said that documents are directories in it; it must know how to handle that.

What does Windows do when you click on a .zip file and WinZIP is installed? It opens the .zip file and lets you explore inside. When you click on a .doc file, though, it opens a viewer or editor -- you don't get the option to look inside.

Quoted 2 lines

What happens when users want to transfer data from Linux to another system ?

This is why I favour storing all essential metadata (about the file's content) inside the file's contents, the primary stream.

We have another problem: what happens when users want to transfer data from Windows (with secondary streams) and MacOS (with resource forks)?

-- Jamie

#735 lore raw
Quoted 6 lines · Jamie Lokier

This is why I favour storing all essential metadata (about the file's content) inside the file's contents, the primary stream.

We have another problem: what happens when users want to transfer data from Windows (with secondary streams) and MacOS (with resource forks)?

I can tell you that - right now we (Samba) has to drop all but the primary stream, because the Linux filesystems don't have data stream support. Oh look, we're losing user data (that's a *bad* thing :-).

Jeremy.

#736 lore raw
Quoted 11 lines · Jeremy Allison
Quoted 6 lines · Jamie Lokier

This is why I favour storing all essential metadata (about the file's content) inside the file's contents, the primary stream.

We have another problem: what happens when users want to transfer data from Windows (with secondary streams) and MacOS (with resource forks)?

I can tell you that - right now we (Samba) has to drop all but the primary stream, because the Linux filesystems don't have data stream support. Oh look, we're losing user data (that's a *bad* thing :-).

Maybe you should learn from netatalk.

#737 lore raw
Quoted 2 lines · Christoph Hellwig

Maybe you should learn from netatalk.

I am well aware of what netatalk does. However netatalk isn't as widely used as Samba. Things they can get away with would cause our user community to flay us alive.

We need a proper solution, not a nasty hack.

That's like me telling you to "learn from *BSD". You have different user constituencies, you have to serve yours, I have to serve mine.

Jeremy.

#738 lore raw
Quoted 9 lines · Jeremy Allison

I am well aware of what netatalk does. However netatalk isn't as widely used as Samba. Things they can get away with would cause our user community to flay us alive.

We need a proper solution, not a nasty hack.

That's like me telling you to "learn from *BSD". You have different user constituencies, you have to serve yours, I have to serve mine.

While I do think that there are some hacks in netatalk, the hacks there are not nearly as bad as the hacks that made it into the kernel to support exporting a native Mac filesystem. Take a look at the fork code in the hfs filesystem from 2.4 or before for exporting to netatalk. It's disgusting, and mostly because there is no easy way to export the resource fork and extra metadata that is essential to the client. The 2.6 code doesn't have that anymore, but I don't think it actually works with netatalk either. Perhaps someone can correct me on that, but I don't see how it could work.

I would hope that something comes out of this whole discussion that will allow a much better interface between the hfs/hfsplus code in the kernel and netatalk.

Brad Boyer
[email protected]

#739 lore raw
In reply to Rik van Riel
Quoted 5 lines · Rik van Riel

Do we really want to have a file paradigm that's different from the other OSes out there ?

What happens when users want to transfer data from Linux to another system ?

I think it depends on the transport being used: some transports could allow for metadata (i.e. a MIME-compatible transport) while some others don't. Of course, the remote side needs to also support metadata (i,e. using a MIME-compatible transport against a remote host that doesn't understand metadata is certainly impossible).

For example, with FTP transports, I guess the only possible option is to transfer the unnamed/default stream. NFS should allow metadata, but this probably needs some kind of extensions to the NFS protocol.

#740 lore raw
In reply to Rik van Riel
Quoted 8 lines · Rik van Riel
Quoted 3 lines · Linus Torvalds

So "/tmp/bash" is _not_ two different things. It is _one_ entity, that contains both a standard data stream (the "file" part) _and_ pointers to other named streams (the "directory" part).

Thinking about it some more, how would file managers and file chosers handle this situation ?

Quoted 8 lines

Currently the user browses the directory tree and when the user clicks on something, one of the following happens:

1) if it is a directory, the file manager/choser changes into that directory

2) if it is a file, the file is opened

And now you have a mess. Is it "really" a file, or a directory? Why not just keep both well apart, and stay happy? I just fail to see what could be gained by having directories that really aren't, and files that aren't either. Use a directory if you want one, use a file elsewhere.

Quoted 4 lines

Now how do we present things to users ?

How will users know when an object can only be chdired into, or only be opened ?

Easy: It is a directory, or it isn't.

Quoted 1 line

For objects that do both, how does the user choose ?

Don't give silly choices.

Quoted 2 lines

Do we really want to have a file paradigm that's different from the other OSes out there ?

I vote "no". There are/have been OSes with weird "files", none of them survived to get anywhere as popular as Unix and "file == stream of bytes". Even with much simpler variants like files as sequences of records. For a good reason: The Unix way is simple, and extremely flexible, as my proggie can define at its own whim how to handle what's inside. If a single stream isn't enough, we have directories. No need to innovate there.

Quoted 2 lines

What happens when users want to transfer data from Linux to another system ?

Or between Linux systems with different kernels that happen to implement different views/metadata on files.

Please do remember devfs: It sounded like a cool idea, got into the kernel just to be thrown out later because nobody used it. Much heat was generated, nothing of permanent value. This looks the same: A very vocal tiny minority is clamoring for something completely non-Unix for totally bogus reasons. What happened to "code talks, bullshit walks"? There is _no_ code (== real-world, user programs that can't be done efficiently enough without this), so this nonsense should just be thrown out, and everybody go back to real hacking.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#741 lore raw
Quoted 2 lines · Horst von Brand

Please do remember devfs: It sounded like a cool idea, got into the kernel just to be thrown out later because nobody used it.

Are you kidding? Lots of distros and many embedded Linuxes use devfs.

And udev _still_ doesn't create device nodes properly. (Hint: I have to run two modprobe commands before pppd works. I have to run modprobe before openvpn or bochs work.)

Quoted 1 line

What happened to "code talks, bullshit walks"?

devfs is a fine example of why code isn't enough. With devfs the code came first, the >1 year of strategic bullshit politics from the "it's not traditional unix" crowd came later, then it went in, then lots of people used it, then it was replaced by something which still doesn't work as well as 2.4 does with or without devfs, and people are still using it despite it's faults.

-- Jamie

#742 lore raw
Quoted 3 lines · Jamie Lokier

And udev _still_ doesn't create device nodes properly. (Hint: I have to run two modprobe commands before pppd works. I have to run modprobe before openvpn or bochs work.)

Just because it is supposed to work that way.

Signature · 7 lines
USB is for mice, FireWire is for men!

sUse lINUX ag, nÜRNBERG
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
#743 lore raw
In reply to Jamie Lokier
Quoted 9 lines · Jamie Lokier
Quoted 1 line · Horst von Brand

What happened to "code talks, bullshit walks"?

devfs is a fine example of why code isn't enough. With devfs the code came first, the >1 year of strategic bullshit politics from the "it's not traditional unix" crowd came later, then it went in, then lots of people used it, then it was replaced by something which still doesn't work as well as 2.4 does with or without devfs, and people are still using it despite it's faults.

What is udev's faults that have an issue with?

Yes, we don't do module autoloading when opening a device node, but that's well known, documented, and the way the kernel has evolved to anyway.

thanks,

greg k-h

#744 lore raw
Quoted 11 lines · Greg KH
Quoted 9 lines · Jamie Lokier
Quoted 1 line · Horst von Brand

What happened to "code talks, bullshit walks"?

devfs is a fine example of why code isn't enough. With devfs the code came first, the >1 year of strategic bullshit politics from the "it's not traditional unix" crowd came later, then it went in, then lots of people used it, then it was replaced by something which still doesn't work as well as 2.4 does with or without devfs, and people are still using it despite it's faults.

Quoted 5 lines

What is udev's faults that have an issue with?

Yes, we don't do module autoloading when opening a device node, but that's well known, documented, and the way the kernel has evolved to anyway.

devfs was very natural, and simple solution. But to have it right, it would have to be the only /dev filesystem.
But no, we like choices, so we have chaos.
Udev is just another thing adding to that chaos.

Someone was numbering things that are good in BSD design, in that thread. One of those things was going for devfs. No cheap solutions. One fs for /dev. And it works great.

Sorry for bit of trolling.

Signature · 1 line
GJ
#745 lore raw
Quoted 5 lines · Grzegorz Jaśkiewicz

devfs was very natural, and simple solution. But to have it right, it
would have to be the only /dev filesystem.
But no, we like choices, so we have chaos. Udev is just another thing adding to that chaos.

Quoted 7 lines

Someone was numbering things that are good in BSD design, in that thread. One of those things was going for devfs. No cheap solutions. One fs for /dev. And it works great.

Sorry for bit of trolling.

Devfs was a ver good idea. The implementation of it was a problem, and after some time nobody maintained it. No surprise it had to go. Now udev+tmpfs can do the same job, and more.

Helge Hafting

#746 lore raw
Quoted 12 lines · Helge Hafting
Quoted 10 lines · Grzegorz Jaśkiewicz

devfs was very natural, and simple solution. But to have it right, it would have to be the only /dev filesystem.
But no, we like choices, so we have chaos.
Udev is just another thing adding to that chaos.

Someone was numbering things that are good in BSD design, in that thread. One of those things was going for devfs. No cheap solutions. One fs for /dev. And it works great.

Quoted 7 lines
Quoted 3 lines

Sorry for bit of trolling.

Devfs was a ver good idea. The implementation of it was a problem, and after some time nobody maintained it. No surprise it had to go. Now udev+tmpfs can do the same job, and more.

udef is a one big mistake, having need for userspace tool to use FS is at least silly.
I can understeand need for some things in kernel to have userspace daemon. But FS is out of question the least one.

I am supprised noone wanted to maintain devfs. Maybe because people didn't want to go to devfs only. But still to have classic /dev. It's also silly, because person writing driver needs to choose between, or implement all. That's more than bad. Once I have loads of time, and no work in KDE, I can take over devfs happily :-)

Signature · 1 line
GJ
#747 lore raw

In article <[email protected]>,

Quoted 2 lines · Grzegorz JaÅ kiewicz

I am supprised noone wanted to maintain devfs. Maybe because people didn't want to go to devfs only.

It's because disk was spelled with a 'c' (disc).

Mike.

Signature · 2 lines
"In times of universal deceit, telling the truth becomes
 a revolutionary act." -- George Orwell.
#748 lore raw
Quoted 4 lines · Miquel van Smoorenburg

In article <[email protected]>, Grzegorz JaÅ kiewicz <[email protected]> wrote:

Quoted 8 lines
Quoted 4 lines

I am supprised noone wanted to maintain devfs. Maybe because people didn't want to go to devfs only.

It's because disk was spelled with a 'c' (disc).

That suppose to be a joke ?

--
GJ

#749 lore raw
Quoted 2 lines · Grzegorz Jaśkiewicz
Quoted 1 line

It's because disk was spelled with a 'c' (disc).

That suppose to be a joke ?

The international joke symbol (8-)) is only mandatory in en_US locales.

#750 lore raw
Quoted 14 lines · Alan Cox
Quoted 6 lines · Grzegorz Jaśkiewicz
Quoted 3 lines

It's because disk was spelled with a 'c' (disc).

That suppose to be a joke ?

The international joke symbol (8-)) is only mandatory in en_US locales.

Ok, Planty of times I did asked Brits and Americans if there's any difference. Neither did say that either disc or disk is more correct. And I am using both all the time, noone has any objections ( I live in UK for 2.5years so far ). So.... me is confused :-0

--
GJ

#751 lore raw
Quoted 4 lines

Ok, Planty of times I did asked Brits and Americans if there's any difference. Neither did say that either disc or disk is more correct.

From http://en.wikipedia.org/wiki/Disk_or_disc

Disk or Disc?

The divergence in spelling is due in part to the way in which the words originated. Disk came into the English language in the mid-17th century, and was modelled on words such as whisk; disc arose some time later, and was based on the original Latin root discus. In the 19th century, disc became the conventional spelling for audio recordings made on a flat plate, such as the gramophone record; this usage gave rise to the modern term disc jockey. Early BBC technicians differentiated between disks (in-house transcription records) and discs (the colloquial term for commercial gramophone records, or what the BBC dubbed CGRs).

By the 20th century, the c-spelling was more popular in British English, while the k-spelling was preferred in American English. In the 1940s, when the American company IBM pioneered the first hard disk storage devices, the k-spelling was used. In 1979 the European company Philips, along with Sony, developed the compact disc medium; here, the c-spelling was chosen, possibly because of the predominating British spelling, or because the compact disc was seen as a successor to the analogue disc record.

Whatever their heritage, in computer jargon today it is common for the k-spelling to refer mainly to magnetic storage devices, while the c-spelling is customary for optical media such as the compact disc and similar technologies. Even in the computing field, however, the terms are used inconsistently; software documentation often uses the k-spelling exclusively.

Signature · 3 lines
                          I won't rest till it's the best ...
                          Programmer, Linux Scalability
                          Paul Jackson <[email protected]> 1.650.933.1373
#752 lore raw
In reply to Grzegorz Jaśkiewicz
Quoted 16 lines · Grzegorz Ja??kiewicz
Quoted 15 lines · Helge Hafting
Quoted 13 lines · Grzegorz Ja??kiewicz

devfs was very natural, and simple solution. But to have it right, it would have to be the only /dev filesystem.
But no, we like choices, so we have chaos.
Udev is just another thing adding to that chaos.

Someone was numbering things that are good in BSD design, in that thread. One of those things was going for devfs. No cheap solutions. One fs for /dev. And it works great.

Sorry for bit of trolling.

Quoted 7 lines
Quoted 4 lines

Devfs was a ver good idea. The implementation of it was a problem, and after some time nobody maintained it. No surprise it had to go. Now udev+tmpfs can do the same job, and more.

udef is a one big mistake, having need for userspace tool to use FS is at least silly.

I have never heard of a program called "udef". Any pointers to it? :)

If it's such a big mistake, then don't use it. No one is forcing you to.

Quoted 2 lines

I can understeand need for some things in kernel to have userspace daemon. But FS is out of question the least one.

It's not a daemon (although to make things a bit easier on itself, it does provide one by default, but some distros don't use it.) It just creates device nodes when the kernel finds a new device, with a name that you pick. Which is something that an in-kernel devfs can not do.

Quoted 5 lines

I am supprised noone wanted to maintain devfs. Maybe because people didn't want to go to devfs only. But still to have classic /dev. It's also silly, because person writing driver needs to choose between, or implement all. That's more than bad. Once I have loads of time, and no work in KDE, I can take over devfs happily :-)

Well, until you do that, baseless criticism and trolling like this will be ignored.

greg k-h

#753 lore raw
In reply to Grzegorz Jaśkiewicz
Quoted 28 lines · Grzegorz Jaśkiewicz
Quoted 18 lines · Helge Hafting
Quoted 5 lines · Grzegorz Jaśkiewicz

devfs was very natural, and simple solution. But to have it right, it would have to be the only /dev filesystem.
But no, we like choices, so we have chaos.
Udev is just another thing adding to that chaos.

Quoted 9 lines

Someone was numbering things that are good in BSD design, in that thread. One of those things was going for devfs. No cheap solutions. One fs for /dev. And it works great.

Sorry for bit of trolling.

Quoted 6 lines

Devfs was a ver good idea. The implementation of it was a problem, and after some time nobody maintained it. No surprise it had to go. Now udev+tmpfs can do the same job, and more.

Quoted 4 lines

udef is a one big mistake, having need for userspace tool to use FS is at least silly.

Well, devfs had devfsd - a userspace tool . . .

Quoted 5 lines

I can understeand need for some things in kernel to have userspace daemon. But FS is out of question the least one.

I am supprised noone wanted to maintain devfs.

I believe it had soemthing to do with the design - in order to fix it you had to rewrite it almost from scratch. People work on whatever they want to, and devfs wasn't it.

Quoted 7 lines

Maybe because people
didn't want to go to devfs only. But still to have classic /dev. It's also silly, because person writing driver needs to choose between, or implement all. That's more than bad. Once I have loads of time, and no work in KDE, I can take over devfs happily :-)

Go ahead! Perhaps you get it right. Then you'll have to convince users of udev (or plain old /dev) that your way is better.

Helge Hafting

#754 lore raw

Salut,

Quoted 4 lines · Helge Hafting
Quoted 2 lines

udef is a one big mistake, having need for userspace tool to use FS is at least silly.

Well, devfs had devfsd - a userspace tool . . .

devfsd was not a requirement, it was only a tool for people who wanted their old device names for devfs devices. I was once doing a miniature environment for a package installation system which used pure devfs
(i.e. no old dev names), and it was doing super well without
devfs. sysinstall was the only process running, and I could install Linux on any writeable device without using devfsd.

Tonnerre

#755 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds

To me, a filesystem that allows this thing doesn't really _have_ the concept of "directory vs file". It's just a "filesystem object", and it can act as _both_ a directory and a file.

I agree.

#756 lore raw
In reply to Rik van Riel
Quoted 4 lines · Rik van Riel

What do we do with O_CREAT ?

Do we always allow both a directory and a file to be created with the same name ?

I haven't analysed it thoroughly. I suspect it's fine as long as:

(1) O_CREAT creates something with "file-like attributes", meaning stat() says it's a regular file.

(2) File-like means it can be unlinked, linked and renamed, even if someone has something inside it open. Nothing that can be created inside it will prevent it from being unlinked (unlike a directory-like object, where rmdir() will refuse if it's not empty).

File-like also means that programs like "cp from to_my_new_file" won't do anything so silly as to write _inside_, the way that "cp from to_my_new_file/from" would.

Basic utilities will need to be checked to make sure they don't try appending "/" as their method of deciding if the target object is a directory and should be entered.

Quoted 2 lines

Does this create a new class of "symlink attack" style security holes ?

Yes, but they don't need O_CREAT.

An adversary creates a symlink to metadata inside your file. You write to it: it has interesting effects that weren't anticipated, such as either modifying another (virtual) file, or altering permissions or other parameters which writing doesn't normally do.

This is very difficult to prevent.

We are creating a way for scripts and classic unix tools to have easy access to fancy attributes which may affect the security: things like being able to change a file's permissions just by writing to the appropriate path, or reading characteristics of files that shouldn't be visible. In Hans Reiser's example of expanded /etc/passwd, atimes and mtimes of individual passwd entries is security information that perhaps shouldn't be exposed.

In a way, these holes are similar to /proc, in that just writing has side effects which might not be acceptable.

The solution is the same as for /proc (I hope): make sure the read permissions on all metadata inside a directory branch are restricted to the permissions of the file branch, and write permissions even more restricted at least by default.

For the paranoid, one way is to make new files "not readable or executable as directories". It's possible for the file and directory branches to have different permission bits, though I could see that getting a bit annoying. Other possibilities abound.

-- Jamie

#757 lore raw

Am Donnerstag, den 26.08.2004, 20:29 +0100 schrieb Jamie Lokier:

Quoted 7 lines

(1) O_CREAT creates something with "file-like attributes", meaning stat() says it's a regular file.

(2) File-like means it can be unlinked, linked and renamed, even if someone has something inside it open. Nothing that can be created inside it will prevent it from being unlinked (unlike a directory-like object, where rmdir() will refuse if it's not empty).

I would say hat the file and all of its directory gets invalidated and deleted as soon as the last opener of the file's main stream or some of its directory content disappears. How would that be?

Quoted 9 lines
Quoted 2 lines

Does this create a new class of "symlink attack" style security holes ?

Yes, but they don't need O_CREAT.

An adversary creates a symlink to metadata inside your file. You write to it: it has interesting effects that weren't anticipated, such as either modifying another (virtual) file, or altering permissions or other parameters which writing doesn't normally do.

Hmm.

Quoted 1 line

This is very difficult to prevent.

Right. But to change some properties you usually need to be the owner of that file anyway, or root.

Quoted 3 lines

In Hans Reiser's example of expanded /etc/passwd, atimes and mtimes of individual passwd entries is security information that perhaps shouldn't be exposed.

Only assuming mtime/ctime/atime/rwx/uid/gid of file contents can actually be independant of those from the main stream. I would say that the rights (read and write) for non-pseudo sub-files should the same as the file, uid and gid 644 (only owner can change these), something like that. mtime, atime and ctime always show the same as the file itself. Hmm.

Quoted 4 lines

The solution is the same as for /proc (I hope): make sure the read permissions on all metadata inside a directory branch are restricted to the permissions of the file branch, and write permissions even more restricted at least by default.

That's what I was thinking.

#758 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 3 lines · Rik van Riel

So you'd have both a file and a directory that just happen to have the same name ? How would this work in the dcache?

There would be only one entry in the dcache. The lookup will select whether it opens the file or the directory based on O_DIRECTORY (and usage, of course - if it's in the middle of a path, it obviously needs to be opened as a directory regardless).

hmm, that might be interesting for applications like apache which allow to append arguments by 'building' a 'virtual' path ...

http://www.kernel.org/list.cgi/some/more/args

where the web server actually decides where the script ends and the arguments start by checking the filesystem ...

Quoted 11 lines

That's not the problem. The problem from a dcache standpoint ends up being when the file has a link, and you have two paths to the same sub-file through two different ways:

.. create file 'x' with named stream 'y' ... ln x z
ls -l x/y z/y /* it's the same attribute!! */

but this is actually exactly the same thing that we already have with mounts, ie it is equivalent (from a dentry standpoint) to

Quoted 8 lines
	.. create directory 'x' with file 'y' ..
	mkdir z
	mount --bind x z
	ls -l x/y z/y	/* It's the same file!! */

so none of this is really anything "new" from a dcache standpoint.

Except for all the details, of course ;)

if the file is removed, but some attributes are 'locked' by access, will the 'other' attributes remain visible or disappear ... leaving a partial 'view' of the contents? or did I get the idea completely wrong?

TIA,
Herbert

Quoted 5 lines

Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#759 lore raw
In reply to Linus Torvalds

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Linus Torvalds wrote:
[...]
| tar cvf file.tar file file/

I don't like it. Imagine this:

$ cat file/perm
040700 drwx------
$ cat file/perm/user/writable
1
$ cat file/perm/uid
0

Do that tar, and you lose the uid. This is an example, and has plenty of flaws. The point is that often there may be files/directories within files/directories.

In order to make this work, you either have to patch tar or split every file/directory into a file/directory pair, recursively, using some sort of mangling. "file" becomes file.f and file.d. Then tar works.

To be honest, I like my "serializer" better. It just feels faster and cleaner.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQS4z1XgHNmZLgCUhAQKXAQ//eJ9tPFFEf1kpFY7gYUhMSUAAs1wIvVrN pOVyLPJwGTv4oIaq8/8Op+XmQt1BWfnFdgwSal5R7vuLlrj0/Lp714I4sTOb35+6 q4hB1Q6X1oKbS1QIVWM9zw8PaebTY1GZDYITqWEN8xFN2cx41m4HZZ9P2TXNGPWx 1TqeBAaR07N1LaDzDHWvXu2v8pP6Wjy8mmWbtHbjN7U8oLEAUVXmT/ZJ5MpVK6z6 FTTA6NnARpEDtWebTJit14IQYG88syl31+lNL/N87iTveYI3+6gAQniajkMda565 PnPbVCd/aLHpx4hNLYDiNZwGkhmCgA/V3/vHJGT1vbTWN56zFLF7XQqE0BMfR5k6 ozwzBZ368+suEs0pDplCJq7ru0BF3j49r2Ad7aILCJOENvN0uVzLNxmu+LfvARTm gG7UNkJYcAjwmUaE1vXEACnwKhtyyxIpRbsb9ALqbDS9KXB7eAaI2FwLPXCU/L61 qHWHN47KllWdfT4fttwPd/Lag7UpKu9McBPnp9ZOTuYGY6QTgflBG1C7kallcJUw 7cZh831PRqmnj2hWcAlSvinD3ClxaF9swewMPfQJ9I10n7p6w9Q9EJEXUX6+gUhv 5MD6q2WxqNVM/i5Y/2s3GQ+3lJ2VvnWHkJgc7JkmP0H1N0vocYWhKaofMOQXAy57 /ocTFfcwzUc=
=YrfX
-----END PGP SIGNATURE-----

#760 lore raw

Why are you guys even considering going to any pain at all to distort semantics for the sake of backup? tar is easy, we'll fix it and send in a patch.

#761 lore raw
Quoted 3 lines · Hans Reiser

Why are you guys even considering going to any pain at all to distort semantics for the sake of backup? tar is easy, we'll fix it and send in a patch.

Because not everybody uses tar. Quite a few people use a network backup system, while others use duplicity, RPM uses cpio internally and big companies tend to use proprietary network backup suites.

Breaking people's setup is something to worry about.

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#762 lore raw
Quoted 8 lines · Rik van Riel
Quoted 3 lines · Hans Reiser

Why are you guys even considering going to any pain at all to distort semantics for the sake of backup? tar is easy, we'll fix it and send in a patch.

Quoted 9 lines

Because not everybody uses tar. Quite a few people use a network backup system, while others use duplicity, RPM uses cpio internally and big companies tend to use proprietary network backup suites.

Breaking people's setup is something to worry about.

It is the tail that should not wag the dog. New semantics are going to break backups other than dd. We need a LOT of new semantics if we don't want to be inferior to Apple and MS.

We should implement backup plugin methods though, so that we only have to break things once....

#763 lore raw
Quoted 12 lines · Hans Reiser
Quoted 11 lines · Rik van Riel
Quoted 3 lines · Hans Reiser

Why are you guys even considering going to any pain at all to distort semantics for the sake of backup? tar is easy, we'll fix it and send in a patch.

Because not everybody uses tar. Quite a few people use a network backup system, while others use duplicity, RPM uses cpio internally and big companies tend to use proprietary network backup suites.

Quoted 8 lines
Quoted 2 lines

Breaking people's setup is something to worry about.

It is the tail that should not wag the dog. New semantics are going to
break backups other than dd. We need a LOT of new semantics if we
don't want to be inferior to Apple and MS.

We should implement backup plugin methods though, so that we only have to break things once....

Every Linux shop I know of uses amanda anyway, which will not be affected by these changes, as it uses dump/restore. restore will need minor changes, I think dump might not have to be changed at all.

If your backup system can't easily handle these changes, that's a pretty good indication that it's poorly designed.

Lee

#764 lore raw
In reply to Hans Reiser
Quoted 1 line · Hans Reiser

It is the tail that should not wag the dog.

That's no reason for dismissing technical questions with vigorous handwaving.

Quoted 2 lines

New semantics are going to break backups other than dd. We need a LOT of new semantics if we don't want to be inferior to Apple and MS.

Technical objections need to be addressed, otherwise Linux will end up inferior to Apple and MS. Whatever choice we end up making, it should be something that's both workable and elegant.

It would be really nice if all the technical issues were worked out on linux-kernel before the files-as-directories stuff gets merged. There are quite a few technical questions in the thread that haven't been answered yet...

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#765 lore raw
Quoted 8 lines · Rik van Riel

It would be really nice if all the technical issues were worked out on linux-kernel before the files-as-directories stuff gets merged. There are quite a few technical questions in the thread that haven't been answered yet...

Well, did I answer the technical question with the filename/pseudos/backup method answer in your opinion?

#766 lore raw
In reply to Rik van Riel
Quoted 4 lines · Rik van Riel

Because not everybody uses tar. Quite a few people use a network backup system, while others use duplicity, RPM uses cpio internally and big companies tend to use proprietary network backup suites.

Big backup systems support scripts that can be run prior to backup and post restore. You aready have to use star to do a metadata backup the acl's/ea into some tgz, which can be unpacked after restore.

Quoted 1 line

Breaking people's setup is something to worry about.

Yeah, maybe. But as usual on UN*X like systems most admins are able to come up with a quick shell/perl script to solve their particular problem.

Signature · 1 line
lg, Chris
#767 lore raw
In reply to Hans Reiser
Quoted 3 lines · Hans Reiser

Why are you guys even considering going to any pain at all to distort semantics for the sake of backup? tar is easy, we'll fix it and send in a patch.

It's not as easy as you make it out, and not just because there are a few dozen backup programs that need fixing.

The problem is more fundamental than that. Some of the file streams proposed need to be backed up, while others are alternative presentations of the file, which should not be backed up.

Currently I see no way to distinguish between the stuff that should be backed up and the stuff that shouldn't.

That problem needs to be resolved before we can even start thinking about fixing archivers...

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#768 lore raw
Quoted 6 lines · Hans Reiser
Quoted 4 lines

Why are you guys even considering going to any pain at all to distort semantics for the sake of backup? tar is easy, we'll fix it and send in a patch.

Quoted 7 lines

It's not as easy as you make it out, and not just because there are a few dozen backup programs that need fixing.

The problem is more fundamental than that. Some of the file streams proposed need to be backed up, while others are alternative presentations of the file, which should not be backed up.

No, not really. This is a user decision and should be options in the backup software. I don't think it is up to the kernel, filesystem, or the OS in general to decide what information the user want to retain or not.

Quoted 5 lines

Currently I see no way to distinguish between the stuff that should be backed up and the stuff that shouldn't.

That problem needs to be resolved before we can even start thinking about fixing archivers...

The archivers should, as I said, allow the user to choose. It shouldn't be automatic. Default, should IMO be to store everything.

~S

#769 lore raw
Quoted 9 lines · Spam
Quoted 4 lines

The problem is more fundamental than that. Some of the file streams proposed need to be backed up, while others are alternative presentations of the file, which should not be backed up.

No, not really. This is a user decision and should be options in the backup software. I don't think it is up to the kernel, filesystem, or the OS in general to decide what information the user want to retain or not.

It is helpful for the OS, or a naming convention, to indicate what _is information_ though.

It makes no sense to backup two or more copies of the _same information_, and it makes even less sense to try to restore them as it'll either be slow, fail (you can't always write to alternative presentations), or cause unwanted side effects.

Just like when you backup a dynamic web site. You store the files which the server is using. You don't use "wget" to store the generated pages, that's not a useful backup and you can't restore from it.

Quoted 8 lines
Quoted 5 lines

Currently I see no way to distinguish between the stuff that should be backed up and the stuff that shouldn't.

That problem needs to be resolved before we can even start thinking about fixing archivers...

The archivers should, as I said, allow the user to choose. It shouldn't be automatic. Default, should IMO be to store everything.

Don't try to store different views of the same thing.
When you try to restore, you _won't_ necessarily get back what you stored.

Whereas if you follow the OS's advice, and skip virtual files, then backup and restore will recreate the filesystem, which is what you want.

_That's_ storing everything. It's what you want from a backup.

-- Jamie

#770 lore raw
Quoted 8 lines · Jamie Lokier

It is helpful for the OS, or a naming convention, to indicate what _is information_ though.

It makes no sense to backup two or more copies of the _same information_, and it makes even less sense to try to restore them as it'll either be slow, fail (you can't always write to alternative presentations), or cause unwanted side effects.

You sure you don't mean what is data (on disk) and what is information, ie. interpretted data?

Following this whole threat is quite interesting, but I think there are two things that need to be pointed out (which I might've missed) is that hardlinks would be backed up multiple times if the archiver doesn't know the semantics of the filesystem. This is the reason why the <fs-of-choice>dump usually are faster than tar/cpio/etc. as they understand the low level and can optimize at that level.

Also, a user might only have access to a certain subset of information, which he might to backup but not the rest of the stream/data, and such the choice should be the user's.

Quoted 3 lines

Just like when you backup a dynamic web site. You store the files which the server is using. You don't use "wget" to store the generated pages, that's not a useful backup and you can't restore from it.

That depends ;^)
But in the general case it's true.

#771 lore raw
In reply to Jamie Lokier
Quoted 17 lines
Quoted 9 lines · Spam
Quoted 4 lines

The problem is more fundamental than that. Some of the file streams proposed need to be backed up, while others are alternative presentations of the file, which should not be backed up.

No, not really. This is a user decision and should be options in the backup software. I don't think it is up to the kernel, filesystem, or the OS in general to decide what information the user want to retain or not.

It is helpful for the OS, or a naming convention, to indicate what _is information_ though.

Quoted 8 lines

It makes no sense to backup two or more copies of the _same information_, and it makes even less sense to try to restore them as it'll either be slow, fail (you can't always write to alternative presentations), or cause unwanted side effects.

Just like when you backup a dynamic web site. You store the files which the server is using. You don't use "wget" to store the generated pages, that's not a useful backup and you can't restore from it.

I do not agree. Everything can be considered information, even if it is derived from an already existing file.

If the user wants to not backup certain things then that is an option to tell the backup program.

Quoted 11 lines
Quoted 8 lines
Quoted 5 lines

Currently I see no way to distinguish between the stuff that should be backed up and the stuff that shouldn't.

That problem needs to be resolved before we can even start thinking about fixing archivers...

The archivers should, as I said, allow the user to choose. It shouldn't be automatic. Default, should IMO be to store everything.

Don't try to store different views of the same thing.
When you try to restore, you _won't_ necessarily get back what you stored.

No, it would be restored as it were before with the meta information intact.

Quoted 4 lines

Whereas if you follow the OS's advice, and skip virtual files, then backup and restore will recreate the filesystem, which is what you want.

_That's_ storing everything. It's what you want from a backup.

It would be storing everything but the virtual files.

Quoted 1 line

-- Jamie

#772 lore raw
Quoted 6 lines · Spam
Quoted 4 lines

Whereas if you follow the OS's advice, and skip virtual files, then backup and restore will recreate the filesystem, which is what you want.

_That's_ storing everything. It's what you want from a backup.

It would be storing everything but the virtual files.

Thing is, there is no way to distinguish between what are virtual files and what are actual streams hidden inside a file. You don't know what should and shouldn't be backed up...

Hans says there must be a way, somehow, but I haven't seen him tell this mailing list what exactly it is ;)

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#773 lore raw
Quoted 5 lines · Rik van Riel

Thing is, there is no way to distinguish between what are virtual files and what are actual streams hidden inside a file. You don't know what should and shouldn't be backed up...

I think that lack of distinguishing poiwer is more serious for directories. The more I think I think about it, the more I wonder whether Solaris did things right - having a special operation to "cross the boundary".

I suspect Solaris did it that way because it's a hell of a lot easier to do it like that, but regardless, it would solve the issue of real directories having both real children _and_ the "extra streams".

Linus

#774 lore raw

Just have a special name instead of a special boundary, or, better, have a filename/pseudos/backup method that outputs everything needed to backup the object "filename".

Hans

Quoted 19 lines · Linus Torvalds
Quoted 6 lines · Rik van Riel

Thing is, there is no way to distinguish between what are virtual files and what are actual streams hidden inside a file. You don't know what should and shouldn't be backed up...

I think that lack of distinguishing poiwer is more serious for directories. The more I think I think about it, the more I wonder whether Solaris did things right - having a special operation to "cross the boundary".

I suspect Solaris did it that way because it's a hell of a lot easier to do it like that, but regardless, it would solve the issue of real directories having both real children _and_ the "extra streams".

Linus

#775 lore raw

I am thinking that is there was a proper API for accessing the filesystem then this problem wouldn't arise because things could be done behind the curtains inside the API, instead of having all the tools to be rewritten to know.

Think of FAT32, for example, the new filenames will be retained even if you use old tools on them. It is only if you mount it as a FAT16 volume they will be lost.

Even if this is fixed for Reiser4 now. Next time someone wants to make changes like this we have the exact same problem yet again.

Quoted 32 lines

Just have a special name instead of a special boundary, or, better, have a filename/pseudos/backup method that outputs everything needed to backup the object "filename".

Hans

Quoted 24 lines · Linus Torvalds
Quoted 6 lines · Rik van Riel

Thing is, there is no way to distinguish between what are virtual files and what are actual streams hidden inside a file. You don't know what should and shouldn't be backed up...

I think that lack of distinguishing poiwer is more serious for directories. The more I think I think about it, the more I wonder whether Solaris did things right - having a special operation to "cross the boundary".

I suspect Solaris did it that way because it's a hell of a lot easier to do it like that, but regardless, it would solve the issue of real directories having both real children _and_ the "extra streams".

Linus

#776 lore raw
In reply to Hans Reiser
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Hans" == Hans Reiser <[email protected]> writes:

Hans> Just have a special name instead of a special boundary, or, Hans> better, have a filename/pseudos/backup method that outputs Hans> everything needed to backup the object "filename".

For the Reiser4 case, this could also be solved using views, right? Just create a view that exposes everything that should be backed up, and nothing else? What "instructions" would a backup program need, other than knowing what files to backup, and what files to not backup?

Where are views going to be handled? It seems like something that would be useful for every other filesystem, and so shouldn't be handled too close to Reiser4. (Or is that a debate/discussion/flamewar we don't want to get into right now? ;-) )

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#777 lore raw
In reply to Linus Torvalds
Quoted 9 lines · Linus Torvalds
Quoted 5 lines · Rik van Riel

Thing is, there is no way to distinguish between what are virtual files and what are actual streams hidden inside a file. You don't know what should and shouldn't be backed up...

Quoted 8 lines

I think that lack of distinguishing poiwer is more serious for directories. The more I think I think about it, the more I wonder whether Solaris did things right - having a special operation to "cross the boundary".

I suspect Solaris did it that way because it's a hell of a lot easier to do it like that, but regardless, it would solve the issue of real directories having both real children _and_ the "extra streams".

There are many ways of doing this. Several extra streams to a directory that aren't ordinary files in the directory?

It seems to me that we can get a lot of nice functionality in a simpler way:
Instead of thinking about a number of streams attached to something that is either an ordinary file or directory, just say that the only change will be that a directory may have a _single_ file stream in addition to being a plain directory.

The conceptual change is smaller. Still you achieve multiple streams, the "main" stream in such a dir+file is the single stream attached to the directory. All the "extra streams" are files inside the (normal) directory.

* This way, the extra streams aren't "special" in any way and should work fine with existing tools.
* Serving such a filesystem to clients that use multiple streams will allow any number of streams per file for them
* Moving the directory+file thing will move both the directory and the special stream, and all the extra streams follow because they're �part of the directory as usual�.

If the VFS is to be extended in order to support file-as-directory (or vice versa) then hopefully it can be done in a simple way.

Helge Hafting

-
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

#778 lore raw
Quoted 6 lines · Helge Hafting
Quoted 6 lines

I think that lack of distinguishing power is more serious for directories. The more I think I think about it, the more I wonder whether Solaris did things right - having a special operation to "cross the boundary".

Quoted 6 lines
Quoted 3 lines

I suspect Solaris did it that way because it's a hell of a lot easier to do it like that, but regardless, it would solve the issue of real directories having both real children _and_ the "extra streams".

There are many ways of doing this. Several extra streams to a directory that aren't ordinary files in the directory?

Well.. Yes. We already have "." and "..", which are "special extra streams" in a sense. However, people expect them, and know to ignore them. The same wouldn't be true of new naming.

Quoted 5 lines

It seems to me that we can get a lot of nice functionality in a simpler way:
Instead of thinking about a number of streams attached to something that is either an ordinary file or directory, just say that the only change will be that a directory may have a _single_ file stream in addition to being a plain directory.

That doesn't really help us. What would the name be, and how could you avoid clashes?

Quoted 2 lines

If the VFS is to be extended in order to support file-as-directory (or vice versa) then hopefully it can be done in a simple way.

I'm pretty confident that we can extend the VFS layer to support named streams (see the technical discussion with Al, rather than the flames in this thread). I also clearly believe that it is worth it, but I'm starting to wonder if we should have a special open flag to make people select the stream.

If you look at the Solaris interface, the _nice_ part about "openat()" is that you can do something like

	file = open(filename, O_RDONLY);
	if (file < 0)
		return -ENOENT;
	icon = openat(file, "icon", O_RDONLY | O_XATTR);
	if (icon < 0)
		icon = default_icon_file;
	..

and it will work regardless of whether "filename" is a directory or a regular file, if I've understood correctly.

Now, I think that makes sense for several reasons:
- single case
- race-free (think "stat()" vs "fstat()" races).
- I think we want to do "openat()" regardless of whether we ever support extended attributes or not ("openat()" is nice for doing "namei()" in user space even in the absense of any attributes or named streams).

So what we can do is
- implement openat() regardless, and expect to do the Solaris thing for it if we ever do streams.
- _also_ support the "implied named attributes" for regular files, so that you don't have to use "openat()" to access them.

Comments? Does anybody hate "openat()" for any reason (regardless of attributes)? We can easily support it, we'd just need to pass in the file to use as part of the "nameidata" thing or add an argument (it would also possibly be cleaner if we made "fs->pwd" be a "struct file").

Linus

#779 lore raw
Quoted 3 lines · Linus Torvalds

Well.. Yes. We already have "." and "..", which are "special extra streams" in a sense. However, people expect them, and know to ignore them. The same wouldn't be true of new naming.

If the streams that don't contain data weren't real files, but instead fifos, backup programs would ignore them fine.

Tar seems to ignore special nodes ...

Signature · 3 lines
"Debugging is twice as hard as writing the code in the first place.
Therefore, if you write the code as cleverly as possible, you are,
by definition, not smart enough to debug it." - Brian W. Kernighan
#780 lore raw
In reply to Linus Torvalds
Quoted 4 lines · Linus Torvalds

Comments? Does anybody hate "openat()" for any reason (regardless of attributes)? We can easily support it, we'd just need to pass in the file to use as part of the "nameidata" thing or add an argument (it would also possibly be cleaner if we made "fs->pwd" be a "struct file").

What would your openat() produce? Normal struct file? Then what's going to be its vfsmount/dentry and what will they be attached to?

#781 lore raw
Quoted 3 lines · On Sat, 28 Aug 2004 [email protected] wrote:

What would your openat() produce? Normal struct file? Then what's going to be its vfsmount/dentry and what will they be attached to?

Normal file descriptor, exactly like "open()".

And it's going to have all the same vfsmount/dentry thing it would have if you looked it up the whole way. I don't understand your question..

Ignore the O_XATTR thing for a while, and assume it's just a convenient combination of "fchdir + open" (plus "fchdir back" of course, but that's beside the point - openat() doesn't ever really change cwd).

Going back to O_XATTR: that would end up doing the "special vfsmount" magic at the beginning of the lookup. If the dentry you started with wasn't marked D_HYBRID, it would just return -ENOTDIR.

So we could do openat() _without_ any of the lookup_mnt() etc special cases. This interface is independent of whether we want to expose the attributes through a normal lookup - we can do either, both, or neither as we choose.

The nice thing about "openat()" is
- people can definitely find uses for it even without attributes.
- it's "portable". Well, at least somebody else does the same thing, which is nice for user-space developers. You don't use a Linux-only interface, you use a Linux/Solaris one, which makes a lot of people a lot more happy.

Remember, portability has always been very important to Linux, and Linux-only features while nice are certainly not as nice as features you can also find in other places.

NIH is a disease.

Linus

#782 lore raw
Quoted 3 lines · Linus Torvalds

Going back to O_XATTR: that would end up doing the "special vfsmount" magic at the beginning of the lookup. If the dentry you started with wasn't marked D_HYBRID, it would just return -ENOTDIR.

OK, let me restate the question - what do we get from pwd if we do fchdir() to such beast?

#783 lore raw
Quoted 8 lines · On Sat, 28 Aug 2004 [email protected] wrote:
Quoted 3 lines · Linus Torvalds

Going back to O_XATTR: that would end up doing the "special vfsmount" magic at the beginning of the lookup. If the dentry you started with wasn't marked D_HYBRID, it would just return -ENOTDIR.

OK, let me restate the question - what do we get from pwd if we do fchdir() to such beast?

We'll see the "attribute path". We could (if we want to) mark the point where we walked into attribute space somehow (since we can see it by just looking at the vfsmount: d_root/d_mountpoint being the same), but even if we don't, we'll get a sane-looking path.

(It's not just "getcwd()", it's /proc interfaces too, for opening files, and we already have the notion of magic markers like "(deleted)" to show human-readable information).

The question about what to do at the "attribute point" (and there may actually be several, if the filesystem supports attributes on attribute files) likely depends on whether we support the previously discussed "lookup()" magic for attributes.

So if we do support it, I think we should just make the attribute point look exactly like a normal directory, since that path would work as-is. If we don't support it (and at directory points), we might want to just consider it a special root.

NOTE! Anybody who tries to use the "getcwd()" string as a real path is already broken - you have pathname permissions that may not make it possible to look up the path you see.

So we have multiple options:

- support "file/attribute" lookup: show the path as-is, so you'd show it as
"/path/to/file/attribute"

- alternatively, even if you _do_ support the normal lookup, show it with a double slash (which will still be a valid path), just as a visual clue:

	"/path/to/file//attribute"

- for directories, or if we do _not_ support the extended lookup format, we could show it the same way we show deleted files, something like this:
"/path/to/file/attribute (attr)"

- using "http notation" for non-standard-namespaces (we already do this for sockets and pipes, for example)

"attr:attribute@/path/to/file"

pick your poison.

Linus

#784 lore raw

On Sat, Aug 28, 2004 at 12:16:47PM -0700, Linus Torvalds wrote:

If I see it correctly, you want to group a file and some ancillary files together.

The Unix way would be to make a directory and put them all there:
xterm/xterm
xterm/xterm.icon

But you are unsatisfied and want
xterm
xterm/xterm.icon

As long as we agree that the latter really means the former, there are no problems in finding out what should happen.

The conclusion is, that a directory carries an additional bit that says "if I am opened as a regular file then use the file of the same name inside".

Now there is no attribute space, just a shorthand.

Andries

#785 lore raw
Quoted 12 lines · Andries Brouwer

If I see it correctly, you want to group a file and some ancillary files together.

The Unix way would be to make a directory and put them all there:
xterm/xterm
xterm/xterm.icon

But you are unsatisfied and want
xterm
xterm/xterm.icon

Quoted 8 lines

As long as we agree that the latter really means the former, there are no problems in finding out what should happen.

The conclusion is, that a directory carries an additional bit that says "if I am opened as a regular file then use the file of the same name inside".

Now there is no attribute space, just a shorthand.

It's more than a shorthand, though. _Much_ more.

There's the small issue of atomicity and locking.

There's the small issue of hardlinks.

Both of those are why it would need special support.

Linus

#786 lore raw
Quoted 5 lines · Linus Torvalds
Quoted 1 line

Now there is no attribute space, just a shorthand.

It's more than a shorthand, though. _Much_ more.
...
Both of those are why it would need special support.

Yes - support, I do not argue against that.
But I argue against an attribute space.

This xterm.icon was xterm.icons - with both large and small icons. And these icons have copyrights. Etc. There is just an arbitrary tree below this xterm filesystem object. The files below are in no way different from the files above.

Unix says: everything is a file. Now we get: everything is a directory. And some directories have a file attached.

Andries

#787 lore raw
In reply to Linus Torvalds
Quoted 8 lines · Linus Torvalds

So if we do support it, I think we should just make the attribute point look exactly like a normal directory, since that path would work as-is. If we don't support it (and at directory points), we might want to just consider it a special root.

NOTE! Anybody who tries to use the "getcwd()" string as a real path is already broken - you have pathname permissions that may not make it possible to look up the path you see.

OK, forget getcwd(). What does lookup of .. do from that point? *Especially* for stuff you've got from regular files. That's the decision that needs to be made.

#788 lore raw
Quoted 4 lines · On Sat, 28 Aug 2004 [email protected] wrote:

OK, forget getcwd(). What does lookup of .. do from that point? *Especially* for stuff you've got from regular files. That's the decision that needs to be made.

I think that will decide on whether we expose attributes through the normal namespace or not.

If we do expose them in the normal namespace, then ".." should work the way the namespace looks: if you do ".." on the "attribute directory" of a file, you get the directory that the file was in. Ie an old-style user-space "getcwd()" would give the right path (well, an old-style user-space getcwd() would probably refuse the file on the base that it is S_IFREG, but ignoring that..)

If we _don't_ expose it in the normal namespace, we should should either just error out (logically you'd get the file itself, but I really don't wan tto have ".." return a non-directory, because _that_ really might confuse things), or you'd just return the same directory (ie it would be a "local root" in the namespace you got moved to).

So let's try to be self-consistent with how we expose it in the normal namespace.

Linus

#789 lore raw
Quoted 7 lines · Linus Torvalds

If we do expose them in the normal namespace, then ".." should work the way the namespace looks: if you do ".." on the "attribute directory" of a file, you get the directory that the file was in. Ie an old-style user-space "getcwd()" would give the right path (well, an old-style user-space getcwd() would probably refuse the file on the base that it is S_IFREG, but ignoring that..)

Errr, why don't we check what Solaris does in these cases and see if that makes sense. They do seem to have made some progress along this path and it wouldn't make sense to create new semantics just for the sake of it.

Jeremy.

#790 lore raw
In reply to [email protected]
Quoted 2 lines · On Sat, 28 Aug 2004 [email protected] wrote:

OK, let me restate the question - what do we get from pwd if we do fchdir() to such beast?

On solaris: ENOTDIR from getcwd(), with 'pwd' executed inside a shell started with runat.

.. entry exists, but doesnt lead anywhere, cd .. returning ENOTDIR too.

regards,

Signature · 3 lines
Paul Jakma	[email protected]	[email protected]	Key ID: 64A2FF6A
Fortune:
If you didn't get caught, did you really do it?
#791 lore raw
In reply to Linus Torvalds

I object to openat().....

My reason is that the things that distinguish between objects should be the names, not the choice of system call. The reason for this is that it improves closure and namespace unification to do so, because it allows all the objects to be accessed within the same namespace.

Yes, it can be useful to allow a namespace to exclude some objects, but that exclusion should not be mandated. If you want to exclude, you should cd or chroot to /proc/nopseudos and find a view of the filesystem that excludes metas, or mount with -nopseuodos.

Do you see why I say this? I can say a lot more about the damage of fragmenting namespaces into multiple apis.... Why look at xattrs....;-)

Hans

Quoted 46 lines · Linus Torvalds

I'm pretty confident that we can extend the VFS layer to support named streams (see the technical discussion with Al, rather than the flames in this thread). I also clearly believe that it is worth it, but I'm starting to wonder if we should have a special open flag to make people select the stream.

If you look at the Solaris interface, the _nice_ part about "openat()" is that you can do something like

	file = open(filename, O_RDONLY);
	if (file < 0)
		return -ENOENT;
	icon = openat(file, "icon", O_RDONLY | O_XATTR);
	if (icon < 0)
		icon = default_icon_file;
	..

and it will work regardless of whether "filename" is a directory or a regular file, if I've understood correctly.

Now, I think that makes sense for several reasons:
- single case
- race-free (think "stat()" vs "fstat()" races).
- I think we want to do "openat()" regardless of whether we ever support extended attributes or not ("openat()" is nice for doing "namei()" in user space even in the absense of any attributes or named streams).

So what we can do is
- implement openat() regardless, and expect to do the Solaris thing for it if we ever do streams.
- _also_ support the "implied named attributes" for regular files, so that you don't have to use "openat()" to access them.

Comments? Does anybody hate "openat()" for any reason (regardless of attributes)? We can easily support it, we'd just need to pass in the file to use as part of the "nameidata" thing or add an argument (it would also possibly be cleaner if we made "fs->pwd" be a "struct file").

Linus
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#792 lore raw

The Necessity of File Types (was silent semantic changes with reiser4)

Quoted 2 lines · Linus Torvalds

I'm pretty confident that we can extend the VFS layer to support named streams [...]

Hans Reiser wrote on Sat, 28 Aug 2004 15:29:33 -0700:

Quoted 6 lines

I object to openat().....

My reason is that the things that distinguish between objects should be the names, not the choice of system call. The reason for this is that it improves closure and namespace unification to do so, because it allows all the objects to be accessed within the same namespace.

I have to agree with Hans there. Just have one main kind of file system object. No separate stream system, other than as an optional alternative way of viewing things. Treat it as a file or a directory or as an attribute as the context implies. Have just one open call to open it, another to enumerate its children, and another to read and write data to it. Paths to file system objects, no matter what their purpose, are all just plain paths (you can get fancy later with sub-name-spaces and name space boundaries).

But how do you tell what kind of thing it is? How do you tell if it is important enough to back up? How do you tell that a thing is intended to be treated as a directory containing other things rather than as a stand-alone object? File types are needed for that!

In BeOS we had a MIME type database and all important files were given an attribute of BEOS:TYPE which contained their MIME string. A system-wide database associated applications that handled each type, and also listed the standard attributes for each MIME type (such as Subject/From/To/... for "text/e-mail" files). It even included some extra info about which attributes should be shown to the user and which were editable by the user in the desktop directory window GUI. By the way, the database was just a tree of files (one for each MIME type using the same structure as MIME names) with attributes attached to them containing the relevant meta-meta-data.

Unclassified files (ones with no type, such as those generated by "cp" and other non-attribute aware programs) were treated as application/octet-stream. There was also a background task that would try to identify them when the computer was idle, or if you tried to open them in a file requestor. It used the file extension and also looked for characteristic signature bytes to pick the type.

Unfortunately BeOS had fragmented the type system by having a separate system of 4 character codes to identify the type of attributes (since attributes had their own API separate from files). That would tell you the attribute was an INT32 or C-String or Date.

I'd like to see one unified type system for the unified file/directory/attribute thing (has anyone named those multipurpose file system objects yet or can I call them Fildirutes?). That means a global list of types and an attribute attached to all files needing a type. That standard child file (used as an attribute) would be named something like "..meta.mimetype". The global list would contain information about whether that type of thing needed to be backed up, what its standard computed attributes would be, and all sorts of common properties that all files of that type need. Plus I'd add the primitive types, so "primitive/int32" and "primitive/date64" would be possible types for things which are simple attributes.

If you're worried about extra files, those file types could be stored as a unique ID (I recommend 8 character codes, using semi-readable text, for portability reasons) in each file inode. The type would only appear as a file to users of the file system.

Another need for the type system is the need to index attributes. Someone has to tell the OS that "MAIL:date-received" is a 64 bit number of microseconds since the start of time, so that the OS knows how to sort the index for that attribute.

Jan Harkes wrote on Sat, 28 Aug 2004 12:11:14 -0400:

Quoted 3 lines

From what I saw in one part of the discussion, it allows for infinite depth recursion (file/metas/metas/metas/...). Some applications are going to have a lot of fun with that.

Some of the type information can be faked out as appearing as files. So looking for the MIME type of a MIME type file (thing/..meta.mimetype/..meta.mimetype) would pretend to be a file that contained "text/mime-string", but not actually represent anything on disk. Instead stock answers for the really primitive data types would be generated by the file system, recursively if needed.

The recursion should be there. Just like it is for "." and "..", which "ls -R" for some reason doesn't traverse :-)

By the way, I disagree on Hans with using a directory for the meta data, I'd prefer to just have "..meta." as the name prefix of the metadata things. Otherwise there's an extra directory level in there, making ..meta/ different from other directories (making browsing it a bit awkward since it doesn't have metadata of its own - like window coordinates for displaying the directory).

- Alex

#793 lore raw

Re: The Necessity of File Types (was silent semantic changes with reiser4)

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander G. M. Smith wrote:
[...]
| By the way, I disagree on Hans with using a directory for the meta data, I'd prefer to just have "..meta." as the name prefix of the metadata things. Otherwise there's an extra directory level in there, making ..meta/ different from other directories (making browsing it a bit awkward since it doesn't have metadata of its own - like window coordinates for displaying the directory).

You'd rather do "cat ..meta.pseudo" than "ls ..metas/"? And we already have weird directories. /proc doesn't do things that /tmp does. I also doubt proc will ever store that kind of metadata.

I don't like either "metas" or "..metas", but you'll have to look in the archives for why. I am NOT starting that again.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTPJW3gHNmZLgCUhAQKuEw/+PjrDfSlZvuTUwiOSa/kU05qjTKkIp05F nBkrSIUKSUfoYN3gILucIGOjjJ5qBRW3wLIwWyO+raOAzHFTdylvR1rEWP9h/JUG Psz3X8RE8cubiuNwjyJawI87cPX8HW1xQxgkiiquZN2L5XbvHsFsbwAJV1tJLnrM 2lL6IBUXcxDIh7pdB0/z+4t945ElT3Rl09CLP/MfP4GNvt/UWjhbeHNWdeLq93C+ w0irn6VIsm+1+GKqEhpYLLMj9V4cbBY6m7XMuBzH+gEyu5exAMqGzPLGDiG/I6y/ rXI6IF94i2p4sLMv0LmtEvQsV37LFIKRBrWVScV5oiQ93LPVLG6pYmAi6RLsHN03 QazMmcRwjVvrTAf0KS21KXaE0qYbvgoQpteUEkPZLRkczSmiry2d3e6gBQlCV7xj 9Hzkm9HxR0BjxmRaMVLEWZ86t08OHkl0Ab186DiHUC5RlL2FPIvdyYthXo9wwtI/ XlA03Rvc3kl6DVRO8UM/+fa8v5MDn58ZMI2QgdPrU0V2Q9bGv/woByszXaDA+IVJ ZiHQ8IgShYlkA279DffOrZnDPnZMOWa7iRYetX5PYPoAyfkrmFKofT8va+dDL7pf HA5crUR5ws1EZZ8lUfrtXXnsBeEn9PS0lUHvJY7zZ4tJfZ65Fa+IiDpK5p+1tJZi ylCBNEidK/8=
=CiZ9
-----END PGP SIGNATURE-----

#794 lore raw
In reply to Hans Reiser
Quoted 2 lines · Hans Reiser

I object to openat().....

Sound slike you object to O_XATTRS, not openat() itself.

Realize that openat() works independently of any special streams, it's fundamentally a "look up name starting from this file" (rather than "starting from root" or "starting from cwd").

Linus

#795 lore raw
Quoted 7 lines · Linus Torvalds
Quoted 3 lines · Hans Reiser

I object to openat().....

Quoted 11 lines

Sound slike you object to O_XATTRS, not openat() itself.

Realize that openat() works independently of any special streams, it's fundamentally a "look up name starting from this file" (rather than "starting from root" or "starting from cwd").

Linus

well, isn't that namespace fragmentation by definition? If you can't go

cat filenameA/metas/permissions > filenameB/metas/permissions

find / -exec cat {}/permissions \; | grep 4777 | wc -l

then aren't you missing the whole point of this exercise which is to allow the whole OS to be better integrated into a more unified namespace so that data can leap from one tool to another, and one container to another, with the greatest of ease? If cat cannot access the metadata without changing the code of cat, then all the elegance goes poof.

It completely baffles me what disabling filenameA/.. does for us. Why add asymmetries? Ease of implementation is no excuse for such asymmetry.

Tomorrow I am going to send a little essay I wrote this evening on these metafiles.

#796 lore raw
Quoted 10 lines · Hans Reiser
Quoted 8 lines · Linus Torvalds
Quoted 1 line · Hans Reiser

I object to openat().....

Sound slike you object to O_XATTRS, not openat() itself.

Realize that openat() works independently of any special streams, it's fundamentally a "look up name starting from this file" (rather than "starting from root" or "starting from cwd").

Quoted 5 lines

well, isn't that namespace fragmentation by definition? If you can't go

cat filenameA/metas/permissions > filenameB/metas/permissions

find / -exec cat {}/permissions \; | grep 4777 | wc -l

My apologies if this question has been raised (and answered) elsewhere. I tried to keep up with the flamefest/discussion but it wasn't always possible.

How would you do it for directories without making certain names illegal? ie taking the above you'd do:

cat dirA/metas/permissions > dirB/metas/permissions

But how does that distinguish an access to the metas for dirA and dirB from an access to a file called permissions within the metas directories of dirA and dirB?

I think what Linus is trying to do, as far as naming, is create a consistant access scheme that works for all types of objects in the fs without making certain names illegal (and with plugins, those names are not even a fixed thing I believe).

Feel free to correct me if I'm wrong, ofcourse. :)

Signature · 1 line
    Red herrings strewn hither and yon.
#797 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"CaT" == CaT <[email protected]> writes:

CaT> My apologies if this question has been raised (and answered) CaT> elsewhere. I tried to keep up with the flamefest/discussion but it CaT> wasn't always possible.

CaT> How would you do it for directories without making certain names CaT> illegal? ie taking the above you'd do:

CaT> cat dirA/metas/permissions > dirB/metas/permissions

Yes. Directories are not allowed to contain objects with a filename "metas". The naming of "metas" has been discussed quite a bit on the Reiserfs list, and I don't think there was much consensus that came out of it. Currently for Reiser4, AFAIK, the "metas" name is a compile-time option that you can change by changing a #define, if you're worried about name clashes.

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#798 lore raw
Quoted 1 line · Hubert Chan

Reiserfs list, and I don't think there was much consensus that came out

It's like Gentoo users and patch sets ;)

Quoted 3 lines

of it. Currently for Reiser4, AFAIK, the "metas" name is a compile-time option that you can change by changing a #define, if you're worried about name clashes.

http://mjt.nysv.org/reiser/reiser4.metas.patch

Signature · 1 line
mjt
#799 lore raw
In reply to Hans Reiser
Quoted 6 lines · Hans Reiser
Quoted 4 lines

Realize that openat() works independently of any special streams, it's fundamentally a "look up name starting from this file" (rather than "starting from root" or "starting from cwd").

well, isn't that namespace fragmentation by definition?

No.

There's no difference between

	fd = open("/usr/bin/yes", O_RDWR);

and

	dirfd = open("/usr/bin", O_RDONLY | O_DIRECTORY);
	fd = openat(dirfd, "yes", O_RDWR);

apart from error checking and permissions..

No new namespace.

Only O_XATTR implies a "namespace change".

Linus

#800 lore raw
Quoted 14 lines · Linus Torvalds
Quoted 8 lines · Hans Reiser
Quoted 5 lines

Realize that openat() works independently of any special streams, it's fundamentally a "look up name starting from this file" (rather than "starting from root" or "starting from cwd").

well, isn't that namespace fragmentation by definition?

No.

Quoted 10 lines

There's no difference between

	fd = open("/usr/bin/yes", O_RDWR);

and

	dirfd = open("/usr/bin", O_RDONLY | O_DIRECTORY);
	fd = openat(dirfd, "yes", O_RDWR);

The difference is that cat uses open() not openat().

With your model, can I do:

cat filenameA/metas/permissions > filenameB/metas/permissions

find / -exec cat {}/permissions \; | grep 4777 | wc -l

If yes, then we are talking past each other somehow rather than disagreeing. If metafiles can be opened with both open and openat() in your model, then we are discussing some small detail.

I think the answer is no though, in which case you are missing the point of the new design. Is the answer no?

#801 lore raw
Quoted 4 lines · Hans Reiser

With your model, can I do:

cat filenameA/metas/permissions > filenameB/metas/permissions

runat filenameA "cat permissions" | runat filenameB "cat > permissions"

Quoted 1 line

find / -exec cat {}/permissions \; | grep 4777 | wc -l

find / -exec runat {} "cat permissions" \; | grep 4777 | wc -l

Although, whether or not the kernel exposes file permissions as an attribute named permissions is up to the filesystem. (And the wrong thing to do, in my opinion, but that's irrelevant.)

Quoted 3 lines

If yes, then we are talking past each other somehow rather than disagreeing. If metafiles can be opened with both open and openat() in your model, then we are discussing some small detail.

It can be opened by both, but in order to do it via open(2), you need to fchdir(2) to the attribute directory as returned by openat(2) and then use a relative pathname.

Quoted 2 lines

I think the answer is no though, in which case you are missing the point of the new design. Is the answer no?

Signature · 1 line
Nicholas Miell <[email protected]>
#802 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Nicholas" == Nicholas Miell <[email protected]> writes:

Quoted 4 lines · Hans Reiser
Quoted 4 lines

With your model, can I do:

cat filenameA/metas/permissions > filenameB/metas/permissions

Nicholas> runat filenameA "cat permissions" | runat filenameB "cat > permissions"

The world is more than the commandline. Say I have an application icon stored in xterm/icon.png. Can I just open xterm/icon.png in Gimp without having to open up a new instance of Gimp, and just use the old instance that I was using for other images? Or Can I use emacs to edit xterm/description.txt without having to open up a new instance of emacs? Will scripts be able to use metafiles and normal files at the same time?

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#803 lore raw
In reply to Hans Reiser
Quoted 2 lines · Hans Reiser

The difference is that cat uses open() not openat().

Hans. Step back a bit. Stop this idiocy about thinking that this is all about you, and all about your reiser4 "metas" etc.

This is about VFS interfaces, and about being generic and _useful_.

Quite frankly, there is absolutely _zero_ point to make attributes just because reiser4 can implement them. Zero. Nada. Zilch. It just won't matter, because people are NOT GOING TO SWITCH TO REISER4 IN MASS.

So in order to make this useful, it has to be:
- stable. This means that if Al Viro asks about locking and aliasing issues, you don't ignore it, you ask "how high?"
- portable. Preferably across operating systems, but at the very _least_ across filesystems. See the NFSv4 thing, for example.
- useful to the people who would actually use it. Right now that's samba, and not a whole lot else. For everybody else, the pain of not having _all_ major filesystems support it is probably so insurmountable that it's just not going to happen.

Do you get it? If you can't get your mind out of your little current reiser4 issues, don't even bother talking about it.

Look through the thread on "possible design issues for hybrids" for me actually talking with Al on how to handle problem #1. That's one important thing, and in fact, that one is important whether we expose it to a normal "open()" call or not, since all the same aliasing issues exist _regardless_ of interfaces.

"openat()" happens to be _one_ solution to #2 and #3. You haven't given any constructive input to _any_ of the problems, and you keep on raising totally irrelevant issues. Face the music, Hans. If you can't solve those three problems, whatever you do simply DOES NOT MATTER.

In particular, in the two last emails to you, I tried to show that "openat()" is _independent_ of streams/attributes. That it has semantics that have absolutely _nothing_ to do with metas. And in BOTH CASES you could not wrap your brain around that fact, and kept on harping about meta's.

There are bigger issues here. I'm perfectly happy to _also_ expose the streams as regular filenames, but I've tried to explain that that doesn't work for directories, for example, so YOU CANNOT -JUST- talk about the filename-only thing. You do need to have a different namespace. Full stop. What's so hard to understand about that?

So stop ignoring the issues. Stop ignoring the fact that not everything revolves around how you happened to implement something. Get with the program.

Linus

#804 lore raw
In reply to Linus Torvalds
Quoted 7 lines · Linus Torvalds
Quoted 2 lines · Hans Reiser

I object to openat().....

Sound slike you object to O_XATTRS, not openat() itself.

xattr deserve to be objected.
quickiehackie design practice.

Signature · 2 lines
"the liberation loophole will make it clear.."
lex lyamin
#805 lore raw
In reply to Linus Torvalds
Quoted 11 lines · Linus Torvalds
Quoted 2 lines · Hans Reiser

I object to openat().....

Sound slike you object to O_XATTRS, not openat() itself.

Realize that openat() works independently of any special streams, it's fundamentally a "look up name starting from this file" (rather than "starting from root" or "starting from cwd").

Linus, openat() is trivial to implement in userland *IF* we are talking about stuff already in namespace. Proof:

int openat(int fd, char *path, int flags, mode_t mode)

{
	int len = snprintf(NULL, 0, "/proc/self/%d/%s", fd, path);
	char name[len + 1];
	sprintf(name, "/proc/self/%d/%s", fd, path);
	return open(name, flags, mode);
}

7 lines. Replace VLA with alloca() or malloc() in pre-C99 environment.

In other words, there's nothing magical about syscall itself - compatibility with Solaris is not an issue here. And basic functionality (i.e. what you've described above) can be trivially achieved with no extra primitives at all.

Now, if you want to use that puppy as explicit indicator of magic steps taken, fine, but that again says nothing about the primitives we want to express the magic steps in question - reproducing openat() as one of the libc functions won't be hard anyway.

So I have to agree with Hans on that one - I believe that as a fundamental primitive it's wrong choice. Convenient to have in libc - sure, why not?

#806 lore raw
In reply to Hans Reiser

Hans> My reason is that the things that distinguish between objects Hans> should be the names, not the choice of system call. The reason Hans> for this is that it improves closure and namespace unification Hans> to do so, because it allows all the objects to be accessed Hans> within the same namespace.

So who decides what the names mean in a specific context? How do we get consistency across applications? In the Apple and MicroSoft cases, it's a single company which sets the standards for the names of the streams and what they mean. I have no knowledge of how Sun is doing this. I don't see us setting up a single naming authority and enforcement mechanism.

I went and read the Pike paper on namespaces which Hans quotes much earlier in this thread, and I think it isn't applicable to this thread. In the paper Pike compares the MVS, VMS, DOS and Unix name spaces for disk layouts. It's an interesting paper, though a bit dated since it's 20 years old. The concepts are good, but the examples aren't any more. How many Unix systems today have /n/faces/... to link email addresses to digitized images of people faces?

The big point of this paper though was to make the syntax simple, and the semantics clear and unambiguous. For example, they point out how the "Cedar" file system added in VMS style version numbers to filenames with the syntax "<name>!<version>" and they pointedly ask what does "/usr/rob!3/bin/cat-v mean? Which I think just points out that the syntax of streams (metas) needs be carefully thought out, and how that syntax implies various semantics.

Lots of people keep asking for ACLs, but time and time again people point out that while the Unix permission model isn't as flexible, it is _simpler_ and easier to keep track of.

As an example of complexity, just keeping track of our /etc/sudoers file with all the users, hosts and commands that can be defined is a huge hassle. You pretty much need to have a 'lint' style program to do the consistency checking.

Hans> Yes, it can be useful to allow a namespace to exclude some Hans> objects, but that exclusion should not be mandated.

Conversely, should a namespace *require* that objects always be there? What happens when I mount an NFS filesystem onto a Resierfs4 tree? How does the user/apps determine when the semantics of the underlying namespace have changed?

Hans> If you want to exclude, you should cd or chroot to Hans> /proc/nopseudos and find a view of the filesystem that excludes Hans> metas, or mount with -nopseuodos.

Hans> Do you see why I say this? I can say a lot more about the Hans> damage of fragmenting namespaces into multiple apis.... Why look Hans> at xattrs....;-)

I think one of the advantages of the xattrs is that the core namespace doesn't change, nor do the semantics. They are layered on top and can be explicitly looked at.

A real life problem of namespaces is Network Appliance Filers. If you have a filer with both NFS and CIFS exports, it will nicely manage the CIFS ACLs in harmony (for some level you determine) with the NFS and Unix permissions. But to backup that data, and to store that information, you need to use either their 'dump' tool or NDMP. If you NFS mount the filesystem to your backup host and then use a simple tar to back it up, you lose the information since that window you use to look into the namespace is restricted.

It's a gotcha. Until we can (or maybe computing in general) come up with a new namespace paradym which we can all agree to, this is going to be a big issue.

John

#807 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"John" == John Stoffel <[email protected]> writes:

[...]

John> I went and read the Pike paper on namespaces which Hans quotes John> much earlier in this thread, and I think it isn't applicable to John> this thread. In the paper Pike compares the MVS, VMS, DOS and John> Unix name spaces for disk layouts. It's an interesting paper, John> though a bit dated since it's 20 years old. The concepts are John> good, but the examples aren't any more. How many Unix systems John> today have /n/faces/... to link email addresses to digitized John> images of people faces?

IIRC, gdm uses some directory in /usr/share to match usernames to pictures (along with ~<username>/<somefile>). But I can't find the directory nor remember exactly what the name is.

Then there's the gconf database, which uses the filesystem to store its hierarchical structure. Or the .cddb/ database that uses filenames to match the discid to the data. Or on Debian systems, there's /var/lib/dpkg/info database that attaches (post|pre)(inst|rm) scripts, file lists, etc. to packages. (Although I would have organized that directory differently.)

And then there's the fact that there would be more examples if filesystems were more expressive. Like my LDAP database that sometimes gets corrupted when the power gets pulled. It's stored in a number of opaque files, so I don't know how it gets corrupted, and I can't use the normal fsck to check it. Wouldn't it be nice if the database could be stored as normal files, using the filesystem hierarchy to mimic the LDAP hierarchy? Then if something gets corrupted, fsck will fix it, or I can easily see which record contains the garbage. The problem is that the filesystem can't do queries, so I'll have to wait until Reiser6.

John> The big point of this paper though was to make the syntax simple, John> and the semantics clear and unambiguous. For example, they point John> out how the "Cedar" file system added in VMS style version numbers John> to filenames with the syntax "<name>!<version>" and they pointedly John> ask what does "/usr/rob!3/bin/cat-v mean? Which I think just John> points out that the syntax of streams (metas) needs be carefully John> thought out, and how that syntax implies various semantics.

The problem isn't just with things being confusing. It's also about making things accessible through a unified interface. If we take the attributes example, if we need to use a special xattr syscall, then I can't use GIMP to edit icon attribute of xterm, or use emacs/vim/nano/ed to edit the comment attribute of sound_of_silence.ogg (or the lyrics attribute of sound_of_silence.ogg). I need to fire up some stupid program to do batch edits of the attributes of my ogg collection because the FreeDB database had the wrong CD title info for the CD that I just ripped and I didn't catch the error before I ripped it. If attributes were exposed through the regular filesystem interface (possibly through some userspace helper to do the actual OGG parsing), I could just type a couple of lines on the command line, and be done with it.

John> Lots of people keep asking for ACLs, but time and time again John> people point out that while the Unix permission model isn't as John> flexible, it is _simpler_ and easier to keep track of.

Not if you need to add a whole bunch of groups just to do the things you want to do. I personally haven't done that. Why? Because I would rather deal with suboptimal permissions than having to add all those groups.

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#808 lore raw
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

"Hubert" == Hubert Chan <[email protected]> writes:
"John" == John Stoffel <[email protected]> writes:

[...]

John> The big point of this paper though was to make the syntax simple, John> and the semantics clear and unambiguous. For example, they point John> out how the "Cedar" file system added in VMS style version numbers John> to filenames with the syntax "<name>!<version>" and they pointedly John> ask what does "/usr/rob!3/bin/cat-v mean? Which I think just John> points out that the syntax of streams (metas) needs be carefully John> thought out, and how that syntax implies various semantics.

Hubert> The problem isn't just with things being confusing. It's also Hubert> about making things accessible through a unified interface. ...

By the way, "The Hideous Name" mentions this just briefly (in the "Properties of Name Spaces" section). A better paper, IMHO, about this is "The Use of Name Spaces in Plan 9" (Rob Pike, Dave Presotto, Ken Thompson, Howard Trickey, Phil Winterbottom; Operating Systems Review, Num. 2, Vol 27, April 1993 pp. 72-76)

http://www.cs.bell-labs.com/sys/doc/names.pdf
or
http://www.cs.bell-labs.com/sys/doc/names.html

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#809 lore raw
In reply to Linus Torvalds
Quoted 23 lines · Linus Torvalds

If you look at the Solaris interface, the _nice_ part about "openat()" is that you can do something like

	file = open(filename, O_RDONLY);
	if (file < 0)
		return -ENOENT;
	icon = openat(file, "icon", O_RDONLY | O_XATTR);
	if (icon < 0)
		icon = default_icon_file;
	..

and it will work regardless of whether "filename" is a directory or a regular file, if I've understood correctly.

Now, I think that makes sense for several reasons:
- single case
- race-free (think "stat()" vs "fstat()" races).
- I think we want to do "openat()" regardless of whether we ever support extended attributes or not ("openat()" is nice for doing "namei()" in user space even in the absense of any attributes or named streams).

Quoted 3 lines

So what we can do is
- implement openat() regardless, and expect to do the Solaris thing for it if we ever do streams.

Hurrah ! Definately the best way to go as far as Samba is concerned. Makes our portability easier. Now the NT interface allows an open on a named data stream without an open on the containing file (ie. you can do CreateFile("filename::STREAMNAME",....) without doing CreateFile("filename",..." first, but I personally think those semantics are a nasty hack. It makes much more sense (IMHO) to ensure that you already have an open fd on a file before allowing access to the named streams inside, Windows be damned. If we have some semantic issues mapping from the Windows semantics to Solaris/Linux semantics then so be it, it's a Samba problem so to speak :-).

Thanks,

Jeremy.

#810 lore raw
In reply to Linus Torvalds
Quoted 26 lines · Linus Torvalds
Quoted 9 lines · Helge Hafting
Quoted 5 lines

I think that lack of distinguishing power is more serious for directories. The more I think I think about it, the more I wonder whether Solaris did things right - having a special operation to "cross the boundary".

I suspect Solaris did it that way because it's a hell of a lot easier to do it like that, but regardless, it would solve the issue of real directories having both real children _and_ the "extra streams".

There are many ways of doing this. Several extra streams to a directory that aren't ordinary files in the directory?

Well.. Yes. We already have "." and "..", which are "special extra
streams" in a sense. However, people expect them, and know to ignore them. The same wouldn't be true of new naming.

Quoted 7 lines

It seems to me that we can get a lot of nice functionality in a simpler way:
Instead of thinking about a number of streams attached to something that is either an ordinary file or directory, just say that the only change will be that a directory may have a _single_ file stream in addition to being a plain directory.

Quoted 3 lines

That doesn't really help us. What would the name be, and how could you avoid clashes?

The name for the file stream and the directory would be the same, distinguished by how they're used. I.e. fopen("filedirname", "rw") gets the stream, while chdir("filedirname") changes into the directory. fopen("filedirname/substream", "rw"); opens some stream inside the directory - as usual.

My idea is that the "extra streams" aren't special at all, they are simply files in a directory and will therefore work with any normal tool. cd'ing into such a directory won't be special either - because it is a plain directory. A collection of named streams is usually a bunch of files in a directory - nothing new there.

Someone serving files for clients using
other os'es may attach special meaning to them, but I don't see the need for a special meaning under linux.

So the only new thing is that the directory have a stream of its own. Then there must be a way to turn a ordinary file into such a directory - people may then implement thumbnails, summaries and so on as substreams. Where the substreams really are plain files in this slightly special directory.

Perhaps mkdir with an existing filename could succeed, creating a mixed file/directory, if the underlying fs support it.

Quoted 23 lines
Quoted 6 lines

If the VFS is to be extended in order to support file-as-directory (or vice versa) then hopefully it can be done in a simple way.

I'm pretty confident that we can extend the VFS layer to support named streams (see the technical discussion with Al, rather than the flames in this thread). I also clearly believe that it is worth it, but I'm starting to wonder if we should have a special open flag to make people select the stream.

If you look at the Solaris interface, the _nice_ part about "openat()" is that you can do something like

	file = open(filename, O_RDONLY);
	if (file < 0)
		return -ENOENT;
	icon = openat(file, "icon", O_RDONLY | O_XATTR);
	if (icon < 0)
		icon = default_icon_file;
	..
Quoted 3 lines

and it will work regardless of whether "filename" is a directory or a regular file, if I've understood correctly.

An alternative: support open("filename/icon", O_CREAT);
and have this turn "filename" into file-as-directory if it wasn't so already. Then it works both with files and directories.
A new flag might be needed if old programs somehow depend on the stuff above failing for plain files.

Quoted 8 lines

Now, I think that makes sense for several reasons:
- single case
- race-free (think "stat()" vs "fstat()" races).
- I think we want to do "openat()" regardless of whether we ever support extended attributes or not ("openat()" is nice for doing "namei()" in user space even in the absense of any attributes or named streams).

Quoted 10 lines

So what we can do is
- implement openat() regardless, and expect to do the Solaris thing for it if we ever do streams.
- _also_ support the "implied named attributes" for regular files, so that you don't have to use "openat()" to access them.

Comments? Does anybody hate "openat()" for any reason (regardless of attributes)? We can easily support it, we'd just need to pass in the file to use as part of the "nameidata" thing or add an argument (it would also possibly be cleaner if we made "fs->pwd" be a "struct file").

No problem with openat()

Helge Hafting

#811 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Helge" == Helge Hafting <[email protected]> writes:

Helge> Linus Torvalds wrote:

[...]

Quoted 2 lines
Quoted 2 lines

That doesn't really help us. What would the name be, and how could you avoid clashes?

Helge> The name for the file stream and the directory would be the same, Helge> distinguished by how they're used. I.e. fopen("filedirname", Helge> "rw") gets the stream, while chdir("filedirname") changes into Helge> the directory. fopen("filedirname/substream", "rw"); opens some Helge> stream inside the directory - as usual.

The problem is that filedirname now cannot contain a file called "substream", or else it will clash with the name of the substream.

One solution is to put all streams into another subdirectory, much like "metas" in Reiser4. So to access a stream named "substream", you open filedirname/streams/substream (replace "streams" with some carefully chosen name that is user-friendly and highly unlikely to produce clashes, such as "..streams", following the original "metas" proposal which has been discussed to death on the ReiserFS list. Or "pseudos/streams", or one of countless variations).

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#812 lore raw
In reply to Spam
Quoted 8 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Spam" == Spam <[email protected]> writes:

Quoted 4 lines · Rik van Riel
Quoted 3 lines · Hans Reiser

Why are you guys even considering going to any pain at all to distort semantics for the sake of backup? tar is easy, we'll fix it and send in a patch.

Quoted 6 lines
Quoted 6 lines

It's not as easy as you make it out, and not just because there are a few dozen backup programs that need fixing.

The problem is more fundamental than that. Some of the file streams proposed need to be backed up, while others are alternative presentations of the file, which should not be backed up.

Spam> No, not really. This is a user decision and should be options in
Spam> the backup software. I don't think it is up to the kernel, Spam> filesystem, or the OS in general to decide what information the Spam> user want to retain or not.

Why not just define an attribute named something like "do-not-backup"? Then whatever program that generates the thumbnail can automatically add the do-not-backup bit, and the backup software knows to ignore it. (Obviously, that bit should apply recursively down the subtree.)

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.