Skip to contentSkip to conversation

silent semantic changes in reiser4 (brief attempt to document the idea of what reiser4 wants to do with metafiles and why

DISCUSSION Hans Reiser 28 participants 60 messages Aug 29, 2004 – Sep 14, 2004 1 message not in the archive VFS

lore
Share & tools
lore
#0 lore raw

The Idea

You should be able to access metadata about a file the same way you access the file's data, but with a name based on the filename followed by a name to select the metadata of interest.

Examples:

cat song_of_silence/metas/owner
cat song_of_silence/metas/permissions
cat 10 > song_of_silence/metas/mixer_defaults/volume cat song_of_silence/metas/license

But wait! This idea has flaws. Let us list them and address them:

Problem: Often we want to access more than one metafile at a time, and a separate system call for each of them is less efficient than stat or xattrs.

Solution: Create a system call that can access more than one file at a time, whether it is a metafile or a data file should be irrelevant.

Problem: Many kinds of metadata need to be updated atomically.

Solution: Atomic filesystem, which allows all files, including data files, to be updated atomically.

Problem: Many kinds of metadata need to be constrained in their allowed values.

Solution: Allow general purpose constraints to constrain all files, including data files also.

Problem: Many kinds of metadata should not be visible to the user unless he wants to see them.

Solution: Allow all files to be hidden from the user unless he wants to see them.

Example: metas subdirectory is invisible to readdir

Problem: this approach can pollute the namespace

Answer: choose obscure names

Problem (all credit to Mr. Demidov for identifying this problem, I argued the other viewpoint, and I can only claim the wisdom to know that I lost the argument): names like "..metas" are ugly to new users, who don't really care for languages that use punctuation in their keywords.

Answer

don't make them too obscure, experienced namespace developers know that the problem of polluting the namespace is not really as big a deal as beginners think it is, and Clearcase and the WAFL filesystem manage to get by just fine, whereas the problem of putting punctuation marks into names and syntax is a big deal for newbies to the system. Name it "metas" not "..metas", and users will never experience it as a real problem, and newbies will never be annoyed by a-rhythmic punctuation. Note: if Linus disagrees, it is not the most important thing in this design, "..metas" isn't the end of the world.

Problem: Many kinds of metadata, such as acls, need to be inherited.
Solution: Allow all files, metafiles and data files both, to be inherited from other files.

The Idea 2:

When implementing a file plugin, very often there are methods supplied by that plugin that are unique to that plugin. These methods are often well interfaced with as files that can be written to or read from. Allow file plugins to define pseudo files that perform these methods.

Examples:

cat complex_document/pseudos/glued gives an archive of the directory "complex_document" and everything below it, suitable for emailing.

cat filename/pseudos/backup gives a set of commands that a backup command can use to create the file "filename" with all of its state preserved, without the backup program having to understand the plugin's internals. (If there is going to be a proliferation of file plugins, then backup plugin methods are a necessity.

cat password_directory/pseudos/cat gives a concatenation of all the files in the directory "password_directory".

Why Is This Design Important?

It is all about namespace fragmentation. If you can access everything in the OS from the same namespace, the OS is much more powerful.

The expressive power of an OS is not proportional to how many objects are in the OS, but rather to how many different ways you can combine them. Separate namespaces for different types of objects is like having different railroad track gauges for the different peoples of the world.

If there are different namespaces for 2 sets of objects, then the number of possible combinations of the objects is (N/2)^2, which is less than the number of combinations of the objects if they are in one namespace, which is N^2.

Why Openat Sucks:

because you can't go cat filenameA/metas/permissions > filenameB/permissions

If cat doesn't work, then we are suffering exactly the problem with namespace fragmentation that this whole scheme was invented to avoid.

The Origins Of This Idea

I disliked stat() way back in 1984. This is just one small piece of my efforts to build tools to allow OS architects to unify namespaces throughout the OS. DARPA funded this under the CHATS program, and it was one of the main components of my proposal.

Schedule:

One piece at a time, I would like to let this mature as a reiser4 only prototype throughout the 2.7 kernel series, and then have it reviewed for use by other filesystems after that. Let us not rush it, please. I have been studying namespace design since 1984, let someone who is deeply familiar with namespace design principles have a try at a coherent design with all the pieces in place first before committee redesign is attempted. Please consider that there are advantages to letting the guy who had the idea and did the work to make it happen be the maintainer.

My approach is to first produce a working prototype, and only then to try to make a standard. It is not an unusual approach in our community. Let me take that path, it works better for persons like me. I have a half working prototype. A little more time please. After 30 years of Unix file system semantics stagnation, there is not a great rush for this little piece of mild innovation I think.

Conclusion:

Namespace fragmentation is the most important single determinant of OS expressive power, just as trade fragmentation is the most important single determinant of national wealth (see Adam Smith's The Wealth of Nations). Making objects combine easily is what it is all about.

#1 lore raw

Hans Reiser wrote on Sun, 29 Aug 2004 13:21:44 -0700:

Quoted 1 line

The Idea [...] Conclusion

Good summary!

Quoted 3 lines

The Idea 2: [...]
cat filename/pseudos/backup gives a set of commands that a backup command can use to create the file "filename" [...]

Are you sneaking in file types there? Just how does a file know which plugins it supports?

- Alex

#2 lore raw
Quoted 17 lines · Alexander G. M. Smith

Hans Reiser wrote on Sun, 29 Aug 2004 13:21:44 -0700:

Quoted 3 lines

The Idea [...] Conclusion

Good summary!

Quoted 5 lines

The Idea 2: [...]
cat filename/pseudos/backup gives a set of commands that a backup command can use to create the file "filename" [...]

Quoted 8 lines

Are you sneaking in file types there? Just how does a file know which plugins it supports?

- Alex

we have plugins with pluginids, is that what you mean by file type? I think they are a bit different from file types.

#7 lore raw
In reply to Hans Reiser

Hi!

Quoted 10 lines

Answer: choose obscure names

Problem (all credit to Mr. Demidov for identifying this problem, I argued the other viewpoint, and I can only claim the wisdom to know that I lost the argument): names like "..metas" are ugly to new users, who don't really care for languages that use punctuation in their keywords.

Answer

Quoted 9 lines

don't make them too obscure, experienced namespace developers know that the problem of polluting the namespace is not really as big a deal as beginners think it is, and Clearcase and the WAFL filesystem manage to get by just fine, whereas the problem of putting punctuation marks into names and syntax is a big deal for newbies to the system. Name it "metas" not "..metas", and users will never experience it as a real problem, and newbies will never be annoyed by a-rhythmic punctuation. Note: if Linus disagrees, it is not the most important thing in this design, "..metas" isn't the end of the world.

What about choosing just "..." instead of "metas"? "metas" is string that needs translation etc, while "..." is nicely neutral.

cat /sound_of_silence.mp3/.../author

does not look bad, either...
Pavel

Signature · 2 lines
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
#44 lore raw
In reply to Hans Reiser

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 9 lines · Hans Reiser

Why Openat Sucks:

because you can't go cat filenameA/metas/permissions > filenameB/permissions

If cat doesn't work, then we are suffering exactly the problem with namespace fragmentation that this whole scheme was invented to avoid.

Sure you can. All you need to do is fix your shell.

- --
Mike Waychison
Sun Microsystems, Inc.
1 (650) 352-5299 voice
1 (416) 202-8336 voice
http://www.sun.com

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE: The opinions expressed in this email are held by me, and may not represent the views of Sun Microsystems, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFBNIavdQs4kOxk3/MRAmvAAJ9sZfEgIOfINZsT23pa+Y3nQ4lfMACgmN3q sdgPzBUg9i/c+roWDrXq5BI=
=poSJ
-----END PGP SIGNATURE-----

#46 lore raw
In reply to Hans Reiser
Quoted 6 lines · Hans Reiser

The Idea

You should be able to access metadata about a file the same way you access the file's data, but with a name based on the filename followed by a name to select the metadata of interest.

Quoted 6 lines

Examples:

cat song_of_silence/metas/owner
cat song_of_silence/metas/permissions
cat 10 > song_of_silence/metas/mixer_defaults/volume cat song_of_silence/metas/license

Maybe I'm crazy but:

You're talking about a major change in the way filesystems work if this is
going to be used by other FSs too. If I understand this correctly it is a completely new thing and trying to do it by patching existing well-known 'primitives' may be wrong.

AFAIK and AFAICS the metadata are not files or directories. You can look at
them as files/dirs but they are not, just like a tar is not a directory. I believe that the correct thing to do (tm) is to add a new 'concept' named 'metadata' (which already exists). This way you'll have files, directories and metadata (or whatever you call them). So, each directory can have metadatas and files and each file can have metadatas. Then you have to provide some new methods of accessing them and not to use chdir() etc. (lets say chdir_meta() to enter the meta dir which will work for files too). After entering the 'metadir' you'll be able to use existing methods etc to access its 'files'.

This approach doesn't mess with existing things and can be extended for
other filesystems too.

(Just a thought)

<<V13>>

#47 lore raw
Quoted 35 lines · Hans Reiser
Quoted 6 lines

The Idea

You should be able to access metadata about a file the same way you access the file's data, but with a name based on the filename followed by a name to select the metadata of interest.

Quoted 6 lines

Examples:

cat song_of_silence/metas/owner
cat song_of_silence/metas/permissions
cat 10 > song_of_silence/metas/mixer_defaults/volume cat song_of_silence/metas/license

Maybe I'm crazy but:

You're talking about a major change in the way filesystems work if this is
going to be used by other FSs too. If I understand this correctly it is a completely new thing and trying to do it by patching existing well-known 'primitives' may be wrong.

AFAIK and AFAICS the metadata are not files or directories. You can look at
them as files/dirs but they are not, just like a tar is not a directory. I believe that the correct thing to do (tm) is to add a new 'concept' named 'metadata' (which already exists). This way you'll have files, directories and metadata (or whatever you call them). So, each directory can have metadatas and files and each file can have metadatas. Then you have to provide some new methods of accessing them and not to use chdir() etc. (lets say chdir_meta() to enter the meta dir which will work for files too). After entering the 'metadir' you'll be able to use existing methods etc to access its 'files'.

Quoted 4 lines

This approach doesn't mess with existing things and can be extended for
other filesystems too.

(Just a thought)

It is a good thought. However I think they are trying to figure out a way to have the metadata and streams to be accesible with legacy applications.

The file-as-directory concept is one way, which still seem to have issues.

How are things done on Windows platforms when there are files and directories with the same name? In Unix that is imposible. How does it work for environments like Cygwin etc? What happen to tools that run in them?

Quoted 1 line

<<V13>>

#45 lore raw
In reply to Mike Waychison
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Mike" == Mike Waychison <[email protected]> writes:

Quoted 8 lines · Mike> Hans Reiser wrote:
Quoted 1 line

Why Openat Sucks:

Quoted 6 lines

because you can't go cat filenameA/metas/permissions > filenameB/permissions

If cat doesn't work, then we are suffering exactly the problem with namespace fragmentation that this whole scheme was invented to avoid.

Mike> Sure you can. All you need to do is fix your shell.

And you'll have to fix every other program that you want to use to edit metadata/attributes/streams.

The whole point of avoiding namespace fragmentation is to avoid having to fix programs.

Like I said in another post, the world is bigger than just the command line. The command line just gives really easy examples of why Hans (and others) thinks about things the way he does.

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#58 lore raw
In reply to V13
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"V13" == V13 <[email protected]> writes:

V13> Maybe I'm crazy but:

V13> You're talking about a major change in the way filesystems work if V13> this is going to be used by other FSs too. If I understand this V13> correctly it is a completely new thing and trying to do it by V13> patching existing well-known 'primitives' may be wrong.

V13> AFAIK and AFAICS the metadata are not files or directories. You
V13> can look at them as files/dirs but they are not, just like a tar is V13> not a directory. I believe that the correct thing to do (tm) is to V13> add a new 'concept' named 'metadata' (which already exists). ...

And you'll need a new 'concept' named 'streams'. And a new concept for every extension you want/need to add to the filesystem. By exposing everything through the normal filesystem interface, you don't need to make any modifications to your programs to use the new concepts. Everything just works as expected. (With the possible exception of backup-like software, but it is easier to fix just the backup software than every other program that's out there.)

The hypothetical object /usr/bin/xterm/icon.png looks like a file, smells like a file, and quacks like a file. Why shouldn't it be a file?

V13> This way you'll have files, directories and metadata (or whatever V13> you call them). So, each directory can have metadatas and files and V13> each file can have metadatas. Then you have to provide some new V13> methods of accessing them and not to use chdir() etc. (lets say V13> chdir_meta() to enter the meta dir which will work for files V13> too). After entering the 'metadir' you'll be able to use existing V13> methods etc to access its 'files'.

V13> This approach doesn't mess with existing things and can be
V13> extended for other filesystems too.

Either way, filesystems that want to implement that interface will need to be changed. The question is just what kind of interface we will expose to the users/programs. And since that's what the question is, it seems to me that we want to expose it through an interface that the users are already used to.

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#48 lore raw
In reply to Spam

Salut,

Quoted 4 lines · Spam

How are things done on Windows platforms when there are files and directories with the same name? In Unix that is imposible. How does it work for environments like Cygwin etc? What happen to tools that run in them?

In NTFS it's illegal IIRC. At least the fs correction utilities complain about a block being assigned to two files.

Same with HFS+.

Sometimes there seem to be several things with the same name. But that's because of hidden extensions (.lnk for example).

I'm talking out of the book here, maybe the real-world implementations of Windows are different. I can't tell, I only used Windows once to ssh into a screwed-up router.

Tonnerre

#52 lore raw
In reply to Spam
Quoted 32 lines · Spam
Quoted 32 lines · Hans Reiser
Quoted 6 lines

The Idea

You should be able to access metadata about a file the same way you access the file's data, but with a name based on the filename followed by a name to select the metadata of interest.

Quoted 6 lines

Examples:

cat song_of_silence/metas/owner
cat song_of_silence/metas/permissions
cat 10 > song_of_silence/metas/mixer_defaults/volume cat song_of_silence/metas/license

Maybe I'm crazy but:

You're talking about a major change in the way filesystems work if this
is going to be used by other FSs too. If I understand this correctly it is a completely new thing and trying to do it by patching existing well-known 'primitives' may be wrong.

AFAIK and AFAICS the metadata are not files or directories. You can
look at them as files/dirs but they are not, just like a tar is not a directory. I believe that the correct thing to do (tm) is to add a new 'concept' named 'metadata' (which already exists). This way you'll have files, directories and metadata (or whatever you call them). So, each directory can have metadatas and files and each file can have metadatas. Then you have to provide some new methods of accessing them and not to use chdir() etc. (lets say chdir_meta() to enter the meta dir which will work for files too). After entering the 'metadir' you'll be able to use existing methods etc to access its 'files'.

Quoted 8 lines
Quoted 4 lines

This approach doesn't mess with existing things and can be extended for
other filesystems too.

(Just a thought)

It is a good thought. However I think they are trying to figure out a way to have the metadata and streams to be accesible with legacy applications.

They will be since after chdir_meta() the user will be able to look at the metadata just like Hans described it. The only thing that changes (from the userland POV) is the way someone can enter the 'metadata directory'. This way you don't have to have a special name, just a special function and no existing application (like tar) can possibly break because it will not know how to enter this 'metadata directory'.

<<V13>>

#49 lore raw
In reply to Tonnerre
Quoted 5 lines

Salut,

Quoted 8 lines
Quoted 4 lines · Spam

How are things done on Windows platforms when there are files and directories with the same name? In Unix that is imposible. How does it work for environments like Cygwin etc? What happen to tools that run in them?

In NTFS it's illegal IIRC. At least the fs correction utilities complain about a block being assigned to two files.

I meant a file and a directory with the same name, not two files with the same name :) subtle but important difference.

ie, you can have a file named "foo" and a directory named "foo" and they won't collide.

Quoted 10 lines

Same with HFS+.

Sometimes there seem to be several things with the same name. But that's because of hidden extensions (.lnk for example).

I'm talking out of the book here, maybe the real-world implementations of Windows are different. I can't tell, I only used Windows once to ssh into a screwed-up router.

Tonnerre

#59 lore raw
In reply to V13

On Tue, 2004-08-31 at 18:55, V13 wrote:
...

Quoted 10 lines

AFAIK and AFAICS the metadata are not files or directories. You can look at
them as files/dirs but they are not, just like a tar is not a directory. I believe that the correct thing to do (tm) is to add a new 'concept' named 'metadata' (which already exists). This way you'll have files, directories and metadata (or whatever you call them). So, each directory can have metadatas and files and each file can have metadatas. Then you have to provide some new methods of accessing them and not to use chdir() etc. (lets say chdir_meta() to enter the meta dir which will work for files too). After entering the 'metadir' you'll be able to use existing methods etc to access its 'files'.

I think this is what Hans is trying to avoid -- it results in creating a separate namespace for the metadata.

I would agree with the notion that metadata nodes are light-weight files in that they themselves have no metadata associated with them, but I don't see why they need separate system-calls (if we can help it).

Chris Dawes.

Quoted 7 lines

This approach doesn't mess with existing things and can be extended for
other filesystems too.

(Just a thought)

<<V13>>

#53 lore raw
In reply to V13
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"V13" == V13 <[email protected]> writes:

Quoted 3 lines · Spam
Quoted 3 lines

It is a good thought. However I think they are trying to figure out a way to have the metadata and streams to be accesible with legacy applications.

V13> They will be since after chdir_meta() the user will be able to look V13> at the metadata just like Hans described it.

Uh, no. Legacy means no changes made. It shouldn't need to be modified to use chdir_meta.

From what I can see, your proposal isn't much different from the openat proposal.

V13> The only thing that changes (from the userland POV) is the way V13> someone can enter the 'metadata directory'. This way you don't have V13> to have a special name, just a special function and no existing V13> application (like tar) can possibly break because it will not know V13> how to enter this 'metadata directory'.

tar won't be able to backup the metadata. That's the major breakage of tar that we're worried about.

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#54 lore raw

V13>> The only thing that changes (from the userland POV) is the way V13>> someone can enter the 'metadata directory'. This way you don't have V13>> to have a special name, just a special function and no existing V13>> application (like tar) can possibly break because it will not know V13>> how to enter this 'metadata directory'.

Quoted 2 lines

tar won't be able to backup the metadata. That's the major breakage of tar that we're worried about.

However, if we do a "cp fileA fileB" then the metadata and streams ought to be copied too, even if "cp" does not support them. This is the real challenge. Backup tools like tar can be patched just like it has so many times before.

#55 lore raw
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Spam" == Spam <[email protected]> writes:

V13> The only thing that changes (from the userland POV) is the way V13> someone can enter the 'metadata directory'. This way you don't have V13> to have a special name, just a special function and no existing V13> application (like tar) can possibly break because it will not know V13> how to enter this 'metadata directory'.

Quoted 2 lines
Quoted 2 lines

tar won't be able to backup the metadata. That's the major breakage of tar that we're worried about.

Spam> However, if we do a "cp fileA fileB" then the metadata and
Spam> streams ought to be copied too, even if "cp" does not support Spam> them.

Huh? How do you plan on pulling that off? Unless cp uses the not-yet-existing copy syscall, if cp can't see the metadata or streams, how is it going to copy it?

Spam> This is the real challenge. Backup tools like tar can be patched Spam> just like it has so many times before.

Yes. And if we can get file-as-dir, then we only need to patch tar once, since everything can be exported through that interface.

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#56 lore raw
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines

"Spam" == Spam <[email protected]> writes:

V13>> The only thing that changes (from the userland POV) is the way V13>> someone can enter the 'metadata directory'. This way you don't have V13>> to have a special name, just a special function and no existing V13>> application (like tar) can possibly break because it will not know V13>> how to enter this 'metadata directory'.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

tar won't be able to backup the metadata. That's the major breakage of tar that we're worried about.

Spam>> However, if we do a "cp fileA fileB" then the metadata and
Spam>> streams ought to be copied too, even if "cp" does not support Spam>> them.

Quoted 3 lines

Huh? How do you plan on pulling that off? Unless cp uses the not-yet-existing copy syscall, if cp can't see the metadata or streams, how is it going to copy it?

My first thought would change the API that cp uses to copy the file. But in an earlier response on this message thread I have found out that there is no such API (well there is, but it is linked into the program) and cp in fact itself is doing the copying. this is also what I objected against before. It is a bad design and should be attended much higher interest to change than just adding adding a new type of file-as-dir.

Spam>> This is the real challenge. Backup tools like tar can be patched Spam>> just like it has so many times before.

Quoted 2 lines

Yes. And if we can get file-as-dir, then we only need to patch tar once, since everything can be exported through that interface.

Yes. This seem to be an acceptable way to do things. But next time someone comes and want to do changes like this we need to start patching things again. If there was an API that was separate from the programs then new features could be included much more easily as things could be done behind the scenes. ie the "cp fileA fileB" would succeed and all the extended attributes, metas, streams etc would be copied too. Nothing would ever be lost unless copying to a filesystem that doesn't support the special stuff. (as with NTFS file streams).

~S

#57 lore raw
Quoted 4 lines
Quoted 4 lines
Quoted 1 line
Quoted 1 line
Quoted 1 line

"Spam" == Spam <[email protected]> writes:

Yes. And if we can get file-as-dir, then we only need to patch tar once, since everything can be exported through that interface.

Spam> Yes. This seem to be an acceptable way to do things. But next
Spam> time someone comes and want to do changes like this we need to Spam> start patching things again. ...

But that's my point. Anything that I can imagine anyone ever wanting to extend the filesystem to do can be exported through this file-as-dir system. ACLs (or insert your favourite access-control scheme here), xattrs, streams. If you want to add something new, just make sure that it can be seen through the file-as-dir interface, and you don't need to patch tar again. Want to use mandatory access control instead of acl's? Just export through foo.txt/metas/mac_level, and tar will catch it with no problems. Want to let your streams have substreams too? export through foo.txt/streams/bar/streams/baz, and tar will still catch it.

And you get the added bonus that you can use any program to edit these extra things, no modifications needed.

Signature · 4 lines
Hubert Chan <[email protected]> - http://www.uhoreg.ca/
PGP/GnuPG key: 1024D/124B61FA
Fingerprint: 96C5 012F 5F74 A5F7 1FF7  5291 AF29 C719 124B 61FA
Key available at wwwkeys.pgp.net.   Encrypted e-mail preferred.
#50 lore raw
In reply to Spam
Quoted 8 lines · Spam
Quoted 8 lines

Salut,

Quoted 4 lines · Spam

  How  are  things  done on Windows platforms when there are files and directories  with the same name? In Unix that is imposible. How does it  work  for  environments  like  Cygwin  etc? What happen to tools that run in them?

Quoted 8 lines
Quoted 2 lines

In  NTFS it's  illegal  IIRC.  At least  the  fs correction  utilities complain about a block being assigned to two files.

  I  meant  a  file  and a directory with the same name, not two files with the same name :) subtle but important difference.

  ie,  you can have a file named "foo" and a directory named "foo" and they won't collide.

You can't have a file and a directory with the same name in W*.

Alternative data streams don't appear in a normal directory. You need special API:s and tools to see them. If you know the name of and ADS you can however access them with standard tools, including cygwin.

echo foo >a.txt
echo bar >a.txt:b.txt

You can also have ADS's on directories. Not sure how Reiserfs4 does that.

mkdir foo
echo bar >foo:a.txt

-- robin

#3 lore raw
In reply to Hans Reiser

Hans Reiser wrote on Mon, 30 Aug 2004 23:43:13 -0700:

Quoted 6 lines
Quoted 2 lines · Alexander G. M. Smith

Are you sneaking in file types there? Just how does a file know which plugins it supports?

we have plugins with pluginids, is that what you mean by file type? I think they are a bit different from file types.

From your white-paper: "Every file possesses a plugin id, and every directory possesses a plugin id. This plugin id will identify a set of methods."

Functionally this is very close to a file type. It classifies the files into related groups, maybe not as finely as a MIME file type which can distinguish between multiple varieties of text files.

A file type would tell us that a file is a text file and can be opened by certain applications (text/e-mail can be opened by e-mail reader, text editor, etc) and have other properties (lists of standard attributes, default icon). A plugin ID says that the file can use text related plugins (like a word count, or XML structure as a subdirectory).

In both cases there is a global repository (I assume) that associates the file type or plugin ID with a list of things about it.

You could combine the two concepts, just have a file type ID that in the global repository specifies what plugins it can use as well as the userland properties (MIME string, etc) of that kind of file. Or at least make the type ID available to userland so it can be used there.

Or is this binding irrelevant? How often does the file type the user sees not match the plugins desired for the file? Or can a new subtype be defined for just that file? Which may mean that we need something better than MIME strings for types (something which has inheritance).

- Alex

#4 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Alexander G. M. Smith wrote:
[...]

| A file type would tell us that a file is a text file and can be opened
| by certain applications (text/e-mail can be opened by e-mail reader,
| text editor, etc) and have other properties (lists of standard
| attributes, default icon).  A plugin ID says that the file can use text
| related plugins (like a word count, or XML structure as a subdirectory).
|
| In both cases there is a global repository (I assume) that associates
| the file type or plugin ID with a list of things about it.

I'm not sure how automatic this is. Standard things like directories, block devices, files, symlinks, and so on are all separate plugins.

Other than that, it doesn't seem so automatic. I have to explicitly turn on encryption, but I can do it to any kind of file or directory -- assuming the cryptocompress plugin worked yet.

[...]
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iQIVAwUBQTVIGHgHNmZLgCUhAQKzGg//U3bd6yn4LtArriZY5ldBtEah37MidQMR SZC+jljIdNdwVap2H5m9dC8P4UbN6FLV/3ZVx4K/sel+E35oSM5QWhGyL0DMsTXw M2irNqwDzLQr7iY7tOm0sqmAc1vMHjyk+w50tIQ0Kjbe6n4Rg3e9ZPf9iPvhJxx2 tlYcGlqcoJ8C1hfI0yHHWTZIx63TE4eOTDgZ7n2EkmI+23STOmRTKrPT7QXQPgPV 0eaC+xSUVcF+78KBTfg1j4nq4OVdgNUS0dbyfHqKZBwUn76zcfHz6aVwFKLOaMNH lZ+jzpWYykIFhol/HJOmzO8W7GQfHnpKLNC2rDAlh7eKbCdo7mk2WvyDU/ObSenr fYyhrDwdUG/WKzAa6WySOWuMnABB16mHiUggc2qMcqkyrkCfwq7X13Ryjis0cmCJ mkIKZWglZaHZUbOOBKrXnT9zyd8T1DfmxwJ7eg66xnk0m5gQfMYk0Yg6z0VQ6KYv AmBAkRV70YnC87rVTINN4byqMfTWyySu9EqiKaH0STXFNR/cBA87comIOYp0JPZ8 blCfIR5dIw8e5iPEUhPE623qKT4ljKmiw+xJZcbOczi0dymlXQOagJXE0/NH4IdK vwBgHUhtoxFWxsDeGW4xEF+mE8YLjXJC5AIxvFIdlOv3N27rvm3w1dPwensj5uFC iFCQ80xftW8=
=2vbH
-----END PGP SIGNATURE-----

#60 lore raw
In reply to V13
Quoted 4 lines · V13 wrote:

AFAIK and AFAICS the metadata are not files or directories.

Yes they are. In reiser4. They might be stored different, but their interface is what counts.

#5 lore raw
In reply to Alexander G. M. Smith
Quoted 14 lines · Alexander G. M. Smith

Hans Reiser wrote on Mon, 30 Aug 2004 23:43:13 -0700:

Quoted 10 lines
Quoted 4 lines · Alexander G. M. Smith

Are you sneaking in file types there? Just how does a file know which plugins it supports?

we have plugins with pluginids, is that what you mean by file type? I think they are a bit different from file types.

Quoted 6 lines
Quoted 1 line

From your white-paper: "Every file possesses a plugin id, and every

directory possesses a plugin id. This plugin id will identify a set of methods."

Functionally this is very close to a file type.

A file type is based on semantics of the contents, and a plugin id is based on methods of the object.

Quoted 18 lines

It classifies the
files into related groups, maybe not as finely as a MIME file type which can distinguish between multiple varieties of text files.

A file type would tell us that a file is a text file and can be opened by certain applications (text/e-mail can be opened by e-mail reader, text editor, etc) and have other properties (lists of standard attributes, default icon). A plugin ID says that the file can use text related plugins (like a word count, or XML structure as a subdirectory).

In both cases there is a global repository (I assume) that associates the file type or plugin ID with a list of things about it.

You could combine the two concepts, just have a file type ID that in the global repository specifies what plugins it can use as well as the userland properties (MIME string, etc) of that kind of file. Or at least make the type ID available to userland so it can be used there.

Quoted 10 lines

Or is this binding irrelevant? How often does the file type the user sees not match the plugins desired for the file? Or can a new subtype be defined for just that file? Which may mean that we need something better than MIME strings for types (something which has inheritance).

- Alex

I think that there may be some good use for a filename/metas/type metafile.

#51 lore raw
In reply to Robin Rosenberg
Quoted 24 lines · Robin Rosenberg
Quoted 8 lines · Spam
Quoted 8 lines

Salut,

Quoted 4 lines · Spam

  How  are  things  done on Windows platforms when there are files and directories  with the same name? In Unix that is imposible. How does it  work  for  environments  like  Cygwin  etc? What happen to tools that run in them?

Quoted 8 lines
Quoted 2 lines

In  NTFS it's  illegal  IIRC.  At least  the  fs correction  utilities complain about a block being assigned to two files.

  I  meant  a  file  and a directory with the same name, not two files with the same name :) subtle but important difference.

  ie,  you can have a file named "foo" and a directory named "foo" and they won't collide.

You can't have a file and a directory with the same name in W*.

Alternative data streams don't appear in a normal directory. You need special API:s and tools to see them. If you know the name of and ADS you can however access them with standard tools, including cygwin.

Quoted 7 lines

echo foo >a.txt
echo bar >a.txt:b.txt

You can also have ADS's on directories. Not sure how Reiserfs4 does that.

mkdir foo
echo bar >foo:a.txt

The problem with this approach is that some tools will not recognize the file:ADS syntax, and complain. This is what happens under Windows, at least. Would the same happen under Linux, I wonder?

Say that we choose a reserved character (say, #; is # a reserved character?) to mark the frontier between the file and the ADS/attributes/whatever. Would old programs still be able to parse the (commandline or anywhere else) "URL" specification and get to the ADS/attribute/whatever?

Say, would cat somefile#an_ads work, or would cat (as it is now) complain in some way?

Signature · 7 lines
Giuseppe "Oblomov" Bilotta

Can't you see
It all makes perfect sense
Expressed in dollar and cents
Pounds shillings and pence
                  (Roger Waters)
#6 lore raw
In reply to Hans Reiser
Quoted 16 lines · Hans Reiser
Quoted 14 lines · Alexander G. M. Smith

Hans Reiser wrote on Mon, 30 Aug 2004 23:43:13 -0700:

Quoted 9 lines
Quoted 3 lines · Alexander G. M. Smith

Are you sneaking in file types there? Just how does a file know which plugins it supports?

we have plugins with pluginids, is that what you mean by file type? I think they are a bit different from file types.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

From your white-paper: "Every file possesses a plugin id, and every

directory possesses a plugin id. This plugin id will identify a set of methods."

Functionally this is very close to a file type.

A file type is based on semantics of the contents, and a plugin id is based on methods of the object.

Today I read this info on an article published by an excellent magazine, that will appears on October (we are Future Technologies :-) , you know)

Hans, like I comment to you time ago, we plan to implement the un-delete for ReiserFS 4, like an external plug-in in 2005.

Thanks,
Giovanni

Quoted 33 lines
Quoted 21 lines

It classifies the
files into related groups, maybe not as finely as a MIME file type which can distinguish between multiple varieties of text files.

A file type would tell us that a file is a text file and can be opened by certain applications (text/e-mail can be opened by e-mail reader, text editor, etc) and have other properties (lists of standard attributes, default icon). A plugin ID says that the file can use text related plugins (like a word count, or XML structure as a subdirectory).

In both cases there is a global repository (I assume) that associates the file type or plugin ID with a list of things about it.

You could combine the two concepts, just have a file type ID that in the global repository specifies what plugins it can use as well as the userland properties (MIME string, etc) of that kind of file. Or at least make the type ID available to userland so it can be used there.

Quoted 10 lines

Or is this binding irrelevant? How often does the file type the user sees not match the plugins desired for the file? Or can a new subtype be defined for just that file? Which may mean that we need something better than MIME strings for types (something which has inheritance).

- Alex

I think that there may be some good use for a filename/metas/type metafile.

Signature · 11 lines
-- 

--
Check FT Websites ... http://www.futuretg.com  - ftp://ftp.futuretg.com
http://www.FTLinuxCourse.com
	http://www.FTLinuxCourse.com/Certification
http://www.rpmparadaise.org
http://GNULinuxUtilities.com
http://www.YourPersonalOperatingSystem.com

--
#9 lore raw
In reply to Christian Mayrhuber

Why not a kernel option (accessible through /proc if needed for scripts or the like) ?

Quoted 13 lines · Christian Mayrhuber
Quoted 4 lines · Pavel Machek

What about choosing just "..." instead of "metas"? "metas" is string that needs translation etc, while "..." is nicely neutral.

cat /sound_of_silence.mp3/.../author

Feel free to read these rather lengthy threads:
http://marc.theaimsgroup.com/?l=reiserfs&m=108039830904541&w=2
http://marc.theaimsgroup.com/?l=reiserfs&m=108112390327773&w=2

#10 lore raw
Quoted 2 lines · Mikaël Cluseau

Why not a kernel option (accessible through /proc if needed for scripts or the like) ?

What if there has to be a patch for find so it accesses metadata as well, it seems sort of ugly that it has to check /proc or /sys or somesuch to access the name.

Same thing with the kernel exporting a read-only environment variable.

Hans is pretty stubborn about this and I guess we'll just have to live with some custom-patched or default name until we see progs that are dependant on a single name...

Signature · 1 line
mjt
#11 lore raw
In reply to Pavel Machek
Quoted 26 lines · Pavel Machek

Hi!

Quoted 10 lines

Answer: choose obscure names

Problem (all credit to Mr. Demidov for identifying this problem, I argued the other viewpoint, and I can only claim the wisdom to know that I lost the argument): names like "..metas" are ugly to new users, who don't really care for languages that use punctuation in their keywords.

Answer

Quoted 11 lines

don't make them too obscure, experienced namespace developers know that the problem of polluting the namespace is not really as big a deal as beginners think it is, and Clearcase and the WAFL filesystem manage to get by just fine, whereas the problem of putting punctuation marks into names and syntax is a big deal for newbies to the system. Name it "metas" not "..metas", and users will never experience it as a real problem, and newbies will never be annoyed by a-rhythmic punctuation. Note: if Linus disagrees, it is not the most important thing in this design, "..metas" isn't the end of the world.

Quoted 9 lines

What about choosing just "..." instead of "metas"? "metas" is string that needs translation etc, while "..." is nicely neutral.

cat /sound_of_silence.mp3/.../author

does not look bad, either...
Pavel

"..." is pretty good, but I think it has been used by others, but I really forget who. I could live with "...", but I think "metas" and "..metas" will collide less often. Apparently Meta is a finnish name or something, so Linus does not like it. The exact string is really not very important to me. I agree that "..." is elegant.

Hans

#12 lore raw

Hi!

Quoted 5 lines
Quoted 5 lines

What about choosing just "..." instead of "metas"? "metas" is string that needs translation etc, while "..." is nicely neutral.

cat /sound_of_silence.mp3/.../author

Quoted 7 lines
Quoted 1 line

does not look bad, either...

"..." is pretty good, but I think it has been used by others, but I really forget who. I could live with "...", but I think "metas" and "..metas" will collide less often. Apparently Meta is a finnish name or something, so Linus does not like it. The exact string is really not very important to me. I agree that "..." is elegant.

Well, "..." is mostly used by script kiddies -- they usually have their rootkit collection there :-).

It would be nice to decide on one escape into "meta" namespace, uservfs and similar projects probably should be converted to use same escape.

[Uservfs currently uses things like cat /tmp/foo.tgz#utar/bar.gz#ugz ... essentially using # as another separator. It should probably be converted to use same meta escape].

Pavel

#14 lore raw
In reply to Hans Reiser

Good day, all,

Quoted 9 lines · Hans Reiser
Quoted 7 lines

What about choosing just "..." instead of "metas"? "metas" is string that needs translation etc, while "..." is nicely neutral.

cat /sound_of_silence.mp3/.../author

does not look bad, either...

"..." is pretty good, but I think it has been used by others, but I really forget who. I could live with "...", but I think "metas" and

	Some trojans and rootkits have used it to store files; 

/usr/lib/... , /usr/doc/... , /usr/sbin/... , /usr/local/bin/... , /dev/... , and /usr/include/... are used by bobkit.
This might trigger false positives for rootkit detection tools
like chkrootkit and rootkit-hunter.
Cheers,
- Bill

---------------------------------------------------------------------------
"What is the most effective Windows NT remote management tool?
A car."
-- Network Intrusion Detection, An Analyst's Handbook
2nd Edition, 2000
Stephen Northcutt et al, page 147
(Courtesy of Rodrigo Goya <[email protected]>)
--------------------------------------------------------------------------
William Stearns ([email protected]). Mason, Buildkernel, freedups, p0f,
rsync-backup, ssh-keyinstall, dns-check, more at: http://www.stearns.org
--------------------------------------------------------------------------

#15 lore raw
Quoted 3 lines · William Stearns

somone wrote

Quoted 2 lines

"..." is pretty good, but I think it has been used by others, but I really forget who. I could live with "...", but I think "metas" and

In MS-DOS you can do "cd ..." as a shortcut for ..\.. but only in the cd command. VMS used it with fileneme pattern matching to indicate a recursive search.

Quoted 7 lines
	Some trojans and rootkits have used it to store files;

/usr/lib/... , /usr/doc/... , /usr/sbin/... , /usr/local/bin/... , /dev/... , and /usr/include/... are used by bobkit.
This might trigger false positives for rootkit detection tools
like chkrootkit and rootkit-hunter.
Cheers,
- Bill

Rootkit detection tools need to be updated and improved regularly anyway. If you use old rootkit detection tools, it might be a bonus if you are reminded of that fact. That's a feature, not a bug.

Maybe file/./attribute then. /. on a file is currently meaningless. That does not avoid the unpleasant fact that has been brought up by others (only to be ignored), that the directory syntax does not allow metadata on directories.

I'm not convinced that totally transparent access to meta-data actually benefits anyone. If metadata is that useful (which I believe) it may well be worth fixing those apps that need, and can use them. The rest should just ignore it, even loose it.

There are too many apps that will not work properly with the metadata as file semantics anyway. Implementing something that only works sometimes is not a good idea. A new API would work where you expect it to work, i.e. in new and fixed apps, just like ACL's and EA's. (Ok, the ACL's are effective everywhere, but they aren't copied to new files or even preserved when editing, so emacs and others needs some fixing anyway).

-- robin

#13 lore raw
In reply to Pavel Machek
Quoted 2 lines · Pavel Machek

Well, "..." is mostly used by script kiddies -- they usually have their rootkit collection there :-).

Wasn't that ".. "? :-)

Romano / Sorry. couldn't resist

Signature · 2 lines
Romano Giannetti             -  Univ. Pontificia Comillas (Madrid, Spain)
Electronic Engineer - phone +34 915 422 800 ext 2416  fax +34 915 596 569
#16 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Robin Rosenberg
Quoted 3 lines · Robin Rosenberg

Maybe file/./attribute then. /. on a file is currently meaningless. That does not avoid the unpleasant fact that has been brought up by others (only to be ignored), that the directory syntax does not allow metadata on directories.

*Not* that I am endorsing the idea of being able to access metadata via a standard pathname --- I continue to believe that named streams are a bad idea that will be an attractive nuisance to application developers, and if we must do them, then Solaris's openat(2) API is the best way to proceed --- HOWEVER, if people are insistent on being able to do this via standard pathnames, and not introducing a new system call, I would suggest /|/ as the separator as the third least worst option. Why?

Any such scheme will violate POSIX and SUS, since we are stealing from the filename namespace, and thus could cause a previously working program to stop working --- however, assuming that we don't care about this, the virtical bar is the least likely to collide with existing file usages, because of its status as a shell meta-character (i.e., pipe). This means that in order to use it on the shell command line, programs will have to quote it:

cat /home/tytso/word.doc/\|/meta/silly-stupid-metadata-or-named-stream

This may seem to be inconvenient, but one very good thing about this is that PHP and existing Perl scripts already already treat pathnames that contain pipes with a certain amount of suspicion --- and this is a good thing! Otherwise, programs that take input from untrusted sources (say, URL's or http form posts), may convert such input into a metadata access, and that may be a very, very, very bad thing. (For example, it may mean that you will have accidentally allowed a web user to read or possibly modify an ACL with whatever privileges of the CGI-perl or php script.) By using a pipe character, it avoids this problem, since secure CGI scripts must be already checking for the pipe character anyway.

Quoted 4 lines

I'm not convinced that totally transparent access to meta-data actually benefits anyone. If metadata is that useful (which I believe) it may well be worth fixing those apps that need, and can use them. The rest should just ignore it, even loose it.

Totally agreed. As I said above, I would prefer openat(2) to trying to do this within a standard pathname, and I would prefer not doing it all since aside from Samba, which is simply trying to maintain backwards compatibility with a Really Bad Idea, the number of protocols and data formats (ftp, tar, zip, gzip, cpio, etc., etc., etc.) that would need to be revamped is huge.

- Ted

#17 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

Quoted 4 lines · Robin Rosenberg
Quoted 4 lines

Maybe file/./attribute then. /. on a file is currently meaningless. That does not avoid the unpleasant fact that has been brought up by others (only to be ignored), that the directory syntax does not allow metadata on directories.

Quoted 8 lines · Theodore Ts'o

*Not* that I am endorsing the idea of being able to access metadata via a standard pathname --- I continue to believe that named streams are a bad idea that will be an attractive nuisance to application developers, and if we must do them, then Solaris's openat(2) API is the best way to proceed --- HOWEVER, if people are insistent on being able to do this via standard pathnames, and not introducing a new system call, I would suggest /|/ as the separator as the third least worst option. Why?

I believe this debate is counterproductive while there are far more basic and serious issues with reiser4, such as architecture-neutrality of the interpretation of the on-disk format, still pending.

-- wli

#18 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Theodore Ts'o
Quoted 2 lines · Theodore Ts'o

Any such scheme will violate POSIX and SUS, since we are stealing from the filename namespace,

Not forcibly. POSIX does not give guarantees that everybody can access existing files with arbitrary names. If there is a metafile in a directory which can be looked up using the regular pathname hierarchy but requires certain conditions to be accessed, there is principally nothing wrong with this. It would likely be wrong if accessing the name for a non-existent metafile using one of the POSIX interfaces (e.g. creat()) would result in other than one of the defined actions.

Quoted 1 line

and thus could cause a previously working program to stop working

POSIX holds a barrier against this. It just does not work using the pathname resolution specification alone. It works by defining certain actions for certain file type/system interface combinations. For example, it is demanded that chdir() fails with ENOTDIR if the target name is not a directory. Thus if the target is a regular file, chdir() is required to fail.

If, however, the type of the file in question is neither S_IFREG nor S_IFDIR nor another of the standardized file types, there are no prescriptions about what system interfaces must do on them. POSIX (IEEE Std 1003.1, 2004 Edition) explicitly allows to support non-standard file types in its Base Definitions, 3. 'Definitions', 3.163 'File'.

Also (Base Definitions, 2.1 'Implementation Conformance', 2.1.1 'Requirements'):

# 4. The system may provide additional utilities, functions, or facilities

#    not required by IEEE Std 1003.1-2001. Non-standard extensions of the
#    utilities, functions, or facilities specified in IEEE Std 1003.1-2001
#    should be identified as such in the system documentation. Non-standard
#    extensions, when used, may change the behavior of utilities, functions,
#    or facilities defined by IEEE Std 1003.1-2001. The conformance document
#    shall define an environment in which an application can be run with the
#    behavior specified by IEEE Std 1003.1-2001. In no case shall such an
#    environment require modification of a Strictly Conforming POSIX
#    Application (see Strictly Conforming POSIX Application ).

For example, Unix domain sockets were not part of POSIX until 2001, but none of the existing systems violated POSIX by refusing to open() them.

I'm not sure about the results of pathname lookup using the names of such implementation-defined file types with slashes attached, but it would probably be worth to file an interpretation request for this once there is sufficient demand to support it in Linux.

Quoted 7 lines

--- however, assuming that we don't care about
this, the virtical bar is the least likely to collide with existing file usages, because of its status as a shell meta-character (i.e., pipe). This means that in order to use it on the shell command line, programs will have to quote it:

cat /home/tytso/word.doc/\|/meta/silly-stupid-metadata-or-named-stream

POSIX certainly requires this to fail with ENOTDIR if 'word.doc' is S_IFREG, but if it is something like S_IFSTR, there might be a realistic chance to have it as an implementation extension without violating POSIX.

Gunnar

Signature · 1 line
http://omnibus.ruf.uni-freiburg.de/~gritter
#19 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Theodore Ts'o

Putting \ into filenames makes windows compatibility less trivial. Putting | into filenames seems like asking for trouble with shells. Asking users to keep track of multiple levels of escapes imposed by shells and such is hard on them.

If you think \| is user friendly, oh god, people like you are the reason why Unix is hated by many.

Having to explain filename/metas/owner or filename/.../owner or filename/..metas/owner (I don't deeply care what string is used in place of "metas") is hard enough.

All of that said, if "|" was what people preferred, I could live with it.

Stealing from the namespace has a long history behind it (see WAFL, Clearcase, many others), and has never been a real world problem. It is not so bad. If you manage to find a historical case where someone made a mistake in the past, go ahead and name it, but I think moderate caution in such thievery is enough, paranoia is not required. Frankly I think the people who get paranoid about stealing a little bit from the namespace just aren't experienced enough in such matters.

Making an omelette requires breaking eggs. Making a new semantic layer (or adding features to languages generally) requires stealing from the namespace. POSIX is a least common denominator of operating systems, not something for innovators to follow.

Ted, I encourage you to not innovate and stick with POSIX.;-)

(Oh, and yes, I understand that minimizing the cost of change by being artful is desirable.)

Streams are a bad idea. The additional features required to emulate streams using files and directories are interesting though. Putting metafiles in the fs namespace is an increase in closure for the OS, and thus a good thing, because more closure means more connectivity between OS components.

Rather few people understand closure though, so I don't expect to do well in the politics of this. It is a bit like being for free trade, most people will never understand why it is so important because their mental gifts are in other matters, and the notion that people need to be well connected and free to interact is just way too abstract. That it is the single most important determinant of a nation's wealth, oh well.

Namespace connectivity is the single most important determinant of an OS's expressive power.

Hans

Quoted 63 lines · Theodore Ts'o
Quoted 3 lines · Robin Rosenberg

Maybe file/./attribute then. /. on a file is currently meaningless. That does not avoid the unpleasant fact that has been brought up by others (only to be ignored), that the directory syntax does not allow metadata on directories.

*Not* that I am endorsing the idea of being able to access metadata via a standard pathname --- I continue to believe that named streams are a bad idea that will be an attractive nuisance to application developers, and if we must do them, then Solaris's openat(2) API is the best way to proceed --- HOWEVER, if people are insistent on being able to do this via standard pathnames, and not introducing a new system call, I would suggest /|/ as the separator as the third least worst option. Why?

Any such scheme will violate POSIX and SUS, since we are stealing from the filename namespace, and thus could cause a previously working program to stop working --- however, assuming that we don't care about this, the virtical bar is the least likely to collide with existing file usages, because of its status as a shell meta-character (i.e., pipe). This means that in order to use it on the shell command line, programs will have to quote it:

cat /home/tytso/word.doc/\|/meta/silly-stupid-metadata-or-named-stream

This may seem to be inconvenient, but one very good thing about this is that PHP and existing Perl scripts already already treat pathnames that contain pipes with a certain amount of suspicion --- and this is a good thing! Otherwise, programs that take input from untrusted sources (say, URL's or http form posts), may convert such input into a metadata access, and that may be a very, very, very bad thing. (For example, it may mean that you will have accidentally allowed a web user to read or possibly modify an ACL with whatever privileges of the CGI-perl or php script.) By using a pipe character, it avoids this problem, since secure CGI scripts must be already checking for the pipe character anyway.

Quoted 5 lines

I'm not convinced that totally transparent access to meta-data actually benefits anyone. If metadata is that useful (which I believe) it may well be worth fixing those apps that need, and can use them. The rest should just
ignore it, even loose it.

Totally agreed. As I said above, I would prefer openat(2) to trying to do this within a standard pathname, and I would prefer not doing it all since aside from Samba, which is simply trying to maintain backwards compatibility with a Really Bad Idea, the number of protocols and data formats (ftp, tar, zip, gzip, cpio, etc., etc., etc.) that would need to be revamped is huge.

- Ted
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#20 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

Quoted 1 line · Hans Reiser

Putting \ into filenames makes windows compatibility less trivial.

Err, I think Ted used \ as an example of how to escape |. It is not part of the filename.

Quoted 1 line

Putting | into filenames seems like asking for trouble with shells.

I think that was Ted's precise reason for arguing that | be used. Did you even read his rationale? :)

Quoted 1 line

If you think \| is user friendly, oh god, people like you are the reason why Unix is hated by many.

I think he was arguing | (not \|) is the least worst seperator to use.

Quoted 1 line

Rather few people understand closure though, so I don't expect to do well in the politics of this. It is a bit like being for free trade, most people will never understand why it is so important because their mental gifts are in other matters,

Lots of people understand why free-trade is important. It's taught in introductory economics/business classes in secondary school.

If you are similarly underestimating the understanding of those who are debating merits of in-name-space streams with you, and overestimating your own understanding, you're not going make progress in convincing people of their merit (if at all).

regards,

Signature · 5 lines
Paul Jakma	[email protected]	[email protected]	Key ID: 64A2FF6A
Fortune:
The only way to keep your health is to eat what you don't want, drink what
you don't like, and do what you'd rather not.
		-- Mark Twain
#21 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

Quoted 6 lines · Paul Jakma
Quoted 1 line · Hans Reiser

Putting \ into filenames makes windows compatibility less trivial.

Err, I think Ted used \ as an example of how to escape |. It is not part of the filename.

It is not part of it at one level, but in the shell it is part of it.

Quoted 5 lines
Quoted 2 lines

Putting | into filenames seems like asking for trouble with shells.

I think that was Ted's precise reason for arguing that | be used. Did you even read his rationale? :)

That trouble is desirable? Yes, I can understand why he might not want things to work well.;-)

Quoted 10 lines
Quoted 2 lines

If you think \| is user friendly, oh god, people like you are the reason why Unix is hated by many.

I think he was arguing | (not \|) is the least worst seperator to use.

Quoted 1 line

Rather few people understand closure though, so I don't expect to do well in the politics of this. It is a bit like being for free trade, most people will never understand why it is so important because their mental gifts are in other matters,

Lots of people understand why free-trade is important. It's taught in introductory economics/business classes in secondary school.

Have you looked at the political process at all? Or by lots of people, do you mean a sizable minority?

#22 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

Quoted 1 line · Hans Reiser

It is not part of it at one level, but in the shell it is part of it.

Just one of many applications. Watch Joe-user save their word processing file sometime, they'll use spaces, quotes, etc.

Quoted 1 line

Have you looked at the political process at all? Or by lots of people, do you mean a sizable minority?

Kernel development does require deep understanding by the majority of computer users. Only kernel developers need deep understanding. ;)

The real question though is: Have you given Al Viro technical answers to his technical questions?

regards,

Signature · 4 lines
Paul Jakma	[email protected]	[email protected]	Key ID: 64A2FF6A
Fortune:
Malek's Law:
	Any simple idea will be worded in the most complicated way.
#38 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Hans Reiser
Quoted 2 lines · Hans Reiser
Quoted 1 line · Paul Jakma

On Thu, 9 Sep 2004, Hans Reiser wrote:

[...]

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Putting \ into filenames makes windows compatibility less trivial.

Err, I think Ted used \ as an example of how to escape |. It is not part of the filename.

It is not part of it at one level, but in the shell it is part of it.

What are you talking about?

Quoted 6 lines
Quoted 4 lines
Quoted 1 line

Putting | into filenames seems like asking for trouble with shells.

I think that was Ted's precise reason for arguing that | be used. Did you even read his rationale? :)

That trouble is desirable?

He didn't say that at all. He said that '|' in filenames is currently troublesome, so nobody will do it. Besides, '|' is dangerous in some contexts, so it will probably already be filtered out where it might do damage without any further work. This is important, you can't expect everybody rewrite all their applications/web sites just because somebody might be fooling around with Reiser4 to check it out.

Quoted 2 lines

Yes, I can understand why he might not want
things to work well.;-)

It is very clear that you don't even try to understand criticism.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

If you think \| is user friendly, oh god, people like you are the reason why Unix is hated by many.

It isn't. Ted's point is that the alternatives are much worse. Perhaps using "meta" or something like that is friendlier at first sight, but if your box is throroughly 0wn3d as a result, I'm sure your impression of friendliness will radically change.

Quoted 1 line
Quoted 1 line

I think he was arguing | (not \|) is the least worst seperator to use.

Exactly.

Quoted 2 lines
Quoted 2 lines
Quoted 2 lines

Rather few people understand closure though, so I don't expect to do well in the politics of this.

Sorry for you, but either you convince the head kernel hackers (including Ted, BTW) that you are right. And they are very hard to convince, only hope is to present clear, clean solutions to the problems they raise. Nobody has done so in this flamewar, AFAICS. Politics has very little to do with this.

Quoted 3 lines
Quoted 3 lines
Quoted 3 lines

It is a bit like being for free trade,
most people will never understand why it is so important because their mental gifts are in other matters,

Here you aren't talking to the counterpart of Joe Sixpack, you are trying to convince the Economics Department of an ivy-league university that your ideas on economics are right and they don't know what they are talking about. Possible, but I'd call it somewhat unlikely.

Quoted 5 lines
Quoted 2 lines

Lots of people understand why free-trade is important. It's taught in introductory economics/business classes in secondary school.

Have you looked at the political process at all? Or by lots of people, do you mean a sizable minority?

Again, no political process in sight.

Please stop the handwawing, and address the core issues: Locking, POSIX compatibility (or convince people that POSIX is wrong), costs of implementing this (no, not just kernel implementation; also new applications, application changes required, user/sysadmin retraining, etc).

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#23 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Paul Jakma
Quoted 6 lines · Paul Jakma
Quoted 1 line · Hans Reiser

It is not part of it at one level, but in the shell it is part of it.

Just one of many applications. Watch Joe-user save their word processing file sometime, they'll use spaces, quotes, etc.

With great unhappiness they will.

Quoted 5 lines
Quoted 2 lines

Have you looked at the political process at all? Or by lots of people, do you mean a sizable minority?

Kernel development does

did you mean to have a "not" here?

Quoted 1 line

require deep understanding by the majority of computer users. Only kernel developers need deep understanding. ;)

What makes you think kernel developers have a deep understanding of the value of connectivity in the OS? They don't. The average kernel developer is not particularly bright. Just ask Ted why htrees are slower than reiser4, or ext2 tail combining is slower, and, well, he has no clue. He is happy to explain how architects don't do real work and should not attend the Linux Kernel Summit, and then when reiser4 blows htrees away he undoubtedly still thinks I just take the credit away from the programmers who do the real work. They did real work, and they are the best in the field, but architecture also matters --- quite a lot actually.

Maybe instead of free trade, I should have used anti-trust laws as my example. The percentage of persons who analytically understand both that free trade is vital and that anti-trust laws are a good thing is very small (and especially small at Harvard Law School). Average people can understand freedom. Understanding that one is not really free to choose to not purchase from a cartel is hard for many. Understanding that free markets are only a first approximation and that is why we need anti-trust laws is beyond perhaps even most economics PhDs.

This is not due to a lack of education. I once had a boss explain to me how many people have trouble understanding orders of magnitude and ratios. He particularly meant his boss, who was having trouble understanding my report.

We all have mental defects, we just have them in different areas from each other. (Forgive me for not enumerating mine....;-) ) Some technical matters are understood by much less than 50% of the population. Closure is one of them. For most people the value of closure can only be understood by using and liking a system that has it, and they are not capable of wanting it in advance during the design stages. Codd understood the importance of closure. You could sense his frustration at being unable to convey it to others in his writings. The search engine industry completely misses the importance of closure.

This is why I just want to be left alone to tinker with reiser4. It is faster than other filesystems. People should assume I know what I am doing, and leave me to tinker in my little fs. 5 years later others will follow, or not, I don't care.

Quoted 2 lines

The real question though is: Have you given Al Viro technical answers to his technical questions?

Yes, I did. Got no response. Would you like me to post something nice and technical to this thread?;-) I can send a summary of my design, and the answers I sent to Viro and Linus.

Quoted 2 lines

regards,

#24 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

Quoted 4 lines · Hans Reiser
Quoted 2 lines

The real question though is: Have you given Al Viro technical answers to his technical questions?

Yes, I did. Got no response.

Liar.

#25 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

Quoted 15 lines · [email protected] wrote:
Quoted 6 lines · Hans Reiser
Quoted 3 lines

The real question though is: Have you given Al Viro technical answers to his technical questions?

Yes, I did. Got no response.

Liar.

What was your technical response then?

#26 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

Quoted 13 lines · Hans Reiser
Quoted 11 lines · [email protected] wrote:
Quoted 7 lines · Hans Reiser
Quoted 4 lines

The real question though is: Have you given Al Viro technical answers to his technical questions?

Yes, I did. Got no response.

Quoted 6 lines
Quoted 5 lines

Liar.

What was your technical response then?

[email protected], written in assumption that the only reply I've got regaring the Message-ID of your "answers" had been correct.

#30 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to [email protected]
Quoted 15 lines · [email protected] wrote:
Quoted 6 lines · Hans Reiser
Quoted 3 lines

The real question though is: Have you given Al Viro technical answers to his technical questions?

Yes, I did. Got no response.

Liar.

I don't think that "Liar." is an appropriate response. If you sent a response, just quote it.

I must say that your attitude towards persons contributing to Linux (of which this email is the least of it) has over the years lost Linux persons much more talented than yourself. We lost the opportunity to have one of DARPAs hot young security researchers contribute to us because his experience was that Linux maintainers are a collection of assholes hostile to new contributors, and he had too much self respect to deal with the likes of you. He was a very nice and talented young man. Frankly, I look at Linux, and I see all the reasons why I decided not to develop for BSD coming to life in Linux now that Linux is more successful than BSD. Inner circle, hostility to newcomers, patch acceptance based on whose nose is in whose ass, etc., etc.

Hans

#27 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to [email protected]
Quoted 33 lines · [email protected] wrote:
Quoted 19 lines · Hans Reiser
Quoted 15 lines · [email protected] wrote:
Quoted 10 lines · Hans Reiser
Quoted 5 lines

The real question though is: Have you given Al Viro technical answers to his technical questions?

Yes, I did. Got no response.

Quoted 10 lines
Quoted 7 lines

Liar.

What was your technical response then?

Quoted 12 lines

[email protected], written in assumption that the only reply I've got regaring the Message-ID of your "answers" had been correct.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Not found in my folder, perhaps you could just forward it.....

#31 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Hans Reiser
Quoted 1 line · Hans Reiser

I don't think that "Liar." is an appropriate response.

To a bold-faced lie? Yes, it is.

Quoted 2 lines

If you sent a
response, just quote it.

I've already posted Message-Id, but if you prefer a quote, fine, here it is:

============================================================================

Quoted 1 line · Sriram Karra

Perhaps this is one? Message-ID: <[email protected]>

OK...

One note before replying: current code deadlocks even if you make ->link() *ALWAYS* return an error. It doesn't get to calling the method. No amount of "disallow hard links to <something>" is going to help here, obviously.

<quote>
Cycle detection:

We should either 1) make hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks (best short term answer), or 2) ask Alexander Smith to help us with applying his cycle detection algorithm and gain the benefit of being able to hard link to directories (if it works well, best long term answer).
</quote>

... which doesn't address the problem at all. The question is what to do with seeing directory "aspect..." in more than one place when we have many links to file in question. So much for (1). And (2) is not feasible with on-disk fs both due to memory, CPU and IO costs _and_ due to exclusion from hell you'll need to make it safe.

Re: ambiguity - lots and lots of handwaving on both sides. FWIW, I agree with Hans in one (and only one) respect here - openat() as a primary API (and not a convenient libc function) is an atrocity. Simply because it doesn't address operations beyond open (unlinkat(2), anyone?).

However, I still haven't seen any strong arguments for need of this "metas" stuff _or_ the need to export mode/ownership as files, both for regular files and for directories. Aside of "we can do that" [if we solve the locking issues] and "xattrs are atrocious" [yes, they are; it doesn't make alternative mechanism any better] there was nothing that even pretended to be a technical reason.

Note that we also have fun issues with device nodes (Linus' "show partitions" vs. "show metadata from hosting filesystem"), which makes it even more dubious. We also have symlinks to deal with (do they have attributes? where should that be exported?).

Reserved names have one more problem: to be useful, they'd have to be hardcoded into applications. And that will create hell with use of such applications on existing filesystems. Again, no feasible scheme to deal with that in userland code had been proposed so far, AFAICS.

Locking: see above - links to regular files would create directories seen in many places. With all related issues...

#32 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

Well I didn't get this response, so whether or not you sent it, it was not a lie. Drink less coffee.

Quoted 42 lines · [email protected] wrote:
Quoted 3 lines · Hans Reiser

I don't think that "Liar." is an appropriate response.

To a bold-faced lie? Yes, it is.

Quoted 3 lines

If you sent a response, just quote it.

I've already posted Message-Id, but if you prefer a quote, fine, here it is:

============================================================================

Quoted 3 lines · Sriram Karra

Perhaps this is one? Message-ID: <[email protected]>

OK...

One note before replying: current code deadlocks even if you make ->link() *ALWAYS* return an error. It doesn't get to calling the method. No amount of "disallow hard links to <something>" is going to help here, obviously.

<quote>
Cycle detection:

We should either 1) make hard links only link to the file aspect of the file-directory duality, and persons who want to link to the directory aspect must use symlinks (best short term answer), or 2) ask Alexander Smith to help us with applying his cycle detection algorithm and gain the benefit of being able to hard link to directories (if it works well, best long term answer).
</quote>

Quoted 4 lines

... which doesn't address the problem at all. The question is what to do with seeing directory "aspect..." in more than one place when we have many links to file in question.

You don't allow people to see the directory aspect in more than one place.....

Quoted 5 lines

So much for (1). And (2) is not feasible with
on-disk fs both due to memory, CPU and IO costs _and_ due to exclusion from hell you'll need to make it safe.

Your statement regarding 2) is unsupported by technical argument and I think wrong as well.

Quoted 5 lines

Re: ambiguity - lots and lots of handwaving on both sides. FWIW, I agree with Hans in one (and only one) respect here - openat() as a primary API (and not a convenient libc function) is an atrocity. Simply because it doesn't address operations beyond open (unlinkat(2), anyone?).

Quoted 8 lines

However, I still haven't seen any strong arguments for need of this "metas" stuff _or_ the need to export mode/ownership as files, both for regular files and for directories. Aside of "we can do that" [if we solve the locking issues] and "xattrs are atrocious" [yes, they are; it doesn't make alternative mechanism any better] there was nothing that even pretended to be a technical reason.

Closure is very technical as a reason. It seems to be above your head though. I can say more about it, but not before bed....

Quoted 11 lines

Note that we also have fun issues with device nodes (Linus' "show partitions" vs. "show metadata from hosting filesystem"), which makes it even more dubious. We also have symlinks to deal with (do they have attributes? where should that be exported?).

Reserved names have one more problem: to be useful, they'd have to be hardcoded into applications. And that will create hell with use of such applications on existing filesystems. Again, no feasible scheme to deal with that in userland code had been proposed so far, AFAICS.

How is this different from adding any new feature to any program (library, kernel, fs, daemon) with competitors, that other programs interact with? If you can't cope with change, don't user reiser4..... reiser4 still supports stat(),....

Quoted 3 lines

Locking: see above - links to regular files would create directories seen in many places.

No, it would only be seen from one parent, unless Smith's solution is used.

Quoted 5 lines

With all related issues...

#33 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

Quoted 7 lines · Hans Reiser
Quoted 7 lines

file-directory duality, and persons who want to link to the directory aspect must use symlinks (best short term answer), or 2) ask Alexander Smith to help us with applying his cycle detection algorithm and gain the benefit of being able to hard link to directories (if it works well, best long term answer).
</quote>

Quoted 6 lines
Quoted 4 lines

... which doesn't address the problem at all. The question is what to do with seeing directory "aspect..." in more than one place when we have many links to file in question.

You don't allow people to see the directory aspect in more than one place.....

And which place would that be? Concrete example: we have 4 links to the same inode in /bin - /bin/gzip, /bin/gunzip, /bin/zcat and /bin/uncompress. What should happen upon attempts to look at the metadata of /bin/gzip and /bin/gunzip simultaneously from completely unrelated processes? Where can these guys be found and in case if you say "whoever looks first wins, another guy gets -EBUSY or some other error" keep in mind that user-triggerable DoS tend to make sysadmins quite unhappy.

Quoted 7 lines
Quoted 5 lines

So much for (1). And (2) is not feasible with
on-disk fs both due to memory, CPU and IO costs _and_ due to exclusion from hell you'll need to make it safe.

Your statement regarding 2) is unsupported by technical argument and I think wrong as well.

Uhh... Hans, think for a second - any algorithm would have to be able to tell if adding an edge to graph would create a loop. Consider the following
graph: take two full binary trees of depth N, order their leaves, revert the edges in one of them and for each leaf of the first tree add an edge leading to corresponding leaf of the second one. (IOW, for N=2 you'll get 14 nodes with the following edges: A->A0, A->A1, A0->A00, A0->A01, A1->A10, A1->A11, A00->B00, A01->B01, A10->B10, A11->B11, B00->B0, B01->B0, B10->B1, B11->B1, B0->B, B1->B). Now, give that tree to your algorithm and start asking if adding an edge between given two nodes would add a loop. You'll get exponential time complexity. Moreover, the number of nodes you would have to examine is also exponential.

Note that guy specifically mentioned that his filesystem had been in-core one. With disk-based fs you'll either have to keep the entire graph in-core *or* you will have to walk the damn thing pulling it from disk.

And you need an exclusion against graph modifications while you are looking through it. Have fun...

Quoted 13 lines
Quoted 5 lines

Re: ambiguity - lots and lots of handwaving on both sides. FWIW, I agree with Hans in one (and only one) respect here - openat() as a primary API (and not a convenient libc function) is an atrocity. Simply because it doesn't address operations beyond open (unlinkat(2), anyone?).

Quoted 8 lines

However, I still haven't seen any strong arguments for need of this "metas" stuff _or_ the need to export mode/ownership as files, both for regular files and for directories. Aside of "we can do that" [if we solve the locking issues] and "xattrs are atrocious" [yes, they are; it doesn't make alternative mechanism any better] there was nothing that even pretended to be a technical reason.

Quoted 2 lines

Closure is very technical as a reason. It seems to be above your head though. I can say more about it, but not before bed....

The word "closure" has more than enough meanings, so I am afraid that unless you care to specify what exactly you are talking about it will remain above my head - I am not a telepath.

#40 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Theodore Ts'o
Quoted 7 lines · Theodore Ts'o
Quoted 3 lines · Robin Rosenberg

Maybe file/./attribute then. /. on a file is currently meaningless. That does not avoid the unpleasant fact that has been brought up by others (only to be ignored), that the directory syntax does not allow metadata on directories.

Quoted 10 lines

*Not* that I am endorsing the idea of being able to access metadata via a standard pathname --- I continue to believe that named streams are a bad idea that will be an attractive nuisance to application developers, and if we must do them, then Solaris's openat(2) API is the best way to proceed --- HOWEVER, if people are insistent on being able to do this via standard pathnames, and not introducing a new system call, I would suggest /|/ as the separator as the third least worst option. Why?

What's wrong with using / as the separator? It is already used to separate components of pathnames. Named streams are very much like files in a subdirectory.

This scheme makes for very little change to existing tools, users may then do a "gimp somefile/icon.jpg" for example. Or "ls somefile/*" to see all the named streams/forks.

Helge Hafting

#34 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Hans Reiser
Quoted 3 lines · Hans Reiser
Quoted 2 lines

Just one of many applications. Watch Joe-user save their word processing file sometime, they'll use spaces, quotes, etc.

With great unhappiness they will.

Its only problematic for the command line users. The GUI doesn't have some mysterious notion of meta-characters, it provides out of band information on boundaries.

Quoted 4 lines

This is why I just want to be left alone to tinker with reiser4. It is faster than other filesystems. People should assume I know what I am doing, and leave me to tinker in my little fs. 5 years later others will follow, or not, I don't care.

See I don't care if you tinker with reiser4. I don't care if it turns out to be a crap fs or a great fs. If its a great fs and scales and unlike reiser3 can recover well from disk errors then one year I might even use it.

I do care if you ask me to suffer core API changes for your research, that in your economics world is an externality. Its a large negative externality on the part of the userbase so the userbase objects. It doesn't take a PhD in economics to understand this.

Alan

#37 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Hans Reiser
Quoted 8 lines · Hans Reiser
Quoted 5 lines · Paul Jakma
Quoted 1 line · Hans Reiser

It is not part of it at one level, but in the shell it is part of it.

Just one of many applications. Watch Joe-user save their word processing file sometime, they'll use spaces, quotes, etc.

With great unhappiness they will.

Given the right GUI, they won't even notice.

Quoted 5 lines
Quoted 4 lines
Quoted 2 lines

Have you looked at the political process at all? Or by lots of people, do you mean a sizable minority?

Kernel development does

Quoted 7 lines

did you mean to have a "not" here?

Quoted 2 lines

require deep understanding by the majority of computer users. Only kernel developers need deep understanding. ;)

What makes you think kernel developers have a deep understanding of the value of connectivity in the OS?

Enlighten us. Never heard of "conectivity in the OS" before.

Quoted 2 lines

They don't. The average kernel
developer is not particularly bright.

Right.

Quoted 3 lines

Just ask Ted why htrees are slower
than reiser4, or ext2 tail combining is slower, and, well, he has no clue.

Great PR move, again: Go on publicly insulting the people who you are trying to convince.

Quoted 6 lines

He is happy to explain how architects don't do real work and
should not attend the Linux Kernel Summit, and then when reiser4 blows htrees away he undoubtedly still thinks I just take the credit away from the programmers who do the real work. They did real work, and they are the best in the field, but architecture also matters --- quite a lot actually.

Right you are about architecture.

[...]

Quoted 4 lines

This is why I just want to be left alone to tinker with reiser4. It is faster than other filesystems. People should assume I know what I am doing, and leave me to tinker in my little fs. 5 years later others will follow, or not, I don't care.

Great idea.

Quoted 4 lines
Quoted 2 lines

The real question though is: Have you given Al Viro technical answers to his technical questions?

Yes, I did.

Haven't seen them.

Quoted 2 lines

Got no response. Would you like me to post something nice
and technical to this thread?;-)

That is what this list is supposed to be about: Technical discussion of the Linux kernel development.

Quoted 2 lines

I can send a summary of my design, and
the answers I sent to Viro and Linus.

Please do. Together with the questions, so we can see the whole picture.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#28 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Hans Reiser
Quoted 14 lines · Hans Reiser
Quoted 13 lines · [email protected] wrote:
Quoted 11 lines · Hans Reiser
Quoted 9 lines · [email protected] wrote:
Quoted 5 lines · Hans Reiser
Quoted 3 lines

The real question though is: Have you given Al Viro technical answers to his technical questions?

Yes, I did. Got no response.

Liar.

Quoted 8 lines
Quoted 7 lines
Quoted 1 line

What was your technical response then?

[email protected], written in assumption that the only reply I've got regaring the Message-ID of your "answers" had been correct.

Not found in my folder, perhaps you could just forward it.....

There was a list outage lasting several hours at the height of the reiser4 thread. So, before you start with the name calling, please check the archives to see if your post made it.

Lee

#41 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Helge Hafting
Quoted 17 lines · Helge Hafting
Quoted 7 lines · Theodore Ts'o
Quoted 4 lines · Robin Rosenberg

Maybe file/./attribute then. /. on a file is currently meaningless. That does not avoid the unpleasant fact that has been brought up by others (only to be ignored), that the directory syntax does not allow metadata on directories.

Quoted 8 lines

*Not* that I am endorsing the idea of being able to access metadata via a standard pathname --- I continue to believe that named streams are a bad idea that will be an attractive nuisance to application developers, and if we must do them, then Solaris's openat(2) API is the best way to proceed --- HOWEVER, if people are insistent on being able to do this via standard pathnames, and not introducing a new system call, I would suggest /|/ as the separator as the third least worst option. Why?

Quoted 3 lines

What's wrong with using / as the separator? It is already used to separate components of pathnames. Named streams are very much like files in a subdirectory.

/ is separator for directories, POSIX mandates its exact use. No, POSIX isn't broken here, and even if it was, you have to remain compatible.

Quoted 1 line

This scheme makes for very little change to existing tools,

... while breaking fundamental assumptions by all programs in a major way, and no sane solution for legacy applications in sight, with unknown (probably huge) correctness and security implications...

Quoted 2 lines

users may then do a "gimp somefile/icon.jpg" for example. Or "ls somefile/*" to see all the named streams/forks.

Please don't rehash this one. It is fundamentally broken.

Signature · 4 lines
Dr. Horst H. von Brand                   User #22616 counter.li.org
Departamento de Informatica                     Fono: +56 32 654431
Universidad Tecnica Federico Santa Maria              +56 32 654239
Casilla 110-V, Valparaiso, Chile                Fax:  +56 32 797513
#35 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Alan Cox
Quoted 6 lines · Alan Cox
Quoted 3 lines · Hans Reiser
Quoted 3 lines

Just one of many applications. Watch Joe-user save their word processing file sometime, they'll use spaces, quotes, etc.

Quoted 9 lines
Quoted 3 lines

With great unhappiness they will.

Its only problematic for the command line users. The GUI doesn't have some mysterious notion of meta-characters, it provides out of band information on boundaries.

Forgive me, what is out of band information on boundaries?

Most people I know don't use the GUI for executing commands, perhaps this is because the existing guis are not good enough yet.

Quoted 12 lines
Quoted 5 lines

This is why I just want to be left alone to tinker with reiser4. It is faster than other filesystems. People should assume I know what I am doing, and leave me to tinker in my little fs. 5 years later others will follow, or not, I don't care.

See I don't care if you tinker with reiser4. I don't care if it turns out to be a crap fs or a great fs. If its a great fs and scales and unlike reiser3 can recover well from disk errors then one year I might even use it.

Is there a technical basis for your claim that we have trouble with disk errors?

Do you mean badblocks support or what?

Quoted 11 lines

I do care if you ask me to suffer core API changes for your research, that in your economics world is an externality. Its a large negative externality on the part of the userbase so the userbase objects. It doesn't take a PhD in economics to understand this.

Alan

I think it would be reasonable for people to say that our approach currently has bugs, we should turn metafiles off until we make the bugs go away.

#36 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

Quoted 4 lines · Hans Reiser

Is there a technical basis for your claim that we have trouble with disk errors?

Do you mean badblocks support or what?

I mean probability of gettng your data back after a disk loses data. And the technical basis for my claims is twofold - painful experience is one, and shooting random blocks of zeros onto a disk and run fsck tools is another.

Quoted 3 lines

I think it would be reasonable for people to say that our approach currently has bugs, we should turn metafiles off until we make the bugs go away.

Well reiserfs4 is a lot more than metafiles and new vfs layer concepts. Clearly those parts of the the fs that don't require core fs changes belong in the kernel as soon as everyone is happy they are clean enough and look correct.

Metafiles and openat() are an argument we can all have later.

#39 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to Hans Reiser
Quoted 11 lines · Hans Reiser

Putting \ into filenames makes windows compatibility less trivial. Putting | into filenames seems like asking for trouble with shells. Asking users to keep track of multiple levels of escapes imposed by shells and such is hard on them.

If you think \| is user friendly, oh god, people like you are the reason why Unix is hated by many.

Having to explain filename/metas/owner or filename/.../owner or filename/..metas/owner (I don't deeply care what string is used in place of "metas") is hard enough.

[Sorry if the tone of this e-mail is particularly heated. I used to use reiserfs and I am really looking towards the insertion of the final version of reiser4 in the linux kernel, but I wish to publish my 2 cents as user (not as kernel developer). TIA for your attention and patience in advance...]

The idea of using "metas" (instead of ".metas" and "..metas") however is ATROCIOUS, as it would mean to have a fine "metas" entry popping up in every dir, be undeletable, and having some random user pissed at his random ${OS} because "it puts stuff that does not belong in the system"...

The main idea about metadata is that you should be able to use metadata only if you care. And if you use something, THEN you need to be bugged by its presence but in a way that does not bug everyone's else... 99% of people out there only care to use metadatas only via those kind of applications that _really_ support it, like Mp3 players, Torrentlike apps and CVS-like programs (if they do and really care to have their metadatas consistent with the reality, that is [1]).

It has no sense for a human being to go and manage thousands of metadata directories by hand, hence having "metas" right there in the filesystem is not calling for user-friendlyness... it is only desperately calling for some n00b# to dd inside it "just for fun" or "just to make it go away". Putting in the spotlight a potentially unreliable OR fragile source of informations is only asking for trouble and is also harrassing for the end-user that couldn't care less about that "metas" stuff. (what is this metas stuff that is between my mariah999.jpg and morena001.jpg in my pr0n/ folder? [2])

In the end, using a UNIX-flavoured system matters because things are shaped to help sysadmin's jobs, and sysadmins are asked for their task to Read The Finely-written (?) Manuals before inserting the root password at the login prompts.

The end-users are to be protected from themselves until they are really ready to read the documentation.

In conclusion: I don't know WAFL, but I won't make Clearcase example a too forward example... Clearcase naming is useful only inside a Clearcase-enabled environment and using Clearcase-enabled tools... I don't think that people at IBM are going to flame me or handle me a DMCA takedown notice if I put on my filesystem filenames that could possibly conflict with their system. Instead Reiserfs4 is going to ask me to sacrifice a viable word for my filenames (no, I don't call my filenames .something), and to create a workaround for my scripts [3] while an alternative like .metas or a ..metas would make even more sense and be more streamlined with Linux/UNIX customs.

Just my 2 cents as random n00b.

Ciao :)

Signature · 5 lines
Davide Inglima

[1] http://www.well.com/~doctorow/metacrap.htm
[2] whoops...
[3] ls -1 | wc -l
#43 lore raw

Re: silent semantic changes in reiser4 (brief attempt to document the idea ofwhat reiser4 wants to do with metafiles and why

In reply to a message not in the archive
Quoted 5 lines · Elladan
Quoted 5 lines

What's wrong with using / as the separator? It is already used to separate components of pathnames. Named streams are very much like files in a subdirectory.

Quoted 9 lines
Quoted 5 lines

This scheme makes for very little change to existing tools, users may then do a "gimp somefile/icon.jpg" for example. Or "ls somefile/*" to see all the named streams/forks.

Directories may have metadata as well.

They can. That doesn't stand in the way of using "/" to separate the named stream's name from the file (or directory) that have the attribute. "Directories may have metadata" pops up now and then, and the solution is so blindlingly obvious that nobody sees it.

A file-as-dir can be implemented as a rather normal directory attached to the file's name. The stuff inside may be interpreted as "attributes" or as something more file-like, such as the often mentioned thumbnails.

What about a directory then? It _is_ a directory, so it support named streams already. They are usually called "files". :-) So, if you really want a thumb for your directory, just store a thumb.jpg in it.

Helge Hafting