FS: hardlinks on directories

Hello all,

although it is very likely I am entering (again :-) an ancient discussion I would like to ask why hardlinks on directories are not allowed/no supported fs action these days. I can't think of a good reason for this, but can think of many good reasons why one would like to have such a function, amongst those:

- restructuring of the fs to meet different sorting criterias (kind of a database view onto the fs)
- relinking of the fs for export to other hosts via nfs or the like (enhanced security through artificially constructed, exported trees)

Would a feature like that be seen as "big action" or rather small enhancement to the fs-layer?
Are there any supporters for this feature out there?

Regards,
Stephan

Hard links on directories are hard to do in the UNIX file system model.

Where ".." should point? How to implement rmdir? You can think about UNIX unlink as some form of reference counter based garbage collector---when last (persistent) reference to the object is removed it is safe to recycle it. It is well-known that reference counting GC cannot cope with cyclical references. Usually this is not a problem for the file system because all cyclical references are very well known---they always involve "." and "..". But as one allows hard links on directories, file system is no longer tree, but generic directed graph and reference counting GC wouldn't work.

although it is very likely I am entering (again :-) an ancient discussion I would like to ask why hardlinks on directories are not allowed/no supported fs action these days. I can't think of a good reason for this, but can think of many good reasons why one would like to have such a function, amongst those:

Quoted 5 lines · Stephan von Krawczynski

although it is very likely I am entering (again :-) an ancient discussion I would like to ask why hardlinks on directories are not allowed/no supported fs action these days. I can't think of a good reason for this, but can think of many good reasons why one would like to have such a function, amongst those:

I don't know the exact reasons it isn't allowed, but you can always use "mount --bind" to get a similar effect.

Quoted 10 lines · On Mon, 04 Aug 2003 14:45:58 +0200 [email protected] (Måns Rullgård)…

Quoted 5 lines · Stephan von Krawczynski

although it is very likely I am entering (again :-) an ancient discussion I would like to ask why hardlinks on directories are not allowed/no supported fs action these days. I can't think of a good reason for this, but can think of many good reasons why one would like to have such a function, amongst those:

I don't know the exact reasons it isn't allowed, but you can always use "mount --bind" to get a similar effect.

I guess this is not really an option if talking about hundreds or thousands of
"links", is it?

Regards,
Stephan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Hi !

the fundamental problem as i know it is that the FS design in unix is based on a directory TREE structure - however if you implement hard links for directories you are breaking this strict treeu structure and can end up with loops/graphs.

(What are the cases where a symlink wouldn't be enough ?)

Quoted 2 lines · On Mon, 4 Aug 2003 15:37:01 +0200 Christian Reichert…

Hi !

Quoted 1 line

the fundamental problem as i know it is that the FS design in unix is

based

Quoted 2 lines

on a directory TREE structure - however if you implement hard links for directories you are breaking this strict treeu structure and can end up

with

Quoted 1 line

loops/graphs.

This is just the same with softlinks, or not?

I guess this is not really an option if talking about hundreds or thousands of "links", is it?

From: Alexander Viro <http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&safe=off&q=author:viro%40math.psu.edu+> ([email protected] <mailto:viro%40math.psu.edu>)
Subject: Re: hundreds of mount --bind mountpoints?

Quoted 8 lines · David L. Parsley

Hi,

I'm still working on a packaging system for diskless (quasi-embedded) devices. The root filesystem is all tmpfs, and I attach packages inside it. Since symlinks in a tmpfs filesystem cost 4k each (ouch!), I'm considering using mount --bind for everything. This appears to use very little memory, but I'm wondering if I'll run into problems when I start having many hundreds of bind mountings. Any feel for this?

Memory use is sizeof(struct vfsmount) per binding. In principle, you can get in trouble when size of /proc/mount will get past 4Kb - you'll get only first 4 (actually 3, IIRC) kilobytes, so stuff that relies on the contents of said file may get unhappy. It's fixable, though.

Quoted 5 lines · Stephan von Krawczynski

I guess this is not really an option if talking about hundreds or thousands of"links", is it?

actually hundreds or thounds still should be ok. See...

Quoted 7 lines · On Mon, 04 Aug 2003 11:31:47 -0400 Jeff Muizelaar <[email protected]> wrote:

Quoted 5 lines · Stephan von Krawczynski

I guess this is not really an option if talking about hundreds or thousands of"links", is it?

Quoted 1 line

actually hundreds or thounds still should be ok. See...

Hm, and I just found out that re-exporting "mount --bind" volumes does not work over nfs...

Is this correct, Neil?

Quoted 12 lines · On Monday August 4, [email protected] wrote:

Quoted 8 lines · On Mon, 04 Aug 2003 11:31:47 -0400 Jeff Muizelaar <[email protected]> wrote:

Quoted 5 lines · Stephan von Krawczynski

I guess this is not really an option if talking about hundreds or thousands of"links", is it?

actually hundreds or thounds still should be ok. See...

Quoted 4 lines

Hm, and I just found out that re-exporting "mount --bind" volumes does not work over nfs...

Is this correct, Neil?

Yes, though there is a reasonable chance that it can be made to work with linux-2.6.0 and nfs-utils-1.1.0 (neither of which have been released yet:-)

Quoted 8 lines

Quoted 4 lines

Hm, and I just found out that re-exporting "mount --bind" volumes does not work over nfs...

Is this correct, Neil?

Yes, though there is a reasonable chance that it can be made to work with linux-2.6.0 and nfs-utils-1.1.0 (neither of which have been released yet:-)

Is this a complex issue? Can you imagine a not-too-big sized patch can make it work in 2.4? What is the basic reason it does in fact not work?

Quoted 11 lines · On Tuesday August 5, [email protected] wrote:

Quoted 8 lines

Quoted 4 lines

Hm, and I just found out that re-exporting "mount --bind" volumes does not work over nfs...

Is this correct, Neil?

Yes, though there is a reasonable chance that it can be made to work with linux-2.6.0 and nfs-utils-1.1.0 (neither of which have been released yet:-)

Is this a complex issue? Can you imagine a not-too-big sized patch can make it work in 2.4? What is the basic reason it does in fact not work?

On reflection, it could probably work in 2.4 and current nfs-utils, but admin might be a bit clumsy.

To allow knfsd to see a mountpoint, you have to export the mounted directory with the "nohide" option. Currently "nohide" only works properly for exports to specific hosts, not to wildcarded hosts or netgroups.
So if your /etc/export contains:

/path/to/some/--bind/mountpoint servername(nohide,....)

for every mountpoint and every server, then it should work.

Quoted 22 lines · On Wed, 6 Aug 2003 11:12:38 +1000 Neil Brown <[email protected]> wrote:

Quoted 11 lines · On Tuesday August 5, [email protected] wrote:

Quoted 8 lines

Quoted 4 lines

Hm, and I just found out that re-exporting "mount --bind" volumes does not work over nfs...

Is this correct, Neil?

Yes, though there is a reasonable chance that it can be made to work with linux-2.6.0 and nfs-utils-1.1.0 (neither of which have been released yet:-)

Is this a complex issue? Can you imagine a not-too-big sized patch can make it work in 2.4? What is the basic reason it does in fact not work?

On reflection, it could probably work in 2.4 and current nfs-utils, but admin might be a bit clumsy.

To allow knfsd to see a mountpoint, you have to export the mounted directory with the "nohide" option. Currently "nohide" only works properly for exports to specific hosts, not to wildcarded hosts or netgroups.
So if your /etc/export contains:

Quoted 3 lines

/path/to/some/--bind/mountpoint servername(nohide,....)

for every mountpoint and every server, then it should work.

Hm, bad luck. I tried and it did not work. I used 2.4.20 kernel, are there chances a later kernel might work?

although it is very likely I am entering (again :-) an ancient discussion I would like to ask why hardlinks on directories are not allowed/no supported fs action these days.

Quoted 3 lines · Stephan von Krawczynski

although it is very likely I am entering (again :-) an ancient discussion I would like to ask why hardlinks on directories are not allowed/no supported fs action these days.

Quite a lot of software thinks that the file hierarchy is a tree, if you wish a forest.

Things would break badly if the hierarchy became an arbitrary graph.

Quoted 8 lines · On Mon, 4 Aug 2003 15:44:15 +0200 Andries Brouwer <[email protected]> wrote:

Quoted 3 lines · Stephan von Krawczynski

although it is very likely I am entering (again :-) an ancient discussion I would like to ask why hardlinks on directories are not allowed/no supported fs action these days.

Quite a lot of software thinks that the file hierarchy is a tree, if you wish a forest.

Quoted 1 line

Things would break badly if the hierarchy became an arbitrary graph.

Care to name one? What exactly is the rule you see broken? Sure you can build loops, but you cannot prevent people from doing braindamaged things to their data anyway. You would not ban dd either for being able to flatten your harddisk only because of one mistyping char...
Every feature can be misused and then damaging, but that is no real reason not to have it - IMHO.

For a start the kernel VFS dcache would break because you end up with multiple entries for each inode, one entry for each parallel directory tree. Read-only you are just about able to get away with it (been there, done that, don't recommend it!) but allow files to be deleted and it will blow up in your face.

You ask for examples of applications? There are millions! Anything that walks the directory tree for a start, e.g. ls -R, find, locatedb, medusa, du, any type of search and/or indexing engine, chown -R, cp -R, scp -R, chmod -R, etc...

Quoted 4 lines · On Mon, 4 Aug 2003 15:04:28 +0100 (BST) Anton Altaparmakov <[email protected]>…

You ask for examples of applications? There are millions! Anything that walks the directory tree for a start, e.g. ls -R, find, locatedb, medusa, du, any type of search and/or indexing engine, chown -R, cp -R, scp -R, chmod -R, etc...

There is a flaw in this argument. If I am told that mount --bind does just about what I want to have as a feature then these applictions must have the same problems already (if I mount braindead). So an implementation in fs cannot do any _additional_ damage to these applications, or not?

On Mon, 4 Aug 2003 15:04:28 +0100 (BST)

Quoted 5 lines · Anton Altaparmakov

For a start the kernel VFS dcache would break because you end up with multiple entries for each inode, one entry for each parallel directory tree. Read-only you are just about able to get away with it (been there, done that, don't recommend it!) but allow files to be deleted and it will blow up in your face.

I cannot comment, I have no inside knowledge of it.

Quoted 4 lines

You ask for examples of applications? There are millions! Anything that walks the directory tree for a start, e.g. ls -R, find, locatedb, medusa, du, any type of search and/or indexing engine, chown -R, cp -R, scp -R, chmod -R, etc...

There is a flaw in this argument. If I am told that mount --bind does just about what I want to have as a feature then these applictions must have the same problems already (if I mount braindead). So an implementation in fs cannot do any _additional_ damage to these applications, or not?

My saying is not "I want to have hardlinks for creating a big mess of loops inside my filesystems". Your view simply drops the fact that there are more possibilities to create and use hardlinks without any loops...

Quoted 5 lines

Quoted 4 lines

You ask for examples of applications? There are millions! Anything that walks the directory tree for a start, e.g. ls -R, find, locatedb, medusa, du, any type of search and/or indexing engine, chown -R, cp -R, scp -R, chmod -R, etc...

Quoted 4 lines

There is a flaw in this argument. If I am told that mount --bind does just about what I want to have as a feature then these applictions must have the same problems already (if I mount braindead). So an implementation in fs cannot do any _additional_ damage to these applications, or not?

Mount -bind only modifies the transient memory storage of a directory. It doesn't change the filesystem. Each bind occupies memory, and on a reboot, the bind is gone.

Quoted 3 lines

My saying is not "I want to have hardlinks for creating a big mess of loops inside my filesystems". Your view simply drops the fact that there are more possibilities to create and use hardlinks without any loops...

been there done that, is is a "big mess of loops".

And you can't prevent the loops either, without scanning the entire graph, or keeping a graph location reference embeded with the file.

Which then breaks "mv" for renaming directories... It would then have to scan the entire graph again to locate a possble creation of a loop, and regenerate the graph location for every file.

On Mon, 4 Aug 2003 16:16:39 -0500

Quoted 11 lines · Jesse Pollard

Quoted 10 lines

Quoted 4 lines

You ask for examples of applications? There are millions! Anything that walks the directory tree for a start, e.g. ls -R, find, locatedb, medusa, du, any type of search and/or indexing engine, chown -R, cp -R, scp -R, chmod -R, etc...

There is a flaw in this argument. If I am told that mount --bind does just about what I want to have as a feature then these applictions must have the same problems already (if I mount braindead). So an implementation in fs cannot do any _additional_ damage to these applications, or not?

Quoted 3 lines

Mount -bind only modifies the transient memory storage of a directory. It doesn't change the filesystem. Each bind occupies memory, and on a reboot, the bind is gone.

What kind of an argument is this? What difference can you see between a transient loop and a permanent loop for the applications? Exactly zero I guess. In my environments nil boots ought to happen.

This is the reason why I would in fact be satisfied with mount -bind if only I could export it via nfs...

Quoted 10 lines

Quoted 5 lines

My saying is not "I want to have hardlinks for creating a big mess of loops inside my filesystems". Your view simply drops the fact that there are more possibilities to create and use hardlinks without any loops...

been there done that, is is a "big mess of loops".

And you can't prevent the loops either, without scanning the entire graph, or keeping a graph location reference embeded with the file.

Or marking the links as type links somehow.

Quoted 3 lines

Which then breaks "mv" for renaming directories... It would then have to scan the entire graph again to locate a possble creation of a loop, and regenerate the graph location for every file.

There should be no difference if only a hardlink is simply marked as such by any kind of marker you possibly can think of.

Quoted 2 lines

This is the reason why I would in fact be satisfied with mount -bind if only I could export it via nfs...

it's a MOUNT point. NFS doesn't export across mount points just as it doesn't allow exporting a NFS mounted directory.

There is a flaw in this argument. If I am told that mount --bind does just about what I want to have as a feature then these applictions must have the same problems already (if I mount braindead). So an implementation in fs cannot do any _additional_ damage to these applications, or not?

Quoted 5 lines · Stephan von Krawczynski

There is a flaw in this argument. If I am told that mount --bind does just about what I want to have as a feature then these applictions must have the same problems already (if I mount braindead). So an implementation in fs cannot do any _additional_ damage to these applications, or not?

There is a flaw in this flaw. :-)

/tmp# mkdir a
/tmp# mkdir a/b
/tmp# mkdir a/c
/tmp# mount --bind a a/b
/tmp# ls a
b c
/tmp# ls a/b
b c
/tmp# ls a/b/b/
/tmp#

It is enlightening in this regard to consider the difference between using unix /etc/fstab and Hurd translators to manage your namespace.

In preparing this example, I discovered that find and ls -R already have hard-link cycle "protection" built in, so they are broken in the presence of bind mounts. :-(

Quoted 4 lines · On Mon, 4 Aug 2003 18:58:19 -0400 Andrew Pimlott <[email protected]> wrote:

In preparing this example, I discovered that find and ls -R already have hard-link cycle "protection" built in, so they are broken in the presence of bind mounts. :-(

Ok, so now we are at: application programmer expected hardlinks to exist, but fs programmer says they won't because they break existing applications.

Now the discussion gets real interesting ;-)

On Tue, Aug 05, 2003 at 02:19:38AM +0200, Stephan von Krawczynski wrote:

Wouldn't bind mounts solve your problem? Why are you still interested in hard links?

Quoted 2 lines

Wouldn't bind mounts solve your problem? Why are you still interested in hard links?

Because mount -bind cannot be exported over nfs (to my current knowledge and testing). Else I would be very pleased to use it.

On Mon, 4 Aug 2003 15:44:15 +0200

Quoted 8 lines · Andries Brouwer

Quoted 3 lines · Stephan von Krawczynski

although it is very likely I am entering (again :-) an ancient discussion I would like to ask why hardlinks on directories are not allowed/no supported fs action these days.

Quite a lot of software thinks that the file hierarchy is a tree, if you wish a forest.

Quoted 1 line

Things would break badly if the hierarchy became an arbitrary graph.

Care to name one? What exactly is the rule you see broken? Sure you can build loops, but you cannot prevent people from doing braindamaged things to their data anyway. You would not ban dd either for being able to flatten your harddisk only because of one mistyping char...
Every feature can be misused and then damaging, but that is no real reason not to have it - IMHO.

Find for one. Any application that must scan the tree in a search. Any application that must backup every file for another (I know, dump bypasses the filesystem to make backups, tar doesn't).

It introduces too many unique problems to be easily handled. That is why symbolic links actually work. Symbolic links are not hard links, therefore they are not processed as part of the tree. and do not cause loops.

It was also done in one of the "popular" code management systems under unix. (it allowed a "mount" of the system root to be under the CVS repository to detect unauthorized modifications...). Unfortunately, the system could not be backed up anymore. 1. A dump of the CVS filesystem turned into a dump of the entire system... 2. You could not restore the backups... The dumps failed (bru at the time) because the pathnames got too long, the restore failed since it ran out of disk space due to the multiple copies of the tree being created.

The KIS principle is the key. A graph is NOT simple to maintain.

Quoted 3 lines · On Mon, 4 Aug 2003 09:33:44 -0500 Jesse Pollard <[email protected]> wrote:

Find for one. Any application that must scan the tree in a search. Any application that must backup every file for another (I know, dump bypasses the filesystem to make backups, tar doesn't).

All that can handle symlinks already have the same problem nowadays. Where is the difference? And yet again: it is no _must_ for the feature to use it for creating complete loops inside your fs.
You _can_ as well dd if=/dev/zero of=/dev/hda, but of course you shouldn't. Have you therefore deleted dd from your bin ?

Quoted 3 lines

It introduces too many unique problems to be easily handled. That is why symbolic links actually work. Symbolic links are not hard links, therefore they are not processed as part of the tree. and do not cause loops.

tar --dereference loops on symlinks _today_, to name an example. All you have to do is to provide a way to find out if a directory is a hardlink, nothing more. And that should be easy.

Quoted 13 lines · Stephan von Krawczynski

On Mon, 4 Aug 2003 09:33:44 -0500

Quoted 3 lines · Jesse Pollard

Find for one. Any application that must scan the tree in a search. Any application that must backup every file for another (I know, dump bypasses the filesystem to make backups, tar doesn't).

All that can handle symlinks already have the same problem nowadays. Where is the difference? And yet again: it is no _must_ for the feature to use it for creating complete loops inside your fs.
You _can_ as well dd if=/dev/zero of=/dev/hda, but of course you shouldn't. Have you therefore deleted dd from your bin ?

Quoted 8 lines

Quoted 4 lines

It introduces too many unique problems to be easily handled. That is why symbolic links actually work. Symbolic links are not hard links, therefore they are not processed as part of the tree. and do not cause loops.

tar --dereference loops on symlinks _today_, to name an example. All you have to do is to provide a way to find out if a directory is a hardlink, nothing more. And that should be easy.

[SNIPPED...]

Reading Denis Howe's Free Online Dictionary of Computing;
http://burks.bton.ac.uk/burks/foldoc/55/51.html, we see that the chief reason for no directory hard-links is that rm and rmdir won't allow you to delete them. There is no POSIX requirement for eliminating them, and it is possible "Some systems provide link and unlink commands which give direct access to the system calls of the same name, for which no such restrictions apply."

Perhaps Linux does support hard-links to directories?

mkdir("foo", 0644) = 0
link("foo", "bar") = -1 EPERM (Operation not
permitted) _exit(0) = ?

Nah... No such luck. I'll bet it's artificial. I think you could remove that S_IFDIR check and get away with it!

All that can handle symlinks already have the same problem nowadays. Where is the difference?

Quoted 5 lines

The KIS principle is the key. A graph is NOT simple to maintain.

This is true. But I am very willing to believe reiserfs is not simple either, still it is there ;-)

Regards,
Stephan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

If you want hard linked directories, send us a patch for v4. Should
be VERY easy to write. If there is some reason it is not simple, let
me know. Discuss it with Vitaly though, it might affect fsck.

If you want hard linked directories, send us a patch for v4. Should be
VERY easy to write. If there is some reason it is not simple, let me
know. Discuss it with Vitaly though, it might affect fsck.

tar --dereference loops on symlinks _today_, to name an example. All you have to do is to provide a way to find out if a directory is a hardlink, nothing more. And that should be easy.

On Mon, 4 Aug 2003 09:33:44 -0500

Quoted 3 lines · Jesse Pollard

Find for one. Any application that must scan the tree in a search. Any application that must backup every file for another (I know, dump bypasses the filesystem to make backups, tar doesn't).

All that can handle symlinks already have the same problem nowadays. Where is the difference? And yet again: it is no _must_ for the feature to use it for creating complete loops inside your fs.
You _can_ as well dd if=/dev/zero of=/dev/hda, but of course you shouldn't. Have you therefore deleted dd from your bin ?

Quoted 5 lines

It introduces too many unique problems to be easily handled. That is why symbolic links actually work. Symbolic links are not hard links, therefore they are not processed as part of the tree. and do not cause loops.

tar --dereference loops on symlinks _today_, to name an example. All you have to do is to provide a way to find out if a directory is a hardlink, nothing more. And that should be easy.

Quoted 9 lines

It was also done in one of the "popular" code management systems under unix. (it allowed a "mount" of the system root to be under the CVS repository to detect unauthorized modifications...). Unfortunately, the system could not be backed up anymore. 1. A dump of the CVS filesystem turned into a dump of the entire system... 2. You could not restore the backups... The dumps failed (bru at the time) because the pathnames got too long, the restore failed since it ran out of disk space due to the multiple copies of the tree being created.

And they never heard of "--exclude" in tar, did they?

Quoted 1 line

The KIS principle is the key. A graph is NOT simple to maintain.

This is true. But I am very willing to believe reiserfs is not simple either, still it is there ;-)

Quoted 9 lines

Quoted 8 lines

It was also done in one of the "popular" code management systems under unix. (it allowed a "mount" of the system root to be under the CVS repository to detect unauthorized modifications...). Unfortunately, the system could not be backed up anymore. 1. A dump of the CVS filesystem turned into a dump of the entire system... 2. You could not restore the backups... The dumps failed (bru at the time) because the pathnames got too long, the restore failed since it ran out of disk space due to the multiple copies of the tree being created.

Quoted 1 line

And they never heard of "--exclude" in tar, did they?

Doesn't work. Remember - you have to --exclude EVERY possible loop. And unless you know ahead of time, you can't exclude it. The only way we found to reliably do the backup was to dismount the CVS.

Quoted 3 lines

tar --dereference loops on symlinks _today_, to name an example. All you have to do is to provide a way to find out if a directory is a hardlink, nothing more. And that should be easy.

Yup - because a symlink is NOT a hard link. it is a file.

If you use hard links then there is no way to determine which is the "proper" link.

Quoted 11 lines

Quoted 10 lines

It was also done in one of the "popular" code management systems under unix. (it allowed a "mount" of the system root to be under the CVS repository to detect unauthorized modifications...). Unfortunately, the system could not be backed up anymore. 1. A dump of the CVS filesystem turned into a dump of the entire system... 2. You could not restore the backups... The dumps failed (bru at the time) because the pathnames got too long, the restore failed since it ran out of disk space due to the multiple copies of the tree being created.

Quoted 1 line

And they never heard of "--exclude" in tar, did they?

Doesn't work. Remember - you have to --exclude EVERY possible loop. And unless you know ahead of time, you can't exclude it. The only way we found to reliably do the backup was to dismount the CVS.

And in just the same way fs could provide a mode bit saying "hi, I am a hardlink", and tar can then easily provide option number 1345 saying "stay away from hardlinks".

Quoted 3 lines · Stephan von Krawczynski

And in just the same way fs could provide a mode bit saying "hi, I am a hardlink", and tar can then easily provide option number 1345 saying "stay away from hardlinks".

Perhaps not a bit, but rather another enumerated value in the type field of an inode. (Are there any values left?)

Ok, lets consider this. Suppose that /a/b and /a/c both refer to the same directory, where /a/b is a traditional link, but /a/c is a "hardlink".

What happens when one executes 'rmdir /a/b'? Does the directory it references disappear? If not, how would tar ever find it? (I have this vision of a disk full of these hardlink-only directories which tar and perhaps fsck cannot find.)

The setup you describe is exactly the reason why I suggested elsewhere (in a private discussion) to single-link all directory entries pointing to the same directory in a list. In case of deletion of the "main" entry, the "main" simply can walk on to the next (former) hardlink, if there are any left the tree is deleted completely. That's it.

Quoted 6 lines · Stephan von Krawczynski

The setup you describe is exactly the reason why I suggested elsewhere (in a private discussion) to single-link all directory entries pointing to the same directory in a list. In case of deletion of the "main" entry, the "main" simply can walk on to the next (former) hardlink, if there are any left the tree is deleted completely. That's it.

Amiga-FFS challenged? ;-)

Quoted 4 lines · On Mon, 4 Aug 2003 16:09:28 -0500 Jesse Pollard <[email protected]> wrote:

Quoted 3 lines

tar --dereference loops on symlinks _today_, to name an example. All you have to do is to provide a way to find out if a directory is a hardlink, nothing more. And that should be easy.

Quoted 4 lines

Yup - because a symlink is NOT a hard link. it is a file.

If you use hard links then there is no way to determine which is the "proper" link.

Where does it say that an fs is not allowed to give you this information in some way?

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

It was also done in one of the "popular" code management systems under unix. (it allowed a "mount" of the system root to be under the CVS repository to detect unauthorized modifications...). Unfortunately, the system could not be backed up anymore. 1. A dump of the CVS filesystem turned into a dump of the entire system... 2. You could not restore the backups... The dumps failed (bru at the time) because the pathnames got too long, the restore failed since it ran out of disk space due to the multiple copies of the tree being created.

Quoted 4 lines

Quoted 1 line

And they never heard of "--exclude" in tar, did they?

Doesn't work. Remember - you have to --exclude EVERY possible loop. And unless you know ahead of time, you can't exclude it. The only way we found to reliably do the backup was to dismount the CVS.

And if you completely run out of ideas in your wild-mounts-throughout-the-tree problem you should simply use "tar -l".

And in just the same way fs could provide a mode bit saying "hi, I am a hardlink", and tar can then easily provide option number 1345 saying "stay away from hardlinks".

tar --dereference loops on symlinks _today_, to name an example. All you have to do is to provide a way to find out if a directory is a hardlink, nothing more. And that should be easy.

Quoted 1 line

Things would break badly if the hierarchy became an arbitrary graph.

Care to name one? What exactly is the rule you see broken? Sure you can build loops,

Even more fun is when you have a directory loop like this:

mkdir A
cd A
mkdir B
cd B
make hard link C back to A

cd ../..
rmdir A

You now removed A from your home directory, but the directory itself did not disappear because it had another hard link from C in B.

Both sound solvable.

We are waiting for your first patch.

Quoted 8 lines · On Tue, 05 Aug 2003 14:51:46 +0200 Helge Hafting <[email protected]> wrote:

Even more fun is when you have a directory loop like this:

mkdir A
cd A
mkdir B
cd B
make hard link C back to A

Quoted 6 lines

cd ../..
rmdir A

You now removed A from your home directory, but the directory itself did not disappear because it had another hard link from C in B.

How about a truly simple idea:

rmdir A says "directory in use" and is rejected

Which means you simply cannot remove the first directory entry before not all other links to it are removed. This implies only two things:
1) you have to know who was first.
2) you have to be able to find out where the links are.

Both sound solvable.

Regards,
Stephan

A hard-link is, by definition, indistinguishable from the original entry. In fact, with fast machines and the course granularity of file-system times, even the creation time may be exactly the same.

Quoted 4 lines · On Tue, 5 Aug 2003 09:36:37 -0400 (EDT) "Richard B. Johnson"…

A hard-link is, by definition, indistinguishable from the original entry. In fact, with fast machines and the course granularity of file-system times, even the creation time may be exactly the same.

Hello Richard,

I really don't mind if you call the thing I am looking for a hardlink or a chicken. And I am really not sticking to creating them by ln or mount or just about anything else. I am, too, not bound to making them permanent on the media. All I really want to do is to _export_ them via nfs.
And guys, looking at mount -bind makes me think someone else (before poor me) needed just about the same thing.
So, instead of constantly feeding my bad conscience, can some kind soul explain the possibilities to make "mount -bind/rbind" work over a network fs of some flavor, please?

Regards,
Stephan

PS: if you ever want to find out what *nix people are carrying guns, just enter the room and cry out loud "directory hardlinks to the left!"
;-)

But symlinks work over NFS. You just have to make sure they are relative to whatever the remote mount-point is:

On Tue, 5 Aug 2003 09:36:37 -0400 (EDT)

Quoted 4 lines · Richard B. Johnson

A hard-link is, by definition, indistinguishable from the original entry. In fact, with fast machines and the course granularity of file-system times, even the creation time may be exactly the same.

Hello Richard,

I really don't mind if you call the thing I am looking for a hardlink or a chicken. And I am really not sticking to creating them by ln or mount or just about anything else. I am, too, not bound to making them permanent on the media. All I really want to do is to _export_ them via nfs. And guys, looking at mount -bind makes me think someone else (before poor me) needed just about the same thing.
So, instead of constantly feeding my bad conscience, can some kind soul explain the possibilities to make "mount -bind/rbind" work over a network fs of some flavor, please?

Quoted 3 lines · Stephan von Krawczynski

So, instead of constantly feeding my bad conscience, can some kind soul explain the possibilities to make "mount -bind/rbind" work over a network fs of some flavor, please?

Not sure, but I suspect there would be a problem IF the -bind mount crosses filesystems. If it doesn't cross the filesystems I wouldn't think there would be much problem.

Quoted 1 line

Quoted 1 line

Quoted 1 line

Quoted 1 line

" " == Jesse Pollard <[email protected]> writes:

Somebody mentioned that VAX/VMS would allow hard-linked directories. This is not true. Even files, created with SET FILE ENTER=DUA0:[1.DIR] DUA0:[000000] that seemed like hard links were not hard links. They were sim-links. The file entry contained the directory information of the file being linked, and was not a clone of the header (inode) itself as would be a hard link. Therefore, these were unique and could not cause recursion to fail.

Quoted 11 lines · On Tue, 05 Aug 2003 14:51:46 +0200 Helge Hafting <[email protected]> wrote:

Even more fun is when you have a directory loop like this:

mkdir A
cd A
mkdir B
cd B
make hard link C back to A

cd ../..
rmdir A

Quoted 3 lines

You now removed A from your home directory, but the directory itself did not disappear because it had another hard link from C in B.

How about a truly simple idea:

rmdir A says "directory in use" and is rejected

Which means you simply cannot remove the first directory entry before not all other links to it are removed. This implies only two things:
1) you have to know who was first.

2) you have to be able to find out where the links are.

Both sound solvable.

Quoted 23 lines · Stephan von Krawczynski

Quoted 8 lines · On Tue, 05 Aug 2003 14:51:46 +0200 Helge Hafting <[email protected]> wrote:

Even more fun is when you have a directory loop like this:

mkdir A
cd A
mkdir B
cd B
make hard link C back to A

Quoted 8 lines

cd ../..
rmdir A

You now removed A from your home directory, but the directory itself did not disappear because it had another hard link from C in B.

How about a truly simple idea:

Quoted 4 lines

rmdir A says "directory in use" and is rejected

Then anybody can prevent you from removing your obsolete directories by creating links to them. Existing hard link don't have such problems.

Quoted 1 line · Helge Hafting

So, he needs links that count as references,

links that don't count as
references but disappear if the object disappears (without dangling like symlinks),

and unlinkall(), which removes an object and all of its
links. He needs for the first reference to a directory to be removable only by removing all links to the object, or designating another link to be the "first" reference.

Sounds clean to me. This is not to say that I am funded to write it.;-) I'd look at a patch though.....;-)

I need to write up a taxonomy of links..... after reiser4 ships.....

Quoted 20 lines · Helge Hafting

Quoted 19 lines · Stephan von Krawczynski

Quoted 8 lines · On Tue, 05 Aug 2003 14:51:46 +0200 Helge Hafting <[email protected]> wrote:

Even more fun is when you have a directory loop like this:

mkdir A
cd A
mkdir B
cd B
make hard link C back to A

Quoted 8 lines

cd ../..
rmdir A

You now removed A from your home directory, but the directory itself did not disappear because it had another hard link from C in B.

How about a truly simple idea:

Quoted 9 lines

Quoted 4 lines

rmdir A says "directory in use" and is rejected

Then anybody can prevent you from removing your obsolete directories by creating links to them. Existing hard link don't have such problems.

So, he needs links that count as references, links that don't count as references but disappear if the object disappears (without dangling like symlinks), and unlinkall(), which removes an object and all of its links. He needs for the first reference to a directory to be removable only by removing all links to the object, or designating another link to be the "first" reference.

Sounds clean to me. This is not to say that I am funded to write it.;-) I'd look at a patch though.....;-)

I need to write up a taxonomy of links..... after reiser4 ships.....

So, he needs links that count as references, links that don't count as references but disappear if the object disappears (without dangling like symlinks), and unlinkall(), which removes an object and all of its links. He needs for the first reference to a directory to be removable only by removing all links to the object, or designating another link to be the "first" reference.

Sounds clean to me.

This is not to say that I am funded to write
it.;-) I'd look at a patch though.....;-)

I need to write up a taxonomy of links..... after reiser4 ships.....

Signature · 2 lines

-- 
Hans

Quoted 1 line · Hans Reiser

[...]

Quoted 3 lines

So, he needs links that count as references, links that don't count as references but disappear if the object disappears (without dangling like symlinks), and unlinkall(), which removes an object and all of its links. He needs for the first reference to a directory to be removable only by removing all links to the object, or designating another link to be the "first" reference.

Sounds clean to me.

Will surely continue to be this way until you start implementing. :)

Quoted 4 lines

This is not to say that I am funded to write
it.;-) I'd look at a patch though.....;-)

I need to write up a taxonomy of links..... after reiser4 ships.....

http://www.namesys.com/v4/links-taxonomy.html

Quoted 4 lines

Signature · 2 lines

-- 
Hans

Nikita.

Quoted 1 line

On Mon, 4 Aug 2003 16:09:28 -0500

Quoted 8 lines · Jesse Pollard

Quoted 3 lines

tar --dereference loops on symlinks _today_, to name an example. All you have to do is to provide a way to find out if a directory is a hardlink, nothing more. And that should be easy.

Yup - because a symlink is NOT a hard link. it is a file.

If you use hard links then there is no way to determine which is the "proper" link.

Where does it say that an fs is not allowed to give you this information in some way?

Quoted 10 lines

Quoted 10 lines

Quoted 9 lines

It was also done in one of the "popular" code management systems under unix. (it allowed a "mount" of the system root to be under the CVS repository to detect unauthorized modifications...). Unfortunately, the system could not be backed up anymore. 1. A dump of the CVS filesystem turned into a dump of the entire system... 2. You could not restore the backups... The dumps failed (bru at the time) because the pathnames got too long, the restore failed since it ran out of disk space due to the multiple copies of the tree being created.

Quoted 5 lines

Quoted 1 line

And they never heard of "--exclude" in tar, did they?

Doesn't work. Remember - you have to --exclude EVERY possible loop. And unless you know ahead of time, you can't exclude it. The only way we found to reliably do the backup was to dismount the CVS.

And if you completely run out of ideas in your wild-mounts-throughout-the-tree problem you should simply use "tar -l".

And in just the same way fs could provide a mode bit saying "hi, I am a hardlink", and tar can then easily provide option number 1345 saying "stay away from hardlinks".

If you can do the fs patch, I can do the tar patch.

Quoted 1 line

If you can do the fs patch, I can do the tar patch.

The patch is FAR too extensive. It would have to be a whole new filesystem

Quoted 3 lines · On Tue, 5 Aug 2003 09:12:31 -0500 Jesse Pollard <[email protected]> wrote:

Quoted 1 line

If you can do the fs patch, I can do the tar patch.

The patch is FAR too extensive. It would have to be a whole new filesystem

Fine. This is a clear and straight forward word. Can you explain why I don't see the mount -bind/rbind stuff through nfs? Is this a problem with a cheaper solution?

Regards,
Stephan

-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

Quoted 5 lines · On Mon, 4 Aug 2003 15:44:15 +0200 Andries Brouwer <[email protected]> wrote:

Quite a lot of software thinks that the file hierarchy is a tree, if you wish a forest.

Things would break badly if the hierarchy became an arbitrary graph.

Care to name one? What exactly is the rule you see broken? Sure you can build loops, but you cannot prevent people from doing braindamaged things to their data anyway. You would not ban dd either for being able to flatten your harddisk only because of one mistyping char...

I'm still waking up, but '..' obviously breaks the "no cycle" observations.

It's just that '..' is well known name by utilities as opposed to arbitrary links.

Quoted 2 lines · On Mon, 4 Aug 2003 08:42:09 -0700 (PDT) Brian Pawlowski <[email protected]>…

I'm still waking up, but '..' obviously breaks the "no cycle" observations.

Hear, hear ...

Quoted 2 lines

It's just that '..' is well known name by utilities as opposed to arbitrary links.

Well, that leads only to the point that ".." implementation is just lousy and it should have been done right in the first place. If there is a need for a loop or a hardlink (like "..") all you have to have is a standard way to find out, be it flags or the like, whatever. But taking the filename or anything not applicable to other cases as matching factor was obviously short-sighted.

Regards,
Stephan
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

on the other hand, if you want somebody to implement this stuff for you, you'll have to provide convincing arguments for it, I for example, would be glad if hardlinks where removed from unix altogether ...

Quoted 4 lines · On Mon, 4 Aug 2003 18:16:57 +0200 Herbert Pötzl <[email protected]> wrote:

on the other hand, if you want somebody to implement this stuff for you, you'll have to provide convincing arguments for it, I for example, would be glad if hardlinks where removed from unix altogether ...

Huh, hard stuff!

Explain your solution for a very common problem:

You have a _big_ fileserver, say some SAN or the like with Gigs. Your data on it is organized according to your basic user structure, because it is very handy to have all data from one user altogether in one directory.

if you managed to link all web-data together in one directory and exported that to your webservers and they are hacked, you just blew up all your web-data but nothing more.
This is a remarkable risk reduction.
And now? Name your idea to export only the data needed to the servers that need it. And keep in mind, we are talking of Gigs and tenthousands of users.

You definitely don't want one mount per user per service.

Can you think of a more elegant way to solve such a problem than hardlinking all web in one single webtree, all sql in one single sql tree ... and then export this single tree (with its artificial structure) to the corresponding server?
I am curiously listening...

Regards,
Stephan

Quoted 8 lines · Stephan von Krawczynski

Quoted 4 lines · On Mon, 4 Aug 2003 18:16:57 +0200 Herbert Pötzl <[email protected]> wrote:

on the other hand, if you want somebody to implement this stuff for you, you'll have to provide convincing arguments for it, I for example, would be glad if hardlinks where removed from unix altogether ...

Quoted 8 lines

Huh, hard stuff!

Explain your solution for a very common problem:

You have a _big_ fileserver, say some SAN or the like with Gigs. Your data on it is organized according to your basic user structure, because it is very handy to have all data from one user altogether in one directory.

I already do something like this, although not for thousands of users, but I guess this would scale well ...

consider a storage device (maybe a partition) for each category of data you want to store/provide/serve

/mnt/webspace, /mnt/database, /mnt/email, /mnt/wossname ...

on the access server (where the customer maintains the data) you'll end up mounting all the storage categories, required for access and you do an additional restructuring for each customer (which of course is automated)

/home/customer/charlie is populated with symlinks to /mnt/*/customer/charlie named by category

/home/customer/charlie/webspace -> /mnt/webspace/customer/charlie /home/customer/charlie/email -> /mnt/email/customer/charlie
...

Quoted 24 lines · On Mon, 4 Aug 2003 18:54:35 +0200 Herbert Pötzl <[email protected]> wrote:

Quoted 8 lines · Stephan von Krawczynski

Quoted 4 lines · On Mon, 4 Aug 2003 18:16:57 +0200 Herbert Pötzl <[email protected]> wrote:

on the other hand, if you want somebody to implement this stuff for you, you'll have to provide convincing arguments for it, I for example, would be glad if hardlinks where removed from unix altogether ...

Quoted 8 lines

Huh, hard stuff!

Explain your solution for a very common problem:

You have a _big_ fileserver, say some SAN or the like with Gigs. Your data on it is organized according to your basic user structure, because it is very handy to have all data from one user altogether in one directory.

I already do something like this, although not for thousands of users, but I guess this would scale well ...

consider a storage device (maybe a partition) for each category of data you want to store/provide/serve

Quoted 1 line

/mnt/webspace, /mnt/database, /mnt/email, /mnt/wossname ...

I have one objection: it is the poorest possible implementation in terms of storage usage. You may have TBs free on database and ran full on webspace, fluctuation on email is dramatic. You waste a damn lot of resources with this approach.

I consider it usable but inferior in terms of costs and scalability. One can very well argue you have to waste bucks to avoid hardlinks (which you don't have at this point). Not having hardlinks costs obviously more. In fact a very good commercial argument pro hardlinks.
If you have all your partitions only logically organised on one SAN you have to remount via nfs quite a bit to get things in the right trees. That does not sound like a performance setup. Did you manage to remount "mount --bind" volumes?

and ...

Quoted 11 lines

on the access server (where the customer maintains the data) you'll end up mounting all the storage categories, required for access and you do an additional restructuring for each customer (which of course is automated)

/home/customer/charlie is populated with symlinks to /mnt/*/customer/charlie named by category

/home/customer/charlie/webspace -> /mnt/webspace/customer/charlie /home/customer/charlie/email -> /mnt/email/customer/charlie
...

which basically means on your access server you have to follow symlinks which is yet another security issue in itself.

Basically you avoid a clean hardlink-setup with a symlink-setup rising possible
security drawbacks and a lot of remounting to get things straight... I am not really convinced.

Regards,
Stephan

On Mon, 4 Aug 2003 18:16:57 +0200

Quoted 4 lines · Herbert Pötzl

on the other hand, if you want somebody to implement this stuff for you, you'll have to provide convincing arguments for it, I for example, would be glad if hardlinks where removed from unix altogether ...

Huh, hard stuff!

Explain your solution for a very common problem:

You have a _big_ fileserver, say some SAN or the like with Gigs. Your data on it is organized according to your basic user structure, because it is very handy to have all data from one user altogether in one directory. You have lots of hosts that use parts of the users' data for a wide range of purposes, lets say web, ftp, sql, name one.
If you cannot re-structure and export your data according to the requirements of your external hosts (web-trees to webserver, sql-trees to sql-server, ftp-trees to ftp-server, name-it to cool-server) you will have to export the total user tree to all your (cluster-) nodes. Do you want that? NO! Of course you don't want that in times of hacked webservers and uncontrollable sql-servers. If anything blows up you likely loose all data at once. On the other hand, if you managed to link all web-data together in one directory and exported that to your webservers and they are hacked, you just blew up all your web-data but nothing more. This is a remarkable risk reduction.

And now? Name your idea to export only the data needed to the servers that need it. And keep in mind, we are talking of Gigs and tenthousands of users. You definitely don't want one mount per user per service. Can you think of a more elegant way to solve such a problem than hardlinking all web in one single webtree, all sql in one single sql tree ... and then export this single tree (with its artificial structure) to the corresponding server?
I am curiously listening...

Quoted 1 line · Stephan von Krawczynski

On Mon, 4 Aug 2003 18:16:57 +0200

[...]
Don't do that. It is too insecure.

1. the structure you describe is FRAGILE. Just adding one more entry could/would break the entire structure.

2. If you mix security structures like this you WILL get a problem.

What you do is copy the declassified data to a nonsecure area (also known as released data). This way the user can modify internal cata without causing the web server potentially catastrophic releases.

Same with the SQL. Do not attmept to mix sensitive and nonsensitive data this way.

If you web server got hacked, how do you prevent the hack from ADDING more links? Or adding SQL injections to other applications...

Quoted 4 lines · On Mon, 4 Aug 2003 18:16:57 +0200 Herbert P�tzl <[email protected]> wrote:

on the other hand, if you want somebody to implement this stuff for you, you'll have to provide convincing arguments for it, I for example, would be glad if hardlinks where removed from unix altogether ...

Huh, hard stuff!

Explain your solution for a very common problem:

You have a _big_ fileserver, say some SAN or the like with Gigs. Your data on it is organized according to your basic user structure, because it is very handy to have all data from one user altogether in one directory. You have lots of hosts that use parts of the users' data for a wide range of purposes, lets say web, ftp, sql, name one.
If you cannot re-structure and export your data according to the requirements of your external hosts (web-trees to webserver, sql-trees to sql-server, ftp-trees to ftp-server, name-it to cool-server) you will have to export the total user tree to all your (cluster-) nodes. Do you want that? NO! Of course you don't want that in times of hacked webservers and uncontrollable sql-servers. If anything blows up you likely loose all data at once. On the other hand, if you managed to link all web-data together in one directory and exported that to your webservers and they are hacked, you just blew up all your web-data but nothing more. This is a remarkable risk reduction.

And now? Name your idea to export only the data needed to the servers that need it. And keep in mind, we are talking of Gigs and tenthousands of users. You definitely don't want one mount per user per service.
Can you think of a more elegant way to solve such a problem than hardlinking all web in one single webtree, all sql in one single sql tree ... and then export this single tree (with its artificial structure) to the corresponding server?
I am curiously listening...

On Mon, 4 Aug 2003 08:42:09 -0700 (PDT)

Quoted 2 lines · Brian Pawlowski

I'm still waking up, but '..' obviously breaks the "no cycle" observations.

Hear, hear ...

Quoted 2 lines

It's just that '..' is well known name by utilities as opposed to arbitrary links.

Well, that leads only to the point that ".." implementation is just lousy and it should have been done right in the first place. If there is a need for a loop or a hardlink (like "..") all you have to have is a standard way to find out, be it flags or the like, whatever. But taking the filename or anything not applicable to other cases as matching factor was obviously short-sighted.

Quoted 11 lines · Stephan von Krawczynski

On Mon, 4 Aug 2003 08:42:09 -0700 (PDT)

Quoted 2 lines · Brian Pawlowski

I'm still waking up, but '..' obviously breaks the "no cycle" observations.

Hear, hear ...

Quoted 2 lines

It's just that '..' is well known name by utilities as opposed to arbitrary links.

Quoted 6 lines

Well, that leads only to the point that ".." implementation is just lousy and it should have been done right in the first place. If there is a need for a loop or a hardlink (like "..") all you have to have is a standard way to find out, be it flags or the like, whatever. But taking the filename or anything not applicable to other cases as matching factor was obviously short-sighted.

Has nothing to do with the loop. It is called an AVL tree.