overlay filesystem: request for inclusion

1. Al Viro and Christoph Hellwig bring up the same locking problems *every single time* someone proposes a copy-up in-kernel file system, and *every single time* they are dismissed or hand-waved. I'd like to see the thread in which one of them says, "Why yes, you have understood and solved that problem to my satisfaction," before even considering merging something.

2. Overlayfs is not the simplest possible solution at present. For example, it currently does not prevent modification of the underlying file system directories, which is absolutely required to prevent bugs according to Al. Al proposed a solution he was happy with (read-only superblocks), I implemented it for union mounts, and I believe it can be ported to overlayfs. But that should happen *before* merging.

3. To my knowledge, Al is not currently able to reply to email on a regular basis and Christoph doesn't frequently comment on the subject these days. Don't take their silence as approval.

I have many more possible comments but I don't think they should be relevant to the discussion about merging. Al and Christoph's judgements should be sufficient.

-VAL
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Perhaps you can detail the locking problem in question ?

Alan

Pretty bizarre things can happen when the topology of the underlying layers change after overlayfs acquired refs to underlying dentries. I think this is the case Val is talking about.

Example:

# mount -toverlayfs x -oupperdir=/upper,lowerdir=/lower /ovl
# mkdir -p /upper/a/b
# ls /ovl/a/b
# mv /upper/a/b /upper/
# mv /upper/a /upper/b/
# ls  /ovl/a/b
a

Apparently "a" became its own ancestor.

Overlayfs is careful not to assume anything about child/parent relationships of underlying dentries.

For example "rmdir /ovl/a/b" will do the following:

1. find the underlying dentry for "a" -> upper-a
2. lock upper-a
3. find the underlying dentry for "b" -> upper-b
4. verify that upper-b is a child of upper-a
5. remove upper-b

With the above example it will fail on step 4. Changes to the underlying filesystems are not supported and result in undefined behavior. But it should never result in BUGs or deadlocks.

The overall locking order is:

-> overlayfs locks
-> upper fs locks
-> lower fs locks

Within each filesystem the usual locking rules apply.

One more difficulty is copy-up. This happens without being protected by i_mutex on overlayfs. The rules here are:

A. directory renames only succeed if both source and destination are
only on the upper fs (never copied up)
B. non-directory renames start with copy-up of source if necessary C. copy-up takes i_mutex on upper parent

During copy-up no ancestor will be renamed because of A.

The file being copied up may be moved concurrently, however. If this happens then copy-up will acquire i_mutex for either the old or the new upper parent. In the latter case the file has already been copied up. In the former case the file may or may not have been copied up. The state of the file is checked after having locked the upper parent and the copy-up skipped if it has already succeeded.

There may be flaws in the above reasoning or the implementation and reviews are very welcome.

Thanks,
Miklos

Miklos Szeredi:

It is good to verify in step 4.
Essentially (or ideally) this verification should be equivalent to all of what VFS does before vfs_rmdir(). I know overlayfs makes the upper mnt_want_write()-ed in early stage and keeps it. So it might be better to lookup again (as step 3 and 4) instead of comparing d_parent simply. If you think it is unnecessary to lookup here, then I'd suggest you to make it option (choosable by user).

I see ovl_rmdir() does,
- lookup and unlink all whiteouts
- rmdir the target dir
- create a whiteout for the target
Right?
But I am afraid that any error can happen in every step on the upper dir. And if it happens, then ovl_rmdir() returns the error but the dir left in incomplete status. It may be one of these.
- some whiteouts are unlinked but others are left
- all whiteouts are gone but the target dir remains
- the target dir is removed but the whiteout is not created Of course, it is bad and makes users really confused, since it will show users things which should not be. At the same time, I don't know how possible it can happen.

Anyway if you have read aufs, then you would know how aufs solves these problems. I don't think the approaches in aufs is best or one and only. I just could not get another good idea.

J. R. Okajima

The parent verification is only to make sure the locking is correct. It's not to make sure that modifications of underlying filesystems will have sane semantics.

Until someone comes up with a sane use case for allowing modification of underlying filesystem I won't bother with that.

Not quite.

- checks if directory is empty (all lower entries are whiteouted)
- marks directory opaque
- unlinks whiteouts
- rmdir
- create whiteout

Atomic whiteout and atomic copy-up would be nice, that's one feature I'm willing to think about.

Rollback on failure is an incomplete solution, rollback itself can fail. And it doesn't protect against machine crashing in the middle of operation.

Thanks,
Miklos

Miklos Szeredi:

Maybe you are right.
But do you think rollback is unnecessary since it is an incomplete solution?

And you might not have read about the approach in aufs, which tries reducing the operations in rollback.

(from '[RFC 2/8] Aufs2: structure' in 2009

      <http://marc.info/?l=linux-kernel&m=123537453514896&w=2>)

----------------------------------------
In aufs, rmdir(2) and rename(2) for dir uses whiteout alternatively. In order to make several functions in a single systemcall to be revertible, aufs adopts an approach to rename a directory to a temporary unique whiteouted name.
For example, in rename(2) dir where the target dir already existed, aufs renames the target dir to a temporary unique whiteouted name before the actual rename on a branch and then handles other actions (make it opaque, update the attributes, etc). If an error happens in these actions, aufs simply renames the whiteouted name back and returns an error. If all are succeeded, aufs registers a function to remove the whiteouted unique temporary name completely and asynchronously to the system global workqueue.
----------------------------------------

J. R. Okajima

This is generally not possible in solutions that don't reserve any filenames.

However, it should be possible to create whiteout of a non-existent entry in a directory while it is locked without affecting userspace.

Removing the whiteout asynchronously does not seem like a good idea. It should be gone before the directory containing the whiteout is unlocked. Otherwise there might be an entry created which conflicts with this whiteout that did not exist when the operation started. Also if you unlock the directory while the artifical whiteout exists an asynchronous process might replace the whiteout and the rollback would fail.

As an alternative way to perform atomic renames I would suggest "fallthrough symlinks". If you want to rename an entry which is "fallthrough" (ie pointing to the entry with the same name in the lower layer in the same directory) you can replace it with a "fallthrough symlink" which points to the lower layer and does not just implicitly say "here" but specifies a path relative to the mountpoint instead. This can then be moved like any other entry. it is in no way special anymore. Moving a directory tree which is partially in the upper layer is still time-consuming but can be performed with reasonable semantics imho. You perform a preparation step during which nothing seems to change from the user's point of view and at the very end you just move the directory.

Thanks

Michal
--
To unsubscribe from this list: send the line "unsubscribe linux-fsdevel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Yes, creation of whiteout and renaming it to target or vice versa works if target is non-directory.

Cases where this trick could make operations atomic:

- create/mknod/symlink/link over whiteout
- rename non-directory to whiteout
- remove of non-directory with whiteout creation
- copy up

Cases where atomicity is not possible with this:

- mkdir over whiteout
- rename directory to whiteout
- rename where source needs whiteout
- rmdir with whiteout creation

This is a nice idea, but doesn't have a lot to do with atomicity. It allows rename of non-pure upper directory (they return EXDEV currently).

Shouldn't be time consuming, really. The upper, mixed directory is renamed and given a "trusted.overlay.redirect" attribute to show where its lower directory resides.

Thanks,
Miklos

Michal Suchanek:

Actually aufs generates a doubly whiteouted unique name dynamically for the target dir. For instance, when rmdir("dirA") aufs does,
- lock i_mutex of the parent dir of dirA on the real fs
- some verifycations for the parent-child relationship
- some tests whether we can do rmdir
- create whiteout for dirA
- rename dirA to .wh..wh.XXXXXXXX (random value in hex), after making sure the name doesn't exist
- unlock the parent dir
- return to VFS
And then the async workqueue removes the .wh..wh.XXXXXXXX dir with some whiteouts under it.

It means the temporary whiteout name is,
- always unique
- always hidden (from users), even if it remains accidentally So even if an error happens in the async work, it doesn't matter.

Additionally there is a userspace script called "auchk" which is like fsck for real fs. auchk script checks the logical consistency on the (writable) real fs, and removes the illegal whiteouts, remained pseudo-links, and remained temp files.

Symlink?
Is it a different thing from DCACHE_FALLTHRU in UnionMount? I am afraid a special symlink is fragile or dangerous.
Its special meaning is valid in inner union world only, is it? If something in outer world gets changed, we may not follow the symlink anymore or follow something different unexpectedly. Is it acceptable?

J. R. Okajima

Probably swap the two above, you can't make a whiteout in presence of the directory, right?
Anyway, you could just mark dirA as whiteout and remove any whiteouts contained in it asynchronously, and only jump through these hoops when trying to create a new entry in place of non-empty whiteout, or sync on emptying the old whiteout before making a new entry.

Yes, it can only cause pollution with whiteouts unrelated to any files that ever existed which is not too much of an issue unless people want to add random stuff to the lower layer and see it in the union when they reconstruct it again.

Yes, the fallthru in unionmount only says "look below here", it cannot point to a different place in the filesystem.

It is only valid when in the upper layer of a union. However, so is whiteout, and so are files that were visible in the union but are not visible in the top layer if examined separately, outside of the union.

It must be accepted that the top layer is different from the union, otherwise you want a copy, not a union.

That' the whole idea behind symlinks, and also unions which implicitly link the lower layer into the upper to present the result as a single directory tree.

Anyway, the motivation behind the "fallthru symlink" is that you need not copy-up on seemingly trivial operations like rename, touch, etc. which both makes them more efficient and easier to get atomic. As I understand it copy-up is the operation that causes the most issues and with "fallthru symlinks" you need it only for operations that are expected to modify something non-trivially.

Obviously, this is not so nice for zero sized files but they should be handled the same way for consistency I guess. Also metada that can be conveniently recorded on the fallthru entry would make touch fast but would hide possible later updates to the lower layer so it might be not good solution for all use cases. For throwaway tmpfs, however, any optimization counts.

Seriously, the overlayfs documents that it can have opaque directories but I don't see what they would be used for. There is no way to turn a directory opaque with normal userspace operation afaict.
It has no explicit fallthrus, at least not documented so to have any level of consistency it should always check the lower layer because it can grow some new directories when the union is deconstructed, offline modified, and reconstructed (which is supported use case according to the docs).

Thanks

Michal

Michal Suchanek:

Unfortunately I cannot understand what you wrote.

First, the order of

is correct and I think it should be, in order to make a little help for fsck/auchk.
And what is "non-empty whiteout" and "emptying the old whiteout"? The whiteout is a size zero-ed and hardlinked regular file in aufs.

??
Do you think that the .wh..wh.XXXXXXXX hides something on the lower layer? If so, it is wrong. Such doubly whiteout hides nothing except itself.

Do you mean that your special symlink has totally different file-type from a symlink?
Anyway what I want to say is, what such symlink refers may differ from what users originally expect. But I may misunderstand what you call "fallthru symlink".

J. R. Okajima

Yes, it's correct for aufs which uses reserved file names for whiteouts.

Filesystems that don't reserve filenames cannot make whiteout for an existing entry but aufs can.

Is there any reason why a directory cannot be whiteout?

It may possibly hide a XXXXXXXX file if it is later added to the lower layer.

But if .wh.XXXXXXXX is in itself a reserved filename that is never brought up from the lower layer then this is a non-issue, it works consistently regardless of existence of the superfluous whiteout.

Just as whiteout has totally different file-type from a file. It's specific to the union.

How is this different from other files that are taken from the lower layer and not copied into the upper layer?

If you are concerned about that you want a full copy, not a union.

Thanks

Michal

Michal Suchanek:

Just to reduce consuming inodes.

No, because it is "doubly" whiteouted.

Ok, we are talking about different whiteouts.

J. R. Okajima

Miklos, I agree with you. I think it makes perfect sense for overlayfs at this point not to bother with users who modify lower files directly, and expect sane semantics at the upper layer. Most unioning users do NOT do that anyway, but the few who do cause unioning code to be much more complex.

That said, if users do go and add/del files below overlayfs, it shouldn't oops...

I've often been bothered by people who suggested that stackable file systems must solve the "cache coherency" problem and must perfectly detect lower-level changes consistently. A lot of code in Unionfs is spent on cache coherency.

Ecryptfs has been around for several years, and I've yet to see the masses scream for upper/lower layer consistency. NFS works just fine for years and no one expects changes to server-side disk-based file system to be reflected immediately and correctly on al clients. Asking overlayfs or other stackable file systems to solve this multi-layer coherency perfectly is somewhat ridiculous: we don't expect file systems like ext3 to detect and correctly handle changes to lower devices — i.e., if someone hand-edited direct blocks in /dev/sda1, do we?

The union mount team also tried to handle this issue (the so-called "really really NFS readonly" idea). And it's really hard — and unnecessary for v1 of a unioning solution.

Skip this can of worms, Miklos. And you'll be happier for it. :-) I think your approach of keeping Overlayfs small for now is absolutely the right way.

Cheers,
Erez.

Erez Zadok:

I agree with you if we discuss about union-type-mount, which handles a block device as its member. As long as the layered-fs handles a directory (mounted filesystem) as its member, it is obviously right that users expect the modification on the member fs (by-passing a union) is available.

Of course I agree it brings complication to us, and I'd suggest three level options to support this issue.
- detect the direct changes and reflect it to union (hardest option)
- skip the detection, but verify the parent-child relationship or more at least. (this is something like overlayfs is trying to do)
- skip both of the detection and verification (lowest option) this option depends how user sets up the union and its member. if user hides the members totally by over-mounting an empty dir on the member (or something), then he can specify this option. otherwise, this option is dangerous. also some symlinks may not work.

  # mkdir /hide
  # mount -o upper=/rw,lower=/ro none /union
  # mount -o bind /hide /rw
  # mount -o bind /hide /ro

J. R. Okajima