non-recursive pathname resolution

Having reread the RFC... What's the problem with intermediates?
Just put GETFH and GETATTR between the LOOKUP for each component in the same compound and be done with that - you've got yourself everything you might possibly need for populating them. Confused...

BTW, I would still very much prefer to allocate a chain of
dentries in fs/namei.c (yes, marking them "in-lookup"), then gave an array of pointers (or beginning and end of the chain, but that can be more delicate due to dentry tree topology changes from e.g. d_materialize_unique(), aka d_splice_alias() these days). With the requirement being "populate them in root-to-leaves order, do nothing for ones that had in-lookup flag already cleared".

Another fun possibility (but that would take somewhat more
restructuring in fs/namei.c) would be to have (on hot cache) a path traced for several components, seeing that they are all on the same fs and delaying revalidation for a while. With bulk revalidate covering all the chain when we stumble across .., mountpoint or something we believe to be a symlink, or when the chain reaches fs-specified limit. Said bulk revalidate should tell how long a prefix had been OK. Permission change handling would be the painful part here...

The problem isn't getting intermediates. The problem is that not having intermediates confuses the dcache. When the dcache is just providing a caching service, and not providing a consistency service, then it shouldn't let itself get confused.

This seems to just entrench the approach that the dcache is in control and you are trying to contort it in some unnecessary way to meet one more need.

In the common case, the rcu_walk version of d_revalidate would be very simple and batching them isn't going to buy much. Once you hit a "needs revalidate" or anything else that trips up rcu_walk, just hand it all to the filesystem and say "your problem". If the filesystem wants to continue one step at a
time, that is easy - there are helpers for that. If the filesystem wants to
send the remainder of the path to the server, it can do that too (being careful of symlinks and mount points of course). Just give the filesystem control for the rare slow path.

NeilBrown

The dcache is much more than just a cache. It *is* in control. You may not like it, but in the big picture, one odd filesystem not liking it isn't a big deal.

Sorry, but that really is how it is. NFS isn't special enough for some badly designed lookup models to matter one whit. And caching si *so* effective, that the actual lookup case isn't the primary thing.

Yes, you can find loads where caching doesn't work well. But they are odd and not all that important. The cases where caches dominate are *much* more common.

Designing things around the 1% case (or the 0.01%) would be completely insane. The dcache and the vfs is designed for the 99.9% case.

Linus

Btw, it's not just about performance, although the whole "we can do cached lookups without ever having to et the filesystem involved" is a big deal.

It's about getting fundamental concpets like mount points etc right, it's about all those those things that the filesystem really doesn't know about, and _cannot_ sanely know about.

It's now about things like overlayfs etc, all those things.

So the filesystem really isn't in control. Never will be. The filesystem is at the mercy of (extended) unix semantics that are bigger than the filesystem.

This is true of IO too. The filesystem does have a bit more flexibility, but in the end, you have to do the readpage thing, because it's the only way you'll get mmap. The filesystem isn't really in control there either, there are strict rules for what it has to do in order to have reasonable coherent mmap semantics.

So the vfs layer often does have a "library" approach, because filesystems may do things in very different ways. But at the same time, the vfs layer really *is* in control, because it's the vfs layer that enforces certain basic semantics. So the dcache very much isn't just sme "slave cache" that you choose to use and is at the control of the filesystem. Like the page cache, you don't get a choice, because you aren't in charge.

When somebody does a lookup of a filename, it is not a "pass this filename to the filesystem". It very much *is* a component-by-component lookup. And in the *vast* majority of the cases, the cached lookup when you don't even get asked is absolutely the right thing to do, and doing anything else wouldn't just be wrong, it would be completely and utterly stupid.

And the fact that somebody doesn't understand that, and has designed bad extensions to do multi-component lookup, isn't actually an argument against the dcache. It's just an argument for "people make bad intterfaces because they hack things up and don't understand things".

Linus

Last I checked, sysfs doesn't use the page cache.
Of course, sysfs is a special case (but then, aren't we all -- deep down).

I like the page cache. Really do. It provides useful services and lots of 'generic' helpers. You can use the 'generic' versions directly, or wrap them in a little bit of extra code, or rewrite them completely. It's lovely.

I think you must have been reading someone else's emails, not mine. I'm totally there with the cached lookups. They are awesome. Don't want anything else. But when the cache doesn't have the answer - what then? The filesystem is most likely to know how to fill the cache most efficiently.

I remember hunting after some problem a while ago. I don't remember the exact details but it was related to when NFS is asked to perform permission checks on the way to opening something. I'm pretty sure it involved atomic_open() as a key part.
Anyway, the code is/was very hairy and seemed to be convoluted in order to try to meet every bodies needs at once.
Having one piece of code that tries to handle the subtle details for all filesystems is, I think, a mistake. Certainly have a block of code that does the 'easy, local filesystem' version. But don't try to combine the necessarily-different NFS version into the same block of code. It becomes (nearly) unreadable.

I know there are interesting complex cases for open: O_EXCL and trailing symlinks and things certainly make it interesting. But pretending that all filesystems can be squashed into the one mould is just a pretence.

NeilBrown

Er... Bad example, that - overlayfs is _not_ fs-agnostic.

And that is complete crap. Multi-component lookups do make sense; once we are at the edge of the area present in dcache, we _know_ there won't be any existing mountpoints involved; parsing the components and feeding them to fs at once, along with an array of dentries to fill makes perfect sense. Why bother with a bunch of roundtrips when we can have one?

Another thing is that the likely situation with revalidaions is that the damn thing had been sitting around long enough for wanting to check with the server, only to get "it's still OK" in response. Again, when that happens it makes perfect sense to do speculative walk for more than one component ("if everything's valid, the next 5 components would be these and they all are on that filesystem; hey, server - are these 5 still valid?") and avoid pointless roundtrips.

As for Neil's point re do_last() and friends being much too convoluted - yes, they are. And it's not a result of trying to shoehorn everything in one model. "Just let NFS have at it" as soon as we reach do_last() won't make things any simpler, unfortunately - we'll end up with the same convoluted mess copied in a bunch of filesystems, each with bugs of its own.

Don't get me wrong - do_last/lookup_open/atomic_open complex is not a pretty sight and it does need cleaning up. Some of that got done lately, but that was only a tangential hit. Untangling the whole damn thing won't be fun, but we'll need to do it. Not this cycle, though - we already have more than enough patches piled in fs/namei.c as it is...

Yes, the edges are easier. And yes, it's fine to do components one by one.

Maybe I misunderstood, but I thought that was exactly what Neil *didn't* want to do, though. It sounded like he wanted to do path-based lookup, not component-based one.

But yes, if it's purely about preloading the cache, then *that* should be reasonably easy. In fact, it should work as-is today, if we just added a "const char *hint" to the lookup callback which told the filesystem what will come after this lookup. But it would be a hint for pre-loading the dcache, nothing more.

So if we have a pathname like "a/b/c" that we don't have in the dcache, and we're doing to look up component "a", we could give "b/c" as the hint, and a filesystem that currently populates the dcache with "a" by doing

        d_instantiate(dentry, inode);

could decide that *before* it does that "d_instantiate()", it could pre-populate the child list of 'dentry' with the lookup information for 'b' (and possibly recursively for 'c' too under 'd').

But you'd still have to do the components one by one, you couldn't just do the "final" tip.

And no, I absolutely refuse to even entertain the thought of the filesystem actually doing any of the do_last crap. It would bt purely about pre-populating the dcache deeper than the one single component, and then the VFS layer would just find the pre-populated dentries and do the normal thing.

Doing things that way means that not only does do_last() at the vfs level already do the right thing, but we get all the per-component semantics (with security checks etc) right, because we'd still be traversing the pathname one component at a time. It's just the filesystem that could prime the cache.

If *that* was what Neil wanted to do (rather than do "a/b/c" as one single lookup to the server), then I withdraw all my complaints and am sorry for having misunderstood.

Linus

Yech, nfs uses d_splice_alias(), so we'd have to add some logic to throw away any preloaded entries there if we actually have an alias and end up merging and using the alias. Or something. Because I think we'd have a dentry leak otherwise.

So it wouldn't work "as-is", but it might still be reasonably easy.

Linus

Just to be crystal clear about what I want:
I want the filesystem to be in control

Any examples, whether about multi-component lookup or path-based lookup or O_EXCL opens are just throw-away examples. I have no desire to implement or re-implement anything like that. I just want the filesystem to have control. The reason I want, is that it will (ultimately) make the code easier to understand and so easier to verify. And it will make implementing unusual filesystems easier.

The dcache is just a cache. It is a great cache, but it isn't the filesystem.

So filesystems should be able to put things in the cache. And the VFS should be able to look up things in the cache. And if the VFS finds everything it needs to follow a full path all the way to the inode at the end, that is great. But as soon as it hits something that the cache doesn't have an answer for, it asks the filesystem.
As a useful simple case it can ask via d_revalidate in RCU mode, in which case the filesystem either says (based on its own caching rules) "Yeah, this one's OK really" and the VFS just keeps going, or the filesystem says "Nope, I need more time with this one" and we drop out or RCU and to the more general case.

In that general case it just hands everything to the filesystem. The filesystem then uses generic helpers (or not) to find the answers and adds more current information to the cache.
It could potentially just return and let the VFS continue down the cache (now with current data), but it probably makes more sense for the filesystem to explicitly return what it has.

So for Al's example of revalidating multiple components at once, once the VFS gets to a point in the path where d_revalidate says "I need more time", the VFS just passes the rest of the path to the filesystem.
The filesystem can then see what is in the cache and revalidate multiple dentries in parallel. Or it could just send the rest of the path to the server requesting attributes for each directory in the path, and then can pop all of that into the dcache/icache and let the lookup complete. Or it can just do one component at a time.

"hint" being a synonym for "layering violation" ??

NeilBrown

Yeah, no. Not going to happen.

You seem to think that the dcache is "just" a cache. It's not. It's a cache, but that is absolutely not all that it is. It's very much a cache with strong semantics.

And no, we're not handing over those semantics over to the filesystem. The dcache is not just a cache, it's the *primary* data structure that we use for pathname validation, local security checking, and for doing things like "getcwd()" and handling ".." etc.

So there's no way the filesystem is "in control". You as a filesystem are not really even doing the actual pathname lookup. The *only* thing you're doing is filling in the dcache. The actual real pathname lookup is done by the VFS layer using the dcache data.

That's how it very fundamentally works. It's *so* much more than a cache - it really *is* the primary path lookup. The filesystem is the slave in this relationship.

You can do that already. There *are* those generic helpers to add data to the cache. That's what "d_instantiate()" and friends _are_ for.

But no, you do *not* control name lookup. You get notified when there's not enough data in the cache, and then you can fill it up any which way you want.

You can populate the dcache with other entries than the one we asked for, and you can ask the dcache to revalidate and throw dentries out.

But no, you do *not* get access to things like do_last() or to the decision to follow symlinks or namespace rules, or mountpoints or things like that.

That's bullshit,. for a very simple and basic reason: "the rest of the path" is not necessarily at all for your filesystem!

Really. There might be mount-points, there might be symlinks, there might be tons of stuff like that.

You're not getting control, for the very simple reason that IT IS NOT YOUR DATA. And it really never ever will be.

Now, this is why I said we can do a "hint" style thing. Part of that "hint" issue is very very much that it has no semantic meaning. You can't screw it up, because if it turns out that the path component we're looking up is a symlink and we actually end up in some other filesystem, if you end up looking up the hint part, it just would never actually get used.

So it's kind of like a prefetch for names. It's semantically much weaker than saying "look up this name". The hint would be "this is likely the next part of the name that the VFS layer will look up".

And the key part of that statement is
(a) "likely" (it might not happen, and even if it does happen, it
migth not be for your filesystem)
and
(b) "the VFS layer will look up" because it won't be the low-level
filesystem doing it.

So it would be the low-level filesystem pre-populating the dcache - if the low-level filesystem decides the hint is worth using for that - and the VFS layer then uses the data in the dcache without further bothering the filesystem.

Exactly because the dcache is *so* much more than "just a cache".

Linus

"Rest of the path" makes no sense, obviously. "More of the path" (and _not_ as a string, TYVM - we have those components in ->d_name.name of dentries we want revalidated, complete with hashes, so WTF redo that?) is fine, though, because the caller knows exactly where the mountpoints are, so we know how far we can go.

For revalidate "this used to be a symlink, now it's not" or vice versa means simply "it's gone stale". Which is fine - again, the caller knows where in that chain the symlinks are (and they obviously terminate the chain to be revalidated).

Anyway, it's a side issue; we _can_ use the capability to do multi-component lookups and with link_path_walk()-related logics getting untangled, we might be able to do just that without messing the code up. It's very clearly not a 4.2 fodder, though, and I'm not sure how high priority it is for 4.3, simply because making the exclusion between lookups weaker seems to be of more general interest. And those two will definitely be stepping on the same area, so it probably makes more sense to sort the parallel lookups out first. _IF_ we manage to get that by 4.3-rc<early>, sure, continuation into multicomponent lookups would start looking as 4.3 material. Hell knows - stranger things have happened...

What we really need is a coherent documentation on the whole pathname-related machinery; I've some preliminary bits and pieces written, but it'll take more work. Hopefully I'll have something postable in a week or less...

Right now I can think of 4 or 5 people familiar with the area, myself included. And we need more - it shouldn't be a fucking black magic, since fs folks really ought to understand what's going on there. The thing is, under the assorted layers of cruft it's simpler than e.g. VMA handling...

For revalidate, yes we kind of have them as dentries. I say kind of, because it may be that we're only revalidating a directory in the middle, and the rest of the path will be all new lookups, and we don't have that part as dentries at all. But nobody is going to care about "revalidate" for those.

HOWEVER.

We haven't actually walked/parsed the rest of the pathname yet at that point, and we generally probably shouldn't, since we don't know if the filesystem really is going to care. Why do extra work that may not be useful?

So if we really do want to do it, and some filesystem cares enough, I think we actually should just pass it in as a string, and then have a helper function to say "ok, filesystem, if you want to revalidate the rest of the path, use this function to turn the hinting string into more dentries".

Because I don't think it's worth doing up-front if it's not clear that the filesystem is going to care.

For example, the /proc filesystem uses d_revalidate() to re-check the pid entries. That does *not* mean that it wants the rest of the dentries pre-parsed at all. So by all means give it a "const char *hint" for the rest, but it's going to just ignore it anyway, so don't wast parsing it as "these will be the following dentries we will ask you to revalidate".

So I do think that "const char *hint" might be the right thing to pass down if we really care about this sufficiently. Both to revalidate and to lookup().

We currently have that "rest" in lookup_slow() as

        nd->last.name + hashlen_len(nd->last.hash_len)

and we could just pass that down to lookup_dcache (which does revalidate) and lookup_real() (which does the actual ->lookup).

But I guess we could just save off 'name' into a new field in nameidata in link_path_walk() after we've removed the slashes? I don't know if it matters. We can mask off the slashes again if/when people want to use the hint, after all.

Linus

A fact that makes it relatively easy to create a situation where 'getcwd()' returns a string for which 'stat' says ENOENT or where "cd .." puts you somewhere that "getcwd" gets quite upset:

$ cd ..
cd: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory

$ ls -l /proc/self/cwd
lrwxrwxrwx 1 neilb users 0 May 19 17:28 /proc/self/cwd -> /mnt/tmp/cdir (deleted)

(and no: I hadn't deleted the cwd, just renamed some things on the server)

This requires the VFS to have knowledge, sometimes intimate knowledge, of how each filesystem works.
DCACHE_NFSFS_RENAMED ?
Oh wait, afs and btrfs know about that too, so it can't be too intimate.

Obviously the important rules that you mention would be handled by library code. But do_last() could be a lot simpler if the filesystem could manage the 'stale dentry' handling and call one version or the other of do_last depending on whether it had an 'atomic_open' callback or not.

For revalidate: probably not, though the filesystem can ask questions of the dcache just as easily as the VFS. For lookup, the rest of the path up to a ".." or symlink (which the filesystem can easily recognise) does belong to the filesystem.

On this topic, Al suggested:

That "fs-specific limit" is what really bothers me. This is feeding more information about the fs into the VFS, and it assumes that a "limit" is the thing that is meaningful for the VFS to know. Just let the FS take over and use the approved interfaces to collect the dentries that it thinks might be useful to revalidate, and then revalidate them.

Well, let's just start with that "in-lookup" or "unknown" dentry that has been mentioned, so that the VFS doesn't have to hold i_mutex across lookup and create, and so that the filesystem can at least control it's own locking.

That would be a big step forward in my mind.

Thanks,
NeilBrown

There is no reason to be so gloomy.
The VFS would provide a generic_do_last() (or whatever) which handles everything correctly for local filesystems which keep the dcache precisely consistent and use it for all the valuable locking it can provide. generic_to_last() would call into other filesystem entry points just like the current do_last() does. It wouldn't bother with 'revalidation' though.

Then there might be a "generic_network_do_last()" which does minimal if any checking because the server will do all that, and just calls back to the filesystem depending on which particular operation is happening - mkdir, or unlink or whatever.

Filesystem developers are both very clever and (I assume) very lazy. If you provide them with tools that work and are easy to understand, they will much rather use them than try to implement their own. This is why the page cache is such a success.

The more I think about this, the more it seems to me that having a dentry that can be "unknown", or as you put it "in-lookup" is potentially transformational.

Pathname lookup just always returns the dentry of the final component (plus vfsmount), but that final component might be "unknown".

Then the various "last" steps are quite separate and each work with this dentry. They unlink or mkdir or mknod or lookup or create or whatever. There are probably races that make this a little more complicated than I am hoping, and rename will doubtlessly be the most interesting. But it feels like a good direction.

That is all a long way off of course. Even just getting a "in-lookup" dentry and starting to explore what can be done with that would be a very positive step I think.

Thanks,
NeilBrown

RTFS.

RTFPOSIX. Semantics of the last step on open is very different from the rest due to symlink handling.

And do_last() has nothing whatsoever with mkdir() or unlink(). _Those_ are much simpler and don't go anywhere near that rats' nest of horrors.

Seriously, read the damn thing. It *is* horrible, all right, but shifting it inside NFS won't help you at all. Especially since you have NFSv3 to cope with, which will take care of bringing in every sodding bit NFSv4 might evade (for non-directories, that is - for directories you get the full shitpile in your face, NFSv4 or not).

And no, server will _not_ "do all that". Again, mkdir and unlink are (almost) trivial (unlink less so, due to NFS-specific shite). The real horrors are on open().

I would be a lot less gloomy about discussing just passing the buck to filesystems, starting with NFS, if NFS folks (you, in particular) had bothered to figure out what the existing code _does_ and why is it doing what it's doing. So far you have not - not even on the level of "which functions are hit in which syscall", let alone what those functions are doing.

And yes, the documentation of the whole thing is piss-poor. What we have there right now is a weird mix of bits and pieces referring to very different periods of evolution. Flat-out contradicting each other *and* the code. I know. Believe me, I know. Fuck, right now the call graph for that thing _finally_ fits into A4. Four years ago it took a goddamned A_1_. As in, eight A4 sheets taped together. Yes, really.

At least now we finally have a reasonable chance of getting that sucker into understandable shape and maybe even getting more folks to understand what that code is doing. Which, unfortunately, _is_ a requirement for serious reworks of that code. Frankly, the burden of keeping dcache consistent is the least of the PITA in there.

Certain - when a symlink or mountpoint is found, the filesystem stops. mountpoints should be rarely hit because the path to a mountpoint will usually be stable enough for RCU-walk to find it....
Thinks: I wonder what happens when a mount-on NFS directory is deleted on the server...

automountpoints would be handled completely by the filesystem. It would mount something and then return saying "Look, I found a mount point - you wanna handle for me?".

NFS directories already need to be revalidated occasionally. Having a dentry in "unknown" state just means a revalidation is that much more likely.

Suppose I cd into a directory, then rename the directory on the server. What happens? What should happen?
I could make a case that the NFS client should lookup ".." on the server and rebuild the path upwards.

There is a (to me) really key point here.
Local filesystems use the dcache for correctness. It prevents concurrent directory renames from creating loops and it ensure that only one file of a given name exists in each directory.

Remote filesystems don't use it for correctness. For them it is simply an optimisation. So getting upset about directories with multiple dentries, or directories that aren't connected to the root is very important for local filesystems, and largely irrelevant for network filesystems.

A local filesystem needs the cache to remain consistent with storage. A network filesystem cannot possible ensure that the cache is consistent with storage, and just need to be able to notice the more offensive inconsistencies reasonably quickly, and repair them.

Any partname lookup which touched these dentries would call d_revalidate() (or similar) which could get the inode etc if it was really needed.

If was remembering RFC3010 in which a LOOKUP had a "pathname4" which was an array of "component4". It could just return the filehandle and attributes of the final target.
RFC3530 and later revised that so "LOOKUP" gets a "component4". That just means that it is easy to get the attributes if you want them.

I'm not really saying that multiple component lookups are a good idea, or that doing the lookup and not getting the intermediate attributes is a sensible approach. What I'm really pointing out is that the current dcache imposes a particular model very strongly on filesystems and I'm far from convinced that that is a good idea.

NeilBrown

rwsem by itself won't do us much good there. Look: for multiple lookups on the same existing entry we could try to teach d_splice_alias() to cope, etc. But what happens when a bunch of processes looks for the same inexistent entry? And no, "who cares about fuckloads of negatives with the same name" isn't a good answer - suppose we do mkdir() after that. OK, so we'll find a negative dentry in dcache. And tell the filesystem to create the sucker. Done. Made it positive. Now, do we hunt down all _other_ negative dentries for it? Or never keep negative ones at all. Or slap some kind of ->d_revalidate() there to catch all negative dentries creates before the last mkdir/creat/mknod/symlink/link in given parent?

One possibility would be a new dentry state - "being looked up". Hashed, treated as "fall out of RCU mode" for lazy pathwalk purposes, and places where we call ->lookup() would (while still holding ->i_mutex on parent shared) wait for that state to end. Places where we call ->d_revalidate() (with or without ->i_mutex on parent) would also wait on those.

It would need a careful analysis of tree-walkers, though. Doable, but there might be dragons. In case of e.g. ceph - swamp ones, with mirror in the line of sight...

Yup, we can do the same to XFS to implement concurrent modifications.

Cheers,

Dave.

Dave Chinner
[email protected]

Let me know what you need :-).

You should, it's very good these days :-). We have to walk the directory trees to do the case insensitive pathname lookups, but we cache these so when we get an incoming pathname we look it up in the cache, see if we can stat the looked up name and if so we're done with pathname resolution.

Of course we tell people to just set their filesystems up using mkfs.xfs -n version=ci :-).

So ASCII-only case-insensitivity is sufficient for you guys?

Doing case-insensitive lookups at a vfs layer level wouldn't be impossible (add some new lookup flag, so it would *not* be per-filesystem, it would be per-operation!), but the *full* case-insensitivity space in utf-8 is too much to expect. Especially since different people have different opinions on what it even should be.

What else is problematic? I think you want an error on symlinks in the middle, right? So that you can do those manually? I also assume you don't like to follow ".." due to containment issues?

Adding (again per-lookup) flags for "no symlinks" and "no dotdot") would be trivial (much more so than the case insensitivity). Would you require "error out on non-ascii characters" too, to then handle the complex cases by hand?

Obviously per-lookup flags means that you can't just use "stat()", it would be "fstatat()".

I dunno. But it *may* be worth it to really try to give samba what it wants. Of course, if samba is happy doing all the name caching in user space, then that's not worth worrying about.

And the reason I don't use samba myself is that I'm not a fan of network filesystems. I want my filesystems low-latency and right there on the local ssd, thank you very much. But if you have some local-machine benchmarkign thing you use, I guess I could use that to see what the profile looks like...

Linus

ENOPARSE. Either two names are equivalent or they are not; it's not a per-operation thing. What do you mean?

We can easily make things per-operation, by adding another flag. We already have per-operation flags like LOOKUP_FOLLOW, which decides if we follow the last symlink or not. We could add a LOOKUP_ICASE, which decides whether we compare case or not. Obviously, we'd have to ad the proper O_ICASE for open (and AT_ICASE for fstatat() and friends). Exactly like we do for LOOKUP_FOLLOW.

HOWEVER.

The reason ASCII-only matters is two-fold:

(a) hashing needs to work, and hash all equivalent names to the same
bucket. And we need to hash the same *regardless* of whether the operation was done with ICASE or not.

With ASCII, this is fairly easy: we could easily make the hashing
just mask bit 5 in each byte, and that wouldn't slow us down at all, and it would hardly change the hash effectiveness either. m

In particular, with ASCII, we can trivially still do the
word-at-a-time hashing. So there's fairly little downside.

(b) The *compare* needs to work too. In particular, right now we very
much try to avoid comparing the names by checking both the full hash and the name length. Again, that's fine with ASCII - two names that differ in case are the same length.

And again, we can still use the word-at-a-time compare, just have
a mask (and at compare time, we can make the mask depend on ICASE). Sure, you'll still have to do a more careful compare (becaue case-insensitivity is not *just* "same except for bit 5 even in ASCII), but we can trivially have a ICASE test up front, and keep the fast case exactly the same as before.

Now, doing full UTF-8 is *much* harder. Part of it is that outside of ASCII, you literally have cases that are ambiguous. Part of it is that outside of ASCII, now the lengths aren't even guaranteed to match. And part of it is that now you have to do things that are much more complex than just masking bits in parallel for multiple bytes at the same time (although you can still have a fast-path that depends on just masking the high bit, to at least say "this is just the ASCII subcase").

But doing ASCII ICASE compares wouldn't be that hard, and wouldn't affect performance.

Btw, don't get me wrong. I'm not saying it's a great idea. I think icase compares are stupid. Really really stupid. But samba might be worth jumping though a few hoops for. The real problem is that even with just ASCII, it does make it much easier to create nasty hash collisions in the dentry hashes (same hash from 256 variations of aAaAAaaA - just repeat the same letter in different variations of lower/upper case).

So even plain ASCII icase has some real problems. But it's conceptually not that hard. True UTF-8 icase? That's an absolute *nightmare*, and causes serious problems. OS X got it very very wrong, for example, by messing up the normalization.

Linus

Hold on. Should
stat("blah", &buf) => ENOENT, OK, let's create it
mkdir("blah", 0) => EEXIST, bugger, looks like a race
stat("blah", &buf) => ENOENT, Whiskey, Tango, Foxtrot
be possible? No per-operation flags passed, doesn't even know of the case-insensitive crap. And if fstatat() without your new flag would find c-i matches, then what does that flag do?

Confused...

No. What I described would not in any way change any of the above. I'm not understanding what your point is.

The only difference - EVER - would be if you pass in the ICASE flag. Nothing I suggested would change semantics without it (the _hash_ changes, but that doesn't change semantics, it's a purely internal random number).

Now, *with* O_ICASE/AT_ICASE, semantics change. Obviously. At that point the dentry lookup would match case-insensitively.

For example, let's say that you have a directory where you already have both "Blah" and "blah", because you created them in a sane environment. They'll be two different dentries (assuming they are cached), but they'll have the same dentry hash.

Now, you open "blah" with O_ICASE, and the end result is that you would randomly open one or the other (it would be the one you find first on the hash chain). Tough. Mixing icase and case-insensitive is by definition going to cause those kinds of issues.

The nasty issue (and the case that samba apparently wants it for) is that ICASE wouldn't be able to trust negative dentries (us having a negative dentry in one case doesn't mean that it's negative in ICASE). And that might be the killer part. Negative dentries are really useful.

Now, the VFS layer support part is I think fairly simple. I might be wrong, but I really think the hashing etc wouldn't be too painful. After all, we already do support ->d_hash() and ->d_compare(), this is "more of the same", just supported at a vfs level directly (and _allowing_ aliases in case).

The real pain is that the low-level filesystem has to support it too. That's simple for some filesystems, but it can be hard for things that hash filenames. Because there - unlike at the VFS layer - the hashes have meaning and you can't just change them to suit a ICASE lookup (because they exist on-disk).

So supporting that is likely trivial on filesystems like FAT or SYSV, which just iterate over the directory anyway at lookup() time. On ext* with hashed directories, it's nasty (and a ICASE lookup would probably have to just walk the whole directory. old-style). But I think all the code to do the nonhashed lookup is still there, since it is a filesystem feature bit. And it would only need to do that linear search thing when the ICASE flag is set in the lookup flags.

Of course, if it ends up just walking the directory linearly anyway, it doesn't fix the one samba performance problem that Jeremy pointed out, so that makes this of dubious value. If we can't do this better than samba can already do it on its own, it's kind of pointless.

Again - the filesystems (and the vfs layer) would remain case sensitive. But I think it might be fairly straightforward to allow per-operation ICASE handling for thins that want it.

Keyword "think". Maybe there's something I didn't think of.

Linus

With c-i mount, unpacking a tarball with tar(1) will not (silently, at that) create a situaiton when your c-i users will get lookups proceeding into randomly picked variant of directory, with variants differing only in case. It will do the right thing and put the files where they would be expected, giving an expected "it already exists" if you try to create a directory with the name that matches that of existing file, etc. With this, OTOH, you'll have to use specialized tools for creating files in that tree, or risk random lossage, because creating /mnt/foo/bar when /mnt/Foo/Bugger already existed will succeed just fine, leaving one hell of a mess for c-i users.

What's the benefit compared to c-i mount? Not hitting filesystem's ->d_hash() and ->d_compare()?

So the reason I'd be interested in per-access flags rather than mount flags are:

- only special apps should use this anyway. IOW, samba and perhaps
things like wine, and that's absolutely it. The argument that it might confuse "tar" is bogus, exactly because not only would tar never do this in the first place, tar absolutely *mustn't* do crap like this anyway. case-insensitive filesystems are insane, the *only* possible valid reason for them is for "emulate insane systems".

- mount flags are bad. They'd be useless for something like wine
(where you want to make part of the users home directory be the filesystem), and they are bad for things like samba too. Having to make a whole filesystem case-insensitive is crazy, because case-insensitivity is crazy. You want to make one application able to use case-insensitivity, not make all accesses so.

- we do have cases where more per-access flags might be a really good
idea. The whole "don't follow any/absolute symlinks" and "don't follow dotdot" are real concerns in various other places that now waste time trying to do it manually (and generally do it badly at that - see all the historical apache issues with dotdot to escape the publicly visible areas). I think it's a mistake in general to think that these kinds of things should be per-mount.

That said, if the main/only reason samba would use this is for the case it would be bad at handling anyway (negative lookups in big directories), it's definitely not worth it.

Linus

No it's not enough really. But for specific Windows apps that use restricted namespaces (and there are such) it works.

ZFS on *BSD does do full case-insenitive lookups (utf8) as part of FreeNAS. I think if it's configured a SMB-only share they turn that on.

Yeah, Apple are the real sinners here. utf8-compose-characters - pah !

We already strip . and .. out of incoming pathnames. Once we've walked the path (following links) we then use realpath() to ensure the full path is under the exported share "path =" directive.

Don't need the dot-dot stuff - "no symlinks" would be useful, but you have to remember Samba is still portable to *BSD and Solaris-clones (all that anyone really cares about these days) so we'll have to keep the old code paths too.

Case insensitive pathname lookup is the one I remember Windows sales guys beating us up on benchmarks (create 1,000,000 files in a directory and then have a client that looks for a file that *doesn't* exist :-).

Hopefully Volker is also on this thread and can chime in with some requirements of his own :-).

You don't know what you're missing mate !

I use it every day via my local NAS to play music (via SONOS Linux clients) and movies/TV (SageTV Linux boxes) everywhere in the house. I don't even have a Windows box (other than VM's for testing) anywhere.

But my house is fully wired for gigabit, I wouldn't do that over wireless :-).

Samba, it's not just for Windows anymore :-).

Ad some people are using this on Linux:

http://oss.sgi.com/archives/xfs/2014-09/msg00169.html

Cheers,

Dave.

Dave Chinner
[email protected]

Almost certainly it should be the netns of NFS mount the referral is on.

Looking at processes causes all kinds of caching problems, and is only really appropriate when we are looking at something deliberate like magic symlinks (i.e. /proc/self).

Automounts are the one case where I can image capturing the current netns during mounting of the filesystem could be an issue.

Eric