UTF-8 and case-insensitivity
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
On Thu, 19 Feb 2004 00:40:21 +0100, you wrote in linux.kernel:
Quoted 2 lines
Because a large number of file operations are on filenames that don't exist. I have to *prove* they don't exist. That includes:
Evil question: do you need to be case-preserving? 'Cause if not, you could simply squash all incoming filenames from case-insensitive clients to some canonical form (say, all lower-case) and use that.
Signature · 2 lines
Ciao, Pascal
Pascal,
Quoted 3 lines
Evil question: do you need to be case-preserving? 'Cause if not, you could simply squash all incoming filenames from case-insensitive clients to some canonical form (say, all lower-case) and use that.
yes, we have to be case preserving, but thats not the problem. Keeping some name mapping in user space or xattrs is tedious but conceptually easy and potentially quite efficient.
The problem is that Samba isn't the only program to be accessing these directories. Multi-protocol file servers and file servers where users also have local access are common. That means we can't assume that some other filesystem user hasn't created a file which matches in a case-insensitive manner. That means we need to do an awful lot of directory scans.
I also understand the decision Linus has made that we won't be doing anything fundamental at the filesystem level to fix this, so we will just have to live with it.
Cheers, Tridge
Quoted 6 lines
The problem is that Samba isn't the only program to be accessing these directories. Multi-protocol file servers and file servers where users also have local access are common. That means we can't assume that some other filesystem user hasn't created a file which matches in a case-insensitive manner. That means we need to do an awful lot of directory scans.
Do you also require NFSD or other file daemons to do the same case-insensitivity check? Say you create a foo, how do you prevent NFSD from creating FOO? What could you do about that?
Hua,
Quoted 2 lines
Do you also require NFSD or other file daemons to do the same case-insensitivity check?
no. That's the point of the per-process check. Only Samba needs to pay the price.
Quoted 2 lines
Say you create a foo, how do you prevent NFSD from creating FOO? What could you do about that?
You don't need to do anything in particular about it. I did explain this earlier in this thread, but here goes again:
* samba always tries the name exactly as given by the client. If that succeeds then we are done.
* if it doesn't find an exact match then it does a directory scan. It uses the first case-insensitive matching name it finds, or if it reaches the end of the directory then it concludes that the file doesn't exist.
So if FOO and foo both exist in the filesystem, and someone asks for FoO then its pretty much random which one they get (ok, not exactly random, but close enough for this argument). The thing is that just making an arbitrary choice is a perfectly fine set of semantics. You can't deal with this situation any more sanely, so don't even try.
well, actually, there is something you could do that we don't do. We could have some special marker that distinguishes files created by windows clients and files created by unix clients, and preferentially return the one created by windows clients, I just don't think this is worth doing. Nobody has even complained (within earshot of me anyway) of the current "pick one" method.
Cheers, Tridge
Quoted 6 lines · On Thu, Feb 19, 2004 at 12:01:53PM +1100, [email protected] wrote:
The problem is that Samba isn't the only program to be accessing these directories. Multi-protocol file servers and file servers where users also have local access are common. That means we can't assume that some other filesystem user hasn't created a file which matches in a case-insensitive manner. That means we need to do an awful lot of directory scans.
Actually, not necessarily. What if Samba gets notifications of all filename renames and creates in the directory, so that after the initial directory scan, it can keep track of what filenames are present in the directory? It can then "prove the negative", as you put it, without having to continuously do directory scans.
Yeah, there can be some race conditions, but Samba already has to deal with the race condition where it tries to create "MaKeFiLe" either just before or just after a Posix process creates "Makefile".
- Ted
Ted,
Quoted 5 lines
Actually, not necessarily. What if Samba gets notifications of all filename renames and creates in the directory, so that after the initial directory scan, it can keep track of what filenames are present in the directory? It can then "prove the negative", as you put it, without having to continuously do directory scans.
Currently dnotify doesn't give you the filename that is being added/deleted/renamed. It just tells you that something has happened, but not enough to actually maintain a name cache in user space.
That could be changed, so that on a dnotify event you do a fcntl() to ask for the name of the file. Or perhaps we could cram it into the structure the signal handler gets passed? I doubt that would make sense, but maybe some signal guru can tell me otherwise. Maybe we could even invent a new dnotify system where you do a read on a file descriptor to get details on what event happened, and give some "everything has changed" error when you run out of buffers.
If that happened then we could build our own dcache in user space, but it will be a very second rate dcache, with a racy and slow update mechanism that will in itself chew cpu. Maybe thats the best we can do, or maybe I should be asking distro vendors if they would consider a case-insensitive patch, especially the vendors aiming for "enterprise" scalability which might include serving windows clients.
Quoted 3 lines
Yeah, there can be some race conditions, but Samba already has to deal with the race condition where it tries to create "MaKeFiLe" either just before or just after a Posix process creates "
Makefile".
yes, thats true.
The races aren't my primary concern really. I've spent the last week doing profiling of a large Samba install, and after fixing a horrendous scalability problem do to with fcntl locking (more on that later) the next thing on the profile is stat() and directory scans. That's why the efficiency of this stuff is a hot topic for me right now.
It's not all as bleak as perhaps I make it seem though. I suspect there is still quite a bit of improvement that can be made in Samba just because our code is so messy that sometimes we do a stat() call or a directory scan when perhaps we can prove that we don't need to. The Samba4 code is much cleaner, and maybe we have room to keep improving things for a couple of years by finding those inefficiencies and fixing them. We will eventually hit a wall, but it could be a fair way off. Maybe windows will be dead by then.
Cheers, Tridge
Quoted 8 lines · [email protected] wrote:
Ted,
Quoted 5 lines
Actually, not necessarily. What if Samba gets notifications of all filename renames and creates in the directory, so that after the initial directory scan, it can keep track of what filenames are present in the directory? It can then "prove the negative", as you put it, without having to continuously do directory scans.
Quoted 4 lines
Currently dnotify doesn't give you the filename that is being added/deleted/renamed. It just tells you that something has happened, but not enough to actually maintain a name cache in user space.
You can still keep per-directory caches that you simply invalidate on each dnotify,
and rebuild when necessary. At least it would help the "repeated
lookup of nonexistant filenames" case. Path searches for executables usually happens on directories that don't see much writing.
Helge Hafting
Quoted 3 lines · [email protected] wrote:
Currently dnotify doesn't give you the filename that is being added/deleted/renamed. It just tells you that something has happened, but not enough to actually maintain a name cache in user space.
This might be a crazy / stupid idea, so flame at will :)
Wouldn't it be possible to do a samba "super-server" mode, in which samba would assume that it controlled the directories it is exporting?
In this mode a "corporate" Samba server, serving Windows clients, could improve performance by assuming that its cache was always up-to-date.
If if we wanted to access the directory locally we could always mount locally using samba, and access the files anyway, albeit a lot slower and without linux permissions, etc.
What we would gain was the ability to say "I want to give priority to my samba server" (and set it to "super-server" mode) or "my priority is to the linux native filesystem, and just want to share my files with windows users anyway" (and keep using samba as always).
Signature · 3 lines
Paulo Marques - www.grupopie.com "In a world without walls and fences who needs windows and gates?"
Quoted 11 lines · Paulo Marques
Quoted 3 lines · [email protected] wrote:
Currently dnotify doesn't give you the filename that is being added/deleted/renamed. It just tells you that something has happened, but not enough to actually maintain a name cache in user space.
This might be a crazy / stupid idea, so flame at will :)
Wouldn't it be possible to do a samba "super-server" mode, in which samba would assume that it controlled the directories it is exporting?
Quoted 7 lines
In this mode a "corporate" Samba server, serving Windows clients, could improve performance by assuming that its cache was always up-to-date.
If if we wanted to access the directory locally we could always mount locally using samba, and access the files anyway, albeit a lot slower and without linux permissions, etc.
You don't really need to go to such extremes. Samba can use dnotify, and run with caching and great performance as long as nobody touch the files in other ways. There is no need to _enforce_ it though, samba can cope by invalidating the cache on those rare occations the files are accessed in other ways. It won't happen often, because:
1. Linux/nfs people have no business in a directory full of windows .dll's and .exe's
2. On a corporate server you simply tell people to stay out. nfs may export another set of homedirs for the unix people.
Quoted 5 lines
What we would gain was the ability to say "I want to give priority to my samba server" (and set it to "super-server" mode) or "my priority is to the linux native filesystem, and just want to share my files with windows users anyway" (and keep using samba as always).
Thanks to dnotify even the "linux priority" setup will be able to benefit from a cache. Particularly if we can get a dnotify that doesn't trip when samba is the one making changes.
Helge Hafting
Quoted 11 lines · On Thu, Feb 19, 2004 at 02:20:44PM +1100, [email protected] wrote:
Currently dnotify doesn't give you the filename that is being added/deleted/renamed. It just tells you that something has happened, but not enough to actually maintain a name cache in user space.
That could be changed, so that on a dnotify event you do a
fcntl()to ask for the name of the file. Or perhaps we could cram it into the structure the signal handler gets passed? I doubt that would make sense, but maybe some signal guru can tell me otherwise. Maybe we could even invent a new dnotify system where you do a read on a file descriptor to get details on what event happened, and give some "everything has changed" error when you run out of buffers.
Yes, that's what I was suggesting. One advantage of such a scheme is that it's not just for Windows compatibility. A more rich directory change notification scheme would also be useful for graphical file managers, automatic indexing tools, and many, many other applications.
No, it's not everything you were requesting, but it may very well represent three-quarters of a loaf, instead of nothing.
Quoted 6 lines
If that happened then we could build our own dcache in user space, but it will be a very second rate dcache, with a racy and slow update mechanism that will in itself chew cpu. Maybe thats the best we can do, or maybe I should be asking distro vendors if they would consider a case-insensitive patch, especially the vendors aiming for "enterprise" scalability which might include serving windows clients.
I don't know that the update mechanism has to seriously chew that much CPU. It can certainly can be designed to minimize the amount of CPU that is consumed, especially if it is read via a file descriptor so that multiple updates can be sent via a single read() system call, instead of sending a signal every single time a directory entry is created, renamed, or deleted.
The problem with a case-insentive patch is that for most modern filesystems (i.e., any filesystem that does better than O(1) directory searches), it will have to involve a format change, since the case insensitivity has to be built into the hash function or the tree comparison fucture, or both. At this point, the filesystem author has to make the choice of whether to try to solve the Windows-specific problem, in which case the fundamental filesystem format would have to be tailored to the Windows case mapping table, or try to solve the more general I18N case mapping problem. (Lots of luck! It's constantly changing over time as new character sets are added or modified...) Yes, a few such filesystems might have this support already, but I doubt distributions would be willing to accept patches that make filesystem format-incompatible changes just for the sake of accelerating Samba operations.
I don't know if the distributions would be willing to accept a case-insensitive patch, but my suspicions is that it would be difficult, and I would argue that it might be more efficient to get a richer directory change notification system, for the reasons I argued above.
- Ted
(I may, of course, be overly naive... but a thought occurs... 8-)
It would seem that the there is a moment of opportunity at the dentry_operations invocation point to harvest all the information you would need to maintain a specialized dcache in a separate module. Unfortunately, since the individual file systems get to tweak their own pointer(s) to this/these struct-of-calls it could get hard to hijack things at that level.
With two changes to core Linux behavior, which could easily be implemented as a configurable kernel option, you could create an advisory hook.
1) add a usually-null pointer(*) to dentry_operations structure to the superblock data structure in vfs (and, of course, an install/remove structure call pair) as a look-aside mechanism, and
2) if-not-null "parallel" invocations of these "advisory" calls are then added to the fixed vfs invocation points along side the normal dentry notices...
You could then add any imaginable advisory behavior to any file system. A well crafted module could then attach to file systems, flag directories (+), and get low-level advisory service at core dentry action time.
A module so attached could answer all your negative enquiries quickly and yet remain nicely segregated. You could probably create the magic_open dream logic of your choice and net near, if not absolute, race elimination.
You still might have to readdir a whole dirctory from time to time just to clean-up a partily aged cache, but there would be no need for the stepwise transfer of this information into the user context.
100% of the native function of each file system is preserved and there are probably other applications for this look-aside feature like low-level security auditing or semantic mirroring (a-la real-time rdist).
But, you know, just a thought...
Rob.
(*) this should, if enabled, be arranged as a linked list of structures so that multiple modules could be installed for different purposes.
(+) flagging and un-flagging directories of interest ad-hoc is needed to prevent saturation of resources.