regressions & flakiness make for ugly symlink links
Subsystems (2)
In my Windows root directory I have several links. This is how they look on the Windows machine's "root". Several months ago I sent an email to this list complimenting you on how they looked. (without user+group)
Quoted 1 line
ll |grep -- '->'
lrwxrwxrwx 1 17 Jun 13 2016 Documents -> //Bliss/Documents/ lrwxrwxrwx 1 6 Jul 13 2009 Documents and Settings -> /Users/ lrwxrwxrwx 1 16 Jun 5 2015 FolderChanger -> /m/FolderChanger/ lrwxrwxrwx 1 5 May 13 2017 Home -> Users/ lrwxrwxrwx 1 20 Nov 6 2014 Prog -> /Program Files (x86)/ lrwxrwxrwx 1 13 Apr 21 2013 Prog64 -> Program Files/ lrwxrwxrwx 1 12 Aug 9 2015 ProgD -> /ProgramData/ lrwxrwxrwx 1 2 Apr 17 2017 Share -> /s/ lrwxrwxrwx 1 22 Aug 22 16:08 Symbols -> //Bliss/Share/Symbols// lrwxrwxrwx 1 7 Mar 1 2018 WINNT -> Windows/ lrwxrwxrwx 1 3 May 13 2017 lib64 -> lib/ lrwxrwxrwx 1 3 Jan 12 2014 temp -> tmp/
Now, I'm getting flakiresults -- with them not resolving 1 time, then resolving, then not again...etc. By flakey I mean output like random can't access messages, followed by question marks in all fields of links.
# ll|more
ls: cannot access 'D': Operation not supported
ls: cannot access 'Documents': Operation not supported
ls: cannot access 'FolderChanger': Operation not supported
ls: cannot access 'Home': Operation not supported
ls: cannot access 'lib64': Operation not supported
ls: cannot access 'M': Operation not supported
ls: cannot access 'P': Operation not supported
ls: cannot access 'pagefile.sys': Device or resource busy
ls: cannot access 'Prog64': Operation not supported
ls: cannot access 'Share': Operation not supported
ls: cannot access 'temp': Operation not supported
ls: cannot access 'WINNT': Operation not supported
d????????? ? ? ? D/ d????????? ? ? ? Documents/ d????????? ? ? ? FolderChanger/ d????????? ? ? ? Home/ d????????? ? ? ? M/ d????????? ? ? ? P/ d????????? ? ? ? Prog64/ d????????? ? ? ? Share/ d????????? ? ? ? WINNT/ d????????? ? ? ? lib64/ d????????? ? ? ? temp/
Depending on timing, Sometimes I will get most or all of them appearing the same as they would on windows.
If I can't resolve them more quickly, is there a know or two to increase cache size and item lifetime?
Thanks
пн, 18 февр. 2019 г. в 22:16, L A Walsh <[email protected]>:
Quoted 51 lines
In my Windows root directory I have several links. This is how they look on the Windows machine's "root". Several months ago I sent an email to this list complimenting you on how they looked. (without user+group)
Quoted 1 line
ll |grep -- '->'
lrwxrwxrwx 1 17 Jun 13 2016 Documents -> //Bliss/Documents/ lrwxrwxrwx 1 6 Jul 13 2009 Documents and Settings -> /Users/ lrwxrwxrwx 1 16 Jun 5 2015 FolderChanger -> /m/FolderChanger/ lrwxrwxrwx 1 5 May 13 2017 Home -> Users/ lrwxrwxrwx 1 20 Nov 6 2014 Prog -> /Program Files (x86)/ lrwxrwxrwx 1 13 Apr 21 2013 Prog64 -> Program Files/ lrwxrwxrwx 1 12 Aug 9 2015 ProgD -> /ProgramData/ lrwxrwxrwx 1 2 Apr 17 2017 Share -> /s/ lrwxrwxrwx 1 22 Aug 22 16:08 Symbols -> //Bliss/Share/Symbols// lrwxrwxrwx 1 7 Mar 1 2018 WINNT -> Windows/ lrwxrwxrwx 1 3 May 13 2017 lib64 -> lib/ lrwxrwxrwx 1 3 Jan 12 2014 temp -> tmp/Now, I'm getting flakiresults -- with them not resolving 1 time, then resolving, then not again...etc. By flakey I mean output like random can't access messages, followed by question marks in all fields of links.
# ll|more
ls: cannot access 'D': Operation not supported
ls: cannot access 'Documents': Operation not supported
ls: cannot access 'FolderChanger': Operation not supported
ls: cannot access 'Home': Operation not supported
ls: cannot access 'lib64': Operation not supported
ls: cannot access 'M': Operation not supported
ls: cannot access 'P': Operation not supported
ls: cannot access 'pagefile.sys': Device or resource busy
ls: cannot access 'Prog64': Operation not supported
ls: cannot access 'Share': Operation not supported
ls: cannot access 'temp': Operation not supported
ls: cannot access 'WINNT': Operation not supportedd????????? ? ? ? D/ d????????? ? ? ? Documents/ d????????? ? ? ? FolderChanger/ d????????? ? ? ? Home/ d????????? ? ? ? M/ d????????? ? ? ? P/ d????????? ? ? ? Prog64/ d????????? ? ? ? Share/ d????????? ? ? ? WINNT/ d????????? ? ? ? lib64/ d????????? ? ? ? temp/
Quoted 8 lines
Depending on timing, Sometimes I will get most or all of them appearing the same as they would on windows.
If I can't resolve them more quickly, is there a know or two to increase cache size and item lifetime?
Thanks
Which kernel version shows symlinks right and which - wrong?
--
Best regards,
Pavel Shilovsky
Quoted 8 lines · Pavel Shilovsky
пн, 18 февр. 2019 г. в 22:16, L A Walsh <[email protected]>:
Quoted 6 lines
In my Windows root directory I have several links...
lrwxrwxrwx 1 17 Jun 13 2016 Documents -> //Bliss/Documents/ lrwxrwxrwx 1 20 Nov 6 2014 Prog -> /Program Files (x86)/ lrwxrwxrwx 1 13 Apr 21 2013 Prog64 -> Program Files/ lrwxrwxrwx 1 12 Aug 9 2015 ProgD -> /ProgramData/
...
Quoted 10 lines
Quoted 10 lines
---
Now, I'm getting flaki results -- with them not resolving 1 time, then resolving, then not again...etc
# ll|more
ls: cannot access 'Documents': Operation not supported
...
ls: cannot access 'Prog64': Operation not supported
...
---
Quoted 7 lines
Quoted 7 lines
d????????? ? ? ? Documents/
...
d????????? ? ? ? Prog64/
...
Depending on timing, Sometimes I will get most or all of them appearing the same as they would on windows.
Quoted 6 lines
Quoted 3 lines
If I can't resolve them more quickly, is there a know or two to increase cache size and item lifetime?
Which kernel version shows symlinks right and which - wrong?
I only recently switched on getting the access info 'live', vs. mounting it all as 1 user. Since it works and doesn't work based on what's in the cache, the same kernel can provide both symptoms.
I don't remember the exact date I switched, maybe in the 4.19.xx series, but at the moment, am running 4.20.3.
Sometimes after a client reboot, I'll see question marks for all files or about 60 entries in the root dir. Vs. only the symlinks where it has to do 2 lookups -- the item and the one pointed to, and see about 10-15 entries.
I figured that it's a timing prob related to whether or not the entries are in some cache, so I need to look more closely at that UID<->RID resolution path, but meanwhile, I thought I'd increase the cache size and maybe look at the timeout value and if I could tolerate a longer timeout.
So -- don't think it is the kernel version, but the fact that it's going a long path to do the lookups -- especially the links -- and that optimizing that path would likely bring the most benefits.
There used to be documentation about doing the resolution through a separate program, but that documenation disappeared as someone wrote a .dll to do the resolving, which got plopped in there after an accidental system update attempt (long story -- power outage, ups needs new batteries, and more).
Anyway, UID/GID<=>RID map is relatively small (<10-15K) with long-lived entries. The server has >100GB mem and the client, 92GB.
That's why I wanted to increase the UID<->RID cache size & raise the timeout to a long value. Maybe ideally with a way to manually invalidate the cache and/or 1 entry for rare updates.
Thanks!
Re:upcalls seem to have problems with symlinks, junctions et al.
Quoted 13 lines · L A Walsh
Quoted 13 lines
Quoted 13 lines
---
ls: cannot access 'Documents': Operation not supported
...
ls: cannot access 'Prog64': Operation not supportedd????????? ? ? ? Documents/
...
d????????? ? ? ? Prog64/
...
Depending on timing, Sometimes I will get most or all of them appearing the same as they would on windows.
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
If I can't resolve them more quickly, is there a know or two to increase cache size and item lifetime?
Which kernel version shows symlinks right and which - wrong?
----
Wrong is 4.20.3... I'm not sure, using the "upcall" for the linux
client reading Metadata(owner+ACL) has ever worked right for the various forms of symlink/symlinkd/mountd/junction, since I see a message from me on the cifs list trying 4.10.1 and having resolution problems related to file-system redirects.
They "should" work the same way...as when I list them locally
since I'm accessing the machine as the same user (the linux machine is a samba4 PDC and I'm using the same user-id to access it on both sides).
The only time they fully work is when I assigned the mount
ownership to my local user that owns or is in a group that owns most of the links. Is there something flakey about reading the link info -- since it works when I mount it using 1 UID.
There may be cache timing issues as well, but they are too
ephemeral to pin down. The question marks seem to be fairly consistent on these names (all of which are some form of NT-pointer to another place - junctions
Re: upcalls seem to have problems with symlinks, junctions et al.
I have wanted to change this code and improve it for a while - one thing which is tricky is showing mode bits when no permissions to read permissions though, and we also need to clean up and simplify some of
this code. Let's follow up on this in a few days, if you are
flexible and can install some test patches
Quoted 47 lines · L A Walsh
Quoted 13 lines · L A Walsh
Quoted 13 lines
Quoted 13 lines
---
ls: cannot access 'Documents': Operation not supported
...
ls: cannot access 'Prog64': Operation not supportedd????????? ? ? ? Documents/
...
d????????? ? ? ? Prog64/
...
Depending on timing, Sometimes I will get most or all of them appearing the same as they would on windows.Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
If I can't resolve them more quickly, is there a know or two to increase cache size and item lifetime?
Which kernel version shows symlinks right and which - wrong?
----
Wrong is 4.20.3... I'm not sure, using the "upcall" for the linux
client reading Metadata(owner+ACL) has ever worked right for the various forms of symlink/symlinkd/mountd/junction, since I see a message from me on the cifs list trying 4.10.1 and having resolution problems related to file-system redirects.They "should" work the same way...as when I list them locally
since I'm accessing the machine as the same user (the linux machine is a samba4 PDC and I'm using the same user-id to access it on both sides).The only time they fully work is when I assigned the mount
ownership to my local user that owns or is in a group that owns most of the links. Is there something flakey about reading the link info -- since it works when I mount it using 1 UID.There may be cache timing issues as well, but they are too
ephemeral to pin down. The question marks seem to be fairly consistent on these names (all of which are some form of NT-pointer to another place - junctions
Signature · 3 lines
Thanks, Steve
Re: upcalls seem to have problems with symlinks, junctions et al.
Quoted 6 lines · Steve French
I have wanted to change this code and improve it for a while - one thing which is tricky is showing mode bits when no permissions to read permissions though, and we also need to clean up and simplify some of
this code. Let's follow up on this in a few days, if you are
flexible and can install some test patches
Usually links on linux have 777 permissions and aren't changeable, though things are changing.
As for testing -- should be able, but maybe not too fast.
Dunno if you rememeber, but I had some probs w/links back around Feb 2-3 2017...and you asked for a wireshark trace that I sent on on the 3rd...then we both got distracted.
(same list)
I sometimes have a long cycle time...you too? :-)
I could send the dump again... ;-) Seriously, I went back to using a single UID mount and that works around the problem, as in my case I have perms to read all of the links.
I just recently tried it again -- and hit the same type of behavior.
Re: upcalls seem to have problems with symlinks, junctions et al.
In reply to Steve FrenchQuoted 4 lines · Steve French
I have wanted to change this code and improve it for a while - one thing which is tricky is showing mode bits when no permissions to read permissions though,
---
How often would it be the case that you could read the content
of a directory, but not be able to read the permissions?
While that can happen on Windows... and not so much on linux
what is the content of the link considered? I.e. where it is pointing? Is that something that would usually be readable because it is pointing to the next item in the path and most people have the privilege to "Bypass Traverse Checking"?
I.e. -- is the access to the path information in a symlink controlled
by read permissions to item content, or is it considered part of the information needed for Traverse Checking?
I think my issue I had in Feb 2017 had to do with getting
different results depending on the type of the redirect -- junctions, mountd/vol, and symlink/symlinkd's -- and perhaps that might be related to how they store store the meta-information? I suppose there is similar redirect information at a DFS mount point?
Part of what I expect, "as a user", to be available to a remote
user, is based on how things look from a user perspective using the POSIXish-designed-Cygwin. If I'm signed in with the same domain-credential (and same SID), remotely, vs. locally, my expectation is that I should see the same access. That forms my expectations with CIFS.
Re: upcalls seem to have problems with symlinks, junctions et al.
In reply to Steve FrenchQuoted 6 lines · Steve French
I have wanted to change this code and improve it for a while - one thing which is tricky is showing mode bits when no permissions to read permissions though, and we also need to clean up and simplify some of
this code. Let's follow up on this in a few days, if you are
flexible and can install some test patches
FWIW, it seems to be symlinks that are unreadable when perms on the mount are set to 'multi-user'.
Using the following two examples from my system,
in cmd.exe they look like:
04/17/2017 08:45 AM <SYMLINKD> Share [S:\] 08/22/2018 03:08 PM <JUNCTION> Symbols [\\Bliss\Share\Symbols\]*
*
in Cygwin, they look like: (domain+user+groups are shortened to fit)
(domain name, B (Bliss), username 'l' and group 'lg')
lrwxrwxrwx 1 B\l B\lg 2 Apr 17 2017 Share -> /s lrwxrwxrwx 1 B\l B\lg 22 Aug 22 2018 Symbols -> //Bliss/Share/Symbols/
So both are links owned by me, one pointing to a locally mounted drive and the other a remotely mounted Share on the Domain server.
On linux with single user mounting, I see:
l--------- 1 law Administrators 0 Apr 17 2017 Share -> /??/S://
drwxr-xr-x 2 law Administrators 0 Dec 17 19:14 Symbols/
Amusingly, I can read directory Symbols which results in a read from the same server I'm doing the 'ls' from -- sorta the long way around.
Oddly, I can't read the permissions on the symlink even though they are usable (created directory named '??' and pointers and directories underneath it as appropriate).
If I mount the cifs share "multiuser", I lose the capacity to see or follow the symlinks as well as getting "Operation not supported" errors, but the junctions still work fine:
ls: cannot access 'Share': Operation not supported
d????????? ? ? ? ? ? Share/
drwxrwxr-x 2 root Domain Admins 0 Dec 17 19:14 Symbols/
1) the Operation not supported message is dubious, since I can read the link in single-user mount, even though the link shows no access.
2) If the permissions on the links really are unreadable, the 'l-----' format is certainly easier on the eyes than 'd?????'.
I'd like to see the multi-user mount give at least as much information as the single-user mount. That would make sense, wouldn't it? Can your patch/cleanup at least solve at problem?
**