Per-entry WSL EA queries during readdir cause severe ls -la slowdown on high-latency links (6.12)
Subsystems (2)
Hi Steve, linux-cifs,
I'm reporting what looks like a missing negative cache for WSL extended- attribute queries during directory listing. On a high-latency link this turns a 3ms operation into 21 seconds.
Environment
-----------
Client: Linux 6.12.0-211.16.1.el10_2.0.1.x86_64 (RHEL 10.2) Server: Windows Server, SMB 3.1.1
Link: site-to-site VPN between datacenters, ~76ms RTT
Share: FTP landing zone, 511 subdirectories in the parent
Mount options:
vers=3.1.1,cache=strict,upcall_target=app,uid=5000,forceuid,gid=5000, forcegid,file_mode=0664,dir_mode=0775,iocharset=utf8,soft,nounix,mapposix, reparse=nfs,nativesocket,symlink=native,rsize=4194304,wsize=4194304, acdirmax=300,acregmax=30
Symptom
-------
# time ls -la /mnt/share > /dev/null real 0m39.031s
An identical mount of the same share from a 5.10 client completes in 0.003s.
/proc/fs/cifs/Stats after a single ls -la (511 entries):
QueryDirectories: 3 total 0 failed
Creates: 512 total 0 failed Closes: 512 total 0 failed QueryInfos: 511 total 0 failed
Enumeration itself is 3 round trips. The client then issues a compounded create/query/close against every single entry.
Root cause
----------
smb3 tracepoints show what those per-entry queries actually are:
ls-766626 [000] ..... smb3_cmd_err: sid=0x... tid=0x29 cmd=16 mid=40511 \
status=0xc0000052 rc=-61 $ grep -oP '(?<=status=)\S+' trace | sort | uniq -c
268 0xc0000052
1 0xc000019c
1 0x800000060xc0000052 is STATUS_NO_EAS_ON_FILE, on cmd=16 (QUERY_INFO), 268 times in one listing. The client is probing for the WSL EAs ($LXUID/$LXGID/$LXMOD/$LXDEV) on every entry. The server has none and says so, every time, at 76ms per round trip.
Notably the mount forces uid/gid, so any ownership data returned would be discarded regardless.
Command distribution for the same listing:
$ grep -oP '(?<=cmd=)\d+' trace | sort -n | uniq -c
548 5 (CREATE)
282 6 (CLOSE)
6 14 (QUERY_DIRECTORY)
553 16 (QUERY_INFO)The CREATE/CLOSE imbalance suggests retries on top of the base cost.
Tracepoint event distribution for the same listing:
$ grep -oP 'smb3_\w+(?=:)' trace | sort | uniq -c | sort -rn | head
830 smb3_cmd_enter
830 smb3_add_credits
295 smb3_cmd_done
292 smb3_waitff_credits
277 smb3_enter
276 smb3_exit_done
270 smb3_cmd_err
9 smb3_tcon_ref
7 smb3_close_enter
3 smb3_query_dir_enterWhat doesn't help
-----------------
noserverino 511 -> 271 queries, 39s -> 21s (only thing that helped) symlink=none no change reparse=wsl no change vers=3.0 no change dir_cache_timeout no change (set post-mount, so possibly not applied)
There is no mount option to disable the WSL EA probing. Checking the module's parameter tables, wsl_query_ea/wsl_query_eas appear to be internal only.
Repeat listings within acdirmax=300 also take ~21s, which suggests the negative result is not being cached per-inode.
Suggestion
----------
Two options that would each solve this:
1. Cache the negative EA result. If a file returns STATUS_NO_EAS_ON_FILE, don't re-probe it on every stat within the attribute cache window. Arguably worth caching per-tcon as well, since a server that has no WSL EAs anywhere is the common case against Windows.
2. Add a mount option to skip WSL EA queries entirely. Mounts that force uid/gid/file_mode discard the results anyway.
Happy to provide full tracepoint output, run patches, or test against other dialects and servers. The reproducer is straightforward: any Windows share with a few hundred entries over a link with meaningful latency.
Thanks,
Gustavo Nobrega
Paulo Alcantara
maintainerQuoted 3 lines · Gustavo Nobrega
I'm reporting what looks like a missing negative cache for WSL extended- attribute queries during directory listing. On a high-latency link this turns a 3ms operation into 21 seconds.
Yes - we currently don't cache xattrs.
Quoted 42 lines
Environment
-----------Client: Linux 6.12.0-211.16.1.el10_2.0.1.x86_64 (RHEL 10.2) Server: Windows Server, SMB 3.1.1Link: site-to-site VPN between datacenters, ~76ms RTT
Share: FTP landing zone, 511 subdirectories in the parentMount options:
vers=3.1.1,cache=strict,upcall_target=app,uid=5000,forceuid,gid=5000, forcegid,file_mode=0664,dir_mode=0775,iocharset=utf8,soft,nounix,mapposix, reparse=nfs,nativesocket,symlink=native,rsize=4194304,wsize=4194304, acdirmax=300,acregmax=30Symptom
-------# time ls -la /mnt/share > /dev/null real 0m39.031sAn identical mount of the same share from a 5.10 client completes in 0.003s.
/proc/fs/cifs/Statsafter a single ls -la (511 entries):QueryDirectories: 3 total 0 failed
Creates: 512 total 0 failed Closes: 512 total 0 failed QueryInfos: 511 total 0 failedEnumeration itself is 3 round trips. The client then issues a compounded create/query/close against every single entry.
Root cause
----------
smb3 tracepoints show what those per-entry queries actually are:ls-766626 [000] ..... smb3_cmd_err: sid=0x... tid=0x29 cmd=16 mid=40511 \ status=0xc0000052 rc=-61$ grep -oP '(?<=status=)\S+' trace | sort | uniq -c 268 0xc0000052 1 0xc000019c 1 0x80000006
Quoted 4 lines
0xc0000052isSTATUS_NO_EAS_ON_FILE, on cmd=16 (QUERY_INFO), 268 times in one listing. The client is probing for the WSL EAs ($LXUID/$LXGID/$LXMOD/$LXDEV) on every entry. The server has none and says so, every time, at 76ms per round trip.
It's not that the client thinks the files have WSL EAs. For some reason the query directory response doesn't seem to satisfy the client in terms of metadata in order to instantiate those files during directory listing. That is why it is sending query info for each file to get more metadata information. The WSL EAs are queried unconditionally when querying information about a file. This is done for saving any potentially additional roundtrips in case they have.
The extra query info usually when the client finds a file with a reparse attribute set, 'cifsacl' or 'modefromsid' mount option being used, 'mfsymlinks' mount option is set and a regular file could be a potential symlink, etc. If that isn't the case, then the bug is somewhere else.
I'll look at the network traces sent off-list and then let you know.
Thanks for the detailed report!
A message that is not in the archive (sent off-list or not yet received): CAJr=uVQ2VGLuHqV2oYsgCxj_EkqdtJAngq8BHUgq_4iU_MNCZg@mail.gmail.com
Paulo Alcantara
maintainerHi Gustavo,
I was able to reproduce your problem with coreutils-9.10-5.fc44.x86_64 and mainline kernel.
The ls -la command is, in fact, calling llistxattr(2) on every file returned by getdents64(2). That's why you're seeing those extra requests being sent to the server. See below
$ mount.cifs //srv/share /mnt/1 -o ...
$ strace -e llistxattr -f ls -la /mnt/1 >/dev/null
llistxattr("/mnt/1", "", 152) = 0
llistxattr("/mnt/1/.", "", 152) = 0
llistxattr("/mnt/1/..", "security.selinux\0", 152) = 17
llistxattr("/mnt/1/008.174041", "", 152) = 0
llistxattr("/mnt/1/008.5301", "", 152) = 0
llistxattr("/mnt/1/008.5307", "", 152) = 0
llistxattr("/mnt/1/008.5313", "", 152) = 0
llistxattr("/mnt/1/008.5384", "", 152) = 0
llistxattr("/mnt/1/008.6320", "", 152) = 0
llistxattr("/mnt/1/009.6781", "", 152) = 0
llistxattr("/mnt/1/012.8056", "", 152) = 0
...That has nothing to do with the client trying to get WSL EAs on the files. By using 'reparse=' mount option will also not help because it is only used for creating special files and symlinks.
I don't know why why llistxattr(2) is being called by default when running ls -la, sorry.
One way to prevent the above from happening is by using 'nouser_xattr' mount option, which would disable getfattr/setfattr entirely, so ls -la wouldn't attempt to llistxattr(2) each file over directory listing.
Hi Paulo,
Confirmed, nouser_xattr fixes it completely:
before: real 0m39.031s
after: real 0m0.367s
QueryInfos: 1 total 0 failed (was 511)
And strace shows no llistxattr(2) calls at all with that option, so ls stops asking once the mount reports no xattr support.
The two hosts:
old: CentOS Linux 7.9.2009
kernel 3.10.0-1160.119.1.el7.x86_64
coreutils-8.22-24.el7_9.2new: Rocky Linux 10.2
kernel 6.12.0-211.16.1.el10_2.0.1.x86_64
coreutils-9.5-7.el10The tell was in the listing output all along, and I missed it:
old: drwxrwxr-x 2 fwservice fwservice ...
new: drwxrwxr-x. 2 fwservice fwservice ...
The trailing dot on the new host is ls reporting a security context on the files, which is exactly the llistxattr(2) result you found. The old host prints no dot and issues no such calls.
I don't know which coreutils change introduced this either, but 8.22 to 9.5 is a wide enough gap that it could be any of several.
Thanks very much for tracking it down, and apologies for pointing you at the WSL EA path, I'd read the STATUS_NO_EAS_ON_FILE responses as the cause rather than a consequence.
All the best,
Gustavo
Quoted 35 lines · Paulo Alcantara
Hi Gustavo,
I was able to reproduce your problem with coreutils-9.10-5.fc44.x86_64 and mainline kernel.
The
ls -lacommand is, in fact, calling llistxattr(2) on every file returned by getdents64(2). That's why you're seeing those extra requests being sent to the server. See below$ mount.cifs //srv/share /mnt/1 -o ... $ strace -e llistxattr -f ls -la /mnt/1 >/dev/null llistxattr("/mnt/1", "", 152) = 0 llistxattr("/mnt/1/.", "", 152) = 0 llistxattr("/mnt/1/..", "security.selinux\0", 152) = 17 llistxattr("/mnt/1/008.174041", "", 152) = 0 llistxattr("/mnt/1/008.5301", "", 152) = 0 llistxattr("/mnt/1/008.5307", "", 152) = 0 llistxattr("/mnt/1/008.5313", "", 152) = 0 llistxattr("/mnt/1/008.5384", "", 152) = 0 llistxattr("/mnt/1/008.6320", "", 152) = 0 llistxattr("/mnt/1/009.6781", "", 152) = 0 llistxattr("/mnt/1/012.8056", "", 152) = 0 ...That has nothing to do with the client trying to get WSL EAs on the files. By using 'reparse=' mount option will also not help because it is only used for creating special files and symlinks.
I don't know why why llistxattr(2) is being called by default when running
ls -la, sorry.One way to prevent the above from happening is by using '
nouser_xattr' mount option, which would disable getfattr/setfattr entirely, sols -lawouldn't attempt to llistxattr(2) each file over directory listing.