Skip to contentSkip to conversation

Per-entry WSL EA queries during readdir cause severe ls -la slowdown on high-latency links (6.12)

DISCUSSION Gustavo Nobrega 2 participants 4 messages Aug 5 – Aug 9 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi Steve, linux-cifs,

I'm reporting what looks like a missing negative cache for WSL extended- attribute queries during directory listing. On a high-latency link this turns a 3ms operation into 21 seconds.

Environment
-----------

Client:  Linux 6.12.0-211.16.1.el10_2.0.1.x86_64 (RHEL 10.2)
Server:  Windows Server, SMB 3.1.1

Link: site-to-site VPN between datacenters, ~76ms RTT
Share: FTP landing zone, 511 subdirectories in the parent

Mount options:
vers=3.1.1,cache=strict,upcall_target=app,uid=5000,forceuid,gid=5000, forcegid,file_mode=0664,dir_mode=0775,iocharset=utf8,soft,nounix,mapposix, reparse=nfs,nativesocket,symlink=native,rsize=4194304,wsize=4194304, acdirmax=300,acregmax=30

Symptom
-------

  # time ls -la /mnt/share > /dev/null
  real    0m39.031s

An identical mount of the same share from a 5.10 client completes in 0.003s.

/proc/fs/cifs/Stats after a single ls -la (511 entries):

QueryDirectories: 3 total 0 failed

  Creates:        512 total 0 failed
  Closes:         512 total 0 failed
  QueryInfos:     511 total 0 failed

Enumeration itself is 3 round trips. The client then issues a compounded create/query/close against every single entry.

Root cause
----------
smb3 tracepoints show what those per-entry queries actually are:

  ls-766626 [000] ..... smb3_cmd_err: sid=0x... tid=0x29 cmd=16 mid=40511 \
      status=0xc0000052 rc=-61
  $ grep -oP '(?<=status=)\S+' trace | sort | uniq -c
      268 0xc0000052
        1 0xc000019c
        1 0x80000006

0xc0000052 is STATUS_NO_EAS_ON_FILE, on cmd=16 (QUERY_INFO), 268 times in one listing. The client is probing for the WSL EAs ($LXUID/$LXGID/$LXMOD/$LXDEV) on every entry. The server has none and says so, every time, at 76ms per round trip.

Notably the mount forces uid/gid, so any ownership data returned would be discarded regardless.

Command distribution for the same listing:

  $ grep -oP '(?<=cmd=)\d+' trace | sort -n | uniq -c
      548 5    (CREATE)
      282 6    (CLOSE)
        6 14   (QUERY_DIRECTORY)
      553 16   (QUERY_INFO)

The CREATE/CLOSE imbalance suggests retries on top of the base cost.

Tracepoint event distribution for the same listing:

  $ grep -oP 'smb3_\w+(?=:)' trace | sort | uniq -c | sort -rn | head
      830 smb3_cmd_enter
      830 smb3_add_credits
      295 smb3_cmd_done
      292 smb3_waitff_credits
      277 smb3_enter
      276 smb3_exit_done
      270 smb3_cmd_err
        9 smb3_tcon_ref
        7 smb3_close_enter
        3 smb3_query_dir_enter

What doesn't help
-----------------

  noserverino          511 -> 271 queries, 39s -> 21s  (only thing that helped)
  symlink=none         no change
  reparse=wsl          no change
  vers=3.0             no change
  dir_cache_timeout    no change (set post-mount, so possibly not applied)

There is no mount option to disable the WSL EA probing. Checking the module's parameter tables, wsl_query_ea/wsl_query_eas appear to be internal only.

Repeat listings within acdirmax=300 also take ~21s, which suggests the negative result is not being cached per-inode.

Suggestion
----------
Two options that would each solve this:

1. Cache the negative EA result. If a file returns STATUS_NO_EAS_ON_FILE, don't re-probe it on every stat within the attribute cache window. Arguably worth caching per-tcon as well, since a server that has no WSL EAs anywhere is the common case against Windows.

2. Add a mount option to skip WSL EA queries entirely. Mounts that force uid/gid/file_mode discard the results anyway.

Happy to provide full tracepoint output, run patches, or test against other dialects and servers. The reproducer is straightforward: any Windows share with a few hundred entries over a link with meaningful latency.

Thanks,
Gustavo Nobrega

maintainer
#1 lore raw
Quoted 3 lines · Gustavo Nobrega

I'm reporting what looks like a missing negative cache for WSL extended- attribute queries during directory listing. On a high-latency link this turns a 3ms operation into 21 seconds.

Yes - we currently don't cache xattrs.

Quoted 42 lines

Environment
-----------

Client:  Linux 6.12.0-211.16.1.el10_2.0.1.x86_64 (RHEL 10.2)
Server:  Windows Server, SMB 3.1.1

Link: site-to-site VPN between datacenters, ~76ms RTT
Share: FTP landing zone, 511 subdirectories in the parent

Mount options:
vers=3.1.1,cache=strict,upcall_target=app,uid=5000,forceuid,gid=5000, forcegid,file_mode=0664,dir_mode=0775,iocharset=utf8,soft,nounix,mapposix, reparse=nfs,nativesocket,symlink=native,rsize=4194304,wsize=4194304, acdirmax=300,acregmax=30

Symptom
-------

  # time ls -la /mnt/share > /dev/null
  real    0m39.031s

An identical mount of the same share from a 5.10 client completes in 0.003s.

/proc/fs/cifs/Stats after a single ls -la (511 entries):

QueryDirectories: 3 total 0 failed

  Creates:        512 total 0 failed
  Closes:         512 total 0 failed
  QueryInfos:     511 total 0 failed

Enumeration itself is 3 round trips. The client then issues a compounded create/query/close against every single entry.

Root cause
----------
smb3 tracepoints show what those per-entry queries actually are:

  ls-766626 [000] ..... smb3_cmd_err: sid=0x... tid=0x29 cmd=16 mid=40511 \
      status=0xc0000052 rc=-61
  $ grep -oP '(?<=status=)\S+' trace | sort | uniq -c
      268 0xc0000052
        1 0xc000019c
        1 0x80000006
Quoted 4 lines

0xc0000052 is STATUS_NO_EAS_ON_FILE, on cmd=16 (QUERY_INFO), 268 times in one listing. The client is probing for the WSL EAs ($LXUID/$LXGID/$LXMOD/$LXDEV) on every entry. The server has none and says so, every time, at 76ms per round trip.

It's not that the client thinks the files have WSL EAs. For some reason the query directory response doesn't seem to satisfy the client in terms of metadata in order to instantiate those files during directory listing. That is why it is sending query info for each file to get more metadata information. The WSL EAs are queried unconditionally when querying information about a file. This is done for saving any potentially additional roundtrips in case they have.

The extra query info usually when the client finds a file with a reparse attribute set, 'cifsacl' or 'modefromsid' mount option being used, 'mfsymlinks' mount option is set and a regular file could be a potential symlink, etc. If that isn't the case, then the bug is somewhere else.

I'll look at the network traces sent off-list and then let you know.

Thanks for the detailed report!

A message that is not in the archive (sent off-list or not yet received): CAJr=uVQ2VGLuHqV2oYsgCxj_EkqdtJAngq8BHUgq_4iU_MNCZg@mail.gmail.com

maintainer
#3 lore raw

Hi Gustavo,

I was able to reproduce your problem with coreutils-9.10-5.fc44.x86_64 and mainline kernel.

The ls -la command is, in fact, calling llistxattr(2) on every file returned by getdents64(2). That's why you're seeing those extra requests being sent to the server. See below

	$ mount.cifs //srv/share /mnt/1 -o ...
	$ strace -e llistxattr -f ls -la /mnt/1 >/dev/null
	llistxattr("/mnt/1", "", 152)           = 0
	llistxattr("/mnt/1/.", "", 152)         = 0
	llistxattr("/mnt/1/..", "security.selinux\0", 152) = 17
	llistxattr("/mnt/1/008.174041", "", 152) = 0
	llistxattr("/mnt/1/008.5301", "", 152)  = 0
	llistxattr("/mnt/1/008.5307", "", 152)  = 0
	llistxattr("/mnt/1/008.5313", "", 152)  = 0
	llistxattr("/mnt/1/008.5384", "", 152)  = 0
	llistxattr("/mnt/1/008.6320", "", 152)  = 0
	llistxattr("/mnt/1/009.6781", "", 152)  = 0
	llistxattr("/mnt/1/012.8056", "", 152)  = 0
	...

That has nothing to do with the client trying to get WSL EAs on the files. By using 'reparse=' mount option will also not help because it is only used for creating special files and symlinks.

I don't know why why llistxattr(2) is being called by default when running ls -la, sorry.

One way to prevent the above from happening is by using 'nouser_xattr' mount option, which would disable getfattr/setfattr entirely, so ls -la wouldn't attempt to llistxattr(2) each file over directory listing.

#4 lore raw

Hi Paulo,

Confirmed, nouser_xattr fixes it completely:

before: real 0m39.031s
after: real 0m0.367s

QueryInfos: 1 total 0 failed (was 511)

And strace shows no llistxattr(2) calls at all with that option, so ls stops asking once the mount reports no xattr support.

The two hosts:

old: CentOS Linux 7.9.2009

        kernel 3.10.0-1160.119.1.el7.x86_64
        coreutils-8.22-24.el7_9.2

new: Rocky Linux 10.2

        kernel 6.12.0-211.16.1.el10_2.0.1.x86_64
        coreutils-9.5-7.el10

The tell was in the listing output all along, and I missed it:

old: drwxrwxr-x 2 fwservice fwservice ...
new: drwxrwxr-x. 2 fwservice fwservice ...

The trailing dot on the new host is ls reporting a security context on the files, which is exactly the llistxattr(2) result you found. The old host prints no dot and issues no such calls.

I don't know which coreutils change introduced this either, but 8.22 to 9.5 is a wide enough gap that it could be any of several.

Thanks very much for tracking it down, and apologies for pointing you at the WSL EA path, I'd read the STATUS_NO_EAS_ON_FILE responses as the cause rather than a consequence.

All the best,

Gustavo

Quoted 35 lines · Paulo Alcantara

Hi Gustavo,

I was able to reproduce your problem with coreutils-9.10-5.fc44.x86_64 and mainline kernel.

The ls -la command is, in fact, calling llistxattr(2) on every file returned by getdents64(2). That's why you're seeing those extra requests being sent to the server. See below

        $ mount.cifs //srv/share /mnt/1 -o ...
        $ strace -e llistxattr -f ls -la /mnt/1 >/dev/null
        llistxattr("/mnt/1", "", 152)           = 0
        llistxattr("/mnt/1/.", "", 152)         = 0
        llistxattr("/mnt/1/..", "security.selinux\0", 152) = 17
        llistxattr("/mnt/1/008.174041", "", 152) = 0
        llistxattr("/mnt/1/008.5301", "", 152)  = 0
        llistxattr("/mnt/1/008.5307", "", 152)  = 0
        llistxattr("/mnt/1/008.5313", "", 152)  = 0
        llistxattr("/mnt/1/008.5384", "", 152)  = 0
        llistxattr("/mnt/1/008.6320", "", 152)  = 0
        llistxattr("/mnt/1/009.6781", "", 152)  = 0
        llistxattr("/mnt/1/012.8056", "", 152)  = 0
        ...

That has nothing to do with the client trying to get WSL EAs on the files. By using 'reparse=' mount option will also not help because it is only used for creating special files and symlinks.

I don't know why why llistxattr(2) is being called by default when running ls -la, sorry.

One way to prevent the above from happening is by using 'nouser_xattr' mount option, which would disable getfattr/setfattr entirely, so ls -la wouldn't attempt to llistxattr(2) each file over directory listing.