Skip to content

Readdir improvements

[email protected] linux-nfs 8 versions – New

Get series with b4
b4 am [email protected]

Versions newest first · linked by author and title

  1. v10 Latest26 patches 0 repliesNew
  2. v927 patches 16 repliesSuperseded
  3. v824 patches 0 repliesSuperseded
Earlier versions · 5
VersionPostedPatchesRepliesStatus
v721 patches35 repliesSuperseded
v613 patches12 repliesSuperseded
v45 patches2 repliesSuperseded
v32 patches0 repliesSuperseded
v12 patches0 repliesSuperseded

Patches in v10 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/26NFS: Return valid errors from nfs2/3_decode_dirent()fs/nfs/nfs2xdr.c fs/nfs/nfs3xdr.c+7 -160 replies
2/26NFS: constify nfs_server_capable() and nfs_have_writebacks()include/linux/nfs_fs.h+3 -40 replies
3/26NFS: Trace lookup revalidation failurefs/nfs/dir.c+5 -120 replies
4/26NFS: Initialise the readdir verifier as best we can in nfs_opendir()fs/nfs/dir.c+1 -00 replies
5/26NFS: Use kzalloc() to avoid initialising the nfs_open_dir_contextfs/nfs/dir.c+4 -70 replies
6/26NFS: Calculate page offsets algorithmicallyfs/nfs/dir.c+13 -50 replies
7/26NFS: Store the change attribute in the directory page cachefs/nfs/dir.c+37 -310 replies
8/26NFS: Don't re-read the entire page cache to find the next cookiefs/nfs/dir.c include/linux/nfs_fs.h+8 -30 replies
9/26NFS: Don't advance the page pointer unless the page is fullfs/nfs/dir.c+22 -100 replies
10/26NFS: Adjust the amount of readahead performed by NFS readdirfs/nfs/dir.c include/linux/nfs_fs.h+53 -10 replies
11/26NFS: If the cookie verifier changes, we must invalidate the page cachefs/nfs/dir.c+6 -10 replies
12/26NFS: Simplify nfs_readdir_xdr_to_array()fs/nfs/dir.c+11 -180 replies
13/26NFS: Reduce use of uncached readdirfs/nfs/dir.c+3 -200 replies
14/26NFS: Improve heuristic for readdirplusfs/nfs/dir.c fs/nfs/inode.c fs/nfs/internal.h +2 more+58 -360 replies
15/26NFS: Don't ask for readdirplus unless it can help nfs_getattr()fs/nfs/inode.c+25 -200 replies
16/26NFSv4: Ask for a full XDR buffer of readdir goodnessfs/nfs/nfs3xdr.c fs/nfs/nfs4xdr.c+7 -60 replies
17/26NFS: Readdirplus can't help lookup for case insensitive filesystemsfs/nfs/dir.c+2 -00 replies
18/26NFS: Don't request readdirplus when revalidation was forcedfs/nfs/dir.c+16 -100 replies
19/26NFS: Add basic readdir tracingfs/nfs/dir.c fs/nfs/nfstrace.h+80 -10 replies
20/26NFS: Trace effects of readdirplus on the dcachefs/nfs/dir.c fs/nfs/nfstrace.h+8 -00 replies
21/26NFS: Trace effects of the readdirplus heuristicfs/nfs/dir.c fs/nfs/nfstrace.h+60 -10 replies
22/26NFS: Clean up page array initialisation/freefs/nfs/dir.c+6 -100 replies
23/26NFS: Convert readdir page cache to use a cookie based indexfs/nfs/Kconfig fs/nfs/dir.c include/linux/nfs_fs.h+69 -860 replies
24/26NFS: Fix up forced readdirplusfs/nfs/dir.c fs/nfs/nfstrace.h include/linux/nfs_fs.h+41 -170 replies
25/26NFS: Optimise away the previous cookie fieldfs/nfs/dir.c fs/nfs/nfs2xdr.c fs/nfs/nfs3xdr.c +2 more+15 -170 replies
26/26NFS: Cache all entries in the readdirplus replyfs/nfs/dir.c+26 -140 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v10 yet.

Read the v10 thread

Cover letter [PATCH v10 00/26] Readdir improvements

From: Trond Myklebust <[email protected]>

The current NFS readdir code will always try to maximise the amount of readahead it performs on the assumption that we can cache anything that isn't immediately read by the process.
There are several cases where this assumption breaks down, including when the 'ls -l' heuristic kicks in to try to force use of readdirplus as a batch replacement for lookup/getattr.

This series also implement Ben's page cache filter to ensure that we can improve the ability to share cached data between processes that are reading the same directory at the same time, and to avoid live-locks when the directory is simultaneously changing.

--
v2: Remove reset of dtsize when NFS_INO_FORCE_READDIR is set
v3: Avoid excessive window shrinking in uncached_readdir case
v4: Track 'ls -l' cache hit/miss statistics
Improved algorithm for falling back to uncached readdir Skip readdirplus when files are being written to
v5: bugfixes
Skip readdirplus when the acdirmax/acregmax values are low Request a full XDR buffer when doing READDIRPLUS
v6: Add tracing
Don't have lookup request readdirplus when it won't help
v7: Implement Ben's page cache filter
Reduce the use of uncached readdir
Change indexing of the page cache to improve seekdir() performance.
v8: Reduce the page cache overhead by shrinking the cookie hashvalue size
Incorporate other feedback from Anna, Ben and Neil
Fix nfs2/3_decode_dirent return values
Fix the change attribute value set in nfs_readdir_page_get_next()
v9: Address bugs that were hit when testing with large directories
Misc cleanups
v10: Reinstate cache invalidations for directories, since Olga showed
that the NFSv3 ctime resolution can be insufficient to record rapid changes to a directory.

Trond Myklebust (26):
NFS: Return valid errors from nfs2/3_decode_dirent()
NFS: constify nfs_server_capable() and nfs_have_writebacks()
NFS: Trace lookup revalidation failure
NFS: Initialise the readdir verifier as best we can in nfs_opendir()
NFS: Use kzalloc() to avoid initialising the nfs_open_dir_context
NFS: Calculate page offsets algorithmically
NFS: Store the change attribute in the directory page cache
NFS: Don't re-read the entire page cache to find the next cookie
NFS: Don't advance the page pointer unless the page is full
NFS: Adjust the amount of readahead performed by NFS readdir
NFS: If the cookie verifier changes, we must invalidate the page cache
NFS: Simplify nfs_readdir_xdr_to_array()
NFS: Reduce use of uncached readdir
NFS: Improve heuristic for readdirplus
NFS: Don't ask for readdirplus unless it can help nfs_getattr()
NFSv4: Ask for a full XDR buffer of readdir goodness
NFS: Readdirplus can't help lookup for case insensitive filesystems
NFS: Don't request readdirplus when revalidation was forced
NFS: Add basic readdir tracing
NFS: Trace effects of readdirplus on the dcache
NFS: Trace effects of the readdirplus heuristic
NFS: Clean up page array initialisation/free
NFS: Convert readdir page cache to use a cookie based index
NFS: Fix up forced readdirplus
NFS: Optimise away the previous cookie field
NFS: Cache all entries in the readdirplus reply

 fs/nfs/Kconfig          |   4 +
 fs/nfs/dir.c            | 595 ++++++++++++++++++++++++----------------
 fs/nfs/inode.c          |  45 +--
 fs/nfs/internal.h       |   4 +-
 fs/nfs/nfs2xdr.c        |   3 +-
 fs/nfs/nfs3xdr.c        |  29 +-
 fs/nfs/nfs4xdr.c        |   7 +-
 fs/nfs/nfstrace.h       | 123 ++++++++-
 include/linux/nfs_fs.h  |  17 +-
 include/linux/nfs_xdr.h |   3 +-
 10 files changed, 535 insertions(+), 295 deletions(-)
Signature · 1 line
2.35.1