Skip to content

NFS client readdir per-page validation

Benjamin Coddington linux-nfs oe-kbuild-all 1 version In review

Get series with b4
b4 am [email protected]

Versions newest first

  1. v1 Latest10 patches 5 repliesIn review

Patches in v1 No applies recorded yet

#Subject and filesSizeRepliesApplied
1/10NFS: save the directory's change attribute on pagecache pagesfs/nfs/dir.c+31 -00 replies
2/10NFSv4: Send GETATTR with READDIRfs/nfs/nfs42proc.c fs/nfs/nfs4proc.c fs/nfs/nfs4xdr.c +2 more+33 -90 replies
3/10NFS: Add a struct to track readdir pagecache locationfs/nfs/dir.c include/linux/nfs_fs.h+23 -170 replies
4/10NFS: Keep the readdir pagecache cursor updatedfs/nfs/dir.c+25 -43 replies
5/10NFS: readdir per-page cache validationfs/nfs/dir.c+57 -140 replies
6/10NFS: stash the readdir pagecache cursor on the open directory contextfs/nfs/dir.c include/linux/nfs_fs.h+9 -80 replies
7/10NFS: Support headless readdir pagecache pagesfs/nfs/dir.c+26 -192 replies
8/10NFS: Reset pagecache cursor on llseekfs/nfs/dir.c+2 -00 replies
9/10NFS: Remove nfs_readdir_dont_search_cache()fs/nfs/dir.c+0 -170 replies
10/10NFS: Revalidate the directory pagecache on every nfs_readdir()fs/nfs/dir.c+3 -50 replies

Reviews trailers per patch

No Reviewed-by, Acked-by or Tested-by trailers for v1 yet.

Read the v1 thread

Cover letter [PATCH v1 00/10] NFS client readdir per-page validation

Due to the constraint that the NFS readdir page cache must contain every entry in cookie order from zero up to the entry of interest, the time or operations required to complete a directory listing increase exponentially with the size of the directory if the client is unable to keep the pagecache stable. The pagecache can be invalidated by a changing directory, or by memory pressure on the client. This can cause some trouble for the NFS client reading large directories over slow connections.

We have a hueristic that allows eventual completion, but it only works as long as there are no other readers simultaneously filling the pagecache.

I think we can resolve this problem by implementing per-page validation. By storing the directory's change version on the page, and checking for changes to the directory on every READDIR, we can validate pages against each reader's version of entry aligment. Rather than attempting to assemble the entire directory in a consistent manner in the pagecache, we can just retrieve the section we're interested in emitting.

This set is a first pass at implementing this idea. Please help me pound it into acceptable shape or point out problems! Thanks for any feedback.

Here's a small program that does a great job of demonstraing the client's current readdir pagecache performance problem by dropping the directory's pagecache at an interval while trying to emit every entry:

#define _GNU_SOURCE
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <sched.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <sys/syscall.h>
#include <signal.h>

#define BUF_SIZE 1024
#define EVICT_INTERVAL 5

int evict_pagecache(int fd) {
                return posix_fadvise(fd, 0, 0, POSIX_FADV_DONTNEED);
}

int main(int argc, char **argv)

{
        int dir_fd;
        pid_t pid;
        cpu_set_t *cpusetp = CPU_ALLOC(2);
        off_t off;

        char buf[BUF_SIZE];

        if (argc < 2) { printf("%s <dir>\n", argv[0]); return 1; }

        dir_fd = open(argv[1], O_RDONLY|O_DIRECTORY|O_CLOEXEC);
        if (dir_fd < 0) { printf("cannot open dir\n"); return 1; }

        pid = fork();
        if (pid == 0) {
                CPU_SET(1, cpusetp);
                sched_setaffinity(0, sizeof(cpu_set_t), cpusetp);
                do {
                        evict_pagecache(dir_fd);
                        off = lseek(dir_fd, 0, SEEK_CUR);
                        printf("currently at %llu\n", off);
                        usleep(EVICT_INTERVAL * 1000000);
                } while (1);
        } else {
                CPU_SET(0, cpusetp);
                sched_setaffinity(0, sizeof(cpu_set_t), cpusetp);
                while (syscall(SYS_getdents, dir_fd, buf, BUF_SIZE)) {}
                kill(pid, SIGINT);
        }

        close(dir_fd);
        return 0;
}

Benjamin Coddington (10):
NFS: save the directory's change attribute on pagecache pages
NFSv4: Send GETATTR with READDIR
NFS: Add a struct to track readdir pagecache location
NFS: Keep the readdir pagecache cursor updated
NFS: readdir per-page cache validation
NFS: stash the readdir pagecache cursor on the open directory context
NFS: Support headless readdir pagecache pages
NFS: Reset pagecache cursor on llseek
NFS: Remove nfs_readdir_dont_search_cache()
NFS: Revalidate the directory pagecache on every nfs_readdir()

 fs/nfs/dir.c              | 210 +++++++++++++++++++++++++++-----------
 fs/nfs/nfs42proc.c        |   2 +-
 fs/nfs/nfs4proc.c         |  27 +++--
 fs/nfs/nfs4xdr.c          |   6 ++
 include/linux/nfs_fs.h    |   8 +-
 include/linux/nfs_fs_sb.h |   5 +
 include/linux/nfs_xdr.h   |   2 +
 7 files changed, 188 insertions(+), 72 deletions(-)
Signature · 1 line
2.25.4