Readdir enhancements
Subsystems (2)
[email protected]
maintainerv5 cover Readdir enhancements
From: Trond Myklebust <[email protected]>
The following patch series performs a number of cleanups on the readdir code.
It also adds support for 1MB readdir RPC calls on-the-wire, and modifies the caching code to ensure that we cache the entire contents of that 1MB call (instead of discarding the data that doesn't fit into a single page).
For filesystems that use ordered readdir cookie schemes (e.g. XFS), it optimises searching for cookies in the client's page cache by skipping over pages that contain cookie values that are not in the range we are searching for.
Finally, it improves scalability when dealing with very large directories by turning off caching when those directories are changing, so as to avoid the need for a linear search on the client of the entire directory when looking for the first entry pointed to by the current file descriptor offset.
v2: Fix the handling of the NFSv3/v4 directory verifier.
v3: Optimise searching when the readdir cookies are seen to be ordered.
v4: Optimise performance for large directories that are changing.
Add in llseek dependency patches.
v5: Integrate Olga's patch for the READDIR security label handling.
Record more entries in the uncached readdir case. Bump the max number of pages to 512, but allocate them on demand in case the readdir RPC call returns fewer entries.
Olga Kornievskaia (1):
NFSv4.2: condition READDIR's mask for security label based on LSM
state
Trond Myklebust (21):
NFS: Remove unnecessary inode locking in nfs_llseek_dir()
NFS: Remove unnecessary inode lock in nfs_fsync_dir()
NFS: Ensure contents of struct nfs_open_dir_context are consistent
NFS: Clean up readdir struct nfs_cache_array
NFS: Clean up nfs_readdir_page_filler()
NFS: Clean up directory array handling
NFS: Don't discard readdir results
NFS: Remove unnecessary kmap in nfs_readdir_xdr_to_array()
NFS: Replace kmap() with kmap_atomic() in nfs_readdir_search_array()
NFS: Simplify struct nfs_cache_array_entry
NFS: Support larger readdir buffers
NFS: More readdir cleanups
NFS: nfs_do_filldir() does not return a value
NFS: Reduce readdir stack usage
NFS: Cleanup to remove nfs_readdir_descriptor_t typedef
NFS: Allow the NFS generic code to pass in a verifier to readdir
NFS: Handle NFS4ERR_NOT_SAME and NFSERR_BADCOOKIE from readdir calls
NFS: Improve handling of directory verifiers
NFS: Optimisations for monotonically increasing readdir cookies
NFS: Reduce number of RPC calls when doing uncached readdir
NFS: Do uncached readdir when we're seeking a cookie in an empty page
cache
fs/nfs/client.c | 4 +- fs/nfs/dir.c | 734 +++++++++++++++++++++++++--------------- fs/nfs/inode.c | 7 - fs/nfs/internal.h | 6 - fs/nfs/nfs3proc.c | 35 +- fs/nfs/nfs4proc.c | 48 +-- fs/nfs/proc.c | 18 +- include/linux/nfs_fs.h | 9 +- include/linux/nfs_xdr.h | 17 +- 9 files changed, 541 insertions(+), 337 deletions(-)
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 1/22 NFS: Remove unnecessary inode locking in nfs_llseek_dir()
From: Trond Myklebust <[email protected]>
Remove the contentious inode lock, and instead provide thread safety using the file->f_lock spinlock.
--- fs/nfs/dir.c | 9 ++++----- 1 file changed, 4 insertions(+), 5 deletions(-)
fs/nfs/dir.c +4 -5
@@ -955,7 +955,6 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)static loff_t nfs_llseek_dir (struct file *filp,loff_t offset,int whence) { -struct inode *inode =file_inode (filp);struct nfs_open_dir_context *dir_ctx = filp->private_data; dfprintk(FILE, "NFS: llseek dir(%pD2, %lld, %d)\n",
@@ -967,15 +966,15 @@ static loff_t nfs_llseek_dir(struct file *filp, loff_t offset, int whence)case SEEK_SET :if (offset <0 )return -EINVAL ; -inode_lock (inode); +spin_lock (&filp->f_lock);break ;case SEEK_CUR :if (offset ==0 )return filp->f_pos; -inode_lock (inode); +spin_lock (&filp->f_lock); offset += filp->f_pos;if (offset <0 ) { -inode_unlock (inode); +spin_unlock (&filp->f_lock);return -EINVAL ; } }
@@ -987,7 +986,7 @@ static loff_t nfs_llseek_dir(struct file *filp, loff_t offset, int whence) dir_ctx->dir_cookie =0 ; dir_ctx->duped =0 ; } -inode_unlock (inode); +spin_unlock (&filp->f_lock);return offset; }
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 2/22 NFS: Remove unnecessary inode lock in nfs_fsync_dir()
From: Trond Myklebust <[email protected]>
nfs_inc_stats() is already thread-safe, and there are no other reasons to hold the inode lock here.
--- fs/nfs/dir.c | 6 +----- 1 file changed, 1 insertion(+), 5 deletions(-)
fs/nfs/dir.c +1 -5
@@ -997,13 +997,9 @@ static loff_t nfs_llseek_dir(struct file *filp, loff_t offset, int whence)static int nfs_fsync_dir (struct file *filp,loff_t start,loff_t end,int datasync) { -struct inode *inode =file_inode (filp); -dfprintk (FILE ,"NFS: fsync dir(%pD2) datasync %d\n" , filp, datasync); -inode_lock (inode); -nfs_inc_stats (inode,NFSIOS_VFSFSYNC ); -inode_unlock (inode); +nfs_inc_stats (file_inode (filp),NFSIOS_VFSFSYNC );return 0 ; }
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 3/22 NFSv4.2: condition READDIR's mask for security label based on LSM state
From: Olga Kornievskaia <[email protected]>
Currently, the client will always ask for security_labels if the server returns that it supports that feature regardless of any LSM modules (such as Selinux) enforcing security policy. This adds performance penalty to the READDIR operation.
Client adjusts superblock's support of the security_label based on the server's support but also current client's configuration of the LSM modules. Thus, prior to using the default bitmask in READDIR, this patch checks the server's capabilities and then instructs READDIR to remove FATTR4_WORD2_SECURITY_LABEL from the bitmask.
v5: fixing silly mistakes of the rushed v4
v4: simplifying logic
v3: changing label's initialization per Ondrej's comment
v2: dropping selinux hook and using the sb cap.
--- fs/nfs/nfs4proc.c | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-)
fs/nfs/nfs4proc.c +8 -2
@@ -4965,12 +4965,12 @@ static int _nfs4_proc_readdir(struct dentry *dentry, const struct cred *cred, u64 cookie, struct page **pages, unsigned int count, bool plus) { struct inode *dir = d_inode(dentry); +struct nfs_server *server =NFS_SERVER (dir);struct nfs4_readdir_arg args = { .fh =NFS_FH (dir), .pages = pages, .pgbase =0 , .count = count, - .bitmask =NFS_SERVER (d_inode (dentry))->attr_bitmask, .plus = plus, };struct nfs4_readdir_res res;
@@ -4985,9 +4985,15 @@ static int _nfs4_proc_readdir(struct dentry *dentry, const struct cred *cred,dprintk ("%s: dentry = %pd2, cookie = %Lu\n" , __func__, dentry, (unsigned long long )cookie); +if (!(server->caps &NFS_CAP_SECURITY_LABEL )) + args.bitmask = server->attr_bitmask_nl; +else + args.bitmask = server->attr_bitmask; +nfs4_setup_readdir (cookie,NFS_I (dir)->cookieverf, dentry, &args); res.pgbase = args.pgbase; - status =nfs4_call_sync (NFS_SERVER (dir)->client,NFS_SERVER (dir), &msg, &args.seq_args, &res.seq_res,0 ); + status =nfs4_call_sync (server->client, server, &msg, &args.seq_args, + &res.seq_res,0 );if (status >=0 ) {memcpy (NFS_I (dir)->cookieverf, res.verifier.data,NFS4_VERIFIER_SIZE ); status += args.pgbase;
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 4/22 NFS: Ensure contents of struct nfs_open_dir_context are consistent
From: Trond Myklebust <[email protected]>
Ensure that the contents of struct nfs_open_dir_context are consistent by setting them under the file->f_lock from a private copy (that is known to be consistent).
--- fs/nfs/dir.c | 72 +++++++++++++++++++++++++++++++--------------------- 1 file changed, 43 insertions(+), 29 deletions(-)
fs/nfs/dir.c +43 -29
@@ -144,20 +144,23 @@ struct nfs_cache_array {struct nfs_cache_array_entry array[]; }; -typedef struct { +typedef struct nfs_readdir_descriptor {struct file *file;struct page *page;struct dir_context *ctx;unsigned long page_index; -u64 *dir_cookie; +u64 dir_cookie;u64 last_cookie; +u64 dup_cookie;loff_t current_index;loff_t prev_index;unsigned long dir_verifier;unsigned long timestamp;unsigned long gencount; +unsigned long attr_gencount;unsigned int cache_entry_index; +signed char duped;bool plus;bool eof; }nfs_readdir_descriptor_t ;
@@ -273,7 +276,7 @@ int nfs_readdir_search_for_pos(struct nfs_cache_array *array, nfs_readdir_descri } index = (unsigned int )diff; - *desc->dir_cookie = array->array[index].cookie; + desc->dir_cookie = array->array[index].cookie; desc->cache_entry_index = index;return 0 ; out_eof:
@@ -298,33 +301,32 @@ int nfs_readdir_search_for_cookie(struct nfs_cache_array *array, nfs_readdir_desint status = -EAGAIN ;for (i =0 ; i < array->size; i++) { -if (array->array[i].cookie == *desc->dir_cookie) { +if (array->array[i].cookie == desc->dir_cookie) {struct nfs_inode *nfsi =NFS_I (file_inode (desc->file)); -struct nfs_open_dir_context *ctx = desc->file->private_data; new_pos = desc->current_index + i; -if (ctx->attr_gencount != nfsi->attr_gencount || +if (desc->attr_gencount != nfsi->attr_gencount || !nfs_readdir_inode_mapping_valid (nfsi)) { - ctx->duped =0 ; - ctx->attr_gencount = nfsi->attr_gencount; + desc->duped =0 ; + desc->attr_gencount = nfsi->attr_gencount; }else if (new_pos < desc->prev_index) { -if (ctx->duped >0 - && ctx->dup_cookie == *desc->dir_cookie) { +if (desc->duped >0 + && desc->dup_cookie == desc->dir_cookie) {if (printk_ratelimit ()) {pr_notice ("NFS: directory %pD2 contains a readdir loop." "Please contact your server vendor. " "The file: %.*s has duplicate cookie %llu\n" , desc->file, array->array[i].string.len, - array->array[i].string.name, *desc->dir_cookie); + array->array[i].string.name, desc->dir_cookie); } status = -ELOOP ; goto out; } - ctx->dup_cookie = *desc->dir_cookie; - ctx->duped =-1 ; + desc->dup_cookie = desc->dir_cookie; + desc->duped =-1 ; }if (nfs_readdir_use_cookie (desc->file)) - desc->ctx->pos = *desc->dir_cookie; + desc->ctx->pos = desc->dir_cookie;else desc->ctx->pos = new_pos; desc->prev_index = new_pos;
@@ -334,7 +336,7 @@ int nfs_readdir_search_for_cookie(struct nfs_cache_array *array, nfs_readdir_des }if (array->eof_index >=0 ) { status = -EBADCOOKIE ; -if (*desc->dir_cookie == array->last_cookie) +if (desc->dir_cookie == array->last_cookie) desc->eof = true; } out:
@@ -349,7 +351,7 @@ int nfs_readdir_search_array(nfs_readdir_descriptor_t *desc) array =kmap (desc->page); -if (*desc->dir_cookie ==0 ) +if (desc->dir_cookie ==0 ) status =nfs_readdir_search_for_pos (array, desc);else status =nfs_readdir_search_for_cookie (array, desc);
@@ -801,7 +803,6 @@ int nfs_do_filldir(nfs_readdir_descriptor_t *desc)int i =0 ;int res =0 ;struct nfs_cache_array *array =NULL ; -struct nfs_open_dir_context *ctx = file->private_data; array =kmap (desc->page);for (i = desc->cache_entry_index; i < array->size; i++) {
@@ -814,22 +815,22 @@ int nfs_do_filldir(nfs_readdir_descriptor_t *desc)break ; }if (i < (array->size-1 )) - *desc->dir_cookie = array->array[i+1 ].cookie; + desc->dir_cookie = array->array[i+1 ].cookie;else - *desc->dir_cookie = array->last_cookie; + desc->dir_cookie = array->last_cookie;if (nfs_readdir_use_cookie (file)) - desc->ctx->pos = *desc->dir_cookie; + desc->ctx->pos = desc->dir_cookie;else desc->ctx->pos++; -if (ctx->duped !=0 ) - ctx->duped =1 ; +if (desc->duped !=0 ) + desc->duped =1 ; }if (array->eof_index >=0 ) desc->eof = true;kunmap (desc->page);dfprintk (DIRCACHE ,"NFS: nfs_do_filldir() filling ended @ cookie %Lu; returning = %d\n" , - (unsigned long long )*desc->dir_cookie, res); + (unsigned long long )desc->dir_cookie, res);return res; }
@@ -851,10 +852,9 @@ int uncached_readdir(nfs_readdir_descriptor_t *desc)struct page *page =NULL ;int status;struct inode *inode =file_inode (desc->file); -struct nfs_open_dir_context *ctx = desc->file->private_data;dfprintk (DIRCACHE ,"NFS: uncached_readdir() searching for cookie %Lu\n" , - (unsigned long long )*desc->dir_cookie); + (unsigned long long )desc->dir_cookie); page =alloc_page (GFP_HIGHUSER );if (!page) {
@@ -863,9 +863,9 @@ int uncached_readdir(nfs_readdir_descriptor_t *desc) } desc->page_index =0 ; - desc->last_cookie = *desc->dir_cookie; + desc->last_cookie = desc->dir_cookie; desc->page = page; - ctx->duped =0 ; + desc->duped =0 ; status =nfs_readdir_xdr_to_array (desc, page, inode);if (status <0 )
@@ -894,7 +894,6 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)nfs_readdir_descriptor_t my_desc = { .file = file, .ctx = ctx, - .dir_cookie = &dir_ctx->dir_cookie, .plus =nfs_use_readdirplus (inode, ctx), }, *desc = &my_desc;
@@ -915,13 +914,20 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)if (res <0 ) goto out; +spin_lock (&file->f_lock); + desc->dir_cookie = dir_ctx->dir_cookie; + desc->dup_cookie = dir_ctx->dup_cookie; + desc->duped = dir_ctx->duped; + desc->attr_gencount = dir_ctx->attr_gencount; +spin_unlock (&file->f_lock); + do { res = readdir_search_pagecache(desc); if (res == -EBADCOOKIE) { res = 0; /* This means either end of directory */ - if (*desc->dir_cookie && !desc->eof) { + if (desc->dir_cookie && !desc->eof) { /* Or that the server has 'lost' a cookie */ res = uncached_readdir(desc); if (res == 0)
@@ -946,6 +952,14 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)if (res <0 )break ; }while (!desc->eof); + +spin_lock (&file->f_lock); + dir_ctx->dir_cookie = desc->dir_cookie; + dir_ctx->dup_cookie = desc->dup_cookie; + dir_ctx->duped = desc->duped; + dir_ctx->attr_gencount = desc->attr_gencount; +spin_unlock (&file->f_lock); + out:if (res >0 ) res =0 ;
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 5/22 NFS: Clean up readdir struct nfs_cache_array
From: Trond Myklebust <[email protected]>
Since the 'eof_index' is only ever used as a flag, make it so. Also add a flag to detect if the page has been completely filled.
--- fs/nfs/dir.c | 66 ++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 49 insertions(+), 17 deletions(-)
fs/nfs/dir.c +49 -17
@@ -138,9 +138,10 @@ struct nfs_cache_array_entry { };struct nfs_cache_array { -int size; -int eof_index;u64 last_cookie; +unsigned int size; +unsigned char page_full :1 , + page_is_eof :1 ;struct nfs_cache_array_entry array[]; };
@@ -172,7 +173,6 @@ void nfs_readdir_init_array(struct page *page) array =kmap_atomic (page);memset (array,0 ,sizeof (struct nfs_cache_array )); - array->eof_index =-1 ;kunmap_atomic (array); }
@@ -192,6 +192,17 @@ void nfs_readdir_clear_array(struct page *page)kunmap_atomic (array); } +static void nfs_readdir_array_set_eof (struct nfs_cache_array *array) +{ + array->page_is_eof =1 ; + array->page_full =1 ; +} + +static bool nfs_readdir_array_is_full (struct nfs_cache_array *array) +{ +return array->page_full; +} +/* * the caller is responsible for freeing qstr.name * when called by nfs_readdir_add_to_array, the strings will be freed in
@@ -213,6 +224,23 @@ int nfs_readdir_make_qstr(struct qstr *string, const char *name, unsigned int lereturn 0 ; } +/* +* Check that the next array entry lies entirely within the page bounds +*/ +static int nfs_readdir_array_can_expand (struct nfs_cache_array *array) +{ +struct nfs_cache_array_entry *cache_entry; + +if (array->page_full) +return -ENOSPC ; + cache_entry = &array->array[array->size +1 ]; +if ((char *)cache_entry - (char *)array >PAGE_SIZE ) { + array->page_full =1 ; +return -ENOSPC ; + } +return 0 ; +} +static int nfs_readdir_add_to_array (struct nfs_entry *entry,struct page *page) {
@@ -220,13 +248,11 @@ int nfs_readdir_add_to_array(struct nfs_entry *entry, struct page *page)struct nfs_cache_array_entry *cache_entry;int ret; - cache_entry = &array->array[array->size]; - -/* Check that this entry lies within the page bounds */ - ret = -ENOSPC ; -if ((char *)&cache_entry[1 ] - (char *)page_address (page) >PAGE_SIZE ) + ret =nfs_readdir_array_can_expand (array); +if (ret) goto out; + cache_entry = &array->array[array->size]; cache_entry->cookie = entry->prev_cookie; cache_entry->ino = entry->ino; cache_entry->d_type = entry->d_type;
@@ -236,12 +262,21 @@ int nfs_readdir_add_to_array(struct nfs_entry *entry, struct page *page) array->last_cookie = entry->cookie; array->size++;if (entry->eof !=0 ) - array->eof_index = array->size; +nfs_readdir_array_set_eof (array); out:kunmap (page);return ret; } +static void nfs_readdir_page_set_eof (struct page *page) +{ +struct nfs_cache_array *array; + + array =kmap_atomic (page); +nfs_readdir_array_set_eof (array); +kunmap_atomic (array); +} +static inline int is_32bit_api (void ) {
@@ -270,7 +305,7 @@ int nfs_readdir_search_for_pos(struct nfs_cache_array *array, nfs_readdir_descriif (diff <0 ) goto out_eof;if (diff >= array->size) { -if (array->eof_index >=0 ) +if (array->page_is_eof) goto out_eof;return -EAGAIN ; }
@@ -334,7 +369,7 @@ int nfs_readdir_search_for_cookie(struct nfs_cache_array *array, nfs_readdir_desreturn 0 ; } } -if (array->eof_index >=0 ) { +if (array->page_is_eof) { status = -EBADCOOKIE ;if (desc->dir_cookie == array->last_cookie) desc->eof = true;
@@ -566,7 +601,6 @@ int nfs_readdir_page_filler(nfs_readdir_descriptor_t *desc, struct nfs_entry *enstruct xdr_stream stream;struct xdr_buf buf;struct page *scratch; -struct nfs_cache_array *array;unsigned int count =0 ;int status;
@@ -604,10 +638,8 @@ int nfs_readdir_page_filler(nfs_readdir_descriptor_t *desc, struct nfs_entry *en out_nopages:if (count ==0 || (status == -EBADCOOKIE && entry->eof !=0 )) { - array =kmap (page); - array->eof_index = array->size; +nfs_readdir_page_set_eof (page); status =0 ; -kunmap (page); }put_page (scratch);
@@ -689,7 +721,7 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page, status =0 ;break ; } - }while (array->eof_index <0 ); + }while (!nfs_readdir_array_is_full (array));nfs_readdir_free_pages (pages, array_size); out_release_array:
@@ -825,7 +857,7 @@ int nfs_do_filldir(nfs_readdir_descriptor_t *desc)if (desc->duped !=0 ) desc->duped =1 ; } -if (array->eof_index >=0 ) +if (array->page_is_eof) desc->eof = true;kunmap (desc->page);
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 6/22 NFS: Clean up nfs_readdir_page_filler()
From: Trond Myklebust <[email protected]>
Clean up handling of the case where there are no entries in the readdir reply.
--- fs/nfs/dir.c | 39 ++++++++++++++++++--------------------- 1 file changed, 18 insertions(+), 21 deletions(-)
fs/nfs/dir.c +18 -21
@@ -601,16 +601,12 @@ int nfs_readdir_page_filler(nfs_readdir_descriptor_t *desc, struct nfs_entry *enstruct xdr_stream stream;struct xdr_buf buf;struct page *scratch; -unsigned int count =0 ;int status; scratch =alloc_page (GFP_KERNEL );if (scratch ==NULL )return -ENOMEM ; -if (buflen ==0 ) - goto out_nopages; -xdr_init_decode_pages (&stream, &buf, xdr_pages, buflen);xdr_set_scratch_buffer (&stream,page_address (scratch),PAGE_SIZE );
@@ -619,27 +615,27 @@ int nfs_readdir_page_filler(nfs_readdir_descriptor_t *desc, struct nfs_entry *en entry->label->len =NFS4_MAXLABELLEN ; status =xdr_decode (desc, entry, &stream); -if (status !=0 ) { -if (status == -EAGAIN ) - status =0 ; +if (status !=0 )break ; - } - - count++;if (desc->plus)nfs_prime_dcache (file_dentry (desc->file), entry, desc->dir_verifier); status =nfs_readdir_add_to_array (entry, page); -if (status !=0 ) -break ; - }while (!entry->eof); + }while (!status && !entry->eof); -out_nopages: -if (count ==0 || (status == -EBADCOOKIE && entry->eof !=0 )) { -nfs_readdir_page_set_eof (page); +switch (status) { +case -EBADCOOKIE : +if (entry->eof) { +nfs_readdir_page_set_eof (page); + status =0 ; + } +break ; +case -ENOSPC : +case -EAGAIN : status =0 ; +break ; }put_page (scratch);
@@ -714,14 +710,15 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page,if (status <0 )break ; + pglen = status; - status =nfs_readdir_page_filler (desc, &entry, pages, page, pglen); -if (status <0 ) { -if (status == -ENOSPC ) - status =0 ; +if (pglen ==0 ) { +nfs_readdir_page_set_eof (page);break ; } - }while (!nfs_readdir_array_is_full (array)); + + status =nfs_readdir_page_filler (desc, &entry, pages, page, pglen); + }while (!status && !nfs_readdir_array_is_full (array));nfs_readdir_free_pages (pages, array_size); out_release_array:
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 7/22 NFS: Clean up directory array handling
From: Trond Myklebust <[email protected]>
Refactor to use pagecache_get_page() so that we can fill the page in multiple stages.
--- fs/nfs/dir.c | 138 ++++++++++++++++++++++++++++----------------------- 1 file changed, 77 insertions(+), 61 deletions(-)
fs/nfs/dir.c +77 -61
@@ -149,7 +149,7 @@ typedef struct nfs_readdir_descriptor {struct file *file;struct page *page;struct dir_context *ctx; -unsigned long page_index; +pgoff_t page_index;u64 dir_cookie;u64 last_cookie;u64 dup_cookie;
@@ -166,13 +166,18 @@ typedef struct nfs_readdir_descriptor {bool eof; } nfs_readdir_descriptor_t; -static -void nfs_readdir_init_array (struct page *page) +static void nfs_readdir_array_init (struct nfs_cache_array *array) +{ +memset (array,0 ,sizeof (struct nfs_cache_array )); +} + +static void nfs_readdir_page_init_array (struct page *page,u64 last_cookie) {struct nfs_cache_array *array; array =kmap_atomic (page); -memset (array,0 ,sizeof (struct nfs_cache_array )); +nfs_readdir_array_init (array); + array->last_cookie = last_cookie;kunmap_atomic (array); }
@@ -188,7 +193,7 @@ void nfs_readdir_clear_array(struct page *page) array =kmap_atomic (page);for (i =0 ; i < array->size; i++)kfree (array->array[i].string.name); - array->size =0 ; +nfs_readdir_array_init (array);kunmap_atomic (array); }
@@ -268,6 +273,44 @@ int nfs_readdir_add_to_array(struct nfs_entry *entry, struct page *page)return ret; } +static struct page *nfs_readdir_page_get_locked (struct address_space *mapping, +pgoff_t index,u64 last_cookie) +{ +struct page *page; + + page =grab_cache_page (mapping, index); +if (page && !PageUptodate (page)) { +nfs_readdir_page_init_array (page, last_cookie); +if (invalidate_inode_pages2_range (mapping, index +1 ,-1 ) <0 ) +nfs_zap_mapping (mapping->host, mapping); +SetPageUptodate (page); + } + +return page; +} + +static u64 nfs_readdir_page_last_cookie (struct page *page) +{ +struct nfs_cache_array *array; +u64 ret; + + array =kmap_atomic (page); + ret = array->last_cookie; +kunmap_atomic (array); +return ret; +} + +static bool nfs_readdir_page_needs_filling (struct page *page) +{ +struct nfs_cache_array *array; +bool ret; + + array =kmap_atomic (page); + ret = !nfs_readdir_array_is_full (array); +kunmap_atomic (array); +return ret; +} +static void nfs_readdir_page_set_eof (struct page *page) {struct nfs_cache_array *array;
@@ -682,10 +725,8 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page,int status = -ENOMEM ;unsigned int array_size =ARRAY_SIZE (pages); -nfs_readdir_init_array (page); - entry.prev_cookie =0 ; - entry.cookie = desc->last_cookie; + entry.cookie =nfs_readdir_page_last_cookie (page); entry.eof =0 ; entry.fh =nfs_alloc_fhandle (); entry.fattr =nfs_alloc_fattr ();
@@ -730,48 +771,25 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page,return status; } -/* -* Now we cache directories properly, by converting xdr information -* to an array that can be used for lookups later. This results in -* fewer cache pages, since we can store more information on each page. -* We only need to convert from xdr once so future lookups are much simpler -*/ -static -int nfs_readdir_filler (void *data,struct page * page) +static void nfs_readdir_page_put (struct nfs_readdir_descriptor *desc) { -nfs_readdir_descriptor_t *desc = data; -struct inode *inode =file_inode (desc->file); -int ret; - - ret =nfs_readdir_xdr_to_array (desc, page, inode); -if (ret <0 ) - goto error; -SetPageUptodate (page); - -if (invalidate_inode_pages2_range (inode->i_mapping, page->index +1 ,-1 ) <0 ) { -/* Should never happen */ -nfs_zap_mapping (inode, inode->i_mapping); - } -unlock_page (page); -return 0 ; - error: -nfs_readdir_clear_array (page); -unlock_page (page); -return ret; +put_page (desc->page); + desc->page =NULL ; } -static -void cache_page_release (nfs_readdir_descriptor_t *desc) +static void +nfs_readdir_page_unlock_and_put_cached (struct nfs_readdir_descriptor *desc) { -put_page (desc->page); - desc->page =NULL ; +unlock_page (desc->page); +nfs_readdir_page_put (desc); } -static -struct page *get_cache_page (nfs_readdir_descriptor_t *desc) +static struct page * +nfs_readdir_page_get_cached (struct nfs_readdir_descriptor *desc) { -return read_cache_page (desc->file->f_mapping, desc->page_index, - nfs_readdir_filler, desc); +return nfs_readdir_page_get_locked (desc->file->f_mapping, + desc->page_index, + desc->last_cookie); }/*
@@ -785,23 +803,21 @@ int find_and_lock_cache_page(nfs_readdir_descriptor_t *desc)struct nfs_inode *nfsi =NFS_I (inode);int res; - desc->page =get_cache_page (desc); -if (IS_ERR (desc->page)) -return PTR_ERR (desc->page); - res =lock_page_killable (desc->page); -if (res !=0 ) - goto error; - res = -EAGAIN ; -if (desc->page->mapping !=NULL ) { - res =nfs_readdir_search_array (desc); -if (res ==0 ) { - nfsi->page_index = desc->page_index; -return 0 ; - } + desc->page =nfs_readdir_page_get_cached (desc); +if (!desc->page) +return -ENOMEM ; +if (nfs_readdir_page_needs_filling (desc->page)) { + res =nfs_readdir_xdr_to_array (desc, desc->page, inode); +if (res <0 ) + goto error; + } + res =nfs_readdir_search_array (desc); +if (res ==0 ) { + nfsi->page_index = desc->page_index; +return 0 ; } -unlock_page (desc->page); error: -cache_page_release (desc); +nfs_readdir_page_unlock_and_put_cached (desc);return res; }
@@ -896,6 +912,7 @@ int uncached_readdir(nfs_readdir_descriptor_t *desc) desc->page = page; desc->duped =0 ; +nfs_readdir_page_init_array (page, desc->dir_cookie); status =nfs_readdir_xdr_to_array (desc, page, inode);if (status <0 ) goto out_release;
@@ -904,7 +921,7 @@ int uncached_readdir(nfs_readdir_descriptor_t *desc) out_release:nfs_readdir_clear_array (desc->page); -cache_page_release (desc); +nfs_readdir_page_put (desc); out:dfprintk (DIRCACHE ,"NFS: %s: returns %d\n" , __func__, status);
@@ -976,8 +993,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)break ; res =nfs_do_filldir (desc); -unlock_page (desc->page); -cache_page_release (desc); +nfs_readdir_page_unlock_and_put_cached (desc);if (res <0 )break ; }while (!desc->eof);
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 8/22 NFS: Don't discard readdir results
From: Trond Myklebust <[email protected]>
If a readdir call returns more data than we can fit into one page cache page, then allocate a new one for that data rather than discarding the data.
--- fs/nfs/dir.c | 46 ++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 42 insertions(+), 4 deletions(-)
fs/nfs/dir.c +42 -4
@@ -320,6 +320,26 @@ static void nfs_readdir_page_set_eof(struct page *page)kunmap_atomic (array); } +static void nfs_readdir_page_unlock_and_put (struct page *page) +{ +unlock_page (page); +put_page (page); +} + +static struct page *nfs_readdir_page_get_next (struct address_space *mapping, +pgoff_t index,u64 cookie) +{ +struct page *page; + + page =nfs_readdir_page_get_locked (mapping, index, cookie); +if (page) { +if (nfs_readdir_page_last_cookie (page) == cookie) +return page; +nfs_readdir_page_unlock_and_put (page); + } +return NULL ; +} +static inline int is_32bit_api (void ) {
@@ -637,13 +657,15 @@ void nfs_prime_dcache(struct dentry *parent, struct nfs_entry *entry, }/* Perform conversion from xdr to cache array */ -static -int nfs_readdir_page_filler (nfs_readdir_descriptor_t *desc,struct nfs_entry *entry, -struct page **xdr_pages,struct page *page,unsigned int buflen) +static int nfs_readdir_page_filler (struct nfs_readdir_descriptor *desc, +struct nfs_entry *entry, +struct page **xdr_pages, +struct page *fillme,unsigned int buflen) { +struct address_space *mapping = desc->file->f_mapping;struct xdr_stream stream;struct xdr_buf buf; -struct page *scratch; +struct page *scratch, *new, *page = fillme;int status; scratch =alloc_page (GFP_KERNEL );
@@ -666,6 +688,19 @@ int nfs_readdir_page_filler(nfs_readdir_descriptor_t *desc, struct nfs_entry *en desc->dir_verifier); status =nfs_readdir_add_to_array (entry, page); +if (status != -ENOSPC ) +continue ; + +if (page->mapping != mapping) +break ; + new =nfs_readdir_page_get_next (mapping, page->index +1 , + entry->prev_cookie); +if (!new) +break ; +if (page != fillme) +nfs_readdir_page_unlock_and_put (page); + page = new; + status =nfs_readdir_add_to_array (entry, page); }while (!status && !entry->eof);switch (status) {
@@ -681,6 +716,9 @@ int nfs_readdir_page_filler(nfs_readdir_descriptor_t *desc, struct nfs_entry *enbreak ; } +if (page != fillme) +nfs_readdir_page_unlock_and_put (page); +put_page (scratch);return status; }
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 9/22 NFS: Remove unnecessary kmap in nfs_readdir_xdr_to_array()
From: Trond Myklebust <[email protected]>
The kmapped pointer is only used once per loop to check if we need to exit.
--- fs/nfs/dir.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-)
fs/nfs/dir.c +3 -7
@@ -759,7 +759,6 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page,struct page *pages[NFS_MAX_READDIR_PAGES ];struct nfs_entry entry;struct file *file = desc->file; -struct nfs_cache_array *array;int status = -ENOMEM ;unsigned int array_size =ARRAY_SIZE (pages);
@@ -778,11 +777,9 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page, goto out; } - array =kmap (page); - status =nfs_readdir_alloc_pages (pages, array_size);if (status <0 ) - goto out_release_array; + goto out_release_label; do { unsigned int pglen; status = nfs_readdir_xdr_filler(pages, desc, &entry, file, inode);
@@ -797,11 +794,10 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page, } status =nfs_readdir_page_filler (desc, &entry, pages, page, pglen); - }while (!status && !nfs_readdir_array_is_full (array)); + }while (!status &&nfs_readdir_page_needs_filling (page));nfs_readdir_free_pages (pages, array_size); -out_release_array: -kunmap (page); +out_release_label:nfs4_label_free (entry.label); out:nfs_free_fattr (entry.fattr);
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 10/22 NFS: Replace kmap() with kmap_atomic() in nfs_readdir_search_array()
From: Trond Myklebust <[email protected]>
--- fs/nfs/dir.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
fs/nfs/dir.c +2 -2
@@ -447,7 +447,7 @@ int nfs_readdir_search_array(nfs_readdir_descriptor_t *desc)struct nfs_cache_array *array;int status; - array =kmap (desc->page); + array =kmap_atomic (desc->page);if (desc->dir_cookie ==0 ) status =nfs_readdir_search_for_pos (array, desc);
@@ -459,7 +459,7 @@ int nfs_readdir_search_array(nfs_readdir_descriptor_t *desc) desc->current_index += array->size; desc->page_index++; } -kunmap (desc->page); +kunmap_atomic (array);return status; }
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 11/22 NFS: Simplify struct nfs_cache_array_entry
From: Trond Myklebust <[email protected]>
We don't need to store a hash, so replace struct qstr with a simple const char pointer and length.
--- fs/nfs/dir.c | 46 +++++++++++++++++++++++++--------------------- 1 file changed, 25 insertions(+), 21 deletions(-)
fs/nfs/dir.c +25 -21
@@ -133,7 +133,8 @@ nfs_closedir(struct inode *inode, struct file *filp)struct nfs_cache_array_entry {u64 cookie;u64 ino; -struct qstr string; +const char *name; +unsigned int name_len;unsigned char d_type; };
@@ -192,7 +193,7 @@ void nfs_readdir_clear_array(struct page *page) array =kmap_atomic (page);for (i =0 ; i < array->size; i++) -kfree (array->array[i].string.name); +kfree (array->array[i].name);nfs_readdir_array_init (array);kunmap_atomic (array); }
@@ -213,20 +214,17 @@ static bool nfs_readdir_array_is_full(struct nfs_cache_array *array)* when called by nfs_readdir_add_to_array, the strings will be freed in * nfs_clear_readdir_array() */ -static -int nfs_readdir_make_qstr (struct qstr *string,const char *name,unsigned int len) +static const char *nfs_readdir_copy_name (const char *name,unsigned int len) { - string->len = len; - string->name =kmemdup_nul (name, len,GFP_KERNEL ); -if (string->name ==NULL ) -return -ENOMEM ; +const char *ret =kmemdup_nul (name, len,GFP_KERNEL ); +/* * Avoid a kmemleak false positive. The pointer to the name is stored * in a page cache page which kmemleak does not scan. */ -kmemleak_not_leak (string->name); - string->hash =full_name_hash (NULL , name, len); -return 0 ; +if (ret !=NULL ) +kmemleak_not_leak (ret); +return ret; }/*
@@ -249,27 +247,34 @@ static int nfs_readdir_array_can_expand(struct nfs_cache_array *array)static int nfs_readdir_add_to_array (struct nfs_entry *entry,struct page *page) { -struct nfs_cache_array *array =kmap (page); +struct nfs_cache_array *array;struct nfs_cache_array_entry *cache_entry; +const char *name;int ret; + name =nfs_readdir_copy_name (entry->name, entry->len); +if (!name) +return -ENOMEM ; + + array =kmap_atomic (page); ret =nfs_readdir_array_can_expand (array); -if (ret) +if (ret) { +kfree (name); goto out; + } cache_entry = &array->array[array->size]; cache_entry->cookie = entry->prev_cookie; cache_entry->ino = entry->ino; cache_entry->d_type = entry->d_type; - ret =nfs_readdir_make_qstr (&cache_entry->string, entry->name, entry->len); -if (ret) - goto out; + cache_entry->name_len = entry->len; + cache_entry->name = name; array->last_cookie = entry->cookie; array->size++;if (entry->eof !=0 )nfs_readdir_array_set_eof (array); out: -kunmap (page); +kunmap_atomic (array);return ret; }
@@ -413,9 +418,8 @@ int nfs_readdir_search_for_cookie(struct nfs_cache_array *array, nfs_readdir_desif (printk_ratelimit ()) {pr_notice ("NFS: directory %pD2 contains a readdir loop." "Please contact your server vendor. " -"The file: %.*s has duplicate cookie %llu\n" , - desc->file, array->array[i].string.len, - array->array[i].string.name, desc->dir_cookie); +"The file: %s has duplicate cookie %llu\n" , + desc->file, array->array[i].name, desc->dir_cookie); } status = -ELOOP ; goto out;
@@ -888,7 +892,7 @@ int nfs_do_filldir(nfs_readdir_descriptor_t *desc)struct nfs_cache_array_entry *ent; ent = &array->array[i]; -if (!dir_emit (desc->ctx, ent->string.name, ent->string.len, +if (!dir_emit (desc->ctx, ent->name, ent->name_len,nfs_compat_user_ino64 (ent->ino), ent->d_type)) { desc->eof = true;break ;
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 12/22 NFS: Support larger readdir buffers
From: Trond Myklebust <[email protected]>
Support readdir buffers of up to 1MB in size so that we can read large directories using few RPC calls.
--- fs/nfs/client.c | 4 ++-- fs/nfs/dir.c | 33 +++++++++++++++++++-------------- fs/nfs/internal.h | 6 ------ 3 files changed, 21 insertions(+), 22 deletions(-)
fs/nfs/client.c +2 -2
@@ -781,8 +781,8 @@ static void nfs_server_set_fsinfo(struct nfs_server *server, server->wtmult =nfs_block_bits (fsinfo->wtmult,NULL ); server->dtsize =nfs_block_size (fsinfo->dtpref,NULL ); -if (server->dtsize >PAGE_SIZE *NFS_MAX_READDIR_PAGES ) - server->dtsize =PAGE_SIZE *NFS_MAX_READDIR_PAGES ; +if (server->dtsize >NFS_MAX_FILE_IO_SIZE ) + server->dtsize =NFS_MAX_FILE_IO_SIZE ;if (server->dtsize > server->rsize) server->dtsize = server->rsize;
fs/nfs/dir.c +19 -14
@@ -727,44 +727,47 @@ static int nfs_readdir_page_filler(struct nfs_readdir_descriptor *desc,return status; } -static -void nfs_readdir_free_pages (struct page **pages,unsigned int npages) +static void nfs_readdir_free_pages (struct page **pages,size_t npages) { -unsigned int i; -for (i =0 ; i < npages; i++) -put_page (pages[i]); +while (npages--) +put_page (pages[npages]); +kfree (pages); }/* * nfs_readdir_alloc_pages() will allocate pages that must be freed with a call * to nfs_readdir_free_pages() */ -static -int nfs_readdir_alloc_pages (struct page **pages,unsigned int npages) +static struct page **nfs_readdir_alloc_pages (size_t npages) { -unsigned int i; +struct page **pages; +size_t i; + pages =kmalloc_array (npages,sizeof (*pages),GFP_KERNEL ); +if (!pages) +return NULL ;for (i =0 ; i < npages; i++) {struct page *page =alloc_page (GFP_KERNEL );if (page ==NULL ) goto out_freepages; pages[i] = page; } -return 0 ; +return pages; out_freepages:nfs_readdir_free_pages (pages, i); -return -ENOMEM ; +return NULL ; }static int nfs_readdir_xdr_to_array (nfs_readdir_descriptor_t *desc,struct page *page,struct inode *inode) { -struct page *pages[NFS_MAX_READDIR_PAGES ]; +struct page **pages;struct nfs_entry entry;struct file *file = desc->file; +size_t array_size; +size_t dtsize =NFS_SERVER (inode)->dtsize;int status = -ENOMEM ; -unsigned int array_size =ARRAY_SIZE (pages); entry.prev_cookie =0 ; entry.cookie =nfs_readdir_page_last_cookie (page);
@@ -781,9 +784,11 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page, goto out; } - status =nfs_readdir_alloc_pages (pages, array_size); -if (status <0 ) + array_size = (dtsize +PAGE_SIZE -1 ) >>PAGE_SHIFT ; + pages =nfs_readdir_alloc_pages (array_size); +if (!pages) goto out_release_label; + do { unsigned int pglen; status = nfs_readdir_xdr_filler(pages, desc, &entry, file, inode);
fs/nfs/internal.h +0 -6
@@ -56,12 +56,6 @@ static inline bool nfs_lookup_is_soft_revalidate(const struct dentry *dentry)#define NFS_UNSPEC_RETRANS (UINT_MAX)#define NFS_UNSPEC_TIMEO (UINT_MAX) -/* -* Maximum number of pages that readdir can use for creating -* a vmapped array of pages. -*/ -#define NFS_MAX_READDIR_PAGES 8 -struct nfs_client_initdata {unsigned long init_flags;const char *hostname;/* Hostname of the server */
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 13/22 NFS: More readdir cleanups
From: Trond Myklebust <[email protected]>
Remove the redundant caching of the credential in struct nfs_open_dir_context.
Pass the buffer size as an argument to nfs_readdir_xdr_filler().
--- fs/nfs/dir.c | 25 +++++++++++-------------- include/linux/nfs_fs.h | 1 - 2 files changed, 11 insertions(+), 15 deletions(-)
fs/nfs/dir.c +11 -14
@@ -68,7 +68,7 @@ const struct address_space_operations nfs_dir_aops = { .freepage = nfs_readdir_clear_array, }; -static struct nfs_open_dir_context *alloc_nfs_open_dir_context (struct inode *dir,const struct cred *cred) +static struct nfs_open_dir_context *alloc_nfs_open_dir_context (struct inode *dir) {struct nfs_inode *nfsi =NFS_I (dir);struct nfs_open_dir_context *ctx;
@@ -78,7 +78,6 @@ static struct nfs_open_dir_context *alloc_nfs_open_dir_context(struct inode *dir ctx->attr_gencount = nfsi->attr_gencount; ctx->dir_cookie =0 ; ctx->dup_cookie =0 ; - ctx->cred =get_cred (cred);spin_lock (&dir->i_lock);if (list_empty (&nfsi->open_files) && (nfsi->cache_validity &NFS_INO_DATA_INVAL_DEFER ))
@@ -96,7 +95,6 @@ static void put_nfs_open_dir_context(struct inode *dir, struct nfs_open_dir_contspin_lock (&dir->i_lock);list_del (&ctx->list);spin_unlock (&dir->i_lock); -put_cred (ctx->cred);kfree (ctx); }
@@ -113,7 +111,7 @@ nfs_opendir(struct inode *inode, struct file *filp)nfs_inc_stats (inode,NFSIOS_VFSOPEN ); - ctx =alloc_nfs_open_dir_context (inode,current_cred ()); + ctx =alloc_nfs_open_dir_context (inode);if (IS_ERR (ctx)) { res =PTR_ERR (ctx); goto out;
@@ -468,12 +466,12 @@ int nfs_readdir_search_array(nfs_readdir_descriptor_t *desc) }/* Fill a page with xdr information before transferring to the cache page */ -static -int nfs_readdir_xdr_filler (struct page **pages,nfs_readdir_descriptor_t *desc, -struct nfs_entry *entry,struct file *file,struct inode *inode) +static int nfs_readdir_xdr_filler (struct nfs_readdir_descriptor *desc, +u64 cookie,struct page **pages, +size_t bufsize) { -struct nfs_open_dir_context *ctx = file->private_data; -const struct cred *cred = ctx->cred; +struct file *file = desc->file; +struct inode *inode =file_inode (file);unsigned long timestamp, gencount;int error;
@@ -481,8 +479,8 @@ int nfs_readdir_xdr_filler(struct page **pages, nfs_readdir_descriptor_t *desc, timestamp = jiffies; gencount =nfs_inc_attr_generation_counter (); desc->dir_verifier =nfs_save_change_attribute (inode); - error =NFS_PROTO (inode)->readdir (file_dentry (file), cred, entry->cookie, pages, -NFS_SERVER (inode)->dtsize, desc->plus); + error =NFS_PROTO (inode)->readdir (file_dentry (file), file->f_cred, + cookie, pages, bufsize, desc->plus);if (error <0 ) {/* We requested READDIRPLUS, but the server doesn't grok it */ if (error == -ENOTSUPP && desc->plus) {
@@ -764,7 +762,6 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page, {struct page **pages;struct nfs_entry entry; -struct file *file = desc->file;size_t array_size;size_t dtsize =NFS_SERVER (inode)->dtsize;int status = -ENOMEM ;
@@ -791,8 +788,8 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page, do { unsigned int pglen; - status = nfs_readdir_xdr_filler(pages, desc, &entry, file, inode); - + status = nfs_readdir_xdr_filler(desc, entry.cookie, + pages, dtsize); if (status < 0) break;
include/linux/nfs_fs.h +0 -1
@@ -88,7 +88,6 @@ struct nfs_open_context {struct nfs_open_dir_context {struct list_head list; -const struct cred *cred;unsigned long attr_gencount;__u64 dir_cookie;__u64 dup_cookie;
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 14/22 NFS: nfs_do_filldir() does not return a value
From: Trond Myklebust <[email protected]>
Clean up nfs_do_filldir().
--- fs/nfs/dir.c | 21 +++++++-------------- 1 file changed, 7 insertions(+), 14 deletions(-)
fs/nfs/dir.c +7 -14
@@ -881,13 +881,11 @@ int readdir_search_pagecache(nfs_readdir_descriptor_t *desc)/* * Once we've found the start of the dirent within a page: fill 'er up... */ -static -int nfs_do_filldir (nfs_readdir_descriptor_t *desc) +static void nfs_do_filldir (struct nfs_readdir_descriptor *desc) {struct file *file = desc->file; -int i =0 ; -int res =0 ; -struct nfs_cache_array *array =NULL ; +struct nfs_cache_array *array; +unsigned int i =0 ; array =kmap (desc->page);for (i = desc->cache_entry_index; i < array->size; i++) {
@@ -914,9 +912,8 @@ int nfs_do_filldir(nfs_readdir_descriptor_t *desc) desc->eof = true;kunmap (desc->page); -dfprintk (DIRCACHE ,"NFS: nfs_do_filldir() filling ended @ cookie %Lu; returning = %d\n" , - (unsigned long long )desc->dir_cookie, res); -return res; +dfprintk (DIRCACHE ,"NFS: nfs_do_filldir() filling ended @ cookie %llu\n" , + (unsigned long long )desc->dir_cookie); }/*
@@ -957,7 +954,7 @@ int uncached_readdir(nfs_readdir_descriptor_t *desc)if (status <0 ) goto out_release; - status =nfs_do_filldir (desc); +nfs_do_filldir (desc); out_release:nfs_readdir_clear_array (desc->page);
@@ -1032,10 +1029,8 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)if (res <0 )break ; - res =nfs_do_filldir (desc); +nfs_do_filldir (desc);nfs_readdir_page_unlock_and_put_cached (desc); -if (res <0 ) -break ; }while (!desc->eof);spin_lock (&file->f_lock);
@@ -1046,8 +1041,6 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)spin_unlock (&file->f_lock); out: -if (res >0 ) - res =0 ;dfprintk (FILE ,"NFS: readdir(%pD2) returns %d\n" , file, res);return res; }
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 15/22 NFS: Reduce readdir stack usage
From: Trond Myklebust <[email protected]>
The descriptor and the struct nfs_entry are both large structures, so don't allocate them from the stack.
--- fs/nfs/dir.c | 58 ++++++++++++++++++++++++++++++---------------------- 1 file changed, 33 insertions(+), 25 deletions(-)
fs/nfs/dir.c +33 -25
@@ -761,23 +761,24 @@ staticint nfs_readdir_xdr_to_array (nfs_readdir_descriptor_t *desc,struct page *page,struct inode *inode) {struct page **pages; -struct nfs_entry entry; +struct nfs_entry *entry;size_t array_size;size_t dtsize =NFS_SERVER (inode)->dtsize;int status = -ENOMEM ; - entry.prev_cookie =0 ; - entry.cookie =nfs_readdir_page_last_cookie (page); - entry.eof =0 ; - entry.fh =nfs_alloc_fhandle (); - entry.fattr =nfs_alloc_fattr (); - entry.server =NFS_SERVER (inode); -if (entry.fh ==NULL || entry.fattr ==NULL ) + entry =kzalloc (sizeof (*entry),GFP_KERNEL ); +if (!entry) +return -ENOMEM ; + entry->cookie =nfs_readdir_page_last_cookie (page); + entry->fh =nfs_alloc_fhandle (); + entry->fattr =nfs_alloc_fattr (); + entry->server =NFS_SERVER (inode); +if (entry->fh ==NULL || entry->fattr ==NULL ) goto out; - entry.label =nfs4_label_alloc (NFS_SERVER (inode),GFP_NOWAIT ); -if (IS_ERR (entry.label)) { - status =PTR_ERR (entry.label); + entry->label =nfs4_label_alloc (NFS_SERVER (inode),GFP_NOWAIT ); +if (IS_ERR (entry->label)) { + status =PTR_ERR (entry->label); goto out; }
@@ -788,7 +789,7 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page, do { unsigned int pglen; - status = nfs_readdir_xdr_filler(desc, entry.cookie, + status = nfs_readdir_xdr_filler(desc, entry->cookie, pages, dtsize); if (status < 0) break;
@@ -799,15 +800,16 @@ int nfs_readdir_xdr_to_array(nfs_readdir_descriptor_t *desc, struct page *page,break ; } - status =nfs_readdir_page_filler (desc, &entry, pages, page, pglen); + status =nfs_readdir_page_filler (desc, entry, pages, page, pglen); }while (!status &&nfs_readdir_page_needs_filling (page));nfs_readdir_free_pages (pages, array_size); out_release_label: -nfs4_label_free (entry.label); +nfs4_label_free (entry->label); out: -nfs_free_fattr (entry.fattr); -nfs_free_fhandle (entry.fh); +nfs_free_fattr (entry->fattr); +nfs_free_fhandle (entry->fh); +kfree (entry);return status; }
@@ -974,13 +976,8 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)struct dentry *dentry =file_dentry (file);struct inode *inode =d_inode (dentry);struct nfs_open_dir_context *dir_ctx = file->private_data; -nfs_readdir_descriptor_t my_desc = { - .file = file, - .ctx = ctx, - .plus =nfs_use_readdirplus (inode, ctx), - }, - *desc = &my_desc; -int res =0 ; +struct nfs_readdir_descriptor *desc; +int res;dfprintk (FILE ,"NFS: readdir(%pD2) starting at cookie %llu\n" , file, (long long )ctx->pos);
@@ -992,10 +989,19 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)* to either find the entry with the appropriate number or * revalidate the cookie. */ -if (ctx->pos ==0 ||nfs_attribute_cache_expired (inode)) +if (ctx->pos ==0 ||nfs_attribute_cache_expired (inode)) { res =nfs_revalidate_mapping (inode, file->f_mapping); -if (res <0 ) +if (res <0 ) + goto out; + } + + res = -ENOMEM ; + desc =kzalloc (sizeof (*desc),GFP_KERNEL ); +if (!desc) goto out; + desc->file = file; + desc->ctx = ctx; + desc->plus =nfs_use_readdirplus (inode, ctx);spin_lock (&file->f_lock); desc->dir_cookie = dir_ctx->dir_cookie;
@@ -1040,6 +1046,8 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) dir_ctx->attr_gencount = desc->attr_gencount;spin_unlock (&file->f_lock); +kfree (desc); + out:dfprintk (FILE ,"NFS: readdir(%pD2) returns %d\n" , file, res);return res;
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 16/22 NFS: Cleanup to remove nfs_readdir_descriptor_t typedef
From: Trond Myklebust <[email protected]>
--- fs/nfs/dir.c | 30 +++++++++++++----------------- 1 file changed, 13 insertions(+), 17 deletions(-)
fs/nfs/dir.c +13 -17
@@ -144,7 +144,7 @@ struct nfs_cache_array {struct nfs_cache_array_entry array[]; }; -typedef struct nfs_readdir_descriptor { +struct nfs_readdir_descriptor {struct file *file;struct page *page;struct dir_context *ctx;
@@ -163,7 +163,7 @@ typedef struct nfs_readdir_descriptor {signed char duped;bool plus;bool eof; -} nfs_readdir_descriptor_t; +};static void nfs_readdir_array_init (struct nfs_cache_array *array) {
@@ -362,8 +362,8 @@ bool nfs_readdir_use_cookie(const struct file *filp)return true; } -static -int nfs_readdir_search_for_pos (struct nfs_cache_array *array,nfs_readdir_descriptor_t *desc) +static int nfs_readdir_search_for_pos (struct nfs_cache_array *array, +struct nfs_readdir_descriptor *desc) {loff_t diff = desc->ctx->pos - desc->current_index;unsigned int index;
@@ -394,8 +394,8 @@ nfs_readdir_inode_mapping_valid(struct nfs_inode *nfsi)return !test_bit (NFS_INO_INVALIDATING , &nfsi->flags); } -static -int nfs_readdir_search_for_cookie (struct nfs_cache_array *array,nfs_readdir_descriptor_t *desc) +static int nfs_readdir_search_for_cookie (struct nfs_cache_array *array, +struct nfs_readdir_descriptor *desc) {int i;loff_t new_pos;
@@ -443,8 +443,7 @@ int nfs_readdir_search_for_cookie(struct nfs_cache_array *array, nfs_readdir_desreturn status; } -static -int nfs_readdir_search_array (nfs_readdir_descriptor_t *desc) +static int nfs_readdir_search_array (struct nfs_readdir_descriptor *desc) {struct nfs_cache_array *array;int status;
@@ -497,7 +496,7 @@ static int nfs_readdir_xdr_filler(struct nfs_readdir_descriptor *desc,return error; } -static int xdr_decode (nfs_readdir_descriptor_t *desc, +static int xdr_decode (struct nfs_readdir_descriptor *desc,struct nfs_entry *entry,struct xdr_stream *xdr) {struct inode *inode =file_inode (desc->file);
@@ -757,8 +756,8 @@ static struct page **nfs_readdir_alloc_pages(size_t npages)return NULL ; } -static -int nfs_readdir_xdr_to_array (nfs_readdir_descriptor_t *desc,struct page *page,struct inode *inode) +static int nfs_readdir_xdr_to_array (struct nfs_readdir_descriptor *desc, +struct page *page,struct inode *inode) {struct page **pages;struct nfs_entry *entry;
@@ -838,8 +837,7 @@ nfs_readdir_page_get_cached(struct nfs_readdir_descriptor *desc)* Returns 0 if desc->dir_cookie was found on page desc->page_index * and locks the page to prevent removal from the page cache. */ -static -int find_and_lock_cache_page (nfs_readdir_descriptor_t *desc) +static int find_and_lock_cache_page (struct nfs_readdir_descriptor *desc) {struct inode *inode =file_inode (desc->file);struct nfs_inode *nfsi =NFS_I (inode);
@@ -864,8 +862,7 @@ int find_and_lock_cache_page(nfs_readdir_descriptor_t *desc) }/* Search for desc->dir_cookie from the beginning of the page cache */ -static inline -int readdir_search_pagecache (nfs_readdir_descriptor_t *desc) +static int readdir_search_pagecache (struct nfs_readdir_descriptor *desc) {int res;
@@ -930,8 +927,7 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc)* we should already have a complete representation of the * directory in the page cache by the time we get here. */ -static inline -int uncached_readdir (nfs_readdir_descriptor_t *desc) +static int uncached_readdir (struct nfs_readdir_descriptor *desc) {struct page *page =NULL ;int status;
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 17/22 NFS: Allow the NFS generic code to pass in a verifier to readdir
From: Trond Myklebust <[email protected]>
If we're ever going to allow support for servers that use the readdir verifier, then that use needs to be managed by the middle layers as those need to be able to reject cookies from other verifiers.
--- fs/nfs/dir.c | 23 ++++++++++++++++++----- fs/nfs/nfs3proc.c | 35 +++++++++++++++++------------------ fs/nfs/nfs4proc.c | 36 +++++++++++++++++------------------- fs/nfs/proc.c | 18 +++++++++--------- include/linux/nfs_xdr.h | 17 +++++++++++++++-- 5 files changed, 76 insertions(+), 53 deletions(-)
fs/nfs/dir.c +18 -5
@@ -469,8 +469,20 @@ static int nfs_readdir_xdr_filler(struct nfs_readdir_descriptor *desc, u64 cookie, struct page **pages, size_t bufsize) { - struct file *file = desc->file; -struct inode *inode =file_inode (file); + struct inode *inode = file_inode(desc->file); +__be32 verf_res[2 ]; +struct nfs_readdir_arg arg = { + .dentry =file_dentry (desc->file), + .cred = desc->file->f_cred, + .verf =NFS_I (inode)->cookieverf, + .cookie = cookie, + .pages = pages, + .page_len = bufsize, + .plus = desc->plus, + }; +struct nfs_readdir_res res = { + .verf = verf_res, + };unsigned long timestamp, gencount;int error;
@@ -478,20 +490,21 @@ static int nfs_readdir_xdr_filler(struct nfs_readdir_descriptor *desc, timestamp = jiffies; gencount =nfs_inc_attr_generation_counter (); desc->dir_verifier =nfs_save_change_attribute (inode); - error =NFS_PROTO (inode)->readdir (file_dentry (file), file->f_cred, - cookie, pages, bufsize, desc->plus); + error =NFS_PROTO (inode)->readdir (&arg, &res);if (error <0 ) {/* We requested READDIRPLUS, but the server doesn't grok it */ if (error == -ENOTSUPP && desc->plus) {NFS_SERVER (inode)->caps &= ~NFS_CAP_READDIRPLUS ;clear_bit (NFS_INO_ADVISE_RDPLUS , &NFS_I (inode)->flags); - desc->plus = false; + desc->plus = arg.plus = false; goto again; } goto error; } desc->timestamp = timestamp; desc->gencount = gencount; +memcpy (NFS_I (inode)->cookieverf, res.verf, +sizeof (NFS_I (inode)->cookieverf)); error:return error; }
fs/nfs/nfs3proc.c +17 -18
@@ -662,37 +662,36 @@ nfs3_proc_rmdir(struct inode *dir, const struct qstr *name)* Also note that this implementation handles both plain readdir and * readdirplus. */ -static int -nfs3_proc_readdir (struct dentry *dentry,const struct cred *cred, -u64 cookie,struct page **pages,unsigned int count,bool plus) +static int nfs3_proc_readdir (struct nfs_readdir_arg *nr_arg, +struct nfs_readdir_res *nr_res) { -struct inode *dir =d_inode (dentry); -__be32 *verf =NFS_I (dir)->cookieverf; +struct inode *dir =d_inode (nr_arg->dentry);struct nfs3_readdirargs arg = { .fh =NFS_FH (dir), - .cookie = cookie, - .verf = {verf[0 ], verf[1 ]}, - .plus = plus, - .count = count, - .pages = pages + .cookie = nr_arg->cookie, + .plus = nr_arg->plus, + .count = nr_arg->page_len, + .pages = nr_arg->pages };struct nfs3_readdirres res = { - .verf = verf, - .plus = plus + .verf = nr_res->verf, + .plus = nr_arg->plus, };struct rpc_message msg = { .rpc_proc = &nfs3_procedures[NFS3PROC_READDIR ], .rpc_argp = &arg, .rpc_resp = &res, - .rpc_cred = cred, + .rpc_cred = nr_arg->cred, };int status = -ENOMEM ; -if (plus) +if (nr_arg->plus) msg.rpc_proc = &nfs3_procedures[NFS3PROC_READDIRPLUS ]; +if (arg.cookie) +memcpy (arg.verf, nr_arg->verf,sizeof (arg.verf)); -dprintk ("NFS call readdir%s %d\n" , - plus?"plus" :"" , (unsigned int ) cookie); +dprintk ("NFS call readdir%s %llu\n" , nr_arg->plus ?"plus" :"" , + (unsigned long long )nr_arg->cookie); res.dir_attr =nfs_alloc_fattr ();if (res.dir_attr ==NULL )
@@ -705,8 +704,8 @@ nfs3_proc_readdir(struct dentry *dentry, const struct cred *cred,nfs_free_fattr (res.dir_attr); out: -dprintk ("NFS reply readdir%s: %d\n" , - plus?"plus" :"" , status); +dprintk ("NFS reply readdir%s: %d\n" , nr_arg->plus ?"plus" :"" , + status);return status; }
fs/nfs/nfs4proc.c +17 -19
@@ -4961,41 +4961,40 @@ static int nfs4_proc_mkdir(struct inode *dir, struct dentry *dentry,return err; } -static int _nfs4_proc_readdir (struct dentry *dentry,const struct cred *cred, -u64 cookie,struct page **pages,unsigned int count,bool plus) +static int _nfs4_proc_readdir (struct nfs_readdir_arg *nr_arg, +struct nfs_readdir_res *nr_res) { -struct inode *dir =d_inode (dentry); +struct inode *dir =d_inode (nr_arg->dentry);struct nfs_server *server =NFS_SERVER (dir);struct nfs4_readdir_arg args = { .fh =NFS_FH (dir), - .pages = pages, + .pages = nr_arg->pages, .pgbase =0 , - .count = count, - .plus = plus, + .count = nr_arg->page_len, + .plus = nr_arg->plus, };struct nfs4_readdir_res res;struct rpc_message msg = { .rpc_proc = &nfs4_procedures[NFSPROC4_CLNT_READDIR ], .rpc_argp = &args, .rpc_resp = &res, - .rpc_cred = cred, + .rpc_cred = nr_arg->cred, };int status; -dprintk ("%s: dentry = %pd2, cookie = %Lu\n" , __func__, - dentry, - (unsigned long long )cookie); +dprintk ("%s: dentry = %pd2, cookie = %llu\n" , __func__, + nr_arg->dentry, (unsigned long long )nr_arg->cookie);if (!(server->caps &NFS_CAP_SECURITY_LABEL )) args.bitmask = server->attr_bitmask_nl;else args.bitmask = server->attr_bitmask; -nfs4_setup_readdir (cookie,NFS_I (dir)->cookieverf, dentry, &args); +nfs4_setup_readdir (nr_arg->cookie, nr_arg->verf, nr_arg->dentry, &args); res.pgbase = args.pgbase; status =nfs4_call_sync (server->client, server, &msg, &args.seq_args, &res.seq_res,0 );if (status >=0 ) { -memcpy (NFS_I (dir)->cookieverf, res.verifier.data,NFS4_VERIFIER_SIZE ); +memcpy (nr_res->verf, res.verifier.data,NFS4_VERIFIER_SIZE ); status += args.pgbase; }
@@ -5005,19 +5004,18 @@ static int _nfs4_proc_readdir(struct dentry *dentry, const struct cred *cred,return status; } -static int nfs4_proc_readdir (struct dentry *dentry,const struct cred *cred, -u64 cookie,struct page **pages,unsigned int count,bool plus) +static int nfs4_proc_readdir (struct nfs_readdir_arg *arg, +struct nfs_readdir_res *res) {struct nfs4_exception exception = { .interruptible = true, };int err;do { - err =_nfs4_proc_readdir (dentry, cred, cookie, - pages, count, plus); -trace_nfs4_readdir (d_inode (dentry), err); - err =nfs4_handle_exception (NFS_SERVER (d_inode (dentry)), err, - &exception); + err =_nfs4_proc_readdir (arg, res); +trace_nfs4_readdir (d_inode (arg->dentry), err); + err =nfs4_handle_exception (NFS_SERVER (d_inode (arg->dentry)), + err, &exception); }while (exception.retry);return err; }
fs/nfs/proc.c +9 -9
@@ -499,26 +499,26 @@ nfs_proc_rmdir(struct inode *dir, const struct qstr *name)* sure it is syntactically correct; the entries itself are decoded * from nfs_readdir by calling the decode_entry function directly. */ -static int -nfs_proc_readdir (struct dentry *dentry,const struct cred *cred, -u64 cookie,struct page **pages,unsigned int count,bool plus) +static int nfs_proc_readdir (struct nfs_readdir_arg *nr_arg, +struct nfs_readdir_res *nr_res) { -struct inode *dir =d_inode (dentry); +struct inode *dir =d_inode (nr_arg->dentry);struct nfs_readdirargs arg = { .fh =NFS_FH (dir), - .cookie = cookie, - .count = count, - .pages = pages, + .cookie = nr_arg->cookie, + .count = nr_arg->page_len, + .pages = nr_arg->pages, };struct rpc_message msg = { .rpc_proc = &nfs_procedures[NFSPROC_READDIR ], .rpc_argp = &arg, - .rpc_cred = cred, + .rpc_cred = nr_arg->cred, };int status; -dprintk ("NFS call readdir %d\n" , (unsigned int )cookie); +dprintk ("NFS call readdir %llu\n" , (unsigned long long )nr_arg->cookie); status =rpc_call_sync (NFS_CLIENT (dir), &msg,0 ); + nr_res->verf[0 ] = nr_res->verf[1 ] =0 ;nfs_invalidate_atime (dir);
include/linux/nfs_xdr.h +15 -2
@@ -750,6 +750,20 @@ struct nfs_entry {struct nfs_server * server; }; +struct nfs_readdir_arg { +struct dentry *dentry; +const struct cred *cred; +__be32 *verf; +u64 cookie; +struct page **pages; +unsigned int page_len; +bool plus; +}; + +struct nfs_readdir_res { +__be32 *verf; +}; +/* * The following types are for NFSv2 only. */
@@ -1744,8 +1758,7 @@ struct nfs_rpc_ops { unsigned int, struct iattr *);int (*mkdir) (struct inode *,struct dentry *,struct iattr *);int (*rmdir) (struct inode *,const struct qstr *); -int (*readdir) (struct dentry *,const struct cred *, -u64 ,struct page **,unsigned int ,bool ); +int (*readdir) (struct nfs_readdir_arg *,struct nfs_readdir_res *);int (*mknod) (struct inode *,struct dentry *,struct iattr *,dev_t ); int (*statfs) (struct nfs_server *, struct nfs_fh *,
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 18/22 NFS: Handle NFS4ERR_NOT_SAME and NFSERR_BADCOOKIE from readdir calls
From: Trond Myklebust <[email protected]>
If the server returns NFS4ERR_NOT_SAME or tells us that the cookie is bad in response to a READDIR call, then we should empty the page cache so that we can fill it from scratch again.
--- fs/nfs/dir.c | 24 ++++++++++++++++-------- fs/nfs/nfs4proc.c | 2 ++ 2 files changed, 18 insertions(+), 8 deletions(-)
fs/nfs/dir.c +16 -8
@@ -861,15 +861,21 @@ static int find_and_lock_cache_page(struct nfs_readdir_descriptor *desc)return -ENOMEM ;if (nfs_readdir_page_needs_filling (desc->page)) { res =nfs_readdir_xdr_to_array (desc, desc->page, inode); -if (res <0 ) - goto error; +if (res <0 ) { +nfs_readdir_page_unlock_and_put_cached (desc); +if (res == -EBADCOOKIE || res == -ENOTSYNC ) { +invalidate_inode_pages2 (desc->file->f_mapping); + desc->page_index =0 ; +return -EAGAIN ; + } +return res; + } } res =nfs_readdir_search_array (desc);if (res ==0 ) { nfsi->page_index = desc->page_index;return 0 ; } -error:nfs_readdir_page_unlock_and_put_cached (desc);return res; }
@@ -879,12 +885,12 @@ static int readdir_search_pagecache(struct nfs_readdir_descriptor *desc) {int res; -if (desc->page_index ==0 ) { - desc->current_index =0 ; - desc->prev_index =0 ; - desc->last_cookie =0 ; - }do { +if (desc->page_index ==0 ) { + desc->current_index =0 ; + desc->prev_index =0 ; + desc->last_cookie =0 ; + } res =find_and_lock_cache_page (desc); }while (res == -EAGAIN );return res;
@@ -1030,6 +1036,8 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) res =uncached_readdir (desc);if (res ==0 )continue ; +if (res == -EBADCOOKIE || res == -ENOTSYNC ) + res =0 ; }break ; }
fs/nfs/nfs4proc.c +2 -0
@@ -184,6 +184,8 @@ static int nfs4_map_errors(int err)return -EPROTONOSUPPORT ;case -NFS4ERR_FILE_OPEN :return -EBUSY ; +case -NFS4ERR_NOT_SAME : +return -ENOTSYNC ;default :dprintk ("%s could not handle NFSv4 error %d\n" , __func__, -err);
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 19/22 NFS: Improve handling of directory verifiers
From: Trond Myklebust <[email protected]>
If the server insists on using the readdir verifiers in order to allow cookies to expire, then we should ensure that we cache the verifier with the cookie, so that we can return an error if the application tries to use the expired cookie.
--- fs/nfs/dir.c | 35 +++++++++++++++++++++++------------ fs/nfs/inode.c | 7 ------- include/linux/nfs_fs.h | 8 +++++++- 3 files changed, 30 insertions(+), 20 deletions(-)
fs/nfs/dir.c +23 -12
@@ -155,6 +155,7 @@ struct nfs_readdir_descriptor {loff_t current_index;loff_t prev_index; +__be32 verf[NFS_DIR_VERIFIER_SIZE ];unsigned long dir_verifier;unsigned long timestamp;unsigned long gencount;
@@ -466,15 +467,15 @@ static int nfs_readdir_search_array(struct nfs_readdir_descriptor *desc)/* Fill a page with xdr information before transferring to the cache page */ static int nfs_readdir_xdr_filler (struct nfs_readdir_descriptor *desc, -u64 cookie,struct page **pages, -size_t bufsize) +__be32 *verf,u64 cookie, +struct page **pages,size_t bufsize, +__be32 *verf_res) {struct inode *inode =file_inode (desc->file); -__be32 verf_res[2 ];struct nfs_readdir_arg arg = { .dentry =file_dentry (desc->file), .cred = desc->file->f_cred, - .verf =NFS_I (inode)->cookieverf, + .verf = verf, .cookie = cookie, .pages = pages, .page_len = bufsize,
@@ -503,8 +504,6 @@ static int nfs_readdir_xdr_filler(struct nfs_readdir_descriptor *desc, } desc->timestamp = timestamp; desc->gencount = gencount; -memcpy (NFS_I (inode)->cookieverf, res.verf, -sizeof (NFS_I (inode)->cookieverf)); error:return error; }
@@ -770,11 +769,13 @@ static struct page **nfs_readdir_alloc_pages(size_t npages) }static int nfs_readdir_xdr_to_array (struct nfs_readdir_descriptor *desc, -struct page *page,struct inode *inode) +struct page *page,__be32 *verf_arg, +__be32 *verf_res) {struct page **pages;struct nfs_entry *entry;size_t array_size; +struct inode *inode =file_inode (desc->file);size_t dtsize =NFS_SERVER (inode)->dtsize;int status = -ENOMEM ;
@@ -801,8 +802,9 @@ static int nfs_readdir_xdr_to_array(struct nfs_readdir_descriptor *desc, do { unsigned int pglen; - status = nfs_readdir_xdr_filler(desc, entry->cookie, - pages, dtsize); + status = nfs_readdir_xdr_filler(desc, verf_arg, entry->cookie, + pages, dtsize, + verf_res); if (status < 0) break;
@@ -854,13 +856,15 @@ static int find_and_lock_cache_page(struct nfs_readdir_descriptor *desc) {struct inode *inode =file_inode (desc->file);struct nfs_inode *nfsi =NFS_I (inode); +__be32 verf[NFS_DIR_VERIFIER_SIZE ];int res; desc->page =nfs_readdir_page_get_cached (desc);if (!desc->page)return -ENOMEM ;if (nfs_readdir_page_needs_filling (desc->page)) { - res =nfs_readdir_xdr_to_array (desc, desc->page, inode); + res =nfs_readdir_xdr_to_array (desc, desc->page, + nfsi->cookieverf, verf);if (res <0 ) {nfs_readdir_page_unlock_and_put_cached (desc);if (res == -EBADCOOKIE || res == -ENOTSYNC ) {
@@ -870,6 +874,7 @@ static int find_and_lock_cache_page(struct nfs_readdir_descriptor *desc) }return res; } +memcpy (nfsi->cookieverf, verf,sizeof (nfsi->cookieverf)); } res =nfs_readdir_search_array (desc);if (res ==0 ) {
@@ -902,6 +907,7 @@ static int readdir_search_pagecache(struct nfs_readdir_descriptor *desc)static void nfs_do_filldir (struct nfs_readdir_descriptor *desc) {struct file *file = desc->file; +struct nfs_inode *nfsi =NFS_I (file_inode (file));struct nfs_cache_array *array;unsigned int i =0 ;
@@ -915,6 +921,7 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc) desc->eof = true;break ; } +memcpy (desc->verf, nfsi->cookieverf,sizeof (desc->verf));if (i < (array->size-1 )) desc->dir_cookie = array->array[i+1 ].cookie;else
@@ -949,8 +956,8 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc)static int uncached_readdir (struct nfs_readdir_descriptor *desc) {struct page *page =NULL ; +__be32 verf[NFS_DIR_VERIFIER_SIZE ];int status; -struct inode *inode =file_inode (desc->file);dfprintk (DIRCACHE ,"NFS: uncached_readdir() searching for cookie %Lu\n" , (unsigned long long )desc->dir_cookie);
@@ -967,7 +974,7 @@ static int uncached_readdir(struct nfs_readdir_descriptor *desc) desc->duped =0 ;nfs_readdir_page_init_array (page, desc->dir_cookie); - status =nfs_readdir_xdr_to_array (desc, page, inode); + status =nfs_readdir_xdr_to_array (desc, page, desc->verf, verf);if (status <0 ) goto out_release;
@@ -1023,6 +1030,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) desc->dup_cookie = dir_ctx->dup_cookie; desc->duped = dir_ctx->duped; desc->attr_gencount = dir_ctx->attr_gencount; +memcpy (desc->verf, dir_ctx->verf,sizeof (desc->verf));spin_unlock (&file->f_lock); do {
@@ -1061,6 +1069,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) dir_ctx->dup_cookie = desc->dup_cookie; dir_ctx->duped = desc->duped; dir_ctx->attr_gencount = desc->attr_gencount; +memcpy (dir_ctx->verf, desc->verf,sizeof (dir_ctx->verf));spin_unlock (&file->f_lock);kfree (desc);
@@ -1101,6 +1110,8 @@ static loff_t nfs_llseek_dir(struct file *filp, loff_t offset, int whence) dir_ctx->dir_cookie = offset; else dir_ctx->dir_cookie = 0; +if (offset ==0 ) +memset (dir_ctx->verf,0 ,sizeof (dir_ctx->verf)); dir_ctx->duped =0 ; }spin_unlock (&filp->f_lock);
fs/nfs/inode.c +0 -7
@@ -229,7 +229,6 @@ static void nfs_zap_caches_locked(struct inode *inode) nfsi->attrtimeo =NFS_MINATTRTIMEO (inode); nfsi->attrtimeo_timestamp = jiffies; -memset (NFS_I (inode)->cookieverf,0 ,sizeof (NFS_I (inode)->cookieverf));if (S_ISREG (mode) ||S_ISDIR (mode) ||S_ISLNK (mode)) {nfs_set_cache_invalid (inode,NFS_INO_INVALID_ATTR | NFS_INO_INVALID_DATA
@@ -1237,7 +1236,6 @@ EXPORT_SYMBOL_GPL(nfs_revalidate_inode);static int nfs_invalidate_mapping (struct inode *inode,struct address_space *mapping) { -struct nfs_inode *nfsi =NFS_I (inode);int ret;if (mapping->nrpages !=0 ) {
@@ -1250,11 +1248,6 @@ static int nfs_invalidate_mapping(struct inode *inode, struct address_space *mapif (ret <0 )return ret; } -if (S_ISDIR (inode->i_mode)) { -spin_lock (&inode->i_lock); -memset (nfsi->cookieverf,0 ,sizeof (nfsi->cookieverf)); -spin_unlock (&inode->i_lock); - }nfs_inc_stats (inode,NFSIOS_DATAINVALIDATE );nfs_fscache_wait_on_invalidate (inode);
include/linux/nfs_fs.h +7 -1
@@ -45,6 +45,11 @@*/ #define NFS_RPC_SWAPFLAGS (RPC_TASK_SWAPPER|RPC_TASK_ROOTCREDS) +/* +* Size of the NFS directory verifier +*/ +#define NFS_DIR_VERIFIER_SIZE 2 +/* * NFSv3/v4 Access mode cache entry */
@@ -89,6 +94,7 @@ struct nfs_open_context {struct nfs_open_dir_context {struct list_head list;unsigned long attr_gencount; +__be32 verf[NFS_DIR_VERIFIER_SIZE ];__u64 dir_cookie;__u64 dup_cookie;signed char duped;
@@ -156,7 +162,7 @@ struct nfs_inode {* This is the cookie verifier used for NFSv3 readdir * operations */ -__be32 cookieverf[2 ]; +__be32 cookieverf[NFS_DIR_VERIFIER_SIZE ];atomic_long_t nrequests;struct nfs_mds_commit_info commit_info;
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 20/22 NFS: Optimisations for monotonically increasing readdir cookies
From: Trond Myklebust <[email protected]>
If the server is handing out monotonically increasing readdir cookie values, then we can optimise away searches through pages that contain cookies that lie outside our search range.
--- fs/nfs/dir.c | 23 ++++++++++++++++++++++- 1 file changed, 22 insertions(+), 1 deletion(-)
fs/nfs/dir.c +22 -1
@@ -140,7 +140,8 @@ struct nfs_cache_array {u64 last_cookie;unsigned int size; unsigned char page_full : 1, - page_is_eof : 1; + page_is_eof : 1, + cookies_are_ordered : 1;struct nfs_cache_array_entry array[]; };
@@ -178,6 +179,7 @@ static void nfs_readdir_page_init_array(struct page *page, u64 last_cookie) array =kmap_atomic (page);nfs_readdir_array_init (array); array->last_cookie = last_cookie; + array->cookies_are_ordered =1 ;kunmap_atomic (array); }
@@ -269,6 +271,8 @@ int nfs_readdir_add_to_array(struct nfs_entry *entry, struct page *page) cache_entry->name_len = entry->len; cache_entry->name = name; array->last_cookie = entry->cookie; +if (array->last_cookie <= cache_entry->cookie) + array->cookies_are_ordered =0 ; array->size++;if (entry->eof !=0 )nfs_readdir_array_set_eof (array);
@@ -395,6 +399,19 @@ nfs_readdir_inode_mapping_valid(struct nfs_inode *nfsi)return !test_bit (NFS_INO_INVALIDATING , &nfsi->flags); } +static bool nfs_readdir_array_cookie_in_range (struct nfs_cache_array *array, +u64 cookie) +{ +if (!array->cookies_are_ordered) +return true; +/* Optimisation for monotonically increasing cookies */ +if (cookie >= array->last_cookie) +return false; +if (array->size && cookie < array->array[0 ].cookie) +return false; +return true; +} +static int nfs_readdir_search_for_cookie (struct nfs_cache_array *array,struct nfs_readdir_descriptor *desc) {
@@ -402,6 +419,9 @@ static int nfs_readdir_search_for_cookie(struct nfs_cache_array *array,loff_t new_pos;int status = -EAGAIN ; +if (!nfs_readdir_array_cookie_in_range (array, desc->dir_cookie)) + goto check_eof; +for (i =0 ; i < array->size; i++) {if (array->array[i].cookie == desc->dir_cookie) {struct nfs_inode *nfsi =NFS_I (file_inode (desc->file));
@@ -435,6 +455,7 @@ static int nfs_readdir_search_for_cookie(struct nfs_cache_array *array,return 0 ; } } +check_eof:if (array->page_is_eof) { status = -EBADCOOKIE ;if (desc->dir_cookie == array->last_cookie)
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 21/22 NFS: Reduce number of RPC calls when doing uncached readdir
From: Trond Myklebust <[email protected]>
If we're doing uncached readdir, allocate multiple pages in order to try to avoid duplicate RPC calls for the same getdents() call.
--- fs/nfs/dir.c | 105 +++++++++++++++++++++++++++++++++------------------ 1 file changed, 69 insertions(+), 36 deletions(-)
fs/nfs/dir.c +69 -36
@@ -199,6 +199,23 @@ void nfs_readdir_clear_array(struct page *page)kunmap_atomic (array); } +static struct page * +nfs_readdir_page_array_alloc (u64 last_cookie,gfp_t gfp_flags) +{ +struct page *page =alloc_page (gfp_flags); +if (page) +nfs_readdir_page_init_array (page, last_cookie); +return page; +} + +static void nfs_readdir_page_array_free (struct page *page) +{ +if (page) { +nfs_readdir_clear_array (page); +put_page (page); + } +} +static void nfs_readdir_array_set_eof (struct nfs_cache_array *array) { array->page_is_eof =1 ;
@@ -694,12 +711,14 @@ void nfs_prime_dcache(struct dentry *parent, struct nfs_entry *entry,static int nfs_readdir_page_filler (struct nfs_readdir_descriptor *desc,struct nfs_entry *entry,struct page **xdr_pages, -struct page *fillme,unsigned int buflen) +unsigned int buflen, +struct page **arrays, +size_t narrays) {struct address_space *mapping = desc->file->f_mapping;struct xdr_stream stream;struct xdr_buf buf; -struct page *scratch, *new, *page = fillme; +struct page *scratch, *new, *page = *arrays;int status; scratch =alloc_page (GFP_KERNEL );
@@ -725,15 +744,25 @@ static int nfs_readdir_page_filler(struct nfs_readdir_descriptor *desc,if (status != -ENOSPC )continue ; -if (page->mapping != mapping) -break ; - new =nfs_readdir_page_get_next (mapping, page->index +1 , - entry->prev_cookie); -if (!new) -break ; -if (page != fillme) -nfs_readdir_page_unlock_and_put (page); - page = new; +if (page->mapping != mapping) { +if (!--narrays) +break ; + new =nfs_readdir_page_array_alloc (entry->prev_cookie, +GFP_KERNEL ); +if (!new) +break ; + arrays++; + *arrays = page = new; + }else { + new =nfs_readdir_page_get_next (mapping, + page->index +1 , + entry->prev_cookie); +if (!new) +break ; +if (page != *arrays) +nfs_readdir_page_unlock_and_put (page); + page = new; + } status =nfs_readdir_add_to_array (entry, page); }while (!status && !entry->eof);
@@ -750,7 +779,7 @@ static int nfs_readdir_page_filler(struct nfs_readdir_descriptor *desc,break ; } -if (page != fillme) +if (page != *arrays)nfs_readdir_page_unlock_and_put (page);put_page (scratch);
@@ -790,10 +819,11 @@ static struct page **nfs_readdir_alloc_pages(size_t npages) }static int nfs_readdir_xdr_to_array (struct nfs_readdir_descriptor *desc, -struct page *page,__be32 *verf_arg, -__be32 *verf_res) +__be32 *verf_arg,__be32 *verf_res, +struct page **arrays,size_t narrays) {struct page **pages; +struct page *page = *arrays;struct nfs_entry *entry;size_t array_size;struct inode *inode =file_inode (desc->file);
@@ -835,7 +865,8 @@ static int nfs_readdir_xdr_to_array(struct nfs_readdir_descriptor *desc,break ; } - status =nfs_readdir_page_filler (desc, entry, pages, page, pglen); + status =nfs_readdir_page_filler (desc, entry, pages, pglen, + arrays, narrays); }while (!status &&nfs_readdir_page_needs_filling (page));nfs_readdir_free_pages (pages, array_size);
@@ -884,8 +915,8 @@ static int find_and_lock_cache_page(struct nfs_readdir_descriptor *desc)if (!desc->page)return -ENOMEM ;if (nfs_readdir_page_needs_filling (desc->page)) { - res =nfs_readdir_xdr_to_array (desc, desc->page, - nfsi->cookieverf, verf); + res =nfs_readdir_xdr_to_array (desc, nfsi->cookieverf, verf, + &desc->page,1 );if (res <0 ) {nfs_readdir_page_unlock_and_put_cached (desc);if (res == -EBADCOOKIE || res == -ENOTSYNC ) {
@@ -976,37 +1007,39 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc)*/ static int uncached_readdir (struct nfs_readdir_descriptor *desc) { -struct page *page =NULL ; +struct page **arrays; +size_t i, sz =512 ;__be32 verf[NFS_DIR_VERIFIER_SIZE ]; -int status; +int status = -ENOMEM ; -dfprintk (DIRCACHE ,"NFS: uncached_readdir() searching for cookie %Lu\n" , +dfprintk (DIRCACHE ,"NFS: uncached_readdir() searching for cookie %llu\n" , (unsigned long long )desc->dir_cookie); - page =alloc_page (GFP_HIGHUSER ); -if (!page) { - status = -ENOMEM ; + arrays =kcalloc (sz,sizeof (*arrays),GFP_KERNEL ); +if (!arrays) + goto out; + arrays[0 ] =nfs_readdir_page_array_alloc (desc->dir_cookie,GFP_KERNEL ); +if (!arrays[0 ]) goto out; - } desc->page_index =0 ; desc->last_cookie = desc->dir_cookie; - desc->page = page; desc->duped =0 ; -nfs_readdir_page_init_array (page, desc->dir_cookie); - status =nfs_readdir_xdr_to_array (desc, page, desc->verf, verf); -if (status <0 ) - goto out_release; + status =nfs_readdir_xdr_to_array (desc, desc->verf, verf, arrays, sz); -nfs_do_filldir (desc); +for (i =0 ; !desc->eof && i < sz && arrays[i]; i++) { + desc->page = arrays[i]; +nfs_do_filldir (desc); + } + desc->page =NULL ; - out_release: -nfs_readdir_clear_array (desc->page); -nfs_readdir_page_put (desc); - out: -dfprintk (DIRCACHE ,"NFS: %s: returns %d\n" , - __func__, status); + +for (i =0 ; i < sz && arrays[i]; i++) +nfs_readdir_page_array_free (arrays[i]); +out: +kfree (arrays); +dfprintk (DIRCACHE ,"NFS: %s: returns %d\n" , __func__, status);return status; }
Signature · 1 line
2.28.0
[email protected]
maintainerPATCH v5 22/22 NFS: Do uncached readdir when we're seeking a cookie in an empty page cache
From: Trond Myklebust <[email protected]>
If the directory is changing, causing the page cache to get invalidated while we are listing the contents, then the NFS client is currently forced to read in the entire directory contents from scratch, because it needs to perform a linear search for the readdir cookie. While this is not an issue for small directories, it does not scale to directories with millions of entries.
In order to be able to deal with large directories that are changing, add a heuristic to ensure that if the page cache is empty, and we are searching for a cookie that is not the zero cookie, we just default to performing uncached readdir.
--- fs/nfs/dir.c | 17 +++++++++++++++++ 1 file changed, 17 insertions(+)
fs/nfs/dir.c +17 -0
@@ -937,11 +937,28 @@ static int find_and_lock_cache_page(struct nfs_readdir_descriptor *desc)return res; } +static bool nfs_readdir_dont_search_cache (struct nfs_readdir_descriptor *desc) +{ +struct address_space *mapping = desc->file->f_mapping; +struct inode *dir =file_inode (desc->file); +unsigned int dtsize =NFS_SERVER (dir)->dtsize; +loff_t size =i_size_read (dir); + +/* +* Default to uncached readdir if the page cache is empty, and +* we're looking for a non-zero cookie in a large directory. +*/ +return desc->dir_cookie !=0 && mapping->nrpages ==0 && size > dtsize; +} +/* Search for desc->dir_cookie from the beginning of the page cache */ static int readdir_search_pagecache (struct nfs_readdir_descriptor *desc) {int res; +if (nfs_readdir_dont_search_cache (desc)) +return -EBADCOOKIE ; + do { if (desc->page_index == 0) { desc->current_index = 0;
Signature · 1 line
2.28.0
Quoted 67 lines · On Tue, Nov 10, 2020 at 4:48 PM <[email protected]> wrote:
From: Trond Myklebust <[email protected]>
The following patch series performs a number of cleanups on the readdir code.
It also adds support for 1MB readdir RPC calls on-the-wire, and modifies the caching code to ensure that we cache the entire contents of that 1MB call (instead of discarding the data that doesn't fit into a single page).
For filesystems that use ordered readdir cookie schemes (e.g. XFS), it optimises searching for cookies in the client's page cache by skipping over pages that contain cookie values that are not in the range we are searching for.
Finally, it improves scalability when dealing with very large directories by turning off caching when those directories are changing, so as to avoid the need for a linear search on the client of the entire directory when looking for the first entry pointed to by the current file descriptor offset.v2: Fix the handling of the NFSv3/v4 directory verifier.
v3: Optimise searching when the readdir cookies are seen to be ordered.
v4: Optimise performance for large directories that are changing.
Add in llseek dependency patches.
v5: Integrate Olga's patch for the READDIR security label handling.
Record more entries in the uncached readdir case. Bump the max number of pages to 512, but allocate them on demand in case the readdir RPC call returns fewer entries.Olga Kornievskaia (1):
NFSv4.2: condition READDIR's mask for security label based on LSM
stateTrond Myklebust (21):
NFS: Remove unnecessary inode locking innfs_llseek_dir()
NFS: Remove unnecessary inode lock innfs_fsync_dir()
NFS: Ensure contents of structnfs_open_dir_contextare consistent
NFS: Clean up readdir structnfs_cache_array
NFS: Clean upnfs_readdir_page_filler()
NFS: Clean up directory array handling
NFS: Don't discard readdir results
NFS: Remove unnecessary kmap innfs_readdir_xdr_to_array()
NFS: Replacekmap()withkmap_atomic()innfs_readdir_search_array()
NFS: Simplify structnfs_cache_array_entry
NFS: Support larger readdir buffers
NFS: More readdir cleanups
NFS:nfs_do_filldir()does not return a value
NFS: Reduce readdir stack usage
NFS: Cleanup to removenfs_readdir_descriptor_ttypedef
NFS: Allow the NFS generic code to pass in a verifier to readdir
NFS: HandleNFS4ERR_NOT_SAMEandNFSERR_BADCOOKIEfrom readdir calls
NFS: Improve handling of directory verifiers
NFS: Optimisations for monotonically increasing readdir cookies
NFS: Reduce number of RPC calls when doing uncached readdir
NFS: Do uncached readdir when we're seeking a cookie in an empty page
cachefs/nfs/client.c | 4 +- fs/nfs/dir.c | 734 +++++++++++++++++++++++++--------------- fs/nfs/inode.c | 7 - fs/nfs/internal.h | 6 - fs/nfs/nfs3proc.c | 35 +- fs/nfs/nfs4proc.c | 48 +-- fs/nfs/proc.c | 18 +- include/linux/nfs_fs.h | 9 +- include/linux/nfs_xdr.h | 17 +- 9 files changed, 541 insertions(+), 337 deletions(-)
Quoted 3 lines
--
2.28.0
I wrote a test script and ran this patchset against 5.10-rc2 in a variety of scenarios listing 1 million files and various directory modification scenarios. The test checked ops, runtime and cookie resets (via tcpdump).
All runs performed very well on all scenarios I threw at it on all NFS versions (I tested 3, 4.0, 4.1, and 4.2) with fairly dramatic
improvements vs the same runs on 5.10-rc2. One scenario I found
where a single 'ls -l' could take a long time (about 5 minutes, but not unbounded time) was when the directory was being modified with both file adds and removed, but this seemed due to other ops unrelated
to readdir performance. A second scenario that this patchset does
not fix is the scenario where the directory is large enough to exceed the acdirmax, is being modified (adds and removes), and _multiple_ processes are listing it. In this scenario a lister can still have unbounded time, but the existing readdir code also has this problem and fixing that probably requires another patch / approach (such as reworking the structure of the cache or an approach such as the flag on per-process dir context).
I think these patches make NFS readdir dramatically better, and they fix a lot of underlying issues with the existing code, like the cookie resets when pagecache is dropped, single page rx data problem, etc. Thank you for doing these patches.
Quoted 78 lines · David Wysochanski
Quoted 67 lines · On Tue, Nov 10, 2020 at 4:48 PM <[email protected]> wrote:
From: Trond Myklebust <[email protected]>
The following patch series performs a number of cleanups on the readdir code.
It also adds support for 1MB readdir RPC calls on-the-wire, and modifies the caching code to ensure that we cache the entire contents of that 1MB call (instead of discarding the data that doesn't fit into a single page).
For filesystems that use ordered readdir cookie schemes (e.g. XFS), it optimises searching for cookies in the client's page cache by skipping over pages that contain cookie values that are not in the range we are searching for.
Finally, it improves scalability when dealing with very large directories by turning off caching when those directories are changing, so as to avoid the need for a linear search on the client of the entire directory when looking for the first entry pointed to by the current file descriptor offset.v2: Fix the handling of the NFSv3/v4 directory verifier.
v3: Optimise searching when the readdir cookies are seen to be ordered.
v4: Optimise performance for large directories that are changing.
Add in llseek dependency patches.
v5: Integrate Olga's patch for the READDIR security label handling.
Record more entries in the uncached readdir case. Bump the max number of pages to 512, but allocate them on demand in case the readdir RPC call returns fewer entries.Olga Kornievskaia (1):
NFSv4.2: condition READDIR's mask for security label based on LSM
stateTrond Myklebust (21):
NFS: Remove unnecessary inode locking innfs_llseek_dir()
NFS: Remove unnecessary inode lock innfs_fsync_dir()
NFS: Ensure contents of structnfs_open_dir_contextare consistent
NFS: Clean up readdir structnfs_cache_array
NFS: Clean upnfs_readdir_page_filler()
NFS: Clean up directory array handling
NFS: Don't discard readdir results
NFS: Remove unnecessary kmap innfs_readdir_xdr_to_array()
NFS: Replacekmap()withkmap_atomic()innfs_readdir_search_array()
NFS: Simplify structnfs_cache_array_entry
NFS: Support larger readdir buffers
NFS: More readdir cleanups
NFS:nfs_do_filldir()does not return a value
NFS: Reduce readdir stack usage
NFS: Cleanup to removenfs_readdir_descriptor_ttypedef
NFS: Allow the NFS generic code to pass in a verifier to readdir
NFS: HandleNFS4ERR_NOT_SAMEandNFSERR_BADCOOKIEfrom readdir calls
NFS: Improve handling of directory verifiers
NFS: Optimisations for monotonically increasing readdir cookies
NFS: Reduce number of RPC calls when doing uncached readdir
NFS: Do uncached readdir when we're seeking a cookie in an empty page
cachefs/nfs/client.c | 4 +- fs/nfs/dir.c | 734 +++++++++++++++++++++++++--------------- fs/nfs/inode.c | 7 - fs/nfs/internal.h | 6 - fs/nfs/nfs3proc.c | 35 +- fs/nfs/nfs4proc.c | 48 +-- fs/nfs/proc.c | 18 +- include/linux/nfs_fs.h | 9 +- include/linux/nfs_xdr.h | 17 +- 9 files changed, 541 insertions(+), 337 deletions(-)Quoted 3 lines
--
2.28.0I wrote a test script and ran this patchset against 5.10-rc2 in a variety of scenarios listing 1 million files and various directory modification scenarios. The test checked ops, runtime and cookie resets (via tcpdump).
Quoted 15 lines
All runs performed very well on all scenarios I threw at it on all NFS versions (I tested 3, 4.0, 4.1, and 4.2) with fairly dramatic
improvements vs the same runs on 5.10-rc2. One scenario I found
where a single 'ls -l' could take a long time (about 5 minutes, but not unbounded time) was when the directory was being modified with both file adds and removed, but this seemed due to other ops unrelated
to readdir performance. A second scenario that this patchset does
not fix is the scenario where the directory is large enough to exceed the acdirmax, is being modified (adds and removes), and _multiple_ processes are listing it. In this scenario a lister can still have unbounded time, but the existing readdir code also has this problem and fixing that probably requires another patch / approach (such as reworking the structure of the cache or an approach such as the flag on per-process dir context).
I think I am mistaken in my previous email that said multiple listers will not make forward progress. After more testing I could not reproduce any indefinite wait but it was many minutes, and I think there is proof in the current code (patch 22) that an indefinite wait should not occur with any process (multiple processes does not change this). Note that I did not trace this, but below is an attempt at a more detailed explanation from the code. The summary of the argument is that periodically nfs_readdir_dont_search_cache() will be true for one iteration of nfs_readdir(), and this ensures forward progress, even if it may be slow.
Some notation for the below explanation.
Tu: Time to list unmodified (idle) directory
Ax: acdirmax
Assume Tu > acdirmax (time to list idle directory is larger than acdirmax) and i_size_read(dir) > NFS_SERVER(dir)->dtsize
PL1: pid1 that is listing the directory
DC1, DC2 and DC3: values of nfs_open_dir_context.dir_cookie for PL1PL1_dir_cookie: the current value of PL1's nfs_open_dir_context.dir_cookie
Then consider the following timeline, with "Tn" various points on a timeline of PL1 listing a very large directory.
T0: PL1 starts listing directory with repeated calls to nfs_readdir()
T1: acdirmax is exceeded, and we drop the pagecache (mapping->nrpages == 0); at this point, PL1_dir_cookie = DC1 != 0
T2: PL1 calls nfs_readdir and nfs_readdir_dont_search_cache() returns true (due to mapping->nrpages == 0), so enters uncached_readdir(); thus, PL1_dir_cookie makes forward progress so PL1_dir_cookie is ahead of DC1 and the exit of nfs_readdir()
T3: PL1 calls nfs_readdir() and nfs_readdir_dont_search_cache() is false, so it must restart filling the pagecache from cookie == 0 (we send READDIRs over the wire); at this point PL1_dir_cookie forward progress is stalled, and PL1's calls to nfs_readdir() will have to fetch entries and fill pages in the pagecache that does not pertain to its listing (they've already been returned to userspace in a previous nfs_readdir call)
T4: acdirmax is exceeded, and we drop the pagecache again, setting mapping->nrpages == 0; at this point though, PL1_dir_cookie = DC2, where DC2 is ahead of DC1 (we made forward progress last time)
T5: PL1 calls nfs_readdir and nfs_readdir_dont_search_cache() returns true, so enters uncached_readdir(); thus, PL1_dir_cookie makes forward progress again for this one call to nfs_readdir(), and PL1_dir_cookie = DC3 is ahead of DC2
Thus, it seems PL1 will eventually complete, even if it is delayed a bit, and even if it has to re-fill pages in the page cache that is "extra work" that doesn't help PL1. Forward progress is guaranteed due to the periodic calls to nfs_readdir() when nfs_readdir_dont_search_cache() returns true due to mapping->nrpages == 0 portion of the condition inside nfs_readdir_dont_search_cache() +static bool nfs_readdir_dont_search_cache(struct nfs_readdir_descriptor *desc) +{
+ struct address_space *mapping = desc->file->f_mapping; + struct inode *dir = file_inode(desc->file); + unsigned int dtsize = NFS_SERVER(dir)->dtsize; + loff_t size = i_size_read(dir); + + /* + * Default to uncached readdir if the page cache is empty, and + * we're looking for a non-zero cookie in a large directory. + */
+ return desc->dir_cookie != 0 && mapping->nrpages == 0 && size > dtsize;
+}
Quoted 6 lines
I think these patches make NFS readdir dramatically better, and they fix a lot of underlying issues with the existing code, like the cookie resets when pagecache is dropped, single page rx data problem, etc. Thank you for doing these patches.
Tested-by Dave Wysochanski
just a general question: since the cache seems to cause many problems when dealing with very large directories, and since all solutions proposed until now don't seem to fully solve those problems, won't an approach such as "if the directory entries count exceeded X - stop using the cache completely" - where X is proportional to the size of the directory entries cache size limit - make the code simpler, and less prone to bugs of this sort?
i *think* we can understand that for a directory with millions of files, we'll not have efficient caching on the client side, while limiting ourselves to reasonable RAM consumption?
--guy keren
Vast data
Quoted 171 lines · David Wysochanski
Quoted 78 lines · David Wysochanski
Quoted 67 lines · On Tue, Nov 10, 2020 at 4:48 PM <[email protected]> wrote:
From: Trond Myklebust <[email protected]>
The following patch series performs a number of cleanups on the readdir code.
It also adds support for 1MB readdir RPC calls on-the-wire, and modifies the caching code to ensure that we cache the entire contents of that 1MB call (instead of discarding the data that doesn't fit into a single page).
For filesystems that use ordered readdir cookie schemes (e.g. XFS), it optimises searching for cookies in the client's page cache by skipping over pages that contain cookie values that are not in the range we are searching for.
Finally, it improves scalability when dealing with very large directories by turning off caching when those directories are changing, so as to avoid the need for a linear search on the client of the entire directory when looking for the first entry pointed to by the current file descriptor offset.v2: Fix the handling of the NFSv3/v4 directory verifier.
v3: Optimise searching when the readdir cookies are seen to be ordered.
v4: Optimise performance for large directories that are changing.
Add in llseek dependency patches.
v5: Integrate Olga's patch for the READDIR security label handling.
Record more entries in the uncached readdir case. Bump the max number of pages to 512, but allocate them on demand in case the readdir RPC call returns fewer entries.Olga Kornievskaia (1):
NFSv4.2: condition READDIR's mask for security label based on LSM
stateTrond Myklebust (21):
NFS: Remove unnecessary inode locking innfs_llseek_dir()
NFS: Remove unnecessary inode lock innfs_fsync_dir()
NFS: Ensure contents of structnfs_open_dir_contextare consistent
NFS: Clean up readdir structnfs_cache_array
NFS: Clean upnfs_readdir_page_filler()
NFS: Clean up directory array handling
NFS: Don't discard readdir results
NFS: Remove unnecessary kmap innfs_readdir_xdr_to_array()
NFS: Replacekmap()withkmap_atomic()innfs_readdir_search_array()
NFS: Simplify structnfs_cache_array_entry
NFS: Support larger readdir buffers
NFS: More readdir cleanups
NFS:nfs_do_filldir()does not return a value
NFS: Reduce readdir stack usage
NFS: Cleanup to removenfs_readdir_descriptor_ttypedef
NFS: Allow the NFS generic code to pass in a verifier to readdir
NFS: HandleNFS4ERR_NOT_SAMEandNFSERR_BADCOOKIEfrom readdir calls
NFS: Improve handling of directory verifiers
NFS: Optimisations for monotonically increasing readdir cookies
NFS: Reduce number of RPC calls when doing uncached readdir
NFS: Do uncached readdir when we're seeking a cookie in an empty page
cachefs/nfs/client.c | 4 +- fs/nfs/dir.c | 734 +++++++++++++++++++++++++--------------- fs/nfs/inode.c | 7 - fs/nfs/internal.h | 6 - fs/nfs/nfs3proc.c | 35 +- fs/nfs/nfs4proc.c | 48 +-- fs/nfs/proc.c | 18 +- include/linux/nfs_fs.h | 9 +- include/linux/nfs_xdr.h | 17 +- 9 files changed, 541 insertions(+), 337 deletions(-)Quoted 3 lines
--
2.28.0I wrote a test script and ran this patchset against 5.10-rc2 in a variety of scenarios listing 1 million files and various directory modification scenarios. The test checked ops, runtime and cookie resets (via tcpdump).
Quoted 15 lines
All runs performed very well on all scenarios I threw at it on all NFS versions (I tested 3, 4.0, 4.1, and 4.2) with fairly dramatic
improvements vs the same runs on 5.10-rc2. One scenario I found
where a single 'ls -l' could take a long time (about 5 minutes, but not unbounded time) was when the directory was being modified with both file adds and removed, but this seemed due to other ops unrelated
to readdir performance. A second scenario that this patchset does
not fix is the scenario where the directory is large enough to exceed the acdirmax, is being modified (adds and removes), and _multiple_ processes are listing it. In this scenario a lister can still have unbounded time, but the existing readdir code also has this problem and fixing that probably requires another patch / approach (such as reworking the structure of the cache or an approach such as the flag on per-process dir context).I think I am mistaken in my previous email that said multiple listers will not make forward progress. After more testing I could not reproduce any indefinite wait but it was many minutes, and I think there is proof in the current code (patch 22) that an indefinite wait should not occur with any process (multiple processes does not change this). Note that I did not trace this, but below is an attempt at a more detailed explanation from the code. The summary of the argument is that periodically
nfs_readdir_dont_search_cache()will be true for one iteration ofnfs_readdir(), and this ensures forward progress, even if it may be slow.Some notation for the below explanation.
Tu: Time to list unmodified (idle) directory
Ax: acdirmax
Assume Tu > acdirmax (time to list idle directory is larger than acdirmax) andi_size_read(dir)> NFS_SERVER(dir)->dtsize
PL1: pid1 that is listing the directory
DC1, DC2 and DC3: values of nfs_open_dir_context.dir_cookie for PL1PL1_dir_cookie: the current value of PL1's nfs_open_dir_context.dir_cookieThen consider the following timeline, with "Tn" various points on a timeline of PL1 listing a very large directory.
T0: PL1 starts listing directory with repeated calls to
nfs_readdir()
T1: acdirmax is exceeded, and we drop the pagecache (mapping->nrpages== 0); at this point,PL1_dir_cookie= DC1 != 0
T2: PL1 callsnfs_readdirandnfs_readdir_dont_search_cache()returns true (due tomapping->nrpages== 0), so entersuncached_readdir(); thus,PL1_dir_cookiemakes forward progress soPL1_dir_cookieis ahead of DC1 and the exit ofnfs_readdir()
T3: PL1 callsnfs_readdir()andnfs_readdir_dont_search_cache()is false, so it must restart filling the pagecache from cookie == 0 (we send READDIRs over the wire); at this pointPL1_dir_cookieforward progress is stalled, and PL1's calls tonfs_readdir()will have to fetch entries and fill pages in the pagecache that does not pertain to its listing (they've already been returned to userspace in a previousnfs_readdircall)
T4: acdirmax is exceeded, and we drop the pagecache again, settingmapping->nrpages== 0; at this point though,PL1_dir_cookie= DC2, where DC2 is ahead of DC1 (we made forward progress last time)
T5: PL1 callsnfs_readdirandnfs_readdir_dont_search_cache()returns true, so entersuncached_readdir(); thus,PL1_dir_cookiemakes forward progress again for this one call tonfs_readdir(), andPL1_dir_cookie= DC3 is ahead of DC2Thus, it seems PL1 will eventually complete, even if it is delayed a bit, and even if it has to re-fill pages in the page cache that is "extra work" that doesn't help PL1. Forward progress is guaranteed due to the periodic calls to
nfs_readdir()whennfs_readdir_dont_search_cache()returns true due tomapping->nrpages== 0 portion of the condition insidenfs_readdir_dont_search_cache()(quoted diff) +13
-0+static bool nfs_readdir_dont_search_cache(struct nfs_readdir_descriptor *desc) +{ + struct address_space *mapping = desc->file->f_mapping; + struct inode *dir = file_inode(desc->file); + unsigned int dtsize = NFS_SERVER(dir)->dtsize; + loff_t size = i_size_read(dir); + + /* + * Default to uncached readdir if the page cache is empty, and + * we're looking for a non-zero cookie in a large directory. + */ + return desc->dir_cookie != 0 && mapping->nrpages == 0 && size > dtsize; +}Quoted 10 lines
I think these patches make NFS readdir dramatically better, and they fix a lot of underlying issues with the existing code, like the cookie resets when pagecache is dropped, single page rx data problem, etc. Thank you for doing these patches.
Tested-by Dave Wysochanski
Quoted 8 lines · Guy Keren
just a general question: since the cache seems to cause many problems when dealing with very large directories, and since all solutions proposed until now don't seem to fully solve those problems, won't an approach such as "if the directory entries count exceeded X - stop using the cache completely" - where X is proportional to the size of the directory entries cache size limit - make the code simpler, and less prone to bugs of this sort?
Quoted 4 lines
i *think* we can understand that for a directory with millions of files, we'll not have efficient caching on the client side, while limiting ourselves to reasonable RAM consumption?
Again, I disagree.
If you have a mostly-read directory with millions of files (e.g. data pool) and lots of processes searching, then caching is both useful and appropriate.
Quoted 1 line
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 190 lines · David Wysochanski
Quoted 109 lines · David Wysochanski
Quoted 88 lines · On Tue, Nov 10, 2020 at 4:48 PM <[email protected]> wrote:
From: Trond Myklebust <[email protected]>
The following patch series performs a number of cleanups on the readdir
code.
It also adds support for 1MB readdir RPC calls on-the-wire, and modifies
the caching code to ensure that we cache the entire contents of that
1MB call (instead of discarding the data that doesn't fit into a single
page).
For filesystems that use ordered readdir cookie schemes (e.g. XFS), it
optimises searching for cookies in the client's page cache by skipping
over pages that contain cookie values that are not in the range we are
searching for.
Finally, it improves scalability when dealing with very large directories by turning off caching when those directories are changing,
so as to avoid the need for a linear search on the client of the entire
directory when looking for the first entry pointed to by the current
file descriptor offset.v2: Fix the handling of the NFSv3/v4 directory verifier.
v3: Optimise searching when the readdir cookies are seen to be ordered.
v4: Optimise performance for large directories that are changing. Add in llseek dependency patches.
v5: Integrate Olga's patch for the READDIR security label handling.
Record more entries in the uncached readdir case. Bump the max
number of pages to 512, but allocate them on demand in case the
readdir RPC call returns fewer entries.Olga Kornievskaia (1):
NFSv4.2: condition READDIR's mask for security label based on LSM
stateTrond Myklebust (21):
NFS: Remove unnecessary inode locking innfs_llseek_dir()
NFS: Remove unnecessary inode lock innfs_fsync_dir()
NFS: Ensure contents of structnfs_open_dir_contextare consistent
NFS: Clean up readdir structnfs_cache_array
NFS: Clean upnfs_readdir_page_filler()
NFS: Clean up directory array handling
NFS: Don't discard readdir results
NFS: Remove unnecessary kmap innfs_readdir_xdr_to_array()
NFS: Replacekmap()withkmap_atomic()innfs_readdir_search_array()
NFS: Simplify structnfs_cache_array_entry
NFS: Support larger readdir buffers
NFS: More readdir cleanups
NFS:nfs_do_filldir()does not return a value
NFS: Reduce readdir stack usage
NFS: Cleanup to removenfs_readdir_descriptor_ttypedef
NFS: Allow the NFS generic code to pass in a verifier to readdir
NFS: HandleNFS4ERR_NOT_SAMEandNFSERR_BADCOOKIEfrom readdir calls
NFS: Improve handling of directory verifiers
NFS: Optimisations for monotonically increasing readdir cookies
NFS: Reduce number of RPC calls when doing uncached readdir
NFS: Do uncached readdir when we're seeking a cookie in an empty page
cachefs/nfs/client.c | 4 +-
fs/nfs/dir.c | 734 +++++++++++++++++++++++++---------
------
fs/nfs/inode.c | 7 -fs/nfs/internal.h | 6 - fs/nfs/nfs3proc.c | 35 +- fs/nfs/nfs4proc.c | 48 +-- fs/nfs/proc.c | 18 +- include/linux/nfs_fs.h | 9 +- include/linux/nfs_xdr.h | 17 +- 9 files changed, 541 insertions(+), 337 deletions(-)Quoted 3 lines
--
2.28.0I wrote a test script and ran this patchset against 5.10-rc2 in a variety of scenarios listing 1 million files and various directory modification scenarios. The test checked ops, runtime and cookie resets (via tcpdump).
All runs performed very well on all scenarios I threw at it on all NFS
versions (I tested 3, 4.0, 4.1, and 4.2) with fairly dramatic improvements vs the same runs on 5.10-rc2. One scenario I found where a single 'ls -l' could take a long time (about 5 minutes, but not unbounded time) was when the directory was being modified with both file adds and removed, but this seemed due to other ops unrelated
to readdir performance. A second scenario that this patchset does not fix is the scenario where the directory is large enough toQuoted 9 lines
exceed
the acdirmax, is being modified (adds and removes), and _multiple_ processes are listing it. In this scenario a lister can still have unbounded time, but the existing readdir code also has this problem and fixing that probably requires another patch / approach (such as reworking the structure of the cache or an approach such as the flag
on per-process dir context).I think I am mistaken in my previous email that said multiple listers will not make forward progress. After more testing I could not reproduce any indefinite wait but it was many minutes, and I think there is proof in the current code (patch 22) that an indefinite wait should not occur with any process (multiple processes does not change this). Note that I did not trace this, but below is an attempt at a more detailed explanation from the code. The summary of the argument is that periodically
nfs_readdir_dont_search_cache()will be true for one iteration ofnfs_readdir(), and this ensures forward progress, even if it may be slow.Some notation for the below explanation.
Tu: Time to list unmodified (idle) directory
Ax: acdirmax
Assume Tu > acdirmax (time to list idle directory is larger than acdirmax) andi_size_read(dir)> NFS_SERVER(dir)->dtsize
PL1: pid1 that is listing the directory
DC1, DC2 and DC3: values of nfs_open_dir_context.dir_cookie for PL1PL1_dir_cookie: the current value of PL1's nfs_open_dir_context.dir_cookieThen consider the following timeline, with "Tn" various points on a timeline of PL1 listing a very large directory.
T0: PL1 starts listing directory with repeated calls to
nfs_readdir()
T1: acdirmax is exceeded, and we drop the pagecache (mapping->nrpages== 0); at this point,PL1_dir_cookie= DC1 != 0
T2: PL1 callsnfs_readdirandnfs_readdir_dont_search_cache()returns true (due tomapping->nrpages== 0), so entersuncached_readdir(); thus,PL1_dir_cookiemakes forward progress soPL1_dir_cookieis ahead
of DC1 and the exit ofnfs_readdir()
T3: PL1 callsnfs_readdir()andnfs_readdir_dont_search_cache()is false, so it must restart filling the pagecache from cookie == 0 (we send READDIRs over the wire); at this pointPL1_dir_cookieforward progress is stalled, and PL1's calls tonfs_readdir()will have to fetch entries and fill pages in the pagecache that does not pertain to
its listing (they've already been returned to userspace in a previousnfs_readdircall)
T4: acdirmax is exceeded, and we drop the pagecache again, settingmapping->nrpages== 0; at this point though,PL1_dir_cookie= DC2, where DC2 is ahead of DC1 (we made forward progress last time)
T5: PL1 callsnfs_readdirandnfs_readdir_dont_search_cache()returns true, so entersuncached_readdir(); thus,PL1_dir_cookiemakes forward
progress again for this one call tonfs_readdir(), andPL1_dir_cookie= DC3 is ahead of DC2Thus, it seems PL1 will eventually complete, even if it is delayed a bit, and even if it has to re-fill pages in the page cache that is "extra work" that doesn't help PL1. Forward progress is guaranteed due to the periodic calls to
nfs_readdir()whennfs_readdir_dont_search_cache()returns true due tomapping->nrpages== 0 portion of the condition insidenfs_readdir_dont_search_cache()+static boolnfs_readdir_dont_search_cache(struct nfs_readdir_descriptor *desc)(quoted diff) +8
-0+{ + struct address_space *mapping = desc->file->f_mapping; + struct inode *dir = file_inode(desc->file); + unsigned int dtsize = NFS_SERVER(dir)->dtsize; + loff_t size = i_size_read(dir); + + /* + * Default to uncached readdir if the page cache is empty,and
+ * we're looking for a non-zero cookie in a large directory. + */
+ returndesc->dir_cookie!= 0 &&mapping->nrpages== 0 && size
Quoted 3 lines
Quoted 1 line
dtsize;
+}
Cool.
I was going to ask you if perhaps reverting Scott's commit 07b5ce8ef2d8 ("NFS: Make nfs_readdir revalidate less often") might help here? My thinking is that will trigger more cache invalidations when the directory is changing underneath us, and will now trigger uncached readdir in those situations.
Quoted 8 lines
Quoted 8 lines
I think these patches make NFS readdir dramatically better, and they
fix a lot of underlying issues with the existing code, like the cookie
resets when pagecache is dropped, single page rx data problem, etc. Thank you for doing these patches.Tested-by Dave Wysochanski
Thanks very much for the discussion and testing!
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 9 lines · Trond Myklebust
I was going to ask you if perhaps reverting Scott's commit
07b5ce8ef2d8
("NFS: Makenfs_readdirrevalidate less often") might help here? My thinking is that will trigger more cache invalidations when the directory is changing underneath us, and will now trigger uncached readdir in those situations.Quoted 1 line
IOW, the suggestion would be to apply something like the following on top of the existing readdir patchset:
--- fs/nfs/dir.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)
fs/nfs/dir.c +0 -0
@@ -956,10 +956,10 @@ static int readdir_search_pagecache(struct
nfs_readdir_descriptor *desc)
{
int res;
- if (nfs_readdir_dont_search_cache(desc))
- return -EBADCOOKIE;
-
do {
+ if (nfs_readdir_dont_search_cache(desc))
+ return -EBADCOOKIE;
+
if (desc->page_index == 0) {
desc->current_index = 0; desc->prev_index = 0;
@@ -1082,11 +1082,9 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)
* to either find the entry with the appropriate number or
* revalidate the cookie.
*/
- if (ctx->pos == 0 || nfs_attribute_cache_expired(inode)) {
- res = nfs_revalidate_mapping(inode, file->f_mapping);
- if (res < 0)
- goto out;
- }
+ res = nfs_revalidate_mapping(inode, file->f_mapping);
+ if (res < 0)
+ goto out;
res = -ENOMEM;
desc = kzalloc(sizeof(*desc), GFP_KERNEL);
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 11 lines · Trond Myklebust
Quoted 9 lines · Trond Myklebust
I was going to ask you if perhaps reverting Scott's commit
07b5ce8ef2d8
("NFS: Makenfs_readdirrevalidate less often") might help here? My thinking is that will trigger more cache invalidations when the directory is changing underneath us, and will now trigger uncached readdir in those situations.Quoted 1 line
Quoted 2 lines
IOW, the suggestion would be to apply something like the following on top of the existing readdir patchset:
I'm all for this approach, but - I'm rarely seeing the mapping->nrpages == 0 since the cache is dropped by a process in nfs_readdir() that immediately starts filling the cache again.
It would make a lot more sense to me if we could do something like stash desc->page_index << PAGE_SHIFT in f_pos after each nfs_readdir, then the hueristic could check f_pos >> PAGE_SHIFT > nrpages.
Yes, f_pos is growing many different meanings in NFS directories, maybe we can stash it on the directory context.
Ben
Quoted 46 lines
--- fs/nfs/dir.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)fs/nfs/dir.c +3
-3@@ -956,10 +956,10 @@ static int readdir_search_pagecache(struct nfs_readdir_descriptor *desc) {int res; -if (nfs_readdir_dont_search_cache (desc)) -return -EBADCOOKIE ; - do { + if (nfs_readdir_dont_search_cache(desc)) + return -EBADCOOKIE; + if (desc->page_index == 0) { desc->current_index = 0;fs/nfs/dir.c +3
-5desc->prev_index =0 ;@@ -1082,11 +1082,9 @@ static int nfs_readdir(struct file *file, structdir_context *ctx) * to either find the entry with the appropriate number or * revalidate the cookie. */ -if (ctx->pos ==0 ||nfs_attribute_cache_expired (inode)) { - res =nfs_revalidate_mapping (inode, file->f_mapping); -if (res <0 ) - goto out; - } + res =nfs_revalidate_mapping (inode, file->f_mapping); +if (res <0 ) + goto out; res = -ENOMEM ;desc =
kzalloc(sizeof(*desc), GFP_KERNEL);Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 20 lines · Benjamin Coddington
Quoted 11 lines · Trond Myklebust
Quoted 9 lines · Trond Myklebust
I was going to ask you if perhaps reverting Scott's commit
07b5ce8ef2d8
("NFS: Makenfs_readdirrevalidate less often") might help here? My thinking is that will trigger more cache invalidations when the directory is changing underneath us, and will now trigger uncached readdir in those situations.Quoted 1 line
Quoted 2 lines
IOW, the suggestion would be to apply something like the following on top of the existing readdir patchset:
I'm all for this approach, but - I'm rarely seeing the
mapping->nrpages== 0 since the cache is dropped by a process innfs_readdir()that immediately starts filling the cache again.
Quoted 6 lines
It would make a lot more sense to me if we could do something like stash
desc->page_index<<PAGE_SHIFTinf_posafter eachnfs_readdir, then the hueristic could checkf_pos>>PAGE_SHIFT> nrpages.Yes,
f_posis growing many different meanings in NFS directories, maybe we can stash it on the directory context.
Ignore the f_pos suggestion -- iterate_dir sets it for us.. would have to use a private value.
Ben
Quoted 21 lines · Benjamin Coddington
Quoted 13 lines · Trond Myklebust
Quoted 11 lines · Trond Myklebust
I was going to ask you if perhaps reverting Scott's commit
07b5ce8ef2d8
("NFS: Makenfs_readdirrevalidate less often") might help here? My thinking is that will trigger more cache invalidations when the
directory is changing underneath us, and will now trigger uncached
readdir in those situations.Quoted 1 line
Quoted 3 lines
IOW, the suggestion would be to apply something like the following on
top of the existing readdir patchset:
Quoted 5 lines
I'm all for this approach, but - I'm rarely seeing the mapping-
Quoted 1 line
nrpages == 0
since the cache is dropped by a process in
nfs_readdir()that immediately
starts filling the cache again.
That's why I moved the check in readdir_search_pagecache. Unless that process has set desc->dir_cookie == 0, then that should prevent the refilling.
Quoted 61 lines
It would make a lot more sense to me if we could do something like stash
desc->page_index<<PAGE_SHIFTinf_posafter eachnfs_readdir, then the
hueristic could checkf_pos>>PAGE_SHIFT> nrpages.Yes,
f_posis growing many different meanings in NFS directories, maybe we
can stash it on the directory context.Ben
Quoted 48 lines
---
fs/nfs/dir.c | 14 ++++++--------1 file changed, 6 insertions(+), 8 deletions(-)fs/nfs/dir.c +3
-3@@ -956,10 +956,10 @@ static int readdir_search_pagecache(struct nfs_readdir_descriptor *desc) { int res; - if (nfs_readdir_dont_search_cache(desc)) - return -EBADCOOKIE; - do { + if (nfs_readdir_dont_search_cache(desc)) + return -EBADCOOKIE; + if (desc->page_index == 0) { desc->current_index = 0;desc->prev_index = 0;
fs/nfs/dir.c +0
-0@@ -1082,11 +1082,9 @@ static int nfs_readdir(struct file *file,struct
dir_context*ctx)
* to either find the entry with the appropriate number or
* revalidate the cookie.
*/
- if (ctx->pos== 0 ||nfs_attribute_cache_expired(inode)) { - res = nfs_revalidate_mapping(inode, file-Quoted 1 line
f_mapping);- if (res < 0)
- goto out;
- }
+ res =nfs_revalidate_mapping(inode, file->f_mapping); + if (res < 0)
+ goto out;res =
-ENOMEM;
desc =kzalloc(sizeof(*desc), GFP_KERNEL);Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 23 lines · Trond Myklebust
Quoted 22 lines · Benjamin Coddington
Quoted 13 lines · Trond Myklebust
Quoted 11 lines · Trond Myklebust
I was going to ask you if perhaps reverting Scott's commit
07b5ce8ef2d8
("NFS: Makenfs_readdirrevalidate less often") might help here? My thinking is that will trigger more cache invalidations when the
directory is changing underneath us, and will now trigger uncached
readdir in those situations.Quoted 1 line
Quoted 3 lines
IOW, the suggestion would be to apply something like the following on
top of the existing readdir patchset:I'm all for this approach, but - I'm rarely seeing the mapping-
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
nrpages == 0
since the cache is dropped by a process in
nfs_readdir()that immediately
starts filling the cache again.That's why I moved the check in
readdir_search_pagecache. Unless that process has setdesc->dir_cookie== 0, then that should prevent the refilling.
My pathological benchmarking does send another process in with desc->dir_cookie == 0.
I'm doing fork(), while(getdents) every second with acdirmin=1 and a listing that takes longer than 1 second to complete uncached, while touching the directory every second.
Ben
Quoted 34 lines · Benjamin Coddington
Quoted 23 lines · Trond Myklebust
Quoted 22 lines · Benjamin Coddington
Quoted 13 lines · Trond Myklebust
Quoted 11 lines · Trond Myklebust
I was going to ask you if perhaps reverting Scott's commit
07b5ce8ef2d8
("NFS: Makenfs_readdirrevalidate less often") might help here? My thinking is that will trigger more cache invalidations when the
directory is changing underneath us, and will now trigger uncached
readdir in those situations.Quoted 1 line
Quoted 3 lines
IOW, the suggestion would be to apply something like the following on
top of the existing readdir patchset:I'm all for this approach, but - I'm rarely seeing the mapping-
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
nrpages == 0
since the cache is dropped by a process in
nfs_readdir()that immediately
starts filling the cache again.That's why I moved the check in
readdir_search_pagecache. Unless that process has setdesc->dir_cookie== 0, then that should prevent the refilling.
Quoted 6 lines
My pathological benchmarking does send another process in with
desc->dir_cookie== 0.I'm doing
fork(), while(getdents) every second with acdirmin=1 and a listing that takes longer than 1 second to complete uncached, while touching the directory every second.
As long as I use a buffer > 16k for getdents(), this hack can keep up with the testcase:
fs/nfs/dir.c +17 -9
@@ -78,6 +78,7 @@ static struct nfs_open_dir_context *alloc_nfs_open_dir_context(struct inode *dir ctx->attr_gencount = nfsi->attr_gencount; ctx->dir_cookie =0 ; ctx->dup_cookie =0 ; + ctx->page_index =0 ;spin_lock (&dir->i_lock);if (list_empty (&nfsi->open_files) && (nfsi->cache_validity &NFS_INO_DATA_INVAL_DEFER ))
@@ -150,6 +151,7 @@ struct nfs_readdir_descriptor {struct page *page;struct dir_context *ctx;pgoff_t page_index; +pgoff_t last_page_index;u64 dir_cookie;u64 last_cookie;u64 dup_cookie;
@@ -928,7 +930,12 @@ static bool nfs_readdir_dont_search_cache(struct nfs_readdir_descriptor *desc)* Default to uncached readdir if the page cache is empty, and * we're looking for a non-zero cookie in a large directory. */ -return desc->dir_cookie !=0 && mapping->nrpages ==0 && size > dtsize; +if (desc->dir_cookie ==0 ) +return false; +else if (mapping->nrpages < desc->last_page_index && size > dtsize) +return true; + +return false; }/* Search for desc->dir_cookie from the beginning of the page cache */
@@ -936,10 +943,10 @@ static int readdir_search_pagecache(struct nfs_readdir_descriptor *desc) {int res; -if (nfs_readdir_dont_search_cache (desc)) -return -EBADCOOKIE ; - do { + if (nfs_readdir_dont_search_cache(desc)) + return -EBADCOOKIE; + if (desc->page_index == 0) { desc->current_index = 0; desc->prev_index = 0;
@@ -1062,11 +1069,9 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)* to either find the entry with the appropriate number or * revalidate the cookie. */ -if (ctx->pos ==0 ||nfs_attribute_cache_expired (inode)) { - res =nfs_revalidate_mapping (inode, file->f_mapping); -if (res <0 ) - goto out; - } + res =nfs_revalidate_mapping (inode, file->f_mapping); +if (res <0 ) + goto out; res = -ENOMEM ; desc =kzalloc (sizeof (*desc),GFP_KERNEL );
@@ -1082,6 +1087,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) desc->duped = dir_ctx->duped; desc->attr_gencount = dir_ctx->attr_gencount;memcpy (desc->verf, dir_ctx->verf,sizeof (desc->verf)); + desc->last_page_index = dir_ctx->page_index;spin_unlock (&file->f_lock); do {
@@ -1121,6 +1127,8 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) dir_ctx->duped = desc->duped; dir_ctx->attr_gencount = desc->attr_gencount;memcpy (dir_ctx->verf, desc->verf,sizeof (dir_ctx->verf)); +if (desc->page_index > dir_ctx->page_index) + dir_ctx->page_index = desc->page_index;spin_unlock (&file->f_lock);kfree (desc);
include/linux/nfs_fs.h +1 -0
@@ -98,6 +98,7 @@ struct nfs_open_dir_context {__u64 dir_cookie;__u64 dup_cookie;signed char duped; +pgoff_t page_index; };/*
Here's the testcase I'm using:
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/syscall.h>
#include <sys/time.h>
/*
* mkdir /exports/10k_dentries
* for i in {1..10000}; do printf "/exports/10k_dentries/file%.12d\n"
$i; done | xargs touch
*/
#define DIR "/mnt/fedora/10k_dentries"
#define BUF_SIZE 1024 * 16
void evict_pagecache() {
int dir_fd = open(DIR, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC);
posix_fadvise(dir_fd, 0, 0, POSIX_FADV_DONTNEED);
close(dir_fd);
}/* call getdents64() until no more, return time taken */
int listdir() {
struct timeval tvs, tve;
char buf[BUF_SIZE];
int dir_fd = open(DIR, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC);
gettimeofday(&tvs, NULL);
while (syscall(SYS_getdents, dir_fd, buf, BUF_SIZE)) { }
gettimeofday(&tve, NULL);
return tve.tv_sec - tvs.tv_sec;
}int main(int argc, char **argv)
{
evict_pagecache();
while (1) {
if (!fork()) {
int secs;
printf("pid %d starts..\n", getpid());
fflush(stdout);
secs = listdir();
printf("pid %d done %d seconds\n", getpid(), secs);
fflush(stdout);
return 0;
}
sleep (1);
}
}Ben
Hi Trond,
I can confirm that with latest patchset we see 3x improvement on directories listings with 100K and 500K files.
The downside is that our users will start to create directories with even more files :). This weeks record 2.5M files at 200 files-per-second rate....
Thanks again,
Tigran.
----- Original Message -----
Quoted 212 lines
From: "Benjamin Coddington" <[email protected]>
To: "trondmy" <[email protected]>
Cc: "linux-nfs" <[email protected]>, "David Wysochanski" <[email protected]>
Sent: Thursday, 12 November, 2020 21:23:32
Subject: Re: [PATCH v5 00/22] Readdir enhancementsQuoted 34 lines · Benjamin Coddington
Quoted 23 lines · Trond Myklebust
Quoted 22 lines · Benjamin Coddington
Quoted 13 lines · Trond Myklebust
Quoted 11 lines · Trond Myklebust
I was going to ask you if perhaps reverting Scott's commit
07b5ce8ef2d8
("NFS: Makenfs_readdirrevalidate less often") might help here? My thinking is that will trigger more cache invalidations when the
directory is changing underneath us, and will now trigger uncached
readdir in those situations.Quoted 1 line
Quoted 3 lines
IOW, the suggestion would be to apply something like the following on
top of the existing readdir patchset:I'm all for this approach, but - I'm rarely seeing the mapping-
Quoted 8 lines
Quoted 4 lines
Quoted 1 line
nrpages == 0
since the cache is dropped by a process in
nfs_readdir()that immediately
starts filling the cache again.That's why I moved the check in
readdir_search_pagecache. Unless that process has setdesc->dir_cookie== 0, then that should prevent the refilling.Quoted 8 lines
My pathological benchmarking does send another process in with
desc->dir_cookie== 0.I'm doing
fork(), while(getdents) every second with acdirmin=1 and a listing
that takes longer than 1 second to complete uncached, while touching the
directory every second.As long as I use a buffer > 16k for
getdents(), this hack can keep up with the testcase:fs/nfs/dir.c +1
-0@@ -78,6 +78,7 @@ static struct nfs_open_dir_context *alloc_nfs_open_dir_context(struct inode *dir ctx->attr_gencount = nfsi->attr_gencount; ctx->dir_cookie =0 ; ctx->dup_cookie =0 ; + ctx->page_index =0 ;spin_lock (&dir->i_lock); if (list_empty(&nfsi->open_files) &&(nfsi->cache_validity & NFS_INO_DATA_INVAL_DEFER))fs/nfs/dir.c +7
-1@@ -150,6 +151,7 @@ struct nfs_readdir_descriptor {struct page *page;struct dir_context *ctx;pgoff_t page_index; +pgoff_t last_page_index;u64 dir_cookie;u64 last_cookie;u64 dup_cookie;@@ -928,7 +930,12 @@ static bool nfs_readdir_dont_search_cache(structnfs_readdir_descriptor *desc) * Default to uncached readdir if the page cache is empty, and * we're looking for a non-zero cookie in a large directory. */ -return desc->dir_cookie !=0 && mapping->nrpages ==0 && size > dtsize; +if (desc->dir_cookie ==0 ) +return false; +else if (mapping->nrpages < desc->last_page_index && size > dtsize) +return true; + +return false;}
/* Search for
desc->dir_cookiefrom the beginning of the page cache */fs/nfs/dir.c +3
-3@@ -936,10 +943,10 @@ static int readdir_search_pagecache(struct nfs_readdir_descriptor *desc) {int res; -if (nfs_readdir_dont_search_cache (desc)) -return -EBADCOOKIE ; - do { + if (nfs_readdir_dont_search_cache(desc)) + return -EBADCOOKIE; + if (desc->page_index == 0) { desc->current_index = 0;desc->prev_index = 0;fs/nfs/dir.c +3
-5@@ -1062,11 +1069,9 @@ static int nfs_readdir(struct file *file, structdir_context *ctx) * to either find the entry with the appropriate number or * revalidate the cookie. */ -if (ctx->pos ==0 ||nfs_attribute_cache_expired (inode)) { - res =nfs_revalidate_mapping (inode, file->f_mapping); -if (res <0 ) - goto out; - } + res =nfs_revalidate_mapping (inode, file->f_mapping); +if (res <0 ) + goto out; res = -ENOMEM ;desc = kzalloc(sizeof(*desc), GFP_KERNEL);fs/nfs/dir.c +1
-0@@ -1082,6 +1087,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) desc->duped = dir_ctx->duped; desc->attr_gencount = dir_ctx->attr_gencount;memcpy (desc->verf, dir_ctx->verf,sizeof (desc->verf)); + desc->last_page_index = dir_ctx->page_index;spin_unlock (&file->f_lock);do {fs/nfs/dir.c +2
-0@@ -1121,6 +1127,8 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) dir_ctx->duped = desc->duped; dir_ctx->attr_gencount = desc->attr_gencount;memcpy (dir_ctx->verf, desc->verf,sizeof (dir_ctx->verf)); +if (desc->page_index > dir_ctx->page_index) + dir_ctx->page_index = desc->page_index;spin_unlock (&file->f_lock);kfree(desc);include/linux/nfs_fs.h +1
-0@@ -98,6 +98,7 @@ struct nfs_open_dir_context {__u64 dir_cookie;__u64 dup_cookie;signed char duped; +pgoff_t page_index; };/* Here's the testcase I'm using:
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
#include <sys/syscall.h>
#include <sys/time.h>/*
* mkdir /exports/10k_dentries
* for i in {1..10000}; do printf "/exports/10k_dentries/file%.12d\n"
$i; done | xargs touch
*/
#define DIR "/mnt/fedora/10k_dentries"
#defineBUF_SIZE1024 * 16void evict_pagecache() { int dir_fd = open(DIR, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC); posix_fadvise(dir_fd, 0, 0, POSIX_FADV_DONTNEED); close(dir_fd); }/* call
getdents64()until no more, return time taken */int listdir() { struct timeval tvs, tve; char buf[BUF_SIZE]; int dir_fd = open(DIR, O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC); gettimeofday(&tvs, NULL); while (syscall(SYS_getdents, dir_fd, buf, BUF_SIZE)) { } gettimeofday(&tve, NULL); return tve.tv_sec - tvs.tv_sec; }int
main(int argc, char **argv){ evict_pagecache(); while (1) { if (!fork()) { int secs; printf("pid %d starts..\n", getpid()); fflush(stdout); secs = listdir(); printf("pid %d done %d seconds\n", getpid(), secs); fflush(stdout); return 0; } sleep (1); } }Ben
Quoted 53 lines · Trond Myklebust
Quoted 9 lines · Trond Myklebust
I was going to ask you if perhaps reverting Scott's commit
07b5ce8ef2d8
("NFS: Makenfs_readdirrevalidate less often") might help here? My thinking is that will trigger more cache invalidations when the directory is changing underneath us, and will now trigger uncached readdir in those situations.Quoted 1 line
IOW, the suggestion would be to apply something like the following on top of the existing readdir patchset:
--- fs/nfs/dir.c | 14 ++++++-------- 1 file changed, 6 insertions(+), 8 deletions(-)fs/nfs/dir.c +3
-3@@ -956,10 +956,10 @@ static int readdir_search_pagecache(struct nfs_readdir_descriptor *desc) {int res; -if (nfs_readdir_dont_search_cache (desc)) -return -EBADCOOKIE ; - do { + if (nfs_readdir_dont_search_cache(desc)) + return -EBADCOOKIE; + if (desc->page_index == 0) { desc->current_index = 0;desc->prev_index = 0;fs/nfs/dir.c +3
-5@@ -1082,11 +1082,9 @@ static int nfs_readdir(struct file *file, structdir_context *ctx) * to either find the entry with the appropriate number or * revalidate the cookie. */ -if (ctx->pos ==0 ||nfs_attribute_cache_expired (inode)) { - res =nfs_revalidate_mapping (inode, file->f_mapping); -if (res <0 ) - goto out; - } + res =nfs_revalidate_mapping (inode, file->f_mapping); +if (res <0 ) + goto out; res = -ENOMEM ;
Quoted 3 lines
desc = kzalloc(sizeof(*desc), GFP_KERNEL);
If you want to pursue this let me know and I will run through some tests with this patch (or some other patch) and all NFS versions. I've mostly been testing with NFSv4 because that has been the problematic case, but this would affect NFSv3 too.
I too had been looking at 07b5ce8ef2d8 because it causes NFSv3 to behave differently than NFSv4 (I saw with NFSv3, nfs_attribute_cache_expired() was always false while a directory is being listed). I pondered whether this was the right direction for NFSv4 as well since that causes us to avoid dropping the pagecache and kinda "neuters" it for a period of time while a large directory listing was in progress. However I didn't see a way to achieve the NFSv3 behavior for NFSv4 and I wasn't convinced NFSv3 behavior was necessarily the best approach.