Readdir improvements
Subsystems (2)
[email protected]
maintainerv10 cover Readdir improvements
From: Trond Myklebust <[email protected]>
The current NFS readdir code will always try to maximise the amount of readahead it performs on the assumption that we can cache anything that isn't immediately read by the process.
There are several cases where this assumption breaks down, including when the 'ls -l' heuristic kicks in to try to force use of readdirplus as a batch replacement for lookup/getattr.
This series also implement Ben's page cache filter to ensure that we can improve the ability to share cached data between processes that are reading the same directory at the same time, and to avoid live-locks when the directory is simultaneously changing.
--
v2: Remove reset of dtsize when NFS_INO_FORCE_READDIR is set
v3: Avoid excessive window shrinking in uncached_readdir case
v4: Track 'ls -l' cache hit/miss statistics
Improved algorithm for falling back to uncached readdir Skip readdirplus when files are being written to
v5: bugfixes
Skip readdirplus when the acdirmax/acregmax values are low Request a full XDR buffer when doing READDIRPLUS
v6: Add tracing
Don't have lookup request readdirplus when it won't help
v7: Implement Ben's page cache filter
Reduce the use of uncached readdir
Change indexing of the page cache to improve seekdir() performance.
v8: Reduce the page cache overhead by shrinking the cookie hashvalue size
Incorporate other feedback from Anna, Ben and Neil
Fix nfs2/3_decode_dirent return values
Fix the change attribute value set in nfs_readdir_page_get_next()
v9: Address bugs that were hit when testing with large directories
Misc cleanups
v10: Reinstate cache invalidations for directories, since Olga showed
that the NFSv3 ctime resolution can be insufficient to record rapid changes to a directory.
Trond Myklebust (26):
NFS: Return valid errors from nfs2/3_decode_dirent()
NFS: constify nfs_server_capable() and nfs_have_writebacks()
NFS: Trace lookup revalidation failure
NFS: Initialise the readdir verifier as best we can in nfs_opendir()
NFS: Use kzalloc() to avoid initialising the nfs_open_dir_context
NFS: Calculate page offsets algorithmically
NFS: Store the change attribute in the directory page cache
NFS: Don't re-read the entire page cache to find the next cookie
NFS: Don't advance the page pointer unless the page is full
NFS: Adjust the amount of readahead performed by NFS readdir
NFS: If the cookie verifier changes, we must invalidate the page cache
NFS: Simplify nfs_readdir_xdr_to_array()
NFS: Reduce use of uncached readdir
NFS: Improve heuristic for readdirplus
NFS: Don't ask for readdirplus unless it can help nfs_getattr()
NFSv4: Ask for a full XDR buffer of readdir goodness
NFS: Readdirplus can't help lookup for case insensitive filesystems
NFS: Don't request readdirplus when revalidation was forced
NFS: Add basic readdir tracing
NFS: Trace effects of readdirplus on the dcache
NFS: Trace effects of the readdirplus heuristic
NFS: Clean up page array initialisation/free
NFS: Convert readdir page cache to use a cookie based index
NFS: Fix up forced readdirplus
NFS: Optimise away the previous cookie field
NFS: Cache all entries in the readdirplus reply
fs/nfs/Kconfig | 4 + fs/nfs/dir.c | 595 ++++++++++++++++++++++++---------------- fs/nfs/inode.c | 45 +-- fs/nfs/internal.h | 4 +- fs/nfs/nfs2xdr.c | 3 +- fs/nfs/nfs3xdr.c | 29 +- fs/nfs/nfs4xdr.c | 7 +- fs/nfs/nfstrace.h | 123 ++++++++- include/linux/nfs_fs.h | 17 +- include/linux/nfs_xdr.h | 3 +- 10 files changed, 535 insertions(+), 295 deletions(-)
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 1/26 NFS: Return valid errors from nfs2/3_decode_dirent()
From: Trond Myklebust <[email protected]>
Valid return values for decode_dirent() callback functions are:
0: Success
-EBADCOOKIE: End of directory
-EAGAIN: End of xdr_stream
All errors need to map into one of those three values.
--- fs/nfs/nfs2xdr.c | 2 +- fs/nfs/nfs3xdr.c | 21 ++++++--------------- 2 files changed, 7 insertions(+), 16 deletions(-)
fs/nfs/nfs2xdr.c +1 -1
@@ -949,7 +949,7 @@ int nfs2_decode_dirent(struct xdr_stream *xdr, struct nfs_entry *entry, error =decode_filename_inline (xdr, &entry->name, &entry->len);if (unlikely (error)) -return error; +return -EAGAIN ;/* * The type (size and byte order) of nfscookie isn't defined in
fs/nfs/nfs3xdr.c +6 -15
@@ -1967,7 +1967,6 @@ int nfs3_decode_dirent(struct xdr_stream *xdr, struct nfs_entry *entry, bool plus) {struct user_namespace *userns =rpc_userns (entry->server->client); -struct nfs_entry old = *entry;__be32 *p;int error;u64 new_cookie;
@@ -1987,15 +1986,15 @@ int nfs3_decode_dirent(struct xdr_stream *xdr, struct nfs_entry *entry, error =decode_fileid3 (xdr, &entry->ino);if (unlikely (error)) -return error; +return -EAGAIN ; error =decode_inline_filename3 (xdr, &entry->name, &entry->len);if (unlikely (error)) -return error; +return -EAGAIN ; error =decode_cookie3 (xdr, &new_cookie);if (unlikely (error)) -return error; +return -EAGAIN ; entry->d_type =DT_UNKNOWN ;
@@ -2003,7 +2002,7 @@ int nfs3_decode_dirent(struct xdr_stream *xdr, struct nfs_entry *entry, entry->fattr->valid =0 ; error =decode_post_op_attr (xdr, entry->fattr, userns);if (unlikely (error)) -return error; +return -EAGAIN ;if (entry->fattr->valid &NFS_ATTR_FATTR_V3 ) entry->d_type =nfs_umode_to_dtype (entry->fattr->mode);
@@ -2018,11 +2017,8 @@ int nfs3_decode_dirent(struct xdr_stream *xdr, struct nfs_entry *entry,return -EAGAIN ;if (*p != xdr_zero) { error =decode_nfs_fh3 (xdr, entry->fh); -if (unlikely (error)) { -if (error == -E2BIG ) - goto out_truncated; -return error; - } +if (unlikely (error)) +return -EAGAIN ; }else zero_nfs_fh3 (entry->fh); }
@@ -2031,11 +2027,6 @@ int nfs3_decode_dirent(struct xdr_stream *xdr, struct nfs_entry *entry, entry->cookie = new_cookie;return 0 ; - -out_truncated: -dprintk ("NFS: directory entry contains invalid file handle\n" ); - *entry = old; -return -EAGAIN ; }/*
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 2/26 NFS: constify nfs_server_capable() and nfs_have_writebacks()
From: Trond Myklebust <[email protected]>
--- include/linux/nfs_fs.h | 7 +++---- 1 file changed, 3 insertions(+), 4 deletions(-)
include/linux/nfs_fs.h +3 -4
@@ -363,7 +363,7 @@ static inline void nfs_mark_for_revalidate(struct inode *inode)spin_unlock (&inode->i_lock); } -static inline int nfs_server_capable (struct inode *inode,int cap) +static inline int nfs_server_capable (const struct inode *inode,int cap) {return NFS_SERVER (inode)->caps & cap; }
@@ -587,12 +587,11 @@ extern struct nfs_commit_data *nfs_commitdata_alloc(bool never_fail);extern void nfs_commit_free (struct nfs_commit_data *data);bool nfs_commit_end (struct nfs_mds_commit_info *cinfo); -static inline int -nfs_have_writebacks (struct inode *inode) +static inline bool nfs_have_writebacks (const struct inode *inode) {if (S_ISREG (inode->i_mode))return atomic_long_read (&NFS_I (inode)->nrequests) !=0 ; -return 0 ; +return false; }/*
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 4/26 NFS: Initialise the readdir verifier as best we can in nfs_opendir()
In reply to [email protected]From: Trond Myklebust <[email protected]>
For the purpose of ensuring that opendir() followed by seekdir() work as correctly as possible, try to initialise the readdir verifier in nfs_opendir().
--- fs/nfs/dir.c | 1 + 1 file changed, 1 insertion(+)
fs/nfs/dir.c +1 -0
@@ -89,6 +89,7 @@ static struct nfs_open_dir_context *alloc_nfs_open_dir_context(struct inode *dir NFS_INO_REVAL_FORCED);list_add (&ctx->list, &nfsi->open_files);clear_bit (NFS_INO_FORCE_READDIR , &nfsi->flags); +memcpy (ctx->verf, nfsi->cookieverf,sizeof (ctx->verf));spin_unlock (&dir->i_lock);return ctx; }
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 5/26 NFS: Use kzalloc() to avoid initialising the nfs_open_dir_context
From: Trond Myklebust <[email protected]>
--- fs/nfs/dir.c | 11 ++++------- 1 file changed, 4 insertions(+), 7 deletions(-)
fs/nfs/dir.c +4 -7
@@ -69,18 +69,15 @@ const struct address_space_operations nfs_dir_aops = { .freepage = nfs_readdir_clear_array, }; -static struct nfs_open_dir_context *alloc_nfs_open_dir_context (struct inode *dir) +static struct nfs_open_dir_context * +alloc_nfs_open_dir_context (struct inode *dir) {struct nfs_inode *nfsi =NFS_I (dir);struct nfs_open_dir_context *ctx; - ctx =kmalloc (sizeof (*ctx),GFP_KERNEL_ACCOUNT ); + + ctx =kzalloc (sizeof (*ctx),GFP_KERNEL_ACCOUNT );if (ctx !=NULL ) { - ctx->duped =0 ; ctx->attr_gencount = nfsi->attr_gencount; - ctx->dir_cookie =0 ; - ctx->dup_cookie =0 ; - ctx->page_index =0 ; - ctx->eof = false;spin_lock (&dir->i_lock);if (list_empty (&nfsi->open_files) && (nfsi->cache_validity &NFS_INO_DATA_INVAL_DEFER ))
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 6/26 NFS: Calculate page offsets algorithmically
From: Trond Myklebust <[email protected]>
Instead of relying on counting the page offsets as we walk through the page cache, switch to calculating them algorithmically.
--- fs/nfs/dir.c | 18 +++++++++++++----- 1 file changed, 13 insertions(+), 5 deletions(-)
fs/nfs/dir.c +13 -5
@@ -249,17 +249,20 @@ static const char *nfs_readdir_copy_name(const char *name, unsigned int len)return ret; } +static size_t nfs_readdir_array_maxentries (void ) +{ +return (PAGE_SIZE -sizeof (struct nfs_cache_array )) / +sizeof (struct nfs_cache_array_entry ); +} +/* * Check that the next array entry lies entirely within the page bounds */ static int nfs_readdir_array_can_expand (struct nfs_cache_array *array) { -struct nfs_cache_array_entry *cache_entry; -if (array->page_full)return -ENOSPC ; - cache_entry = &array->array[array->size +1 ]; -if ((char *)cache_entry - (char *)array >PAGE_SIZE ) { +if (array->size ==nfs_readdir_array_maxentries ()) { array->page_full =1 ;return -ENOSPC ; }
@@ -318,6 +321,11 @@ static struct page *nfs_readdir_page_get_locked(struct address_space *mapping,return page; } +static loff_t nfs_readdir_page_offset (struct page *page) +{ +return (loff_t )page->index * (loff_t )nfs_readdir_array_maxentries (); +} +static u64 nfs_readdir_page_last_cookie (struct page *page) {struct nfs_cache_array *array;
@@ -448,7 +456,7 @@ static int nfs_readdir_search_for_cookie(struct nfs_cache_array *array,if (array->array[i].cookie == desc->dir_cookie) {struct nfs_inode *nfsi =NFS_I (file_inode (desc->file)); - new_pos = desc->current_index + i; + new_pos =nfs_readdir_page_offset (desc->page) + i;if (desc->attr_gencount != nfsi->attr_gencount || !nfs_readdir_inode_mapping_valid (nfsi)) { desc->duped =0 ;
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 3/26 NFS: Trace lookup revalidation failure
In reply to [email protected]From: Trond Myklebust <[email protected]>
Enable tracing of lookup revalidation failures.
--- fs/nfs/dir.c | 17 +++++------------ 1 file changed, 5 insertions(+), 12 deletions(-)
fs/nfs/dir.c +5 -12
@@ -1474,9 +1474,7 @@ nfs_lookup_revalidate_done(struct inode *dir, struct dentry *dentry, {switch (error) {case 1 : -dfprintk (LOOKUPCACHE ,"NFS: %s(%pd2) is valid\n" , - __func__, dentry); -return 1 ; +break ;case 0 :/* * We can't d_drop the root of a disconnected tree:
@@ -1485,13 +1483,10 @@ nfs_lookup_revalidate_done(struct inode *dir, struct dentry *dentry,* inodes on unmount and further oopses. */ if (inode &&IS_ROOT (dentry)) -return 1 ; -dfprintk (LOOKUPCACHE ,"NFS: %s(%pd2) is invalid\n" , - __func__, dentry); -return 0 ; + error =1 ; +break ; } -dfprintk (LOOKUPCACHE ,"NFS: %s(%pd2) lookup returned error %d\n" , - __func__, dentry, error); +trace_nfs_lookup_revalidate_exit (dir, dentry,0 , error);return error; }
@@ -1623,9 +1618,7 @@ nfs_do_lookup_revalidate(struct inode *dir, struct dentry *dentry, goto out_bad;trace_nfs_lookup_revalidate_enter (dir, dentry, flags); - error =nfs_lookup_revalidate_dentry (dir, dentry, inode); -trace_nfs_lookup_revalidate_exit (dir, dentry, flags, error); -return error; +return nfs_lookup_revalidate_dentry (dir, dentry, inode); out_valid:return nfs_lookup_revalidate_done (dir, dentry, inode,1 ); out_bad:
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 8/26 NFS: Don't re-read the entire page cache to find the next cookie
In reply to [email protected]From: Trond Myklebust <[email protected]>
If the page cache entry that was last read gets invalidated for some reason, then make sure we can re-create it on the next call to readdir. This, combined with the cache page validation, allows us to reuse the cached value of page-index on successive calls to nfs_readdir.
Credit is due to Benjamin Coddington for showing that the concept works, and that it allows for improved cache sharing between processes even in the case where pages are lost due to LRU or active invalidation.
--- fs/nfs/dir.c | 10 +++++++--- include/linux/nfs_fs.h | 1 + 2 files changed, 8 insertions(+), 3 deletions(-)
fs/nfs/dir.c +7 -3
@@ -1120,6 +1120,8 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) desc->dup_cookie = dir_ctx->dup_cookie; desc->duped = dir_ctx->duped; page_index = dir_ctx->page_index; + desc->page_index = page_index; + desc->last_cookie = dir_ctx->last_cookie; desc->attr_gencount = dir_ctx->attr_gencount; desc->eof = dir_ctx->eof;memcpy (desc->verf, dir_ctx->verf,sizeof (desc->verf));
@@ -1168,6 +1170,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)spin_lock (&file->f_lock); dir_ctx->dir_cookie = desc->dir_cookie; dir_ctx->dup_cookie = desc->dup_cookie; + dir_ctx->last_cookie = desc->last_cookie; dir_ctx->duped = desc->duped; dir_ctx->attr_gencount = desc->attr_gencount; dir_ctx->page_index = desc->page_index;
@@ -1209,10 +1212,11 @@ static loff_t nfs_llseek_dir(struct file *filp, loff_t offset, int whence) }if (offset != filp->f_pos) { filp->f_pos = offset; -if (nfs_readdir_use_cookie (filp)) - dir_ctx->dir_cookie = offset; -else +if (!nfs_readdir_use_cookie (filp)) { dir_ctx->dir_cookie =0 ; + dir_ctx->page_index =0 ; + }else + dir_ctx->dir_cookie = offset;if (offset ==0 )memset (dir_ctx->verf,0 ,sizeof (dir_ctx->verf)); dir_ctx->duped =0 ;
include/linux/nfs_fs.h +1 -0
@@ -105,6 +105,7 @@ struct nfs_open_dir_context {__be32 verf[NFS_DIR_VERIFIER_SIZE ];__u64 dir_cookie;__u64 dup_cookie; +__u64 last_cookie;pgoff_t page_index;signed char duped;bool eof;
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 10/26 NFS: Adjust the amount of readahead performed by NFS readdir
In reply to [email protected]From: Trond Myklebust <[email protected]>
The current NFS readdir code will always try to maximise the amount of readahead it performs on the assumption that we can cache anything that isn't immediately read by the process.
There are several cases where this assumption breaks down, including when the 'ls -l' heuristic kicks in to try to force use of readdirplus as a batch replacement for lookup/getattr.
This patch therefore tries to tone down the amount of readahead we perform, and adjust it to try to match the amount of data being requested by user space.
--- fs/nfs/dir.c | 53 +++++++++++++++++++++++++++++++++++++++++- include/linux/nfs_fs.h | 1 + 2 files changed, 53 insertions(+), 1 deletion(-)
fs/nfs/dir.c +52 -1
@@ -69,6 +69,8 @@ const struct address_space_operations nfs_dir_aops = { .freepage = nfs_readdir_clear_array, }; +#define NFS_INIT_DTSIZE PAGE_SIZE +static struct nfs_open_dir_context *alloc_nfs_open_dir_context (struct inode *dir) {
@@ -78,6 +80,7 @@ alloc_nfs_open_dir_context(struct inode *dir) ctx =kzalloc (sizeof (*ctx),GFP_KERNEL_ACCOUNT );if (ctx !=NULL ) { ctx->attr_gencount = nfsi->attr_gencount; + ctx->dtsize =NFS_INIT_DTSIZE ;spin_lock (&dir->i_lock);if (list_empty (&nfsi->open_files) && (nfsi->cache_validity &NFS_INO_DATA_INVAL_DEFER ))
@@ -154,6 +157,7 @@ struct nfs_readdir_descriptor {struct page *page;struct dir_context *ctx;pgoff_t page_index; +pgoff_t page_index_max;u64 dir_cookie;u64 last_cookie;u64 dup_cookie;
@@ -166,12 +170,36 @@ struct nfs_readdir_descriptor {unsigned long gencount;unsigned long attr_gencount;unsigned int cache_entry_index; +unsigned int buffer_fills; +unsigned int dtsize;signed char duped;bool plus;bool eob;bool eof; }; +static void nfs_set_dtsize (struct nfs_readdir_descriptor *desc,unsigned int sz) +{ +struct nfs_server *server =NFS_SERVER (file_inode (desc->file)); +unsigned int maxsize = server->dtsize; + +if (sz > maxsize) + sz = maxsize; +if (sz <NFS_MIN_FILE_IO_SIZE ) + sz =NFS_MIN_FILE_IO_SIZE ; + desc->dtsize = sz; +} + +static void nfs_shrink_dtsize (struct nfs_readdir_descriptor *desc) +{ +nfs_set_dtsize (desc, desc->dtsize >>1 ); +} + +static void nfs_grow_dtsize (struct nfs_readdir_descriptor *desc) +{ +nfs_set_dtsize (desc, desc->dtsize <<1 ); +} +static void nfs_readdir_array_init (struct nfs_cache_array *array) {memset (array,0 ,sizeof (struct nfs_cache_array ));
@@ -784,6 +812,7 @@ static int nfs_readdir_page_filler(struct nfs_readdir_descriptor *desc,break ; arrays++; *arrays = page = new; + desc->page_index_max++; }else { new = nfs_readdir_page_get_next(mapping, page->index + 1,
@@ -793,6 +822,7 @@ static int nfs_readdir_page_filler(struct nfs_readdir_descriptor *desc,if (page != *arrays)nfs_readdir_page_unlock_and_put (page); page = new; + desc->page_index_max = new->index; } status =nfs_readdir_add_to_array (entry, page); }while (!status && !entry->eof);
@@ -858,7 +888,7 @@ static int nfs_readdir_xdr_to_array(struct nfs_readdir_descriptor *desc,struct nfs_entry *entry;size_t array_size;struct inode *inode =file_inode (desc->file); -size_t dtsize =NFS_SERVER (inode)->dtsize; +unsigned int dtsize = desc->dtsize;int status = -ENOMEM ; entry =kzalloc (sizeof (*entry),GFP_KERNEL );
@@ -894,6 +924,7 @@ static int nfs_readdir_xdr_to_array(struct nfs_readdir_descriptor *desc, status =nfs_readdir_page_filler (desc, entry, pages, pglen, arrays, narrays); + desc->buffer_fills++; }while (!status &&nfs_readdir_page_needs_filling (page) &&page_mapping (page));
@@ -941,6 +972,10 @@ static int find_and_lock_cache_page(struct nfs_readdir_descriptor *desc)if (!desc->page)return -ENOMEM ;if (nfs_readdir_page_needs_filling (desc->page)) { +/* Grow the dtsize if we had to go back for more pages */ +if (desc->page_index == desc->page_index_max) +nfs_grow_dtsize (desc); + desc->page_index_max = desc->page_index; res =nfs_readdir_xdr_to_array (desc, nfsi->cookieverf, verf, &desc->page,1 );if (res <0 ) {
@@ -1075,6 +1110,7 @@ static int uncached_readdir(struct nfs_readdir_descriptor *desc) desc->cache_entry_index =0 ; desc->last_cookie = desc->dir_cookie; desc->duped =0 ; + desc->page_index_max =0 ; status =nfs_readdir_xdr_to_array (desc, desc->verf, verf, arrays, sz);
@@ -1084,10 +1120,22 @@ static int uncached_readdir(struct nfs_readdir_descriptor *desc) } desc->page =NULL ; +/* +* Grow the dtsize if we have to go back for more pages, +* or shrink it if we're reading too many. +*/ +if (!desc->eof) { +if (!desc->eob) +nfs_grow_dtsize (desc); +else if (desc->buffer_fills ==1 && + i < (desc->page_index_max >>1 )) +nfs_shrink_dtsize (desc); + }for (i =0 ; i < sz && arrays[i]; i++)nfs_readdir_page_array_free (arrays[i]); out: + desc->page_index_max =-1 ;kfree (arrays);dfprintk (DIRCACHE ,"NFS: %s: returns %d\n" , __func__, status);return status;
@@ -1126,6 +1174,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) desc->file = file; desc->ctx = ctx; desc->plus =nfs_use_readdirplus (inode, ctx); + desc->page_index_max =-1 ;spin_lock (&file->f_lock); desc->dir_cookie = dir_ctx->dir_cookie;
@@ -1136,6 +1185,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) desc->last_cookie = dir_ctx->last_cookie; desc->attr_gencount = dir_ctx->attr_gencount; desc->eof = dir_ctx->eof; +nfs_set_dtsize (desc, dir_ctx->dtsize);memcpy (desc->verf, dir_ctx->verf,sizeof (desc->verf));spin_unlock (&file->f_lock);
@@ -1187,6 +1237,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) dir_ctx->attr_gencount = desc->attr_gencount; dir_ctx->page_index = desc->page_index; dir_ctx->eof = desc->eof; + dir_ctx->dtsize = desc->dtsize;memcpy (dir_ctx->verf, desc->verf,sizeof (dir_ctx->verf));spin_unlock (&file->f_lock); out_free:
include/linux/nfs_fs.h +1 -0
@@ -107,6 +107,7 @@ struct nfs_open_dir_context {__u64 dup_cookie;__u64 last_cookie;pgoff_t page_index; +unsigned int dtsize;signed char duped;bool eof; };
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 7/26 NFS: Store the change attribute in the directory page cache
In reply to [email protected]From: Trond Myklebust <[email protected]>
Use the change attribute and the first cookie in a directory page cache entry to validate that the page is up to date.
--- fs/nfs/dir.c | 68 ++++++++++++++++++++++++++++------------------------ 1 file changed, 37 insertions(+), 31 deletions(-)
fs/nfs/dir.c +37 -31
@@ -140,6 +140,7 @@ struct nfs_cache_array_entry { };struct nfs_cache_array { +u64 change_attr;u64 last_cookie;unsigned int size; unsigned char page_full : 1,
@@ -176,12 +177,14 @@ static void nfs_readdir_array_init(struct nfs_cache_array *array)memset (array,0 ,sizeof (struct nfs_cache_array )); } -static void nfs_readdir_page_init_array (struct page *page,u64 last_cookie) +static void nfs_readdir_page_init_array (struct page *page,u64 last_cookie, +u64 change_attr) {struct nfs_cache_array *array; array =kmap_atomic (page);nfs_readdir_array_init (array); + array->change_attr = change_attr; array->last_cookie = last_cookie; array->cookies_are_ordered =1 ;kunmap_atomic (array);
@@ -208,7 +211,7 @@ nfs_readdir_page_array_alloc(u64 last_cookie, gfp_t gfp_flags) {struct page *page =alloc_page (gfp_flags);if (page) -nfs_readdir_page_init_array (page, last_cookie); +nfs_readdir_page_init_array (page, last_cookie,0 );return page; }
@@ -305,19 +308,43 @@ int nfs_readdir_add_to_array(struct nfs_entry *entry, struct page *page)return ret; } +static bool nfs_readdir_page_validate (struct page *page,u64 last_cookie, +u64 change_attr) +{ +struct nfs_cache_array *array =kmap_atomic (page); +int ret = true; + +if (array->change_attr != change_attr) + ret = false; +if (array->size >0 && array->array[0 ].cookie != last_cookie) + ret = false; +kunmap_atomic (array); +return ret; +} + +static void nfs_readdir_page_unlock_and_put (struct page *page) +{ +unlock_page (page); +put_page (page); +} +static struct page *nfs_readdir_page_get_locked (struct address_space *mapping,pgoff_t index,u64 last_cookie) {struct page *page; +u64 change_attr; page =grab_cache_page (mapping, index); -if (page && !PageUptodate (page)) { -nfs_readdir_page_init_array (page, last_cookie); -if (invalidate_inode_pages2_range (mapping, index +1 ,-1 ) <0 ) -nfs_zap_mapping (mapping->host, mapping); -SetPageUptodate (page); +if (!page) +return NULL ; + change_attr =inode_peek_iversion_raw (mapping->host); +if (PageUptodate (page)) { +if (nfs_readdir_page_validate (page, last_cookie, change_attr)) +return page; +nfs_readdir_clear_array (page); } - +nfs_readdir_page_init_array (page, last_cookie, change_attr); +SetPageUptodate (page);return page; }
@@ -357,12 +384,6 @@ static void nfs_readdir_page_set_eof(struct page *page)kunmap_atomic (array); } -static void nfs_readdir_page_unlock_and_put (struct page *page) -{ -unlock_page (page); -put_page (page); -} -static struct page *nfs_readdir_page_get_next (struct address_space *mapping,pgoff_t index,u64 cookie) {
@@ -419,16 +440,6 @@ static int nfs_readdir_search_for_pos(struct nfs_cache_array *array,return -EBADCOOKIE ; } -static bool -nfs_readdir_inode_mapping_valid (struct nfs_inode *nfsi) -{ -if (nfsi->cache_validity & (NFS_INO_INVALID_CHANGE | -NFS_INO_INVALID_DATA )) -return false; -smp_rmb (); -return !test_bit (NFS_INO_INVALIDATING , &nfsi->flags); -} -static bool nfs_readdir_array_cookie_in_range (struct nfs_cache_array *array,u64 cookie) {
@@ -457,8 +468,7 @@ static int nfs_readdir_search_for_cookie(struct nfs_cache_array *array,struct nfs_inode *nfsi =NFS_I (file_inode (desc->file)); new_pos =nfs_readdir_page_offset (desc->page) + i; -if (desc->attr_gencount != nfsi->attr_gencount || - !nfs_readdir_inode_mapping_valid (nfsi)) { +if (desc->attr_gencount != nfsi->attr_gencount) { desc->duped =0 ; desc->attr_gencount = nfsi->attr_gencount; }else if (new_pos < desc->prev_index) {
@@ -1095,11 +1105,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)* to either find the entry with the appropriate number or * revalidate the cookie. */ -if (ctx->pos ==0 ||nfs_attribute_cache_expired (inode)) { - res =nfs_revalidate_mapping (inode, file->f_mapping); -if (res <0 ) - goto out; - } +nfs_revalidate_mapping (inode, file->f_mapping); res = -ENOMEM ; desc =kzalloc (sizeof (*desc),GFP_KERNEL );
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 9/26 NFS: Don't advance the page pointer unless the page is full
In reply to [email protected]From: Trond Myklebust <[email protected]>
When we hit the end of the data in the readdir page, we don't want to start filling a new page, unless this one is full.
--- fs/nfs/dir.c | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-)
fs/nfs/dir.c +22 -10
@@ -417,6 +417,18 @@ bool nfs_readdir_use_cookie(const struct file *filp)return true; } +static void nfs_readdir_seek_next_array (struct nfs_cache_array *array, +struct nfs_readdir_descriptor *desc) +{ +if (array->page_full) { + desc->last_cookie = array->last_cookie; + desc->current_index += array->size; + desc->cache_entry_index =0 ; + desc->page_index++; + }else + desc->last_cookie = array->array[0 ].cookie; +} +static int nfs_readdir_search_for_pos (struct nfs_cache_array *array,struct nfs_readdir_descriptor *desc) {
@@ -428,6 +440,7 @@ static int nfs_readdir_search_for_pos(struct nfs_cache_array *array,if (diff >= array->size) {if (array->page_is_eof) goto out_eof; +nfs_readdir_seek_next_array (array, desc);return -EAGAIN ; }
@@ -500,7 +513,8 @@ static int nfs_readdir_search_for_cookie(struct nfs_cache_array *array, status = -EBADCOOKIE ;if (desc->dir_cookie == array->last_cookie) desc->eof = true; - } + }else +nfs_readdir_seek_next_array (array, desc); out:return status; }
@@ -517,11 +531,6 @@ static int nfs_readdir_search_array(struct nfs_readdir_descriptor *desc)else status =nfs_readdir_search_for_cookie (array, desc); -if (status == -EAGAIN ) { - desc->last_cookie = array->last_cookie; - desc->current_index += array->size; - desc->page_index++; - }kunmap_atomic (array);return status; }
@@ -998,7 +1007,7 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc, {struct file *file = desc->file;struct nfs_cache_array *array; -unsigned int i =0 ; +unsigned int i; array =kmap (desc->page);for (i = desc->cache_entry_index; i < array->size; i++) {
@@ -1011,10 +1020,13 @@ static void nfs_do_filldir(struct nfs_readdir_descriptor *desc,break ; }memcpy (desc->verf, verf,sizeof (desc->verf)); -if (i < (array->size-1 )) - desc->dir_cookie = array->array[i+1 ].cookie; -else +if (i == array->size -1 ) { desc->dir_cookie = array->last_cookie; +nfs_readdir_seek_next_array (array, desc); + }else { + desc->dir_cookie = array->array[i +1 ].cookie; + desc->last_cookie = array->array[0 ].cookie; + }if (nfs_readdir_use_cookie (file)) desc->ctx->pos = desc->dir_cookie;else
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 11/26 NFS: If the cookie verifier changes, we must invalidate the page cache
In reply to [email protected]From: Trond Myklebust <[email protected]>
Ensure that if the cookie verifier changes when we use the zero-valued cookie, then we invalidate any cached pages.
--- fs/nfs/dir.c | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-)
fs/nfs/dir.c +6 -1
@@ -990,9 +990,14 @@ static int find_and_lock_cache_page(struct nfs_readdir_descriptor *desc)/* * Set the cookie verifier if the page cache was empty */ -if (desc->page_index ==0 ) +if (desc->last_cookie ==0 && +memcmp (nfsi->cookieverf, verf,sizeof (nfsi->cookieverf))) {memcpy (nfsi->cookieverf, verf,sizeof (nfsi->cookieverf)); +invalidate_inode_pages2_range (desc->file->f_mapping, + desc->page_index_max +1 , +-1 ); + } } res =nfs_readdir_search_array (desc);if (res ==0 )
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 13/26 NFS: Reduce use of uncached readdir
In reply to [email protected]From: Trond Myklebust <[email protected]>
When reading a very large directory, we want to try to keep the page cache up to date if doing so is inexpensive. With the change to allow readdir to continue reading even when the cache is incomplete, we no longer need to fall back to uncached readdir in order to scale to large directories.
--- fs/nfs/dir.c | 23 +++-------------------- 1 file changed, 3 insertions(+), 20 deletions(-)
fs/nfs/dir.c +3 -20
@@ -999,28 +999,11 @@ static int find_and_lock_cache_page(struct nfs_readdir_descriptor *desc)return res; } -static bool nfs_readdir_dont_search_cache (struct nfs_readdir_descriptor *desc) -{ -struct address_space *mapping = desc->file->f_mapping; -struct inode *dir =file_inode (desc->file); -unsigned int dtsize =NFS_SERVER (dir)->dtsize; -loff_t size =i_size_read (dir); - -/* -* Default to uncached readdir if the page cache is empty, and -* we're looking for a non-zero cookie in a large directory. -*/ -return desc->dir_cookie !=0 && mapping->nrpages ==0 && size > dtsize; -} -/* Search for desc->dir_cookie from the beginning of the page cache */ static int readdir_search_pagecache (struct nfs_readdir_descriptor *desc) {int res; -if (nfs_readdir_dont_search_cache (desc)) -return -EBADCOOKIE ; - do { if (desc->page_index == 0) { desc->current_index = 0;
@@ -1273,10 +1256,10 @@ static loff_t nfs_llseek_dir(struct file *filp, loff_t offset, int whence) }if (offset != filp->f_pos) { filp->f_pos = offset; -if (!nfs_readdir_use_cookie (filp)) { + dir_ctx->page_index =0 ; +if (!nfs_readdir_use_cookie (filp)) dir_ctx->dir_cookie =0 ; - dir_ctx->page_index =0 ; - }else +else dir_ctx->dir_cookie = offset;if (offset ==0 )memset (dir_ctx->verf,0 ,sizeof (dir_ctx->verf));
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 12/26 NFS: Simplify nfs_readdir_xdr_to_array()
In reply to [email protected]From: Trond Myklebust <[email protected]>
Recent changes to readdir mean that we can cope with partially filled page cache entries, so we no longer need to rely on looping in nfs_readdir_xdr_to_array().
--- fs/nfs/dir.c | 29 +++++++++++------------------ 1 file changed, 11 insertions(+), 18 deletions(-)
fs/nfs/dir.c +11 -18
@@ -889,6 +889,7 @@ static int nfs_readdir_xdr_to_array(struct nfs_readdir_descriptor *desc,size_t array_size;struct inode *inode =file_inode (desc->file);unsigned int dtsize = desc->dtsize; +unsigned int pglen;int status = -ENOMEM ; entry =kzalloc (sizeof (*entry),GFP_KERNEL );
@@ -906,28 +907,20 @@ static int nfs_readdir_xdr_to_array(struct nfs_readdir_descriptor *desc,if (!pages) goto out; -do { -unsigned int pglen; - status =nfs_readdir_xdr_filler (desc, verf_arg, entry->cookie, - pages, dtsize, - verf_res); -if (status <0 ) -break ; - - pglen = status; -if (pglen ==0 ) { -nfs_readdir_page_set_eof (page); -break ; - } - - verf_arg = verf_res; + status =nfs_readdir_xdr_filler (desc, verf_arg, entry->cookie, pages, + dtsize, verf_res); +if (status <0 ) + goto free_pages; + pglen = status; +if (pglen !=0 ) status =nfs_readdir_page_filler (desc, entry, pages, pglen, arrays, narrays); - desc->buffer_fills++; - }while (!status &&nfs_readdir_page_needs_filling (page) && -page_mapping (page)); +else +nfs_readdir_page_set_eof (page); + desc->buffer_fills++; +free_pages:nfs_readdir_free_pages (pages, array_size); out:nfs_free_fattr (entry->fattr);
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 14/26 NFS: Improve heuristic for readdirplus
In reply to [email protected]From: Trond Myklebust <[email protected]>
The heuristic for readdirplus is designed to try to detect 'ls -l' and similar patterns. It does so by looking for cache hit/miss patterns in both the attribute cache and in the dcache of the files in a given directory, and then sets a flag for the readdirplus code to interpret.
The problem with this approach is that a single attribute or dcache miss can cause the NFS code to force a refresh of the attributes for the entire set of files contained in the directory.
To be able to make a more nuanced decision, let's sample the number of hits and misses in the set of open directory descriptors. That allows us to set thresholds at which we start preferring READDIRPLUS over regular READDIR, or at which we start to force a re-read of the remaining readdir cache using READDIRPLUS.
--- fs/nfs/dir.c | 80 +++++++++++++++++++++++++++--------------- fs/nfs/inode.c | 4 +-- fs/nfs/internal.h | 4 +-- fs/nfs/nfstrace.h | 1 - include/linux/nfs_fs.h | 5 +-- 5 files changed, 58 insertions(+), 36 deletions(-)
fs/nfs/dir.c +51 -29
@@ -87,8 +87,7 @@ alloc_nfs_open_dir_context(struct inode *dir)nfs_set_cache_invalid (dir,NFS_INO_INVALID_DATA |NFS_INO_REVAL_FORCED ); -list_add (&ctx->list, &nfsi->open_files); -clear_bit (NFS_INO_FORCE_READDIR , &nfsi->flags); +list_add_tail_rcu (&ctx->list, &nfsi->open_files);memcpy (ctx->verf, nfsi->cookieverf,sizeof (ctx->verf));spin_unlock (&dir->i_lock);return ctx;
@@ -99,9 +98,9 @@ alloc_nfs_open_dir_context(struct inode *dir)static void put_nfs_open_dir_context (struct inode *dir,struct nfs_open_dir_context *ctx) {spin_lock (&dir->i_lock); -list_del (&ctx->list); +list_del_rcu (&ctx->list);spin_unlock (&dir->i_lock); -kfree (ctx); +kfree_rcu (ctx, rcu_head); }/*
@@ -594,7 +593,6 @@ static int nfs_readdir_xdr_filler(struct nfs_readdir_descriptor *desc,/* We requested READDIRPLUS, but the server doesn't grok it */ if (error == -ENOTSUPP && desc->plus) {NFS_SERVER (inode)->caps &= ~NFS_CAP_READDIRPLUS ; -clear_bit (NFS_INO_ADVISE_RDPLUS , &NFS_I (inode)->flags); desc->plus = arg.plus = false; goto again; }
@@ -644,51 +642,63 @@ int nfs_same_file(struct dentry *dentry, struct nfs_entry *entry)return 1 ; } -static -bool nfs_use_readdirplus (struct inode *dir,struct dir_context *ctx) +#define NFS_READDIR_CACHE_USAGE_THRESHOLD (8UL) + +static bool nfs_use_readdirplus (struct inode *dir,struct dir_context *ctx, +unsigned int cache_hits, +unsigned int cache_misses) {if (!nfs_server_capable (dir,NFS_CAP_READDIRPLUS ))return false; -if (test_and_clear_bit (NFS_INO_ADVISE_RDPLUS , &NFS_I (dir)->flags)) -return true; -if (ctx->pos ==0 ) +if (ctx->pos ==0 || + cache_hits + cache_misses >NFS_READDIR_CACHE_USAGE_THRESHOLD )return true;return false; }/* -* This function is called by the lookup and getattr code to request the +* This function is called by the getattr code to request the * use of readdirplus to accelerate any future lookups in the same * directory. */ -void nfs_advise_use_readdirplus (struct inode *dir) +void nfs_readdir_record_entry_cache_hit (struct inode *dir) {struct nfs_inode *nfsi =NFS_I (dir); +struct nfs_open_dir_context *ctx;if (nfs_server_capable (dir,NFS_CAP_READDIRPLUS ) && - !list_empty (&nfsi->open_files)) -set_bit (NFS_INO_ADVISE_RDPLUS , &nfsi->flags); +S_ISDIR (dir->i_mode)) { +rcu_read_lock (); + list_for_each_entry_rcu (ctx, &nfsi->open_files, list) +atomic_inc (&ctx->cache_hits); +rcu_read_unlock (); + } }/* * This function is mainly for use by nfs_getattr(). * * If this is an 'ls -l', we want to force use of readdirplus. -* Do this by checking if there is an active file descriptor -* and calling nfs_advise_use_readdirplus, then forcing a -* cache flush. */ -void nfs_force_use_readdirplus (struct inode *dir) +void nfs_readdir_record_entry_cache_miss (struct inode *dir) {struct nfs_inode *nfsi =NFS_I (dir); +struct nfs_open_dir_context *ctx;if (nfs_server_capable (dir,NFS_CAP_READDIRPLUS ) && - !list_empty (&nfsi->open_files)) { -set_bit (NFS_INO_ADVISE_RDPLUS , &nfsi->flags); -set_bit (NFS_INO_FORCE_READDIR , &nfsi->flags); +S_ISDIR (dir->i_mode)) { +rcu_read_lock (); + list_for_each_entry_rcu (ctx, &nfsi->open_files, list) +atomic_inc (&ctx->cache_misses); +rcu_read_unlock (); } } +static void nfs_lookup_advise_force_readdirplus (struct inode *dir) +{ +nfs_readdir_record_entry_cache_miss (dir); +} +static void nfs_prime_dcache (struct dentry *parent,struct nfs_entry *entry,unsigned long dir_verifier)
@@ -1122,6 +1132,19 @@ static int uncached_readdir(struct nfs_readdir_descriptor *desc)return status; } +#define NFS_READDIR_CACHE_MISS_THRESHOLD (16UL) + +static void nfs_readdir_handle_cache_misses (struct inode *inode, +struct nfs_readdir_descriptor *desc, +pgoff_t page_index, +unsigned int cache_misses) +{ +if (desc->ctx->pos ==0 || + cache_misses <=NFS_READDIR_CACHE_MISS_THRESHOLD ) +return ; +invalidate_mapping_pages (inode->i_mapping, page_index +1 ,-1 ); +} +/* The file offset position represents the dirent entry number. A last cookie cache takes care of the common case of reading the whole directory.
@@ -1133,6 +1156,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)struct nfs_inode *nfsi =NFS_I (inode);struct nfs_open_dir_context *dir_ctx = file->private_data;struct nfs_readdir_descriptor *desc; +unsigned int cache_hits, cache_misses;pgoff_t page_index;int res;
@@ -1154,7 +1178,6 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) goto out; desc->file = file; desc->ctx = ctx; - desc->plus =nfs_use_readdirplus (inode, ctx); desc->page_index_max =-1 ;spin_lock (&file->f_lock);
@@ -1168,6 +1191,8 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) desc->eof = dir_ctx->eof;nfs_set_dtsize (desc, dir_ctx->dtsize);memcpy (desc->verf, dir_ctx->verf,sizeof (desc->verf)); + cache_hits =atomic_xchg (&dir_ctx->cache_hits,0 ); + cache_misses =atomic_xchg (&dir_ctx->cache_misses,0 );spin_unlock (&file->f_lock);if (desc->eof) {
@@ -1175,9 +1200,8 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) goto out_free; } -if (test_and_clear_bit (NFS_INO_FORCE_READDIR , &nfsi->flags) && -list_is_singular (&nfsi->open_files)) -invalidate_mapping_pages (inode->i_mapping, page_index +1 ,-1 ); + desc->plus =nfs_use_readdirplus (inode, ctx, cache_hits, cache_misses); +nfs_readdir_handle_cache_misses (inode, desc, page_index, cache_misses); do { res = readdir_search_pagecache(desc);
@@ -1196,7 +1220,6 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)break ; }if (res == -ETOOSMALL && desc->plus) { -clear_bit (NFS_INO_ADVISE_RDPLUS , &nfsi->flags);nfs_zap_caches (inode); desc->page_index =0 ; desc->plus = false;
@@ -1610,7 +1633,7 @@ nfs_lookup_revalidate_dentry(struct inode *dir, struct dentry *dentry,nfs_set_verifier (dentry, dir_verifier);/* set a readdirplus hint that we had a cache miss */ -nfs_force_use_readdirplus (dir); +nfs_lookup_advise_force_readdirplus (dir); ret =1 ; out:nfs_free_fattr (fattr);
@@ -1667,7 +1690,6 @@ nfs_do_lookup_revalidate(struct inode *dir, struct dentry *dentry,nfs_mark_dir_for_revalidate (dir); goto out_bad; } -nfs_advise_use_readdirplus (dir); goto out_valid; }
@@ -1872,7 +1894,7 @@ struct dentry *nfs_lookup(struct inode *dir, struct dentry * dentry, unsigned in goto out;/* Notify readdir to use READDIRPLUS */ -nfs_force_use_readdirplus (dir); +nfs_lookup_advise_force_readdirplus (dir); no_entry: res =d_splice_alias (inode, dentry);
fs/nfs/inode.c +2 -2
@@ -787,7 +787,7 @@ static void nfs_readdirplus_parent_cache_miss(struct dentry *dentry)if (!nfs_server_capable (d_inode (dentry),NFS_CAP_READDIRPLUS ))return ; parent =dget_parent (dentry); -nfs_force_use_readdirplus (d_inode (parent)); +nfs_readdir_record_entry_cache_miss (d_inode (parent));dput (parent); }
@@ -798,7 +798,7 @@ static void nfs_readdirplus_parent_cache_hit(struct dentry *dentry)if (!nfs_server_capable (d_inode (dentry),NFS_CAP_READDIRPLUS ))return ; parent =dget_parent (dentry); -nfs_advise_use_readdirplus (d_inode (parent)); +nfs_readdir_record_entry_cache_hit (d_inode (parent));dput (parent); }
fs/nfs/internal.h +2 -2
@@ -366,8 +366,8 @@ extern struct nfs_client *nfs_init_client(struct nfs_client *clp, const struct nfs_client_initdata *);/* dir.c */ -extern void nfs_advise_use_readdirplus (struct inode *dir); -extern void nfs_force_use_readdirplus (struct inode *dir); +extern void nfs_readdir_record_entry_cache_hit (struct inode *dir); +extern void nfs_readdir_record_entry_cache_miss (struct inode *dir);extern unsigned long nfs_access_cache_count (struct shrinker *shrink,struct shrink_control *sc); extern unsigned long nfs_access_cache_scan(struct shrinker *shrink,
fs/nfs/nfstrace.h +0 -1
@@ -36,7 +36,6 @@#define nfs_show_nfsi_flags (v) \ __print_flags(v, "|", \ - { BIT(NFS_INO_ADVISE_RDPLUS), "ADVISE_RDPLUS" }, \ { BIT(NFS_INO_STALE), "STALE" }, \ { BIT(NFS_INO_ACL_LRU_SET), "ACL_LRU_SET" }, \ { BIT(NFS_INO_INVALIDATING), "INVALIDATING" }, \
include/linux/nfs_fs.h +3 -2
@@ -101,6 +101,8 @@ struct nfs_open_context {struct nfs_open_dir_context {struct list_head list; +atomic_t cache_hits; +atomic_t cache_misses;unsigned long attr_gencount;__be32 verf[NFS_DIR_VERIFIER_SIZE ];__u64 dir_cookie;
@@ -110,6 +112,7 @@ struct nfs_open_dir_context {unsigned int dtsize;signed char duped;bool eof; +struct rcu_head rcu_head; };/*
@@ -274,13 +277,11 @@ struct nfs4_copy_state {/* * Bit offsets in flags field */ -#define NFS_INO_ADVISE_RDPLUS (0)/* advise readdirplus */ #define NFS_INO_STALE (1)/* possible stale inode */ #define NFS_INO_ACL_LRU_SET (2)/* Inode is on the LRU list */ #define NFS_INO_INVALIDATING (3)/* inode is being invalidated */ #define NFS_INO_PRESERVE_UNLINKED (4)/* preserve file if removed while open */ #define NFS_INO_FSCACHE (5)/* inode can be cached by FS-Cache */ -#define NFS_INO_FORCE_READDIR (7)/* force readdirplus */ #define NFS_INO_LAYOUTCOMMIT (9)/* layoutcommit required */ #define NFS_INO_LAYOUTCOMMITTING (10)/* layoutcommit inflight */ #define NFS_INO_LAYOUTSTATS (11)/* layoutstats inflight */
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 16/26 NFSv4: Ask for a full XDR buffer of readdir goodness
In reply to [email protected]From: Trond Myklebust <[email protected]>
Instead of pretending that we know the ratio of directory info vs readdirplus attribute info, just set the 'dircount' field to the same value as the 'maxcount' field.
--- fs/nfs/nfs3xdr.c | 7 ++++--- fs/nfs/nfs4xdr.c | 6 +++--- 2 files changed, 7 insertions(+), 6 deletions(-)
fs/nfs/nfs3xdr.c +4 -3
@@ -1261,6 +1261,8 @@ static void nfs3_xdr_enc_readdir3args(struct rpc_rqst *req,static void encode_readdirplus3args (struct xdr_stream *xdr,const struct nfs3_readdirargs *args) { +uint32_t dircount = args->count; +uint32_t maxcount = args->count;__be32 *p;encode_nfs_fh3 (xdr, args->fh);
@@ -1273,9 +1275,8 @@ static void encode_readdirplus3args(struct xdr_stream *xdr,* readdirplus: need dircount + buffer size. * We just make sure we make dircount big enough */ - *p++ =cpu_to_be32 (args->count >>3 ); - - *p =cpu_to_be32 (args->count); + *p++ =cpu_to_be32 (dircount); + *p =cpu_to_be32 (maxcount); } static void nfs3_xdr_enc_readdirplus3args(struct rpc_rqst *req,
fs/nfs/nfs4xdr.c +3 -3
@@ -1605,7 +1605,8 @@ static void encode_readdir(struct xdr_stream *xdr, const struct nfs4_readdir_argFATTR4_WORD0_RDATTR_ERROR , FATTR4_WORD1_MOUNTED_ON_FILEID, }; -uint32_t dircount = readdir->count >>1 ; +uint32_t dircount = readdir->count; +uint32_t maxcount = readdir->count;__be32 *p, verf[2 ];uint32_t attrlen =0 ;unsigned int i;
@@ -1618,7 +1619,6 @@ static void encode_readdir(struct xdr_stream *xdr, const struct nfs4_readdir_argFATTR4_WORD1_SPACE_USED |FATTR4_WORD1_TIME_ACCESS |FATTR4_WORD1_TIME_METADATA |FATTR4_WORD1_TIME_MODIFY ; attrs[2 ] |=FATTR4_WORD2_SECURITY_LABEL ; - dircount >>=1 ; }/* Use mounted_on_fileid only if the server supports it */ if (!(readdir->bitmask[1 ] &FATTR4_WORD1_MOUNTED_ON_FILEID ))
@@ -1634,7 +1634,7 @@ static void encode_readdir(struct xdr_stream *xdr, const struct nfs4_readdir_argencode_nfs4_verifier (xdr, &readdir->verifier); p =reserve_space (xdr,12 + (attrlen <<2 )); *p++ =cpu_to_be32 (dircount); - *p++ =cpu_to_be32 (readdir->count); + *p++ =cpu_to_be32 (maxcount); *p++ =cpu_to_be32 (attrlen);for (i =0 ; i < attrlen; i++) *p++ =cpu_to_be32 (attrs[i]);
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 15/26 NFS: Don't ask for readdirplus unless it can help nfs_getattr()
In reply to [email protected]From: Trond Myklebust <[email protected]>
If attribute caching is turned off, then use of readdirplus is not going to help stat() performance.
Readdirplus also doesn't help if a file is being written to, since we will have to flush those writes in order to sync the mtime/ctime.
--- fs/nfs/inode.c | 45 +++++++++++++++++++++++++-------------------- 1 file changed, 25 insertions(+), 20 deletions(-)
fs/nfs/inode.c +25 -20
@@ -780,26 +780,32 @@ void nfs_setattr_update_inode(struct inode *inode, struct iattr *attr, }EXPORT_SYMBOL_GPL (nfs_setattr_update_inode); -static void nfs_readdirplus_parent_cache_miss (struct dentry *dentry) +/* +* Don't request help from readdirplus if the file is being written to, +* or if attribute caching is turned off +*/ +static bool nfs_getattr_readdirplus_enable (const struct inode *inode) { -struct dentry *parent; +return nfs_server_capable (inode,NFS_CAP_READDIRPLUS ) && + !nfs_have_writebacks (inode) &&NFS_MAXATTRTIMEO (inode) >5 *HZ ; +} -if (!nfs_server_capable (d_inode (dentry),NFS_CAP_READDIRPLUS )) -return ; - parent =dget_parent (dentry); -nfs_readdir_record_entry_cache_miss (d_inode (parent)); -dput (parent); +static void nfs_readdirplus_parent_cache_miss (struct dentry *dentry) +{ +if (!IS_ROOT (dentry)) { +struct dentry *parent =dget_parent (dentry); +nfs_readdir_record_entry_cache_miss (d_inode (parent)); +dput (parent); + } }static void nfs_readdirplus_parent_cache_hit (struct dentry *dentry) { -struct dentry *parent; - -if (!nfs_server_capable (d_inode (dentry),NFS_CAP_READDIRPLUS )) -return ; - parent =dget_parent (dentry); -nfs_readdir_record_entry_cache_hit (d_inode (parent)); -dput (parent); +if (!IS_ROOT (dentry)) { +struct dentry *parent =dget_parent (dentry); +nfs_readdir_record_entry_cache_hit (d_inode (parent)); +dput (parent); + } }static u32 nfs_get_valid_attrmask (struct inode *inode)
@@ -835,6 +841,7 @@ int nfs_getattr(struct user_namespace *mnt_userns, const struct path *path,int err =0 ;bool force_sync = query_flags &AT_STATX_FORCE_SYNC ;bool do_update = false; +bool readdirplus_enabled =nfs_getattr_readdirplus_enable (inode);trace_nfs_getattr_enter (inode);
@@ -843,7 +850,8 @@ int nfs_getattr(struct user_namespace *mnt_userns, const struct path *path,STATX_INO |STATX_SIZE |STATX_BLOCKS ;if ((query_flags &AT_STATX_DONT_SYNC ) && !force_sync) { -nfs_readdirplus_parent_cache_hit (path->dentry); +if (readdirplus_enabled) +nfs_readdirplus_parent_cache_hit (path->dentry); goto out_no_revalidate; }
@@ -893,15 +901,12 @@ int nfs_getattr(struct user_namespace *mnt_userns, const struct path *path, do_update |= cache_validity &NFS_INO_INVALID_BLOCKS ;if (do_update) { -/* Update the attribute cache */ -if (!(server->flags &NFS_MOUNT_NOAC )) +if (readdirplus_enabled)nfs_readdirplus_parent_cache_miss (path->dentry); -else -nfs_readdirplus_parent_cache_hit (path->dentry); err =__nfs_revalidate_inode (server, inode);if (err) goto out; - }else + }else if (readdirplus_enabled)nfs_readdirplus_parent_cache_hit (path->dentry); out_no_revalidate:/* Only return attributes that were revalidated. */
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 17/26 NFS: Readdirplus can't help lookup for case insensitive filesystems
In reply to [email protected]From: Trond Myklebust <[email protected]>
If the filesystem is case insensitive, then readdirplus can't help with cache misses, since it won't return case folded variants of the filename.
--- fs/nfs/dir.c | 2 ++ 1 file changed, 2 insertions(+)
fs/nfs/dir.c +2 -0
@@ -696,6 +696,8 @@ void nfs_readdir_record_entry_cache_miss(struct inode *dir)static void nfs_lookup_advise_force_readdirplus (struct inode *dir) { +if (nfs_server_capable (dir,NFS_CAP_CASE_INSENSITIVE )) +return ;nfs_readdir_record_entry_cache_miss (dir); }
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 18/26 NFS: Don't request readdirplus when revalidation was forced
From: Trond Myklebust <[email protected]>
If the revalidation was forced, due to the presence of a LOOKUP_EXCL or a LOOKUP_REVAL flag, then readdirplus won't help. It also can't help when we're doing a path component lookup.
--- fs/nfs/dir.c | 26 ++++++++++++++++---------- 1 file changed, 16 insertions(+), 10 deletions(-)
fs/nfs/dir.c +16 -10
@@ -694,10 +694,13 @@ void nfs_readdir_record_entry_cache_miss(struct inode *dir) } } -static void nfs_lookup_advise_force_readdirplus (struct inode *dir) +static void nfs_lookup_advise_force_readdirplus (struct inode *dir, +unsigned int flags) {if (nfs_server_capable (dir,NFS_CAP_CASE_INSENSITIVE ))return ; +if (flags & (LOOKUP_EXCL |LOOKUP_PARENT |LOOKUP_REVAL )) +return ;nfs_readdir_record_entry_cache_miss (dir); }
@@ -1596,15 +1599,17 @@ nfs_lookup_revalidate_delegated(struct inode *dir, struct dentry *dentry,return nfs_lookup_revalidate_done (dir, dentry, inode,1 ); } -static int -nfs_lookup_revalidate_dentry (struct inode *dir,struct dentry *dentry, -struct inode *inode) +static int nfs_lookup_revalidate_dentry (struct inode *dir, +struct dentry *dentry, +struct inode *inode,unsigned int flags) {struct nfs_fh *fhandle;struct nfs_fattr *fattr;unsigned long dir_verifier;int ret; +trace_nfs_lookup_revalidate_enter (dir, dentry, flags); + ret = -ENOMEM ; fhandle =nfs_alloc_fhandle (); fattr =nfs_alloc_fattr_with_label (NFS_SERVER (inode));
@@ -1625,6 +1630,10 @@ nfs_lookup_revalidate_dentry(struct inode *dir, struct dentry *dentry, } goto out; } + +/* Request help from readdirplus */ +nfs_lookup_advise_force_readdirplus (dir, flags); + ret =0 ;if (nfs_compare_fh (NFS_FH (inode), fhandle)) goto out;
@@ -1634,8 +1643,6 @@ nfs_lookup_revalidate_dentry(struct inode *dir, struct dentry *dentry,nfs_setsecurity (inode, fattr);nfs_set_verifier (dentry, dir_verifier); -/* set a readdirplus hint that we had a cache miss */ -nfs_lookup_advise_force_readdirplus (dir); ret =1 ; out:nfs_free_fattr (fattr);
@@ -1701,8 +1708,7 @@ nfs_do_lookup_revalidate(struct inode *dir, struct dentry *dentry,if (NFS_STALE (inode)) goto out_bad; -trace_nfs_lookup_revalidate_enter (dir, dentry, flags); -return nfs_lookup_revalidate_dentry (dir, dentry, inode); +return nfs_lookup_revalidate_dentry (dir, dentry, inode, flags); out_valid:return nfs_lookup_revalidate_done (dir, dentry, inode,1 ); out_bad:
@@ -1896,7 +1902,7 @@ struct dentry *nfs_lookup(struct inode *dir, struct dentry * dentry, unsigned in goto out;/* Notify readdir to use READDIRPLUS */ -nfs_lookup_advise_force_readdirplus (dir); +nfs_lookup_advise_force_readdirplus (dir, flags); no_entry: res =d_splice_alias (inode, dentry);
@@ -2159,7 +2165,7 @@ nfs4_do_lookup_revalidate(struct inode *dir, struct dentry *dentry, reval_dentry:if (flags &LOOKUP_RCU )return -ECHILD ; -return nfs_lookup_revalidate_dentry (dir, dentry, inode); +return nfs_lookup_revalidate_dentry (dir, dentry, inode, flags); full_reval:return nfs_do_lookup_revalidate (dir, dentry, flags);
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 19/26 NFS: Add basic readdir tracing
From: Trond Myklebust <[email protected]>
Add tracing to track how often the client goes to the server for updated readdir information.
--- fs/nfs/dir.c | 13 ++++++++- fs/nfs/nfstrace.h | 68 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+), 1 deletion(-)
fs/nfs/dir.c +12 -1
@@ -984,10 +984,14 @@ static int find_and_lock_cache_page(struct nfs_readdir_descriptor *desc)if (desc->page_index == desc->page_index_max)nfs_grow_dtsize (desc); desc->page_index_max = desc->page_index; +trace_nfs_readdir_cache_fill (desc->file, nfsi->cookieverf, + desc->last_cookie, + desc->page->index, desc->dtsize); res =nfs_readdir_xdr_to_array (desc, nfsi->cookieverf, verf, &desc->page,1 );if (res <0 ) {nfs_readdir_page_unlock_and_put_cached (desc); +trace_nfs_readdir_cache_fill_done (inode, res);if (res == -EBADCOOKIE || res == -ENOTSYNC ) {invalidate_inode_pages2 (desc->file->f_mapping); desc->page_index =0 ;
@@ -1108,7 +1112,14 @@ static int uncached_readdir(struct nfs_readdir_descriptor *desc) desc->duped =0 ; desc->page_index_max =0 ; +trace_nfs_readdir_uncached (desc->file, desc->verf, desc->last_cookie, +-1 , desc->dtsize); + status =nfs_readdir_xdr_to_array (desc, desc->verf, verf, arrays, sz); +if (status <0 ) { +trace_nfs_readdir_uncached_done (file_inode (desc->file), status); + goto out_free; + }for (i =0 ; !desc->eob && i < sz && arrays[i]; i++) { desc->page = arrays[i];
@@ -1127,7 +1138,7 @@ static int uncached_readdir(struct nfs_readdir_descriptor *desc) i < (desc->page_index_max >> 1))nfs_shrink_dtsize (desc); } - +out_free:for (i =0 ; i < sz && arrays[i]; i++)nfs_readdir_page_array_free (arrays[i]); out:
fs/nfs/nfstrace.h +68 -0
80 lines not on this page · raw message
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 21/26 NFS: Trace effects of the readdirplus heuristic
In reply to [email protected]From: Trond Myklebust <[email protected]>
Enable tracking of when the readdirplus heuristic causes a page cache invalidation.
--- fs/nfs/dir.c | 11 ++++++++++- fs/nfs/nfstrace.h | 50 +++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 60 insertions(+), 1 deletion(-)
fs/nfs/dir.c +10 -1
@@ -1000,6 +1000,8 @@ static int find_and_lock_cache_page(struct nfs_readdir_descriptor *desc)if (res == -EBADCOOKIE || res == -ENOTSYNC ) {invalidate_inode_pages2 (desc->file->f_mapping); desc->page_index =0 ; +trace_nfs_readdir_invalidate_cache_range ( + inode,0 ,MAX_LFS_FILESIZE );return -EAGAIN ; }return res;
@@ -1014,6 +1016,9 @@ static int find_and_lock_cache_page(struct nfs_readdir_descriptor *desc)invalidate_inode_pages2_range (desc->file->f_mapping, desc->page_index_max +1 ,-1 ); +trace_nfs_readdir_invalidate_cache_range ( + inode, desc->page_index_max +1 , +MAX_LFS_FILESIZE ); } } res =nfs_readdir_search_array (desc);
@@ -1163,7 +1168,11 @@ static void nfs_readdir_handle_cache_misses(struct inode *inode,if (desc->ctx->pos ==0 || cache_misses <=NFS_READDIR_CACHE_MISS_THRESHOLD )return ; -invalidate_mapping_pages (inode->i_mapping, page_index +1 ,-1 ); +if (invalidate_mapping_pages (inode->i_mapping, page_index +1 ,-1 ) ==0 ) +return ; +trace_nfs_readdir_invalidate_cache_range ( + inode, (loff_t)(page_index +1 ) <<PAGE_SHIFT , +MAX_LFS_FILESIZE ); }/* The file offset position represents the dirent entry number. A
fs/nfs/nfstrace.h +50 -0
56 lines not on this page · raw message
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 20/26 NFS: Trace effects of readdirplus on the dcache
In reply to [email protected]From: Trond Myklebust <[email protected]>
Trace the effects of readdirplus on attribute and dentry revalidation.
--- fs/nfs/dir.c | 5 +++++ fs/nfs/nfstrace.h | 3 +++ 2 files changed, 8 insertions(+)
fs/nfs/dir.c +5 -0
@@ -754,8 +754,12 @@ void nfs_prime_dcache(struct dentry *parent, struct nfs_entry *entry, status =nfs_refresh_inode (d_inode (dentry), entry->fattr);if (!status)nfs_setsecurity (d_inode (dentry), entry->fattr); +trace_nfs_readdir_lookup_revalidate (d_inode (parent), + dentry,0 , status); goto out; }else { +trace_nfs_readdir_lookup_revalidate_failed ( +d_inode (parent), dentry,0 );d_invalidate (dentry);dput (dentry); dentry =NULL ;
@@ -777,6 +781,7 @@ void nfs_prime_dcache(struct dentry *parent, struct nfs_entry *entry, dentry = alias; }nfs_set_verifier (dentry, dir_verifier); +trace_nfs_readdir_lookup (d_inode (parent), dentry,0 ); out:dput (dentry); }
fs/nfs/nfstrace.h +3 -0
@@ -432,6 +432,9 @@ DEFINE_NFS_LOOKUP_EVENT(nfs_lookup_enter);DEFINE_NFS_LOOKUP_EVENT_DONE (nfs_lookup_exit);DEFINE_NFS_LOOKUP_EVENT (nfs_lookup_revalidate_enter);DEFINE_NFS_LOOKUP_EVENT_DONE (nfs_lookup_revalidate_exit); +DEFINE_NFS_LOOKUP_EVENT (nfs_readdir_lookup); +DEFINE_NFS_LOOKUP_EVENT (nfs_readdir_lookup_revalidate_failed); +DEFINE_NFS_LOOKUP_EVENT_DONE (nfs_readdir_lookup_revalidate); TRACE_EVENT(nfs_atomic_open_enter, TP_PROTO(
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 22/26 NFS: Clean up page array initialisation/free
In reply to [email protected]From: Trond Myklebust <[email protected]>
--- fs/nfs/dir.c | 16 ++++++---------- 1 file changed, 6 insertions(+), 10 deletions(-)
fs/nfs/dir.c +6 -10
@@ -199,20 +199,17 @@ static void nfs_grow_dtsize(struct nfs_readdir_descriptor *desc)nfs_set_dtsize (desc, desc->dtsize <<1 ); } -static void nfs_readdir_array_init (struct nfs_cache_array *array) -{ -memset (array,0 ,sizeof (struct nfs_cache_array )); -} -static void nfs_readdir_page_init_array (struct page *page,u64 last_cookie,u64 change_attr) {struct nfs_cache_array *array; array =kmap_atomic (page); -nfs_readdir_array_init (array); array->change_attr = change_attr; array->last_cookie = last_cookie; + array->size =0 ; + array->page_full =0 ; + array->page_is_eof =0 ; array->cookies_are_ordered =1 ;kunmap_atomic (array); }
@@ -220,16 +217,15 @@ static void nfs_readdir_page_init_array(struct page *page, u64 last_cookie,/* * we are freeing strings created by nfs_add_to_readdir_array() */ -static -void nfs_readdir_clear_array (struct page *page) +static void nfs_readdir_clear_array (struct page *page) {struct nfs_cache_array *array; -int i; +unsigned int i; array =kmap_atomic (page);for (i =0 ; i < array->size; i++)kfree (array->array[i].name); -nfs_readdir_array_init (array); + array->size =0 ;kunmap_atomic (array); }
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 25/26 NFS: Optimise away the previous cookie field
In reply to [email protected]From: Trond Myklebust <[email protected]>
Replace the 'previous cookie' field in struct nfs_entry with the array->last_cookie.
--- fs/nfs/dir.c | 26 ++++++++++++++------------ fs/nfs/nfs2xdr.c | 1 - fs/nfs/nfs3xdr.c | 1 - fs/nfs/nfs4xdr.c | 1 - include/linux/nfs_xdr.h | 3 +-- 5 files changed, 15 insertions(+), 17 deletions(-)
fs/nfs/dir.c +14 -12
@@ -301,19 +301,20 @@ static int nfs_readdir_array_can_expand(struct nfs_cache_array *array)return 0 ; } -static -int nfs_readdir_add_to_array (struct nfs_entry *entry,struct page *page) +static int nfs_readdir_page_array_append (struct page *page, +const struct nfs_entry *entry, +u64 *cookie) {struct nfs_cache_array *array;struct nfs_cache_array_entry *cache_entry;const char *name; -int ret; +int ret = -ENOMEM ; name =nfs_readdir_copy_name (entry->name, entry->len); -if (!name) -return -ENOMEM ; array =kmap_atomic (page); +if (!name) + goto out; ret =nfs_readdir_array_can_expand (array);if (ret) {kfree (name);
@@ -321,7 +322,7 @@ int nfs_readdir_add_to_array(struct nfs_entry *entry, struct page *page) } cache_entry = &array->array[array->size]; - cache_entry->cookie = entry->prev_cookie; + cache_entry->cookie = array->last_cookie; cache_entry->ino = entry->ino; cache_entry->d_type = entry->d_type; cache_entry->name_len = entry->len;
@@ -333,6 +334,7 @@ int nfs_readdir_add_to_array(struct nfs_entry *entry, struct page *page)if (entry->eof !=0 )nfs_readdir_array_set_eof (array); out: + *cookie = array->last_cookie;kunmap_atomic (array);return ret; }
@@ -798,6 +800,7 @@ static int nfs_readdir_page_filler(struct nfs_readdir_descriptor *desc,struct xdr_stream stream;struct xdr_buf buf;struct page *scratch, *new, *page = *arrays; +u64 cookie;int status; scratch =alloc_page (GFP_KERNEL );
@@ -819,22 +822,21 @@ static int nfs_readdir_page_filler(struct nfs_readdir_descriptor *desc,nfs_prime_dcache (file_dentry (desc->file), entry, desc->dir_verifier); - status =nfs_readdir_add_to_array (entry, page); + status =nfs_readdir_page_array_append (page, entry, &cookie);if (status != -ENOSPC )continue ;if (page->mapping != mapping) {if (!--narrays)break ; - new =nfs_readdir_page_array_alloc (entry->prev_cookie, -GFP_KERNEL ); + new =nfs_readdir_page_array_alloc (cookie,GFP_KERNEL );if (!new)break ; arrays++; *arrays = page = new; }else { - new =nfs_readdir_page_get_next ( - mapping, entry->prev_cookie, change_attr); + new =nfs_readdir_page_get_next (mapping, cookie, + change_attr);if (!new)break ;if (page != *arrays)
@@ -842,7 +844,7 @@ static int nfs_readdir_page_filler(struct nfs_readdir_descriptor *desc, page = new; } desc->page_index_max++; - status =nfs_readdir_add_to_array (entry, page); + status =nfs_readdir_page_array_append (page, entry, &cookie); }while (!status && !entry->eof);switch (status) {
fs/nfs/nfs2xdr.c +0 -1
@@ -955,7 +955,6 @@ int nfs2_decode_dirent(struct xdr_stream *xdr, struct nfs_entry *entry,* The type (size and byte order) of nfscookie isn't defined in * RFC 1094. This implementation assumes that it's an XDR uint32. */ - entry->prev_cookie = entry->cookie; p =xdr_inline_decode (xdr,4 );if (unlikely (!p))return -EAGAIN ;
fs/nfs/nfs3xdr.c +0 -1
@@ -2024,7 +2024,6 @@ int nfs3_decode_dirent(struct xdr_stream *xdr, struct nfs_entry *entry,zero_nfs_fh3 (entry->fh); } - entry->prev_cookie = entry->cookie; entry->cookie = new_cookie;return 0 ;
fs/nfs/nfs4xdr.c +0 -1
@@ -7508,7 +7508,6 @@ int nfs4_decode_dirent(struct xdr_stream *xdr, struct nfs_entry *entry,if (entry->fattr->valid &NFS_ATTR_FATTR_TYPE ) entry->d_type =nfs_umode_to_dtype (entry->fattr->mode); - entry->prev_cookie = entry->cookie; entry->cookie = new_cookie;return 0 ;
include/linux/nfs_xdr.h +1 -2
@@ -745,8 +745,7 @@ struct nfs_auth_info {*/ struct nfs_entry {__u64 ino; -__u64 cookie, - prev_cookie; +__u64 cookie;const char * name;unsigned int len;int eof;
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 23/26 NFS: Convert readdir page cache to use a cookie based index
In reply to [email protected]From: Trond Myklebust <[email protected]>
Instead of using a linear index to address the pages, use the cookie of the first entry, since that is what we use to match the page anyway.
This allows us to avoid re-reading the entire cache on a seekdir() type of operation. The latter is very common when re-exporting NFS, and is a major performance drain.
The change does affect our duplicate cookie detection, since we can no longer rely on the page index as a linear offset for detecting whether we looped backwards. However since we no longer do a linear search through all the pages on each call to nfs_readdir(), this is less of a concern than it was previously.
The other downside is that invalidate_mapping_pages() no longer can use the page index to avoid clearing pages that have been read. A subsequent patch will restore the functionality this provides to the 'ls -l' heuristic.
--- fs/nfs/Kconfig | 4 ++ fs/nfs/dir.c | 149 ++++++++++++++++++----------------------- include/linux/nfs_fs.h | 2 - 3 files changed, 69 insertions(+), 86 deletions(-)
fs/nfs/Kconfig +4 -0
@@ -4,6 +4,10 @@ config NFS_FS depends on INET && FILE_LOCKING && MULTIUSER select LOCKD select SUNRPC + select CRYPTO + select CRYPTO_HASH + select XXHASH + select CRYPTO_XXHASH select NFS_ACL_SUPPORT if NFS_V3_ACL help Choose Y here if you want to access files residing on other
fs/nfs/dir.c +65 -84
360 lines not on this page · raw message
include/linux/nfs_fs.h +0 -2
@@ -106,11 +106,9 @@ struct nfs_open_dir_context {unsigned long attr_gencount;__be32 verf[NFS_DIR_VERIFIER_SIZE ];__u64 dir_cookie; -__u64 dup_cookie;__u64 last_cookie;pgoff_t page_index;unsigned int dtsize; -signed char duped;bool eof;struct rcu_head rcu_head; };
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 24/26 NFS: Fix up forced readdirplus
From: Trond Myklebust <[email protected]>
Avoid clearing the entire readdir page cache if we're just doing forced readdirplus for the 'ls -l' heuristic.
--- fs/nfs/dir.c | 56 +++++++++++++++++++++++++++++------------- fs/nfs/nfstrace.h | 1 + include/linux/nfs_fs.h | 1 + 3 files changed, 41 insertions(+), 17 deletions(-)
fs/nfs/dir.c +39 -17
@@ -170,6 +170,7 @@ struct nfs_readdir_descriptor {unsigned int cache_entry_index;unsigned int buffer_fills;unsigned int dtsize; +bool clear_cache;bool plus;bool eob;bool eof;
@@ -227,6 +228,13 @@ static void nfs_readdir_clear_array(struct page *page)kunmap_atomic (array); } +static void nfs_readdir_page_reinit_array (struct page *page,u64 last_cookie, +u64 change_attr) +{ +nfs_readdir_clear_array (page); +nfs_readdir_page_init_array (page, last_cookie, change_attr); +} +static struct page *nfs_readdir_page_array_alloc (u64 last_cookie,gfp_t gfp_flags) {
@@ -428,12 +436,11 @@ static struct page *nfs_readdir_page_get_next(struct address_space *mapping,struct page *page; page =nfs_readdir_page_get_locked (mapping, cookie, change_attr); -if (page) { -if (nfs_readdir_page_last_cookie (page) == cookie) -return page; -nfs_readdir_page_unlock_and_put (page); - } -return NULL ; +if (!page) +return NULL ; +if (nfs_readdir_page_last_cookie (page) != cookie) +nfs_readdir_page_reinit_array (page, cookie, change_attr); +return page; } static inline
@@ -960,9 +967,15 @@ nfs_readdir_page_get_cached(struct nfs_readdir_descriptor *desc) {struct address_space *mapping = desc->file->f_mapping;u64 change_attr =inode_peek_iversion_raw (mapping->host); +u64 cookie = desc->last_cookie; +struct page *page; -return nfs_readdir_page_get_locked (mapping, desc->last_cookie, - change_attr); + page =nfs_readdir_page_get_locked (mapping, cookie, change_attr); +if (!page) +return NULL ; +if (desc->clear_cache && !nfs_readdir_page_needs_filling (page)) +nfs_readdir_page_reinit_array (page, cookie, change_attr); +return page; }/*
@@ -1013,6 +1026,7 @@ static int find_and_lock_cache_page(struct nfs_readdir_descriptor *desc)trace_nfs_readdir_invalidate_cache_range ( inode,1 ,MAX_LFS_FILESIZE ); } + desc->clear_cache = false; } res =nfs_readdir_search_array (desc);if (res ==0 )
@@ -1147,16 +1161,17 @@ static int uncached_readdir(struct nfs_readdir_descriptor *desc)#define NFS_READDIR_CACHE_MISS_THRESHOLD (16UL) -static void nfs_readdir_handle_cache_misses (struct inode *inode, +static bool nfs_readdir_handle_cache_misses (struct inode *inode,struct nfs_readdir_descriptor *desc, -unsigned int cache_misses) +unsigned int cache_misses, +bool force_clear) { -if (desc->ctx->pos ==0 || - cache_misses <=NFS_READDIR_CACHE_MISS_THRESHOLD ) -return ; -if (invalidate_mapping_pages (inode->i_mapping,0 ,-1 ) ==0 ) -return ; -trace_nfs_readdir_invalidate_cache_range (inode,0 ,MAX_LFS_FILESIZE ); +if (desc->ctx->pos ==0 || !desc->plus) +return false; +if (cache_misses <=NFS_READDIR_CACHE_MISS_THRESHOLD && !force_clear) +return false; +trace_nfs_readdir_force_readdirplus (inode); +return true; }/* The file offset position represents the dirent entry number. A
@@ -1171,6 +1186,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)struct nfs_open_dir_context *dir_ctx = file->private_data;struct nfs_readdir_descriptor *desc;unsigned int cache_hits, cache_misses; +bool force_clear;int res; dfprintk(FILE, "NFS: readdir(%pD2) starting at cookie %llu\n",
@@ -1203,6 +1219,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)memcpy (desc->verf, dir_ctx->verf,sizeof (desc->verf)); cache_hits =atomic_xchg (&dir_ctx->cache_hits,0 ); cache_misses =atomic_xchg (&dir_ctx->cache_misses,0 ); + force_clear = dir_ctx->force_clear;spin_unlock (&file->f_lock);if (desc->eof) {
@@ -1211,7 +1228,9 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) } desc->plus =nfs_use_readdirplus (inode, ctx, cache_hits, cache_misses); -nfs_readdir_handle_cache_misses (inode, desc, cache_misses); + force_clear =nfs_readdir_handle_cache_misses (inode, desc, cache_misses, + force_clear); + desc->clear_cache = force_clear; do { res = readdir_search_pagecache(desc);
@@ -1240,6 +1259,8 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx)nfs_do_filldir (desc, nfsi->cookieverf);nfs_readdir_page_unlock_and_put_cached (desc); +if (desc->page_index == desc->page_index_max) + desc->clear_cache = force_clear; }while (!desc->eob && !desc->eof);spin_lock (&file->f_lock);
@@ -1247,6 +1268,7 @@ static int nfs_readdir(struct file *file, struct dir_context *ctx) dir_ctx->last_cookie = desc->last_cookie; dir_ctx->attr_gencount = desc->attr_gencount; dir_ctx->page_index = desc->page_index; + dir_ctx->force_clear = force_clear; dir_ctx->eof = desc->eof; dir_ctx->dtsize = desc->dtsize;memcpy (dir_ctx->verf, desc->verf,sizeof (dir_ctx->verf));
fs/nfs/nfstrace.h +1 -0
@@ -160,6 +160,7 @@ DEFINE_NFS_INODE_EVENT(nfs_fsync_enter);DEFINE_NFS_INODE_EVENT_DONE (nfs_fsync_exit);DEFINE_NFS_INODE_EVENT (nfs_access_enter);DEFINE_NFS_INODE_EVENT_DONE (nfs_set_cache_invalid); +DEFINE_NFS_INODE_EVENT (nfs_readdir_force_readdirplus);DEFINE_NFS_INODE_EVENT_DONE (nfs_readdir_cache_fill_done);DEFINE_NFS_INODE_EVENT_DONE (nfs_readdir_uncached_done);
include/linux/nfs_fs.h +1 -0
@@ -109,6 +109,7 @@ struct nfs_open_dir_context {__u64 last_cookie;pgoff_t page_index;unsigned int dtsize; +bool force_clear;bool eof;struct rcu_head rcu_head; };
Signature · 1 line
2.35.1
[email protected]
maintainerPATCH v10 26/26 NFS: Cache all entries in the readdirplus reply
In reply to [email protected]From: Trond Myklebust <[email protected]>
Even if we're not able to cache all the entries in the readdir buffer, let's ensure that we do prime the dcache.
--- fs/nfs/dir.c | 40 ++++++++++++++++++++++++++-------------- 1 file changed, 26 insertions(+), 14 deletions(-)
fs/nfs/dir.c +26 -14
@@ -789,6 +789,21 @@ void nfs_prime_dcache(struct dentry *parent, struct nfs_entry *entry,dput (dentry); } +static int nfs_readdir_entry_decode (struct nfs_readdir_descriptor *desc, +struct nfs_entry *entry, +struct xdr_stream *stream) +{ +int ret; + +if (entry->fattr->label) + entry->fattr->label->len =NFS4_MAXLABELLEN ; + ret =xdr_decode (desc, entry, stream); +if (ret || !desc->plus) +return ret; +nfs_prime_dcache (file_dentry (desc->file), entry, desc->dir_verifier); +return 0 ; +} +/* Perform conversion from xdr to cache array */ static int nfs_readdir_page_filler(struct nfs_readdir_descriptor *desc, struct nfs_entry *entry,
@@ -811,17 +826,10 @@ static int nfs_readdir_page_filler(struct nfs_readdir_descriptor *desc,xdr_set_scratch_page (&stream, scratch); do { - if (entry->fattr->label) - entry->fattr->label->len = NFS4_MAXLABELLEN; - - status = xdr_decode(desc, entry, &stream); + status = nfs_readdir_entry_decode(desc, entry, &stream); if (status != 0) break; - if (desc->plus) - nfs_prime_dcache(file_dentry(desc->file), entry, - desc->dir_verifier); - status = nfs_readdir_page_array_append(page, entry, &cookie); if (status != -ENOSPC) continue;
@@ -849,15 +857,19 @@ static int nfs_readdir_page_filler(struct nfs_readdir_descriptor *desc,switch (status) {case -EBADCOOKIE : -if (entry->eof) { -nfs_readdir_page_set_eof (page); - status =0 ; - } -break ; -case -ENOSPC : +if (!entry->eof) +break ; +nfs_readdir_page_set_eof (page); + fallthrough;case -EAGAIN : status =0 ;break ; +case -ENOSPC : + status =0 ; +if (!desc->plus) +break ; +while (!nfs_readdir_entry_decode (desc, entry, &stream)) + ; }if (page != *arrays)
Signature · 1 line
2.35.1