NFS: Fix missing files in `ls` command output
Subsystems (2)
RFC PATCH 1/1 NFS: Fix missing files in `ls` command output
In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:
$ cd nfs $ ls a b c e f <<<< 'd' is missing $ cd d <<<< success
I verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit 580f236737d1 ("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.
Upon further analysis, it appears that there is a bad entry being decoded in nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.
However, I am unable to reproduce this issue with a simple example; it only occurs on our production servers.
--- fs/nfs/dir.c | 22 +++++++++++++++++----- 1 file changed, 17 insertions(+), 5 deletions(-)
fs/nfs/dir.c +17 -5
@@ -310,7 +310,7 @@ static int nfs_readdir_array_can_expand(struct nfs_cache_array *array)static int nfs_readdir_folio_array_append (struct folio *folio,const struct nfs_entry *entry, -u64 *cookie) +u64 *cookie,u64 *prev_cookie) {struct nfs_cache_array *array;struct nfs_cache_array_entry *cache_entry;
@@ -342,6 +342,7 @@ static int nfs_readdir_folio_array_append(struct folio *folio,nfs_readdir_array_set_eof (array); out: *cookie = array->last_cookie; + *prev_cookie = cache_entry->cookie;kunmap_local (array);return ret; }
@@ -826,10 +827,11 @@ static int nfs_readdir_folio_filler(struct nfs_readdir_descriptor *desc, {struct address_space *mapping = desc->file->f_mapping;struct folio *new, *folio = *arrays; +struct nfs_cache_array *array;struct xdr_stream stream; +u64 cookie, prev_cookie;struct page *scratch;struct xdr_buf buf; -u64 cookie;int status; scratch =alloc_page (GFP_KERNEL );
@@ -841,10 +843,20 @@ static int nfs_readdir_folio_filler(struct nfs_readdir_descriptor *desc, do { status = nfs_readdir_entry_decode(desc, entry, &stream); - if (status != 0) + if (status != 0) { + if (status == -EAGAIN && entry->cookie == cookie) { + /* Revert the bad entry */ + array = kmap_local_folio(folio, 0); + array->last_cookie = prev_cookie; + desc->last_cookie = 0; + desc->dir_cookie = 0; + array->size--; + kunmap_local(array); + } break; + } - status = nfs_readdir_folio_array_append(folio, entry, &cookie); + status = nfs_readdir_folio_array_append(folio, entry, &cookie, &prev_cookie); if (status != -ENOSPC) continue;
@@ -866,7 +878,7 @@ static int nfs_readdir_folio_filler(struct nfs_readdir_descriptor *desc, folio = new; } desc->folio_index_max++; - status =nfs_readdir_folio_array_append (folio, entry, &cookie); + status =nfs_readdir_folio_array_append (folio, entry, &cookie, &prev_cookie); }while (!status && !entry->eof);switch (status) {
Signature · 1 line
2.30.1 (Apple Git-130)
Quoted 17 lines · Yafang Shao
In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:
$ cd nfs $ ls a b c e f <<<< 'd' is missing $ cd d <<<< successI verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit
580f236737d1("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.
Quoted 5 lines
Upon further analysis, it appears that there is a bad entry being decoded in
nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.
It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?
Ben
Quoted 20 lines · Benjamin Coddington
Quoted 17 lines · Yafang Shao
In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:
$ cd nfs $ ls a b c e f <<<< 'd' is missing $ cd d <<<< successI verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit
580f236737d1("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.
Quoted 8 lines
Quoted 5 lines
Upon further analysis, it appears that there is a bad entry being decoded in
nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?
I've used tcpdump to analyze the packets but didn't find anything suspicious. Do you have any suggestions?
Interestingly, when we increase the dtsize, the issue goes away. This suggests that the problem might not be with the server itself, but rather with the NFS readdir operation. The change in dtsize is as follows,
fs/nfs/dir.c +1 -1
@@ -81,7 +81,7 @@ ctx =kzalloc (sizeof (*ctx),GFP_KERNEL_ACCOUNT );if (ctx !=NULL ) { ctx->attr_gencount = nfsi->attr_gencount; - ctx->dtsize =NFS_INIT_DTSIZE ; + ctx->dtsize =2 *NFS_INIT_DTSIZE ;// 8K spin_lock (&dir->i_lock);if (list_empty (&nfsi->open_files) && (nfsi->cache_validity &NFS_INO_DATA_INVAL_DEFER ))
--
Regards
Yafang
Quoted 27 lines · Yafang Shao
Quoted 26 lines · Benjamin Coddington
Quoted 22 lines · Yafang Shao
In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:
$ cd nfs $ ls a b c e f <<<< 'd' is missing $ cd d <<<< successI verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit
580f236737d1("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.Upon further analysis, it appears that there is a bad entry being decoded in
nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.
Quoted 5 lines
Quoted 2 lines
It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?
I've used tcpdump to analyze the packets but didn't find anything suspicious. Do you have any suggestions?
I'd check to make sure the server isn't overrunning the READDIR request's dircount and maxcount (they should be the same for the linux client). If the server isn't exceeding them, then there's a likely client bug.
Ben
Quoted 33 lines · Benjamin Coddington
Quoted 30 lines · Yafang Shao
Quoted 28 lines · Benjamin Coddington
Quoted 17 lines · Yafang Shao
In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:
$ cd nfs $ ls a b c e f <<<< 'd' is missing $ cd d <<<< successI verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit
580f236737d1("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.Quoted 5 lines
Upon further analysis, it appears that there is a bad entry being decoded in
nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?
Quoted 6 lines
Quoted 2 lines
I've used tcpdump to analyze the packets but didn't find anything suspicious. Do you have any suggestions?
I'd check to make sure the server isn't overrunning the READDIR request's dircount and maxcount (they should be the same for the linux client). If the server isn't exceeding them, then there's a likely client bug.
Thank you for the suggestion. I have captured and analyzed the NFS RPC traffic using Wireshark. I noticed that the ls command is being split into two NFS READDIR operations. In the first READDIR request, both the dircount and maxcount parameters are set to 4008. In the subsequent READDIR request, both dircount and maxcount are set to 8192.
Interestingly, when I increase the value of ctx->dtsize to 8192, the ls command now generates only a single NFS READDIR RPC call. In this case, both the dircount and maxcount parameters are set to 8104. This issue disappears as well.
--
Regards
Yafang
Quoted 33 lines · Benjamin Coddington
Quoted 30 lines · Yafang Shao
Quoted 28 lines · Benjamin Coddington
Quoted 17 lines · Yafang Shao
In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:
$ cd nfs $ ls a b c e f <<<< 'd' is missing $ cd d <<<< successI verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit
580f236737d1("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.Quoted 5 lines
Upon further analysis, it appears that there is a bad entry being decoded in
nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?
Quoted 9 lines
Quoted 2 lines
I've used tcpdump to analyze the packets but didn't find anything suspicious. Do you have any suggestions?
I'd check to make sure the server isn't overrunning the READDIR request's dircount and maxcount (they should be the same for the linux client). If the server isn't exceeding them, then there's a likely client bug.
Ben
Hello Ben,
Upon thorough examination, we have identified the root cause of the issue to lie within the NFS server, specifically its behavior of truncating file listings to match the client's READDIR RPC args->size parameter without appropriately adjusting the cookie value. After implementing a fix on the server side, the issue has been resolved. However, to enhance resilience and mitigate future server-side vulnerabilities, it may be prudent to implement client-side handling mechanisms for such issues. What do you think?
Signature · 2 lines
Regards Yafang
Quoted 48 lines · Yafang Shao
Quoted 33 lines · Benjamin Coddington
Quoted 30 lines · Yafang Shao
Quoted 28 lines · Benjamin Coddington
Quoted 17 lines · Yafang Shao
In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:
$ cd nfs $ ls a b c e f <<<< 'd' is missing $ cd d <<<< successI verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit
580f236737d1("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.Quoted 5 lines
Upon further analysis, it appears that there is a bad entry being decoded in
nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?
Quoted 9 lines
Quoted 2 lines
I've used tcpdump to analyze the packets but didn't find anything suspicious. Do you have any suggestions?
I'd check to make sure the server isn't overrunning the READDIR request's dircount and maxcount (they should be the same for the linux client). If the server isn't exceeding them, then there's a likely client bug.
Ben
Quoted 7 lines
Hello Ben,
Upon thorough examination, we have identified the root cause of the issue to lie within the NFS server, specifically its behavior of truncating file listings to match the client's READDIR RPC
args->sizeparameter without appropriately adjusting the cookie value. After implementing a fix on the server side, the issue has been resolved.
Please post your server fix on this mailing list. Thanks!
Quoted 3 lines
However, to enhance resilience and mitigate future server-side vulnerabilities, it may be prudent to implement client-side handling mechanisms for such issues. What do you think?
The general policy we follow is to avoid fixing server bugs via client-side workarounds. Fix the server in that case.
--
Chuck Lever
Quoted 54 lines · Chuck Lever III
Quoted 51 lines · Yafang Shao
Quoted 33 lines · Benjamin Coddington
Quoted 30 lines · Yafang Shao
Quoted 28 lines · Benjamin Coddington
Quoted 17 lines · Yafang Shao
In our production environment, we noticed that some files are missing when running the ls command in an NFS directory. However, we can still successfully cd into the missing directories. This issue can be illustrated as follows:
$ cd nfs $ ls a b c e f <<<< 'd' is missing $ cd d <<<< successI verified the issue with the latest upstream kernel, and it still persists. Further analysis reveals that files go missing when the dtsize is expanded. The default dtsize was reduced from 1MB to 4KB in commit
580f236737d1("NFS: Adjust the amount of readahead performed by NFS readdir"). After restoring the default size to 1MB, the issue disappears. I also tried setting the default size to 8KB, and the issue similarly disappears.Quoted 5 lines
Upon further analysis, it appears that there is a bad entry being decoded in
nfs_readdir_entry_decode(). When a bad entry is encountered, the decoding process breaks without handling the error. We should revert the bad entry in such cases. After implementing this change, the issue is resolved.It seems like you're trying to handle a server bug of some sort. Have you been able to look at a wire capture to determine why there's a bad entry?
Quoted 9 lines
Quoted 2 lines
I've used tcpdump to analyze the packets but didn't find anything suspicious. Do you have any suggestions?
I'd check to make sure the server isn't overrunning the READDIR request's dircount and maxcount (they should be the same for the linux client). If the server isn't exceeding them, then there's a likely client bug.
Ben
Hello Ben,
Quoted 7 lines
Quoted 5 lines
Upon thorough examination, we have identified the root cause of the issue to lie within the NFS server, specifically its behavior of truncating file listings to match the client's READDIR RPC
args->sizeparameter without appropriately adjusting the cookie value. After implementing a fix on the server side, the issue has been resolved.Please post your server fix on this mailing list. Thanks!
I was assuming your test server was Linux NFSD. If not, then please ignore me!
--
Chuck Lever
Quoted 7 lines · Yafang Shao
Hello Ben,
Upon thorough examination, we have identified the root cause of the issue to lie within the NFS server, specifically its behavior of truncating file listings to match the client's READDIR RPC
args->sizeparameter without appropriately adjusting the cookie value. After implementing a fix on the server side, the issue has been resolved.
Nice work! Out of curiosity, what server implemenation did you fix?
Quoted 3 lines
However, to enhance resilience and mitigate future server-side vulnerabilities, it may be prudent to implement client-side handling mechanisms for such issues. What do you think?
We have in the past modified the client to be more resilient, but usually only for cases where the server can cause the client to crash and/or corrupt data. For a bug like this, the maintainers usually assert "we do not fix the client for server bugs", since doing so can paper over protocol correctness issues created by the server. I think that's what would happen here if you keep working to justify your fix -- which you're free to do, of course!
Regards,
Ben
kernel test robot
build botkernel test robot noticed the following build warnings:
Hi Yafang,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on trondmy-nfs/linux-next]
[also build test WARNING on linus/master v6.11-rc6 next-20240830] [If your patch is applied to the wrong git tree, kindly drop us a note. And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Yafang-Shao/NFS-Fix-missing-files-in-ls-command-output/20240829-171711 base: git://git.linux-nfs.org/projects/trondmy/linux-nfs.git linux-next patch link: https://lore.kernel.org/r/20240829091340.2043-1-laoar.shao%40gmail.com patch subject: [RFC PATCH] NFS: Fix missing files in `ls` command output config: i386-randconfig-001-20240901 (https://download.01.org/0day-ci/archive/20240901/[email protected]/config) compiler: clang version 18.1.5 (https://github.com/llvm/llvm-project 617a15a9eac96088ae5e9134248d8236e34b91b1) reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240901/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> fs/nfs/dir.c:326:6: warning: variable 'cache_entry' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
326 | if (ret) {
| ^~~
fs/nfs/dir.c:345:17: note: uninitialized use occurs here
345 | *prev_cookie = cache_entry->cookie;
| ^~~~~~~~~~~
fs/nfs/dir.c:326:2: note: remove the 'if' if its condition is always false
326 | if (ret) {
| ^~~~~~~~~~
327 | kfree(name);
| ~~~~~~~~~~~~
328 | goto out;
| ~~~~~~~~~
329 | }
| ~
fs/nfs/dir.c:323:6: warning: variable 'cache_entry' is used uninitialized whenever 'if' condition is true [-Wsometimes-uninitialized]
323 | if (!name)
| ^~~~~
fs/nfs/dir.c:345:17: note: uninitialized use occurs here
345 | *prev_cookie = cache_entry->cookie;
| ^~~~~~~~~~~
fs/nfs/dir.c:323:2: note: remove the 'if' if its condition is always false
323 | if (!name)
| ^~~~~~~~~~
324 | goto out;
| ~~~~~~~~
fs/nfs/dir.c:316:43: note: initialize the variable 'cache_entry' to silence this warning
316 | struct nfs_cache_array_entry *cache_entry;
| ^
| = NULL2 warnings generated.
vim +326 fs/nfs/dir.c
d1bacf9eb2fd0e Bryan Schumaker 2010-09-24 310
ec108d3cc7663d Anna Schumaker 2023-04-04 311 static int nfs_readdir_folio_array_append(struct folio *folio,
0adf85b445c7fb Trond Myklebust 2022-02-27 312 const struct nfs_entry *entry,
b04660a1b4ac01 Yafang Shao 2024-08-29 313 u64 *cookie, u64 *prev_cookie)
d1bacf9eb2fd0e Bryan Schumaker 2010-09-24 314 {
a52a8a6adad99e Trond Myklebust 2020-11-01 315 struct nfs_cache_array *array;
4a201d6e3f4253 Trond Myklebust 2010-10-23 316 struct nfs_cache_array_entry *cache_entry;
a52a8a6adad99e Trond Myklebust 2020-11-01 317 const char *name;
0adf85b445c7fb Trond Myklebust 2022-02-27 318 int ret = -ENOMEM;
4a201d6e3f4253 Trond Myklebust 2010-10-23 319
a52a8a6adad99e Trond Myklebust 2020-11-01 320 name = nfs_readdir_copy_name(entry->name, entry->len);
3020093f578fb6 Trond Myklebust 2010-11-20 321
4b71e2416ec4cb Fabio M. De Francesco 2023-05-03 322 array = kmap_local_folio(folio, 0);
0adf85b445c7fb Trond Myklebust 2022-02-27 323 if (!name)
0adf85b445c7fb Trond Myklebust 2022-02-27 324 goto out;
b1e21c97437f64 Trond Myklebust 2020-11-01 325 ret = nfs_readdir_array_can_expand(array);
a52a8a6adad99e Trond Myklebust 2020-11-01 @326 if (ret) {
a52a8a6adad99e Trond Myklebust 2020-11-01 327 kfree(name);
4a201d6e3f4253 Trond Myklebust 2010-10-23 328 goto out;
a52a8a6adad99e Trond Myklebust 2020-11-01 329 }
d1bacf9eb2fd0e Bryan Schumaker 2010-09-24 330
b1e21c97437f64 Trond Myklebust 2020-11-01 331 cache_entry = &array->array[array->size];
0adf85b445c7fb Trond Myklebust 2022-02-27 332 cache_entry->cookie = array->last_cookie;
4a201d6e3f4253 Trond Myklebust 2010-10-23 333 cache_entry->ino = entry->ino;
0b26a0bf6ff398 Trond Myklebust 2010-11-20 334 cache_entry->d_type = entry->d_type;
a52a8a6adad99e Trond Myklebust 2020-11-01 335 cache_entry->name_len = entry->len;
a52a8a6adad99e Trond Myklebust 2020-11-01 336 cache_entry->name = name;
d1bacf9eb2fd0e Bryan Schumaker 2010-09-24 337 array->last_cookie = entry->cookie;
762567b7c798af Trond Myklebust 2020-11-04 338 if (array->last_cookie <= cache_entry->cookie)
762567b7c798af Trond Myklebust 2020-11-04 339 array->cookies_are_ordered = 0;
8cd51a0ccd1bed Trond Myklebust 2010-11-15 340 array->size++;
47c716cbf638a1 Trond Myklebust 2010-12-07 341 if (entry->eof != 0)
b1e21c97437f64 Trond Myklebust 2020-11-01 342 nfs_readdir_array_set_eof(array);
4a201d6e3f4253 Trond Myklebust 2010-10-23 343 out:
0adf85b445c7fb Trond Myklebust 2022-02-27 344 *cookie = array->last_cookie;
b04660a1b4ac01 Yafang Shao 2024-08-29 345 *prev_cookie = cache_entry->cookie;
4b71e2416ec4cb Fabio M. De Francesco 2023-05-03 346 kunmap_local(array);
4a201d6e3f4253 Trond Myklebust 2010-10-23 347 return ret;
d1bacf9eb2fd0e Bryan Schumaker 2010-09-24 348 }
d1bacf9eb2fd0e Bryan Schumaker 2010-09-24 349 Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki