'ls -lrt' performance issue on large dir while dir is being modified
Subsystems (2)
Dai Ngo
reviewerHi,
I'd like to report an issue with 'ls -lrt' on NFSv3 client takes a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.
The problem can be reproduced using 3 systems. One system serves as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
Client1 creates files using this simple script:
Quoted 10 lines
#!/bin/sh
if [ $# -lt 2 ]; then
echo "Usage: $0number_of_filesbase_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..." i=0
Quoted 6 lines
while [ i -lt $nfiles ] ;
doi=`expr $i + 1` echo "xyz" > $fname$i echo "$fname$i"done
Client2 runs 'time ls -lrt /tmp/mnt/bd1 |wc -l' in a loop.
The network traces and dtrace probes showed numerous READDIRPLUS3 requests restarting from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages to be refilled starting from cookie 0 until the current requested cookie. The cached page invalidation were tracked to nfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.
The modification is:
fs/nfs/dir.c void nfs_force_use_readdirplus(struct inode *dir) · line 449
Quoted 12 lines
fs/nfs/dir.c +1
-1@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct inode *dir)if (nfs_server_capable (dir,NFS_CAP_READDIRPLUS ) && !list_empty (&nfsi->open_files)) {set_bit (NFS_INO_ADVISE_RDPLUS , &nfsi->flags); -invalidate_mapping_pages (dir->i_mapping,0 ,-1 ); +nfs_zap_mapping (dir, dir->i_mapping); } }
Note that after this change, I did not see READDIRPLUS3 restarting with cookie 0 anymore.
Below are the summary results of 'ls -lrt'. For each kernel version to be compared, one row for the original kernel and one row for the kernel with the above modification.
I cloned dtrace-linux from here:
github.com/oracle/dtrace-linux-kernel
dtrace-linux 5.1.0-rc4 [ORI] 89191: 2m59.32s 193071: 6m7.810s
dtrace-linux 5.1.0-rc4 [MOD] 98771: 1m55.900s 191322: 3m48.668s
I cloned upstream Linux from here:
git.kernel.org/pub/scm/linux/kernel/git/tovards/linux.git
Upstream Linux 5.5.0-rc1 [ORI] 87891: 5m11.089s 160974: 14m4.384s
Upstream Linux 5.5.0-rc1 [MOD] 87075: 5m2.057s 161421: 14m33.615s
Please note that these are relative performance numbers and are used to illustrate the issue only.
For reference, on the original dtrace-linux it takes about 9s for 'ls -ltr' to complete on a directory with 200k files if the directory is not modified while 'ls' is running.
The number of the original Upstream Linux is *really* bad, and the modification did not seem to have any effect, not sure why... it could be something else is going on here.
The cache invalidation in nfs_force_use_readdirplus seems too drastic and might need to be reviewed. Even though this change helps but it did not get the 'ls' performance to where it's expected to be. I think even though READDIRPLUS3 was used, the attribute cache was invalidated due to the directory modification, causing attribute cache misses resulting in the calls to nfs_force_use_readdirplus as shown in this stack trace:
0 17586 page_cache_tree_delete:entry
vmlinux`remove_mapping+0x14
vmlinux`invalidate_inode_page+0x7c
vmlinux`invalidate_mapping_pages+0x1dd
nfs`nfs_force_use_readdirplus+0x47
nfs`__dta_nfs_lookup_revalidate_478+0x5dd
vmlinux`d_revalidate.part.24+0x10
vmlinux`lookup_fast+0x254
vmlinux`walk_component+0x49
vmlinux`path_lookupat+0x79
vmlinux`filename_lookup+0xaf
vmlinux`user_path_at_empty+0x36
vmlinux`vfs_statx+0x77
vmlinux`SYSC_newlstat+0x3d
vmlinux`SyS_newlstat+0xe
vmlinux`do_syscall_64+0x79
vmlinux`entry_SYSCALL_64+0x18dBesides the overhead of refilling the page caches from cookie 0, I think the reason 'ls' still takes so long to compete because the client has to send a bunch of additional LOOKUP/ACCESS requests over the wire to service the stat(2) calls from 'ls' due to the attribute cache misses.
Please let me know you what you think and if there is any addition information is needed.
Thanks,
-Dai
Dai Ngo
reviewerHi Anna, Trond,
I made a mistake with the 5.5 numbers. The VM that runs 5.5 has some problems. There is no regression with 5.5, here are the new numbers:
Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891: 10m35.789s Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801: 3m55.003s
My apologies for the mistake.
Now there is no regression with 5.5, I'd like to get your opinion regarding the change to revert the call from invalidate_mapping_pages to nfs_zap_mapping in nfs_force_use_readdirplus to prevent the current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm not quite sure about the intention of the prior change from nfs_zap_mapping to invalidate_mapping_pages so that is why I'm seeking advise. Or do you have any suggestions to achieve the same?
Thanks,
-Dai
Quoted 121 lines · Dai Ngo
Hi,
I'd like to report an issue with 'ls -lrt' on NFSv3 client takes a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.
The problem can be reproduced using 3 systems. One system serves as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
Client1 creates files using this simple script:Quoted 12 lines
#!/bin/sh
if [ $# -lt 2 ]; then
echo "Usage: $0number_of_filesbase_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..." i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" doneClient2 runs 'time ls -lrt
/tmp/mnt/bd1|wc -l' in a loop.The network traces and dtrace probes showed numerous READDIRPLUS3 requests restarting from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages to be refilled starting from cookie 0 until the current requested cookie. The cached page invalidation were tracked to nfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.
The modification is:
Quoted 5 lines
fs/nfs/dir.c +0
-0@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct inode *dir)Quoted 7 lines
if (
nfs_server_capable(dir, NFS_CAP_READDIRPLUS)&& !list_empty(&nfsi->open_files)) {set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -invalidate_mapping_pages(dir->i_mapping, 0, -1); +nfs_zap_mapping(dir, dir->i_mapping);
}
}Note that after this change, I did not see READDIRPLUS3 restarting with cookie 0 anymore.
Below are the summary results of 'ls -lrt'. For each kernel version to be compared, one row for the original kernel and one row for the kernel with the above modification.
I cloned dtrace-linux from here:
github.com/oracle/dtrace-linux-kerneldtrace-linux 5.1.0-rc4 [ORI] 89191: 2m59.32s 193071: 6m7.810s dtrace-linux 5.1.0-rc4 [MOD] 98771: 1m55.900s 191322: 3m48.668s
I cloned upstream Linux from here:
git.kernel.org/pub/scm/linux/kernel/git/tovards/linux.gitUpstream Linux 5.5.0-rc1 [ORI] 87891: 5m11.089s 160974: 14m4.384s Upstream Linux 5.5.0-rc1 [MOD] 87075: 5m2.057s 161421: 14m33.615s
Please note that these are relative performance numbers and are used to illustrate the issue only.
For reference, on the original dtrace-linux it takes about 9s for 'ls -ltr' to complete on a directory with 200k files if the directory is not modified while 'ls' is running.
The number of the original Upstream Linux is *really* bad, and the modification did not seem to have any effect, not sure why... it could be something else is going on here.
The cache invalidation in
nfs_force_use_readdirplusseems too drastic and might need to be reviewed. Even though this change helps but it did not get the 'ls' performance to where it's expected to be. I think even though READDIRPLUS3 was used, the attribute cache was invalidated due to the directory modification, causing attribute cache misses resulting in the calls tonfs_force_use_readdirplusas shown in this stack trace:0 17586 page_cache_tree_delete:entry vmlinux
remove_mapping+0x14 vmlinuxinvalidate_inode_page+0x7cvmlinuxinvalidate_mapping_pages+0x1dd nfsnfs_force_use_readdirplus+0x47nfs__dta_nfs_lookup_revalidate_478+0x5dd vmlinuxd_revalidate.part.24+0x10 vmlinuxlookup_fast+0x254 vmlinuxwalk_component+0x49vmlinuxpath_lookupat+0x79 vmlinuxfilename_lookup+0xafvmlinuxuser_path_at_empty+0x36 vmlinuxvfs_statx+0x77vmlinuxSYSC_newlstat+0x3d vmlinuxSyS_newlstat+0xevmlinuxdo_syscall_64+0x79 vmlinuxentry_SYSCALL_64+0x18dBesides the overhead of refilling the page caches from cookie 0, I think the reason 'ls' still takes so long to compete because the client has to send a bunch of additional LOOKUP/ACCESS requests over the wire to service the stat(2) calls from 'ls' due to the attribute cache misses.
Please let me know you what you think and if there is any addition information is needed.
Thanks,
-Dai
Dai Ngo
reviewerHi Anna, Trond,
Would you please let me know your opinion regarding reverting the change in nfs_force_use_readdirplus to call nfs_zap_mapping instead of invalidate_mapping_pages. This change is to prevent the cookie of the READDIRPLUS to be reset to 0 while an instance of 'ls' is running and the directory is being modified.
fs/nfs/dir.c index a73e2f8bd8ec..5d4a64555fa7 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct inode *dir) if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); - invalidate_mapping_pages(dir->i_mapping, 0, -1); + nfs_zap_mapping(dir, dir->i_mapping); } }
Quoted 1 line
fs/nfs/dir.c index a73e2f8bd8ec..5d4a64555fa7 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct inode *dir) if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); - invalidate_mapping_pages(dir->i_mapping, 0, -1); + nfs_zap_mapping(dir, dir->i_mapping); } } renamed from fs/nfs/dir.c +0
-0
Thanks,
-Dai
Quoted 143 lines · Dai Ngo
Hi Anna, Trond,
I made a mistake with the 5.5 numbers. The VM that runs 5.5 has some problems. There is no regression with 5.5, here are the new numbers:
Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891: 10m35.789s Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801: 3m55.003s
My apologies for the mistake.
Now there is no regression with 5.5, I'd like to get your opinion regarding the change to revert the call from
invalidate_mapping_pagestonfs_zap_mappinginnfs_force_use_readdirplusto prevent the current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm not quite sure about the intention of the prior change fromnfs_zap_mappingtoinvalidate_mapping_pagesso that is why I'm seeking advise. Or do you have any suggestions to achieve the same?Thanks,
-DaiQuoted 121 lines · Dai Ngo
Hi,
I'd like to report an issue with 'ls -lrt' on NFSv3 client takes a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.
The problem can be reproduced using 3 systems. One system serves as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
Client1 creates files using this simple script:Quoted 12 lines
#!/bin/sh
if [ $# -lt 2 ]; then
echo "Usage: $0number_of_filesbase_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..." i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" doneClient2 runs 'time ls -lrt
/tmp/mnt/bd1|wc -l' in a loop.The network traces and dtrace probes showed numerous READDIRPLUS3 requests restarting from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages to be refilled starting from cookie 0 until the current requested cookie. The cached page invalidation were tracked to nfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.
The modification is:
Quoted 5 lines
fs/nfs/dir.c +0
-0@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct inode *dir)Quoted 7 lines
if (
nfs_server_capable(dir, NFS_CAP_READDIRPLUS)&& !list_empty(&nfsi->open_files)) {set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -invalidate_mapping_pages(dir->i_mapping, 0, -1); +nfs_zap_mapping(dir, dir->i_mapping);
}
}Note that after this change, I did not see READDIRPLUS3 restarting with cookie 0 anymore.
Below are the summary results of 'ls -lrt'. For each kernel version to be compared, one row for the original kernel and one row for the kernel with the above modification.
I cloned dtrace-linux from here:
github.com/oracle/dtrace-linux-kerneldtrace-linux 5.1.0-rc4 [ORI] 89191: 2m59.32s 193071: 6m7.810s dtrace-linux 5.1.0-rc4 [MOD] 98771: 1m55.900s 191322: 3m48.668s
I cloned upstream Linux from here:
git.kernel.org/pub/scm/linux/kernel/git/tovards/linux.gitUpstream Linux 5.5.0-rc1 [ORI] 87891: 5m11.089s 160974: 14m4.384s Upstream Linux 5.5.0-rc1 [MOD] 87075: 5m2.057s 161421: 14m33.615s
Please note that these are relative performance numbers and are used to illustrate the issue only.
For reference, on the original dtrace-linux it takes about 9s for 'ls -ltr' to complete on a directory with 200k files if the directory is not modified while 'ls' is running.
The number of the original Upstream Linux is *really* bad, and the modification did not seem to have any effect, not sure why... it could be something else is going on here.
The cache invalidation in
nfs_force_use_readdirplusseems too drastic and might need to be reviewed. Even though this change helps but it did not get the 'ls' performance to where it's expected to be. I think even though READDIRPLUS3 was used, the attribute cache was invalidated due to the directory modification, causing attribute cache misses resulting in the calls tonfs_force_use_readdirplusas shown in this stack trace:0 17586 page_cache_tree_delete:entry vmlinux
remove_mapping+0x14 vmlinuxinvalidate_inode_page+0x7cvmlinuxinvalidate_mapping_pages+0x1dd nfsnfs_force_use_readdirplus+0x47nfs__dta_nfs_lookup_revalidate_478+0x5dd vmlinuxd_revalidate.part.24+0x10 vmlinuxlookup_fast+0x254 vmlinuxwalk_component+0x49vmlinuxpath_lookupat+0x79 vmlinuxfilename_lookup+0xafvmlinuxuser_path_at_empty+0x36 vmlinuxvfs_statx+0x77vmlinuxSYSC_newlstat+0x3d vmlinuxSyS_newlstat+0xevmlinuxdo_syscall_64+0x79 vmlinuxentry_SYSCALL_64+0x18dBesides the overhead of refilling the page caches from cookie 0, I think the reason 'ls' still takes so long to compete because the client has to send a bunch of additional LOOKUP/ACCESS requests over the wire to service the stat(2) calls from 'ls' due to the attribute cache misses.
Please let me know you what you think and if there is any addition information is needed.
Thanks,
-Dai
Quoted 88 lines · Dai Ngo
Hi Anna, Trond,
Would you please let me know your opinion regarding reverting the change in
nfs_force_use_readdirplusto callnfs_zap_mappinginstead ofinvalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be reset to 0 while
an instance of 'ls' is running and the directory is being modified.Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0
-0
a73e2f8bd8ec..5d4a64555fa7100644 ---a/fs/nfs/dir.c+++b/fs/nfs/dir.c@@ -444,7 +444,7 @@ voidnfs_force_use_readdirplus(struct inode *dir)if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS)&& !list_empty(&nfsi->open_files)) {set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -invalidate_mapping_pages(dir->i_mapping, 0, -1); +nfs_zap_mapping(dir, dir->i_mapping); } }Thanks,
-DaiQuoted 64 lines · Dai Ngo
Hi Anna, Trond,
I made a mistake with the 5.5 numbers. The VM that runs 5.5 has some
problems. There is no regression with 5.5, here are the new numbers:Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891: 10m35.789s Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801: 3m55.003s
My apologies for the mistake.
Now there is no regression with 5.5, I'd like to get your opinion regarding the change to revert the call from
invalidate_mapping_pages
tonfs_zap_mappinginnfs_force_use_readdirplusto prevent the current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm not quite sure about the intention of the prior change fromnfs_zap_mappingtoinvalidate_mapping_pagesso that is why I'm seeking advise. Or do you have any suggestions to achieve the same?Thanks,
-DaiQuoted 39 lines · Dai Ngo
Hi,
I'd like to report an issue with 'ls -lrt' on NFSv3 client takes a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.
The problem can be reproduced using 3 systems. One system serves as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
Client1 creates files using this simple script:Quoted 12 lines
#!/bin/sh
if [ $# -lt 2 ]; then
echo "Usage: $0number_of_filesbase_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" doneClient2 runs 'time ls -lrt
/tmp/mnt/bd1|wc -l' in a loop.The network traces and dtrace probes showed numerous READDIRPLUS3 requests restarting from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages to be refilled starting from cookie 0 until the current requested cookie. The cached page invalidation were tracked to
nfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.The modification is:
Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
fs/nfs/dir.c +0
-0@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct inodeQuoted 8 lines
Quoted 8 lines
fs/nfs/dir.c void nfs_force_use_readdirplus(struct inode · line 450
Quoted 8 lines
fs/nfs/dir.c +1
-1*dir)if (nfs_server_capable (dir,NFS_CAP_READDIRPLUS ) && !list_empty (&nfsi->open_files)) {set_bit (NFS_INO_ADVISE_RDPLUS , &nfsi->flags); -invalidate_mapping_pages (dir->i_mapping,0 ,-1 ); +nfs_zap_mapping (dir, dir->i_mapping); } }
This change is only reverting part of commit 79f687a3de9e. My problem with that is as follows:
RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must match those returned by previous READDIRPLUS calls, and READDIR cookies and verifiers must match those returned by previous READDIR calls. It says nothing about being able to assume cookies from READDIR and READDIRPLUS calls are interchangeable. So the only reason I can see for the invalidate_mapping_pages() is to ensure that we do separate the two cookie caches.
OTOH, for NFSv4, there is no separate READDIRPLUS function, so there really does not appear to be any reason to clear the page cache at all as we're switching between requesting attributes or not.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 126 lines · Trond Myklebust
Quoted 96 lines · Dai Ngo
Hi Anna, Trond,
Would you please let me know your opinion regarding reverting the change in
nfs_force_use_readdirplusto callnfs_zap_mappinginstead ofinvalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be reset to 0 while
an instance of 'ls' is running and the directory is being modified.Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0
-0
a73e2f8bd8ec..5d4a64555fa7100644 ---a/fs/nfs/dir.c+++b/fs/nfs/dir.c@@ -444,7 +444,7 @@ voidnfs_force_use_readdirplus(struct inode *dir)if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS)&& !list_empty(&nfsi->open_files)) {set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -invalidate_mapping_pages(dir->i_mapping, 0, -1); +nfs_zap_mapping(dir, dir->i_mapping); } }Thanks,
-DaiQuoted 72 lines · Dai Ngo
Hi Anna, Trond,
I made a mistake with the 5.5 numbers. The VM that runs 5.5 has some
problems. There is no regression with 5.5, here are the new numbers:Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003sMy apologies for the mistake.
Now there is no regression with 5.5, I'd like to get your opinion regarding the change to revert the call from
invalidate_mapping_pages
tonfs_zap_mappinginnfs_force_use_readdirplusto prevent the current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm not quite sure about the intention of the prior change fromnfs_zap_mappingtoinvalidate_mapping_pagesso that is why I'm seeking advise. Or do you have any suggestions to achieve the same?Thanks,
-DaiQuoted 44 lines · Dai Ngo
Hi,
I'd like to report an issue with 'ls -lrt' on NFSv3 client takes
a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.The problem can be reproduced using 3 systems. One system serves
as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
Client1 creates files using this simple script:Quoted 12 lines
#!/bin/sh
if [ $# -lt 2 ]; then
echo "Usage: $0number_of_filesbase_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" doneClient2 runs 'time ls -lrt
/tmp/mnt/bd1|wc -l' in a loop.The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked tonfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.The modification is:
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
fs/nfs/dir.c +0
-0@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(structinode
Quoted 8 lines
Quoted 8 lines
fs/nfs/dir.c
Quoted 8 lines
*dir)
if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);fs/nfs/dir.c +1
-1-invalidate_mapping_pages (dir->i_mapping,0 ,-1 ); +nfs_zap_mapping (dir, dir->i_mapping); } }This change is only reverting part of commit
79f687a3de9e. My problem with that is as follows:RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must match
those returned by previous READDIRPLUS calls, and READDIR cookies and verifiers must match those returned by previous READDIR calls. It says
nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for theinvalidate_mapping_pages()is to ensure that we do separate the two cookie caches.
Quoted 5 lines
OTOH, for NFSv4, there is no separate READDIRPLUS function, so there really does not appear to be any reason to clear the page cache at all
as we're switching between requesting attributes or not.
Sorry... To spell out my objection to this change more clearly: The call to nfs_zap_mapping() makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3 concern.
* It would appear to be entirely superfluous for the NFSv4 case.
So a change that might be acceptable would be to keep the existing call to invalidate_mapping_pages() for NFSv3, but to remove it for NFSv4.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Dai Ngo
reviewerQuoted 129 lines · Trond Myklebust
Quoted 123 lines · Trond Myklebust
Quoted 94 lines · Dai Ngo
Hi Anna, Trond,
Would you please let me know your opinion regarding reverting the change in
nfs_force_use_readdirplusto callnfs_zap_mappinginstead ofinvalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be reset to 0 while
an instance of 'ls' is running and the directory is being modified.Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0
-0
a73e2f8bd8ec..5d4a64555fa7100644 ---a/fs/nfs/dir.c+++b/fs/nfs/dir.c@@ -444,7 +444,7 @@ voidnfs_force_use_readdirplus(struct inode *dir)if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS)&& !list_empty(&nfsi->open_files)) {set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -invalidate_mapping_pages(dir->i_mapping, 0, -1); +nfs_zap_mapping(dir, dir->i_mapping); } }Thanks,
-DaiQuoted 71 lines · Dai Ngo
Hi Anna, Trond,
I made a mistake with the 5.5 numbers. The VM that runs 5.5 has some
problems. There is no regression with 5.5, here are the new numbers:Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003sMy apologies for the mistake.
Now there is no regression with 5.5, I'd like to get your opinion regarding the change to revert the call from
invalidate_mapping_pages
tonfs_zap_mappinginnfs_force_use_readdirplusto prevent the current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm not quite sure about the intention of the prior change fromnfs_zap_mappingtoinvalidate_mapping_pagesso that is why I'm seeking advise. Or do you have any suggestions to achieve the same?Thanks,
-DaiQuoted 43 lines · Dai Ngo
Hi,
I'd like to report an issue with 'ls -lrt' on NFSv3 client takes
a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.The problem can be reproduced using 3 systems. One system serves
as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
Client1 creates files using this simple script:Quoted 12 lines
#!/bin/sh
if [ $# -lt 2 ]; then
echo "Usage: $0number_of_filesbase_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" doneClient2 runs 'time ls -lrt
/tmp/mnt/bd1|wc -l' in a loop.The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked tonfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.The modification is:
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
fs/nfs/dir.c +0
-0@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(structinode
Quoted 8 lines
Quoted 8 lines
fs/nfs/dir.c
Quoted 8 lines
*dir)
if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);fs/nfs/dir.c +1
-1-invalidate_mapping_pages (dir->i_mapping,0 ,-1 ); +nfs_zap_mapping (dir, dir->i_mapping); } }This change is only reverting part of commit
79f687a3de9e. My problem with that is as follows:RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must match
those returned by previous READDIRPLUS calls, and READDIR cookies and verifiers must match those returned by previous READDIR calls. It says
nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for theinvalidate_mapping_pages()is to ensure that we do separate the two cookie caches.Quoted 5 lines
OTOH, for NFSv4, there is no separate READDIRPLUS function, so there really does not appear to be any reason to clear the page cache at all
as we're switching between requesting attributes or not.
Quoted 8 lines
Sorry... To spell out my objection to this change more clearly: The call to
nfs_zap_mapping()makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3 concern.
* It would appear to be entirely superfluous for the NFSv4 case.So a change that might be acceptable would be to keep the existing call to
invalidate_mapping_pages()for NFSv3, but to remove it for NFSv4.
Thank you Trond, I'll make your suggested change, test it and resubmit.
-Dai
Quoted 1 line
Dai Ngo
reviewerHi Trond,
Quoted 113 lines · Trond Myklebust
Quoted 112 lines · Trond Myklebust
Quoted 94 lines · Dai Ngo
Hi Anna, Trond,
Would you please let me know your opinion regarding reverting the change in
nfs_force_use_readdirplusto callnfs_zap_mappinginstead ofinvalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be reset to 0 while
an instance of 'ls' is running and the directory is being modified.Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0
-0
a73e2f8bd8ec..5d4a64555fa7100644 ---a/fs/nfs/dir.c+++b/fs/nfs/dir.c@@ -444,7 +444,7 @@ voidnfs_force_use_readdirplus(struct inode *dir)if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS)&& !list_empty(&nfsi->open_files)) {set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -invalidate_mapping_pages(dir->i_mapping, 0, -1); +nfs_zap_mapping(dir, dir->i_mapping); } }Thanks,
-DaiQuoted 71 lines · Dai Ngo
Hi Anna, Trond,
I made a mistake with the 5.5 numbers. The VM that runs 5.5 has some
problems. There is no regression with 5.5, here are the new numbers:Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003sMy apologies for the mistake.
Now there is no regression with 5.5, I'd like to get your opinion regarding the change to revert the call from
invalidate_mapping_pages
tonfs_zap_mappinginnfs_force_use_readdirplusto prevent the current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm not quite sure about the intention of the prior change fromnfs_zap_mappingtoinvalidate_mapping_pagesso that is why I'm seeking advise. Or do you have any suggestions to achieve the same?Thanks,
-DaiQuoted 43 lines · Dai Ngo
Hi,
I'd like to report an issue with 'ls -lrt' on NFSv3 client takes
a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.The problem can be reproduced using 3 systems. One system serves
as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
Client1 creates files using this simple script:Quoted 12 lines
#!/bin/sh
if [ $# -lt 2 ]; then
echo "Usage: $0number_of_filesbase_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" doneClient2 runs 'time ls -lrt
/tmp/mnt/bd1|wc -l' in a loop.The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked tonfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.The modification is:
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
fs/nfs/dir.c +0
-0@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(structinode
Quoted 8 lines
Quoted 8 lines
fs/nfs/dir.c
Quoted 8 lines
*dir)
if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);fs/nfs/dir.c +1
-1-invalidate_mapping_pages (dir->i_mapping,0 ,-1 ); +nfs_zap_mapping (dir, dir->i_mapping); } }This change is only reverting part of commit
79f687a3de9e. My problem with that is as follows:
Quoted 10 lines
Quoted 10 lines
RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must match
those returned by previous READDIRPLUS calls, and READDIR cookies and verifiers must match those returned by previous READDIR calls. It says
nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for theinvalidate_mapping_pages()is to ensure that we do separate the two cookie caches.
If I understand your concern correctly that in NFSv3 the client must maintain valid cookies and cookie verifiers when switching between READDIR and READDIRPLUS, or vice sersa, then I think the current client code handles this condition ok.
On the client, both READDIR and READDIRPLUS requests use the cookie values from the same cached pages of the directory so I don't think they can be out of sync when the client switches between READDIRPLUS and READDIR requests for different nfs_readdir calls.
In fact, currently the first nfs_readdir uses READDIRPLUS's to fill read the entries and if there is no LOOKUP/GETATTR on one of the directory entries then the client reverts to READDIR's for subsequent nfs_readdir calls without invalidating any cached pages of the directory. If there are LOOKUP/GETATTR done on one of the directory entries then nfs_advise_use_readdirplus is called which forces the client to use READDIRPLUS again for the next nfs_readdir.
Unless the user mounts the export with 'nordirplus' option then the client uses only READDIRs for all requests, no switching takes place.
Thanks,
-Dai
Quoted 15 lines
Quoted 6 lines
OTOH, for NFSv4, there is no separate READDIRPLUS function, so there really does not appear to be any reason to clear the page cache at all
as we're switching between requesting attributes or not.Sorry... To spell out my objection to this change more clearly: The call to
nfs_zap_mapping()makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3 concern.
* It would appear to be entirely superfluous for the NFSv4 case.So a change that might be acceptable would be to keep the existing call to
invalidate_mapping_pages()for NFSv3, but to remove it for NFSv4.
Quoted 167 lines · Dai Ngo
Hi Trond,
Quoted 127 lines · Trond Myklebust
Quoted 126 lines · Trond Myklebust
Quoted 107 lines · Dai Ngo
Hi Anna, Trond,
Would you please let me know your opinion regarding reverting the
change innfs_force_use_readdirplusto callnfs_zap_mappinginstead ofinvalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be reset
to 0 while
an instance of 'ls' is running and the directory is being modified.Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0
-0
a73e2f8bd8ec..5d4a64555fa7100644 ---a/fs/nfs/dir.c+++b/fs/nfs/dir.c@@ -444,7 +444,7 @@ voidnfs_force_use_readdirplus(struct inode *dir)if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS)&& !list_empty(&nfsi->open_files)) {set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -invalidate_mapping_pages(dir->i_mapping, 0, -1); +nfs_zap_mapping(dir, dir->i_mapping); } }Thanks,
-DaiQuoted 81 lines · Dai Ngo
Hi Anna, Trond,
I made a mistake with the 5.5 numbers. The VM that runs 5.5 has
some
problems. There is no regression with 5.5, here are the new numbers:Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003sMy apologies for the mistake.
Now there is no regression with 5.5, I'd like to get your opinion
regarding the change to revert the call frominvalidate_mapping_pages
tonfs_zap_mappinginnfs_force_use_readdirplusto prevent the
current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm
not quite sure about the intention of the prior change fromnfs_zap_mappingtoinvalidate_mapping_pagesso that is why I'm
seeking advise. Or do you have any suggestions to achieve the same?Thanks,
-DaiQuoted 48 lines · Dai Ngo
Hi,
I'd like to report an issue with 'ls -lrt' on NFSv3 client takes
a very long time to display the content of a large directory
(100k - 200k files) while the directory is being modified by
another NFSv3 client.The problem can be reproduced using 3 systems. One system serves
as the NFS server, one system runs as the client that doing the
'ls -lrt' and another system runs the client that creates files
on the server.
Client1 creates files using this simple script:Quoted 12 lines
#!/bin/sh
if [ $# -lt 2 ]; then
echo "Usage: $0number_of_filesbase_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" doneClient2 runs 'time ls -lrt
/tmp/mnt/bd1|wc -l' in a loop.The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate the
cached pages of the directory were invalidated causing the pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked tonfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.The modification is:
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
fs/nfs/dir.c +0
-0@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(structinode
Quoted 8 lines
Quoted 8 lines
fs/nfs/dir.c
Quoted 8 lines
*dir)
if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);fs/nfs/dir.c +1
-1-invalidate_mapping_pages (dir->i_mapping,0 ,-1 ); +nfs_zap_mapping (dir, dir->i_mapping); } }This change is only reverting part of commit
79f687a3de9e. My problem
with that is as follows:Quoted 12 lines
Quoted 4 lines
RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must match
those returned by previous READDIRPLUS calls, and READDIR cookies andQuoted 8 lines
verifiers must match those returned by previous READDIR calls. It says
nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for theinvalidate_mapping_pages()is to ensure that we do separate the two
cookie caches.If I understand your concern correctly that in NFSv3 the client must maintain valid cookies and cookie verifiers when switching between READDIR and READDIRPLUS, or vice sersa, then I think the current client
code handles this condition ok.On the client, both READDIR and READDIRPLUS requests use the cookie values
from the same cached pages of the directory so I don't think they can be
out of sync when the client switches between READDIRPLUS and READDIR requests for differentnfs_readdircalls.In fact, currently the first
nfs_readdiruses READDIRPLUS's to fill read
the entries and if there is no LOOKUP/GETATTR on one of the directory entries then the client reverts to READDIR's for subsequentnfs_readdir
calls without invalidating any cached pages of the directory. If there
are LOOKUP/GETATTR done on one of the directory entries thennfs_advise_use_readdirplusis called which forces the client to use READDIRPLUS again for the nextnfs_readdir.
Quoted 2 lines
Unless the user mounts the export with 'nordirplus' option then the client uses only READDIRs for all requests, no switching takes place.
I don't understand your point. The issue is that nfs_advise_use_readdirplus() can cause the behaviour to switch between use of READDIRPLUS and use of READDIR from one syscall to getdents() to the next.
If the client is using the same page cache, across those syscalls, then it will end up caching a mixture of cookies. Furthermore, since the cookie that is used as an argument to the next call to READDIR/READDIRPLUS is taken from that page cache, then we can end up calling READDIRPLUS with a cookie that came from READDIR and vice versa.
As I said, I'm not convinced that is legal in RFC1813 (NFSv3).
That is why we want to clear the page cache when we swap between use of READDIR and use of READDIRPLUS for the case of NFSv3.
Quoted 23 lines
Thanks,
-DaiQuoted 19 lines
Quoted 7 lines
OTOH, for NFSv4, there is no separate READDIRPLUS function, so there
really does not appear to be any reason to clear the page cache at
all
as we're switching between requesting attributes or not.Sorry... To spell out my objection to this change more clearly: The call to
nfs_zap_mapping()makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3
concern.
* It would appear to be entirely superfluous for the NFSv4 case.So a change that might be acceptable would be to keep the existing call
toinvalidate_mapping_pages()for NFSv3, but to remove it for NFSv4.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
Quoted 172 lines · Trond Myklebust
Quoted 167 lines · Dai Ngo
Hi Trond,
Quoted 127 lines · Trond Myklebust
Quoted 126 lines · Trond Myklebust
Quoted 107 lines · Dai Ngo
Hi Anna, Trond,
Would you please let me know your opinion regarding reverting the
change innfs_force_use_readdirplusto callnfs_zap_mappinginstead ofinvalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be reset
to 0 while
an instance of 'ls' is running and the directory is being modified.Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0
-0
a73e2f8bd8ec..5d4a64555fa7100644 ---a/fs/nfs/dir.c+++b/fs/nfs/dir.c@@ -444,7 +444,7 @@ voidnfs_force_use_readdirplus(struct inode *dir)if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS)&& !list_empty(&nfsi->open_files)) {set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -invalidate_mapping_pages(dir->i_mapping, 0, -1); +nfs_zap_mapping(dir, dir->i_mapping); } }Thanks,
-DaiQuoted 81 lines · Dai Ngo
Hi Anna, Trond,
I made a mistake with the 5.5 numbers. The VM that runs 5.5 has
some
problems. There is no regression with 5.5, here are the new numbers:Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003sMy apologies for the mistake.
Now there is no regression with 5.5, I'd like to get your opinion
regarding the change to revert the call frominvalidate_mapping_pages
tonfs_zap_mappinginnfs_force_use_readdirplusto prevent the
current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm
not quite sure about the intention of the prior change fromnfs_zap_mappingtoinvalidate_mapping_pagesso that is why I'm
seeking advise. Or do you have any suggestions to achieve the same?Thanks,
-DaiQuoted 48 lines · Dai Ngo
Hi,
I'd like to report an issue with 'ls -lrt' on NFSv3 client takes
a very long time to display the content of a large directory
(100k - 200k files) while the directory is being modified by
another NFSv3 client.The problem can be reproduced using 3 systems. One system serves
as the NFS server, one system runs as the client that doing the
'ls -lrt' and another system runs the client that creates files
on the server.
Client1 creates files using this simple script:Quoted 12 lines
#!/bin/sh
if [ $# -lt 2 ]; then
echo "Usage: $0number_of_filesbase_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" doneClient2 runs 'time ls -lrt
/tmp/mnt/bd1|wc -l' in a loop.The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate the
cached pages of the directory were invalidated causing the pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked tonfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.The modification is:
Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
fs/nfs/dir.c +0
-0@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(structinode
Quoted 8 lines
Quoted 8 lines
fs/nfs/dir.c
Quoted 8 lines
*dir)
if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);fs/nfs/dir.c +1
-1-invalidate_mapping_pages (dir->i_mapping,0 ,-1 ); +nfs_zap_mapping (dir, dir->i_mapping); } }This change is only reverting part of commit
79f687a3de9e. My problem
with that is as follows:Quoted 12 lines
Quoted 4 lines
RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must match
those returned by previous READDIRPLUS calls, and READDIR cookies andQuoted 8 lines
verifiers must match those returned by previous READDIR calls. It says
nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for theinvalidate_mapping_pages()is to ensure that we do separate the two
cookie caches.If I understand your concern correctly that in NFSv3 the client must maintain valid cookies and cookie verifiers when switching between READDIR and READDIRPLUS, or vice sersa, then I think the current client
code handles this condition ok.On the client, both READDIR and READDIRPLUS requests use the cookie values
from the same cached pages of the directory so I don't think they can be
out of sync when the client switches between READDIRPLUS and READDIR requests for differentnfs_readdircalls.In fact, currently the first
nfs_readdiruses READDIRPLUS's to fill read
the entries and if there is no LOOKUP/GETATTR on one of the directory entries then the client reverts to READDIR's for subsequentnfs_readdir
calls without invalidating any cached pages of the directory. If there
are LOOKUP/GETATTR done on one of the directory entries thennfs_advise_use_readdirplusis called which forces the client to use READDIRPLUS again for the nextnfs_readdir.
Quoted 5 lines
Quoted 2 lines
Unless the user mounts the export with 'nordirplus' option then the client uses only READDIRs for all requests, no switching takes place.
I don't understand your point.
The original point was that the directory's page cache seems to be cleared a little too often (quite apart from switching between READDIRPLUS and READDIR).
I think Dai is saying that cache clearing is appropriate to defer when the directory's mtime has changed but the READ method remains the same. Otherwise repeatedly adding a new file to a very large directory that is being read can trigger a situation where the reading getdents loop never completes.
My two cents Euro.
Quoted 11 lines
The issue is that
nfs_advise_use_readdirplus()can cause the behaviour to switch between use of READDIRPLUS and use of READDIR from one syscall togetdents()to the next.
If the client is using the same page cache, across those syscalls, then it will end up caching a mixture of cookies. Furthermore, since the cookie that is used as an argument to the next call to READDIR/READDIRPLUS is taken from that page cache, then we can end up calling READDIRPLUS with a cookie that came from READDIR and vice versa.
Quoted 4 lines
As I said, I'm not convinced that is legal in RFC1813 (NFSv3).
That is why we want to clear the page cache when we swap between use of READDIR and use of READDIRPLUS for the case of NFSv3.
Just curious, are you aware of an NFSv3 server implementation that would have a problem with a client that mixes the cookies?
Quoted 17 lines
Quoted 17 lines
Thanks,
-DaiQuoted 14 lines
Quoted 7 lines
OTOH, for NFSv4, there is no separate READDIRPLUS function, so there
really does not appear to be any reason to clear the page cache at
all
as we're switching between requesting attributes or not.Sorry... To spell out my objection to this change more clearly: The call to
nfs_zap_mapping()makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3
concern.
* It would appear to be entirely superfluous for the NFSv4 case.
Quoted 9 lines
Quoted 5 lines
Quoted 5 lines
So a change that might be acceptable would be to keep the existing call
toinvalidate_mapping_pages()for NFSv3, but to remove it for NFSv4.Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]
--
Chuck Lever
Quoted 221 lines · Chuck Lever
Quoted 210 lines
On Jan 18, 2020, at 10:58 AM, Trond Myklebust <
[email protected]> wrote:Quoted 206 lines · Dai Ngo
Hi Trond,
Quoted 162 lines · Trond Myklebust
Quoted 161 lines · Trond Myklebust
Quoted 28 lines · Dai Ngo
Hi Anna, Trond,
Would you please let me know your opinion regarding reverting
the
change innfs_force_use_readdirplusto callnfs_zap_mappinginstead ofinvalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be
reset
to 0 while
an instance of 'ls' is running and the directory is being modified.Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0
-0
a73e2f8bd8ec..5d4a64555fa7100644 ---a/fs/nfs/dir.c+++b/fs/nfs/dir.c@@ -444,7 +444,7 @@ voidnfs_force_use_readdirplus(struct inode *dir)if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS)&& !list_empty(&nfsi->open_files)) {set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -invalidate_mapping_pages(dir->i_mapping, 0, -1); +nfs_zap_mapping(dir, dir->i_mapping); } }Thanks,
-DaiQuoted 119 lines
Quoted 118 lines · Dai Ngo
Hi Anna, Trond,
I made a mistake with the 5.5 numbers. The VM that runs 5.5
has
some
problems. There is no regression with 5.5, here are the new
numbers:Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003sMy apologies for the mistake.
Now there is no regression with 5.5, I'd like to get your opinion
regarding the change to revert the call frominvalidate_mapping_pages
tonfs_zap_mappinginnfs_force_use_readdirplusto prevent
the
current 'ls' from restarting the READDIRPLUS3 from cookie 0.
I'm
not quite sure about the intention of the prior change fromnfs_zap_mappingtoinvalidate_mapping_pagesso that is why
I'm
seeking advise. Or do you have any suggestions to achieve the
same?Thanks,
-DaiQuoted 78 lines · Dai Ngo
Hi,
I'd like to report an issue with 'ls -lrt' on NFSv3 client
takes
a very long time to display the content of a large directory
(100k - 200k files) while the directory is being modified
by
another NFSv3 client.The problem can be reproduced using 3 systems. One system
serves
as the NFS server, one system runs as the client that doing
the
'ls -lrt' and another system runs the client that creates
files
on the server.
Client1 creates files using this simple script:Quoted 14 lines
#!/bin/sh
if [ $# -lt 2 ]; then
echo "Usage: $0number_of_filesbase_filename"
exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using
filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" doneClient2 runs 'time ls -lrt
/tmp/mnt/bd1|wc -l' in a loop.The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate
the
cached pages of the directory were invalidated causing the
pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked tonfs_force_use_readdirplus(). To verify, I made the below
modification, ran the test for various kernel versions and
captured the results shown below.The modification is:
Quoted 18 lines
fs/nfs/dir.c +0
-0@@ -444,7 +444,7 @@ voidnfs_force_use_readdirplus(struct
inode
*dir)
if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS)
&&!list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi-Quoted 1 line
flags);
fs/nfs/dir.c +1
-2-invalidate_mapping_pages (dir->i_mapping,0 , -1 ); +nfs_zap_mapping (dir, dir->i_mapping); } }This change is only reverting part of commit
79f687a3de9e. My problem
with that is as follows:RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must
match
those returned by previous READDIRPLUS calls, and READDIR cookies
and
verifiers must match those returned by previous READDIR calls. It
saysQuoted 8 lines
Quoted 8 lines
nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for theinvalidate_mapping_pages()is to ensure that we do separate the
two
cookie caches.If I understand your concern correctly that in NFSv3 the client must
maintain valid cookies and cookie verifiers when switching between
READDIR and READDIRPLUS, or vice sersa, then I think the current client
code handles this condition ok.On the client, both READDIR and READDIRPLUS requests use the cookie
values
from the same cached pages of the directory so I don't think they can
be
out of sync when the client switches between READDIRPLUS and READDIR
requests for differentnfs_readdircalls.In fact, currently the first
nfs_readdiruses READDIRPLUS's to fill
read
the entries and if there is no LOOKUP/GETATTR on one of the directory
entries then the client reverts to READDIR's for subsequentnfs_readdir
calls without invalidating any cached pages of the directory. If there
are LOOKUP/GETATTR done on one of the directory entries thennfs_advise_use_readdirplusis called which forces the client to use
READDIRPLUS again for the nextnfs_readdir.Quoted 6 lines
Quoted 4 lines
Unless the user mounts the export with 'nordirplus' option then the
client uses only READDIRs for all requests, no switching takes place.I don't understand your point.
The original point was that the directory's page cache seems to be cleared a little too often (quite apart from switching between READDIRPLUS and READDIR).
Quoted 6 lines
I think Dai is saying that cache clearing is appropriate to defer when the directory's mtime has changed but the READ method remains the same. Otherwise repeatedly adding a new file to a very large directory that is being read can trigger a situation where the reading getdents loop never completes.
Fair enough, but the section of code he was touching with his patch should be only about switching between READDIR/PLUS methods.
The mtime monitoring happens elsewhere and is already set up to invalidate the cache only on opendir()/rewinddir().
Quoted 65 lines
My two cents Euro.
Quoted 15 lines
The issue is that
nfs_advise_use_readdirplus()can cause the behaviour to switch between
use of READDIRPLUS and use of READDIR from one syscall togetdents()to
the next.
If the client is using the same page cache, across those syscalls, then
it will end up caching a mixture of cookies. Furthermore, since the cookie that is used as an argument to the next call to READDIR/READDIRPLUS is taken from that page cache, then we can end up
calling READDIRPLUS with a cookie that came from READDIR and vice versa.Quoted 5 lines
As I said, I'm not convinced that is legal in RFC1813 (NFSv3).
That is why we want to clear the page cache when we swap between use of
READDIR and use of READDIRPLUS for the case of NFSv3.Just curious, are you aware of an NFSv3 server implementation that would have a problem with a client that mixes the cookies?
Quoted 21 lines
Quoted 21 lines
Thanks,
-DaiQuoted 18 lines
Quoted 9 lines
OTOH, for NFSv4, there is no separate READDIRPLUS function, so
there
really does not appear to be any reason to clear the page cache
at
all
as we're switching between requesting attributes or not.Sorry... To spell out my objection to this change more clearly:
The
call tonfs_zap_mapping()makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3
concern.
* It would appear to be entirely superfluous for the NFSv4
case.Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
So a change that might be acceptable would be to keep the existing
call
toinvalidate_mapping_pages()for NFSv3, but to remove it for NFSv4.Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]--
Chuck Lever
Signature · 5 lines
Trond Myklebust CTO, Hammerspace Inc 4300 El Camino Real, Suite 105 Los Altos, CA 94022 www.hammer.space
Dai Ngo
reviewerQuoted 219 lines · Trond Myklebust
Quoted 218 lines · Chuck Lever
Quoted 209 lines
On Jan 18, 2020, at 10:58 AM, Trond Myklebust <
[email protected]> wrote:Quoted 205 lines · Dai Ngo
Hi Trond,
Quoted 162 lines · Trond Myklebust
Quoted 161 lines · Trond Myklebust
Quoted 28 lines · Dai Ngo
Hi Anna, Trond,
Would you please let me know your opinion regarding reverting
the
change innfs_force_use_readdirplusto callnfs_zap_mappinginstead ofinvalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be
reset
to 0 while
an instance of 'ls' is running and the directory is being modified.Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0
-0
a73e2f8bd8ec..5d4a64555fa7100644 ---a/fs/nfs/dir.c+++b/fs/nfs/dir.c@@ -444,7 +444,7 @@ voidnfs_force_use_readdirplus(struct inode *dir)if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS)&& !list_empty(&nfsi->open_files)) {set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -invalidate_mapping_pages(dir->i_mapping, 0, -1); +nfs_zap_mapping(dir, dir->i_mapping); } }Thanks,
-DaiQuoted 119 lines
Quoted 118 lines · Dai Ngo
Hi Anna, Trond,
I made a mistake with the 5.5 numbers. The VM that runs 5.5
has
some
problems. There is no regression with 5.5, here are the new
numbers:Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003sMy apologies for the mistake.
Now there is no regression with 5.5, I'd like to get your opinion
regarding the change to revert the call frominvalidate_mapping_pages
tonfs_zap_mappinginnfs_force_use_readdirplusto prevent
the
current 'ls' from restarting the READDIRPLUS3 from cookie 0.
I'm
not quite sure about the intention of the prior change fromnfs_zap_mappingtoinvalidate_mapping_pagesso that is why
I'm
seeking advise. Or do you have any suggestions to achieve the
same?Thanks,
-DaiQuoted 78 lines · Dai Ngo
Hi,
I'd like to report an issue with 'ls -lrt' on NFSv3 client
takes
a very long time to display the content of a large directory
(100k - 200k files) while the directory is being modified
by
another NFSv3 client.The problem can be reproduced using 3 systems. One system
serves
as the NFS server, one system runs as the client that doing
the
'ls -lrt' and another system runs the client that creates
files
on the server.
Client1 creates files using this simple script:Quoted 14 lines
#!/bin/sh
if [ $# -lt 2 ]; then
echo "Usage: $0number_of_filesbase_filename"
exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using
filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" doneClient2 runs 'time ls -lrt
/tmp/mnt/bd1|wc -l' in a loop.The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate
the
cached pages of the directory were invalidated causing the
pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked tonfs_force_use_readdirplus(). To verify, I made the below
modification, ran the test for various kernel versions and
captured the results shown below.The modification is:
Quoted 18 lines
fs/nfs/dir.c +0
-0@@ -444,7 +444,7 @@ voidnfs_force_use_readdirplus(struct
inode
*dir)
if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS)
&&!list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi-Quoted 1 line
flags);
fs/nfs/dir.c +1
-2-invalidate_mapping_pages (dir->i_mapping,0 , -1 ); +nfs_zap_mapping (dir, dir->i_mapping); } }This change is only reverting part of commit
79f687a3de9e. My problem
with that is as follows:RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must
match
those returned by previous READDIRPLUS calls, and READDIR cookies
and
verifiers must match those returned by previous READDIR calls. It
saysQuoted 8 lines
Quoted 8 lines
nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for theinvalidate_mapping_pages()is to ensure that we do separate the
two
cookie caches.If I understand your concern correctly that in NFSv3 the client must
maintain valid cookies and cookie verifiers when switching between
READDIR and READDIRPLUS, or vice sersa, then I think the current client
code handles this condition ok.On the client, both READDIR and READDIRPLUS requests use the cookie
values
from the same cached pages of the directory so I don't think they can
be
out of sync when the client switches between READDIRPLUS and READDIR
requests for differentnfs_readdircalls.In fact, currently the first
nfs_readdiruses READDIRPLUS's to fill
read
the entries and if there is no LOOKUP/GETATTR on one of the directory
entries then the client reverts to READDIR's for subsequentnfs_readdir
calls without invalidating any cached pages of the directory. If there
are LOOKUP/GETATTR done on one of the directory entries thennfs_advise_use_readdirplusis called which forces the client to use
READDIRPLUS again for the nextnfs_readdir.Quoted 5 lines
Quoted 4 lines
Unless the user mounts the export with 'nordirplus' option then the
client uses only READDIRs for all requests, no switching takes place.I don't understand your point.
The original point was that the directory's page cache seems to be cleared a little too often (quite apart from switching between READDIRPLUS and READDIR).
Quoted 8 lines
Quoted 6 lines
I think Dai is saying that cache clearing is appropriate to defer when the directory's mtime has changed but the READ method remains the same. Otherwise repeatedly adding a new file to a very large directory that is being read can trigger a situation where the reading getdents loop never completes.
Fair enough, but the section of code he was touching with his patch should be only about switching between READDIR/PLUS methods.
The code that I'd to patch, nfs_force_use_readdirplus, is called when there is an attribute cache miss which causes the client to make sure READDIRPLUS is being used on the directory, regardless which READ method is currently being used using for the directory. So it's possible that the call to nfs_force_use_readdirplus will cause the next getdent/nfs_readdir to switch to from READDIR to READDIRPLUS.
Quoted 23 lines
The mtime monitoring happens elsewhere and is already set up to invalidate the cache only on opendir()/rewinddir().
Quoted 19 lines
My two cents Euro.
Quoted 16 lines
The issue is that
nfs_advise_use_readdirplus()can cause the behaviour to switch between
use of READDIRPLUS and use of READDIR from one syscall togetdents()to
the next.
If the client is using the same page cache, across those syscalls, then
it will end up caching a mixture of cookies. Furthermore, since the cookie that is used as an argument to the next call to READDIR/READDIRPLUS is taken from that page cache, then we can end up
calling READDIRPLUS with a cookie that came from READDIR and vice versa.As I said, I'm not convinced that is legal in RFC1813 (NFSv3).
I think this is the contention point: the spec did not explicitly mention anything regarding mixing of cookies from READDIR & READDIRPLUS.
However, as I mentioned, the current client implementation already mixing cookies between READDIRPLUS and READDIR, everytime the user does a simple 'ls' on a large directory, without invalidating any mapping.
Also, as Chuck mentioned, we're not aware of any server implementation that has problems with this mixing of cookies.
-Dai
Quoted 44 lines
Quoted 44 lines
Quoted 4 lines
That is why we want to clear the page cache when we swap between use of
READDIR and use of READDIRPLUS for the case of NFSv3.Just curious, are you aware of an NFSv3 server implementation that would have a problem with a client that mixes the cookies?
Quoted 21 lines
Quoted 21 lines
Thanks,
-DaiQuoted 18 lines
Quoted 9 lines
OTOH, for NFSv4, there is no separate READDIRPLUS function, so
there
really does not appear to be any reason to clear the page cache
at
all
as we're switching between requesting attributes or not.Sorry... To spell out my objection to this change more clearly:
The
call tonfs_zap_mapping()makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3
concern.
* It would appear to be entirely superfluous for the NFSv4
case.Quoted 10 lines
Quoted 6 lines
Quoted 6 lines
So a change that might be acceptable would be to keep the existing
call
toinvalidate_mapping_pages()for NFSv3, but to remove it for NFSv4.Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]--
Chuck Lever
Quoted 11 lines · Dai Ngo
I think this is the contention point: the spec did not explicitly mention anything regarding mixing of cookies from READDIR & READDIRPLUS.
However, as I mentioned, the current client implementation already mixing
cookies between READDIRPLUS and READDIR, everytime the user does a simple
'ls' on a large directory, without invalidating any mapping.
Quoted 4 lines
Also, as Chuck mentioned, we're not aware of any server implementation
that has problems with this mixing of cookies.
OK I did a little time warp and went back to the original emails around this behaviour:
https://linux-nfs.vger.kernel.narkive.com/O0Xhnqxe/readdir-vs-getattr
Part of the problem that needed solving at the time was that even when the directory and its contents were not changing, people were still needing to do reams of GETATTR calls in order to typically satisfy an 'ls -l' or even 'ls --color'. When we see that pattern, we want to switch from using GETATTR on all these files to using READDIRPLUS.
The cache invalidation was introduced in order to force the NFS client to do these READDIRPLUS calls so we avoid the GETATTRs.
So one optimisation we could definitely do is try to track the index of the last page our descriptor accessed on readdir(), and truncate only the remaining pages. That way we don't keep re-reading the beginning of the directory.
Signature · 3 lines
Trond Myklebust Linux NFS client maintainer, Hammerspace [email protected]