Skip to contentSkip to conversation

'ls -lrt' performance issue on large dir while dir is being modified

DISCUSSION Dai Ngo 4 participants 12 messages Dec 18, 2019 – Jan 20, 2020

Subsystems (2)
lore
Share & tools
lore

Dai Ngo

author reviewer
#0 lore raw

Hi,

I'd like to report an issue with 'ls -lrt' on NFSv3 client takes a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.

The problem can be reproduced using 3 systems. One system serves as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.

Client1 creates files using this simple script:

Quoted 10 lines

#!/bin/sh

if [ $# -lt 2 ]; then
echo "Usage: $0 number_of_files base_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..." i=0

Quoted 6 lines

while [ i -lt $nfiles ] ;
do

        i=`expr $i + 1`
        echo "xyz" > $fname$i
        echo "$fname$i" 

done

Client2 runs 'time ls -lrt /tmp/mnt/bd1 |wc -l' in a loop.

The network traces and dtrace probes showed numerous READDIRPLUS3 requests restarting  from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages to be refilled starting from cookie 0 until the current requested cookie.  The cached page invalidation were tracked to nfs_force_use_readdirplus().  To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.

The modification is:

fs/nfs/dir.c void nfs_force_use_readdirplus(struct inode *dir) · line 449

Quoted 12 lines
fs/nfs/dir.c +1 -1
index a73e2f8bd8ec..5d4a64555fa7 100644
@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct inode *dir)
     if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) &&
         !list_empty(&nfsi->open_files)) {
         set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);
-        invalidate_mapping_pages(dir->i_mapping, 0, -1);
+        nfs_zap_mapping(dir, dir->i_mapping);
     }
 }

Note that after this change, I did not see READDIRPLUS3 restarting with cookie 0 anymore.

Below are the summary results of 'ls -lrt'.  For each kernel version to be compared, one row for the original kernel and one row for the kernel with the above modification.

I cloned dtrace-linux from here:
github.com/oracle/dtrace-linux-kernel

dtrace-linux 5.1.0-rc4 [ORI] 89191: 2m59.32s 193071: 6m7.810s
dtrace-linux 5.1.0-rc4 [MOD] 98771: 1m55.900s 191322: 3m48.668s

I cloned upstream Linux from here:
git.kernel.org/pub/scm/linux/kernel/git/tovards/linux.git

Upstream Linux 5.5.0-rc1 [ORI] 87891: 5m11.089s 160974: 14m4.384s
Upstream Linux 5.5.0-rc1 [MOD] 87075: 5m2.057s 161421: 14m33.615s

Please note that these are relative performance numbers and are used to illustrate the issue only.

For reference, on the original dtrace-linux it takes about 9s for 'ls -ltr' to complete on a directory with 200k files if the directory is not modified while 'ls' is running.

The number of the original Upstream Linux is *really* bad, and the modification did not seem to have any effect, not sure why... it could be something else is going on here.

The cache invalidation in nfs_force_use_readdirplus seems too drastic and might need to be reviewed. Even though this change helps but it did not get the 'ls' performance to where it's expected to be. I think even though READDIRPLUS3 was used, the attribute cache was invalidated due to the directory modification, causing attribute cache misses resulting in the calls to nfs_force_use_readdirplus as shown in this stack trace:

0 17586 page_cache_tree_delete:entry

              vmlinux`remove_mapping+0x14
              vmlinux`invalidate_inode_page+0x7c
              vmlinux`invalidate_mapping_pages+0x1dd
              nfs`nfs_force_use_readdirplus+0x47
              nfs`__dta_nfs_lookup_revalidate_478+0x5dd
              vmlinux`d_revalidate.part.24+0x10
              vmlinux`lookup_fast+0x254
              vmlinux`walk_component+0x49
              vmlinux`path_lookupat+0x79
              vmlinux`filename_lookup+0xaf
              vmlinux`user_path_at_empty+0x36
              vmlinux`vfs_statx+0x77
              vmlinux`SYSC_newlstat+0x3d
              vmlinux`SyS_newlstat+0xe
              vmlinux`do_syscall_64+0x79
              vmlinux`entry_SYSCALL_64+0x18d

Besides the overhead of refilling the page caches from cookie 0, I think the reason 'ls' still takes so long to compete because the client has to send a bunch of additional LOOKUP/ACCESS requests over the wire to service the stat(2) calls from 'ls' due to the attribute cache misses.

Please let me know you what you think and if there is any addition information is needed.

Thanks,
-Dai

author reviewer
#1 lore raw

Hi Anna, Trond,

I made a mistake with the 5.5 numbers. The VM that runs 5.5 has some problems. There is no regression with 5.5, here are the new numbers:

Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:  10m35.789s Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:  3m55.003s

My apologies for the mistake.

Now there is no regression with 5.5, I'd like to get your opinion regarding the change to revert the call from invalidate_mapping_pages to nfs_zap_mapping in nfs_force_use_readdirplus to prevent the current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm not quite sure about the intention of the prior change from nfs_zap_mapping to invalidate_mapping_pages so that is why I'm seeking advise. Or do you have any suggestions to achieve the same?

Thanks,
-Dai

Quoted 121 lines · Dai Ngo

Hi,

I'd like to report an issue with 'ls -lrt' on NFSv3 client takes a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.

The problem can be reproduced using 3 systems. One system serves as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
    Client1 creates files using this simple script:

Quoted 12 lines

#!/bin/sh

if [ $# -lt 2 ]; then
        echo "Usage: $0 number_of_files base_filename" exit
fi    nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..." i=0         while [ i -lt $nfiles ] ;
do            i=expr $i + 1
        echo "xyz" > $fname$i
        echo "$fname$i" done

Client2 runs 'time ls -lrt /tmp/mnt/bd1 |wc -l' in a loop.

The network traces and dtrace probes showed numerous READDIRPLUS3 requests restarting  from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages to be refilled starting from cookie 0 until the current requested cookie.  The cached page invalidation were tracked to nfs_force_use_readdirplus().  To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.

The modification is:

Quoted 5 lines
fs/nfs/dir.c +0 -0
index a73e2f8bd8ec..5d4a64555fa7 100644
@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct inode *dir)
Quoted 7 lines

     if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -        invalidate_mapping_pages(dir->i_mapping, 0, -1); +        nfs_zap_mapping(dir, dir->i_mapping);
     }
 }

Note that after this change, I did not see READDIRPLUS3 restarting with cookie 0 anymore.

Below are the summary results of 'ls -lrt'.  For each kernel version to be compared, one row for the original kernel and one row for the kernel with the above modification.

I cloned dtrace-linux from here:
github.com/oracle/dtrace-linux-kernel

dtrace-linux 5.1.0-rc4 [ORI] 89191: 2m59.32s   193071: 6m7.810s dtrace-linux 5.1.0-rc4 [MOD] 98771: 1m55.900s  191322: 3m48.668s

I cloned upstream Linux from here:
git.kernel.org/pub/scm/linux/kernel/git/tovards/linux.git

Upstream Linux 5.5.0-rc1 [ORI] 87891: 5m11.089s  160974: 14m4.384s Upstream Linux 5.5.0-rc1 [MOD] 87075: 5m2.057s   161421: 14m33.615s

Please note that these are relative performance numbers and are used to illustrate the issue only.

For reference, on the original dtrace-linux it takes about 9s for 'ls -ltr' to complete on a directory with 200k files if the directory is not modified while 'ls' is running.

The number of the original Upstream Linux is *really* bad, and the modification did not seem to have any effect, not sure why... it could be something else is going on here.

The cache invalidation in nfs_force_use_readdirplus seems too drastic and might need to be reviewed. Even though this change helps but it did not get the 'ls' performance to where it's expected to be. I think even though READDIRPLUS3 was used, the attribute cache was invalidated due to the directory modification, causing attribute cache misses resulting in the calls to nfs_force_use_readdirplus as shown in this stack trace:

  0  17586     page_cache_tree_delete:entry vmlinuxremove_mapping+0x14 vmlinuxinvalidate_inode_page+0x7c vmlinuxinvalidate_mapping_pages+0x1dd nfsnfs_force_use_readdirplus+0x47 nfs__dta_nfs_lookup_revalidate_478+0x5dd vmlinuxd_revalidate.part.24+0x10 vmlinuxlookup_fast+0x254 vmlinuxwalk_component+0x49 vmlinuxpath_lookupat+0x79 vmlinuxfilename_lookup+0xaf vmlinuxuser_path_at_empty+0x36 vmlinuxvfs_statx+0x77 vmlinuxSYSC_newlstat+0x3d vmlinuxSyS_newlstat+0xe vmlinuxdo_syscall_64+0x79 vmlinuxentry_SYSCALL_64+0x18d

Besides the overhead of refilling the page caches from cookie 0, I think the reason 'ls' still takes so long to compete because the client has to send a bunch of additional LOOKUP/ACCESS requests over the wire to service the stat(2) calls from 'ls' due to the attribute cache misses.

Please let me know you what you think and if there is any addition information is needed.

Thanks,
-Dai

author reviewer
#2 lore raw

Hi Anna, Trond,

Would you please let me know your opinion regarding reverting the change in nfs_force_use_readdirplus to call nfs_zap_mapping instead of invalidate_mapping_pages. This change is to prevent the cookie of the READDIRPLUS to be reset to 0 while an instance of 'ls' is running and the directory is being modified.

fs/nfs/dir.c index a73e2f8bd8ec..5d4a64555fa7 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct inode *dir)      if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) &&          !list_empty(&nfsi->open_files)) {          set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -        invalidate_mapping_pages(dir->i_mapping, 0, -1); +        nfs_zap_mapping(dir, dir->i_mapping);      }  }

Quoted 1 line
fs/nfs/dir.c index a73e2f8bd8ec..5d4a64555fa7 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct inode *dir)      if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) &&          !list_empty(&nfsi->open_files)) {          set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -        invalidate_mapping_pages(dir->i_mapping, 0, -1); +        nfs_zap_mapping(dir, dir->i_mapping);      }  } renamed from fs/nfs/dir.c +0 -0

Thanks,
-Dai

Quoted 143 lines · Dai Ngo

Hi Anna, Trond,

I made a mistake with the 5.5 numbers. The VM that runs 5.5 has some problems. There is no regression with 5.5, here are the new numbers:

Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s  197891: 10m35.789s Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s  192801: 3m55.003s

My apologies for the mistake.

Now there is no regression with 5.5, I'd like to get your opinion regarding the change to revert the call from invalidate_mapping_pages to nfs_zap_mapping in nfs_force_use_readdirplus to prevent the current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm not quite sure about the intention of the prior change from nfs_zap_mapping to invalidate_mapping_pages so that is why I'm seeking advise. Or do you have any suggestions to achieve the same?

Thanks,
-Dai

Quoted 121 lines · Dai Ngo

Hi,

I'd like to report an issue with 'ls -lrt' on NFSv3 client takes a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.

The problem can be reproduced using 3 systems. One system serves as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
    Client1 creates files using this simple script:

Quoted 12 lines

#!/bin/sh

if [ $# -lt 2 ]; then
        echo "Usage: $0 number_of_files base_filename" exit
fi    nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..." i=0         while [ i -lt $nfiles ] ;
do            i=expr $i + 1
        echo "xyz" > $fname$i
        echo "$fname$i" done

Client2 runs 'time ls -lrt /tmp/mnt/bd1 |wc -l' in a loop.

The network traces and dtrace probes showed numerous READDIRPLUS3 requests restarting  from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages to be refilled starting from cookie 0 until the current requested cookie.  The cached page invalidation were tracked to nfs_force_use_readdirplus().  To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.

The modification is:

Quoted 5 lines
fs/nfs/dir.c +0 -0
index a73e2f8bd8ec..5d4a64555fa7 100644
@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct inode *dir)
Quoted 7 lines

     if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); -        invalidate_mapping_pages(dir->i_mapping, 0, -1); +        nfs_zap_mapping(dir, dir->i_mapping);
     }
 }

Note that after this change, I did not see READDIRPLUS3 restarting with cookie 0 anymore.

Below are the summary results of 'ls -lrt'.  For each kernel version to be compared, one row for the original kernel and one row for the kernel with the above modification.

I cloned dtrace-linux from here:
github.com/oracle/dtrace-linux-kernel

dtrace-linux 5.1.0-rc4 [ORI] 89191: 2m59.32s   193071: 6m7.810s dtrace-linux 5.1.0-rc4 [MOD] 98771: 1m55.900s  191322: 3m48.668s

I cloned upstream Linux from here:
git.kernel.org/pub/scm/linux/kernel/git/tovards/linux.git

Upstream Linux 5.5.0-rc1 [ORI] 87891: 5m11.089s  160974: 14m4.384s Upstream Linux 5.5.0-rc1 [MOD] 87075: 5m2.057s   161421: 14m33.615s

Please note that these are relative performance numbers and are used to illustrate the issue only.

For reference, on the original dtrace-linux it takes about 9s for 'ls -ltr' to complete on a directory with 200k files if the directory is not modified while 'ls' is running.

The number of the original Upstream Linux is *really* bad, and the modification did not seem to have any effect, not sure why... it could be something else is going on here.

The cache invalidation in nfs_force_use_readdirplus seems too drastic and might need to be reviewed. Even though this change helps but it did not get the 'ls' performance to where it's expected to be. I think even though READDIRPLUS3 was used, the attribute cache was invalidated due to the directory modification, causing attribute cache misses resulting in the calls to nfs_force_use_readdirplus as shown in this stack trace:

  0  17586     page_cache_tree_delete:entry vmlinuxremove_mapping+0x14 vmlinuxinvalidate_inode_page+0x7c vmlinuxinvalidate_mapping_pages+0x1dd nfsnfs_force_use_readdirplus+0x47 nfs__dta_nfs_lookup_revalidate_478+0x5dd vmlinuxd_revalidate.part.24+0x10 vmlinuxlookup_fast+0x254 vmlinuxwalk_component+0x49 vmlinuxpath_lookupat+0x79 vmlinuxfilename_lookup+0xaf vmlinuxuser_path_at_empty+0x36 vmlinuxvfs_statx+0x77 vmlinuxSYSC_newlstat+0x3d vmlinuxSyS_newlstat+0xe vmlinuxdo_syscall_64+0x79 vmlinuxentry_SYSCALL_64+0x18d

Besides the overhead of refilling the page caches from cookie 0, I think the reason 'ls' still takes so long to compete because the client has to send a bunch of additional LOOKUP/ACCESS requests over the wire to service the stat(2) calls from 'ls' due to the attribute cache misses.

Please let me know you what you think and if there is any addition information is needed.

Thanks,
-Dai

#3 lore raw
Quoted 88 lines · Dai Ngo

Hi Anna, Trond,

Would you please let me know your opinion regarding reverting the change in
nfs_force_use_readdirplus to call nfs_zap_mapping instead of invalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be reset to 0 while
an instance of 'ls' is running and the directory is being modified.

Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0 -0

a73e2f8bd8ec..5d4a64555fa7 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -444,7 +444,7 @@ void
nfs_force_use_readdirplus(struct inode *dir) if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); - invalidate_mapping_pages(dir->i_mapping, 0, -1); +
nfs_zap_mapping(dir, dir->i_mapping); } }

Thanks,
-Dai

Quoted 64 lines · Dai Ngo

Hi Anna, Trond,

I made a mistake with the 5.5 numbers. The VM that runs 5.5 has some
problems. There is no regression with 5.5, here are the new numbers:

Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891: 10m35.789s Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801: 3m55.003s

My apologies for the mistake.

Now there is no regression with 5.5, I'd like to get your opinion regarding the change to revert the call from invalidate_mapping_pages
to nfs_zap_mapping in nfs_force_use_readdirplus to prevent the current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm not quite sure about the intention of the prior change from nfs_zap_mapping to invalidate_mapping_pages so that is why I'm seeking advise. Or do you have any suggestions to achieve the same?

Thanks,
-Dai

Quoted 39 lines · Dai Ngo

Hi,

I'd like to report an issue with 'ls -lrt' on NFSv3 client takes a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.

The problem can be reproduced using 3 systems. One system serves as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
Client1 creates files using this simple script:

Quoted 12 lines

#!/bin/sh

if [ $# -lt 2 ]; then
echo "Usage: $0 number_of_files base_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" done

Client2 runs 'time ls -lrt /tmp/mnt/bd1 |wc -l' in a loop.

The network traces and dtrace probes showed numerous READDIRPLUS3 requests restarting from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages to be refilled starting from cookie 0 until the current requested cookie. The cached page invalidation were tracked to nfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.

The modification is:

Quoted 13 lines
Quoted 5 lines
Quoted 5 lines
Quoted 5 lines
fs/nfs/dir.c +0 -0
index a73e2f8bd8ec..5d4a64555fa7 100644
@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct inode
Quoted 8 lines
Quoted 8 lines

fs/nfs/dir.c void nfs_force_use_readdirplus(struct inode · line 450

Quoted 8 lines
fs/nfs/dir.c +1 -1
 *dir)
     if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) &&
         !list_empty(&nfsi->open_files)) {
         set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);
-        invalidate_mapping_pages(dir->i_mapping, 0, -1);
+        nfs_zap_mapping(dir, dir->i_mapping);
     }
 }

This change is only reverting part of commit 79f687a3de9e. My problem with that is as follows:

RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must match those returned by previous READDIRPLUS calls, and READDIR cookies and verifiers must match those returned by previous READDIR calls. It says nothing about being able to assume cookies from READDIR and READDIRPLUS calls are interchangeable. So the only reason I can see for the invalidate_mapping_pages() is to ensure that we do separate the two cookie caches.

OTOH, for NFSv4, there is no separate READDIRPLUS function, so there really does not appear to be any reason to clear the page cache at all as we're switching between requesting attributes or not.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#4 lore raw
Quoted 126 lines · Trond Myklebust
Quoted 96 lines · Dai Ngo

Hi Anna, Trond,

Would you please let me know your opinion regarding reverting the change in
nfs_force_use_readdirplus to call nfs_zap_mapping instead of invalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be reset to 0 while
an instance of 'ls' is running and the directory is being modified.

Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0 -0

a73e2f8bd8ec..5d4a64555fa7 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -444,7 +444,7 @@ void
nfs_force_use_readdirplus(struct inode *dir) if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); - invalidate_mapping_pages(dir->i_mapping, 0, -1); +
nfs_zap_mapping(dir, dir->i_mapping); } }

Thanks,
-Dai

Quoted 72 lines · Dai Ngo

Hi Anna, Trond,

I made a mistake with the 5.5 numbers. The VM that runs 5.5 has some
problems. There is no regression with 5.5, here are the new numbers:

Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003s

My apologies for the mistake.

Now there is no regression with 5.5, I'd like to get your opinion regarding the change to revert the call from invalidate_mapping_pages
to nfs_zap_mapping in nfs_force_use_readdirplus to prevent the current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm not quite sure about the intention of the prior change from nfs_zap_mapping to invalidate_mapping_pages so that is why I'm seeking advise. Or do you have any suggestions to achieve the same?

Thanks,
-Dai

Quoted 44 lines · Dai Ngo

Hi,

I'd like to report an issue with 'ls -lrt' on NFSv3 client takes
a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.

The problem can be reproduced using 3 systems. One system serves
as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
Client1 creates files using this simple script:

Quoted 12 lines

#!/bin/sh

if [ $# -lt 2 ]; then
echo "Usage: $0 number_of_files base_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" done

Client2 runs 'time ls -lrt /tmp/mnt/bd1 |wc -l' in a loop.

The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked to nfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.

The modification is:

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
fs/nfs/dir.c +0 -0
index a73e2f8bd8ec..5d4a64555fa7 100644
@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct

inode

Quoted 8 lines
Quoted 8 lines

fs/nfs/dir.c

Quoted 8 lines

*dir)

     if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) &&
         !list_empty(&nfsi->open_files)) {
         set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);
fs/nfs/dir.c +1 -1
-        invalidate_mapping_pages(dir->i_mapping, 0, -1);
+        nfs_zap_mapping(dir, dir->i_mapping);
     }
 }

This change is only reverting part of commit 79f687a3de9e. My problem with that is as follows:

RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must match
those returned by previous READDIRPLUS calls, and READDIR cookies and verifiers must match those returned by previous READDIR calls. It says
nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for the invalidate_mapping_pages() is to ensure that we do separate the two cookie caches.

Quoted 5 lines

OTOH, for NFSv4, there is no separate READDIRPLUS function, so there really does not appear to be any reason to clear the page cache at all
as we're switching between requesting attributes or not.

Sorry... To spell out my objection to this change more clearly: The call to nfs_zap_mapping() makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3 concern.
* It would appear to be entirely superfluous for the NFSv4 case.

So a change that might be acceptable would be to keep the existing call to invalidate_mapping_pages() for NFSv3, but to remove it for NFSv4.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
author reviewer
#5 lore raw
Quoted 129 lines · Trond Myklebust
Quoted 123 lines · Trond Myklebust
Quoted 94 lines · Dai Ngo

Hi Anna, Trond,

Would you please let me know your opinion regarding reverting the change in
nfs_force_use_readdirplus to call nfs_zap_mapping instead of invalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be reset to 0 while
an instance of 'ls' is running and the directory is being modified.

Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0 -0

a73e2f8bd8ec..5d4a64555fa7 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -444,7 +444,7 @@ void
nfs_force_use_readdirplus(struct inode *dir) if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); - invalidate_mapping_pages(dir->i_mapping, 0, -1); +
nfs_zap_mapping(dir, dir->i_mapping); } }

Thanks,
-Dai

Quoted 71 lines · Dai Ngo

Hi Anna, Trond,

I made a mistake with the 5.5 numbers. The VM that runs 5.5 has some
problems. There is no regression with 5.5, here are the new numbers:

Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003s

My apologies for the mistake.

Now there is no regression with 5.5, I'd like to get your opinion regarding the change to revert the call from invalidate_mapping_pages
to nfs_zap_mapping in nfs_force_use_readdirplus to prevent the current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm not quite sure about the intention of the prior change from nfs_zap_mapping to invalidate_mapping_pages so that is why I'm seeking advise. Or do you have any suggestions to achieve the same?

Thanks,
-Dai

Quoted 43 lines · Dai Ngo

Hi,

I'd like to report an issue with 'ls -lrt' on NFSv3 client takes
a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.

The problem can be reproduced using 3 systems. One system serves
as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
Client1 creates files using this simple script:

Quoted 12 lines

#!/bin/sh

if [ $# -lt 2 ]; then
echo "Usage: $0 number_of_files base_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" done

Client2 runs 'time ls -lrt /tmp/mnt/bd1 |wc -l' in a loop.

The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked to nfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.

The modification is:

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
fs/nfs/dir.c +0 -0
index a73e2f8bd8ec..5d4a64555fa7 100644
@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct

inode

Quoted 8 lines
Quoted 8 lines

fs/nfs/dir.c

Quoted 8 lines

*dir)

      if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) &&
          !list_empty(&nfsi->open_files)) {
          set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);
fs/nfs/dir.c +1 -1
-        invalidate_mapping_pages(dir->i_mapping, 0, -1);
+        nfs_zap_mapping(dir, dir->i_mapping);
      }
  }

This change is only reverting part of commit 79f687a3de9e. My problem with that is as follows:

RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must match
those returned by previous READDIRPLUS calls, and READDIR cookies and verifiers must match those returned by previous READDIR calls. It says
nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for the invalidate_mapping_pages() is to ensure that we do separate the two cookie caches.

Quoted 5 lines

OTOH, for NFSv4, there is no separate READDIRPLUS function, so there really does not appear to be any reason to clear the page cache at all
as we're switching between requesting attributes or not.

Quoted 8 lines

Sorry... To spell out my objection to this change more clearly: The call to nfs_zap_mapping() makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3 concern.
* It would appear to be entirely superfluous for the NFSv4 case.

So a change that might be acceptable would be to keep the existing call to invalidate_mapping_pages() for NFSv3, but to remove it for NFSv4.

Thank you Trond, I'll make your suggested change, test it and resubmit.

-Dai

Quoted 1 line
author reviewer
#6 lore raw
In reply to Trond Myklebust

Hi Trond,

Quoted 113 lines · Trond Myklebust
Quoted 112 lines · Trond Myklebust
Quoted 94 lines · Dai Ngo

Hi Anna, Trond,

Would you please let me know your opinion regarding reverting the change in
nfs_force_use_readdirplus to call nfs_zap_mapping instead of invalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be reset to 0 while
an instance of 'ls' is running and the directory is being modified.

Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0 -0

a73e2f8bd8ec..5d4a64555fa7 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -444,7 +444,7 @@ void
nfs_force_use_readdirplus(struct inode *dir) if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); - invalidate_mapping_pages(dir->i_mapping, 0, -1); +
nfs_zap_mapping(dir, dir->i_mapping); } }

Thanks,
-Dai

Quoted 71 lines · Dai Ngo

Hi Anna, Trond,

I made a mistake with the 5.5 numbers. The VM that runs 5.5 has some
problems. There is no regression with 5.5, here are the new numbers:

Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003s

My apologies for the mistake.

Now there is no regression with 5.5, I'd like to get your opinion regarding the change to revert the call from invalidate_mapping_pages
to nfs_zap_mapping in nfs_force_use_readdirplus to prevent the current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm not quite sure about the intention of the prior change from nfs_zap_mapping to invalidate_mapping_pages so that is why I'm seeking advise. Or do you have any suggestions to achieve the same?

Thanks,
-Dai

Quoted 43 lines · Dai Ngo

Hi,

I'd like to report an issue with 'ls -lrt' on NFSv3 client takes
a very long time to display the content of a large directory (100k - 200k files) while the directory is being modified by another NFSv3 client.

The problem can be reproduced using 3 systems. One system serves
as the NFS server, one system runs as the client that doing the 'ls -lrt' and another system runs the client that creates files on the server.
Client1 creates files using this simple script:

Quoted 12 lines

#!/bin/sh

if [ $# -lt 2 ]; then
echo "Usage: $0 number_of_files base_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" done

Client2 runs 'time ls -lrt /tmp/mnt/bd1 |wc -l' in a loop.

The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate the cached pages of the directory were invalidated causing the pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked to nfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.

The modification is:

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
fs/nfs/dir.c +0 -0
index a73e2f8bd8ec..5d4a64555fa7 100644
@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct

inode

Quoted 8 lines
Quoted 8 lines

fs/nfs/dir.c

Quoted 8 lines

*dir)

      if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) &&
          !list_empty(&nfsi->open_files)) {
          set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);
fs/nfs/dir.c +1 -1
-        invalidate_mapping_pages(dir->i_mapping, 0, -1);
+        nfs_zap_mapping(dir, dir->i_mapping);
      }
  }

This change is only reverting part of commit 79f687a3de9e. My problem with that is as follows:

Quoted 10 lines
Quoted 10 lines

RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must match
those returned by previous READDIRPLUS calls, and READDIR cookies and verifiers must match those returned by previous READDIR calls. It says
nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for the invalidate_mapping_pages() is to ensure that we do separate the two cookie caches.

If I understand your concern correctly that in NFSv3 the client must maintain valid cookies and cookie verifiers when switching between READDIR and READDIRPLUS, or vice sersa, then I think the current client code handles this condition ok.

On the client, both READDIR and READDIRPLUS requests use the cookie values from the same cached pages of the directory so I don't think they can be out of sync when the client switches between READDIRPLUS and READDIR requests for different nfs_readdir calls.

In fact, currently the first nfs_readdir uses READDIRPLUS's to fill read the entries and if there is no LOOKUP/GETATTR on one of the directory entries then the client reverts to READDIR's for subsequent nfs_readdir calls without invalidating any cached pages of the directory. If there are LOOKUP/GETATTR done on one of the directory entries then nfs_advise_use_readdirplus is called which forces the client to use READDIRPLUS again for the next nfs_readdir.

Unless the user mounts the export with 'nordirplus' option then the client uses only READDIRs for all requests, no switching takes place.

Thanks,
-Dai

Quoted 15 lines
Quoted 6 lines

OTOH, for NFSv4, there is no separate READDIRPLUS function, so there really does not appear to be any reason to clear the page cache at all
as we're switching between requesting attributes or not.

Sorry... To spell out my objection to this change more clearly: The call to nfs_zap_mapping() makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3 concern.
* It would appear to be entirely superfluous for the NFSv4 case.

So a change that might be acceptable would be to keep the existing call to invalidate_mapping_pages() for NFSv3, but to remove it for NFSv4.

#7 lore raw
Quoted 167 lines · Dai Ngo

Hi Trond,

Quoted 127 lines · Trond Myklebust
Quoted 126 lines · Trond Myklebust
Quoted 107 lines · Dai Ngo

Hi Anna, Trond,

Would you please let me know your opinion regarding reverting the
change in
nfs_force_use_readdirplus to call nfs_zap_mapping instead of invalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be reset
to 0 while
an instance of 'ls' is running and the directory is being modified.

Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0 -0

a73e2f8bd8ec..5d4a64555fa7 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -444,7 +444,7 @@ void
nfs_force_use_readdirplus(struct inode *dir) if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); - invalidate_mapping_pages(dir->i_mapping, 0, -1); +
nfs_zap_mapping(dir, dir->i_mapping); } }

Thanks,
-Dai

Quoted 81 lines · Dai Ngo

Hi Anna, Trond,

I made a mistake with the 5.5 numbers. The VM that runs 5.5 has
some
problems. There is no regression with 5.5, here are the new numbers:

Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003s

My apologies for the mistake.

Now there is no regression with 5.5, I'd like to get your opinion
regarding the change to revert the call from invalidate_mapping_pages
to nfs_zap_mapping in nfs_force_use_readdirplus to prevent the
current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm
not quite sure about the intention of the prior change from nfs_zap_mapping to invalidate_mapping_pages so that is why I'm
seeking advise. Or do you have any suggestions to achieve the same?

Thanks,
-Dai

Quoted 48 lines · Dai Ngo

Hi,

I'd like to report an issue with 'ls -lrt' on NFSv3 client takes
a very long time to display the content of a large directory
(100k - 200k files) while the directory is being modified by
another NFSv3 client.

The problem can be reproduced using 3 systems. One system serves
as the NFS server, one system runs as the client that doing the
'ls -lrt' and another system runs the client that creates files
on the server.
Client1 creates files using this simple script:

Quoted 12 lines

#!/bin/sh

if [ $# -lt 2 ]; then
echo "Usage: $0 number_of_files base_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" done

Client2 runs 'time ls -lrt /tmp/mnt/bd1 |wc -l' in a loop.

The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate the
cached pages of the directory were invalidated causing the pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked to nfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.

The modification is:

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
fs/nfs/dir.c +0 -0
index a73e2f8bd8ec..5d4a64555fa7 100644
@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct

inode

Quoted 8 lines
Quoted 8 lines

fs/nfs/dir.c

Quoted 8 lines

*dir)

      if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) &&
          !list_empty(&nfsi->open_files)) {
          set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);
fs/nfs/dir.c +1 -1
-        invalidate_mapping_pages(dir->i_mapping, 0, -1);
+        nfs_zap_mapping(dir, dir->i_mapping);
      }
  }

This change is only reverting part of commit 79f687a3de9e. My problem
with that is as follows:

Quoted 12 lines
Quoted 4 lines

RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must match
those returned by previous READDIRPLUS calls, and READDIR cookies and

Quoted 8 lines

verifiers must match those returned by previous READDIR calls. It says
nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for the invalidate_mapping_pages() is to ensure that we do separate the two
cookie caches.

If I understand your concern correctly that in NFSv3 the client must maintain valid cookies and cookie verifiers when switching between READDIR and READDIRPLUS, or vice sersa, then I think the current client
code handles this condition ok.

On the client, both READDIR and READDIRPLUS requests use the cookie values
from the same cached pages of the directory so I don't think they can be
out of sync when the client switches between READDIRPLUS and READDIR requests for different nfs_readdir calls.

In fact, currently the first nfs_readdir uses READDIRPLUS's to fill read
the entries and if there is no LOOKUP/GETATTR on one of the directory entries then the client reverts to READDIR's for subsequent nfs_readdir
calls without invalidating any cached pages of the directory. If there
are LOOKUP/GETATTR done on one of the directory entries then nfs_advise_use_readdirplus is called which forces the client to use READDIRPLUS again for the next nfs_readdir.

Quoted 2 lines

Unless the user mounts the export with 'nordirplus' option then the client uses only READDIRs for all requests, no switching takes place.

I don't understand your point. The issue is that nfs_advise_use_readdirplus() can cause the behaviour to switch between use of READDIRPLUS and use of READDIR from one syscall to getdents() to the next.
If the client is using the same page cache, across those syscalls, then it will end up caching a mixture of cookies. Furthermore, since the cookie that is used as an argument to the next call to READDIR/READDIRPLUS is taken from that page cache, then we can end up calling READDIRPLUS with a cookie that came from READDIR and vice versa.

As I said, I'm not convinced that is legal in RFC1813 (NFSv3).

That is why we want to clear the page cache when we swap between use of READDIR and use of READDIRPLUS for the case of NFSv3.

Quoted 23 lines

Thanks,
-Dai

Quoted 19 lines
Quoted 7 lines

OTOH, for NFSv4, there is no separate READDIRPLUS function, so there
really does not appear to be any reason to clear the page cache at
all
as we're switching between requesting attributes or not.

Sorry... To spell out my objection to this change more clearly: The call to nfs_zap_mapping() makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3
concern.
* It would appear to be entirely superfluous for the NFSv4 case.

So a change that might be acceptable would be to keep the existing call
to invalidate_mapping_pages() for NFSv3, but to remove it for NFSv4.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]
#8 lore raw
Quoted 172 lines · Trond Myklebust
Quoted 167 lines · Dai Ngo

Hi Trond,

Quoted 127 lines · Trond Myklebust
Quoted 126 lines · Trond Myklebust
Quoted 107 lines · Dai Ngo

Hi Anna, Trond,

Would you please let me know your opinion regarding reverting the
change in
nfs_force_use_readdirplus to call nfs_zap_mapping instead of invalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be reset
to 0 while
an instance of 'ls' is running and the directory is being modified.

Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0 -0

a73e2f8bd8ec..5d4a64555fa7 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -444,7 +444,7 @@ void
nfs_force_use_readdirplus(struct inode *dir) if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); - invalidate_mapping_pages(dir->i_mapping, 0, -1); +
nfs_zap_mapping(dir, dir->i_mapping); } }

Thanks,
-Dai

Quoted 81 lines · Dai Ngo

Hi Anna, Trond,

I made a mistake with the 5.5 numbers. The VM that runs 5.5 has
some
problems. There is no regression with 5.5, here are the new numbers:

Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003s

My apologies for the mistake.

Now there is no regression with 5.5, I'd like to get your opinion
regarding the change to revert the call from invalidate_mapping_pages
to nfs_zap_mapping in nfs_force_use_readdirplus to prevent the
current 'ls' from restarting the READDIRPLUS3 from cookie 0. I'm
not quite sure about the intention of the prior change from nfs_zap_mapping to invalidate_mapping_pages so that is why I'm
seeking advise. Or do you have any suggestions to achieve the same?

Thanks,
-Dai

Quoted 48 lines · Dai Ngo

Hi,

I'd like to report an issue with 'ls -lrt' on NFSv3 client takes
a very long time to display the content of a large directory
(100k - 200k files) while the directory is being modified by
another NFSv3 client.

The problem can be reproduced using 3 systems. One system serves
as the NFS server, one system runs as the client that doing the
'ls -lrt' and another system runs the client that creates files
on the server.
Client1 creates files using this simple script:

Quoted 12 lines

#!/bin/sh

if [ $# -lt 2 ]; then
echo "Usage: $0 number_of_files base_filename" exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" done

Client2 runs 'time ls -lrt /tmp/mnt/bd1 |wc -l' in a loop.

The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate the
cached pages of the directory were invalidated causing the pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked to nfs_force_use_readdirplus(). To verify, I made the below modification, ran the test for various kernel versions and captured the results shown below.

The modification is:

Quoted 14 lines
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
fs/nfs/dir.c +0 -0
index a73e2f8bd8ec..5d4a64555fa7 100644
@@ -444,7 +444,7 @@ void nfs_force_use_readdirplus(struct

inode

Quoted 8 lines
Quoted 8 lines

fs/nfs/dir.c

Quoted 8 lines

*dir)

     if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS) &&
         !list_empty(&nfsi->open_files)) {
         set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags);
fs/nfs/dir.c +1 -1
-        invalidate_mapping_pages(dir->i_mapping, 0, -1);
+        nfs_zap_mapping(dir, dir->i_mapping);
     }
 }

This change is only reverting part of commit 79f687a3de9e. My problem
with that is as follows:

Quoted 12 lines
Quoted 4 lines

RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must match
those returned by previous READDIRPLUS calls, and READDIR cookies and

Quoted 8 lines

verifiers must match those returned by previous READDIR calls. It says
nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for the invalidate_mapping_pages() is to ensure that we do separate the two
cookie caches.

If I understand your concern correctly that in NFSv3 the client must maintain valid cookies and cookie verifiers when switching between READDIR and READDIRPLUS, or vice sersa, then I think the current client
code handles this condition ok.

On the client, both READDIR and READDIRPLUS requests use the cookie values
from the same cached pages of the directory so I don't think they can be
out of sync when the client switches between READDIRPLUS and READDIR requests for different nfs_readdir calls.

In fact, currently the first nfs_readdir uses READDIRPLUS's to fill read
the entries and if there is no LOOKUP/GETATTR on one of the directory entries then the client reverts to READDIR's for subsequent nfs_readdir
calls without invalidating any cached pages of the directory. If there
are LOOKUP/GETATTR done on one of the directory entries then nfs_advise_use_readdirplus is called which forces the client to use READDIRPLUS again for the next nfs_readdir.

Quoted 5 lines
Quoted 2 lines

Unless the user mounts the export with 'nordirplus' option then the client uses only READDIRs for all requests, no switching takes place.

I don't understand your point.

The original point was that the directory's page cache seems to be cleared a little too often (quite apart from switching between READDIRPLUS and READDIR).

I think Dai is saying that cache clearing is appropriate to defer when the directory's mtime has changed but the READ method remains the same. Otherwise repeatedly adding a new file to a very large directory that is being read can trigger a situation where the reading getdents loop never completes.

My two cents Euro.

Quoted 11 lines

The issue is that
nfs_advise_use_readdirplus() can cause the behaviour to switch between use of READDIRPLUS and use of READDIR from one syscall to getdents() to the next.
If the client is using the same page cache, across those syscalls, then it will end up caching a mixture of cookies. Furthermore, since the cookie that is used as an argument to the next call to READDIR/READDIRPLUS is taken from that page cache, then we can end up calling READDIRPLUS with a cookie that came from READDIR and vice versa.

Quoted 4 lines

As I said, I'm not convinced that is legal in RFC1813 (NFSv3).

That is why we want to clear the page cache when we swap between use of READDIR and use of READDIRPLUS for the case of NFSv3.

Just curious, are you aware of an NFSv3 server implementation that would have a problem with a client that mixes the cookies?

Quoted 17 lines
Quoted 17 lines

Thanks,
-Dai

Quoted 14 lines
Quoted 7 lines

OTOH, for NFSv4, there is no separate READDIRPLUS function, so there
really does not appear to be any reason to clear the page cache at
all
as we're switching between requesting attributes or not.

Sorry... To spell out my objection to this change more clearly: The call to nfs_zap_mapping() makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3
concern.
* It would appear to be entirely superfluous for the NFSv4 case.

Quoted 9 lines
Quoted 5 lines
Quoted 5 lines

So a change that might be acceptable would be to keep the existing call
to invalidate_mapping_pages() for NFSv3, but to remove it for NFSv4.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

--
Chuck Lever

#9 lore raw
Quoted 221 lines · Chuck Lever
Quoted 210 lines

On Jan 18, 2020, at 10:58 AM, Trond Myklebust <
[email protected]> wrote:

Quoted 206 lines · Dai Ngo

Hi Trond,

Quoted 162 lines · Trond Myklebust
Quoted 161 lines · Trond Myklebust
Quoted 28 lines · Dai Ngo

Hi Anna, Trond,

Would you please let me know your opinion regarding reverting
the
change in
nfs_force_use_readdirplus to call nfs_zap_mapping instead of
invalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be
reset
to 0 while
an instance of 'ls' is running and the directory is being modified.

Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0 -0

a73e2f8bd8ec..5d4a64555fa7 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -444,7 +444,7 @@ void
nfs_force_use_readdirplus(struct inode *dir) if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); - invalidate_mapping_pages(dir->i_mapping, 0, -1); +
nfs_zap_mapping(dir, dir->i_mapping); } }

Thanks,
-Dai

Quoted 119 lines
Quoted 118 lines · Dai Ngo

Hi Anna, Trond,

I made a mistake with the 5.5 numbers. The VM that runs 5.5
has
some
problems. There is no regression with 5.5, here are the new
numbers:

Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003s

My apologies for the mistake.

Now there is no regression with 5.5, I'd like to get your opinion
regarding the change to revert the call from invalidate_mapping_pages
to nfs_zap_mapping in nfs_force_use_readdirplus to prevent
the
current 'ls' from restarting the READDIRPLUS3 from cookie 0.
I'm
not quite sure about the intention of the prior change from
nfs_zap_mapping to invalidate_mapping_pages so that is why
I'm
seeking advise. Or do you have any suggestions to achieve the
same?

Thanks,
-Dai

Quoted 78 lines · Dai Ngo

Hi,

I'd like to report an issue with 'ls -lrt' on NFSv3 client
takes
a very long time to display the content of a large directory
(100k - 200k files) while the directory is being modified
by
another NFSv3 client.

The problem can be reproduced using 3 systems. One system
serves
as the NFS server, one system runs as the client that doing
the
'ls -lrt' and another system runs the client that creates
files
on the server.
Client1 creates files using this simple script:

Quoted 14 lines

#!/bin/sh

if [ $# -lt 2 ]; then
echo "Usage: $0 number_of_files
base_filename"
exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using
filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" done

Client2 runs 'time ls -lrt /tmp/mnt/bd1 |wc -l' in a loop.

The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate
the
cached pages of the directory were invalidated causing the
pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked to nfs_force_use_readdirplus(). To verify, I made the below
modification, ran the test for various kernel versions and
captured the results shown below.

The modification is:

Quoted 18 lines
fs/nfs/dir.c +0 -0
index a73e2f8bd8ec..5d4a64555fa7 100644
@@ -444,7 +444,7 @@ void

nfs_force_use_readdirplus(struct
inode
*dir)
if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS)
&&

         !list_empty(&nfsi->open_files)) {
         set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi-
Quoted 1 line

flags);

fs/nfs/dir.c +1 -2
-        invalidate_mapping_pages(dir->i_mapping, 0,
-1);
+        nfs_zap_mapping(dir, dir->i_mapping);
     }
 }

This change is only reverting part of commit 79f687a3de9e. My problem
with that is as follows:

RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must
match
those returned by previous READDIRPLUS calls, and READDIR cookies
and
verifiers must match those returned by previous READDIR calls. It
says

Quoted 8 lines
Quoted 8 lines

nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for the
invalidate_mapping_pages() is to ensure that we do separate the
two
cookie caches.

If I understand your concern correctly that in NFSv3 the client must
maintain valid cookies and cookie verifiers when switching between
READDIR and READDIRPLUS, or vice sersa, then I think the current client
code handles this condition ok.

On the client, both READDIR and READDIRPLUS requests use the cookie
values
from the same cached pages of the directory so I don't think they can
be
out of sync when the client switches between READDIRPLUS and READDIR
requests for different nfs_readdir calls.

In fact, currently the first nfs_readdir uses READDIRPLUS's to fill
read
the entries and if there is no LOOKUP/GETATTR on one of the directory
entries then the client reverts to READDIR's for subsequent nfs_readdir
calls without invalidating any cached pages of the directory. If there
are LOOKUP/GETATTR done on one of the directory entries then nfs_advise_use_readdirplus is called which forces the client to use
READDIRPLUS again for the next nfs_readdir.

Quoted 6 lines
Quoted 4 lines

Unless the user mounts the export with 'nordirplus' option then the
client uses only READDIRs for all requests, no switching takes place.

I don't understand your point.

The original point was that the directory's page cache seems to be cleared a little too often (quite apart from switching between READDIRPLUS and READDIR).

Quoted 6 lines

I think Dai is saying that cache clearing is appropriate to defer when the directory's mtime has changed but the READ method remains the same. Otherwise repeatedly adding a new file to a very large directory that is being read can trigger a situation where the reading getdents loop never completes.

Fair enough, but the section of code he was touching with his patch should be only about switching between READDIR/PLUS methods.

The mtime monitoring happens elsewhere and is already set up to invalidate the cache only on opendir()/rewinddir().

Quoted 65 lines

My two cents Euro.

Quoted 15 lines

The issue is that
nfs_advise_use_readdirplus() can cause the behaviour to switch between
use of READDIRPLUS and use of READDIR from one syscall to getdents() to
the next.
If the client is using the same page cache, across those syscalls, then
it will end up caching a mixture of cookies. Furthermore, since the cookie that is used as an argument to the next call to READDIR/READDIRPLUS is taken from that page cache, then we can end up
calling READDIRPLUS with a cookie that came from READDIR and vice versa.

Quoted 5 lines

As I said, I'm not convinced that is legal in RFC1813 (NFSv3).

That is why we want to clear the page cache when we swap between use of
READDIR and use of READDIRPLUS for the case of NFSv3.

Just curious, are you aware of an NFSv3 server implementation that would have a problem with a client that mixes the cookies?

Quoted 21 lines
Quoted 21 lines

Thanks,
-Dai

Quoted 18 lines
Quoted 9 lines

OTOH, for NFSv4, there is no separate READDIRPLUS function, so
there
really does not appear to be any reason to clear the page cache
at
all
as we're switching between requesting attributes or not.

Sorry... To spell out my objection to this change more clearly:
The
call to nfs_zap_mapping() makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3
concern.
* It would appear to be entirely superfluous for the NFSv4
case.

Quoted 10 lines
Quoted 6 lines
Quoted 6 lines

So a change that might be acceptable would be to keep the existing
call
to invalidate_mapping_pages() for NFSv3, but to remove it for NFSv4.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

--
Chuck Lever

Signature · 5 lines
Trond Myklebust
CTO, Hammerspace Inc
4300 El Camino Real, Suite 105
Los Altos, CA 94022
www.hammer.space
author reviewer
#10 lore raw
Quoted 219 lines · Trond Myklebust
Quoted 218 lines · Chuck Lever
Quoted 209 lines

On Jan 18, 2020, at 10:58 AM, Trond Myklebust <
[email protected]> wrote:

Quoted 205 lines · Dai Ngo

Hi Trond,

Quoted 162 lines · Trond Myklebust
Quoted 161 lines · Trond Myklebust
Quoted 28 lines · Dai Ngo

Hi Anna, Trond,

Would you please let me know your opinion regarding reverting
the
change in
nfs_force_use_readdirplus to call nfs_zap_mapping instead of
invalidate_mapping_pages.
This change is to prevent the cookie of the READDIRPLUS to be
reset
to 0 while
an instance of 'ls' is running and the directory is being modified.

Quoted 9 lines
fs/nfs/dir.c index renamed from fs/nfs/dir.c +0 -0

a73e2f8bd8ec..5d4a64555fa7 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -444,7 +444,7 @@ void
nfs_force_use_readdirplus(struct inode *dir) if
(nfs_server_capable(dir, NFS_CAP_READDIRPLUS) && !list_empty(&nfsi->open_files)) { set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi->flags); - invalidate_mapping_pages(dir->i_mapping, 0, -1); +
nfs_zap_mapping(dir, dir->i_mapping); } }

Thanks,
-Dai

Quoted 119 lines
Quoted 118 lines · Dai Ngo

Hi Anna, Trond,

I made a mistake with the 5.5 numbers. The VM that runs 5.5
has
some
problems. There is no regression with 5.5, here are the new
numbers:

Upstream Linux 5.5.0-rc1 [ORI] 93296: 3m10.917s 197891:
10m35.789s
Upstream Linux 5.5.0-rc1 [MOD] 98614: 1m59.649s 192801:
3m55.003s

My apologies for the mistake.

Now there is no regression with 5.5, I'd like to get your opinion
regarding the change to revert the call from invalidate_mapping_pages
to nfs_zap_mapping in nfs_force_use_readdirplus to prevent
the
current 'ls' from restarting the READDIRPLUS3 from cookie 0.
I'm
not quite sure about the intention of the prior change from
nfs_zap_mapping to invalidate_mapping_pages so that is why
I'm
seeking advise. Or do you have any suggestions to achieve the
same?

Thanks,
-Dai

Quoted 78 lines · Dai Ngo

Hi,

I'd like to report an issue with 'ls -lrt' on NFSv3 client
takes
a very long time to display the content of a large directory
(100k - 200k files) while the directory is being modified
by
another NFSv3 client.

The problem can be reproduced using 3 systems. One system
serves
as the NFS server, one system runs as the client that doing
the
'ls -lrt' and another system runs the client that creates
files
on the server.
Client1 creates files using this simple script:

Quoted 14 lines

#!/bin/sh

if [ $# -lt 2 ]; then
echo "Usage: $0 number_of_files
base_filename"
exit
fi nfiles=$1
fname=$2
echo "creating $nfiles files using
filename[$fname]..."
i=0 while [ i -lt $nfiles ] ;
do i=expr $i + 1
echo "xyz" > $fname$i
echo "$fname$i" done

Client2 runs 'time ls -lrt /tmp/mnt/bd1 |wc -l' in a loop.

The network traces and dtrace probes showed numerous READDIRPLUS3
requests restarting from cookie 0 which seemed to indicate
the
cached pages of the directory were invalidated causing the
pages
to be refilled starting from cookie 0 until the current requested
cookie. The cached page invalidation were tracked to nfs_force_use_readdirplus(). To verify, I made the below
modification, ran the test for various kernel versions and
captured the results shown below.

The modification is:

Quoted 18 lines
fs/nfs/dir.c +0 -0
index a73e2f8bd8ec..5d4a64555fa7 100644
@@ -444,7 +444,7 @@ void

nfs_force_use_readdirplus(struct
inode
*dir)
if (nfs_server_capable(dir, NFS_CAP_READDIRPLUS)
&&

          !list_empty(&nfsi->open_files)) {
          set_bit(NFS_INO_ADVISE_RDPLUS, &nfsi-
Quoted 1 line

flags);

fs/nfs/dir.c +1 -2
-        invalidate_mapping_pages(dir->i_mapping, 0,
-1);
+        nfs_zap_mapping(dir, dir->i_mapping);
      }
  }

This change is only reverting part of commit 79f687a3de9e. My problem
with that is as follows:

RFC1813 states that NFSv3 READDIRPLUS cookies and verifiers must
match
those returned by previous READDIRPLUS calls, and READDIR cookies
and
verifiers must match those returned by previous READDIR calls. It
says

Quoted 8 lines
Quoted 8 lines

nothing about being able to assume cookies from READDIR and READDIRPLUS
calls are interchangeable. So the only reason I can see for the
invalidate_mapping_pages() is to ensure that we do separate the
two
cookie caches.

If I understand your concern correctly that in NFSv3 the client must
maintain valid cookies and cookie verifiers when switching between
READDIR and READDIRPLUS, or vice sersa, then I think the current client
code handles this condition ok.

On the client, both READDIR and READDIRPLUS requests use the cookie
values
from the same cached pages of the directory so I don't think they can
be
out of sync when the client switches between READDIRPLUS and READDIR
requests for different nfs_readdir calls.

In fact, currently the first nfs_readdir uses READDIRPLUS's to fill
read
the entries and if there is no LOOKUP/GETATTR on one of the directory
entries then the client reverts to READDIR's for subsequent nfs_readdir
calls without invalidating any cached pages of the directory. If there
are LOOKUP/GETATTR done on one of the directory entries then nfs_advise_use_readdirplus is called which forces the client to use
READDIRPLUS again for the next nfs_readdir.

Quoted 5 lines
Quoted 4 lines

Unless the user mounts the export with 'nordirplus' option then the
client uses only READDIRs for all requests, no switching takes place.

I don't understand your point.

The original point was that the directory's page cache seems to be cleared a little too often (quite apart from switching between READDIRPLUS and READDIR).

Quoted 8 lines
Quoted 6 lines

I think Dai is saying that cache clearing is appropriate to defer when the directory's mtime has changed but the READ method remains the same. Otherwise repeatedly adding a new file to a very large directory that is being read can trigger a situation where the reading getdents loop never completes.

Fair enough, but the section of code he was touching with his patch should be only about switching between READDIR/PLUS methods.

The code that I'd to patch, nfs_force_use_readdirplus, is called when there is an attribute cache miss which causes the client to make sure READDIRPLUS is being used on the directory, regardless which READ method is currently being used using for the directory. So it's possible that the call to nfs_force_use_readdirplus will cause the next getdent/nfs_readdir to switch to from READDIR to READDIRPLUS.

Quoted 23 lines

The mtime monitoring happens elsewhere and is already set up to invalidate the cache only on opendir()/rewinddir().

Quoted 19 lines

My two cents Euro.

Quoted 16 lines

The issue is that
nfs_advise_use_readdirplus() can cause the behaviour to switch between
use of READDIRPLUS and use of READDIR from one syscall to getdents() to
the next.
If the client is using the same page cache, across those syscalls, then
it will end up caching a mixture of cookies. Furthermore, since the cookie that is used as an argument to the next call to READDIR/READDIRPLUS is taken from that page cache, then we can end up
calling READDIRPLUS with a cookie that came from READDIR and vice versa.

As I said, I'm not convinced that is legal in RFC1813 (NFSv3).

I think this is the contention point: the spec did not explicitly mention anything regarding mixing of cookies from READDIR & READDIRPLUS.

However, as I mentioned, the current client implementation already mixing cookies between READDIRPLUS and READDIR, everytime the user does a simple 'ls' on a large directory, without invalidating any mapping.

Also, as Chuck mentioned, we're not aware of any server implementation that has problems with this mixing of cookies.

-Dai

Quoted 44 lines
Quoted 44 lines
Quoted 4 lines

That is why we want to clear the page cache when we swap between use of
READDIR and use of READDIRPLUS for the case of NFSv3.

Just curious, are you aware of an NFSv3 server implementation that would have a problem with a client that mixes the cookies?

Quoted 21 lines
Quoted 21 lines

Thanks,
-Dai

Quoted 18 lines
Quoted 9 lines

OTOH, for NFSv4, there is no separate READDIRPLUS function, so
there
really does not appear to be any reason to clear the page cache
at
all
as we're switching between requesting attributes or not.

Sorry... To spell out my objection to this change more clearly:
The
call to nfs_zap_mapping() makes no sense in either case.
* It defers the cache invalidation until the next call to rewinddir()/opendir(), so it does not address the NFSv3
concern.
* It would appear to be entirely superfluous for the NFSv4
case.

Quoted 10 lines
Quoted 6 lines
Quoted 6 lines

So a change that might be acceptable would be to keep the existing
call
to invalidate_mapping_pages() for NFSv3, but to remove it for NFSv4.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]

--
Chuck Lever

#11 lore raw
Quoted 11 lines · Dai Ngo

I think this is the contention point: the spec did not explicitly mention anything regarding mixing of cookies from READDIR & READDIRPLUS.

However, as I mentioned, the current client implementation already mixing
cookies between READDIRPLUS and READDIR, everytime the user does a simple
'ls' on a large directory, without invalidating any mapping.

Quoted 4 lines

Also, as Chuck mentioned, we're not aware of any server implementation
that has problems with this mixing of cookies.

OK I did a little time warp and went back to the original emails around this behaviour:

https://linux-nfs.vger.kernel.narkive.com/O0Xhnqxe/readdir-vs-getattr

Part of the problem that needed solving at the time was that even when the directory and its contents were not changing, people were still needing to do reams of GETATTR calls in order to typically satisfy an 'ls -l' or even 'ls --color'. When we see that pattern, we want to switch from using GETATTR on all these files to using READDIRPLUS.

The cache invalidation was introduced in order to force the NFS client to do these READDIRPLUS calls so we avoid the GETATTRs.

So one optimisation we could definitely do is try to track the index of the last page our descriptor accessed on readdir(), and truncate only the remaining pages. That way we don't keep re-reading the beginning of the directory.

Signature · 3 lines
Trond Myklebust
Linux NFS client maintainer, Hammerspace
[email protected]