FLushing cached writes in nfs_getattr() and stat() delay
Subsystems (2)
Hello,
I am an HP engineer participating in L3 Linux support. Recently we have found that current design of nfs_getattr() might create huge delays in stat() on the file we are writing to (this is important for big files only, >2Gb).
The problem
-----------
Assuming that /nfs is an NFS-mounted FS:
1. In one shell, start
$ dd if=/dev/zero of=/nfs/dir/big bs=1G count=20
2. In another shell, start
$ ls -l /nfs/dir
or
$ ls -l /nfs/dir/big
'ls' does not return until the whole /nfs/dir/big is written.
Analysis
--------
Kernel 2.6.16 has introduced the following change in nfs_getattr():
NFS: Make stat() return updated mtimes after a write()
The SuS states that a call to write() will cause mtime to be updated on the file. In order to satisfy that requirement, we need to flush out any cached writes in nfs_getattr().
Speed things up slightly by not committing the writes.
+ /* Flush out writes to the server in order to update c/mtime */
+ nfs_sync_inode(inode, 0, 0, FLUSH_WAIT|FLUSH_NOCOMMIT);
Then later:
2.6.16-rc6:
http://www.linux-nfs.org/Linux-2.6.x/2.6.16-rc6/linux-2.6.16-99-fix_nfs_sync_inode_race.dif
/* Flush out writes to the server in order to update c/mtime */
- nfs_sync_inode(inode, 0, 0, FLUSH_WAIT|FLUSH_NOCOMMIT);
+ nfs_sync_inode_wait(inode, 0, 0, FLUSH_NOCOMMIT);
I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch stat() will return incorrect results, both for mtime and file length. Some applications may depend on stat() results being correct.
At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).
The problem is still there in 2.6.27. I am not sure what can be done to both reduce the stat() delay and guarantee reasonable stat() results.
It is interesting that with 'noac' stat() returns much faster (just 1-3s delay).
Best regards,
Alex
Signature · 5 lines
------------------------------------------------------------------ Alexandre Sidorenko email: [email protected] Global Solutions Engineering: Unix Networking Hewlett-Packard (Canada) ------------------------------------------------------------------
Hi Alex-
Quoted 62 lines · Alex Sidorenko
Hello,
I am an HP engineer participating in L3 Linux support. Recently we have found that current design of
nfs_getattr()might create huge delays instat()on the file we are writing to (this is important for big files only,Quoted 1 line
2Gb).
The problem
-----------Assuming that /nfs is an NFS-mounted FS:
1. In one shell, start
$ dd if=/dev/zero of=/nfs/dir/big bs=1G count=202. In another shell, start
$ ls -l /nfs/diror
$ ls -l /nfs/dir/big'ls' does not return until the whole /nfs/dir/big is written.
Analysis
--------Kernel 2.6.16 has introduced the following change in
nfs_getattr():NFS: Make
stat()return updated mtimes after awrite()The SuS states that a call to
write()will cause mtime to be
updated on
the file. In order to satisfy that requirement, we need to flush
out
any cached writes innfs_getattr().
Speed things up slightly by not committing the writes.Signed-off-by Trond Myklebust <[email protected]>(quoted diff) +2
-0+ /* Flush out writes to the server in order to update c/mtime */ + nfs_sync_inode(inode, 0, 0, FLUSH_WAIT|FLUSH_NOCOMMIT);Then later:
2.6.16-rc6:
http://www.linux-nfs.org/Linux-2.6.x/2.6.16-rc6/linux-2.6.16-99-fix_nfs_sync_inode_race.dif
/* Flush out writes to the server in order to update c/mtime */(quoted diff) +1
-1- nfs_sync_inode(inode, 0, 0, FLUSH_WAIT|FLUSH_NOCOMMIT); + nfs_sync_inode_wait(inode, 0, 0, FLUSH_NOCOMMIT);I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch
stat()will return incorrect results, both for mtime and file length. Some applications may depend onstat()results being correct.
Quoted 7 lines
At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).
The problem is still there in 2.6.27. I am not sure what can be done to both reduce the
stat()delay and guarantee reasonablestat()results.
The goal is to meet the POSIX requirement that the mtime of the returned stat(2) results must reflect the mtime of the latest application write(2) request. If the client is caching writes, then those must be flushed to the server first because only the server determines the file's mtime.
If the client limited its write cache to a few dozen megabytes, the delay during stat(2) would be nearly unnoticeable.
Quoted 2 lines
It is interesting that with 'noac'
stat()returns much faster (just 1-3s delay).
That's because "noac" never caches writes on the client.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 10 lines · Alex Sidorenko
I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch
stat()will return incorrect results, both for mtime and file length. Some applications may depend onstat()results being correct.At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).
In order to relax that requirement, we'd have to introduce some mechanism for the application to notify the filesystem that they don't care about strictly correct c/mtimes. As you noted above, returning incorrect mtimes may trip up some applications (backup applications, and mail readers are a couple of business critical cases that come to mind).
Quoted 5 lines
The problem is still there in 2.6.27. I am not sure what can be done to both reduce the
stat()delay and guarantee reasonablestat()results.It is interesting that with 'noac'
stat()returns much faster (just 1-3s delay).
That would be because 'noac' enforces synchronous writes. If you don't care about the degraded write performance, you can do the same thing without all the extra getattr clutter that noac introduces, by simply mounting with -osync.
Cheers
Trond
A message that is not in the archive (sent off-list or not yet received): 1225997396.387.30.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org
Quoted 12 lines · Trond Myklebust
Quoted 10 lines · Alex Sidorenko
I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch
stat()will return incorrect results, both for mtime and file length. Some applications may depend onstat()results being correct.At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).
Quoted 5 lines
In order to relax that requirement, we'd have to introduce some mechanism for the application to notify the filesystem that they don't care about strictly correct c/mtimes. As you noted above, returning incorrect mtimes may trip up some applications (backup applications, and mail readers are a couple of business critical cases that come to mind).
I thought the preferred way to address this issue was to limit the per-file write cache size on the client. That is effectively what you do with "-osync" as you suggest below.
Especially on big systems, the client will delay writes until the cows come home. Then someone does a stat(2) and the lights dim... Really the client shouldn't need to cache that aggressively, and there are good reasons to keep a cap on it.
Quoted 10 lines
Quoted 5 lines
The problem is still there in 2.6.27. I am not sure what can be done to both reduce the
stat()delay and guarantee reasonablestat()results.It is interesting that with 'noac'
stat()returns much faster (just 1-3s delay).That would be because 'noac' enforces synchronous writes. If you don't care about the degraded write performance, you can do the same thing without all the extra getattr clutter that noac introduces, by simply mounting with -osync.
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 21 lines · Trond Myklebust
Quoted 10 lines · Alex Sidorenko
I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch
stat()will return incorrect results, both for mtime and file length. Some applications may depend onstat()results being correct.At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).
In order to relax that requirement, we'd have to introduce some mechanism for the application to notify the filesystem that they don't care about strictly correct c/mtimes. As you noted above, returning incorrect mtimes may trip up some applications (backup applications, and mail readers are a couple of business critical cases that come to mind).
Quoted 3 lines
The problem is still there in 2.6.27. I am not sure what can be done to both reduce the
stat()delay and guarantee reasonablestat()results.
Quoted 7 lines
Quoted 2 lines
It is interesting that with 'noac'
stat()returns much faster (just 1-3s delay).That would be because 'noac' enforces synchronous writes. If you don't care about the degraded write performance, you can do the same thing without all the extra getattr clutter that noac introduces, by simply mounting with -osync.
Hi Trond,
In my experiments on 2.6.24 I saw practically no performance degradation while doing 'cp' of a 4Gb file with 'noac', with 'sync' the performance is really bad. And writes are still definitely ASYNC, here is what I see using Systemtap script on entry to rpc_execute
from /etc/mtab:
cats:/data /mnt nfs rw,udp,noac,hard,intr,addr=192.168.0.33 0 0
$ dd if=/dev/zero of=/mnt/win/big bs=100m count=1
Quoted 1 line
From stap output:
rpc_execute p_proc=7 WRITE qlen=0 prio=1 flags=0x1 --ts=4rpc_execute p_proc=7 WRITE qlen=0 prio=1 flags=0x1
...
So we still have RPC_TASK_ASYNC set.
I did not check experimentally 'noac' on 2.6.27 but I still think that 'noac' does not make writes sync. nfs_commit_rpcsetup() still sets RPC_TASK_ASYNC by default and I don't see NFS_MOUNT_NOACL setting FLUSH_SYNC anywhere.
So I still don't quite understand why 'noac' eliminates the delay. Chuck Lever says that "noac" never caches writes on the client. Printing xprt->backlog->qlen in my experiments I can still see a significant backlog even with 'noac', e.g.
--ts=32rpc_execute p_proc=7 WRITE qlen=3086 prio=1 flags=0x1
but 'stat' delay is just 1-2s.
Regards,
Alex
Signature · 5 lines
------------------------------------------------------------------ Alexandre Sidorenko email: [email protected] Global Solutions Engineering: Unix Networking Hewlett-Packard (Canada) ------------------------------------------------------------------
Quoted 49 lines · Alex Sidorenko
Quoted 21 lines · Trond Myklebust
Quoted 10 lines · Alex Sidorenko
I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch
stat()will return incorrect results, both for mtime and file length. Some applications may depend onstat()results being correct.At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).
In order to relax that requirement, we'd have to introduce some mechanism for the application to notify the filesystem that they don't care about strictly correct c/mtimes. As you noted above, returning incorrect mtimes may trip up some applications (backup applications, and mail readers are a couple of business critical cases that come to mind).
Quoted 3 lines
The problem is still there in 2.6.27. I am not sure what can be done to both reduce the
stat()delay and guarantee reasonablestat()results.Quoted 7 lines
Quoted 2 lines
It is interesting that with 'noac'
stat()returns much faster (just 1-3s delay).That would be because 'noac' enforces synchronous writes. If you don't care about the degraded write performance, you can do the same thing without all the extra getattr clutter that noac introduces, by simply mounting with -osync.
Hi Trond,
In my experiments on 2.6.24 I saw practically no performance degradation while doing 'cp' of a 4Gb file with 'noac', with 'sync' the performance is really bad. And writes are still definitely ASYNC, here is what I see using Systemtap script on entry to
rpc_executefrom
/etc/mtab:cats:/data /mnt nfs rw,udp,noac,hard,intr,addr=192.168.0.33 0 0
$ dd if=/dev/zero of=/mnt/win/big bs=100m count=1From stap output:
rpc_executep_proc=7 WRITE qlen=0 prio=1 flags=0x1 --ts=4rpc_executep_proc=7 WRITE qlen=0 prio=1 flags=0x1
...
Quoted 5 lines
So we still have
RPC_TASK_ASYNCset.I did not check experimentally 'noac' on 2.6.27 but I still think that 'noac' does not make writes sync.
nfs_commit_rpcsetup()still setsRPC_TASK_ASYNCby default and I don't seeNFS_MOUNT_NOACLsettingFLUSH_SYNCanywhere.
I repeat: 'noac' automatically sets the 'sync' flag, as you can see below:
# mount -t nfs -onoac fas960-1:/vol/san /mnt # cat /proc/mounts | grep nfs fas960-1:/vol/san /mnt nfs rw,sync,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,nointr,noac,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=141.211.133.16,mountvers=3,mountproto=tcp,addr=141.211.133.16 0 0
Trond
Quoted 30 lines · Alex Sidorenko
Quoted 21 lines · Trond Myklebust
Quoted 10 lines · Alex Sidorenko
I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch
stat()will return incorrect results, both for mtime and file length. Some applications may depend onstat()results being correct.At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).
In order to relax that requirement, we'd have to introduce some mechanism for the application to notify the filesystem that they don't care about strictly correct c/mtimes. As you noted above, returning incorrect mtimes may trip up some applications (backup applications, and mail readers are a couple of business critical cases that come to mind).
Quoted 3 lines
The problem is still there in 2.6.27. I am not sure what can be done to both reduce the
stat()delay and guarantee reasonablestat()results.Quoted 7 lines
Quoted 2 lines
It is interesting that with 'noac'
stat()returns much faster (just 1-3s delay).That would be because 'noac' enforces synchronous writes. If you don't care about the degraded write performance, you can do the same thing without all the extra getattr clutter that noac introduces, by simply mounting with -osync.
Quoted 6 lines
Hi Trond,
In my experiments on 2.6.24 I saw practically no performance degradation while doing 'cp' of a 4Gb file with 'noac', with 'sync' the performance is really bad. And writes are still definitely ASYNC, here is what I see using Systemtap script on entry to
rpc_execute
There's a difference between an asynchronous RPC request, and an asynchronous write request.
An async RPC means the process doesn't wait for the request to finish, it can perform other housekeeping.
An async write means that the client delays sending NFS writes, maintaining the dirty data in its memory. It can send the NFS write requests by means of an async RPC if it wishes. A synchronous write means that the client will block the application until the server has replied that the dirty data is on the server's disk.
Quoted 6 lines
from
/etc/mtab:cats:/data /mnt nfs rw,udp,noac,hard,intr,addr=192.168.0.33 0 0
$ dd if=/dev/zero of=/mnt/win/big bs=100m count=1
Quoted 7 lines
From stap output:
rpc_executep_proc=7 WRITE qlen=0 prio=1 flags=0x1 --ts=4rpc_executep_proc=7 WRITE qlen=0 prio=1 flags=0x1
...So we still have
RPC_TASK_ASYNCset.
See above.
Quoted 3 lines
I did not check experimentally 'noac' on 2.6.27 but I still think that 'noac' does not make writes sync.
nfs_commit_rpcsetup()still setsRPC_TASK_ASYNCby default and I don't seeNFS_MOUNT_NOACLsettingFLUSH_SYNCanywhere.
Again, RPC_TASK_ASYNC has nothing to do with whether the application is blocked until the server says the write is permanent.
Quoted 23 lines
So I still don't quite understand why 'noac' eliminates the delay. Chuck Lever says that "noac" never caches writes on the client. Printing
xprt->backlog->qlenin my experiments I can still see a significant backlog even with 'noac', e.g.--ts=32
rpc_executep_proc=7 WRITE qlen=3086 prio=1 flags=0x1but 'stat' delay is just 1-2s.
Regards,
AlexSignature · 9 lines
------------------------------------------------------------------ Alexandre Sidorenko email: [email protected] Global Solutions Engineering: Unix Networking Hewlett-Packard (Canada) ------------------------------------------------------------------ -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Chuck Lever chuck[dot]lever[at]oracle[dot]com
Quoted 4 lines · Chuck Lever
There's a difference between an asynchronous RPC request, and an asynchronous write request.
An async RPC means the process doesn't wait for the request to finish=
,
Quoted 4 lines
it can perform other housekeeping.
An async write means that the client delays sending NFS writes, maintaining the dirty data in its memory. =A0It can send the NFS writ=
e
Quoted 1 line
requests by means of an async RPC if it wishes. =A0A synchronous writ=
e
Quoted 2 lines
means that the client will block the application until the server has replied that the dirty data is on the server's disk.
Thank you for clarification. I knew what RPC-async means but for some r= eason=20
forgot what SYNC-write means, sorry for confusion. And yes, I should ha= ve=20
looked at /proc/mounts instead of /etc/mtab
Regards,
Alex
--=20
------------------------------------------------------------------
Alexandre Sidorenko email: [email protected]
Global Solutions Engineering: Unix Networking
Hewlett-Packard (Canada)
------------------------------------------------------------------