Skip to contentSkip to conversation

FLushing cached writes in nfs_getattr() and stat() delay

DISCUSSION Alex Sidorenko 4 participants 8 messages Nov 6, 2008 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello,

I am an HP engineer participating in L3 Linux support. Recently we have found that current design of nfs_getattr() might create huge delays in stat() on the file we are writing to (this is important for big files only, >2Gb).

The problem
-----------

Assuming that /nfs is an NFS-mounted FS:

1. In one shell, start

$ dd if=/dev/zero of=/nfs/dir/big bs=1G count=20

2. In another shell, start

$ ls -l /nfs/dir

or

$ ls -l /nfs/dir/big

'ls' does not return until the whole /nfs/dir/big is written.

Analysis
--------

Kernel 2.6.16 has introduced the following change in nfs_getattr():

NFS: Make stat() return updated mtimes after a write()

The SuS states that a call to write() will cause mtime to be updated on the file. In order to satisfy that requirement, we need to flush out any cached writes in nfs_getattr().
Speed things up slightly by not committing the writes.

+ /* Flush out writes to the server in order to update c/mtime */
+ nfs_sync_inode(inode, 0, 0, FLUSH_WAIT|FLUSH_NOCOMMIT);

Then later:

2.6.16-rc6:

http://www.linux-nfs.org/Linux-2.6.x/2.6.16-rc6/linux-2.6.16-99-fix_nfs_sync_inode_race.dif

/* Flush out writes to the server in order to update c/mtime */
- nfs_sync_inode(inode, 0, 0, FLUSH_WAIT|FLUSH_NOCOMMIT);
+ nfs_sync_inode_wait(inode, 0, 0, FLUSH_NOCOMMIT);

I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch stat() will return incorrect results, both for mtime and file length. Some applications may depend on stat() results being correct.

At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).

The problem is still there in 2.6.27. I am not sure what can be done to both reduce the stat() delay and guarantee reasonable stat() results.

It is interesting that with 'noac' stat() returns much faster (just 1-3s delay).

Best regards,
Alex

Signature · 5 lines
------------------------------------------------------------------
Alexandre Sidorenko             email: [email protected]
Global Solutions Engineering:   Unix Networking
Hewlett-Packard (Canada)
------------------------------------------------------------------
#1 lore raw

Hi Alex-

Quoted 62 lines · Alex Sidorenko

Hello,

I am an HP engineer participating in L3 Linux support. Recently we have found that current design of nfs_getattr() might create huge delays in stat() on the file we are writing to (this is important for big files only,

Quoted 1 line

2Gb).

The problem
-----------

Assuming that /nfs is an NFS-mounted FS:

1. In one shell, start

$ dd if=/dev/zero of=/nfs/dir/big bs=1G count=20

2. In another shell, start

$ ls -l /nfs/dir

or

$ ls -l /nfs/dir/big

'ls' does not return until the whole /nfs/dir/big is written.

Analysis
--------

Kernel 2.6.16 has introduced the following change in nfs_getattr():

NFS: Make stat() return updated mtimes after a write()

The SuS states that a call to write() will cause mtime to be
updated on
the file. In order to satisfy that requirement, we need to flush
out
any cached writes in nfs_getattr().
Speed things up slightly by not committing the writes.

(quoted diff) +2 -0
+	/* Flush out writes to the server in order to update c/mtime */
+	nfs_sync_inode(inode, 0, 0, FLUSH_WAIT|FLUSH_NOCOMMIT);

Then later:

2.6.16-rc6:

http://www.linux-nfs.org/Linux-2.6.x/2.6.16-rc6/linux-2.6.16-99-fix_nfs_sync_inode_race.dif

	/* Flush out writes to the server in order to update c/mtime */
(quoted diff) +1 -1
-	nfs_sync_inode(inode, 0, 0, FLUSH_WAIT|FLUSH_NOCOMMIT);
+	nfs_sync_inode_wait(inode, 0, 0, FLUSH_NOCOMMIT);

I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch stat() will return incorrect results, both for mtime and file length. Some applications may depend on stat() results being correct.

Quoted 7 lines

At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).

The problem is still there in 2.6.27. I am not sure what can be done to both reduce the stat() delay and guarantee reasonable stat() results.

The goal is to meet the POSIX requirement that the mtime of the returned stat(2) results must reflect the mtime of the latest application write(2) request. If the client is caching writes, then those must be flushed to the server first because only the server determines the file's mtime.

If the client limited its write cache to a few dozen megabytes, the delay during stat(2) would be nearly unnoticeable.

Quoted 2 lines

It is interesting that with 'noac' stat() returns much faster (just 1-3s delay).

That's because "noac" never caches writes on the client.

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#2 lore raw
In reply to Alex Sidorenko
Quoted 10 lines · Alex Sidorenko

I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch stat() will return incorrect results, both for mtime and file length. Some applications may depend on stat() results being correct.

At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).

In order to relax that requirement, we'd have to introduce some mechanism for the application to notify the filesystem that they don't care about strictly correct c/mtimes. As you noted above, returning incorrect mtimes may trip up some applications (backup applications, and mail readers are a couple of business critical cases that come to mind).

Quoted 5 lines

The problem is still there in 2.6.27. I am not sure what can be done to both reduce the stat() delay and guarantee reasonable stat() results.

It is interesting that with 'noac' stat() returns much faster (just 1-3s delay).

That would be because 'noac' enforces synchronous writes. If you don't care about the degraded write performance, you can do the same thing without all the extra getattr clutter that noac introduces, by simply mounting with -osync.

Cheers
Trond

A message that is not in the archive (sent off-list or not yet received): 1225997396.387.30.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#4 lore raw
Quoted 12 lines · Trond Myklebust
Quoted 10 lines · Alex Sidorenko

I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch stat() will return incorrect results, both for mtime and file length. Some applications may depend on stat() results being correct.

At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).

Quoted 5 lines

In order to relax that requirement, we'd have to introduce some mechanism for the application to notify the filesystem that they don't care about strictly correct c/mtimes. As you noted above, returning incorrect mtimes may trip up some applications (backup applications, and mail readers are a couple of business critical cases that come to mind).

I thought the preferred way to address this issue was to limit the per-file write cache size on the client. That is effectively what you do with "-osync" as you suggest below.

Especially on big systems, the client will delay writes until the cows come home. Then someone does a stat(2) and the lights dim... Really the client shouldn't need to cache that aggressively, and there are good reasons to keep a cap on it.

Quoted 10 lines
Quoted 5 lines

The problem is still there in 2.6.27. I am not sure what can be done to both reduce the stat() delay and guarantee reasonable stat() results.

It is interesting that with 'noac' stat() returns much faster (just 1-3s delay).

That would be because 'noac' enforces synchronous writes. If you don't care about the degraded write performance, you can do the same thing without all the extra getattr clutter that noac introduces, by simply mounting with -osync.

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#5 lore raw
In reply to a message not in the archive
Quoted 21 lines · Trond Myklebust
Quoted 10 lines · Alex Sidorenko

I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch stat() will return incorrect results, both for mtime and file length. Some applications may depend on stat() results being correct.

At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).

In order to relax that requirement, we'd have to introduce some mechanism for the application to notify the filesystem that they don't care about strictly correct c/mtimes. As you noted above, returning incorrect mtimes may trip up some applications (backup applications, and mail readers are a couple of business critical cases that come to mind).

Quoted 3 lines

The problem is still there in 2.6.27. I am not sure what can be done to both reduce the stat() delay and guarantee reasonable stat() results.

Quoted 7 lines
Quoted 2 lines

It is interesting that with 'noac' stat() returns much faster (just 1-3s delay).

That would be because 'noac' enforces synchronous writes. If you don't care about the degraded write performance, you can do the same thing without all the extra getattr clutter that noac introduces, by simply mounting with -osync.

Hi Trond,

In my experiments on 2.6.24 I saw practically no performance degradation while doing 'cp' of a 4Gb file with 'noac', with 'sync' the performance is really bad. And writes are still definitely ASYNC, here is what I see using Systemtap script on entry to rpc_execute

from /etc/mtab:

cats:/data /mnt nfs rw,udp,noac,hard,intr,addr=192.168.0.33 0 0

$ dd if=/dev/zero of=/mnt/win/big bs=100m count=1
Quoted 1 line

From stap output:

rpc_execute p_proc=7 WRITE qlen=0 prio=1 flags=0x1 --ts=4
rpc_execute p_proc=7 WRITE qlen=0 prio=1 flags=0x1
...

So we still have RPC_TASK_ASYNC set.

I did not check experimentally 'noac' on 2.6.27 but I still think that 'noac' does not make writes sync. nfs_commit_rpcsetup() still sets RPC_TASK_ASYNC by default and I don't see NFS_MOUNT_NOACL setting FLUSH_SYNC anywhere.

So I still don't quite understand why 'noac' eliminates the delay. Chuck Lever says that "noac" never caches writes on the client. Printing xprt->backlog->qlen in my experiments I can still see a significant backlog even with 'noac', e.g.

--ts=32
rpc_execute p_proc=7 WRITE qlen=3086 prio=1 flags=0x1

but 'stat' delay is just 1-2s.

Regards,
Alex

Signature · 5 lines
------------------------------------------------------------------
Alexandre Sidorenko             email: [email protected]
Global Solutions Engineering:   Unix Networking
Hewlett-Packard (Canada)
------------------------------------------------------------------
#6 lore raw
Quoted 49 lines · Alex Sidorenko
Quoted 21 lines · Trond Myklebust
Quoted 10 lines · Alex Sidorenko

I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch stat() will return incorrect results, both for mtime and file length. Some applications may depend on stat() results being correct.

At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).

In order to relax that requirement, we'd have to introduce some mechanism for the application to notify the filesystem that they don't care about strictly correct c/mtimes. As you noted above, returning incorrect mtimes may trip up some applications (backup applications, and mail readers are a couple of business critical cases that come to mind).

Quoted 3 lines

The problem is still there in 2.6.27. I am not sure what can be done to both reduce the stat() delay and guarantee reasonable stat() results.

Quoted 7 lines
Quoted 2 lines

It is interesting that with 'noac' stat() returns much faster (just 1-3s delay).

That would be because 'noac' enforces synchronous writes. If you don't care about the degraded write performance, you can do the same thing without all the extra getattr clutter that noac introduces, by simply mounting with -osync.

Hi Trond,

In my experiments on 2.6.24 I saw practically no performance degradation while doing 'cp' of a 4Gb file with 'noac', with 'sync' the performance is really bad. And writes are still definitely ASYNC, here is what I see using Systemtap script on entry to rpc_execute

from /etc/mtab:

cats:/data /mnt nfs rw,udp,noac,hard,intr,addr=192.168.0.33 0 0

$ dd if=/dev/zero of=/mnt/win/big bs=100m count=1

From stap output:
rpc_execute p_proc=7 WRITE qlen=0 prio=1 flags=0x1 --ts=4
rpc_execute p_proc=7 WRITE qlen=0 prio=1 flags=0x1
...

Quoted 5 lines

So we still have RPC_TASK_ASYNC set.

I did not check experimentally 'noac' on 2.6.27 but I still think that 'noac' does not make writes sync. nfs_commit_rpcsetup() still sets RPC_TASK_ASYNC by default and I don't see NFS_MOUNT_NOACL setting FLUSH_SYNC anywhere.

I repeat: 'noac' automatically sets the 'sync' flag, as you can see below:

# mount -t nfs -onoac fas960-1:/vol/san /mnt
# cat /proc/mounts | grep nfs
fas960-1:/vol/san /mnt nfs rw,sync,vers=3,rsize=65536,wsize=65536,namlen=255,acregmin=0,acregmax=0,acdirmin=0,acdirmax=0,hard,nointr,noac,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=141.211.133.16,mountvers=3,mountproto=tcp,addr=141.211.133.16 0 0

Trond

#7 lore raw
In reply to Alex Sidorenko
Quoted 30 lines · Alex Sidorenko
Quoted 21 lines · Trond Myklebust
Quoted 10 lines · Alex Sidorenko

I understand the reasoning behind that. From application point of view, NFS file/directory should behave the same as on local FS. If we have queued many writes, without this patch stat() will return incorrect results, both for mtime and file length. Some applications may depend on stat() results being correct.

At the same time, the fact that we have to wait forever while copying big files and doing 'ls -l' on that directory (or on the file being written) is not very good either (two HP customers have complained about this after migrating from RHEL4 to RHEL5).

In order to relax that requirement, we'd have to introduce some mechanism for the application to notify the filesystem that they don't care about strictly correct c/mtimes. As you noted above, returning incorrect mtimes may trip up some applications (backup applications, and mail readers are a couple of business critical cases that come to mind).

Quoted 3 lines

The problem is still there in 2.6.27. I am not sure what can be done to both reduce the stat() delay and guarantee reasonable stat() results.

Quoted 7 lines
Quoted 2 lines

It is interesting that with 'noac' stat() returns much faster (just 1-3s delay).

That would be because 'noac' enforces synchronous writes. If you don't care about the degraded write performance, you can do the same thing without all the extra getattr clutter that noac introduces, by simply mounting with -osync.

Quoted 6 lines

Hi Trond,

In my experiments on 2.6.24 I saw practically no performance degradation while doing 'cp' of a 4Gb file with 'noac', with 'sync' the performance is really bad. And writes are still definitely ASYNC, here is what I see using Systemtap script on entry to rpc_execute

There's a difference between an asynchronous RPC request, and an asynchronous write request.

An async RPC means the process doesn't wait for the request to finish, it can perform other housekeeping.

An async write means that the client delays sending NFS writes, maintaining the dirty data in its memory. It can send the NFS write requests by means of an async RPC if it wishes. A synchronous write means that the client will block the application until the server has replied that the dirty data is on the server's disk.

Quoted 6 lines

from /etc/mtab:

cats:/data /mnt nfs rw,udp,noac,hard,intr,addr=192.168.0.33 0 0

$ dd if=/dev/zero of=/mnt/win/big bs=100m count=1
Quoted 7 lines

From stap output:
rpc_execute p_proc=7 WRITE qlen=0 prio=1 flags=0x1 --ts=4
rpc_execute p_proc=7 WRITE qlen=0 prio=1 flags=0x1
...

So we still have RPC_TASK_ASYNC set.

See above.

Quoted 3 lines

I did not check experimentally 'noac' on 2.6.27 but I still think that 'noac' does not make writes sync. nfs_commit_rpcsetup() still sets RPC_TASK_ASYNC by default and I don't see NFS_MOUNT_NOACL setting FLUSH_SYNC anywhere.

Again, RPC_TASK_ASYNC has nothing to do with whether the application is blocked until the server says the write is permanent.

Quoted 23 lines

So I still don't quite understand why 'noac' eliminates the delay. Chuck Lever says that "noac" never caches writes on the client. Printing xprt->backlog->qlen in my experiments I can still see a significant backlog even with 'noac', e.g.

--ts=32
rpc_execute p_proc=7 WRITE qlen=3086 prio=1 flags=0x1

but 'stat' delay is just 1-2s.

Regards,
Alex

Signature · 9 lines
------------------------------------------------------------------
Alexandre Sidorenko             email: [email protected]
Global Solutions Engineering:   Unix Networking
Hewlett-Packard (Canada)
------------------------------------------------------------------
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
#8 lore raw
Quoted 4 lines · Chuck Lever

There's a difference between an asynchronous RPC request, and an asynchronous write request.

An async RPC means the process doesn't wait for the request to finish=

,

Quoted 4 lines

it can perform other housekeeping.

An async write means that the client delays sending NFS writes, maintaining the dirty data in its memory. =A0It can send the NFS writ=

e

Quoted 1 line

requests by means of an async RPC if it wishes. =A0A synchronous writ=

e

Quoted 2 lines

means that the client will block the application until the server has replied that the dirty data is on the server's disk.

Thank you for clarification. I knew what RPC-async means but for some r= eason=20
forgot what SYNC-write means, sorry for confusion. And yes, I should ha= ve=20
looked at /proc/mounts instead of /etc/mtab

Regards,
Alex

--=20
------------------------------------------------------------------
Alexandre Sidorenko email: [email protected]
Global Solutions Engineering: Unix Networking
Hewlett-Packard (Canada)
------------------------------------------------------------------