PROBLEM: knfsd misses occasional writes
Subsystems (2)
Hello everyone,
When copying lots of small files from multiple NFS clients to a kNFSd filesystem (i.e. doing backup of a cluster), exported with sync', I find that some few files (1 out of 1000) were silently truncated to zero size when checking locally with ls' (the clients reported total success). With `asynch' instead, all files were correctly copied.
I have seen this behaviour in 2.4.17 (UP and SMP builds, UP hardware) as well as 2.4.18, when using the NFSv2 protocol. I have not tried 2.5.x and NFSv3 yet. The full /etc/exports line is:
/opt/telorb 172.16.0.0/255.255.0.0(rw,sync,no_wdelay)
Removing `no_wdelay' makes no difference.
The clients are all 2.4.17, and the relevant .config lines (for both server and clients) are:
CONFIG_NFS_FS=y
CONFIG_NFS_V3=y
CONFIG_ROOT_NFS=y
CONFIG_NFSD=y
CONFIG_NFSD_V3=y
CONFIG_SUNRPC=y
CONFIG_LOCKD=y
CONFIG_LOCKD_V4=y
Reading the source (fs/nfsd/*) seems to show that knfsd tries to do the right thing.
/Sverker Wiberg
Quoted 8 lines · Neil Brown
Quoted 6 lines · On Wednesday May 15, [email protected] wrote:
Hello everyone,
When copying lots of small files from multiple NFS clients to a kNFSd filesystem (i.e. doing backup of a cluster), exported with `sync', I find that some few files (1 out of 1000) were silently truncated to zero
^^^^^^^^ no errors reported
Quoted 8 lines
Quoted 2 lines
size when checking locally with
ls' (the clients reported total success). Withasynch' instead, all files were correctly copied.How are you mounting the file systems on the clients?
The symptoms sound exactly like you are using "soft" mounts. "soft" is a very bad mount option. Use "hard".If you aren't using "soft", let me know and I will look harder.
Errrm, I am using "soft" mounts, as I (we) want the clients to survive server restarts.
But shouldn't those timeouts become errors over at the clients?
/Sverker
Quoted 2 lines · Neil Brown
On Thursday May 16, [email protected] wrote:
[on soft mount timeouts]
Quoted 4 lines
Quoted 1 line
But shouldn't those timeouts become errors over at the clients?
Yes... but "write" won't see an error. Only 'fsync' or maybe 'close', and many applications ignore errors from these operations.
How come? Isn't the client side innately synchronous (as RPC clients in general)?
Or is this one of thost thing that are now done differently?
/Sverker
Quoted 12 lines · Sverker Wiberg
Quoted 4 lines · Neil Brown
Quoted 3 lines · On Wednesday May 15, [email protected] wrote:
When copying lots of small files from multiple NFS clients to a kNFSd filesystem (i.e. doing backup of a cluster), exported with `sync', I find that some few files (1 out of 1000) were silently truncated to zero
^^^^^^^^ no errors reported
Quoted 5 lines
How are you mounting the file systems on the clients?
The symptoms sound exactly like you are using "soft" mounts. "soft" is a very bad mount option. Use "hard".
Quoted 5 lines
Quoted 1 line
If you aren't using "soft", let me know and I will look harder.
Errrm, I am using "soft" mounts, as I (we) want the clients to survive server restarts.
But shouldn't those timeouts become errors over at the clients?
I have seen this too, with a file system exported with rw,no_root_squash and mounted hard,intr. We were running vanilla 2.4.18 on the server and clients. We have a text file on the server serving to record employees' time, and one day the time clock file remained a text file but was truncated to zero. All further punch ins/punch outs did not record in the truncated file (user names, dates, and times should have appended). Deleting and recreating the text file on a client returned behavior to normal. No error messages whatsoever, and it has worked fine for two weeks as we watch for the behavior to repeat.
Regards,
Gary S.
Quoted 4 lines · G Sandine
[...]and one day the time clock file remained a text file but was truncated to zero. All further punch ins/punch outs did not record in the truncated file (user names, dates, and times should have appended).
Sounds as you've got the file's ownership and perms clobbered. I'll check if that happens over here as well.
Quoted 3 lines
Deleting and recreating the text file on a client returned behavior to normal. No error messages whatsoever, and it has worked fine for two weeks as we watch for the behavior to repeat.
Then we're (cough!) luckier over here, we can recreate the problem in about an hour. But at least it's nice to know you're no alone.
[To the lkml:]
Over here, we started to log the conversations, and saw the client opening a file, writing 272 bytes into it (one write), and then closing it, with the server replying full success all the time. printk()'s in knfsd and the vfs's generic_write() also reported that 272 bytes had been successfully written. Yet the file was truncated.
We switched from soft to hard mount: It didn't help. We are now experimenting with disabling SCSI's disconnect/reconnect feature. Are there any more straws to grasp at?
/Sverker
Re: PROBLEM: knfsd misses occasional writes (w/ WORKAROUND)
Quoted 9 lines · Sverker Wiberg
Over here, we started to log the conversations, and saw the client opening a file, writing 272 bytes into it (one write), and then closing it, with the server replying full success all the time. printk()'s in knfsd and the vfs's
generic_write()also reported that 272 bytes had been successfully written. Yet the file was truncated.We switched from soft to hard mount: It didn't help. We are now experimenting with disabling SCSI's disconnect/reconnect feature. Are there any more straws to grasp at?
With one single knfsd thread running, the problem went away (for a price in performance). This seems to indicate there is some kind of race between the knfsd threads.
/Sverker
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
_______________________________________________________________
Hundreds of nodes, one monster rendering program.
Now that’s a super model! Visit http://clustering.foundries.sf.net/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Re: PROBLEM: knfsd misses occasional writes (w/ WORKAROUND)
In reply to Sverker WibergQuoted 9 lines · Sverker Wiberg
Over here, we started to log the conversations, and saw the client opening a file, writing 272 bytes into it (one write), and then closing it, with the server replying full success all the time. printk()'s in knfsd and the vfs's
generic_write()also reported that 272 bytes had been successfully written. Yet the file was truncated.We switched from soft to hard mount: It didn't help. We are now experimenting with disabling SCSI's disconnect/reconnect feature. Are there any more straws to grasp at?
With one single knfsd thread running, the problem went away (for a price in performance). This seems to indicate there is some kind of race between the knfsd threads.
/Sverker
Quoted 8 lines · On Wednesday May 15, [email protected] wrote:
Hello everyone,
When copying lots of small files from multiple NFS clients to a kNFSd filesystem (i.e. doing backup of a cluster), exported with
sync', I find that some few files (1 out of 1000) were silently truncated to zero size when checking locally withls' (the clients reported total success). With `asynch' instead, all files were correctly copied.
How are you mounting the file systems on the clients?
The symptoms sound exactly like you are using "soft" mounts. "soft" is a very bad mount option. Use "hard".
If you aren't using "soft", let me know and I will look harder.
NeilBrown
Quoted 30 lines
I have seen this behaviour in 2.4.17 (UP and SMP builds, UP hardware) as well as 2.4.18, when using the NFSv2 protocol. I have not tried 2.5.x and NFSv3 yet. The full
/etc/exportsline is:/opt/telorb 172.16.0.0/255.255.0.0(rw,sync,no_wdelay)Removing `no_wdelay' makes no difference.
The clients are all 2.4.17, and the relevant .config lines (for both server and clients) are:
CONFIG_NFS_FS=y CONFIG_NFS_V3=y CONFIG_ROOT_NFS=y CONFIG_NFSD=y CONFIG_NFSD_V3=y CONFIG_SUNRPC=y CONFIG_LOCKD=y CONFIG_LOCKD_V4=yReading the source (
fs/nfsd/*) seems to show that knfsd tries to do the right thing./Sverker Wiberg
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/