knsfd - files don't sync to disk

DISCUSSION Jörgen Karlsson 3 participants 4 messages May 16, 2002 – May 17, 2002

Subsystems (2)

NFS client NFS server

lore

Share & tools

lore

Jörgen Karlsson

author

2002-05-16 19:36 UTC #0 lore raw

Hi,

We have a serious problem with knfsd and kernel 2.4.17.

When doing a database backup in our linux cluster we have noticed that a few files are not written to disk properly. The files have zero file size when checked with the 'ls' command

Several thousands of files are written to the nfs server during a short period of time. Average file size is 300-400 bytes.

We have noticed that usually 1-2 out of 3000 files will have their sizes truncated to 0.

File system is exported with (rw,sync)

The disk filesystem is ext2.

E.g. this is what is happening:
- client writes 272 bytes to the server
- mm/file.c:generic_file_write() returns 272 bytes written
- fs/nfsd/vfs.c:nfs_write() increments nfsdstats.io_write +=272 and returns that the write was sucessful.
- knfsd returns to client that 272 bytes was written.
- when checking with 'ls' command on server the file size is zero.

Apparently the nfs server lies to the client and the files are not properly synced/written to disk.

Setting no_wdelay makes no difference.

When running with async set (or sync removed) the problem disappears (no files have their sizes truncated to zero).

The nfs server is a PIII-700/ 1GB RAM

Any ideas what is going on ?

/Jörgen Karlsson

_______________________________________________________________

Have big pipes? SourceForge.net is looking for download mirrors. We supply the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Neil Brown

2002-05-17 11:58 UTC #1 lore raw

Are you using NFSv3?

Does this patch:
http://www.cse.unsw.edu.au/~neilb/patches/linux-stable/2.4.19-pre5/patch-I-NfsdVfsTidyup

make a difference?

NeilBrown

Quoted 29 lines · On Thursday May 16, [email protected] wrote:

Hi,
We have a serious problem with knfsd and kernel 2.4.17.
When doing a database backup in our linux cluster we have noticed that a few files are not written to disk properly. The files have zero file size when checked with the 'ls' command
Several thousands of files are written to the nfs server during a short period of time. Average file size is 300-400 bytes.
We have noticed that usually 1-2 out of 3000 files will have their sizes truncated to 0.
File system is exported with (rw,sync)
The disk filesystem is ext2.
E.g. this is what is happening:
- client writes 272 bytes to the server
- mm/file.c:generic_file_write() returns 272 bytes written
- fs/nfsd/vfs.c:nfs_write() increments nfsdstats.io_write +=272 and returns that the write was sucessful.
- knfsd returns to client that 272 bytes was written.
- when checking with 'ls' command on server the file size is zero.
Apparently the nfs server lies to the client and the files are not properly synced/written to disk.

Quoted 8 lines

Setting no_wdelay makes no difference.
When running with async set (or sync removed) the problem disappears (no files have their sizes truncated to zero).
The nfs server is a PIII-700/ 1GB RAM
Any ideas what is going on ?

_______________________________________________________________

Jörgen Karlsson

2002-05-17 17:11 UTC #2 lore raw

We are using NFSv2.

We have found a temporary workaround by reducing the number of nfs threads to 1.

We have a 30 processor cluster that gave zero size files everytime we made a backup. By reducing the number of threads from 8 to 1 we were able to make 25 backups without any problem. Switching back to 8 threads we immediately got the corrupted files back.

Maybe some down() and up() are missing.......

If you think the the patch may fix it for v2 as well as v3 we can try the patch next week.

Can running with only one thread cause any performance problems (with sync the bottleneck probably is disk io....)?

/Jorgen Karlsson

Quoted 50 lines · Neil Brown

Are you using NFSv3?
Does this patch:
http://www.cse.unsw.edu.au/~neilb/patches/linux-stable/2.4.19-pre5/patch-I-NfsdVfsTidyup
make a difference?
NeilBrown
Quoted 29 lines · On Thursday May 16, [email protected] wrote:
Hi,
We have a serious problem with knfsd and kernel 2.4.17.
When doing a database backup in our linux cluster we have noticed that a few files are not written to disk properly. The files have zero file size when checked with the 'ls' command
Several thousands of files are written to the nfs server during a short period of time. Average file size is 300-400 bytes.
We have noticed that usually 1-2 out of 3000 files will have their sizes truncated to 0.
File system is exported with (rw,sync)
The disk filesystem is ext2.
E.g. this is what is happening:
- client writes 272 bytes to the server
- mm/file.c:generic_file_write() returns 272 bytes written
- fs/nfsd/vfs.c:nfs_write() increments nfsdstats.io_write +=272 and returns that the write was sucessful.
- knfsd returns to client that 272 bytes was written.
- when checking with 'ls' command on server the file size is zero.
Apparently the nfs server lies to the client and the files are not properly synced/written to disk.
Quoted 9 lines
Setting no_wdelay makes no difference.
When running with async set (or sync removed) the problem disappears (no files have their sizes truncated to zero).
The nfs server is a PIII-700/ 1GB RAM
Any ideas what is going on ?

Quoted 8 lines

_______________________________________________________________
Have big pipes? SourceForge.net is looking for download mirrors. We supply the hardware. You get the recognition. Email Us: [email protected]
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

_______________________________________________________________

Hundreds of nodes, one monster rendering program.
Now that’s a super model! Visit http://clustering.foundries.sf.net/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Neil Brown

2002-05-17 20:56 UTC #3 lore raw

Quoted 10 lines · On Friday May 17, [email protected] wrote:

We are using NFSv2.
We have found a temporary workaround by reducing the number of nfs threads to 1.
We have a 30 processor cluster that gave zero size files everytime we made a backup. By reducing the number of threads from 8 to 1 we were able to make 25 backups without any problem. Switching back to 8 threads we immediately got the corrupted files back.

Quoted 4 lines

Maybe some down() and up() are missing.......
If you think the the patch may fix it for v2 as well as v3 we can try the patch next week.

The patch would not affect NFSv2 with the no_wdelay export option, so if that combination still have problems, don't bother with the patch.

Quoted 3 lines

Can running with only one thread cause any performance problems (with sync the bottleneck probably is disk io....)?

The more threads, the more concurrent IO you can be waiting on, so I would definately expect a reduction in performance.

The whole scenario is very odd..
You have confirmed that nfsd does actually write data to the file, but if you look afterwards, the file is empty.
This suggests that one of:
It wrote to the wrong file by mistake
A subsequent "truncate" request was received
The filesystem lied when it said that it had written data.

Can you get a complete tcpdump trace of NFS activity on the server and let me have a look at it?
Something like

tcpdump -w /var/tmp/file -s 1500 port 2049
bzip2 /var/tmp/file

on the server, and let me know where to pick it up??

NeilBrown

_______________________________________________________________