nfs/mmap/rename file corruption

There is a fairly easily reproducible bug in NFS in 2.4.22 that can cause files to read back as full of nulls. I have a tcpdump that shows what is going wrong.

Gavrie Philipson reported corruption happening when distcc and ccache are used together with the cache on NFS.

http://lists.samba.org/pipermail/distcc/2003q3/001556.html

To reproduce the bug you need to just install ccache 2.2 and distcc 2.10.1. Set CCACHE_DIR to an empty directory on an NFS filesystem mounted with default/rw options. Build a file with a command like this:

ccache distcc -c ./hello.c

The first (only the first) time that you run this, the output file (hello.o) will be the correct size, but contain only \0 bytes.

What is basically happening here is

- ccache runs distcc with output to a temporary file
- distcc opens, mmaps, writes to, munmaps, and closes the temporary file
- distcc exits
- ccache renames the temporary file to its proper location in the ccache
- ccache opens the file read only, and reads from it

ccache ought to see the proper contents as written by mmap, but when the cache is on NFS it just sees \0s. It works correctly and reliably on reiserfs and ext3. However, if you look at the file ccache was trying to read a second later then it seems to have the right contents.

I tried writing a standalone test case but I couldn't reproduce it, perhaps because of some timing issue. It is quite reproducible both on my machine and Gavrie's.

If distcc is configured to not use mmap for writing, the problem is hidden.

A tcpdump of the problem is available here:

http://distcc.samba.org/ftp/distcc/misc/mmap-bug/nfs-20030827T1351.pcap.gz

Here are the significant bits:

frame 79

renames tmp.hash.vexed.7897.o to the final object filename, cbfc5ca42b1a693a5bca9bb8b23c5b-17387

frame 105

also frame 107

look up a filehandle for the final object filename, and gets the hash 0xed8222404

frame 115

reads back from the final object file, 0xed8222404

frame 116

is the reply to the read and it is full of nulls

frame 127

writes the ELF output into the temporary object file, tmp.hash.vexed.7897.o, which has file hash 0xf27c2204.

The problem is that the NFS client tries to read from the destination file before it has written to the temporary file! Frame 127 is far too late.

It seems to me like there are two possible solutions: either flush out all cached data for a file before it's renamed, or make the rename smart enough to 'take over' any data cached under an old name. To me the first seems more robust if a little slower.

You can see something similar going on in this NFS log:

http://distcc.samba.org/ftp/distcc/misc/mmap-bug/nfsdebug-20030827T1609.log.gz

The flush(b/49777) call comes long after the rename and the attempt to read from the new file.

I'll try to draft a patch for this.

Martin


-------------------------------------------------------
This sf.net email is sponsored by:ThinkGeek
Welcome to geek heaven.
http://thinkgeek.com/sf
_______________________________________________
NFS maillist  -  [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs