make -j4 gets stuck w/ ccache over NFS
Hello:
I'm using ccache version 2.4 [1]. I just changed ~/.ccache to a symbolic link to a directory which is NFS mounted [2]. The kernel source itself is on a local FS. With the ccache suitably primed, when I do a kernel compile using 'make -j4' it seems to get stuck for seconds at a time. When it gets unstuck, it blows through a handful of files and then gets stuck again.
When it is stuck, both NFS client and server are almost totally idle. The network itself has almost no other traffic. It doesn't seem to matter if I mount NFS w/ udp or tcp (v3 in both cases).
If I move ~/.ccache to a local FS, it never gets stuck. If I just use 'make' or even 'make -j2', (I'm pretty sure but not 100%) it never gets stuck.
The NFS client is FC3 with kernel.org 2.6.10-rc3. I also tried it with -rc1, 2.6.9, and 2.6.8.1 - same problem.
Weirdest part: when I boot the distro stock kernel (2.6.9-1.681_FC3smp) the problem goes away. Configs for all of the above are here [3].
Please CC me with replies - thanks!
[2] The NFS server uses kernel 2.4.28-rc2.
[3] http://members.dca.net/mhoffman/lkml-20041213/
Signature · 2 lines
Mark M. Hoffman [email protected]
Quoted 9 lines · Mark M. Hoffman
Hello:
I'm using ccache version 2.4 [1]. I just changed ~/.ccache to a symbolic link to a directory which is NFS mounted [2]. The kernel source itself is on a local FS. With the ccache suitably primed, when I do a kernel compile using 'make -j4' it seems to get stuck for seconds at a time. When it gets unstuck, it blows through a handful of files and then gets stuck again.
Quoted 7 lines
When it is stuck, both NFS client and server are almost totally idle. The network itself has almost no other traffic. It doesn't seem to matter if I mount NFS w/ udp or tcp (v3 in both cases).
If I move ~/.ccache to a local FS, it never gets stuck. If I just use 'make' or even 'make -j2', (I'm pretty sure but not 100%) it never gets stuck.
Perhaps ccache is getting jammed/deadlocked trying to take a lock on NFS. Maybe you should try getting an ethereal dump of the network traffic.
Signature · 1 line
Martin
Mark,
Quoted 5 lines
I'm using ccache version 2.4 [1]. I just changed ~/.ccache to a symbolic link to a directory which is NFS mounted [2]. The kernel source itself is on a local FS. With the ccache suitably primed, when I do a kernel compile using 'make -j4' it seems to get stuck for seconds at a time. When it gets unstuck, it blows through a handful of files and then gets stuck again.
I'd suggest you first narrow down the problem to either being a locking problem or a file IO problem. To do that, change lock_fd() in util.c in ccache to just "return 0;". That will mean the ccache stats file could become corrupted, but if it runs fast then you know that it is a locking problem. I have noticed severe speed problem with NFS locking on Linux previosly, which is why I mention this as a possibility.
Note that removing this locking will not cause ccache to produce incorrect object files, it will just mean the stats printed with "ccache -s" may be inaccurate.
Cheers, Tridge
PS: I also wonder why you're not just using distcc. It's usually a lot more appropriate in a distributed environment.
Re: make -j4 gets stuck w/ ccache over NFS - solved!
Hi Tridge, Greg, et. al.:
I wrote, some months ago:
Quoted 5 lines
Quoted 5 lines
I'm using ccache version 2.4 [1]. I just changed ~/.ccache to a symbolic link to a directory which is NFS mounted [2]. The kernel source itself is on a local FS. With the ccache suitably primed, when I do a kernel compile using 'make -j4' it seems to get stuck for seconds at a time. When it gets unstuck, it blows through a handful of files and then gets stuck again.
Quoted 11 lines · * [email protected] <[email protected]> [2004-12-08 18:25:59 +1100]:
I'd suggest you first narrow down the problem to either being a locking problem or a file IO problem. To do that, change
lock_fd()inutil.cin ccache to just "return 0;". That will mean the ccache stats file could become corrupted, but if it runs fast then you know that it is a locking problem. I have noticed severe speed problem with NFS locking on Linux previosly, which is why I mention this as a possibility.Note that removing this locking will not cause ccache to produce incorrect object files, it will just mean the stats printed with "ccache -s" may be inaccurate.
Thanks for the suggestions. It wasn't very important to me so I didn't make time to follow up on it. I was just playing w/ ccache at the time.
Finally I noticed this patch from -mm1... and it solves the problem.
nfsd--lockd-dont-try-to-match-callback-requests-against-export-table.patch
How I tested: I applied the first 12 patches in 2.6.11-mm1; the above mentioned was last - couldn't reproduce the bug. When I unapplied just that one, I saw it again.
original bug report:
http://marc.theaimsgroup.com/?l=linux-kernel&m=110238645132535&w=3
Greg: have you considered this one for 2.6.11.x?
Thanks,
Signature · 2 lines
Mark M. Hoffman [email protected]
Re: make -j4 gets stuck w/ ccache over NFS - solved!
Quoted 7 lines · Mark M. Hoffman
Thanks for the suggestions. It wasn't very important to me so I didn't make time to follow up on it. I was just playing w/ ccache at the time.
Finally I noticed this patch from -mm1... and it solves the problem.
nfsd--lockd-dont-try-to-match-callback-requests-against-export-table.patch
Quoted 8 lines
How I tested: I applied the first 12 patches in 2.6.11-mm1; the above mentioned was last - couldn't reproduce the bug. When I unapplied just that one, I saw it again.
original bug report:
http://marc.theaimsgroup.com/?l=linux-kernel&m=110238645132535&w=3Greg: have you considered this one for 2.6.11.x?
That patch depends on 3 of the previous 4 patches. Taken together I doubt they meet the criteria for 2.6.11.x.
It's probably possible to write a shorter and more obvious one-off fix just for that tree, but I'm not sure it's worth it for a bug that, while it's obviously extremely annoying for some workloads, doesn't quite reach the level of, say, a root exploit.
--b.
Re: make -j4 gets stuck w/ ccache over NFS - solved!
In reply to Mark M. HoffmanQuoted 4 lines · Mark M. Hoffman
Finally I noticed this patch from -mm1... and it solves the problem.
nfsd--lockd-dont-try-to-match-callback-requests-against-export-table.patch
Quoted 8 lines
How I tested: I applied the first 12 patches in 2.6.11-mm1; the above mentioned was last - couldn't reproduce the bug. When I unapplied just that one, I saw it again.
original bug report:
http://marc.theaimsgroup.com/?l=linux-kernel&m=110238645132535&w=3Greg: have you considered this one for 2.6.11.x?
No, it hasn't been submitted to the [email protected] address :)
thanks,
greg k-h