Apparent race condition w/ NFSv3 mktemp+write+rename
Subsystems (2)
Howdy. With a very heavily loaded (CentOS 5 - Linux 2.6.18-128.el5) NFSv3 client and server, I've experienced the following sporadic issue:
function atomic_write() {
local filename="$1"
local tempfile
tempfile="$(mktemp "${filename}.XXXXXX")"
cat >"${tempfile}" && mv "${tempfile}" "${filename}"
}$ atomic_write /mnt/foobar <<<"baz" mv: cannot move `/mnt/foobar.gO1881' to `/mnt/foobar': No such file or directory
Are my assumptions regarding available semantics fair and reasonable?
Quoted 10 lines · On Tue, 24 Nov 2009 13:26:44 -0600 Charles Duffy…
Howdy. With a very heavily loaded (CentOS 5 - Linux 2.6.18-128.el5) NFSv3 client and server, I've experienced the following sporadic issue:
function atomic_write() { local filename="$1" local tempfile tempfile="$(mktemp "${filename}.XXXXXX")" cat >"${tempfile}" && mv "${tempfile}" "${filename}" }
Quoted 6 lines
$ atomic_write /mnt/foobar <<<"baz" mv: cannot move `/mnt/foobar.gO1881' to `/mnt/foobar': No such file or directoryAre my assumptions regarding available semantics fair and reasonable?
Known problem with O_EXCL creates. Fixed in RHEL fairly recently. You probably want a newer kernel.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 10 lines · Charles Duffy
Howdy. With a very heavily loaded (CentOS 5 - Linux 2.6.18-128.el5) NFSv3 client and server, I've experienced the following sporadic issue:
function atomic_write() { local filename="$1" local tempfile tempfile="$(mktemp "${filename}.XXXXXX")" cat >"${tempfile}" && mv "${tempfile}" "${filename}" }
Quoted 6 lines
$ atomic_write /mnt/foobar <<<"baz" mv: cannot move `/mnt/foobar.gO1881' to `/mnt/foobar': No such file or directoryAre my assumptions regarding available semantics fair and reasonable?
At the end, does /mnt/foobar exist and contain the string, "baz"?
What mount options are being used on the client?
It sounds to me as if the duplicate request cache on the server is not working, perhaps because entries are being recycled too quickly, and does not catch a retransmitted RENAME operation.
Thanx... ps
Quoted 17 lines · On Tue, 24 Nov 2009 15:03:01 -0500 Jeff Layton <[email protected]> wrote:
Quoted 14 lines · On Tue, 24 Nov 2009 13:26:44 -0600 Charles Duffy <[email protected]> wrote:
Howdy. With a very heavily loaded (CentOS 5 - Linux 2.6.18-128.el5) NFSv3 client and server, I've experienced the following sporadic issue:
function atomic_write() { local filename="$1" local tempfile tempfile="$(mktemp "${filename}.XXXXXX")" cat >"${tempfile}" && mv "${tempfile}" "${filename}" }$ atomic_write /mnt/foobar <<<"baz" mv: cannot move `/mnt/foobar.gO1881' to `/mnt/foobar': No such file or directory
Quoted 6 lines
Quoted 2 lines
Are my assumptions regarding available semantics fair and reasonable?
Known problem with
O_EXCLcreates. Fixed in RHEL fairly recently. You probably want a newer kernel.
My mistake...I didn't read the version closely enough. I was thinking of another issue. I'm not sure if a newer kernel will help you.
Signature · 1 line
Jeff Layton <[email protected]>
Quoted 2 lines · Peter Staubach
At the end, does
/mnt/foobarexist and contain the string, "baz"?
I'll follow up with that information with that after the issue is next reproduced. (The mv operation is replacing a prior instance of /mnt/foobar, so content will need to be inspected to determine whether it was successful).
Quoted 2 lines
What mount options are being used on the client?
rw,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=<redacted>
Quoted 12 lines · Charles Duffy
Quoted 2 lines · Peter Staubach
At the end, does
/mnt/foobarexist and contain the string, "baz"?I'll follow up with that information with that after the issue is next reproduced. (The mv operation is replacing a prior instance of
/mnt/foobar, so content will need to be inspected to determine whether it was successful).Quoted 2 lines
What mount options are being used on the client?
rw,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=<redacted>
Hmmm. This appears to be the defaults.
Does this take a very long time to run when it fails like this?
Thanx... ps
Quoted 2 lines · Peter Staubach
Does this take a very long time to run when it fails like this?
Total runtime for script invocations where the failure occurred has in all cases (two today -- one at 7:59AM, the other at 12:08PM) been less than three seconds.
Quoted 6 lines · Charles Duffy
Quoted 2 lines · Peter Staubach
Does this take a very long time to run when it fails like this?
Total runtime for script invocations where the failure occurred has in all cases (two today -- one at 7:59AM, the other at 12:08PM) been less than three seconds.
Interesting. And non-deterministic, I'll bet?
Thanx... ps