Skip to contentSkip to conversation

Apparent race condition w/ NFSv3 mktemp+write+rename

DISCUSSION Charles Duffy 4 participants 8 messages Nov 24, 2009 2 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Howdy. With a very heavily loaded (CentOS 5 - Linux 2.6.18-128.el5) NFSv3 client and server, I've experienced the following sporadic issue:

function atomic_write() {
    local filename="$1"
    local tempfile
    tempfile="$(mktemp "${filename}.XXXXXX")"
    cat >"${tempfile}" && mv "${tempfile}" "${filename}"
}
$ atomic_write /mnt/foobar <<<"baz"
mv: cannot move `/mnt/foobar.gO1881' to `/mnt/foobar': No such file or directory

Are my assumptions regarding available semantics fair and reasonable?

#1 lore raw
Quoted 10 lines · On Tue, 24 Nov 2009 13:26:44 -0600 Charles Duffy…

Howdy. With a very heavily loaded (CentOS 5 - Linux 2.6.18-128.el5) NFSv3 client and server, I've experienced the following sporadic issue:

function atomic_write() {
     local filename="$1"
     local tempfile
     tempfile="$(mktemp "${filename}.XXXXXX")"
     cat >"${tempfile}" && mv "${tempfile}" "${filename}"
}
Quoted 6 lines
$ atomic_write /mnt/foobar <<<"baz"
mv: cannot move `/mnt/foobar.gO1881' to `/mnt/foobar': No such file or 
directory

Are my assumptions regarding available semantics fair and reasonable?

Known problem with O_EXCL creates. Fixed in RHEL fairly recently. You probably want a newer kernel.

Signature · 1 line
Jeff Layton <[email protected]>
#2 lore raw
Quoted 17 lines · On Tue, 24 Nov 2009 15:03:01 -0500 Jeff Layton <[email protected]> wrote:
Quoted 14 lines · On Tue, 24 Nov 2009 13:26:44 -0600 Charles Duffy <[email protected]> wrote:

Howdy. With a very heavily loaded (CentOS 5 - Linux 2.6.18-128.el5) NFSv3 client and server, I've experienced the following sporadic issue:

function atomic_write() {
     local filename="$1"
     local tempfile
     tempfile="$(mktemp "${filename}.XXXXXX")"
     cat >"${tempfile}" && mv "${tempfile}" "${filename}"
}
$ atomic_write /mnt/foobar <<<"baz"
mv: cannot move `/mnt/foobar.gO1881' to `/mnt/foobar': No such file or 
directory
Quoted 6 lines
Quoted 2 lines

Are my assumptions regarding available semantics fair and reasonable?

Known problem with O_EXCL creates. Fixed in RHEL fairly recently. You probably want a newer kernel.

My mistake...I didn't read the version closely enough. I was thinking of another issue. I'm not sure if a newer kernel will help you.

Signature · 1 line
Jeff Layton <[email protected]>
#3 lore raw
In reply to Charles Duffy
Quoted 10 lines · Charles Duffy

Howdy. With a very heavily loaded (CentOS 5 - Linux 2.6.18-128.el5) NFSv3 client and server, I've experienced the following sporadic issue:

function atomic_write() {
    local filename="$1"
    local tempfile
    tempfile="$(mktemp "${filename}.XXXXXX")"
    cat >"${tempfile}" && mv "${tempfile}" "${filename}"
}
Quoted 6 lines
$ atomic_write /mnt/foobar <<<"baz"
mv: cannot move `/mnt/foobar.gO1881' to `/mnt/foobar': No such file or
directory

Are my assumptions regarding available semantics fair and reasonable?

At the end, does /mnt/foobar exist and contain the string, "baz"?

What mount options are being used on the client?

It sounds to me as if the duplicate request cache on the server is not working, perhaps because entries are being recycled too quickly, and does not catch a retransmitted RENAME operation.

	Thanx...

		ps
#4 lore raw
Quoted 2 lines · Peter Staubach

At the end, does /mnt/foobar exist and contain the string, "baz"?

I'll follow up with that information with that after the issue is next reproduced. (The mv operation is replacing a prior instance of /mnt/foobar, so content will need to be inspected to determine whether it was successful).

Quoted 2 lines

What mount options are being used on the client?

rw,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=<redacted>

A message that is not in the archive (sent off-list or not yet received): [email protected]

#6 lore raw
Quoted 12 lines · Charles Duffy
Quoted 2 lines · Peter Staubach

At the end, does /mnt/foobar exist and contain the string, "baz"?

I'll follow up with that information with that after the issue is next reproduced. (The mv operation is replacing a prior instance of /mnt/foobar, so content will need to be inspected to determine whether it was successful).

Quoted 2 lines

What mount options are being used on the client?

rw,vers=3,rsize=32768,wsize=32768,hard,intr,proto=tcp,timeo=600,retrans=2,sec=sys,addr=<redacted>

Hmmm. This appears to be the defaults.

Does this take a very long time to run when it fails like this?

	Thanx...

		ps
#7 lore raw
Quoted 2 lines · Peter Staubach

Does this take a very long time to run when it fails like this?

Total runtime for script invocations where the failure occurred has in all cases (two today -- one at 7:59AM, the other at 12:08PM) been less than three seconds.

A message that is not in the archive (sent off-list or not yet received): [email protected]

#9 lore raw
Quoted 6 lines · Charles Duffy
Quoted 2 lines · Peter Staubach

Does this take a very long time to run when it fails like this?

Total runtime for script invocations where the failure occurred has in all cases (two today -- one at 7:59AM, the other at 12:08PM) been less than three seconds.

Interesting. And non-deterministic, I'll bet?

	Thanx...

		ps