Skip to contentSkip to conversation

NFSERR_NOSPC nfs-client bug

DISCUSSION Ray Ferguson 4 participants 7 messages Mar 18, 2008 6 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I've discovered a bug in the linux nfs client. Specifically it ignores NFSERR_NOSPC messages (code 28) from an NFS server and happily continues pounding it with data.

This causes some rather unfortunate consequences on linux nfs servers by exhausting resources. In 2.4, all cpus peg at 100% usage under the system catagory. In 2.6, at least one core gets pegged at 100% iowait, but this still triggers cascading load issues.

So far I've tested:

Opensuse-10.3 = Linux 2.6.22 (client bug confirmed) RHAS4 = 2.6.9 (client bug confirmed)
RHAS3 = 2.4.21(No Bug: Pre-nfs4)
Solaris 9 = (No Bug)

This can be reproduced by creating a small filesystem and exporting it via nfs. Then mount it with a buggy client and "cat /dev/zero > /nfs-share/foo"

The expected behavior is for the client to error out the write with a message informing you that the filesystem is out of space. Instead, the client keeps sending data and the servers kernel take a beating.

I've checked the wire and confirmed that the server is sending the NOSPC message back to the client. Most of my testing has been nfs3 though I did some brief testing w/ nfs2 (bug still present). I have kernel sysrq debug data and packet captures if anyone is interested.

If this is not the correct place to report this, I would be grateful if anyone could redirect me.

Thank you for your help.

-
Ray Ferguson

#2 lore raw
In reply to a message not in the archive
Quoted 4 lines · Ray Ferguson

I've discovered a bug in the linux nfs client. Specifically it ignores NFSERR_NOSPC messages (code 28) from an NFS server and happily continues pounding it with data.

It doesn't ignore ENOSPC, it reports it on close(). Of course this is often several gigabytes of lost data too late.

Sensible clients (e.g. Irix) store that error on the inode and report it on the next call to write().

Signature · 3 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
The cake is *not* a lie.
I don't speak for SGI.
#4 lore raw
In reply to a message not in the archive
Quoted 2 lines · Greg Banks

It doesn't ignore ENOSPC, it reports it on close(). Of course this is often several gigabytes of lost data too late.

In our case, the thrashing it gave our Linux NFS cluster was severe enough to take it out of commission. The lost data from the file transfer that triggered the event was the least of our worries.

-
Ray Ferguson

#6 lore raw
In reply to a message not in the archive
Quoted 11 lines · Ray Ferguson
Quoted 3 lines · Greg Banks

It doesn't ignore ENOSPC, it reports it on close(). Of course this is often several gigabytes of lost data too late.

In our case, the thrashing it gave our Linux NFS cluster was severe enough to take it out of commission. The lost data from the file transfer that triggered the event was the least of our worries.

Agreed, it sucks.

Signature · 3 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
The cake is *not* a lie.
I don't speak for SGI.
#8 lore raw
In reply to a message not in the archive
Quoted 7 lines · Greg Banks
Quoted 6 lines · Ray Ferguson
Quoted 3 lines · Greg Banks

It doesn't ignore ENOSPC, it reports it on close(). Of course this is often several gigabytes of lost data too late.

Quoted 6 lines
Quoted 5 lines

In our case, the thrashing it gave our Linux NFS cluster was severe enough to take it out of commission. The lost data from the file transfer that triggered the event was the least of our worries.

Agreed, it sucks.

Try a more recent kernel: 2.6.24 and more recent will report these errors more promptly.

Trond

#10 lore raw
In reply to a message not in the archive
Quoted 17 lines · Trond Myklebust
Quoted 15 lines · Greg Banks
Quoted 13 lines · Ray Ferguson
Quoted 4 lines · Greg Banks

It doesn't ignore ENOSPC, it reports it on close(). Of course this is often several gigabytes of lost data too late.

In our case, the thrashing it gave our Linux NFS cluster was severe enough to take it out of commission. The lost data from the file transfer that triggered the event was the least of our worries.

Quoted 6 lines
Quoted 2 lines

Agreed, it sucks.

Try a more recent kernel: 2.6.24 and more recent will report these errors more promptly.

So that would be
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=7b159fc18d417980f57aef64cab3417ee6af70f8

?
Is my reading right, that a solitary transient ENOSPC is still reported at close(), but a sequence of two or more ENOSPC is reported in the next write() call?

Signature · 3 lines
Greg Banks, R&D Software Engineer, SGI Australian Software Group.
The cake is *not* a lie.
I don't speak for SGI.
#12 lore raw
In reply to a message not in the archive
Quoted 7 lines · Greg Banks

So that would be
http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commitdiff;h=7b159fc18d417980f57aef64cab3417ee6af70f8

?
Is my reading right, that a solitary transient ENOSPC is still reported at close(), but a sequence of two or more ENOSPC is reported in the next write() call?

I don't think so.
After an error report is received by the client, the next write/fsync/close will report the error (even if that write didn't actually have an error).

Also, further writes will be attempted synchronously, so the correct error is reported, until a write succeeds. At this point we go back to async writes.

So a solitary transient ENOSPC will be reported against a subsequent write.

When a write fails, the page remains DIRTY, so a flush will be attempted on every subsequent 'sync', including those triggered by a write while the error flag is set.
So if you keep writing, you will keep getting an error until all dirty pages have been safely written to the server.

If you give up and close the file, you won't be able to tell just by looking at the error codes which pages were successfully written and which aren't. But it would seem unwise to expect to be able to do that in any case.

NeilBrown