Short write in nfsd becomes a full write to the client
Subsystems (2)
Quoted 9 lines · David Shaw
If a filesystem being written to via NFS returns a short write count (as opposed to an error) to nfsd, nfsd treats that as a success for the entire write, rather than the short count that actually succeeded.
For example, given a 8192 byte write, if the underlying filesystem only writes 4096 bytes, nfsd will ack back to the nfs client that all 8192 bytes were written. The nfs client does have retry logic for short writes, but this is never called as the client is told the complete write succeeded.
...
Quoted 2 lines
Here is a patch to properly return the short write count to the client.
[patch elided]
I bring this to your attention so you may, if you choose, look into this further:
Problem synopsis:
An old client (running RHL 9 with kernel "2.4.20-43.9.legacy") attempts to seek on a file mounted over nfs. The operation fails with "Illegal seek" or "Input/Output error". The server is running Fedora 11 kernel-PAE-2.6.29.5-191.fc11.i686, which includes the short write patch. When this kernel is re-built without the short write patch, everything works as before.
Detais are at
https://bugzilla.redhat.com/show_bug.cgi?id=508174
Caveats: I am specifically referring to the patch file "linux-2.6-nfsd-report-short-writes.patch" as newly included in Fedora's file kernel-2.6.29.5-191.fc11.src.rpm . I can't vouch that that patch file is identical to what was posted to this list or merged for 2.6.30-rc1.
(As an aside, in this case, the client was attempting a simple gcc compile and link. The failing programs (invoked by fcc) were the assember ("as)" and "ld".)
Quoted 18 lines · Dale Stimson
Quoted 9 lines · David Shaw
If a filesystem being written to via NFS returns a short write count (as opposed to an error) to nfsd, nfsd treats that as a success for the entire write, rather than the short count that actually succeeded.
For example, given a 8192 byte write, if the underlying filesystem only writes 4096 bytes, nfsd will ack back to the nfs client that all 8192 bytes were written. The nfs client does have retry logic for short writes, but this is never called as the client is told the complete write succeeded.
...
Quoted 2 lines
Here is a patch to properly return the short write count to the client.
[patch elided]
I bring this to your attention so you may, if you choose, look into this further:
Quoted 7 lines
Problem synopsis:
An old client (running RHL 9 with kernel "2.4.20-43.9.legacy") attempts to seek on a file mounted over nfs. The operation fails with "Illegal seek" or "Input/Output error". The server is running Fedora 11 kernel-PAE-2.6.29.5-191.fc11.i686, which includes the short write patch. When this kernel is re-built without the short write patch, everything works as before.
Does that server kernel have some version of a0d24b295aed7a9daf4ca36bd4784e4d40f82303 "nfsd: fix hung up of nfs client while sync write data to nfs server" applied?
If not, would it be possible to get a network trace? (Run "tcpdump -s0 -wTMP", then run the test case, then kill tcpdump and mail me a copy of TMP.)
--b.
Quoted 13 lines
Detais are at
https://bugzilla.redhat.com/show_bug.cgi?id=508174Caveats: I am specifically referring to the patch file "linux-2.6-nfsd-report-short-writes.patch" as newly included in Fedora's file kernel-2.6.29.5-191.fc11.src.rpm . I can't vouch that that patch file is identical to what was posted to this list or merged for 2.6.30-rc1.
(As an aside, in this case, the client was attempting a simple gcc compile and link. The failing programs (invoked by fcc) were the assember ("as)" and "ld".)
Quoted 27 lines · J. Bruce Fields
Quoted 18 lines · Dale Stimson
Quoted 9 lines · David Shaw
If a filesystem being written to via NFS returns a short write count (as opposed to an error) to nfsd, nfsd treats that as a success for the entire write, rather than the short count that actually succeeded.
For example, given a 8192 byte write, if the underlying filesystem only writes 4096 bytes, nfsd will ack back to the nfs client that all 8192 bytes were written. The nfs client does have retry logic for short writes, but this is never called as the client is told the complete write succeeded.
...
Quoted 2 lines
Here is a patch to properly return the short write count to the client.
[patch elided]
I bring this to your attention so you may, if you choose, look into this further:
Quoted 7 lines
Problem synopsis:
An old client (running RHL 9 with kernel "2.4.20-43.9.legacy") attempts to seek on a file mounted over nfs. The operation fails with "Illegal seek" or "Input/Output error". The server is running Fedora 11 kernel-PAE-2.6.29.5-191.fc11.i686, which includes the short write patch. When this kernel is re-built without the short write patch, everything works as before.
Quoted 3 lines
Does that server kernel have some version of
a0d24b295aed7a9daf4ca36bd4784e4d40f82303"nfsd: fix hung up of nfs client while sync write data to nfs server" applied?
No. As far as I can see that patch has been merged only in 2.6.30 and is not in any of the 2.6.29.? releases. There is definitely no Fedora-specific version of that patch present. The only Fedora-specific patch related to NFS applied in kernel-2.6.29.5-191.fc11.src.rpm is linux-2.6-nfsd-report-short-writes.patch.
Quoted 5 lines
If not, would it be possible to get a network trace? (Run "tcpdump -s0 -wTMP", then run the test case, then kill tcpdump and mail me a copy of TMP.)
--b.
Capture done, to be emailed off-list.
Quoted 12 lines
Quoted 12 lines
Detais are at
https://bugzilla.redhat.com/show_bug.cgi?id=508174Caveats: I am specifically referring to the patch file "linux-2.6-nfsd-report-short-writes.patch" as newly included in Fedora's file kernel-2.6.29.5-191.fc11.src.rpm . I can't vouch that that patch file is identical to what was posted to this list or merged for 2.6.30-rc1.
(As an aside, in this case, the client was attempting a simple gcc compile and link. The failing programs (invoked by fcc) were the assember ("as)" and "ld".)
Quoted 28 lines · Dale Stimson
Quoted 27 lines · J. Bruce Fields
Quoted 25 lines · Dale Stimson
Quoted 9 lines · David Shaw
If a filesystem being written to via NFS returns a short write count (as opposed to an error) to nfsd, nfsd treats that as a success for the entire write, rather than the short count that actually succeeded.
For example, given a 8192 byte write, if the underlying filesystem only writes 4096 bytes, nfsd will ack back to the nfs client that all 8192 bytes were written. The nfs client does have retry logic for short writes, but this is never called as the client is told the complete write succeeded.
...
Quoted 2 lines
Here is a patch to properly return the short write count to the client.
[patch elided]
I bring this to your attention so you may, if you choose, look into this further:
Problem synopsis:
An old client (running RHL 9 with kernel "2.4.20-43.9.legacy") attempts to seek on a file mounted over nfs. The operation fails with "Illegal seek" or "Input/Output error". The server is running Fedora 11 kernel-PAE-2.6.29.5-191.fc11.i686, which includes the short write patch. When this kernel is re-built without the short write patch, everything works as before.
Quoted 5 lines
Quoted 3 lines
Does that server kernel have some version of
a0d24b295aed7a9daf4ca36bd4784e4d40f82303"nfsd: fix hung up of nfs client while sync write data to nfs server" applied?No. As far as I can see that patch has been merged only in 2.6.30 and is not in any of the 2.6.29.? releases. There is definitely no Fedora-specific version of that patch present. The only Fedora-specific patch related to NFS applied in kernel-2.6.29.5-191.fc11.src.rpm is linux-2.6-nfsd-report-short-writes.patch.
That patch fixes a problem with the showrt write patch, so it would be worth retesting with it applied.
--b.
Quoted 29 lines · bfields
Quoted 28 lines · Dale Stimson
Quoted 27 lines · J. Bruce Fields
Quoted 25 lines · Dale Stimson
Quoted 9 lines · David Shaw
If a filesystem being written to via NFS returns a short write count (as opposed to an error) to nfsd, nfsd treats that as a success for the entire write, rather than the short count that actually succeeded.
For example, given a 8192 byte write, if the underlying filesystem only writes 4096 bytes, nfsd will ack back to the nfs client that all 8192 bytes were written. The nfs client does have retry logic for short writes, but this is never called as the client is told the complete write succeeded.
...
Quoted 2 lines
Here is a patch to properly return the short write count to the client.
[patch elided]
I bring this to your attention so you may, if you choose, look into this further:
Problem synopsis:
An old client (running RHL 9 with kernel "2.4.20-43.9.legacy") attempts to seek on a file mounted over nfs. The operation fails with "Illegal seek" or "Input/Output error". The server is running Fedora 11 kernel-PAE-2.6.29.5-191.fc11.i686, which includes the short write patch. When this kernel is re-built without the short write patch, everything works as before.
Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Does that server kernel have some version of
a0d24b295aed7a9daf4ca36bd4784e4d40f82303"nfsd: fix hung up of nfs client while sync write data to nfs server" applied?No. As far as I can see that patch has been merged only in 2.6.30 and is not in any of the 2.6.29.? releases. There is definitely no Fedora-specific version of that patch present. The only Fedora-specific patch related to NFS applied in kernel-2.6.29.5-191.fc11.src.rpm is linux-2.6-nfsd-report-short-writes.patch.
That patch fixes a problem with the showrt write patch, so it would be worth retesting with it applied.
After looking at the trace: yes, looks like the same problem. (The (succesful) FILE_SYNC write request returned with a 0 length.) So retest with the new patch applied, then poke the Fedora people to apply it as well.
--b.
Quoted 39 lines · J. Bruce Fields
Quoted 29 lines · bfields
Quoted 28 lines · Dale Stimson
Quoted 27 lines · J. Bruce Fields
Quoted 25 lines · Dale Stimson
Quoted 9 lines · David Shaw
If a filesystem being written to via NFS returns a short write count (as opposed to an error) to nfsd, nfsd treats that as a success for the entire write, rather than the short count that actually succeeded.
For example, given a 8192 byte write, if the underlying filesystem only writes 4096 bytes, nfsd will ack back to the nfs client that all 8192 bytes were written. The nfs client does have retry logic for short writes, but this is never called as the client is told the complete write succeeded.
...
Quoted 2 lines
Here is a patch to properly return the short write count to the client.
[patch elided]
I bring this to your attention so you may, if you choose, look into this further:
Problem synopsis:
An old client (running RHL 9 with kernel "2.4.20-43.9.legacy") attempts to seek on a file mounted over nfs. The operation fails with "Illegal seek" or "Input/Output error". The server is running Fedora 11 kernel-PAE-2.6.29.5-191.fc11.i686, which includes the short write patch. When this kernel is re-built without the short write patch, everything works as before.Quoted 8 lines
Quoted 5 lines
Quoted 3 lines
Does that server kernel have some version of
a0d24b295aed7a9daf4ca36bd4784e4d40f82303"nfsd: fix hung up of nfs client while sync write data to nfs server" applied?No. As far as I can see that patch has been merged only in 2.6.30 and is not in any of the 2.6.29.? releases. There is definitely no Fedora-specific version of that patch present. The only Fedora-specific patch related to NFS applied in kernel-2.6.29.5-191.fc11.src.rpm is linux-2.6-nfsd-report-short-writes.patch.
That patch fixes a problem with the showrt write patch, so it would be worth retesting with it applied.
Quoted 6 lines
After looking at the trace: yes, looks like the same problem. (The (succesful)
FILE_SYNCwrite request returned with a 0 length.) So retest with the new patch applied, then poke the Fedora people to apply it as well.--b.
Success.
I applied patch a0d24b295aed7a9daf4ca36bd4784e4d40f82303 "nfsd: fix hung up of nfs client while sync write data to nfs server" to the otherwise unmodified Fedora 11 kernel 2.6.29.5-191.fc11.i686.PAE and the problem was resolved.
Thank you for your help. I will update
https://bugzilla.redhat.com/show_bug.cgi?id=508174
and suggest that the patch be applied to future Fedora 2.6.29-based kernels.
Quoted 10 lines · Dale Stimson
Quoted 6 lines · J. Bruce Fields
After looking at the trace: yes, looks like the same problem. (The (succesful)
FILE_SYNCwrite request returned with a 0 length.) So retest with the new patch applied, then poke the Fedora people to apply it as well.--b.
Success.
Quoted 8 lines
I applied patch
a0d24b295aed7a9daf4ca36bd4784e4d40f82303"nfsd: fix hung up of nfs client while sync write data to nfs server" to the otherwise unmodified Fedora 11 kernel 2.6.29.5-191.fc11.i686.PAE and the problem was resolved.Thank you for your help. I will update
https://bugzilla.redhat.com/show_bug.cgi?id=508174
and suggest that the patch be applied to future Fedora 2.6.29-based kernels.
Thanks for following up.--b.