Skip to contentSkip to conversation

NFS EINVAL on open(... | O_TRUNC) on 2.6.23.9

DISCUSSION Gianluca Alberici 7 participants 25 messages Dec 22, 2007 – Feb 6, 2008 7 messages not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

Sorry for repeating but as far as i can see 2.6.23.11/12 got no changes into NFS.

I've run into this problem 2.6.23.9. The open syscall will return "Invalid argument" when O_TRUNC is set on existing files.

The same file can be opened for append or removed.

The evidence is for example:

mars:~# mount localhost:/opt/nfs/ /mnt/tmp
mars:~# echo "Hello" > /mnt/tmp/test-file
mars:~# echo "Hello" > /mnt/tmp/test-file
bash: /mnt/tmp/test-file: Invalid argument
mars:~# echo "Hello" > /mnt/tmp/test-file
bash: /mnt/tmp/test-file: Invalid argument
mars:~# echo "Hello" > /mnt/tmp/test-file
bash: /mnt/tmp/test-file: Invalid argument
mars:~# rm /mnt/tmp/test-file
mars:~# echo "Hello" > /mnt/tmp/test-file
mars:~# echo "Hello" > /mnt/tmp/test-file
bash: /mnt/tmp/test-file: Invalid argument

This is not present in 2.6.21.7.
How can this be fixed ?

TIA

Gianluca
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#1 lore raw
Quoted 11 lines · Gianluca Alberici

I've run into this problem 2.6.23.9. The open syscall will return "Invalid argument" when O_TRUNC is set on existing files.

The same file can be opened for append or removed.

The evidence is for example:

mars:~# mount localhost:/opt/nfs/ /mnt/tmp
mars:~# echo "Hello" > /mnt/tmp/test-file
mars:~# echo "Hello" > /mnt/tmp/test-file
bash: /mnt/tmp/test-file: Invalid argument

Can you show an strace of this please? Also, might suggest "rpcdebug -m nfsd -s all"

--
Scott

#2 lore raw

Hello,

I can do better. I have investigated a bit the problem:

1) The problem arises only with the userspace nfsd (Universal nfsd 2.2). I have realized that the latest patches introduced in 2.6.22 have changed a lot of things into NFS.

2) The following code has been debugged with sunrpc.rpc_debug and sunrpc.nfs_debug

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
       int fp;
       if ((fp=open("/mnt/tmp/test",O_WRONLY | O_CREAT | O_TRUNC, S_IRWXU )) == -1) printf("ERR: %d\n",errno);
       else {
               write(fp, argv[1], strlen(argv[1]));
               printf("OK\n");
               close(fp);
       };
}

2b) The output

[...]

<8>NFS call setattr
<8>RPC: new task initialized, procpid 18656
<8>RPC: allocated task f7bec500
<8>RPC: 0 looking up UNIX cred

<8>RPC:   740 __rpc_execute flags=0x480
<8>RPC:   740 call_start nfs2 proc 2 (sync)
<8>RPC:   740 call_reserve (status 0)
<8>RPC:   740 reserved req f1416000 xid 643f3c42
<8>RPC:   740 call_reserveresult (status 0)
<8>RPC:   740 call_allocate (status 0)
<8>RPC:   740 allocated buffer of size 528 at e627b800
<8>RPC:   740 call_bind (status 0)
<8>RPC:   740 call_connect xprt f70d4000 is connected
<8>RPC:   740 call_transmit (status 0)
<8>RPC:   740 xprt_prepare_transmit
<8>RPC:   740 xprt_cwnd_limited cong = 0 cwnd = 512
<8>RPC:   740 call_encode (status 0)
<8>RPC:   740 marshaling UNIX cred f4efcb40
<8>RPC:   740 using AUTH_UNIX cred f4efcb40 to wrap rpc data
<8>RPC:   740 xprt_transmit(148)

<8>RPC: xs_udp_data_ready...
<8>RPC: cong 256, cwnd was 512, now 512
<8>RPC: 740 xid 643f3c42 complete (28 bytes received)
<8>RPC: xs_udp_send_request(148) = 148
<8>RPC: 740 xmit complete
<8>RPC: wake_up_next(f70d4114 "xprt_resend")
<8>RPC: wake_up_next(f70d40e4 "xprt_sending")

<8>RPC:   740 call_status (status 28)
<8>RPC:   740 call_decode (status 28)
<8>RPC:   740 validating UNIX cred f4efcb40
<8>RPC:   740 using AUTH_UNIX cred f4efcb40 to unwrap rpc data
<8>RPC:   740 call_decode result -22
<8>RPC:   740 return 0, status -22
<8>RPC:   740 release task

<8>RPC: freeing buffer of size 528 at e627b800
<8>RPC: 740 release request f1416000
<8>RPC: wake_up_next(f70d4174 "xprt_backlog")
<8>RPC: 740 releasing UNIX cred f4efcb40
<8>RPC: rpc_release_client(f6f1f880)
<8>NFS reply setattr: -22

[...]

3) It turns out the following:

- setattr returns EINVAL due to...
- RPC call_decode returns status 22

4) In conclusion in my understanding:

- At present linux nfs filesystem support is not anymore compatible with old userspace servers like universal nfsd and crypto filesystems as cfsd (which is an nfs server pretty old fashioned).
- This problem makes UNFSD and CFSD unusable on 2.6.22 and up (this doesnt sound good to me)

The question are:

- Is this wanted or is a bug ?
- Can this be solved with some backwards compat stuff into the kernel or all the old packages as UNFSD have to be bugfixed/upgraded
- I have found other strange behaviors of the new NFS filesystem support that i am investigating. All are bound to the user of old userspace servers onto the new NFS (since 2.6.22). What to do ?

Thanks

Gianluca

Quoted 24 lines · Scott
Quoted 11 lines · Gianluca Alberici

I've run into this problem 2.6.23.9. The open syscall will return "Invalid argument" when O_TRUNC is set on existing files.

The same file can be opened for append or removed.

The evidence is for example:

mars:~# mount localhost:/opt/nfs/ /mnt/tmp
mars:~# echo "Hello" > /mnt/tmp/test-file
mars:~# echo "Hello" > /mnt/tmp/test-file
bash: /mnt/tmp/test-file: Invalid argument

Can you show an strace of this please? Also, might suggest "rpcdebug - m nfsd -s all"

Signature · 5 lines
Scott --
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
#3 lore raw
Quoted 90 lines · On Sun, 23 Dec 2007 12:35:17 +0100 Gianluca Alberici…

Hello,

I can do better. I have investigated a bit the problem:

1) The problem arises only with the userspace nfsd (Universal nfsd 2.2). I have realized that the latest patches introduced in 2.6.22 have changed a lot of things into NFS.

2) The following code has been debugged with sunrpc.rpc_debug and sunrpc.nfs_debug

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
        int fp;
        if ((fp=open("/mnt/tmp/test",O_WRONLY | O_CREAT | O_TRUNC, S_IRWXU )) == -1) printf("ERR: %d\n",errno);
        else {
                write(fp, argv[1], strlen(argv[1]));
                printf("OK\n");
                close(fp);
        };
}

2b) The output

[...]

<8>NFS call setattr
<8>RPC: new task initialized, procpid 18656
<8>RPC: allocated task f7bec500
<8>RPC: 0 looking up UNIX cred

<8>RPC:   740 __rpc_execute flags=0x480
<8>RPC:   740 call_start nfs2 proc 2 (sync)
<8>RPC:   740 call_reserve (status 0)
<8>RPC:   740 reserved req f1416000 xid 643f3c42
<8>RPC:   740 call_reserveresult (status 0)
<8>RPC:   740 call_allocate (status 0)
<8>RPC:   740 allocated buffer of size 528 at e627b800
<8>RPC:   740 call_bind (status 0)
<8>RPC:   740 call_connect xprt f70d4000 is connected
<8>RPC:   740 call_transmit (status 0)
<8>RPC:   740 xprt_prepare_transmit
<8>RPC:   740 xprt_cwnd_limited cong = 0 cwnd = 512
<8>RPC:   740 call_encode (status 0)
<8>RPC:   740 marshaling UNIX cred f4efcb40
<8>RPC:   740 using AUTH_UNIX cred f4efcb40 to wrap rpc data
<8>RPC:   740 xprt_transmit(148)

<8>RPC: xs_udp_data_ready...
<8>RPC: cong 256, cwnd was 512, now 512
<8>RPC: 740 xid 643f3c42 complete (28 bytes received)
<8>RPC: xs_udp_send_request(148) = 148
<8>RPC: 740 xmit complete
<8>RPC: wake_up_next(f70d4114 "xprt_resend")
<8>RPC: wake_up_next(f70d40e4 "xprt_sending")

<8>RPC:   740 call_status (status 28)
<8>RPC:   740 call_decode (status 28)
<8>RPC:   740 validating UNIX cred f4efcb40
<8>RPC:   740 using AUTH_UNIX cred f4efcb40 to unwrap rpc data
<8>RPC:   740 call_decode result -22
<8>RPC:   740 return 0, status -22
<8>RPC:   740 release task

<8>RPC: freeing buffer of size 528 at e627b800
<8>RPC: 740 release request f1416000
<8>RPC: wake_up_next(f70d4174 "xprt_backlog")
<8>RPC: 740 releasing UNIX cred f4efcb40
<8>RPC: rpc_release_client(f6f1f880)
<8>NFS reply setattr: -22

[...]

3) It turns out the following:

- setattr returns EINVAL due to...
- RPC call_decode returns status 22

4) In conclusion in my understanding:

- At present linux nfs filesystem support is not anymore compatible with old userspace servers like universal nfsd and crypto filesystems as cfsd (which is an nfs server pretty old fashioned).
- This problem makes UNFSD and CFSD unusable on 2.6.22 and up (this doesnt sound good to me)

The question are:

Quoted 8 lines

- Is this wanted or is a bug ?
- Can this be solved with some backwards compat stuff into the kernel or all the old packages as UNFSD have to be
bugfixed/upgraded
- I have found other strange behaviors of the new NFS filesystem support that i am investigating. All are bound to the user of
old userspace servers onto the new NFS (since 2.6.22). What to do ?

I'm not sure what the NFS client's policy is regarding support for userspace servers. But I'd certainly hope that it is "don't break them".

Which would make this an NFS client regression.

#4 lore raw

Hello all,

I've seen that Trond and the other guys at Netapp are working hard on the NFS support and there are dozens of patches in latest 2.6.24rcX s kernels regarding both RPC and NFS.
I can only hope, as Andrew Morton said, that the policy of new NFS impl was 'dont break'.
In the end those buggy, dreaded, old fashioned NFSs have done their job excellently for over 15 years and will be around for another 5 IMHO.

Anyways, id really would be so thankful if somebody from the NFS team was givin us a clue about the whole thing.

TIA

Gianluca

Quoted 112 lines · Andrew Morton
Quoted 86 lines · On Sun, 23 Dec 2007 12:35:17 +0100 Gianluca Alberici…

Hello,

I can do better. I have investigated a bit the problem:

1) The problem arises only with the userspace nfsd (Universal nfsd 2.2). I have realized that the latest patches introduced in 2.6.22 have changed a lot of things into NFS.

2) The following code has been debugged with sunrpc.rpc_debug and sunrpc.nfs_debug

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
       int fp;
       if ((fp=open("/mnt/tmp/test",O_WRONLY | O_CREAT | O_TRUNC, S_IRWXU )) == -1) printf("ERR: %d\n",errno);
       else {
               write(fp, argv[1], strlen(argv[1]));
               printf("OK\n");
               close(fp);
       };
}

2b) The output

[...]

<8>NFS call setattr
<8>RPC: new task initialized, procpid 18656
<8>RPC: allocated task f7bec500
<8>RPC: 0 looking up UNIX cred

<8>RPC:   740 __rpc_execute flags=0x480
<8>RPC:   740 call_start nfs2 proc 2 (sync)
<8>RPC:   740 call_reserve (status 0)
<8>RPC:   740 reserved req f1416000 xid 643f3c42
<8>RPC:   740 call_reserveresult (status 0)
<8>RPC:   740 call_allocate (status 0)
<8>RPC:   740 allocated buffer of size 528 at e627b800
<8>RPC:   740 call_bind (status 0)
<8>RPC:   740 call_connect xprt f70d4000 is connected
<8>RPC:   740 call_transmit (status 0)
<8>RPC:   740 xprt_prepare_transmit
<8>RPC:   740 xprt_cwnd_limited cong = 0 cwnd = 512
<8>RPC:   740 call_encode (status 0)
<8>RPC:   740 marshaling UNIX cred f4efcb40
<8>RPC:   740 using AUTH_UNIX cred f4efcb40 to wrap rpc data
<8>RPC:   740 xprt_transmit(148)

<8>RPC: xs_udp_data_ready...
<8>RPC: cong 256, cwnd was 512, now 512
<8>RPC: 740 xid 643f3c42 complete (28 bytes received)
<8>RPC: xs_udp_send_request(148) = 148
<8>RPC: 740 xmit complete
<8>RPC: wake_up_next(f70d4114 "xprt_resend")
<8>RPC: wake_up_next(f70d40e4 "xprt_sending")

<8>RPC:   740 call_status (status 28)
<8>RPC:   740 call_decode (status 28)
<8>RPC:   740 validating UNIX cred f4efcb40
<8>RPC:   740 using AUTH_UNIX cred f4efcb40 to unwrap rpc data
<8>RPC:   740 call_decode result -22
<8>RPC:   740 return 0, status -22
<8>RPC:   740 release task

<8>RPC: freeing buffer of size 528 at e627b800
<8>RPC: 740 release request f1416000
<8>RPC: wake_up_next(f70d4174 "xprt_backlog")
<8>RPC: 740 releasing UNIX cred f4efcb40
<8>RPC: rpc_release_client(f6f1f880)
<8>NFS reply setattr: -22

[...]

3) It turns out the following:

- setattr returns EINVAL due to...
- RPC call_decode returns status 22

4) In conclusion in my understanding:

- At present linux nfs filesystem support is not anymore compatible with
old userspace servers like universal nfsd and crypto filesystems as cfsd (which is an nfs server pretty old fashioned).
- This problem makes UNFSD and CFSD unusable on 2.6.22 and up (this
doesnt sound good to me)

The question are:

Quoted 10 lines

- Is this wanted or is a bug ?
- Can this be solved with some backwards compat stuff into the kernel or all the old packages as UNFSD have to be
bugfixed/upgraded
- I have found other strange behaviors of the new NFS filesystem support that i am investigating. All are bound to the user of
old userspace servers onto the new NFS (since 2.6.22). What to do ?

I'm not sure what the NFS client's policy is regarding support for userspace servers. But I'd certainly hope that it is "don't break them".

Which would make this an NFS client regression.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#5 lore raw
In reply to Andrew Morton
Quoted 83 lines · Andrew Morton
Quoted 81 lines · On Sun, 23 Dec 2007 12:35:17 +0100 Gianluca Alberici…

Hello,

I can do better. I have investigated a bit the problem:

1) The problem arises only with the userspace nfsd (Universal nfsd 2.2). I have realized that the latest patches introduced in 2.6.22 have changed a lot of things into NFS.

2) The following code has been debugged with sunrpc.rpc_debug and sunrpc.nfs_debug

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
        int fp;
        if ((fp=open("/mnt/tmp/test",O_WRONLY | O_CREAT | O_TRUNC,  
S_IRWXU )) == -1) printf("ERR: %d\n",errno);
        else {
                write(fp, argv[1], strlen(argv[1]));
                printf("OK\n");
                close(fp);
        };
}

2b) The output

[...]

<8>NFS call setattr
<8>RPC: new task initialized, procpid 18656
<8>RPC: allocated task f7bec500
<8>RPC: 0 looking up UNIX cred

<8>RPC:   740 __rpc_execute flags=0x480
<8>RPC:   740 call_start nfs2 proc 2 (sync)
<8>RPC:   740 call_reserve (status 0)
<8>RPC:   740 reserved req f1416000 xid 643f3c42
<8>RPC:   740 call_reserveresult (status 0)
<8>RPC:   740 call_allocate (status 0)
<8>RPC:   740 allocated buffer of size 528 at e627b800
<8>RPC:   740 call_bind (status 0)
<8>RPC:   740 call_connect xprt f70d4000 is connected
<8>RPC:   740 call_transmit (status 0)
<8>RPC:   740 xprt_prepare_transmit
<8>RPC:   740 xprt_cwnd_limited cong = 0 cwnd = 512
<8>RPC:   740 call_encode (status 0)
<8>RPC:   740 marshaling UNIX cred f4efcb40
<8>RPC:   740 using AUTH_UNIX cred f4efcb40 to wrap rpc data
<8>RPC:   740 xprt_transmit(148)

<8>RPC: xs_udp_data_ready...
<8>RPC: cong 256, cwnd was 512, now 512
<8>RPC: 740 xid 643f3c42 complete (28 bytes received)
<8>RPC: xs_udp_send_request(148) = 148
<8>RPC: 740 xmit complete
<8>RPC: wake_up_next(f70d4114 "xprt_resend")
<8>RPC: wake_up_next(f70d40e4 "xprt_sending")

<8>RPC:   740 call_status (status 28)
<8>RPC:   740 call_decode (status 28)
<8>RPC:   740 validating UNIX cred f4efcb40
<8>RPC:   740 using AUTH_UNIX cred f4efcb40 to unwrap rpc data
<8>RPC:   740 call_decode result -22
<8>RPC:   740 return 0, status -22
<8>RPC:   740 release task

<8>RPC: freeing buffer of size 528 at e627b800
<8>RPC: 740 release request f1416000
<8>RPC: wake_up_next(f70d4174 "xprt_backlog")
<8>RPC: 740 releasing UNIX cred f4efcb40
<8>RPC: rpc_release_client(f6f1f880)
<8>NFS reply setattr: -22

[...]

3) It turns out the following:

- setattr returns EINVAL due to...
- RPC call_decode returns status 22

Quoted 6 lines
Quoted 6 lines

4) In conclusion in my understanding:

- At present linux nfs filesystem support is not anymore compatible with
old userspace servers like universal nfsd and crypto filesystems as cfsd (which is an nfs server pretty old fashioned).

Linux NFS client issues with servers that are old or not widely used should be reported to [email protected] (thanks for forwarding this, Andrew).

User space servers are usually not tested with NFS client changes, since their use is infrequent compared with Solaris, NetApp filers, and the Linux knfsd (and several others). We have to draw the line somewhere to make NFS client development a manageable process.

Quoted 2 lines
Quoted 2 lines

- This problem makes UNFSD and CFSD unusable on 2.6.22 and up (this doesnt sound good to me)

It's likely the NFS client team is not aware of problems with these servers simply because none of us run them, and nothing has been reported so far.

Quoted 3 lines
Quoted 3 lines

The question are:

- Is this wanted or is a bug ?

An EINVAL return is fairly generic, but since it is coming from the reply path on the client, that probably indicates that decoding the reply failed somehow. That could be a client problem, or the server reply was incorrect or corrupt.

More specific information about the problem is needed. Could we see a wire trace? A raw tcpdump or tethereal dump captured during the problematic interaction would be helpful. Maybe even an strace of the failing application would tell us what the arguments for the setattr() call are. And what are your mount options, especially which NFS version? (cat /proc/mounts)

Have you tried a 'git bisect' or something similar to track down exactly when client behavior changed?

Quoted 4 lines
Quoted 4 lines

- Can this be solved with some backwards compat stuff into the kernel or
all the old packages as UNFSD have to be bugfixed/upgraded

If NFS servers don't conform to the NFS protocol specification, then it's unlikely that the NFS client will be changed to fix such issues. Ie: server bugs need to be fixed on the server. That is sometimes difficult if the server is not maintained, for instance.

Quoted 4 lines
Quoted 4 lines

- I have found other strange behaviors of the new NFS filesystem support
that i am investigating. All are bound to the user of old userspace servers onto the new NFS (since 2.6.22). What to do ?

Report the problems on the [email protected] mailing list, or
document them in the official bug databases (either the Linux kernel bugzilla or the NFSv4 bug tracker at http://bugzilla.linux-nfs.org/

Quoted 2 lines

I'm not sure what the NFS client's policy is regarding support for userspace servers. But I'd certainly hope that it is "don't break them".

The general policy is that if a server behaves in ways that don't conform to the NFS spec, then the Linux NFS client probably won't work with it. If the client works with a broken server today, there is no guarantee that it will continue to work.

Quoted 1 line

Which would make this an NFS client regression.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#6 lore raw

Hello all,

Sorry for being down for so long but i have decided to test the latest 2.6.24 kernel because of the huge number of NFS patches (especially at RPC UDP protocol level) introduced mainly by Netapp people (Trond Myklebust et al.), hoping that maybe the problem would have magically disappeared, but...

On kernel 2.6.24 rc7:

- Debian nfs-user-server (v2) is giving the very same problem i reported last times: EINVAL on open() syscall with O|TRUNC on existing file
- CFSD, same as debian nfs-user-server (v2)

To answer Chuck's questions:

i have already straced all the applications, which puts in evidence that the problem is into the open() syscall with O|TRUNC on existing files, always. I prepared that very basic piece of c code to rise the problem, as you can see it just 'syscalls' open(), and that is enough. NFS version is 2, no options (for CFSD [ -oport=3049,intr ] as i ever did)

When im done with this email i will work on finding where exactly the problem arises, for now i can say that 2.6.21.7 is working so well, and 2.6.22.16 is not working exactly as 2.6.24rc7 does.

I am really interested in finding out whats wrong with NFSD/CFSD and if it was a problem of NFS compliance i would really be glad to find out what to do to patch those servers, but im really far to be able to do it alone.
But: dont you think that we should find out that the problem is surely into the userspace servers before saying that everything is alright with linux kernel ?

I propose to anybody interested to test cfsd or nfs-user-server 2.2beta47-23 (from debian etch) on any kernel >= 2.6.23. You ca use something like the piece of code included in this email, or you can simply:

echo "Hello" > /mnt/nfsmount/test

which is gonna fail after the first time (the first it creates a new file and it works) with EINVAL.

Now as i already said the problem is an RPC call_decode which fails, this is evident from an RPC debug. I have noticed a different number of bytes in the response from knfsd to userspace (28 instead of 96).

Somebody can help ?

Gianluca

Quoted 147 lines · Chuck Lever
Quoted 82 lines · Andrew Morton
Quoted 80 lines · On Sun, 23 Dec 2007 12:35:17 +0100 Gianluca Alberici…

Hello,

I can do better. I have investigated a bit the problem:

1) The problem arises only with the userspace nfsd (Universal nfsd 2.2). I have realized that the latest patches introduced in 2.6.22 have changed a lot of things into NFS.

2) The following code has been debugged with sunrpc.rpc_debug and sunrpc.nfs_debug

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
int fp;
if ((fp=open("/mnt/tmp/test",O_WRONLY | O_CREAT | O_TRUNC, S_IRWXU )) == -1) printf("ERR: %d\n",errno); else {
write(fp, argv[1], strlen(argv[1]));
printf("OK\n");
close(fp);
};
}

2b) The output

[...]

<8>NFS call setattr
<8>RPC: new task initialized, procpid 18656 <8>RPC: allocated task f7bec500
<8>RPC: 0 looking up UNIX cred
<8>RPC: 740 __rpc_execute flags=0x480
<8>RPC: 740 call_start nfs2 proc 2 (sync)
<8>RPC: 740 call_reserve (status 0)
<8>RPC: 740 reserved req f1416000 xid 643f3c42 <8>RPC: 740 call_reserveresult (status 0)
<8>RPC: 740 call_allocate (status 0)
<8>RPC: 740 allocated buffer of size 528 at e627b800 <8>RPC: 740 call_bind (status 0)
<8>RPC: 740 call_connect xprt f70d4000 is connected <8>RPC: 740 call_transmit (status 0)
<8>RPC: 740 xprt_prepare_transmit
<8>RPC: 740 xprt_cwnd_limited cong = 0 cwnd = 512 <8>RPC: 740 call_encode (status 0)
<8>RPC: 740 marshaling UNIX cred f4efcb40
<8>RPC: 740 using AUTH_UNIX cred f4efcb40 to wrap rpc data <8>RPC: 740 xprt_transmit(148)
<8>RPC: xs_udp_data_ready...
<8>RPC: cong 256, cwnd was 512, now 512
<8>RPC: 740 xid 643f3c42 complete (28 bytes received) <8>RPC: xs_udp_send_request(148) = 148
<8>RPC: 740 xmit complete
<8>RPC: wake_up_next(f70d4114 "xprt_resend") <8>RPC: wake_up_next(f70d40e4 "xprt_sending") <8>RPC: 740 call_status (status 28)
<8>RPC: 740 call_decode (status 28)
<8>RPC: 740 validating UNIX cred f4efcb40
<8>RPC: 740 using AUTH_UNIX cred f4efcb40 to unwrap rpc data <8>RPC: 740 call_decode result -22
<8>RPC: 740 return 0, status -22
<8>RPC: 740 release task
<8>RPC: freeing buffer of size 528 at e627b800 <8>RPC: 740 release request f1416000
<8>RPC: wake_up_next(f70d4174 "xprt_backlog") <8>RPC: 740 releasing UNIX cred f4efcb40
<8>RPC: rpc_release_client(f6f1f880)
<8>NFS reply setattr: -22

[...]

3) It turns out the following:

- setattr returns EINVAL due to...
- RPC call_decode returns status 22

Quoted 7 lines
Quoted 6 lines

4) In conclusion in my understanding:

- At present linux nfs filesystem support is not anymore compatible with
old userspace servers like universal nfsd and crypto filesystems as cfsd (which is an nfs server pretty old fashioned).

Linux NFS client issues with servers that are old or not widely used should be reported to [email protected] (thanks for forwarding this, Andrew).

User space servers are usually not tested with NFS client changes, since their use is infrequent compared with Solaris, NetApp filers, and the Linux knfsd (and several others). We have to draw the line somewhere to make NFS client development a manageable process.

Quoted 3 lines
Quoted 2 lines

- This problem makes UNFSD and CFSD unusable on 2.6.22 and up (this doesnt sound good to me)

It's likely the NFS client team is not aware of problems with these servers simply because none of us run them, and nothing has been reported so far.

Quoted 4 lines
Quoted 3 lines

The question are:

- Is this wanted or is a bug ?

An EINVAL return is fairly generic, but since it is coming from the reply path on the client, that probably indicates that decoding the reply failed somehow. That could be a client problem, or the server reply was incorrect or corrupt.

More specific information about the problem is needed. Could we see a wire trace? A raw tcpdump or tethereal dump captured during the problematic interaction would be helpful. Maybe even an strace of the failing application would tell us what the arguments for the setattr() call are. And what are your mount options, especially which NFS version? (cat /proc/mounts)

Have you tried a 'git bisect' or something similar to track down exactly when client behavior changed?

Quoted 5 lines
Quoted 4 lines

- Can this be solved with some backwards compat stuff into the kernel or
all the old packages as UNFSD have to be bugfixed/upgraded

If NFS servers don't conform to the NFS protocol specification, then it's unlikely that the NFS client will be changed to fix such issues. Ie: server bugs need to be fixed on the server. That is sometimes difficult if the server is not maintained, for instance.

Quoted 5 lines
Quoted 4 lines

- I have found other strange behaviors of the new NFS filesystem support
that i am investigating. All are bound to the user of old userspace servers onto the new NFS (since 2.6.22). What to do ?

Report the problems on the [email protected] mailing list, or
document them in the official bug databases (either the Linux kernel bugzilla or the NFSv4 bug tracker at http://bugzilla.linux-nfs.org/

Quoted 2 lines

I'm not sure what the NFS client's policy is regarding support for userspace servers. But I'd certainly hope that it is "don't break them".

The general policy is that if a server behaves in ways that don't conform to the NFS spec, then the Linux NFS client probably won't work with it. If the client works with a broken server today, there is no guarantee that it will continue to work.

Quoted 1 line

Which would make this an NFS client regression.

Signature · 7 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
-- 
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
#7 lore raw
In reply to Chuck Lever

Hello all,

Sorry for being down for so long but i have decided to test the latest 2.6.24 kernel because of the huge number of NFS patches (especially at RPC UDP protocol level) introduced mainly by Netapp people (Trond Myklebust et al.), hoping that maybe the problem would have magically disappeared, but...

On kernel 2.6.24 rc7:

- Debian nfs-user-server (v2) is giving the very same problem i reported last times: EINVAL on open() syscall with O|TRUNC on existing file
- CFSD, same as debian nfs-user-server (v2)

To answer Chuck's questions:

i have already straced all the applications, which puts in evidence that the problem is into the open() syscall with O|TRUNC on existing files, always.
I prepared that very basic piece of c code to rise the problem, as you can see it just 'syscalls' open(), and that is enough.
NFS version is 2, no options (for CFSD [ -oport=3049,intr ] as i ever did)

When im done with this email i will work on finding where exactly the problem arises, for now i can say that 2.6.21.7 is working so well, and 2.6.22.16 is not working exactly as 2.6.24rc7 does.

I am really interested in finding out whats wrong with NFSD/CFSD and if it was a problem of NFS compliance i would really be glad to find out what to do to patch those servers, but im really far to be able to do it alone.
But: dont you think that we should find out that the problem is surely into the userspace servers before saying that everything is alright with linux kernel ?

I propose to anybody interested to test cfsd or nfs-user-server 2.2beta47-23 (from debian etch) on any kernel >= 2.6.23. You ca use something like the piece of code included in this email, or you can simply:

echo "Hello" > /mnt/nfsmount/test

which is gonna fail after the first time (the first it creates a new file and it works) with EINVAL.

Now as i already said the problem is an RPC call_decode which fails, this is evident from an RPC debug. I have noticed a different number of bytes in the response from knfsd to userspace (28 instead of 96).

Somebody can help ?

Gianluca

Quoted 147 lines · Chuck Lever
Quoted 82 lines · Andrew Morton
Quoted 80 lines · On Sun, 23 Dec 2007 12:35:17 +0100 Gianluca Alberici…

Hello,

I can do better. I have investigated a bit the problem:

1) The problem arises only with the userspace nfsd (Universal nfsd 2.2). I have realized that the latest patches introduced in 2.6.22 have changed a lot of things into NFS.

2) The following code has been debugged with sunrpc.rpc_debug and sunrpc.nfs_debug

#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <errno.h>
#include <stdio.h>

int main(int argc, char *argv[]) {
int fp;
if ((fp=open("/mnt/tmp/test",O_WRONLY | O_CREAT | O_TRUNC, S_IRWXU )) == -1) printf("ERR: %d\n",errno); else {
write(fp, argv[1], strlen(argv[1]));
printf("OK\n");
close(fp);
};
}

2b) The output

[...]

<8>NFS call setattr
<8>RPC: new task initialized, procpid 18656 <8>RPC: allocated task f7bec500
<8>RPC: 0 looking up UNIX cred
<8>RPC: 740 __rpc_execute flags=0x480
<8>RPC: 740 call_start nfs2 proc 2 (sync)
<8>RPC: 740 call_reserve (status 0)
<8>RPC: 740 reserved req f1416000 xid 643f3c42 <8>RPC: 740 call_reserveresult (status 0)
<8>RPC: 740 call_allocate (status 0)
<8>RPC: 740 allocated buffer of size 528 at e627b800 <8>RPC: 740 call_bind (status 0)
<8>RPC: 740 call_connect xprt f70d4000 is connected <8>RPC: 740 call_transmit (status 0)
<8>RPC: 740 xprt_prepare_transmit
<8>RPC: 740 xprt_cwnd_limited cong = 0 cwnd = 512 <8>RPC: 740 call_encode (status 0)
<8>RPC: 740 marshaling UNIX cred f4efcb40
<8>RPC: 740 using AUTH_UNIX cred f4efcb40 to wrap rpc data <8>RPC: 740 xprt_transmit(148)
<8>RPC: xs_udp_data_ready...
<8>RPC: cong 256, cwnd was 512, now 512
<8>RPC: 740 xid 643f3c42 complete (28 bytes received) <8>RPC: xs_udp_send_request(148) = 148
<8>RPC: 740 xmit complete
<8>RPC: wake_up_next(f70d4114 "xprt_resend") <8>RPC: wake_up_next(f70d40e4 "xprt_sending") <8>RPC: 740 call_status (status 28)
<8>RPC: 740 call_decode (status 28)
<8>RPC: 740 validating UNIX cred f4efcb40
<8>RPC: 740 using AUTH_UNIX cred f4efcb40 to unwrap rpc data <8>RPC: 740 call_decode result -22
<8>RPC: 740 return 0, status -22
<8>RPC: 740 release task
<8>RPC: freeing buffer of size 528 at e627b800 <8>RPC: 740 release request f1416000
<8>RPC: wake_up_next(f70d4174 "xprt_backlog") <8>RPC: 740 releasing UNIX cred f4efcb40
<8>RPC: rpc_release_client(f6f1f880)
<8>NFS reply setattr: -22

[...]

3) It turns out the following:

- setattr returns EINVAL due to...
- RPC call_decode returns status 22

Quoted 7 lines
Quoted 6 lines

4) In conclusion in my understanding:

- At present linux nfs filesystem support is not anymore compatible with
old userspace servers like universal nfsd and crypto filesystems as cfsd (which is an nfs server pretty old fashioned).

Linux NFS client issues with servers that are old or not widely used should be reported to [email protected] (thanks for forwarding this, Andrew).

User space servers are usually not tested with NFS client changes, since their use is infrequent compared with Solaris, NetApp filers, and the Linux knfsd (and several others). We have to draw the line somewhere to make NFS client development a manageable process.

Quoted 3 lines
Quoted 2 lines

- This problem makes UNFSD and CFSD unusable on 2.6.22 and up (this doesnt sound good to me)

It's likely the NFS client team is not aware of problems with these servers simply because none of us run them, and nothing has been reported so far.

Quoted 4 lines
Quoted 3 lines

The question are:

- Is this wanted or is a bug ?

An EINVAL return is fairly generic, but since it is coming from the reply path on the client, that probably indicates that decoding the reply failed somehow. That could be a client problem, or the server reply was incorrect or corrupt.

More specific information about the problem is needed. Could we see a wire trace? A raw tcpdump or tethereal dump captured during the problematic interaction would be helpful. Maybe even an strace of the failing application would tell us what the arguments for the setattr() call are. And what are your mount options, especially which NFS version? (cat /proc/mounts)

Have you tried a 'git bisect' or something similar to track down exactly when client behavior changed?

Quoted 5 lines
Quoted 4 lines

- Can this be solved with some backwards compat stuff into the kernel or
all the old packages as UNFSD have to be bugfixed/upgraded

If NFS servers don't conform to the NFS protocol specification, then it's unlikely that the NFS client will be changed to fix such issues. Ie: server bugs need to be fixed on the server. That is sometimes difficult if the server is not maintained, for instance.

Quoted 5 lines
Quoted 4 lines

- I have found other strange behaviors of the new NFS filesystem support
that i am investigating. All are bound to the user of old userspace servers onto the new NFS (since 2.6.22). What to do ?

Report the problems on the [email protected] mailing list, or
document them in the official bug databases (either the Linux kernel bugzilla or the NFSv4 bug tracker at http://bugzilla.linux-nfs.org/

Quoted 2 lines

I'm not sure what the NFS client's policy is regarding support for userspace servers. But I'd certainly hope that it is "don't break them".

The general policy is that if a server behaves in ways that don't conform to the NFS spec, then the Linux NFS client probably won't work with it. If the client works with a broken server today, there is no guarantee that it will continue to work.

Quoted 1 line

Which would make this an NFS client regression.

Signature · 7 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
-- 
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
#8 lore raw
In reply to Chuck Lever

Hello Chuck,

As you requested i have finally found out that the problem was introduced in 2.6.22 (2.6.21.7 works, after 2.6.21.7 theres 2.6.22)

I have also examined the changeslogs: 2.6.22 introduce a certain number of patches into NFS client. Many of them are yours and Trond's.

Pls can you help with this ?

TIA

Gianluca

A message that is not in the archive (sent off-list or not yet received): [email protected]

#10 lore raw

Hi Gianluca-

Quoted 7 lines · Gianluca Alberici

Hello Chuck,

As you requested i have finally found out that the problem was introduced in 2.6.22 (2.6.21.7 works, after 2.6.21.7 theres 2.6.22)

I have also examined the changeslogs: 2.6.22 introduce a certain number of patches into NFS client. Many of them are yours and Trond's.

Pls can you help with this ?

[ I trimmed the Linux kernel mailing list because this is an NFS specific problem. No need to add more noise on lkml. ]

The first thing to do is to capture a network trace during the failing operation. Capturing a trace with a working kernel, and one with a failing kernel, would let us see how the behavior changes between the two.

To capture a network trace, you can use "tcpdump" on your client:

sudo tcpdump -s0 -w /tmp/dumpfile hostname-of-server

Substitute the fully-qualified domain name of your server for "hostname-of-server".

Try this with a working client and a broken client. The /tmp/dumpfile will be a binary file. You can post it on a convenient web site, or attach it to e-mail.

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#11 lore raw

Hello Chuck,

i have produced the output you requested using the code i used to show you last time (which simply tries to open(... | O_TRUNC) a file onto the nfs mount and writes "Hello" into it. I simply iterate execution 2 times. The mount is a loop mount on 127.0.0.1 Since the second execution (the first time it creates the file) you get EINVAL:

FILE CREATION:

hydra:~# tcpdump -s0 -i lo port 2049
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
12:15:06.306619 IP localhost.251828621 > localhost.nfs: 120 getattr fh Unknown/47521E2B0223C100000000000000000000000000000000000000000000000000
12:15:06.306666 IP localhost.nfs > localhost.251828621: reply ok 96 getattr DIR 40777 ids 0/0 sz 4096
12:15:06.306705 IP localhost.268605837 > localhost.nfs: 128 lookup fh Unknown/47521E2B0223C100000000000000000000000000000000000000000000000000 "test"
12:15:06.306752 IP localhost.nfs > localhost.268605837: reply ok 28 lookup ERROR: No such file or directory
12:15:06.306786 IP localhost.285383053 > localhost.nfs: 160 create fh Unknown/47521E2B0223C100000000000000000000000000000000000000000000000000 "test"
12:15:06.306917 IP localhost.nfs > localhost.285383053: reply ok 128 create fh Unknown/48521E2B0323C120000000000000000000000000000000000000000000000000
12:15:06.307179 IP localhost.302160269 > localhost.nfs: 144 write fh Unknown/48521E2B0323C120000000000000000000000000000000000000000000000000 5 (5) bytes @ 0 (0)
12:15:06.307283 IP localhost.nfs > localhost.302160269: reply ok 96 write

...EINVAL ON OPEN

12:15:14.825249 IP localhost.318937485 > localhost.nfs: 120 getattr fh Unknown/48521E2B0323C120000000000000000000000000000000000000000000000000 12:15:14.825337 IP localhost.nfs > localhost.318937485: reply ok 96 getattr REG 100700 ids 99/99 sz 5
12:15:14.825403 IP localhost.335714701 > localhost.nfs: 152 setattr fh Unknown/48521E2B0323C120000000000000000000000000000000000000000000000000 12:15:14.825496 IP localhost.nfs > localhost.335714701: reply ok 28 setattr ERROR: Invalid argument

This is a 2.6.23.9 (which shouldnt be different from 2.6.22 at least under nfs client POV)

Now for a working 2.6.21.7 kernel (FILE CREATION):

ios:~# tcpdump -s0 -i lo port 2049
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
12:37:09.463537 IP debian.1135581207 > debian.nfs: 116 getattr fh Unknown/0100182B0223D900000000000000000000000000000000000000000000000000
12:37:09.463845 IP debian.nfs > debian.1135581207: reply ok 96 getattr DIR 40777 ids 0/0 sz 4096
12:37:09.463939 IP debian.1152358423 > debian.nfs: 124 lookup fh Unknown/0100182B0223D900000000000000000000000000000000000000000000000000 "test"
12:37:09.464012 IP debian.nfs > debian.1152358423: reply ok 28 lookup ERROR: No such file or directory
12:37:09.464083 IP debian.1169135639 > debian.nfs: 156 create fh Unknown/0100182B0223D900000000000000000000000000000000000000000000000000 "test"
12:37:09.464293 IP debian.nfs > debian.1169135639: reply ok 128 create fh Unknown/0200182B0323D932000000000000000000000000000000000000000000000000
12:37:09.464488 IP debian.1185912855 > debian.nfs: 140 write fh Unknown/0200182B0323D932000000000000000000000000000000000000000000000000 5 (5) bytes @ 0 (0)
12:37:09.464674 IP debian.nfs > debian.1185912855: reply ok 96 write

and open O | TRUNC going perfect

12:37:14.442789 IP debian.1202690071 > debian.nfs: 116 getattr fh Unknown/0200182B0323D932000000000000000000000000000000000000000000000000 12:37:14.442868 IP debian.nfs > debian.1202690071: reply ok 96 getattr REG 100700 ids 99/99 sz 5
12:37:14.442998 IP debian.1219467287 > debian.nfs: 148 setattr fh Unknown/0200182B0323D932000000000000000000000000000000000000000000000000 12:37:14.443165 IP debian.nfs > debian.1219467287: reply ok 96 setattr REG 100700 ids 99/99 sz 0
12:37:14.443394 IP debian.1236244503 > debian.nfs: 140 write fh Unknown/0200182B0323D932000000000000000000000000000000000000000000000000 5 (5) bytes @ 0 (0) 12:37:14.443480 IP debian.nfs > debian.1236244503: reply ok 96 write
12:37:14.443511 IP debian.1253021719 > debian.nfs: 116 getattr fh Unknown/0200182B0323D932000000000000000000000000000000000000000000000000 12:37:14.443551 IP debian.nfs > debian.1253021719: reply ok 96 getattr REG 100700 ids 99/99 sz 5

Here follows the kernel debug info @ setattr for the 'broken' kernel/uspace-server:

Quoted 32 lines
Quoted 32 lines
Quoted 32 lines

<8>NFS call setattr
<8>RPC: new task initialized, procpid 18656 <8>RPC: allocated task f7bec500
<8>RPC: 0 looking up UNIX cred
<8>RPC: 740 __rpc_execute flags=0x480
<8>RPC: 740 call_start nfs2 proc 2 (sync)
<8>RPC: 740 call_reserve (status 0)
<8>RPC: 740 reserved req f1416000 xid 643f3c42 <8>RPC: 740 call_reserveresult (status 0)
<8>RPC: 740 call_allocate (status 0)
<8>RPC: 740 allocated buffer of size 528 at e627b800 <8>RPC: 740 call_bind (status 0)
<8>RPC: 740 call_connect xprt f70d4000 is connected <8>RPC: 740 call_transmit (status 0)
<8>RPC: 740 xprt_prepare_transmit
<8>RPC: 740 xprt_cwnd_limited cong = 0 cwnd = 512 <8>RPC: 740 call_encode (status 0)
<8>RPC: 740 marshaling UNIX cred f4efcb40
<8>RPC: 740 using AUTH_UNIX cred f4efcb40 to wrap rpc data <8>RPC: 740 xprt_transmit(148)
<8>RPC: xs_udp_data_ready...
<8>RPC: cong 256, cwnd was 512, now 512
<8>RPC: 740 xid 643f3c42 complete (28 bytes received) <8>RPC: xs_udp_send_request(148) = 148
<8>RPC: 740 xmit complete
<8>RPC: wake_up_next(f70d4114 "xprt_resend") <8>RPC: wake_up_next(f70d40e4 "xprt_sending") <8>RPC: 740 call_status (status 28)
<8>RPC: 740 call_decode (status 28)
<8>RPC: 740 validating UNIX cred f4efcb40
<8>RPC: 740 using AUTH_UNIX cred f4efcb40 to unwrap rpc data <8>RPC: 740 call_decode result -22

Quoted 8 lines
Quoted 8 lines
Quoted 8 lines

<8>RPC: 740 return 0, status -22
<8>RPC: 740 release task
<8>RPC: freeing buffer of size 528 at e627b800
<8>RPC: 740 release request f1416000
<8>RPC: wake_up_next(f70d4174 "xprt_backlog")
<8>RPC: 740 releasing UNIX cred f4efcb40
<8>RPC: rpc_release_client(f6f1f880)
<8>NFS reply setattr: -22

...and for the working one:

<4>NFS call setattr
<4>RPC: new task initialized, procpid 12112
<4>RPC: allocated task f3ae2140
<4>RPC: 0 looking up UNIX cred

<4>RPC:   171 __rpc_execute flags=0x480
<4>RPC:   171 call_start nfs2 proc 2 (sync)
<4>RPC:   171 call_reserve (status 0)
<4>RPC:   171 reserved req f3cb4000 xid 4daf9817
<4>RPC:   171 call_reserveresult (status 0)
<4>RPC:   171 call_allocate (status 0)
<4>RPC:   171 call_bind (status 0)
<4>RPC:   171 call_connect xprt f6b83400 is connected
<4>RPC:   171 call_transmit (status 0)
<4>RPC:   171 xprt_prepare_transmit
<4>RPC:   171 xprt_cwnd_limited cong = 0 cwnd = 512
<4>RPC:   171 call_encode (status 0)
<4>RPC:   171 marshaling UNIX cred e984f1c0
<4>RPC:   171 using AUTH_UNIX cred e984f1c0 to wrap rpc data
<4>RPC:   171 xprt_transmit(148)
<4>RPC:       xs_udp_send_request(148) = 148
<4>RPC:   171 xmit complete
<4>RPC:   171 sleep_on(queue "xprt_pending" time 81013324)
<4>RPC:   171 added to queue f6b83544 "xprt_pending"
<4>RPC:   171 setting alarm for 700 ms
<4>RPC:       wake_up_next(f6b83514 "xprt_resend")
<4>RPC:       wake_up_next(f6b834e4 "xprt_sending")
<4>RPC:   171 sync task going to sleep
<4>RPC:       xs_udp_data_ready...
<4>RPC:       cong 256, cwnd was 512, now 512
<4>RPC:       wake_up_next(f6b83514 "xprt_resend")
<4>RPC:       wake_up_next(f6b834e4 "xprt_sending")
<4>RPC:   171 xid 4daf9817 complete (96 bytes received)
<4>RPC:   171 __rpc_wake_up_task (now 81013324)
<4>RPC:   171 disabling timer
<4>RPC:   171 removed from queue f6b83544 "xprt_pending"
<4>RPC:       __rpc_wake_up_task done
<4>RPC:   171 sync task resuming
<4>RPC:   171 deleting timer
<4>RPC:   171 call_status (status 96)
<4>RPC:   171 call_decode (status 96)
<4>RPC:   171 validating UNIX cred e984f1c0
<4>RPC:   171 using AUTH_UNIX cred e984f1c0 to unwrap rpc data
<4>RPC:   171 call_decode result 0
<4>RPC:   171 return 0, status 0
<4>RPC:   171 release task
<4>RPC:   171 release request f3cb4000

<4>RPC: wake_up_next(f6b83574 "xprt_backlog")
<4>RPC: 171 releasing UNIX cred e984f1c0
<4>RPC: rpc_release_client(cab12680, 1)
<4>NFS reply setattr: 0

I can notice the error returned by call_decode and the different size of reply (which is an error reply sized 28). Might this be bound to the different packet lengths ? In the working solution packets are always 4 bytes shorter than in the broken one.

thanks for any help,

Gianluca

Quoted 32 lines · Chuck Lever

Hi Gianluca-

Quoted 7 lines · Gianluca Alberici

Hello Chuck,

As you requested i have finally found out that the problem was introduced in 2.6.22 (2.6.21.7 works, after 2.6.21.7 theres 2.6.22)

I have also examined the changeslogs: 2.6.22 introduce a certain number of patches into NFS client. Many of them are yours and Trond's.

Pls can you help with this ?

[ I trimmed the Linux kernel mailing list because this is an NFS specific problem. No need to add more noise on lkml. ]

The first thing to do is to capture a network trace during the failing operation. Capturing a trace with a working kernel, and one with a failing kernel, would let us see how the behavior changes between the two.

To capture a network trace, you can use "tcpdump" on your client:

sudo tcpdump -s0 -w /tmp/dumpfile hostname-of-server

Substitute the fully-qualified domain name of your server for "hostname-of-server".

Try this with a working client and a broken client. The /tmp/ dumpfile will be a binary file. You can post it on a convenient web site, or attach it to e-mail.

Signature · 2 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#13 lore raw

Hi Gianluca-

Quoted 7 lines · Gianluca Alberici

Hello Chuck,

i have produced the output you requested using the code i used to show you last time (which simply tries to open(... | O_TRUNC) a file onto the nfs mount and writes "Hello" into it. I simply iterate execution 2 times. The mount is a loop mount on 127.0.0.1 Since the second execution (the first time it creates the file) you get EINVAL:

FILE CREATION:

Quoted 11 lines
hydra:~# tcpdump -s0 -i lo port 2049
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
12:15:06.306619 IP localhost.251828621 > localhost.nfs: 120 getattr fh Unknown/47521E2B0223C100000000000000000000000000000000000000000000000000
12:15:06.306666 IP localhost.nfs > localhost.251828621: reply ok 96 getattr DIR 40777 ids 0/0 sz 4096
12:15:06.306705 IP localhost.268605837 > localhost.nfs: 128 lookup fh Unknown/47521E2B0223C100000000000000000000000000000000000000000000000000 "test"
12:15:06.306752 IP localhost.nfs > localhost.268605837: reply ok 28 lookup ERROR: No such file or directory
12:15:06.306786 IP localhost.285383053 > localhost.nfs: 160 create fh Unknown/47521E2B0223C100000000000000000000000000000000000000000000000000 "test"
12:15:06.306917 IP localhost.nfs > localhost.285383053: reply ok 128 create fh Unknown/48521E2B0323C120000000000000000000000000000000000000000000000000
12:15:06.307179 IP localhost.302160269 > localhost.nfs: 144 write fh Unknown/48521E2B0323C120000000000000000000000000000000000000000000000000 5 (5) bytes @ 0 (0)
12:15:06.307283 IP localhost.nfs > localhost.302160269: reply ok 96 write

We need to have the raw output of tcpdump. Please use "-w dumpfile" and send the raw output.

Quoted 1 line
Quoted 1 line

sudo tcpdump -s0 -w /tmp/dumpfile hostname-of-server

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#14 lore raw

Hello Chuck,

I attach as you requested the two dumpfiles obtained by

tcpdump -s0 -i lo -w /tmp/dump-(not-)working port 2049

They contain the dump relative to the usual double try: at first the open() syscall creates the file, while in the second tries to truncate to zero length.

Hope these are ok with you.

I am waiting for istructions if you got any.

Thanks,

Gianluca

Quoted 37 lines · Chuck Lever

Hi Gianluca-

Quoted 7 lines · Gianluca Alberici

Hello Chuck,

i have produced the output you requested using the code i used to show you last time (which simply tries to open(... | O_TRUNC) a file onto the nfs mount and writes "Hello" into it. I simply iterate execution 2 times. The mount is a loop mount on 127.0.0.1 Since the second execution (the first time it creates the file) you get EINVAL:

FILE CREATION:

Quoted 11 lines
hydra:~# tcpdump -s0 -i lo port 2049
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
12:15:06.306619 IP localhost.251828621 > localhost.nfs: 120 getattr fh Unknown/ 47521E2B0223C100000000000000000000000000000000000000000000000000
12:15:06.306666 IP localhost.nfs > localhost.251828621: reply ok 96 getattr DIR 40777 ids 0/0 sz 4096
12:15:06.306705 IP localhost.268605837 > localhost.nfs: 128 lookup fh Unknown/ 47521E2B0223C100000000000000000000000000000000000000000000000000 "test"
12:15:06.306752 IP localhost.nfs > localhost.268605837: reply ok 28 lookup ERROR: No such file or directory
12:15:06.306786 IP localhost.285383053 > localhost.nfs: 160 create fh Unknown/ 47521E2B0223C100000000000000000000000000000000000000000000000000 "test"
12:15:06.306917 IP localhost.nfs > localhost.285383053: reply ok 128 create fh Unknown/ 48521E2B0323C120000000000000000000000000000000000000000000000000
12:15:06.307179 IP localhost.302160269 > localhost.nfs: 144 write fh Unknown/ 48521E2B0323C120000000000000000000000000000000000000000000000000 5 (5) bytes @ 0 (0)
12:15:06.307283 IP localhost.nfs > localhost.302160269: reply ok 96 write

We need to have the raw output of tcpdump. Please use "-w dumpfile" and send the raw output.

Quoted 2 lines
Quoted 1 line

sudo tcpdump -s0 -w /tmp/dumpfile hostname-of-server

Signature · 6 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
-
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

#16 lore raw
Quoted 9 lines · Gianluca Alberici

Hello Chuck,

I attach as you requested the two dumpfiles obtained by

tcpdump -s0 -i lo -w /tmp/dump-(not-)working port 2049

They contain the dump relative to the usual double try: at first the open() syscall creates the file, while in the second tries to truncate to zero length.

The client is doing a SETATTR to truncate the file, and the server returns NFSERR_INVAL (EINVAL) which is error code 22. This is not an RPC decoding problem: the server is genuinely returning an error.

If this were a client problem, we would see it with other servers as well, but this is the only reported case I am aware of. The client's SETATTR request looks valid in both cases (and the two requests are very nearly identical), so the next step is to look closely at your server to determine why the request fails in the "not working" case.

NFSERR_INVAL is quite uncommon, and in fact is not defined in the NFS version 2 specification (RFC 1094). This suggests that the server has encountered some kind of internal problem, or that it is simply a broken implementation.

I think you mentioned previously that the server is the Debian user-space server. You should contact Debian and ask for their help to diagnose the problem. (As far as I know there are no user-space server developers on this list, but I could be incorrect).

Quoted 41 lines
Quoted 24 lines · Chuck Lever

Hi Gianluca-

Quoted 7 lines · Gianluca Alberici

Hello Chuck,

i have produced the output you requested using the code i used to show you last time (which simply tries to open(... | O_TRUNC) a file onto the nfs mount and writes "Hello" into it. I simply iterate execution 2 times. The mount is a loop mount on 127.0.0.1 Since the second execution (the first time it creates the file) you get EINVAL:

FILE CREATION:

Quoted 11 lines
hydra:~# tcpdump -s0 -i lo port 2049
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on lo, link-type EN10MB (Ethernet), capture size 65535 bytes
12:15:06.306619 IP localhost.251828621 > localhost.nfs: 120 getattr fh Unknown/ 47521E2B0223C100000000000000000000000000000000000000000000000000
12:15:06.306666 IP localhost.nfs > localhost.251828621: reply ok 96 getattr DIR 40777 ids 0/0 sz 4096
12:15:06.306705 IP localhost.268605837 > localhost.nfs: 128 lookup fh Unknown/ 47521E2B0223C100000000000000000000000000000000000000000000000000 "test"
12:15:06.306752 IP localhost.nfs > localhost.268605837: reply ok 28 lookup ERROR: No such file or directory
12:15:06.306786 IP localhost.285383053 > localhost.nfs: 160 create fh Unknown/ 47521E2B0223C100000000000000000000000000000000000000000000000000 "test"
12:15:06.306917 IP localhost.nfs > localhost.285383053: reply ok 128 create fh Unknown/ 48521E2B0323C120000000000000000000000000000000000000000000000000
12:15:06.307179 IP localhost.302160269 > localhost.nfs: 144 write fh Unknown/ 48521E2B0323C120000000000000000000000000000000000000000000000000 5 (5) bytes @ 0 (0)
12:15:06.307283 IP localhost.nfs > localhost.302160269: reply ok 96 write
Quoted 13 lines

We need to have the raw output of tcpdump. Please use "-w dumpfile" and send the raw output.

Quoted 2 lines
Quoted 1 line

sudo tcpdump -s0 -w /tmp/dumpfile hostname-of-server

Signature · 6 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
-
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 1 line

<dump-working><dump-not-working>

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#17 lore raw
Quoted 4 lines · Chuck Lever

I think you mentioned previously that the server is the Debian user-space server. You should contact Debian and ask for their help to diagnose the problem. (As far as I know there are no user-space server developers on this list, but I could be incorrect).

(I've thought I've been advertising unfs3 almost too much :-))

If this problem shows up with unfs3, I can probably take a look at it.

Regards,
---
Peter Åstrand ThinLinc Chief Developer
Cendio AB http://www.cendio.se
Wallenbergs gata 4
583 30 Linköping Phone: +46-13-21 46 00

A message that is not in the archive (sent off-list or not yet received): Pine.LNX.4.64.0801291851110.4820-K9BqGu7AvB3wj5YHdwD3Ga2PxDmRETKR@public.gmane.org

#19 lore raw
Quoted 10 lines · On Jan 29, 2008, at 12:54 PM, Peter =C5strand wrote:
Quoted 7 lines · Chuck Lever

I think you mentioned previously that the server is the Debian =20 user-space
server. You should contact Debian and ask for their help to =20 diagnose the
problem. (As far as I know there are no user-space server =20 developers on this
list, but I could be incorrect).

Quoted 3 lines

(I've thought I've been advertising unfs3 almost too much :-))

If this problem shows up with unfs3, I can probably take a look at it=

=2E

From one of Gianluca's previous e-mails, the server is:

"I propose to anybody interested to test cfsd or nfs-user-server =20 2.2beta47-23 (from debian etch) on any kernel >=3D 2.6.23."

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#20 lore raw

Hello,

I confirm that i have encountered this same problem (EINVAL on open=20 (...O | TRUNC) with the following userspace servers:

- nfs-user-server shipped with debian sarge/etch etc...
- cfsd (crypto file system which is an nfs server)

I want to underline again that these userspace servers have been woking= =20
perfectly until 2.6.21.7 (which is the last 2.6.21)
Since 2.6.22 the problem came out and it is still present into 2.6.24=20 rc7 (last i tested). Conclusion: there must have been something that is changed in 2.6.22 that caused the problem.

I thought about nfs client because of the only difference between the=20 userspace server and the knfsd was the RPC call_decode() exit status=20 (see sysctl debug traces).

I have also tried unfs3 which, if i remember well, used to give me=20 EINVAL at mount time (!). But i should retest.
Also, i have tried to warn debian mantainers (both nfs-user-server and=20 cfsd) but without success.

Anyway any help is appreciated, and many thanks a lot for the help up t= o=20
now.

Regards,

Gianluca

Quoted 14 lines · Chuck Lever
Quoted 11 lines · On Jan 29, 2008, at 12:54 PM, Peter =C5strand wrote:
Quoted 7 lines · Chuck Lever

I think you mentioned previously that the server is the Debian =20 user-space
server. You should contact Debian and ask for their help to =20 diagnose the
problem. (As far as I know there are no user-space server =20 developers on this
list, but I could be incorrect).

Quoted 3 lines
Quoted 3 lines

(I've thought I've been advertising unfs3 almost too much :-))

If this problem shows up with unfs3, I can probably take a look at i=

t.

Quoted 10 lines

From one of Gianluca's previous e-mails, the server is:

"I propose to anybody interested to test cfsd or nfs-user-server =20 2.2beta47-23 (from debian etch) on any kernel >=3D 2.6.23."

--=20
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

A message that is not in the archive (sent off-list or not yet received): [email protected]

#22 lore raw
Quoted 5 lines · Gianluca Alberici

Hello,

I confirm that i have encountered this same problem (EINVAL on open (...O | TRUNC) with the following userspace servers:

Quoted 8 lines

- nfs-user-server shipped with debian sarge/etch etc...
- cfsd (crypto file system which is an nfs server)

I want to underline again that these userspace servers have been woking perfectly until 2.6.21.7 (which is the last 2.6.21)
Since 2.6.22 the problem came out and it is still present into 2.6.24 rc7 (last i tested). Conclusion: there must have been something that is changed in 2.6.22 that caused the problem.

The only difference between these two dumps are the fact that the first one isn't using the Sun convention for telling NFSv2 servers to set to the current time (see the code in xdr_encode_current_server_time).

I don't see why this would be new behaviour after 2.6.21. The code for this has been in the NFS client since 2.6.15 at least...

Trond

A message that is not in the archive (sent off-list or not yet received): 1201638661.7969.7.camel-rJ7iovZKK19ZJLDQqaL3InhyD016LWXt@public.gmane.org

#24 lore raw
Quoted 15 lines · Trond Myklebust
Quoted 5 lines · Gianluca Alberici

Hello,

I confirm that i have encountered this same problem (EINVAL on open (...O | TRUNC) with the following userspace servers:

Quoted 8 lines

- nfs-user-server shipped with debian sarge/etch etc...
- cfsd (crypto file system which is an nfs server)

I want to underline again that these userspace servers have been woking perfectly until 2.6.21.7 (which is the last 2.6.21)
Since 2.6.22 the problem came out and it is still present into 2.6.24 rc7 (last i tested). Conclusion: there must have been something that is changed in 2.6.22 that caused the problem.

Quoted 3 lines

The only difference between these two dumps are the fact that the first one isn't using the Sun convention for telling NFSv2 servers to set to the current time (see the code in xdr_encode_current_server_time).

I thought I saw that on both SETATTRs, but I could be wrong.

Quoted 2 lines

I don't see why this would be new behaviour after 2.6.21. The code for this has been in the NFS client since 2.6.15 at least...

A mount option is set on one test client, and not the other, perhaps?

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#25 lore raw

Hello again everybody

Here follows the testbench:

- I got two mirrors, same machine, same disk etc...chaged hostname, IP, and on the second i have recompiled kernel.
- First: 2.6.21.7 on debian sarge
- Second: 2.6.22 same system.
- Onto both i got nfs-user-server and cfsd last versions
- The export file is the same (localhost /opt/nfs (rw, async), stripping off the async option does not changes anything)
- Mount options are exactly the same.

The problem arises in the very same manner with both nfs and cfsd:

NFS:setattr {
...
...
RPC:call_decode {
return 22;
}
...
return 22;
}

I have tried these kernels:

2.6.16.11 works
2.6.20 works
2.6.21 works
2.6.21.7 works
2.6.22 doesnt work (contiguous to previous version)
2.6.23 doesnt work (same behavior as previous)
2.6.23.9 doesnt work (as above)
2.6.24rc7 doesnt work (as above)

I would really like to do more, client or server side, if you ave any suggestions.
Can we find out what is the change (doesnt matter if it is a buf or bug fix) that caused this problem ?

Thanks,

Gianluca

Quoted 40 lines · Chuck Lever
Quoted 17 lines · Trond Myklebust
Quoted 5 lines · Gianluca Alberici

Hello,

I confirm that i have encountered this same problem (EINVAL on open (...O | TRUNC) with the following userspace servers:

Quoted 8 lines

- nfs-user-server shipped with debian sarge/etch etc...
- cfsd (crypto file system which is an nfs server)

I want to underline again that these userspace servers have been woking perfectly until 2.6.21.7 (which is the last 2.6.21)
Since 2.6.22 the problem came out and it is still present into 2.6.24 rc7 (last i tested). Conclusion: there must have been something that is changed in 2.6.22 that caused the problem.

Quoted 3 lines

The only difference between these two dumps are the fact that the first one isn't using the Sun convention for telling NFSv2 servers to set to the current time (see the code in xdr_encode_current_server_time).

I thought I saw that on both SETATTRs, but I could be wrong.

Quoted 2 lines

I don't see why this would be new behaviour after 2.6.21. The code for this has been in the NFS client since 2.6.15 at least...

A mount option is set on one test client, and not the other, perhaps?

Signature · 6 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
-
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

A message that is not in the archive (sent off-list or not yet received): [email protected]

#27 lore raw

Hi Gianluca-

Quoted 15 lines · Gianluca Alberici

Hello again everybody

Here follows the testbench:

- I got two mirrors, same machine, same disk etc...chaged hostname, IP, and on the second i have recompiled kernel.
- First: 2.6.21.7 on debian sarge
- Second: 2.6.22 same system.
- Onto both i got nfs-user-server and cfsd last versions
- The export file is the same (localhost /opt/nfs (rw, async), stripping off the async option does not changes anything)
- Mount options are exactly the same.

The problem arises in the very same manner with both nfs and cfsd:

Quoted 9 lines

NFS:setattr {
...
...
RPC:call_decode {
return 22;
}
...
return 22;
}

Again, there is nothing wrong with the RPC client or call_decode. The *server* is returning NFSERR_INVAL (22) to a SETATTR request; the RPC client is simply passing that along to the NFS client, as it is designed to do.

Quoted 11 lines

I have tried these kernels:

2.6.16.11 works
2.6.20 works
2.6.21 works
2.6.21.7 works
2.6.22 doesnt work (contiguous to previous version)
2.6.23 doesnt work (same behavior as previous)
2.6.23.9 doesnt work (as above)
2.6.24rc7 doesnt work (as above)

Quoted 2 lines

I would really like to do more, client or server side, if you ave any suggestions.
Can we find out what is the change (doesnt matter if it is a buf or bug fix) that caused this problem ?

The goal here is to identify the kernel change between 2.6.21 and 2.6.22 that makes the client generate SETATTR requests the user-space server chokes on. It may be a change in the NFS client, or it could be somewhere else in the file system stack, like the VFS.

The usual procedure is to use "git bisect". It does a binary search on the kernel patches between the working kernel version and the kernel version that is known not to work. It works like this:

1. You clone a linux kernel git repository (if you don't have a git repository already)

2. You tell git bisect which kernel version is working, and which isn't. git bisect then selects a commit about half way in between the
working
and non-working versions, and checks out that version of the kernel

3. You build that kernel, and run your test case

4. You tell git bisect whether the resulting kernel passes your test case, it selects a new commit, and checks out that version of the kernel.

5. Repeat steps 3 and 4 until git bisect has identified the commit that causes the kernel to stop passing your test case

If the number of patches between 2.6.21 and 2.6.22 is N, then git bisect will find the faulty patch in O(log2(N)) steps. For example, if there are 250 patches between 2.6.21 and 2.6.22, it will take about 8 iterations of steps 3 and 4 to find the faulty patch, if all goes well; far fewer than the total number of patches you would need to test one at a time.

Naturally you can also do this by applying and reverting patches with "patch -p1", but it's a little more work.

Quoted 44 lines
Quoted 24 lines · Chuck Lever
Quoted 17 lines · Trond Myklebust
Quoted 5 lines · Gianluca Alberici

Hello,

I confirm that i have encountered this same problem (EINVAL on open (...O | TRUNC) with the following userspace servers:

Quoted 8 lines

- nfs-user-server shipped with debian sarge/etch etc...
- cfsd (crypto file system which is an nfs server)

I want to underline again that these userspace servers have been woking perfectly until 2.6.21.7 (which is the last 2.6.21)
Since 2.6.22 the problem came out and it is still present into 2.6.24 rc7 (last i tested). Conclusion: there must have been something that is changed in 2.6.22 that caused the problem.

Quoted 3 lines

The only difference between these two dumps are the fact that the first one isn't using the Sun convention for telling NFSv2 servers to set to the current time (see the code in xdr_encode_current_server_time).

Quoted 16 lines

I thought I saw that on both SETATTRs, but I could be wrong.

Quoted 2 lines

I don't see why this would be new behaviour after 2.6.21. The code for this has been in the NFS client since 2.6.15 at least...

A mount option is set on one test client, and not the other, perhaps?

Signature · 6 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
-
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Quoted 4 lines

-
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#28 lore raw

Hello all,

Thanks to Chuck's help i finally decided to proceed to a git bisect and found the bad patch. Is there anybody that has an idea why it breaks userspace nfs servers as we have seen ? Sorry for emailing directly Chuck Lever and Andrew Morton but i really wanted to thank Chuck for his precious help and thought that /akpm/ having signed this commit maybe he's going to figure out whats wrong easily

This is what i finally get from git:

1c710c896eb461895d3c399e15bb5f20b39c9073 is first bad commit commit 1c710c896eb461895d3c399e15bb5f20b39c9073

Author: Ulrich Drepper <[email protected]>
Date:   Tue May 8 00:33:25 2007 -0700

utimensat implementation

Implement utimensat(2) which is an extension to futimesat(2) in that it

a) supports nano-second resolution for the timestamps
b) allows to selectively ignore the atime/mtime value
c) allows to selectively use the current time for either atime or mtime
d) supports changing the atime/mtime of a symlink itself along the lines of the BSD lutimes(3) functions

[...]

[[email protected]: add missing i386 syscall table entry]
Cc Alexey Dobriyan <[email protected]>
Cc Michael Kerrisk <[email protected]>
:040000 040000 3bedbc7fd919ba167b8e5f208a630261570853bb 927002a9423dcb51ba4f7bee53e60cdca6c1df43 M      arch
:040000 040000 fd688c5b534efd3111cbf1e1095d6ff631738325 3d0fbf20fb3da1cb380c92f5b2b39815897376d3 M      fs
:040000 040000 bfb1a907a9a842db4fa3543e12a8381d4e11b1eb 9c1d99324db12e066c0d17870fe48457809ad43b M      include

Thanks in advance, regards,

Gianluca

Quoted 134 lines

Hi Gianluca-

Quoted 15 lines · Gianluca Alberici

Hello again everybody

Here follows the testbench:

- I got two mirrors, same machine, same disk etc...chaged hostname, IP, and on the second i have recompiled kernel.
- First: 2.6.21.7 on debian sarge
- Second: 2.6.22 same system.
- Onto both i got nfs-user-server and cfsd last versions
- The export file is the same (localhost /opt/nfs (rw, async), stripping off the async option does not changes anything)
- Mount options are exactly the same.

The problem arises in the very same manner with both nfs and cfsd:

Quoted 9 lines

NFS:setattr {
...
...
RPC:call_decode {
return 22;
}
...
return 22;
}

Again, there is nothing wrong with the RPC client or call_decode. The *server* is returning NFSERR_INVAL (22) to a SETATTR request; the RPC client is simply passing that along to the NFS client, as it is designed to do.

Quoted 11 lines

I have tried these kernels:

2.6.16.11 works
2.6.20 works
2.6.21 works
2.6.21.7 works
2.6.22 doesnt work (contiguous to previous version) 2.6.23 doesnt work (same behavior as previous) 2.6.23.9 doesnt work (as above)
2.6.24rc7 doesnt work (as above)

Quoted 2 lines

I would really like to do more, client or server side, if you ave any suggestions.
Can we find out what is the change (doesnt matter if it is a buf or bug fix) that caused this problem ?

The goal here is to identify the kernel change between 2.6.21 and 2.6.22 that makes the client generate SETATTR requests the user-space server chokes on. It may be a change in the NFS client, or it could be somewhere else in the file system stack, like the VFS.

The usual procedure is to use "git bisect". It does a binary search on the kernel patches between the working kernel version and the kernel version that is known not to work. It works like this:

1. You clone a linux kernel git repository (if you don't have a git repository already)

2. You tell git bisect which kernel version is working, and which isn't. git bisect then selects a commit about half way in between the working and non-working versions, and checks out that version of the kernel

3. You build that kernel, and run your test case

4. You tell git bisect whether the resulting kernel passes your test case, it selects a new commit, and checks out that version of the kernel.

5. Repeat steps 3 and 4 until git bisect has identified the commit that causes the kernel to stop passing your test case

If the number of patches between 2.6.21 and 2.6.22 is N, then git bisect will find the faulty patch in O(log2(N)) steps. For example, if there are 250 patches between 2.6.21 and 2.6.22, it will take about 8 iterations of steps 3 and 4 to find the faulty patch, if all goes well; far fewer than the total number of patches you would need to test one at a time.

Naturally you can also do this by applying and reverting patches with "patch -p1", but it's a little more work.

Quoted 52 lines
Quoted 28 lines · Chuck Lever
Quoted 18 lines · Trond Myklebust
Quoted 5 lines · Gianluca Alberici

Hello,

I confirm that i have encountered this same problem (EINVAL on open (...O | TRUNC) with the following userspace servers:

Quoted 8 lines

- nfs-user-server shipped with debian sarge/etch etc...
- cfsd (crypto file system which is an nfs server)

I want to underline again that these userspace servers have been woking perfectly until 2.6.21.7 (which is the last 2.6.21)
Since 2.6.22 the problem came out and it is still present into 2.6.24 rc7 (last i tested). Conclusion: there must have been something that is changed in 2.6.22 that caused the problem.

Quoted 3 lines

The only difference between these two dumps are the fact that the first one isn't using the Sun convention for telling NFSv2 servers to set to the current time (see the code in xdr_encode_current_server_time).

I thought I saw that on both SETATTRs, but I could be wrong.

Quoted 15 lines
Quoted 2 lines

I don't see why this would be new behaviour after 2.6.21. The code for this has been in the NFS client since 2.6.15 at least...

A mount option is set on one test client, and not the other, perhaps?

Signature · 6 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
-
To unsubscribe from this list: send the line "unsubscribe linux- nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

-
To unsubscribe from this list: send the line "unsubscribe linux- nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 9 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com



-
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
#29 lore raw

Hi Gianluca-

Quoted 3 lines · Gianluca Alberici

Hello all,

Thanks to Chuck's help i finally decided to proceed to a git bisect and found the bad patch. Is there anybody that has an idea why it breaks userspace nfs servers as we have seen ? Sorry for emailing directly Chuck Lever and Andrew Morton but i really wanted to thank Chuck for his precious help and thought that /akpm/ having signed this commit maybe he's going to figure out whats wrong easily

The commit you found is a plausible source of the trouble (based on our current theory about the problem).

What isn't quite clear to me is whether this commit causes your user-space server to start failing suddenly, or it causes the client to start sending the special non-standard time stamps in the SETATTR request. My guess is the latter, but I want to confirm this guess against reality :-)

Are you running the client and server concurrently on the same system? If so, it would be helpful if you could run this test with a constant kernel version on one side while varying it on the other. If client and server are already on different systems, can you tell us which system and which kernel combinations caused the failure?

A matrix of combinations might be:

1. server kernel is before 1c710c89, client kernel is before 1c710c89
2. server kernel is before 1c710c89, client kernel is after 1c710c89
3. server kernel is after 1c710c89, client kernel is before 1c710c89
4. server kernel is after 1c710c89, client kernel is after 1c710c89

Thanks.

Quoted 30 lines

This is what i finally get from git:

1c710c896eb461895d3c399e15bb5f20b39c9073 is first bad commit commit 1c710c896eb461895d3c399e15bb5f20b39c9073

Author: Ulrich Drepper <[email protected]>
Date:   Tue May 8 00:33:25 2007 -0700

utimensat implementation

Implement utimensat(2) which is an extension to futimesat(2) in
that it

a) supports nano-second resolution for the timestamps
b) allows to selectively ignore the atime/mtime value
c) allows to selectively use the current time for either atime
or mtime
d) supports changing the atime/mtime of a symlink itself along
the lines
of the BSD lutimes(3) functions

[...]

[[email protected]: add missing i386 syscall table entry]
Cc Alexey Dobriyan <[email protected]>
Cc Michael Kerrisk <[email protected]>
Quoted 144 lines
:040000 040000 3bedbc7fd919ba167b8e5f208a630261570853bb 927002a9423dcb51ba4f7bee53e60cdca6c1df43 M      arch
:040000 040000 fd688c5b534efd3111cbf1e1095d6ff631738325 3d0fbf20fb3da1cb380c92f5b2b39815897376d3 M      fs
:040000 040000 bfb1a907a9a842db4fa3543e12a8381d4e11b1eb 9c1d99324db12e066c0d17870fe48457809ad43b M      include

Thanks in advance, regards,

Gianluca

Quoted 134 lines

Hi Gianluca-

Quoted 15 lines · Gianluca Alberici

Hello again everybody

Here follows the testbench:

- I got two mirrors, same machine, same disk etc...chaged hostname, IP, and on the second i have recompiled kernel.
- First: 2.6.21.7 on debian sarge
- Second: 2.6.22 same system.
- Onto both i got nfs-user-server and cfsd last versions
- The export file is the same (localhost /opt/nfs (rw, async), stripping off the async option does not changes anything)
- Mount options are exactly the same.

The problem arises in the very same manner with both nfs and cfsd:

Quoted 9 lines

NFS:setattr {
...
...
RPC:call_decode {
return 22;
}
...
return 22;
}

Again, there is nothing wrong with the RPC client or call_decode. The *server* is returning NFSERR_INVAL (22) to a SETATTR request; the RPC client is simply passing that along to the NFS client, as it is designed to do.

Quoted 11 lines

I have tried these kernels:

2.6.16.11 works
2.6.20 works
2.6.21 works
2.6.21.7 works
2.6.22 doesnt work (contiguous to previous version) 2.6.23 doesnt work (same behavior as previous) 2.6.23.9 doesnt work (as above)
2.6.24rc7 doesnt work (as above)

Quoted 2 lines

I would really like to do more, client or server side, if you ave any suggestions.
Can we find out what is the change (doesnt matter if it is a buf or bug fix) that caused this problem ?

The goal here is to identify the kernel change between 2.6.21 and 2.6.22 that makes the client generate SETATTR requests the user-space server chokes on. It may be a change in the NFS client, or it could be somewhere else in the file system stack, like the VFS.

The usual procedure is to use "git bisect". It does a binary search on the kernel patches between the working kernel version and the kernel version that is known not to work. It works like this:

1. You clone a linux kernel git repository (if you don't have a git repository already)

2. You tell git bisect which kernel version is working, and which isn't. git bisect then selects a commit about half way in between the working and non-working versions, and checks out that version of the kernel

3. You build that kernel, and run your test case

4. You tell git bisect whether the resulting kernel passes your test case, it selects a new commit, and checks out that version of the kernel.

5. Repeat steps 3 and 4 until git bisect has identified the commit that causes the kernel to stop passing your test case

If the number of patches between 2.6.21 and 2.6.22 is N, then git bisect will find the faulty patch in O(log2(N)) steps. For example, if there are 250 patches between 2.6.21 and 2.6.22, it will take about 8 iterations of steps 3 and 4 to find the faulty patch, if all goes well; far fewer than the total number of patches you would need to test one at a time.

Naturally you can also do this by applying and reverting patches with "patch -p1", but it's a little more work.

Quoted 52 lines
Quoted 28 lines · Chuck Lever
Quoted 18 lines · Trond Myklebust
Quoted 5 lines · Gianluca Alberici

Hello,

I confirm that i have encountered this same problem (EINVAL on open (...O | TRUNC) with the following userspace servers:

Quoted 8 lines

- nfs-user-server shipped with debian sarge/etch etc...
- cfsd (crypto file system which is an nfs server)

I want to underline again that these userspace servers have been woking perfectly until 2.6.21.7 (which is the last 2.6.21)
Since 2.6.22 the problem came out and it is still present into 2.6.24 rc7 (last i tested). Conclusion: there must have been something that is changed in 2.6.22 that caused the problem.

Quoted 3 lines

The only difference between these two dumps are the fact that the first one isn't using the Sun convention for telling NFSv2 servers to set to the current time (see the code in xdr_encode_current_server_time).

I thought I saw that on both SETATTRs, but I could be wrong.

Quoted 15 lines
Quoted 2 lines

I don't see why this would be new behaviour after 2.6.21. The code for this has been in the NFS client since 2.6.15 at least...

A mount option is set on one test client, and not the other, perhaps?

Signature · 6 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
-
To unsubscribe from this list: send the line "unsubscribe linux- nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

-
To unsubscribe from this list: send the line "unsubscribe linux- nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 9 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com



-
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

--
Chuck Lever
chuck[dot]lever[at]oracle[dot]com

#30 lore raw

Hi Chuck,

I finally got it. Problem and solution have been found from 6 month but nobody cared...up to now those servers have not been mantained, this problem is not discussed anywhere else than the following link. The bug (userspace server side i would say at this point) is well described from the author of an nfs-user-server patch which has not been managed yet. The magic hint to find it on google was 'nfs server utimensat' :-)

http://marc.info/?l=linux-nfs&m=118724649406144&w=2

I have already prepared a working patch for cfsd based upon the one ive listed. The nfs patch is of course waiting for commit since august, 2007. Ill submit it to debian cfsd mantainers, hoping to have more chance than my predecessor. It doesnt seem to me that there was any kernel related issue.

Thanks a lot again, sorry for the lots of noise i have done. I will try to be more appropriate next time.

Cheers,

Gianluca

Quoted 214 lines · Chuck Lever

Hi Gianluca-

Quoted 3 lines · Gianluca Alberici

Hello all,

Thanks to Chuck's help i finally decided to proceed to a git bisect and found the bad patch. Is there anybody that has an idea why it breaks userspace nfs servers as we have seen ? Sorry for emailing directly Chuck Lever and Andrew Morton but i really wanted to thank Chuck for his precious help and thought that /akpm/ having signed this commit maybe he's going to figure out whats wrong easily

The commit you found is a plausible source of the trouble (based on our current theory about the problem).

What isn't quite clear to me is whether this commit causes your user- space server to start failing suddenly, or it causes the client to start sending the special non-standard time stamps in the SETATTR request. My guess is the latter, but I want to confirm this guess against reality :-)

Are you running the client and server concurrently on the same system? If so, it would be helpful if you could run this test with a constant kernel version on one side while varying it on the other. If client and server are already on different systems, can you tell us which system and which kernel combinations caused the failure?

A matrix of combinations might be:

1. server kernel is before 1c710c89, client kernel is before 1c710c89
2. server kernel is before 1c710c89, client kernel is after 1c710c89
3. server kernel is after 1c710c89, client kernel is before 1c710c89
4. server kernel is after 1c710c89, client kernel is after 1c710c89

Thanks.

Quoted 179 lines

This is what i finally get from git:

1c710c896eb461895d3c399e15bb5f20b39c9073 is first bad commit commit 1c710c896eb461895d3c399e15bb5f20b39c9073
Author: Ulrich Drepper <[email protected]>
Date: Tue May 8 00:33:25 2007 -0700

utimensat implementation

Implement utimensat(2) which is an extension to futimesat(2) in that it

a) supports nano-second resolution for the timestamps
b) allows to selectively ignore the atime/mtime value
c) allows to selectively use the current time for either atime or mtime
d) supports changing the atime/mtime of a symlink itself along the lines of the BSD lutimes(3) functions

[...]

[[email protected]: add missing i386 syscall table entry]
Cc Alexey Dobriyan <[email protected]>
Cc Michael Kerrisk <[email protected]>

:040000 040000 3bedbc7fd919ba167b8e5f208a630261570853bb 927002a9423dcb51ba4f7bee53e60cdca6c1df43 M arch :040000 040000 fd688c5b534efd3111cbf1e1095d6ff631738325 3d0fbf20fb3da1cb380c92f5b2b39815897376d3 M fs :040000 040000 bfb1a907a9a842db4fa3543e12a8381d4e11b1eb 9c1d99324db12e066c0d17870fe48457809ad43b M include

Thanks in advance, regards,

Gianluca

Quoted 141 lines

Hi Gianluca-

Quoted 15 lines · Gianluca Alberici

Hello again everybody

Here follows the testbench:

- I got two mirrors, same machine, same disk etc...chaged hostname, IP, and on the second i have recompiled kernel.
- First: 2.6.21.7 on debian sarge
- Second: 2.6.22 same system.
- Onto both i got nfs-user-server and cfsd last versions
- The export file is the same (localhost /opt/nfs (rw, async), stripping off the async option does not changes anything)
- Mount options are exactly the same.

The problem arises in the very same manner with both nfs and cfsd:

Quoted 9 lines

NFS:setattr {
...
...
RPC:call_decode {
return 22;
}
...
return 22;
}

Again, there is nothing wrong with the RPC client or call_decode. The *server* is returning NFSERR_INVAL (22) to a SETATTR request; the RPC client is simply passing that along to the NFS client, as it is designed to do.

Quoted 11 lines

I have tried these kernels:

2.6.16.11 works
2.6.20 works
2.6.21 works
2.6.21.7 works
2.6.22 doesnt work (contiguous to previous version) 2.6.23 doesnt work (same behavior as previous) 2.6.23.9 doesnt work (as above)
2.6.24rc7 doesnt work (as above)

Quoted 2 lines

I would really like to do more, client or server side, if you ave any suggestions.
Can we find out what is the change (doesnt matter if it is a buf or bug fix) that caused this problem ?

The goal here is to identify the kernel change between 2.6.21 and 2.6.22 that makes the client generate SETATTR requests the user- space server chokes on. It may be a change in the NFS client, or it could be somewhere else in the file system stack, like the VFS.

The usual procedure is to use "git bisect". It does a binary search on the kernel patches between the working kernel version and the kernel version that is known not to work. It works like this:

1. You clone a linux kernel git repository (if you don't have a git repository already)

2. You tell git bisect which kernel version is working, and which isn't. git bisect then selects a commit about half way in between the working and non-working versions, and checks out that version of the kernel

3. You build that kernel, and run your test case

4. You tell git bisect whether the resulting kernel passes your test case, it selects a new commit, and checks out that version of the kernel.

5. Repeat steps 3 and 4 until git bisect has identified the commit that causes the kernel to stop passing your test case

If the number of patches between 2.6.21 and 2.6.22 is N, then git bisect will find the faulty patch in O(log2(N)) steps. For example, if there are 250 patches between 2.6.21 and 2.6.22, it will take about 8 iterations of steps 3 and 4 to find the faulty patch, if all goes well; far fewer than the total number of patches you would need to test one at a time.

Naturally you can also do this by applying and reverting patches with "patch -p1", but it's a little more work.

Quoted 56 lines
Quoted 30 lines · Chuck Lever
Quoted 19 lines · Trond Myklebust
Quoted 5 lines · Gianluca Alberici

Hello,

I confirm that i have encountered this same problem (EINVAL on open (...O | TRUNC) with the following userspace servers:

Quoted 8 lines

- nfs-user-server shipped with debian sarge/etch etc...
- cfsd (crypto file system which is an nfs server)

I want to underline again that these userspace servers have been woking perfectly until 2.6.21.7 (which is the last 2.6.21)
Since 2.6.22 the problem came out and it is still present into 2.6.24 rc7 (last i tested). Conclusion: there must have been something that is changed in 2.6.22 that caused the problem.

Quoted 3 lines

The only difference between these two dumps are the fact that the first one isn't using the Sun convention for telling NFSv2 servers to set to the current time (see the code in xdr_encode_current_server_time).

I thought I saw that on both SETATTRs, but I could be wrong.

Quoted 16 lines
Quoted 2 lines

I don't see why this would be new behaviour after 2.6.21. The code for this has been in the NFS client since 2.6.15 at least...

A mount option is set on one test client, and not the other, perhaps?

Signature · 6 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com
-
To unsubscribe from this list: send the line "unsubscribe linux- nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

-
To unsubscribe from this list: send the line "unsubscribe linux- nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 9 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com



-
To unsubscribe from this list: send the line "unsubscribe linux- nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Signature · 9 lines
Chuck Lever
chuck[dot]lever[at]oracle[dot]com



-
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in
the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
#31 lore raw
Quoted 9 lines · On Wed, 06 Feb 2008 22:55:02 +0100 Gianluca Alberici…

I finally got it. Problem and solution have been found from 6 month but nobody cared...up to now those servers have not been mantained, this problem is not discussed anywhere else than the following link. The bug (userspace server side i would say at this point) is well described from the author of an nfs-user-server patch which has not been managed yet. The magic hint to find it on google was 'nfs server utimensat' :-)

http://marc.info/?l=linux-nfs&m=118724649406144&w=2

This is pretty significant. We have on several occasions in recent years tightened up the argument checking on long-standing system calls and it's always a concern that this will break previously-working applications.

And now it has happened.

If we put buggy code into the kernel then we're largely stuck with it: we need to be back-compatible with our bugs so we don't break things like this.

Quoted 8 lines

I have already prepared a working patch for cfsd based upon the one ive listed. The nfs patch is of course waiting for commit since august, 2007. Ill submit it to debian cfsd mantainers, hoping to have more chance than my predecessor.
It doesnt seem to me that there was any kernel related issue.

Thanks a lot again, sorry for the lots of noise i have done. I will try to be more appropriate next time.

That wasn't noise - it was quite valuable. Thanks for all the work you did on this.

Given that our broken-by-unbreaking code has been out there in several releases there isn't really any point in rebreaking it to fix this - the offending applications need to be repaired so they'll work on 2.6.22 and 2.6.23 anyway.