Move (mv) hangs with NFSv4
Subsystems (2)
I've been trying out NFSv4 instead of NFSv3 on the systems here and have had mixed success. First of there was an uid/gid mapping bug which was solved in 2.6.9.
Now I've run into another serious bug; I cannot move files to the NFS filesystem. It stalls when the file is finished. I've tried with big and small files (even 0 bytes) and the effect is the same.
I started guessing that it sets some attributes during the final stage
so i tried cp -a, which has the same effect. So this seems to support that theory. chmod and chown work fine. I don't know if there is some other command I can try that only messes with attributes.
There is no error message on the server. On the client I get "RPC: rpciod waiting on sync task!" now and then. Most of the time it doesn't say a thing though so I'm not sure it's related.
The server is a FC2 machine running 2.6.9-1.6_FC2 (also tried 2.6.8.1 + uid/gid patch) and the clients are a FC2 machine with the same kernel and a FC3 machine with a clean 2.6.9.
Rgds
Pierre
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
fr den 03.12.2004 Klokka 18:51 (+0100) skreiv Pierre Ossman:
Quoted 3 lines
There is no error message on the server. On the client I get "RPC:
rpciod waiting on sync task!" now and then. Most of the time it doesn't say a thing though so I'm not sure it's related.
The "rpciod waiting on sync task" is a known problem that should normally only occur if you ^C out. I'm still working on a complete fix for that.
As for the other hang: I'm not reproducing it against non-linux servers. Bruce, any ideas?
Cheers,
Trond
Cheers,
Trond
Signature · 12 lines
Trond Myklebust <[email protected]> ------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/ _______________________________________________ NFS maillist - [email protected] https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 6 lines · Trond Myklebust
fr den 03.12.2004 Klokka 18:51 (+0100) skreiv Pierre Ossman:
Quoted 3 lines
There is no error message on the server. On the client I get "RPC:
rpciod waiting on sync task!" now and then. Most of the time it doesn't say a thing though so I'm not sure it's related.
Quoted 6 lines
The "rpciod waiting on sync task" is a known problem that should normally only occur if you ^C out. I'm still working on a complete fix for that.
As for the other hang: I'm not reproducing it against non-linux servers. Bruce, any ideas?
Hm. With our latest kernel and user utilities I actually get a segfault on cp -a from the getxattr to get the acl on the source file. Probably a bug our modified libacl, ugh. In any case, probably also a different bug from what he's seeing. Maybe I could see a pcap file or an strace of the cp -a showing where it hangs?
--b.
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 9 lines · J. Bruce Fields
Hm. With our latest kernel and user utilities I actually get a segfault on cp -a from the getxattr to get the acl on the source file. Probably a bug our modified libacl, ugh. In any case, probably also a different bug from what he's seeing. Maybe I could see a pcap file or an strace of the cp -a showing where it hangs?
Here's a tcpdump of the transfer (I figure that's what you mean with a pcap file). The last couple of packets keep repeating themselves so the line never goes quiet (until I ^C). The transfer is a mv of a empty file named 'foo'.
Rgds
Pierre
Quoted 4 lines · Pierre Ossman
Here's a tcpdump of the transfer (I figure that's what you mean with a pcap file). The last couple of packets keep repeating themselves so the line never goes quiet (until I ^C).
The transfer is a mv of a empty file named 'foo'.
Thanks. It's a problem with the idmapping on the server, argh. Is rpc.idmapd still running after this, or does it die?--b.
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 12 lines · J. Bruce Fields
Quoted 4 lines · Pierre Ossman
Here's a tcpdump of the transfer (I figure that's what you mean with a pcap file). The last couple of packets keep repeating themselves so the line never goes quiet (until I ^C). The transfer is a mv of a empty file named 'foo'.
Thanks. It's a problem with the idmapping on the server, argh. Is rpc.idmapd still running after this, or does it die?--b.
It's still running. The effect is only local to the mv process. Other traffic to the same mount continue to work perfectly. There has been a few incidents where the entire mount hangs. But in these cases I have to restart the client, not the server.
Can I get some debug output from idmapd that will pinpoint where the problem is?
Rgds
Pierre
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users.
Discover which products truly live up to the hype. Start reading now. http://productguide.itmanagersjournal.com/
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 11 lines · Pierre Ossman
Quoted 5 lines · J. Bruce Fields
Thanks. It's a problem with the idmapping on the server, argh. Is rpc.idmapd still running after this, or does it die?--b.
It's still running. The effect is only local to the mv process. Other traffic to the same mount continue to work perfectly. There has been a few incidents where the entire mount hangs. But in these cases I have to restart the client, not the server.
Can I get some debug output from idmapd that will pinpoint where the problem is?
Any updates to this issue?
Rgds
Pierre
-------------------------------------------------------
The SF.Net email is sponsored by: Beat the post-holiday blues Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek. It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 10 lines · Pierre Ossman
Quoted 10 lines
Quoted 4 lines · J. Bruce Fields
Thanks. It's a problem with the idmapping on the server, argh. Is rpc.idmapd still running after this, or does it die?--b.
It's still running. The effect is only local to the mv process. Other traffic to the same mount continue to work perfectly. There has been a few incidents where the entire mount hangs. But in these cases I have to restart the client, not the server.
Quoted 5 lines
Quoted 3 lines
Can I get some debug output from idmapd that will pinpoint where the problem is?
Any updates to this issue?
Sorry for the overly long delay on the response. Cleaning out my inbox prompted me to look back at this issue. I did find one bug in the ACL code which could cause a "mv" with the target on an NFSv4 partition to fail. Other than that I can't reproduce any problems with move.
It's possible we've fixed something since the beginning of the year, or it's possible something else is going on. If you get the chance to retest with our latest patches, let me know.
--b.
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 31 lines · J. Bruce Fields
Quoted 14 lines · Pierre Ossman
Quoted 14 lines
Quoted 6 lines · J. Bruce Fields
Thanks. It's a problem with the idmapping on the server, argh. Is rpc.idmapd still running after this, or does it die?--b.
It's still running. The effect is only local to the mv process. Other traffic to the same mount continue to work perfectly. There has been a few incidents where the entire mount hangs. But in these cases I have to restart the client, not the server.
Quoted 8 lines
Quoted 5 lines
Can I get some debug output from idmapd that will pinpoint where the problem is?
Any updates to this issue?
Sorry for the overly long delay on the response. Cleaning out my inbox prompted me to look back at this issue. I did find one bug in the ACL code which could cause a "mv" with the target on an NFSv4 partition to fail. Other than that I can't reproduce any problems with move.
Quoted 7 lines
It's possible we've fixed something since the beginning of the year, or it's possible something else is going on. If you get the chance to retest with our latest patches, let me know.
--b.
Ehm... where would these patches be located? I didn't see anything obvious on the web page for any recent kernel. Or perhaps you meant the lats nfs-utils? or both? =)
Rgds
Pierre
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 3 lines · Pierre Ossman
Ehm... where would these patches be located? I didn't see anything obvious on the web page for any recent kernel. Or perhaps you meant the lats nfs-utils? or both? =)
Sorry, I meant the latest citi kernel patch:
--b.
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 9 lines · J. Bruce Fields
Quoted 5 lines · Pierre Ossman
Ehm... where would these patches be located? I didn't see anything obvious on the web page for any recent kernel. Or perhaps you meant the lats nfs-utils? or both? =)
Quoted 7 lines
Sorry, I meant the latest citi kernel patch:
--b.
I've tried the patch and the problem remains.
[root@apollo home]# ~/mv ~/freedesktop.org.xml .
Updating timestamps...done
Updating ownership...
(this is my hacked mv which lists each operation as it is performed)
Rgds
Pierre
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 7 lines · Pierre Ossman
I've tried the patch and the problem remains.
[root@apollo home]# ~/mv
~/freedesktop.org.xml.
Updating timestamps...done
Updating ownership...(this is my hacked mv which lists each operation as it is performed)
Hm. Maybe my "mv" doesn't work quite the same as yours, but I'm not seeing this. Could I see a raw packet trace? (tcpdump s0 -wtmp, send me tmp.pcap). Are you having any problems with rpc.idmapd dying on the server?
--b.
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs
Quoted 9 lines · J. Bruce Fields
Quoted 6 lines · Pierre Ossman
I've tried the patch and the problem remains.
[root@apollo home]# ~/mv
~/freedesktop.org.xml.
Updating timestamps...done
Updating ownership...
Quoted 10 lines
Quoted 3 lines
(this is my hacked mv which lists each operation as it is performed)
Hm. Maybe my "mv" doesn't work quite the same as yours, but I'm not seeing this. Could I see a raw packet trace? (tcpdump s0 -wtmp, send me tmp.pcap). Are you having any problems with rpc.idmapd dying on the server?
No change in behaviour from the last attempt. idmapd keeps running just fine.
Dump from new run included.
Previous test:
http://marc.theaimsgroup.com/?l=linux-nfs&m=110245477428548&w=4
http://marc.theaimsgroup.com/?l=linux-nfs&m=110250639332016&w=4
As I was playing around with idmapd (after running the test) it started to work just fine. I am unable to reproduce the error now, even though idmapd is started the same way as originally. A reboot might get the error back but I can't test that on this machine. I'll see if I can find another machine to torture.
Rgds
Pierre
Quoted 3 lines · Pierre Ossman
No change in behaviour from the last attempt. idmapd keeps running just fine.
Dump from new run included.
OK, thanks for the data. It still looks like a server-side idmapping problem. Next time it gets into this state, it would be interesting to know whether sending rpc.idmapd a SIGHUP (or just restarting it) makes any difference. I'll try to find the time to investigate further later this week....--b.
-------------------------------------------------------
SF email is sponsored by - The IT Product Guide
Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now.
http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click
_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs