NFS4: ssh + unlink(~/.Xauthority) delays
Subsystems (2)
Hello,
I'm about to setup a Linux based fileserver for home directories and a couple of Linux clients with kerberos and NFS4 (no NFS3 for security reasons).
The whole stuff is currently based on debian stable (with a backported current NFS userland 1.2.5) and a recent vanilla kernel (3.5.2).
So far I have the NFS4 Server up and running as well as a couple of clients with NFS4+autofs mounted home directories.
All this stuff mostly works now, but unfortunately I ran into some strange bahaviour now.
When I ssh from one machine to another the system hangs (a delay of up to 60 seconds) while running xauth.
Replacing xauth by a wrapper script I have been able to trace this behaviour to a hang of an unlink("/home/<user>/.Xauthority") system call.
So the question is what cases hangs in NFS4 based Linux systems in general an in this particular case?
Any hint on how to debug this?
Sven
Signature · 4 lines
"C Is Quirky, Flawed, And An Enormous Success." (Dennis M. Ritchie) /me is giggls@ircnet, http://sven.gegg.us/ on the Web
NFS4ERR_DELAY (was: NFS4: ssh + unlink(~/.Xauthority) delays)
Quoted 2 lines · Sven Geggus
So the question is what cases hangs in NFS4 based Linux systems in general an in this particular case?
Digging this down a little bit further using wireshark I figured out that the server responds to the REMOVE .Xauthority call with an NFS4ERR_DELAY for whatever reason.
This is a currently a test-machine with one single user and next to zero load.
So I will probably need a hint on how to debug the server behaviour, as a NFS4ERR_DELAY response is definitely not expected in this case.
Sven
Signature · 4 lines
"Das Einzige wovor wir Angst haben müssen ist die Angst selbst" (Franklin D. Roosevelt) /me is giggls@ircnet, http://sven.gegg.us/ on the Web
Re: NFS4ERR_DELAY (was: NFS4: ssh + unlink(~/.Xauthority) delays)
Quoted 9 lines · On Tue, 21 Aug 2012 09:19:03 +0000 (UTC) Sven Geggus…
Quoted 2 lines · Sven Geggus
So the question is what cases hangs in NFS4 based Linux systems in general an in this particular case?
Digging this down a little bit further using wireshark I figured out that the server responds to the REMOVE .Xauthority call with an
NFS4ERR_DELAYfor whatever reason.
Quoted 8 lines
This is a currently a test-machine with one single user and next to zero load.
So I will probably need a hint on how to debug the server behaviour, as a
NFS4ERR_DELAYresponse is definitely not expected in this case.Sven
It's often the case that this indicates a problem communicating over the callback channel. For instance, the server is trying to recall a delegation but the client isn't responding, so the server has to wait until the recall attempt times out before proceeding.
Signature · 1 line
Jeff Layton <[email protected]>
Re: NFS4ERR_DELAY
Quoted 4 lines · Jeff Layton
It's often the case that this indicates a problem communicating over the callback channel. For instance, the server is trying to recall a delegation but the client isn't responding, so the server has to wait until the recall attempt times out before proceeding.
Hm I'm not shure if I understand this correctly.
I am talking about exactly 3 machines (and one single user for now) here:
clientA, clientB and the NFS-server.
"user" is logged in on clientA any now opens a shell to ssh to clientB.
The NFS-server is providing the home for the user on both machines.
The ssh is now getting delayed for up to 1 minute because the NFS server does not allow for the .Xauthority file to be deleted immediately.
It is probably worth to mention, that I'm currently experimenting with btrfs on the server. Is there a chance that this bug will disapper when I change the underlaying filesystem of the server to ext4?
Sven
Signature · 4 lines
Trotz der zunehmenden Verbreitung von Linux erfreut sich der Bär, und - dank Knut - insbesondere der Eisbär, deutlich größerer Beliebtheit als der Pinguin. (Gefunden bei http://telepolis.de/) /me is giggls@ircnet, http://sven.gegg.us/ on the Web
Re: NFS4ERR_DELAY
Quoted 7 lines · On Tue, 21 Aug 2012 12:52:16 +0000 (UTC) Sven Geggus…
Quoted 4 lines · Jeff Layton
It's often the case that this indicates a problem communicating over the callback channel. For instance, the server is trying to recall a delegation but the client isn't responding, so the server has to wait until the recall attempt times out before proceeding.
Quoted 7 lines
Hm I'm not shure if I understand this correctly.
I am talking about exactly 3 machines (and one single user for now) here:
clientA, clientB and the NFS-server."user" is logged in on clientA any now opens a shell to ssh to clientB.
Right, so you probably opened ~/Xauthority on clientA and got a delegation. Then you ssh'ed to clientB and opened the file there. At that point, the server has to recall the delegation. Usually that's pretty quick, but if the server can't talk to clientA on the callback port then it has to wait and eventually time out before it can allow the open on clientB to proceed.
Quoted 11 lines
The NFS-server is providing the home for the user on both machines.
The ssh is now getting delayed for up to 1 minute because the NFS server does not allow for the .Xauthority file to be deleted immediately.
It is probably worth to mention, that I'm currently experimenting with btrfs on the server. Is there a chance that this bug will disapper when I change the underlaying filesystem of the server to ext4?
Sven
You asked for hints on how to debug it, and I gave one. The server will often return NFS4ERR_DELAY when it's waiting for a delegation recall to complete. I'd make sure that that's all working as expected.
Signature · 1 line
Jeff Layton <[email protected]>
Re: NFS4ERR_DELAY
Quoted 11 lines · Jeff Layton
Quoted 9 lines · On Tue, 21 Aug 2012 12:52:16 +0000 (UTC) Sven Geggus…
The NFS-server is providing the home for the user on both machines.
The ssh is now getting delayed for up to 1 minute because the NFS server does not allow for the .Xauthority file to be deleted immediately.
It is probably worth to mention, that I'm currently experimenting with btrfs on the server. Is there a chance that this bug will disapper when I change the underlaying filesystem of the server to ext4?
Quoted 6 lines
Quoted 2 lines
Sven
You asked for hints on how to debug it, and I gave one. The server will often return
NFS4ERR_DELAYwhen it's waiting for a delegation recall to complete. I'd make sure that that's all working as expected.
Yep. There's a recent regression which could cause this; could you try:
https://lkml.org/lkml/2012/8/16/531
?
--b.
Re: NFS4ERR_DELAY
Quoted 3 lines · J. Bruce Fields
Yep. There's a recent regression which could cause this; could you try:
https://lkml.org/lkml/2012/8/16/531
Patch applied, problem solved.
Thanks
Sven
Signature · 4 lines
Um Kontrolle Ihres Kontos wiederzugewinnen, klicken Sie bitte auf das Verbindungsgebrüll. (aus einer Ebay fishing Mail) /me is giggls@ircnet, http://sven.gegg.us/ on the Web
Sven Geggus wrote:
Hello,
I'm about to setup a Linux based fileserver for home directories and a couple of Linux clients with kerberos and NFS4 (no NFS3 for security reasons).
The whole stuff is currently based on debian stable (with a backported current NFS userland 1.2.5) and a recent vanilla kernel (3.5.2).
So far I have the NFS4 Server up and running as well as a couple of clients with NFS4+autofs mounted home directories.
All this stuff mostly works now, but unfortunately I ran into some strange bahaviour now.
When I ssh from one machine to another the system hangs (a delay of up to 60 seconds) while running xauth.
Replacing xauth by a wrapper script I have been able to trace this behaviour to a hang of an unlink("/home/<user>/.Xauthority") system call.
So the question is what cases hangs in NFS4 based Linux systems in general an in this particular case?
No, the question is why did X put that file in your home directory. That's not where it belongs. I've got this in my .xinitrc:
setenv XAUTHORITY /tmp/Xauthorityid -u
Quoted 4 lines · Jim Rees
No, the question is why did X put that file in your home directory. That's not where it belongs. I've got this in my .xinitrc:
setenv XAUTHORITY
/tmp/Xauthorityid -u
I agree, that ~/.Xauthority is probably not the best place for this file, but it is the default (at least in Debian).
If it is expected behaviour that this will not work correctly with shared Homedirectories, I wonder why this did not cause me any trouble with NFS3 for many years.
Anyway I did not yet figure out how to globally change the XAUTHORITY Variable in kdm and ssh :(
Sven
Signature · 4 lines
Microsoft ist offenbar die einzige Firma, die in der Lage ist, ein mit
Office nicht kompatibles Bürosoftwarepaket einzuführen.
(Florian Weimer in de.alt.sysadmin.recovery)
/me is giggls@ircnet, http://sven.gegg.us/ on the WebSven Geggus wrote:
Quoted 4 lines · Jim Rees
No, the question is why did X put that file in your home directory. That's not where it belongs. I've got this in my .xinitrc:
setenv XAUTHORITY
/tmp/Xauthorityid -u
I agree, that ~/.Xauthority is probably not the best place for this file, but it is the default (at least in Debian).
If it is expected behaviour that this will not work correctly with shared Homedirectories, I wonder why this did not cause me any trouble with NFS3 for many years.
Are you sure it didn't cause you any trouble? You were giving root access to anyone who could snoop the Xauthority file off the wire. That's the main reason you don't want it in your home dir. As to why that's still the default, laziness I guess.
Anyway I did not yet figure out how to globally change the XAUTHORITY Variable in kdm and ssh :(
I don't use kdm. For ssh:
% cat ~/.ssh/environment
XAUTHORITY=/tmp/Xauthority1234
You might also have to fix /etc/ssh/sshd_config.
Quoted 7 lines
Sven Geggus wrote:
Quoted 3 lines · Jim Rees
No, the question is why did X put that file in your home directory. That's not where it belongs. I've got this in my .xinitrc:
Quoted 8 lines
Quoted 1 line
setenv XAUTHORITY
/tmp/Xauthorityid -uI agree, that ~/.Xauthority is probably not the best place for this file, but it is the default (at least in Debian).
If it is expected behaviour that this will not work correctly with shared Homedirectories, I wonder why this did not cause me any trouble with NFS3 for many years.
NFS3 doesn't have delegations and this file placement should not be an issue in NFSv4 either if it is working correctly!
Regards, Malahal.
Malahal Naineni wrote:
NFS3 doesn't have delegations and this file placement should not be an issue in NFSv4 either if it is working correctly!
Xauthority doesn't belong in $HOME, it belongs on the local disk. That has nothing to do with NFS v3/v4. It has to do with limiting distribution and exposure of sensitive information.
Quoted 8 lines · Jim Rees
Malahal Naineni wrote:
NFS3 doesn't have delegations and this file placement should not be an issue in NFSv4 either if it is working correctly!
Xauthority doesn't belong in $HOME, it belongs on the local disk. That has nothing to do with NFS v3/v4. It has to do with limiting distribution and exposure of sensitive information.
Yah. Unlinking a file shouldn't take a minute, though, so we do have a bug, however it was found. Let me know if the patch I referred to fixes it.
--b.