NooB Assitance with debugging NFSv4 client requested
Subsystems (2)
Hi,
I have an NFSv4 client set up on Ubuntu 10.04.1 LTS x86. The NFSv4 server is running Centos 5.5 and we use MIT kerberos and LDAP for users/groups. This seems to work well with Centos 5.5 clients
All works fine with my Ubuntu client, except after a while my client acts like it loses its authentication - symptom: home directory mount drops to "nobody" - I see the mount as "other" - no write access, can read files that have world read bit set etc.
This can happen anytime between 48 hours and 2 hours after a full client reboot. It seems to be triggered by active use of thunderbird via the NFSv4 mounted home dir which suggests it may be load sensitive.
When it happens, if I unmount my home dir (killing the desktop of course) , then remount the fault is cleared and I can work again.
What doesn't work is just doing a kinit -f or restarting idmapd or gssd.
I have run rpc.gssd in foreground debug mode and that doesn't say much during the problem times, ditto idmapd. We are using openldap for passwd and group lookups cached locally with nscd.
I have tried upping kernel debugging:
rpcdebug -m nfs -s vfs dircache lookupcache pagecache proc xdr file root callback client mount all
but I'm not sure what I'm looking for.
The symptoms feel like the kernel is losing the ticket or timing it out or possibly the ID mapping is failing - is there any way to examining the state of the kernel ticket cache or anything else I could be looking for?
I am tempted to say this is a bug, possibly in the Ubuntu build, but I would like to investigate further.
Any pointers much appreciated as to how I might isolate the fault further.
Cheers
Tim
Signature · 1 line
Tim Watts
A message that is not in the archive (sent off-list or not yet received): 4D024A8E.50407-AQ/[email protected]
Bump - anyone :)
Please note - I'm not requesting anyone fix a deeply obscure problem. I'm a sysadmin - I'd just be really grateful for some pointers to help me continue debugging this myself...
Ta,
Tim
Quoted 44 lines · Tim Watts
Hi,
I have an NFSv4 client set up on Ubuntu 10.04.1 LTS x86. The NFSv4 server is running Centos 5.5 and we use MIT kerberos and LDAP for users/groups. This seems to work well with Centos 5.5 clients
All works fine with my Ubuntu client, except after a while my client acts like it loses its authentication - symptom: home directory mount drops to "nobody" - I see the mount as "other" - no write access, can read files that have world read bit set etc.
This can happen anytime between 48 hours and 2 hours after a full client reboot. It seems to be triggered by active use of thunderbird via the NFSv4 mounted home dir which suggests it may be load sensitive.
When it happens, if I unmount my home dir (killing the desktop of course) , then remount the fault is cleared and I can work again.
What doesn't work is just doing a kinit -f or restarting idmapd or gssd.
I have run rpc.gssd in foreground debug mode and that doesn't say much during the problem times, ditto idmapd. We are using openldap for passwd and group lookups cached locally with nscd.
I have tried upping kernel debugging:
rpcdebug -m nfs -s vfs dircache lookupcache pagecache proc xdr file root callback client mount all
but I'm not sure what I'm looking for.
The symptoms feel like the kernel is losing the ticket or timing it out or possibly the ID mapping is failing - is there any way to examining the state of the kernel ticket cache or anything else I could be looking for?
I am tempted to say this is a bug, possibly in the Ubuntu build, but I would like to investigate further.
Any pointers much appreciated as to how I might isolate the fault further.
Cheers
Tim
Signature · 3 lines
Tim Watts Linux Sysadmin, High Energy Physics, Imperial College London Tel: 020 759 47809
Hmm,
Wonder if this is a clue (not noticed this before, but running idmapd in the foreground, it just whined):
rpc.idmapd: nfscb: read(/var/lib/nfs/rpc_pipefs/nfs/clnt0/idmap): No such file or directory
(Repeated several times).
I'm veering towards this issue being an idmap problem.
Cheers
tim
Quoted 58 lines · Tim Watts
Bump - anyone :)
Please note - I'm not requesting anyone fix a deeply obscure problem. I'm a sysadmin - I'd just be really grateful for some pointers to help me continue debugging this myself...
Ta,
Tim
Quoted 45 lines · Tim Watts
Hi,
I have an NFSv4 client set up on Ubuntu 10.04.1 LTS x86. The NFSv4 server is running Centos 5.5 and we use MIT kerberos and LDAP for users/groups. This seems to work well with Centos 5.5 clients
All works fine with my Ubuntu client, except after a while my client acts like it loses its authentication - symptom: home directory mount drops to "nobody" - I see the mount as "other" - no write access, can read files that have world read bit set etc.
This can happen anytime between 48 hours and 2 hours after a full client reboot. It seems to be triggered by active use of thunderbird via the NFSv4 mounted home dir which suggests it may be load sensitive.
When it happens, if I unmount my home dir (killing the desktop of course) , then remount the fault is cleared and I can work again.
What doesn't work is just doing a kinit -f or restarting idmapd or gssd.
I have run rpc.gssd in foreground debug mode and that doesn't say much during the problem times, ditto idmapd. We are using openldap for passwd and group lookups cached locally with nscd.
I have tried upping kernel debugging:
rpcdebug -m nfs -s vfs dircache lookupcache pagecache proc xdr file root callback client mount all
but I'm not sure what I'm looking for.
The symptoms feel like the kernel is losing the ticket or timing it out or possibly the ID mapping is failing - is there any way to examining the state of the kernel ticket cache or anything else I could be looking for?
I am tempted to say this is a bug, possibly in the Ubuntu build, but I would like to investigate further.
Any pointers much appreciated as to how I might isolate the fault further.
Cheers
Tim
Signature · 3 lines
Tim Watts Linux Sysadmin, High Energy Physics, Imperial College London Tel: 020 759 47809
Quoted 11 lines · Tim Watts
Hmm,
Wonder if this is a clue (not noticed this before, but running idmapd in the foreground, it just whined):
rpc.idmapd: nfscb:
read(/var/lib/nfs/rpc_pipefs/nfs/clnt0/idmap): No such file or directory(Repeated several times).
I'm veering towards this issue being an idmap problem.
All permissions are enforced on the server side--the effect of an idmapper failure should just be that "ls -l" shows bad results (e.g., "nobody" as the owner of every file) and that commands like "chown" fail.
If you're actually being prevented from performing normal filesystem operations, then the most likely culprit is with gssd and krb5.
Might be worth filing a bug with ubuntu.
rpc.gssd debugging you've tried. Sniffing client/server and client/kdc traffic with wireshark might also show some more information.
I dunno. Maybe strace rpc.gssd during the failure?
--b.
Quoted 74 lines
Cheers
tim
Quoted 58 lines · Tim Watts
Bump - anyone :)
Please note - I'm not requesting anyone fix a deeply obscure problem. I'm a sysadmin - I'd just be really grateful for some pointers to help me continue debugging this myself...
Ta,
Tim
Quoted 45 lines · Tim Watts
Hi,
I have an NFSv4 client set up on Ubuntu 10.04.1 LTS x86. The NFSv4 server is running Centos 5.5 and we use MIT kerberos and LDAP for users/groups. This seems to work well with Centos 5.5 clients
All works fine with my Ubuntu client, except after a while my client acts like it loses its authentication - symptom: home directory mount drops to "nobody" - I see the mount as "other" - no write access, can read files that have world read bit set etc.
This can happen anytime between 48 hours and 2 hours after a full client reboot. It seems to be triggered by active use of thunderbird via the NFSv4 mounted home dir which suggests it may be load sensitive.
When it happens, if I unmount my home dir (killing the desktop of course) , then remount the fault is cleared and I can work again.
What doesn't work is just doing a kinit -f or restarting idmapd or gssd.
I have run rpc.gssd in foreground debug mode and that doesn't say much during the problem times, ditto idmapd. We are using openldap for passwd and group lookups cached locally with nscd.
I have tried upping kernel debugging:
rpcdebug -m nfs -s vfs dircache lookupcache pagecache proc xdr file root callback client mount all
but I'm not sure what I'm looking for.
The symptoms feel like the kernel is losing the ticket or timing it out or possibly the ID mapping is failing - is there any way to examining the state of the kernel ticket cache or anything else I could be looking for?
I am tempted to say this is a bug, possibly in the Ubuntu build, but I would like to investigate further.
Any pointers much appreciated as to how I might isolate the fault further.
Cheers
Tim
Signature · 7 lines
Tim Watts Linux Sysadmin, High Energy Physics, Imperial College London Tel: 020 759 47809 -- To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected] More majordomo info at http://vger.kernel.org/majordomo-info.html
Hi Bruce,
Quoted 9 lines · J. Bruce Fields
Quoted 8 lines · Tim Watts
Hmm,
Wonder if this is a clue (not noticed this before, but running idmapd in the foreground, it just whined):
rpc.idmapd: nfscb:
read(/var/lib/nfs/rpc_pipefs/nfs/clnt0/idmap): No such file or directory
Quoted 8 lines
Quoted 3 lines
(Repeated several times).
I'm veering towards this issue being an idmap problem.
All permissions are enforced on the server side--the effect of an idmapper failure should just be that "ls -l" shows bad results (e.g., "nobody" as the owner of every file) and that commands like "chown" fail.
Thanks very much for your reply.
Thanks for confirming the logic - I certainly now believe I have an issue with idmapd - sometimes the user will change to "nobody" and the group is good, and sometimes the reverse will happen.
We are using LDAP - with nscd - I might give myself a static /etc/passwd and /etc/groups entry to see if that makes the problem go away.
I'll run idmapd in debug plus under strace to see what's happening.
Quoted 2 lines
If you're actually being prevented from performing normal filesystem operations, then the most likely culprit is with gssd and krb5.
I think I *may* have discovered the cause, eventually.
I have a couple of bash functions that maintain a user kerb principle and a root one - the root one being in /tmp/krb5cc_<uid>_root - and the bash function switches the env var to suit the mode I want to be in.
Seems that having two files of the form /tmp/krb5cc_<uid>* might be confusing something, so I have made the root one have a totally non conforming name to see if that was the cause.
Quoted 1 line
Might be worth filing a bug with ubuntu.
I did - it got classified under "nfs4_acl_tools"(!). I don't think Ubuntu are big on NFSv4 and I'm happy to try to get to the bottom of this, but it's pretty new to me (I've run a lot of NFSv3 before though).
Quoted 2 lines
rpc.gssd debugging you've tried. Sniffing client/server and client/kdc traffic with wireshark might also show some more information.
Yes - that would be worth doing.
Quoted 1 line
I dunno. Maybe strace rpc.gssd during the failure?
Yes - might be worth doing that too.
Luckily during the failure conditions, the desktop is usable enough so I can at least get around and look at stuff.
Many thanks for the pointers :)
Cheers
Tim
Signature · 3 lines
Tim Watts Linux Sysadmin, High Energy Physics, Imperial College London Tel: 020 759 47809
Quoted 36 lines · Tim Watts
Hi,
I have an NFSv4 client set up on Ubuntu 10.04.1 LTS x86. The NFSv4 server is running Centos 5.5 and we use MIT kerberos and LDAP for users/groups. This seems to work well with Centos 5.5 clients
All works fine with my Ubuntu client, except after a while my client acts like it loses its authentication - symptom: home directory mount drops to "nobody" - I see the mount as "other" - no write access, can read files that have world read bit set etc.
This can happen anytime between 48 hours and 2 hours after a full client reboot. It seems to be triggered by active use of thunderbird via the NFSv4 mounted home dir which suggests it may be load sensitive.
When it happens, if I unmount my home dir (killing the desktop of course) , then remount the fault is cleared and I can work again.
What doesn't work is just doing a kinit -f or restarting idmapd or gssd.
I have run rpc.gssd in foreground debug mode and that doesn't say much during the problem times, ditto idmapd. We are using openldap for passwd and group lookups cached locally with nscd.
I have tried upping kernel debugging:
rpcdebug -m nfs -s vfs dircache lookupcache pagecache proc xdr file root callback client mount all
but I'm not sure what I'm looking for.
The symptoms feel like the kernel is losing the ticket or timing it out or possibly the ID mapping is failing - is there any way to examining the state of the kernel ticket cache or anything else I could be looking for?
Quoted 8 lines
I am tempted to say this is a bug, possibly in the Ubuntu build, but I would like to investigate further.
Any pointers much appreciated as to how I might isolate the fault further.
Cheers
Tim
Right - thanks for all the discussions...
The security (krb tickets) seem stable now I have renamed my root principle cache file to /tmp/kerberos5cc_tjw_root rather than /tmp/krb5cc_myuid_root
idmap seems still unstable - often my file groups go to nobody despite my file owners being correct.
Our LDAP server setup is new (replaced NIS) and we see sometimes that other servers (eg the mailserver) occasionally don't get an LDAP query through in a timely manner once in a blue moon.
I would like to see idmapd try a bit harder and/or not cache bad/missing answers for so long but I think that is roughly where the other problem lies - thus is debuggable and soluable.
Thanks for the help guys!
Cheers
Tim
Signature · 3 lines
Tim Watts Linux Sysadmin, High Energy Physics, Imperial College London Tel: 020 759 47809
A message that is not in the archive (sent off-list or not yet received): 4D0A8CC9.9050306-AQ/[email protected]
Quoted 55 lines · Tim Watts
Quoted 36 lines · Tim Watts
Hi,
I have an NFSv4 client set up on Ubuntu 10.04.1 LTS x86. The NFSv4 server is running Centos 5.5 and we use MIT kerberos and LDAP for users/groups. This seems to work well with Centos 5.5 clients
All works fine with my Ubuntu client, except after a while my client acts like it loses its authentication - symptom: home directory mount drops to "nobody" - I see the mount as "other" - no write access, can read files that have world read bit set etc.
This can happen anytime between 48 hours and 2 hours after a full client reboot. It seems to be triggered by active use of thunderbird via the NFSv4 mounted home dir which suggests it may be load sensitive.
When it happens, if I unmount my home dir (killing the desktop of course) , then remount the fault is cleared and I can work again.
What doesn't work is just doing a kinit -f or restarting idmapd or gssd.
I have run rpc.gssd in foreground debug mode and that doesn't say much during the problem times, ditto idmapd. We are using openldap for passwd and group lookups cached locally with nscd.
I have tried upping kernel debugging:
rpcdebug -m nfs -s vfs dircache lookupcache pagecache proc xdr file root callback client mount all
but I'm not sure what I'm looking for.
The symptoms feel like the kernel is losing the ticket or timing it out or possibly the ID mapping is failing - is there any way to examining the state of the kernel ticket cache or anything else I could be looking for?
Quoted 8 lines
I am tempted to say this is a bug, possibly in the Ubuntu build, but I would like to investigate further.
Any pointers much appreciated as to how I might isolate the fault further.
Cheers
Tim
Right - thanks for all the discussions...
The security (krb tickets) seem stable now I have renamed my root principle cache file to
/tmp/kerberos5cc_tjw_rootrather than/tmp/krb5cc_myuid_rootidmap seems still unstable - often my file groups go to nobody despite my file owners being correct.
Quoted 7 lines
Our LDAP server setup is new (replaced NIS) and we see sometimes that other servers (eg the mailserver) occasionally don't get an LDAP query through in a timely manner once in a blue moon.
I would like to see idmapd try a bit harder and/or not cache bad/missing answers for so long but I think that is roughly where the other problem lies - thus is debuggable and soluable.
Let us know what you figure out.
--b.