Trouble with kerberized NFS client after upgrading from nfs-utils 1.2.0 to 1.2.5
Subsystems (2)
Hello everyone, I am stuck trying to figure out why I cannot get sec=krb5 Linux clients working after upgrading from Ubuntu 10.04 LTS (Lucid) to 12.04 (Precise)
I suspect the same problem is with the newer nfs-utils, but cannot be sure.
On the old (working) Lucid system, I think the important software is:
# dpkg -l |grep nfs-common ii nfs-common 1:1.2.0-4ubuntu4.2 NFS support files common to client and serve # uname -a Linux cselin3 2.6.32-29-generic #58-Ubuntu SMP Fri Feb 11 20:52:10 UTC 2011 x86_64 GNU/Linux
And on the newer (sec=krb5 mounts fail) system, the important software is:
# dpkg -l |grep nfs-common ii nfs-common 1:1.2.5-3ubuntu3.1 NFS support files common to client and server # uname -a Linux c27-00 3.2.0-51-generic #77-Ubuntu SMP Wed Jul 24 20:18:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
The NFS server we are using is a Hitachi BlueARC, and like I said, older Linux clients work fine. After upgrading to new kernel and
nfs-utils, any attempt to mount yields an error:
# mount hnas.engr.uconn.edu:/EngrUser/users/rohitm /foo -o sec=krb5 mount.nfs: access denied by server while mounting hnas.engr.uconn.edu:/EngrUser/users/rohitm
I've reproduced the same behavior with both -t nfs4 and -t nfs. (Both nfsv3 and nfsv4 work with kerberos security in our configuration with Lucid, but not Precise) I've checked the Kerberos credential cache:
root@c27-00:~# klist -e -f -c /tmp/krb5cc_machine_ENGR.UCONN.EDU Ticket cache: FILE:/tmp/krb5cc_machine_ENGR.UCONN.EDU Default principal: nfs/[email protected]
Valid starting Expires Service principal
01/08/2013 15:40 02/08/2013 01:40 krbtgt/[email protected]
renew until 02/08/2013 15:40, Flags: FRI
Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1
01/08/2013 15:40 02/08/2013 01:40 nfs/[email protected]
renew until 02/08/2013 15:40, Flags: FRT
Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1I also have rpc.idmapd and rpc.gssd running with extra verbosity. I don't see anything blatantly wrong. This looks slightly suspicious:
Aug 1 16:32:50 c27-00 rpc.gssd[780]: creating tcp client for server hnas.engr.uconn.edu Aug 1 16:32:50 c27-00 rpc.gssd[780]: DEBUG: port already set to 2049 Aug 1 16:32:50 c27-00 rpc.gssd[780]: creating context with server [email protected] Aug 1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create krb5 context for user with uid 0 for server hnas.engr.uconn.edu Aug 1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_ENGR.UCONN.EDU for server hnas.engr.uconn.edu Aug 1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create machine krb5 context with any credentials cache for server hnas.engr.uconn.edu Aug 1 16:32:50 c27-00 rpc.gssd[780]: doing error downcall Aug 1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si 0x7fffdf0135b0 data 0x7fffdf013480 Aug 1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si 0x7fffdf0135b0 data 0x7fffdf013480 Aug 1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si 0x7fffdf0134f0 data 0x7fffdf0133c0 Aug 1 16:32:50 rpc.gssd[780]: last message repeated 4 times Aug 1 16:32:50 c27-00 rpc.gssd[780]: destroying client /run/rpc_pipefs/nfs/clnt5 Aug 1 16:32:50 c27-00 rpc.gssd[780]: destroying client /run/rpc_pipefs/nfs/clnt4
I am able to successfuly get the nfs principal for the client from /etc/krb5.keytab "nfs/c27-00.engr.uconn.edu" and I can see the principal for the server "nfs/hnas.engr.uconn.edu" in cache /tmp/krb5cc_machine_ENGR.UCONN.EDU.
I appreciate any advice or assistance. Thanks in advance! Rohit
Signature · 9 lines
Rohit Mehta Computer Engineer University of Connecticut Engineering Computing Services 371 Fairfield Road Unit 4031 Storrs, CT 06269-4031 Office: (860) 486 - 2331 Fax: (860) 486 - 1273
Quoted 74 lines · Rohit Kumar Mehta
Hello everyone, I am stuck trying to figure out why I cannot get sec=krb5 Linux clients working after upgrading from Ubuntu 10.04 LTS (Lucid) to 12.04 (Precise)
I suspect the same problem is with the newer nfs-utils, but cannot be sure.
On the old (working) Lucid system, I think the important software is:
# dpkg -l |grep nfs-common ii nfs-common 1:1.2.0-4ubuntu4.2 NFS support files common to client and serve # uname -a Linux cselin3 2.6.32-29-generic #58-Ubuntu SMP Fri Feb 11 20:52:10 UTC 2011 x86_64 GNU/LinuxAnd on the newer (sec=krb5 mounts fail) system, the important software is:
# dpkg -l |grep nfs-common ii nfs-common 1:1.2.5-3ubuntu3.1 NFS support files common to client and server # uname -a Linux c27-00 3.2.0-51-generic #77-Ubuntu SMP Wed Jul 24 20:18:19 UTC 2013 x86_64 x86_64 x86_64 GNU/LinuxThe NFS server we are using is a Hitachi BlueARC, and like I said, older Linux clients work fine. After upgrading to new kernel and
nfs-utils, any attempt to mount yields an error:
# mount hnas.engr.uconn.edu:/EngrUser/users/rohitm /foo -o sec=krb5 mount.nfs: access denied by server while mounting hnas.engr.uconn.edu:/EngrUser/users/rohitmI've reproduced the same behavior with both -t nfs4 and -t nfs. (Both nfsv3 and nfsv4 work with kerberos security in our configuration with Lucid, but not Precise) I've checked the Kerberos credential cache:
root@c27-00:~# klist -e -f -c /tmp/krb5cc_machine_ENGR.UCONN.EDU Ticket cache: FILE:/tmp/krb5cc_machine_ENGR.UCONN.EDU Default principal: nfs/[email protected]Valid starting Expires Service principal
01/08/2013 15:40 02/08/2013 01:40 krbtgt/[email protected]renew until 02/08/2013 15:40, Flags: FRI Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1 01/08/2013 15:40 02/08/2013 01:40 nfs/[email protected] renew until 02/08/2013 15:40, Flags: FRT Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1I also have rpc.idmapd and rpc.gssd running with extra verbosity. I don't see anything blatantly wrong. This looks slightly suspicious:
Aug 1 16:32:50 c27-00 rpc.gssd[780]: creating tcp client for server hnas.engr.uconn.edu Aug 1 16:32:50 c27-00 rpc.gssd[780]: DEBUG: port already set to 2049 Aug 1 16:32:50 c27-00 rpc.gssd[780]: creating context with server [email protected] Aug 1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create krb5 context for user with uid 0 for server hnas.engr.uconn.edu Aug 1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_ENGR.UCONN.EDU for server hnas.engr.uconn.edu Aug 1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create machine krb5 context with any credentials cache for server hnas.engr.uconn.edu Aug 1 16:32:50 c27-00 rpc.gssd[780]: doing error downcall Aug 1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si 0x7fffdf0135b0 data 0x7fffdf013480 Aug 1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si 0x7fffdf0135b0 data 0x7fffdf013480 Aug 1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si 0x7fffdf0134f0 data 0x7fffdf0133c0 Aug 1 16:32:50 rpc.gssd[780]: last message repeated 4 times Aug 1 16:32:50 c27-00 rpc.gssd[780]: destroying client /run/rpc_pipefs/nfs/clnt5 Aug 1 16:32:50 c27-00 rpc.gssd[780]: destroying client
/run/rpc_pipefs/nfs/clnt4
Quoted 7 lines
I am able to successfuly get the nfs principal for the client from
/etc/krb5.keytab"nfs/c27-00.engr.uconn.edu" and I can see the principal for the server "nfs/hnas.engr.uconn.edu" in cache/tmp/krb5cc_machine_ENGR.UCONN.EDU.I appreciate any advice or assistance. Thanks in advance! Rohit
Was libtirpc also updated ?
There has beena change recently where we eliminated the use of libgssglue and you will get issues if both nfs-utils and libtirpc are not compiled to use the same gssapi library.
Please post ldd output on rpc.gssd and libtirpc.so to verify if this is the issue.
Simo.
Signature · 1 line
Simo Sorce * Red Hat, Inc * New York
Thanks Simo, it doesn't look like libtirpc was available in previous (10.04) release of Ubuntu.
root@c27-00:~# ldd /usr/sbin/
linux-vdso.so.1 => (0x00007fffeedff000)
libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb06778e000)
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3
(0x00007fb0674c0000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2
(0x00007fb0672bb000)
libtirpc.so.1 => /lib/x86_64-linux-gnu/libtirpc.so.1
(0x00007fb067093000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb066cd4000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb066acf000)
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
(0x00007fb0668a7000)
libkrb5support.so.0 =>
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fb06669f000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
(0x00007fb06649a000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
(0x00007fb06627e000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fb066061000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb0679b8000)
root@c27-00:~# locate libtirpc.so
/lib/x86_64-linux-gnu/libtirpc.so.1
/lib/x86_64-linux-gnu/libtirpc.so.1.0.10
root@c27-00:~# ldd /lib/x86_64-linux-gnu/libtirpc.so.1
linux-vdso.so.1 => (0x00007fff01dc1000)
libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb701d54000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fb701b37000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb701777000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb701573000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb7021a6000)
root@c27-00:~#Quoted 12 lines · Simo Sorce
Rohit
Was libtirpc also updated ?
There has beena change recently where we eliminated the use of libgssglue and you will get issues if both nfs-utils and libtirpc are not compiled to use the same gssapi library.Please post ldd output on rpc.gssd and libtirpc.so to verify if this is the issue.
Simo.
Signature · 6 lines
Rohit Mehta Computer Engineer University of Connecticut Engineering Computing Services 371 Fairfield Road Unit 2031 Storrs, CT 06269-2031
Quoted 3 lines · Rohit Mehta
Thanks Simo, it doesn't look like libtirpc was available in previous (10.04) release of Ubuntu.
root@c27-00:~# ldd /usr/sbin/
I assume this had ^^ rpc.gssd at the end and is just a copy&paste error.
Quoted 25 lines
linux-vdso.so.1 => (0x00007fffeedff000)
libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb06778e000)
libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3
(0x00007fb0674c0000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2
(0x00007fb0672bb000)
libtirpc.so.1 => /lib/x86_64-linux-gnu/libtirpc.so.1
(0x00007fb067093000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb066cd4000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb066acf000)
libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
(0x00007fb0668a7000)
libkrb5support.so.0 =>
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fb06669f000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
(0x00007fb06649a000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
(0x00007fb06627e000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fb066061000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb0679b8000)
root@c27-00:~# locate libtirpc.so
/lib/x86_64-linux-gnu/libtirpc.so.1
/lib/x86_64-linux-gnu/libtirpc.so.1.0.10
root@c27-00:~# ldd /lib/x86_64-linux-gnu/libtirpc.so.1Quoted 8 lines
linux-vdso.so.1 => (0x00007fff01dc1000)
libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb701d54000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fb701b37000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb701777000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb701573000)
/lib64/ld-linux-x86-64.so.2 (0x00007fb7021a6000)
root@c27-00:~#Both are built against libgssglue so at least they are consistent, and that is not the source of the problem as I suspected.
More information on the actual error would help.
Simo.
Quoted 15 lines · Simo Sorce
Quoted 12 lines
Rohit
Was libtirpc also updated ?
There has beena change recently where we eliminated the use of libgssglue and you will get issues if both nfs-utils and libtirpc are not compiled to use the same gssapi library.Please post ldd output on rpc.gssd and libtirpc.so to verify if this is the issue.
Simo.
Signature · 1 line
Simo Sorce * Red Hat, Inc * New York
Thanks Simo, I apologize for the copy and paste error. You are correct about it being "ldd /usr/sbin/rpc.gssd" command.
The actual error message we get is from the mount command:
mount.nfs: access denied by server while mounting hnas.engr.uconn.edu:/EngrUser/users/rohitm
I got a little more output with mount -v
root@c27-00:~# mount -vvv hnas.engr.uconn.edu:/EngrUser/users/rohitm /foo -o sec=krb5 mount: fstab path: "/etc/fstab" mount: mtab path: "/etc/mtab" mount: lock path: "/etc/mtab~" mount: temp path: "/etc/mtab.tmp" mount: UID: 0 mount: eUID: 0 mount: no type was given - I'll assume nfs because of the colon mount: spec: "hnas.engr.uconn.edu:/EngrUser/users/rohitm" mount: node: "/foo" mount: types: "nfs" mount: opts: "sec=krb5" mount: external mount: argv[0] = "/sbin/mount.nfs" mount: external mount: argv[1] = "hnas.engr.uconn.edu:/EngrUser/users/rohitm" mount: external mount: argv[2] = "/foo" mount: external mount: argv[3] = "-v" mount: external mount: argv[4] = "-o" mount: external mount: argv[5] = "rw,sec=krb5" mount.nfs: timeout set for Fri Aug 2 08:04:08 2013 mount.nfs: trying text-based options 'sec=krb5,vers=4,addr=137.99.203.4,clientaddr=137.99.2.29' mount.nfs: mount(2): Permission denied mount.nfs: access denied by server while mounting hnas.engr.uconn.edu:/EngrUser/users/rohitm
So there is unfortunately no more info available from mount command and rpc.gssd output did not have any red flags for me. I'm trying to find out if there is a way to get more information from the NFS server itself, but as of now I'm not sure how to do that.
Thanks,
Rohit
Quoted 61 lines · Simo Sorce
Quoted 3 lines · Rohit Mehta
Thanks Simo, it doesn't look like libtirpc was available in previous (10.04) release of Ubuntu.
root@c27-00:~# ldd /usr/sbin/I assume this had ^^ rpc.gssd at the end and is just a copy&paste error.
Quoted 25 lines
linux-vdso.so.1 => (0x00007fffeedff000) libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb06778e000) libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007fb0674c0000) libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007fb0672bb000) libtirpc.so.1 => /lib/x86_64-linux-gnu/libtirpc.so.1 (0x00007fb067093000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb066cd4000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb066acf000) libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007fb0668a7000) libkrb5support.so.0 => /usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fb06669f000) libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007fb06649a000) libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007fb06627e000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb066061000) /lib64/ld-linux-x86-64.so.2 (0x00007fb0679b8000) root@c27-00:~# locate libtirpc.so /lib/x86_64-linux-gnu/libtirpc.so.1 /lib/x86_64-linux-gnu/libtirpc.so.1.0.10 root@c27-00:~# ldd /lib/x86_64-linux-gnu/libtirpc.so.1Quoted 8 lines
linux-vdso.so.1 => (0x00007fff01dc1000) libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb701d54000) libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fb701b37000) libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb701777000) libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb701573000) /lib64/ld-linux-x86-64.so.2 (0x00007fb7021a6000) root@c27-00:~#Both are built against libgssglue so at least they are consistent, and that is not the source of the problem as I suspected.
More information on the actual error would help.
Simo.
Quoted 13 lines · Simo Sorce
Quoted 11 lines
Rohit
Was libtirpc also updated ?
There has beena change recently where we eliminated the use of libgssglue and you will get issues if both nfs-utils and libtirpc are not compiled to use the same gssapi library.Please post ldd output on rpc.gssd and libtirpc.so to verify if this is the issue.
Simo.
Signature · 6 lines
Rohit Mehta Computer Engineer University of Connecticut Engineering Computing Services 371 Fairfield Road Unit 2031 Storrs, CT 06269-2031
Quoted 35 lines · Rohit Mehta
Thanks Simo, I apologize for the copy and paste error. You are correct about it being "ldd
/usr/sbin/rpc.gssd" command.The actual error message we get is from the mount command:
mount.nfs: access denied by server while mounting hnas.engr.uconn.edu:/EngrUser/users/rohitmI got a little more output with mount -v
root@c27-00:~# mount -vvv hnas.engr.uconn.edu:/EngrUser/users/rohitm /foo -o sec=krb5 mount: fstab path: "/etc/fstab" mount: mtab path: "/etc/mtab" mount: lock path: "/etc/mtab~" mount: temp path: "/etc/mtab.tmp" mount: UID: 0 mount: eUID: 0 mount: no type was given - I'll assume nfs because of the colon mount: spec: "hnas.engr.uconn.edu:/EngrUser/users/rohitm" mount: node: "/foo" mount: types: "nfs" mount: opts: "sec=krb5" mount: external mount: argv[0] = "/sbin/mount.nfs" mount: external mount: argv[1] = "hnas.engr.uconn.edu:/EngrUser/users/rohitm" mount: external mount: argv[2] = "/foo" mount: external mount: argv[3] = "-v" mount: external mount: argv[4] = "-o" mount: external mount: argv[5] = "rw,sec=krb5" mount.nfs: timeout set for Fri Aug 2 08:04:08 2013 mount.nfs: trying text-based options 'sec=krb5,vers=4,addr=137.99.203.4,clientaddr=137.99.2.29' mount.nfs: mount(2): Permission denied mount.nfs: access denied by server while mounting hnas.engr.uconn.edu:/EngrUser/users/rohitm
Quoted 4 lines
So there is unfortunately no more info available from mount command and rpc.gssd output did not have any red flags for me. I'm trying to find out if there is a way to get more information from the NFS server itself, but as of now I'm not sure how to do that.
It looks like you have to look at the server side here.
Bump up rpc.svcgssd (or whatever is used on your server) debug and maybe even the kernel log with the rpcdebug command and see why it is refusing your authentication.
Simo.
Signature · 1 line
Simo Sorce * Red Hat, Inc * New York