Skip to contentSkip to conversation

Trouble with kerberized NFS client after upgrading from nfs-utils 1.2.0 to 1.2.5

DISCUSSION Rohit Kumar Mehta 2 participants 6 messages Aug 1, 2013 – Aug 2, 2013

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hello everyone, I am stuck trying to figure out why I cannot get sec=krb5 Linux clients working after upgrading from Ubuntu 10.04 LTS (Lucid) to 12.04 (Precise)

I suspect the same problem is with the newer nfs-utils, but cannot be sure.

On the old (working) Lucid system, I think the important software is:

# dpkg -l |grep nfs-common
ii  nfs-common 1:1.2.0-4ubuntu4.2                              NFS support files common to client and serve
# uname -a
Linux cselin3 2.6.32-29-generic #58-Ubuntu SMP Fri Feb 11 20:52:10 UTC 2011 x86_64 GNU/Linux

And on the newer (sec=krb5 mounts fail) system, the important software is:

# dpkg -l |grep nfs-common
ii  nfs-common 1:1.2.5-3ubuntu3.1                      NFS support files common to client and server
# uname -a
Linux c27-00 3.2.0-51-generic #77-Ubuntu SMP Wed Jul 24 20:18:19 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

The NFS server we are using is a Hitachi BlueARC, and like I said, older Linux clients work fine. After upgrading to new kernel and

nfs-utils, any attempt to mount yields an error:

# mount hnas.engr.uconn.edu:/EngrUser/users/rohitm /foo -o sec=krb5
mount.nfs: access denied by server while mounting hnas.engr.uconn.edu:/EngrUser/users/rohitm

I've reproduced the same behavior with both -t nfs4 and -t nfs. (Both nfsv3 and nfsv4 work with kerberos security in our configuration with Lucid, but not Precise) I've checked the Kerberos credential cache:

root@c27-00:~# klist -e -f -c /tmp/krb5cc_machine_ENGR.UCONN.EDU
Ticket cache: FILE:/tmp/krb5cc_machine_ENGR.UCONN.EDU
Default principal: nfs/[email protected]

Valid starting Expires Service principal
01/08/2013 15:40 02/08/2013 01:40 krbtgt/[email protected]

    renew until 02/08/2013 15:40, Flags: FRI
    Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1
01/08/2013 15:40  02/08/2013 01:40 nfs/[email protected]
    renew until 02/08/2013 15:40, Flags: FRT
    Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1

I also have rpc.idmapd and rpc.gssd running with extra verbosity. I don't see anything blatantly wrong. This looks slightly suspicious:

Aug  1 16:32:50 c27-00 rpc.gssd[780]: creating tcp client for server hnas.engr.uconn.edu
Aug  1 16:32:50 c27-00 rpc.gssd[780]: DEBUG: port already set to 2049
Aug  1 16:32:50 c27-00 rpc.gssd[780]: creating context with server [email protected]
Aug  1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create krb5 context for user with uid 0 for server hnas.engr.uconn.edu
Aug  1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create machine krb5 context with credentials cache FILE:/tmp/krb5cc_machine_ENGR.UCONN.EDU for server hnas.engr.uconn.edu
Aug  1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create machine krb5 context with any credentials cache for server hnas.engr.uconn.edu
Aug  1 16:32:50 c27-00 rpc.gssd[780]: doing error downcall
Aug  1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si 0x7fffdf0135b0 data 0x7fffdf013480
Aug  1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si 0x7fffdf0135b0 data 0x7fffdf013480
Aug  1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si 0x7fffdf0134f0 data 0x7fffdf0133c0
Aug  1 16:32:50  rpc.gssd[780]: last message repeated 4 times
Aug  1 16:32:50 c27-00 rpc.gssd[780]: destroying client /run/rpc_pipefs/nfs/clnt5
Aug  1 16:32:50 c27-00 rpc.gssd[780]: destroying client /run/rpc_pipefs/nfs/clnt4

I am able to successfuly get the nfs principal for the client from /etc/krb5.keytab "nfs/c27-00.engr.uconn.edu" and I can see the principal for the server "nfs/hnas.engr.uconn.edu" in cache /tmp/krb5cc_machine_ENGR.UCONN.EDU.

I appreciate any advice or assistance. Thanks in advance! Rohit

Signature · 9 lines
Rohit Mehta
Computer Engineer
University of Connecticut
Engineering Computing Services
371 Fairfield Road Unit 4031
Storrs, CT 06269-4031

Office: (860) 486 - 2331
Fax: (860) 486 - 1273
#1 lore raw
Quoted 74 lines · Rohit Kumar Mehta

Hello everyone, I am stuck trying to figure out why I cannot get sec=krb5 Linux clients working after upgrading from Ubuntu 10.04 LTS (Lucid) to 12.04 (Precise)

I suspect the same problem is with the newer nfs-utils, but cannot be sure.

On the old (working) Lucid system, I think the important software is:

# dpkg -l |grep nfs-common
ii  nfs-common 1:1.2.0-4ubuntu4.2                              NFS 
support files common to client and serve
# uname -a
Linux cselin3 2.6.32-29-generic #58-Ubuntu SMP Fri Feb 11 20:52:10 UTC 
2011 x86_64 GNU/Linux

And on the newer (sec=krb5 mounts fail) system, the important software is:

# dpkg -l |grep nfs-common
ii  nfs-common 1:1.2.5-3ubuntu3.1                      NFS support files 
common to client and server
# uname -a
Linux c27-00 3.2.0-51-generic #77-Ubuntu SMP Wed Jul 24 20:18:19 UTC 
2013 x86_64 x86_64 x86_64 GNU/Linux

The NFS server we are using is a Hitachi BlueARC, and like I said, older Linux clients work fine. After upgrading to new kernel and

nfs-utils, any attempt to mount yields an error:

# mount hnas.engr.uconn.edu:/EngrUser/users/rohitm /foo -o sec=krb5
mount.nfs: access denied by server while mounting 
hnas.engr.uconn.edu:/EngrUser/users/rohitm

I've reproduced the same behavior with both -t nfs4 and -t nfs. (Both nfsv3 and nfsv4 work with kerberos security in our configuration with Lucid, but not Precise) I've checked the Kerberos credential cache:

root@c27-00:~# klist -e -f -c /tmp/krb5cc_machine_ENGR.UCONN.EDU
Ticket cache: FILE:/tmp/krb5cc_machine_ENGR.UCONN.EDU
Default principal: nfs/[email protected]

Valid starting Expires Service principal
01/08/2013 15:40 02/08/2013 01:40 krbtgt/[email protected]

     renew until 02/08/2013 15:40, Flags: FRI
     Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1
01/08/2013 15:40  02/08/2013 01:40 nfs/[email protected]
     renew until 02/08/2013 15:40, Flags: FRT
     Etype (skey, tkt): des3-cbc-sha1, des3-cbc-sha1

I also have rpc.idmapd and rpc.gssd running with extra verbosity. I don't see anything blatantly wrong. This looks slightly suspicious:

Aug  1 16:32:50 c27-00 rpc.gssd[780]: creating tcp client for server 
hnas.engr.uconn.edu
Aug  1 16:32:50 c27-00 rpc.gssd[780]: DEBUG: port already set to 2049
Aug  1 16:32:50 c27-00 rpc.gssd[780]: creating context with server 
[email protected]
Aug  1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create krb5 
context for user with uid 0 for server hnas.engr.uconn.edu
Aug  1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create machine 
krb5 context with credentials cache 
FILE:/tmp/krb5cc_machine_ENGR.UCONN.EDU for server hnas.engr.uconn.edu
Aug  1 16:32:50 c27-00 rpc.gssd[780]: WARNING: Failed to create machine 
krb5 context with any credentials cache for server hnas.engr.uconn.edu
Aug  1 16:32:50 c27-00 rpc.gssd[780]: doing error downcall
Aug  1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si 
0x7fffdf0135b0 data 0x7fffdf013480
Aug  1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si 
0x7fffdf0135b0 data 0x7fffdf013480
Aug  1 16:32:50 c27-00 rpc.gssd[780]: dir_notify_handler: sig 37 si 
0x7fffdf0134f0 data 0x7fffdf0133c0
Aug  1 16:32:50  rpc.gssd[780]: last message repeated 4 times
Aug  1 16:32:50 c27-00 rpc.gssd[780]: destroying client 
/run/rpc_pipefs/nfs/clnt5
Aug  1 16:32:50 c27-00 rpc.gssd[780]: destroying client 

/run/rpc_pipefs/nfs/clnt4

Quoted 7 lines

I am able to successfuly get the nfs principal for the client from /etc/krb5.keytab "nfs/c27-00.engr.uconn.edu" and I can see the principal for the server "nfs/hnas.engr.uconn.edu" in cache /tmp/krb5cc_machine_ENGR.UCONN.EDU.

I appreciate any advice or assistance. Thanks in advance! Rohit

Was libtirpc also updated ?
There has beena change recently where we eliminated the use of libgssglue and you will get issues if both nfs-utils and libtirpc are not compiled to use the same gssapi library.

Please post ldd output on rpc.gssd and libtirpc.so to verify if this is the issue.

Simo.

Signature · 1 line
Simo Sorce * Red Hat, Inc * New York
#2 lore raw

Thanks Simo, it doesn't look like libtirpc was available in previous (10.04) release of Ubuntu.

root@c27-00:~# ldd /usr/sbin/
        linux-vdso.so.1 =>  (0x00007fffeedff000)
        libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb06778e000)
        libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 
(0x00007fb0674c0000)
        libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 
(0x00007fb0672bb000)
        libtirpc.so.1 => /lib/x86_64-linux-gnu/libtirpc.so.1 
(0x00007fb067093000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb066cd4000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb066acf000)
        libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 
(0x00007fb0668a7000)
        libkrb5support.so.0 => 
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fb06669f000)
        libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 
(0x00007fb06649a000)
        libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 
(0x00007fb06627e000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x00007fb066061000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb0679b8000)
root@c27-00:~# locate libtirpc.so
/lib/x86_64-linux-gnu/libtirpc.so.1
/lib/x86_64-linux-gnu/libtirpc.so.1.0.10
root@c27-00:~# ldd /lib/x86_64-linux-gnu/libtirpc.so.1
        linux-vdso.so.1 =>  (0x00007fff01dc1000)
        libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb701d54000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x00007fb701b37000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb701777000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb701573000)
        /lib64/ld-linux-x86-64.so.2 (0x00007fb7021a6000)
root@c27-00:~#
Quoted 12 lines · Simo Sorce

Rohit
Was libtirpc also updated ?
There has beena change recently where we eliminated the use of libgssglue and you will get issues if both nfs-utils and libtirpc are not compiled to use the same gssapi library.

Please post ldd output on rpc.gssd and libtirpc.so to verify if this is the issue.

Simo.

Signature · 6 lines
Rohit Mehta
Computer Engineer
University of Connecticut
Engineering Computing Services
371 Fairfield Road Unit 2031
Storrs, CT 06269-2031
#3 lore raw
Quoted 3 lines · Rohit Mehta

Thanks Simo, it doesn't look like libtirpc was available in previous (10.04) release of Ubuntu.

root@c27-00:~# ldd /usr/sbin/

I assume this had ^^ rpc.gssd at the end and is just a copy&paste error.

Quoted 25 lines
         linux-vdso.so.1 =>  (0x00007fffeedff000)
         libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb06778e000)
         libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3 
(0x00007fb0674c0000)
         libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 
(0x00007fb0672bb000)
         libtirpc.so.1 => /lib/x86_64-linux-gnu/libtirpc.so.1 
(0x00007fb067093000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb066cd4000)
         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb066acf000)
         libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3 
(0x00007fb0668a7000)
         libkrb5support.so.0 => 
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fb06669f000)
         libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 
(0x00007fb06649a000)
         libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 
(0x00007fb06627e000)
         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x00007fb066061000)
         /lib64/ld-linux-x86-64.so.2 (0x00007fb0679b8000)
root@c27-00:~# locate libtirpc.so
/lib/x86_64-linux-gnu/libtirpc.so.1
/lib/x86_64-linux-gnu/libtirpc.so.1.0.10
root@c27-00:~# ldd /lib/x86_64-linux-gnu/libtirpc.so.1
Quoted 8 lines
         linux-vdso.so.1 =>  (0x00007fff01dc1000)
         libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb701d54000)
         libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 
(0x00007fb701b37000)
         libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb701777000)
         libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb701573000)
         /lib64/ld-linux-x86-64.so.2 (0x00007fb7021a6000)
root@c27-00:~#

Both are built against libgssglue so at least they are consistent, and that is not the source of the problem as I suspected.

More information on the actual error would help.

Simo.

Quoted 15 lines · Simo Sorce
Quoted 12 lines

Rohit
Was libtirpc also updated ?
There has beena change recently where we eliminated the use of libgssglue and you will get issues if both nfs-utils and libtirpc are not compiled to use the same gssapi library.

Please post ldd output on rpc.gssd and libtirpc.so to verify if this is the issue.

Simo.

Signature · 1 line
Simo Sorce * Red Hat, Inc * New York
#4 lore raw

Thanks Simo, I apologize for the copy and paste error. You are correct about it being "ldd /usr/sbin/rpc.gssd" command.

The actual error message we get is from the mount command:
mount.nfs: access denied by server while mounting hnas.engr.uconn.edu:/EngrUser/users/rohitm

I got a little more output with mount -v

root@c27-00:~# mount -vvv hnas.engr.uconn.edu:/EngrUser/users/rohitm /foo -o sec=krb5
mount: fstab path: "/etc/fstab"
mount: mtab path:  "/etc/mtab"
mount: lock path:  "/etc/mtab~"
mount: temp path:  "/etc/mtab.tmp"
mount: UID:        0
mount: eUID:       0
mount: no type was given - I'll assume nfs because of the colon
mount: spec: "hnas.engr.uconn.edu:/EngrUser/users/rohitm"
mount: node:  "/foo"
mount: types: "nfs"
mount: opts:  "sec=krb5"
mount: external mount: argv[0] = "/sbin/mount.nfs"
mount: external mount: argv[1] = "hnas.engr.uconn.edu:/EngrUser/users/rohitm"
mount: external mount: argv[2] = "/foo"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw,sec=krb5"
mount.nfs: timeout set for Fri Aug  2 08:04:08 2013
mount.nfs: trying text-based options 'sec=krb5,vers=4,addr=137.99.203.4,clientaddr=137.99.2.29'
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting hnas.engr.uconn.edu:/EngrUser/users/rohitm

So there is unfortunately no more info available from mount command and rpc.gssd output did not have any red flags for me. I'm trying to find out if there is a way to get more information from the NFS server itself, but as of now I'm not sure how to do that.

Thanks,

Rohit

Quoted 61 lines · Simo Sorce
Quoted 3 lines · Rohit Mehta

Thanks Simo, it doesn't look like libtirpc was available in previous (10.04) release of Ubuntu.

root@c27-00:~# ldd /usr/sbin/

I assume this had ^^ rpc.gssd at the end and is just a copy&paste error.

Quoted 25 lines
          linux-vdso.so.1 =>  (0x00007fffeedff000)
          libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb06778e000)
          libkrb5.so.3 => /usr/lib/x86_64-linux-gnu/libkrb5.so.3
(0x00007fb0674c0000)
          libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2
(0x00007fb0672bb000)
          libtirpc.so.1 => /lib/x86_64-linux-gnu/libtirpc.so.1
(0x00007fb067093000)
          libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb066cd4000)
          libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb066acf000)
          libk5crypto.so.3 => /usr/lib/x86_64-linux-gnu/libk5crypto.so.3
(0x00007fb0668a7000)
          libkrb5support.so.0 =>
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007fb06669f000)
          libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1
(0x00007fb06649a000)
          libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2
(0x00007fb06627e000)
          libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fb066061000)
          /lib64/ld-linux-x86-64.so.2 (0x00007fb0679b8000)
root@c27-00:~# locate libtirpc.so
/lib/x86_64-linux-gnu/libtirpc.so.1
/lib/x86_64-linux-gnu/libtirpc.so.1.0.10
root@c27-00:~# ldd /lib/x86_64-linux-gnu/libtirpc.so.1
Quoted 8 lines
          linux-vdso.so.1 =>  (0x00007fff01dc1000)
          libgssglue.so.1 => /lib/libgssglue.so.1 (0x00007fb701d54000)
          libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0
(0x00007fb701b37000)
          libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fb701777000)
          libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fb701573000)
          /lib64/ld-linux-x86-64.so.2 (0x00007fb7021a6000)
root@c27-00:~#

Both are built against libgssglue so at least they are consistent, and that is not the source of the problem as I suspected.

More information on the actual error would help.

Simo.

Quoted 13 lines · Simo Sorce
Quoted 11 lines

Rohit
Was libtirpc also updated ?
There has beena change recently where we eliminated the use of libgssglue and you will get issues if both nfs-utils and libtirpc are not compiled to use the same gssapi library.

Please post ldd output on rpc.gssd and libtirpc.so to verify if this is the issue.

Simo.

Signature · 6 lines
Rohit Mehta
Computer Engineer
University of Connecticut
Engineering Computing Services
371 Fairfield Road Unit 2031
Storrs, CT 06269-2031
#5 lore raw
Quoted 35 lines · Rohit Mehta

Thanks Simo, I apologize for the copy and paste error. You are correct about it being "ldd /usr/sbin/rpc.gssd" command.

The actual error message we get is from the mount command:
mount.nfs: access denied by server while mounting hnas.engr.uconn.edu:/EngrUser/users/rohitm

I got a little more output with mount -v

root@c27-00:~# mount -vvv hnas.engr.uconn.edu:/EngrUser/users/rohitm 
/foo -o sec=krb5
mount: fstab path: "/etc/fstab"
mount: mtab path:  "/etc/mtab"
mount: lock path:  "/etc/mtab~"
mount: temp path:  "/etc/mtab.tmp"
mount: UID:        0
mount: eUID:       0
mount: no type was given - I'll assume nfs because of the colon
mount: spec: "hnas.engr.uconn.edu:/EngrUser/users/rohitm"
mount: node:  "/foo"
mount: types: "nfs"
mount: opts:  "sec=krb5"
mount: external mount: argv[0] = "/sbin/mount.nfs"
mount: external mount: argv[1] = 
"hnas.engr.uconn.edu:/EngrUser/users/rohitm"
mount: external mount: argv[2] = "/foo"
mount: external mount: argv[3] = "-v"
mount: external mount: argv[4] = "-o"
mount: external mount: argv[5] = "rw,sec=krb5"
mount.nfs: timeout set for Fri Aug  2 08:04:08 2013
mount.nfs: trying text-based options 
'sec=krb5,vers=4,addr=137.99.203.4,clientaddr=137.99.2.29'
mount.nfs: mount(2): Permission denied
mount.nfs: access denied by server while mounting 
hnas.engr.uconn.edu:/EngrUser/users/rohitm
Quoted 4 lines

So there is unfortunately no more info available from mount command and rpc.gssd output did not have any red flags for me. I'm trying to find out if there is a way to get more information from the NFS server itself, but as of now I'm not sure how to do that.

It looks like you have to look at the server side here.
Bump up rpc.svcgssd (or whatever is used on your server) debug and maybe even the kernel log with the rpcdebug command and see why it is refusing your authentication.

Simo.

Signature · 1 line
Simo Sorce * Red Hat, Inc * New York