mount hangs in NFS4+Kerberos setup

Hi,

I want to setup a file server with NFS4+Kerberos and Debian squeeze for clients running Ubuntu 11.10.

What is already working:
1) Mount NFS4 on client without krb5 option works. Users are able to access files and uids/gids are correct. 2) KDC works. Access from client, get tickets, user authentication/change password through pam is ok.

Now I want to mount with sec=krb5 but this time the command hangs and does not return to shell. See also logs below.

Any hints to fix the issue or to get more helpful debug information are welcome.

regards
knut

=== server status ===

Debian Linux squeeze

# uname -a
Linux tm 2.6.32-5-686 #1 SMP Mon Jan 16 16:04:25 UTC 2012 i686 GNU/Linux

# cat /etc/exports 
/srv        *(rw,sync,no_subtree_check,fsid=0,crossmnt)
/srv/opt    *(sec=sys:krb5,rw,sync,no_subtree_check)

# ps -ef|grep -e rpc -e portmap -e nfs
daemon    1019     1  0 10:26 ?        00:00:00 /sbin/portmap
root      1043     2  0 10:26 ?        00:00:00 [rpciod/0]
root      1044     2  0 10:26 ?        00:00:00 [rpciod/1]
root      1055     2  0 10:26 ?        00:00:00 [nfsiod]
root      1079     1  0 10:26 ?        00:00:00 /usr/sbin/rpc.idmapd -vvv
root      1083     1  0 10:26 ?        00:00:00 /usr/sbin/rpc.gssd -vvv
root      1309     2  0 10:26 ?        00:00:00 [nfsd4]
root      1310     2  0 10:26 ?        00:00:00 [nfsd]
...
root      1336     1  0 10:26 ?        00:00:00 /usr/sbin/rpc.svcgssd -vvv
root      1338     1  0 10:26 ?        00:00:00 /usr/sbin/rpc.mountd --manage-gids

# rpcdebug -m nfsd -s all

=== client status ===

Ubuntu Linux 10.11

# uname -a
Linux orest 3.0.0-15-generic #26-Ubuntu SMP Fri Jan 20 15:59:53 UTC 2012 i686 i686 i386 GNU/Linux

# ps -ef |grep rpc
root       702     1  0 13:16 ?        00:00:00 rpcbind -w
root       712     2  0 13:16 ?        00:00:00 [rpciod]
root       805     1  0 13:16 ?        00:00:00 rpc.gssd -vvv
root       890     1  0 13:16 ?        00:00:00 rpc.idmapd -vvv

=== client log ===

# mount -t nfs4 <server-fqdn>:/opt /opt
# umount /opt

# mount -t nfs4 -osec=krb5 <server-fqdn>:/opt /opt
Feb 10 14:45:17 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7584c data 0xbfe758cc
Feb 10 14:45:17 orest rpc.idmapd[728]: New client: 0
Feb 10 14:45:17 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7102c data 0xbfe710ac
Feb 10 14:45:17 orest rpc.idmapd[728]: Opened /var/lib/nfs/rpc_pipefs/nfs/clnt0/idmap
Feb 10 14:45:17 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7102c data 0xbfe710ac
Feb 10 14:45:17 orest rpc.idmapd[728]: New client: 1
Feb 10 14:45:17 orest rpc.gssd[883]: handling gssd upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
Feb 10 14:45:17 orest rpc.gssd[883]: handle_gssd_upcall: 'mech=krb5 uid=0 enctypes=18,17,16,23,3,1,2 '
Feb 10 14:45:17 orest rpc.gssd[883]: handling krb5 upcall (/var/lib/nfs/rpc_pipefs/nfs/clnt0)
Feb 10 14:45:17 orest rpc.gssd[883]: process_krb5_upcall: service is '<null>'
Feb 10 14:45:17 orest rpc.gssd[883]: Full hostname for '<server-fqdn>' is '<server-fqdn>'
Feb 10 14:45:17 orest rpc.gssd[883]: Full hostname for '<client-fqdn>' is '<client-fqdn>'
Feb 10 14:45:17 orest rpc.gssd[883]: No key table entry found for <CLIENT-HOSTNAME>$@<MYREALM> while getting keytab entry for '<CLIENT-HOSTNAME>$@<MYREALM>'
Feb 10 14:45:17 orest rpc.gssd[883]: Success getting keytab entry for 'root/<client-fqdn>@<MYREALM>'
Feb 10 14:45:17 orest rpc.gssd[883]: Successfully obtained machine credentials for principal 'root/<client-fqdn>@<MYREALM>' stored in ccache 'FILE:/tmp/krb5cc_machine_<MYREALM>'
Feb 10 14:45:17 orest rpc.gssd[883]: INFO: Credentials in CC 'FILE:/tmp/krb5cc_machine_<MYREALM>' are good until 1328967917
Feb 10 14:45:17 orest rpc.gssd[883]: using FILE:/tmp/krb5cc_machine_<MYREALM> as credentials cache for machine creds
Feb 10 14:45:17 orest rpc.gssd[883]: using environment variable to select krb5 ccache FILE:/tmp/krb5cc_machine_<MYREALM>
Feb 10 14:45:17 orest rpc.gssd[883]: creating context using fsuid 0 (save_uid 0)
Feb 10 14:45:17 orest rpc.gssd[883]: creating tcp client for server <server-fqdn>
Feb 10 14:45:17 orest rpc.gssd[883]: DEBUG: port already set to 2049
Feb 10 14:45:17 orest rpc.gssd[883]: creating context with server nfs@<server-fqdn>
Feb 10 14:45:17 orest rpc.gssd[883]: DEBUG: serialize_krb5_ctx: lucid version!
Feb 10 14:45:17 orest rpc.gssd[883]: prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length 8
Feb 10 14:45:17 orest rpc.gssd[883]: doing downcall

(waiting...)

Feb 10 14:48:17 orest kernel: [  249.312058] nfs: server <server-fqdn> not responding, still trying

(press Ctrl-C, >10 min later)

Feb 10 14:59:40 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7584c data 0xbfe758cc
Feb 10 14:59:40 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7584c data 0xbfe758cc
Feb 10 14:59:40 orest rpc.gssd[883]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt1
Feb 10 14:59:40 orest rpc.idmapd[728]: Stale client: 1
Feb 10 14:59:40 orest rpc.idmapd[728]: #011-> closed /var/lib/nfs/rpc_pipefs/nfs/clnt1/idmap
Feb 10 14:59:40 orest rpc.gssd[883]: dir_notify_handler: sig 37 si 0xbfe7584c data 0xbfe758cc
Feb 10 14:59:40  rpc.gssd[883]: last message repeated 4 times
Feb 10 14:59:40 orest rpc.idmapd[728]: Stale client: 0
Feb 10 14:59:40 orest rpc.idmapd[728]: #011-> closed /var/lib/nfs/rpc_pipefs/nfs/clnt0/idmap
Feb 10 14:59:40 orest rpc.gssd[883]: destroying client /var/lib/nfs/rpc_pipefs/nfs/clnt0

=== server log ===

The following messages appeared while initiating mount command on client:

==> auth.log <==

Feb 10 14:45:17 tm krb5kdc[1429]: AS_REQ (7 etypes {18 17 16 23 1 3 2}) <client-IP>: NEEDED_PREAUTH: root/<client-fqdn>@<MYREALM> for krbtgt/<MYREALM>@<MYREALM>, Additional pre-authentication required
Feb 10 14:45:17 tm krb5kdc[1429]: AS_REQ (7 etypes {18 17 16 23 1 3 2}) <client-IP>: ISSUE: authtime 1328881517, etypes {rep=1 tkt=18 ses=18}, root/<client-fqdn>@<MYREALM> for krbtgt/<MYREALM>@<MYREALM>
Feb 10 14:45:17 tm krb5kdc[1429]: TGS_REQ (7 etypes {18 17 16 23 1 3 2}) <client-IP>: ISSUE: authtime 1328881517, etypes {rep=18 tkt=1 ses=1}, root/<client-fqdn>@<MYREALM> for nfs/<server-fqdn>@<MYREALM>

==> syslog <==

Feb 10 14:45:17 tm rpc.svcgssd[1335]: leaving poll
Feb 10 14:45:17 tm rpc.svcgssd[1335]: handling null request
Feb 10 14:45:17 tm rpc.svcgssd[1335]: sname = root/<client-fqdn>@<MYREALM>
Feb 10 14:45:17 tm rpc.svcgssd[1335]: libnfsidmap: using (default) domain: cs.uni-potsdam.de
Feb 10 14:45:17 tm rpc.svcgssd[1335]: libnfsidmap: loaded plugin /usr/lib/libnfsidmap/nsswitch.so for method nsswitch
Feb 10 14:45:17 tm rpc.svcgssd[1335]: nss_gss_princ_to_ids: Local-Realm '<MYREALM>': NOT FOUND
Feb 10 14:45:17 tm rpc.svcgssd[1335]: DEBUG: serialize_krb5_ctx: lucid version!
Feb 10 14:45:17 tm rpc.svcgssd[1335]: prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length 8
Feb 10 14:45:17 tm rpc.svcgssd[1335]: doing downcall
Feb 10 14:45:17 tm rpc.svcgssd[1335]: mech: krb5, hndl len: 4, ctx len 85, timeout: 1328967917 (86400 from now), clnt: root@<client-fqdn>, uid: -1, gid: -1, num aux grps: 0:
Feb 10 14:45:17 tm rpc.svcgssd[1335]: sending null reply
Feb 10 14:45:17 tm rpc.svcgssd[1335]: writing message: \x \x..<lots of digits> 1328881577 0 0 \x01000000 \x607006092a864886f71201020202006f61305fa003020105a10302010fa2533051a003020101a24a0448d85b3308847004f703039fd5f315f81fcbb287c6a53821eb2ea8660169c687bbb0f1d7fbdecb9016284f3adf11b122949b98488862d956d352c6c14e6fd2beb565ba1de7afc99d1c 
Feb 10 14:45:17 tm rpc.svcgssd[1335]: finished handling null request
Feb 10 14:45:17 tm rpc.svcgssd[1335]: entering poll

(waiting...)

I would at least recomend using a recent kernel and backports of nfs-common, rpcbind and libtirpc (compiling them is very straight forward).

Sven

"Ich fürchte mich nicht vor der Rückkehr der Faschisten in der Maske der
Faschisten, sondern vor der Rückkehr der Faschisten in der Maske der
Demokraten" (Theodor W. Adorno)
/me is giggls@ircnet, http://sven.gegg.us/ on the Web

Hi

It appears that the RPCSEC_GSS Kerberos calls were successful, but that the Kerberos principal to id mapping failed.

What kernel is the server running?
What nfs-utils version is the server using?
What libnfsidmap version is the server using?

Thanks

-->Andy

Yes. libnfsidmapd.023 nss_gss_princ_to_id checks the Kerberos REALM passed in the sname against the configured REALMS in /etc/idmapd.conf as explained:

[General]
#Verbosity = 0
# The following should be set to the local NFSv4 domain name # The default is the host's DNS domain name. #Domain = local.domain.edu

# The following is a comma-separated list of Kerberos realm

# names that should be considered to be equivalent to the
# local realm, such that <user>@REALM.A can be assumed to
# be the same user as <user>@REALM.B
# If not specified, the default local realm is the domain name,
# which defaults to the host's DNS domain name,
# translated to upper-case.
# Note that if this value is specified, the local realm name
# must be included in the list!
#Local-Realms =

I believe you have not set the Local-Realms, so libnfsidmapd.023 uses the default of the upper-case of the local domain-name. Thus the nss_gss_print_ids error message:

Try setting the Local-Realms = in /etc/idmapd.conf.

-->Andy

I'm a bit confused because <MYREALM> in the error message is the correct realm.

I altered the idmapd.conf (server and client). Just be be sure we speak about the same things:

<MYREALM>: MYSERVERHOSTNAME.SUB1.DOMAIN.TLD

server:         myserverhostname.sub1.domain.tld
client:    myclienthostname.sub2.sub1.domain.tld

/etc/idmapd.conf
Domain = myserverhostname.sub1.domain.tld
Local-Realm = MYSERVERHOSTNAME.SUB1.DOMAIN.TLD

I wondered if a "local domain name" should include the hostname or not, thus I tried sub1.domain.tld and also myserverhostname.sub1.domain.tld The later worked and let the error message disappear from the log. The rest is the same and mount still hangs.

regards
knut

server log:

Feb 13 10:23:29 tm rpc.svcgssd[18043]: leaving poll
Feb 13 10:23:29 tm rpc.svcgssd[18043]: handling null request
Feb 13 10:23:29 tm rpc.svcgssd[18043]: sname = root/<client-fqdn>@<MYREALM>
Feb 13 10:23:29 tm rpc.svcgssd[18043]: DEBUG: serialize_krb5_ctx: lucid version!
Feb 13 10:23:29 tm rpc.svcgssd[18043]: prepare_krb5_rfc1964_buffer: serializing keys with enctype 4 and length 8
Feb 13 10:23:29 tm rpc.svcgssd[18043]: doing downcall
Feb 13 10:23:29 tm rpc.svcgssd[18043]: mech: krb5, hndl len: 4, ctx len 85, timeout: 1329211409 (86400 from now), clnt: root@<client-fqdn>, uid: -1, gid: -1, num aux grps: 0:
Feb 13 10:23:29 tm rpc.svcgssd[18043]: sending null reply
Feb 13 10:23:29 tm rpc.svcgssd[18043]: writing message: \x... 1329125069 0 0 \x02000000 \x...
Feb 13 10:23:29 tm rpc.svcgssd[18043]: finished handling null request
Feb 13 10:23:29 tm rpc.svcgssd[18043]: entering poll

Ubuntu 11.10
uname -r
3.0.0-15-generic

Some older kernels do not support strong keys. Try adding:
allow_weak_crypto = true
to the
[libdefaults]
in /etc/krb5.conf

Here it is using the machine principal with arcfour:

Kerberos: AS-REQ nfs/[email protected] from ipv4:192.168.1.3:49650 for krbtgt/[email protected]
Kerberos: UNKNOWN -- nfs/[email protected]: no such entry found in hdb
Kerberos: AS-REQ [email protected] from ipv4:192.168.1.3:43041 for krbtgt/[email protected]
Kerberos: Client sent patypes: 149
Kerberos: Looking for PKINIT pa-data -- [email protected]
Kerberos: Looking for ENC-TS pa-data -- [email protected]
Kerberos: No preauth found, returning PREAUTH-REQUIRED -- [email protected]
Kerberos: AS-REQ [email protected] from ipv4:192.168.1.3:32850 for krbtgt/[email protected]
Kerberos: Client sent patypes: encrypted-timestamp, 149
Kerberos: Looking for PKINIT pa-data -- [email protected]
Kerberos: Looking for ENC-TS pa-data -- [email protected]
Kerberos: ENC-TS Pre-authentication succeeded -- [email protected] using arcfour-hmac-md5
Kerberos: AS-REQ authtime: 2012-02-10T18:00:16 starttime: unset endtime: 2012-02-11T04:00:16 renew till: 2012-02-11T18:00:15
Kerberos: Client supported enctypes: aes256-cts-hmac-sha1-96, aes128-cts-hmac-sha1-96, des3-cbc-sha1, arcfour-hmac-md5, using arcfour-hmac-md5/arcfour-hmac-md5
Kerberos: Requested flags: renewable-ok
Kerberos: TGS-REQ [email protected] from ipv4:192.168.1.3:41288 for nfs/[email protected] [canonicalize, renewable]
Kerberos: TGS-REQ authtime: 2012-02-10T18:00:16 starttime: 2012-02-10T18:00:16 endtime: 2012-02-11T04:00:16 renew till: 2012-02-11T18:00:15

Also it's not recommended to use the pseudo-root fsid=0 method for nfs exports under Linux:
http://wiki.linux-nfs.org/wiki/index.php/Nfsv4_configuration

HTH,
Steve

yes. I painfully (mount only says access denied) found out this already and I use allow_weak_crypto to limit to DES. More encryption types have been introduced with kernel 2.6.39...

I tried to use kernel 3.2 from squeeze-backports but this introduced new errors, thus I decided to try with 2.6 first.

hmm, as far as I have understood I have to:
- export the root folder /exports explicitly beside the "real" exports p.ex. /exports/opt
- use fsid=0 for the root folder to force version 4 of NFS

What's your suggestion to improve/secure my configuration?

regards
knut

Officially, you should not export from a pseudo root. Please see the last few lines in the link I sent.
man rpc.gssd(8) adds:
<quote>
Previous versions of
rpc.gssd used only "nfs/*" keys found within the keytab. To be more consistent with other implementations, we now look for specific keytab entries. The search order for keytabs to be used for "machine credentials" is now:
<HOSTNAME>$@<REALM>
root/<hostname>@<REALM>

nfs/<hostname>@<REALM>

host/<hostname>@<REALM>

root/<anyname>@<REALM>

nfs/<anyname>@<REALM>

host/<anyname>@<REALM>
</quote>

I see your setup uses the root principal. If you still get access denied, create another keytab with just the machine$ and host/fqdn keys. I can remember having to fiddle with nfs-utils and keytabs on openSUSE at some stage last year.

If none of this works you can either stick with the old kernel and accept he security, get an up to date nfs-utils and see if hat fixes it with the DES keys or grab an up to date distro where all this stuff will work out of the box. Cheers,
Steve

That link offers the same assertion but no explanation for it. Can you explain why that's the recommendation?

If an admin were to remove fsid=0 from their exports line, they'd also need to adjust the paths in all the connecting clients, right?

	--dkg

Yep.

The main problem with the fsid=0 trick is that your v3 and v4 clients end up with different paths.

--b.

Sorry, no. I can however explain why we changed: We were having problems with idmapd uid:gid mappings from the bind mount and just as an experiment we tried the old way. Problem solved.

No. If you are exporting /export fsid-0 and crossmnt-ing /export/opt, you mount server:/opt client/somewhere. The only file you change is /etc/exports on the server, to contain the old style mount spec. /opt yourIPs(rw,sec=none:sys:krb5. . .)
The client sees exactly he same path.

HTH
Steve

That almost had to be a coincidence--there's no connection between the two things.

--b.

I removed the pseudo root entry.
/etc/exports:
/srv/opt *(sec=sys:krb5,rw,sync,no_subtree_check)

but now I get:

Feb 13 10:43:45 tm mountd[18045]: Kernel does not have pseudo root support.
Feb 13 10:43:45 tm mountd[18045]: NFS v4 mounts will be disabled unless fsid=0
Feb 13 10:43:45 tm mountd[18045]: is specfied in /etc/exports file.

Also I can't mount on client anymore:

# mount -t nfs4 <server-fqdn>:/opt /opt
mount.nfs4: mounting ...:/opt failed, reason given by server:
  No such file or directory

# mount -t nfs4 :/srv/opt /opt
mount.nfs4: mounting ...:/srv/opt failed, reason given by server:
  No such file or directory

I started with host/... and nfs/... principals and got an access denied error while mounting. Thus I added the root principal also.

Which of the four mentioned keys are necessary, resp. which combinations are sufficient? Or do I always need host, nfs and root?

regards
knut

I already told you that you need a backport of the latest version of libtirpc. The Version included in squeeze is broken.

Sven

If you can spend five minutes on the Internet and do not run Linux,
you're a genius. (Dirk Hohndel)

/me is giggls@ircnet, http://sven.gegg.us/ on the Web

You recommended to use backports. As you now say the lib is broken things are different and I finally solved my problem.

For those who face similar issues I sum up my last steps:

- include squeeze-backports and upgrade nfs-common, nfs-kernel-server to version 1.2.4 and Linux kernel to 3.2
- replace portmap by rpcbind
- install version 0.2.2 of libtirpc from unstable (forced new libc6)
- remove pseudo root from /etc/exports
- use AES keys for Kerberos

Thanks to all for your helpful hints! :)

regards
knut

------

# dpkg -l ...
ii  libnfsidmap2                                0.23-2                                      An nfs idmapping library
ii  nfs-common                                  1:1.2.4-1~bpo60+1                           NFS support files common to client and server
ii  nfs-kernel-server                           1:1.2.4-1~bpo60+1                           support for NFS kernel server

ii  libgssrpc4                                  1.8.3+dfsg-4squeeze5                        MIT Kerberos runtime libraries - GSS enabled ONCRPC
ii  librpcsecgss3                               0.19-2                                      allows secure rpc communication using the rpcsec_gss protocol
ii  libtirpc1                                   0.2.2-5                                     transport-independent RPC library
ii  rpcbind                                     0.2.0-4.1                                   converts RPC program numbers into universal addresses

ii linux-image-3.2.0-0.bpo.1-686-pae 3.2.4-1~bpo60+1 Linux 3.2 for modern PCs

It sounds like you'd like to request a backport of libtirpc to squeeze-backports. is that right? that is best done on the debian backports mailing list (cc'ed here)

I've set reply-to [email protected], so please follow up there if you'd like to see this package backported.

Regards,

	--dkg

Don't do this. Just backport yourself. This is trivial in that case (apt-get source + dpkg-buildpackage)

Sven

Das Internet ist kein rechtsfreier Raum, das Internet ist aber auch
kein bürgerrechtsfreier Raum. (Wolfgang Wieland Bündnis 90/Die Grünen)

/me is giggls@ircnet, http://sven.gegg.us/ on the Web