Debugging mount failure with netgroup

Trying to mount:

[root@wont ~]# mount -o vers=3,hard,intr adept:/fooper /t/adept
mount.nfs: access denied by server while mounting adept:/fooper
[root@wont ~]# uname -a
Linux wont.internal.excfb.com 2.6.35.10-74.fc14.x86_64 #1 SMP Thu Dec 23 16:04:50 UTC 2010 x86_64 x86_64 x86_64 GNU/Linux [root@wont ~]# showmount -e adept
Export list for adept:
/fooper @adeptya

The server tells us:

[root@adept log]# more /etc/exports
/fooper @titan(rw)
[root@adept log]# vi /etc/exports
[root@adept log]# more /etc/exports
/fooper @adeptya(rw)
[root@adept log]# exportfs -au
[root@adept log]# exportfs -a
[root@adept log]# exportfs
/fooper @adeptya
[root@adept log]# ypmatch adeptya netgroup
(adept,,) (adept.internal.excfb.com,,) (wont,,) (wont.internal.excfb.com,,) [root@adept log]# host wont.internal.excfb.com wont.internal.excfb.com has address 192.168.2.105
[root@adept log]# host 192.168.2.105
105.2.168.192.in-addr.arpa domain name pointer wont.internal.excfb.com. [root@adept log]# tail messages

Feb  7 13:33:41 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:34:58 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:36:02 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:36:32 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:37:14 adept mountd[13050]: authenticated mount request from 192.168.2.105:698 for /fooper (/fooper)
Feb  7 13:38:12 adept mountd[13050]: authenticated unmount request from 192.168.2.105:657 for /fooper (/fooper)
Feb  7 13:39:06 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:43:52 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:45:24 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host
Feb  7 13:48:17 adept mountd[13050]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host

The success at 13:37:14 was from when I changed the export to:

[root@adept ~]# exportfs
/fooper 192.168.2.0/255.255.255.0

to verify that NFS was working. At that point I was able to mount, create a file, etc.

Oh yeah, the NFS server is also the NIS and DNS server, so there shouldn't be any connectivity issues.

What can I do to further triage this?

Thanks!

Tom Haynes
ex-cfb

Alas, I've never used netgroups, much less tried to debug problems with them.... Have you tried looking at the nis traffic over "lo" to see if mountd's doing the right queries? You could also try seeing if adding "-dall" to the rpc.mountd commandline turns up any useful debugging.

After that, I don't know, I suppose I'd start looking at the mountd source....

--b.

[tdh@adept ~]> sudo tshark -i lo
Running as user "root" and group "root". This could be dangerous.
Capturing on lo
0.000000 192.168.2.108 -> 192.168.2.108 DNS Standard query PTR 105.2.168.192.in-addr.arpa 0.002862 192.168.2.108 -> 192.168.2.108 DNS Standard query response PTR wont.internal.excfb.com 0.003455 192.168.2.108 -> 192.168.2.108 DNS Standard query A wont.internal.excfb.com 0.003499 192.168.2.108 -> 192.168.2.108 DNS Standard query AAAA wont.internal.excfb.com 0.003819 192.168.2.108 -> 192.168.2.108 DNS Standard query response A 192.168.2.105 0.003939 192.168.2.108 -> 192.168.2.108 DNS Standard query response
0.004427 192.168.2.108 -> 192.168.2.108 DNS Standard query A wont.internal.excfb.com 0.004761 192.168.2.108 -> 192.168.2.108 DNS Standard query response A 192.168.2.105 7.472292 192.168.2.108 -> 192.168.2.108 YPSERV V2 DOMAIN Call
7.472413 192.168.2.108 -> 192.168.2.108 YPSERV V2 DOMAIN Reply (Call In 9)

The above seems to suggest that the host lookup went fine.

-dall:

Feb  7 14:35:27 adept mountd[13285]: check_default: access by 192.168.2.105 ALLOWED (cached)
Feb  7 14:35:27 adept mountd[13285]: Received NULL request from 192.168.2.105
Feb  7 14:35:27 adept mountd[13285]: check_default: access by 192.168.2.105 ALLOWED (cached)
Feb  7 14:35:27 adept mountd[13285]: Received NULL request from 192.168.2.105
Feb  7 14:35:27 adept mountd[13285]: check_default: access by 192.168.2.105 ALLOWED (cached)
Feb  7 14:35:27 adept mountd[13285]: Received MNT3(/fooper) request from 192.168.2.105
Feb  7 14:35:27 adept mountd[13285]: refused mount request from 192.168.2.105 for /fooper (/fooper): unmatched host

Yeah, I'm not sure if I can do that. :->

I did google and did not find much on netgroups - but I did find a bunch on "unmatched host".

It seems to be black magic from some of the answers.

And I agree:

1) Modified the export:
[root@adept /]# more /etc/exports
/fooper @adeptya(rw,insecure,no_root_squash,sync)

2) Did other mumbo-jumbo

3) Really, really did not want to reboot.

4) service nfs stop ; exportfs ; service nfs start

And that last one worked!

Feb  7 14:44:31 adept mountd[13712]: Received MNT3(/fooper) request from 192.168.2.105
Feb  7 14:44:31 adept mountd[13712]: authenticated mount request from wont.internal.excfb.com:972 for /fooper (/fooper)
Feb  7 14:44:31 adept mountd[13712]: nfsd_fh: inbuf '@adeptya 7 \x0100060000000000a9a3be796f4c43878b99ea75025e5384'
Feb  7 14:44:31 adept mountd[13712]: nfsd_fh: found 0x20da950 path /fooper

Thanks

Tom Haynes
ex-cfb

So it was probably the "insecure"?

Are we returning the right error to the mount client in that case?

And I suppose the logged error should be more helpful. I'm not really sure mountd should be logging every mount failure by default, but if you request it, it should at be able to tell you the reason for the failure.

--b.

Sorry, I forgot to mention after every step that I tried the mount and it failed. It was only restarting nfs that worked.

I'd have to check to see what port number the requests are coming in on, but I don't think it is this.

And I've actually got the system horked up right now - I restarted one time too many. Frustrating as there isn't anything logged beyond "unmatched host".

Tom Haynes
ex-cfb

Changes to /etc/exports aren't automatically detected, so you need a server restart or an exportfs -ra after any change.

I forgot you said that was the error. And looking at the code we actually do have a different message for the port number case ("illegal port").

OK, so maybe the problem was just that mountd never found out about your original /etc/exports file?

--b.

Yep:

exportfs -au  # unload old rules
exportfs      # verify it is empty
exportfs -a   # load it up
exportfs      # verify it loaded

So I would have expected it to have reloaded.

That wouldn't explain why it has stopped working now.

Neither the exportfs commands above nor a service nfs stop/start yields a working mount from the client.

Restarting nfsd with an export of:

[root@adept internal.excfb.com]# exportfs
/fooper 192.168.2.0/255.255.255.0
[root@adept internal.excfb.com]# uname -a
Linux adept.internal.excfb.com 2.6.34.7-66.fc13.i686.PAE #1 SMP Wed Dec 15 07:21:49 UTC 2010 i686 i686 i386 GNU/Linux

Does yield a mount.

Tom Haynes
ex-cfb

Sorry, and you said that in your original email. OK.

And it's failing in the same way? ("unmatched host"?)

Like I say, I suppose my next step would be to get out the nfs-utils source and figure out where it's failing (probably somewhere in auth_authenticate_newcache -> get_client_hostname -> client_compose -> client_check -> check_netgroup()).

If it's reporting "unmatched host", and if you can see it making the right NIS request (and getting the right response), then I'm out of other debugging ideas....

--b.