Use of /etc/netgroup appears to be broken in the NFS server version which ships with Ubuntu 20.04
Subsystems (2)
Sadly, it took me a couple of days to track this down. The /etc/netgroup file I'm using works perfectly on another NFS server (Ubuntu 18.04) in production, so this wasn't an immediate suspicion. However, if I use this /etc/exports:
/srv/nfs @cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check) /srv/nfs/cryosparc @cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)
Client mounts fail:
root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021 mount.nfs4: trying text-based options 'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx' mount.nfs4: mount(2): Permission denied mount.nfs4: access denied by server while mounting cerebro:/cryosparc
and if I switch to specifying the host explicitly:
/srv/nfs javelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)
/srv/nfs/cryosparcjavelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)
the mount just works. The tcpdump error message isn't terribly helpful here:
11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 0
11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth Bogus Credentials (seal broken)
but after figuring out the cause of the problem, I did find a corroborating RHEL error report (which you'll need a RHEL account to access):
https://access.redhat.com/solutions/3563601
I couldn't figure out how to determine the exact version of the NFS server that ships with Ubuntu 20.04. Maybe someone could explain how to do this. Running
/usr/sbin/rpc.nfsd --version
doesn't do it.
Quoted 40 lines · Patrick Goetz
Sadly, it took me a couple of days to track this down. The
/etc/netgroupfile I'm using works perfectly on another NFS server (Ubuntu 18.04) in production, so this wasn't an immediate suspicion. However, if I use this/etc/exports:
/srv/nfs@cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check)/srv/nfs/cryosparc@cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)Client mounts fail:
root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021 mount.nfs4: trying text-based options 'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx' mount.nfs4: mount(2): Permission denied mount.nfs4: access denied by server while mounting cerebro:/cryosparcand if I switch to specifying the host explicitly:
/srv/nfsjavelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)
/srv/nfs/cryosparcjavelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)the mount just works. The tcpdump error message isn't terribly helpful here:
11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 0
11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth Bogus Credentials (seal broken)but after figuring out the cause of the problem, I did find a corroborating RHEL error report (which you'll need a RHEL account to access):
Quoted 9 lines
https://access.redhat.com/solutions/3563601I couldn't figure out how to determine the exact version of the NFS server that ships with Ubuntu 20.04. Maybe someone could explain how to do this. Running
/usr/sbin/rpc.nfsd --versiondoesn't do it.
The problem is unlikely to be the implementation of netgroups - that hasn't changed in a long time. It is more likely to be some subtle configuration difference.
Can you provide the verbatim /etc/netgroups file, and the extact host name that a DNS lookup of the client IP adress results in?
NeilBrown
Hi Neil -
This is extremely embarrassing, but chalk this one up to user error (technically, PEBKAC). I'm used to /etc/nsswitch.conf always including the files option, so didn't think to check this when in fact on Ubuntu 20.04 they ship the nsswitch.conf with this netgroup entry.
netgroup: nis
Who still uses NIS? Beats me; but once I added files to the option list it starting working as advertised. Very sorry to dump random noise onto the list.
Quoted 60 lines · NeilBrown
Quoted 40 lines · Patrick Goetz
Sadly, it took me a couple of days to track this down. The
/etc/netgroupfile I'm using works perfectly on another NFS server (Ubuntu 18.04) in production, so this wasn't an immediate suspicion. However, if I use this/etc/exports:
/srv/nfs@cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check)/srv/nfs/cryosparc@cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)Client mounts fail:
root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021 mount.nfs4: trying text-based options 'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx' mount.nfs4: mount(2): Permission denied mount.nfs4: access denied by server while mounting cerebro:/cryosparcand if I switch to specifying the host explicitly:
/srv/nfsjavelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)
/srv/nfs/cryosparcjavelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)the mount just works. The tcpdump error message isn't terribly helpful here:
11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 0
11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth Bogus Credentials (seal broken)but after figuring out the cause of the problem, I did find a corroborating RHEL error report (which you'll need a RHEL account to access):
Quoted 9 lines
https://access.redhat.com/solutions/3563601I couldn't figure out how to determine the exact version of the NFS server that ships with Ubuntu 20.04. Maybe someone could explain how to do this. Running
/usr/sbin/rpc.nfsd --versiondoesn't do it.
The problem is unlikely to be the implementation of netgroups - that hasn't changed in a long time. It is more likely to be some subtle configuration difference.
Can you provide the verbatim
/etc/netgroupsfile, and the extact host name that a DNS lookup of the client IP adress results in?NeilBrown
Quoted 7 lines · Patrick Goetz
Hi Neil -
This is extremely embarrassing, but chalk this one up to user error (technically, PEBKAC). I'm used to
/etc/nsswitch.confalways including the files option, so didn't think to check this when in fact on Ubuntu 20.04 they ship the nsswitch.conf with this netgroup entry.
Quoted 5 lines
netgroup: nis
Who still uses NIS? Beats me; but once I added files to the option list it starting working as advertised. Very sorry to dump random noise onto the list.
:-)
NIS has a certain elegant simplicity. LDAP is probably objective better in every way ... except the elegant simplicity.
Glad to know I wasn't missing anything important, and that there was an easy solution.
NeilBrown
Quoted 65 lines · NeilBrown
Quoted 60 lines · Patrick Goetz
Quoted 40 lines
Sadly, it took me a couple of days to track this down. The
/etc/netgroupfile I'm using works perfectly on another NFS server (Ubuntu 18.04) in production, so this wasn't an immediate suspicion. However, if I use this/etc/exports:
/srv/nfs@cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check)/srv/nfs/cryosparc@cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)Client mounts fail:
root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021 mount.nfs4: trying text-based options 'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx' mount.nfs4: mount(2): Permission denied mount.nfs4: access denied by server while mounting cerebro:/cryosparcand if I switch to specifying the host explicitly:
/srv/nfsjavelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)
/srv/nfs/cryosparcjavelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)the mount just works. The tcpdump error message isn't terribly helpful here:
11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 0
11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth Bogus Credentials (seal broken)but after figuring out the cause of the problem, I did find a corroborating RHEL error report (which you'll need a RHEL account to access):
Quoted 9 lines
https://access.redhat.com/solutions/3563601I couldn't figure out how to determine the exact version of the NFS server that ships with Ubuntu 20.04. Maybe someone could explain how to do this. Running
/usr/sbin/rpc.nfsd --versiondoesn't do it.
The problem is unlikely to be the implementation of netgroups - that hasn't changed in a long time. It is more likely to be some subtle configuration difference.
Can you provide the verbatim
/etc/netgroupsfile, and the extact host name that a DNS lookup of the client IP adress results in?NeilBrown
Quoted 10 lines · NeilBrown
Quoted 9 lines · Patrick Goetz
Hi Neil -
This is extremely embarrassing, but chalk this one up to user error (technically, PEBKAC). I'm used to
/etc/nsswitch.confalways including the files option, so didn't think to check this when in fact on Ubuntu 20.04 they ship the nsswitch.conf with this netgroup entry.netgroup: nis
Quoted 8 lines
Quoted 3 lines
Who still uses NIS? Beats me; but once I added files to the option list it starting working as advertised. Very sorry to dump random noise onto the list.
:-)
NIS has a certain elegant simplicity. LDAP is probably objective better in every way ... except the elegant simplicity.
I agree. I hate LDAP (so inelegant) and I find ASN.1 to be especially inelegant. (I've never seen anything to indicate this is anything other than an anachronism.)
Quoted 71 lines
Glad to know I wasn't missing anything important, and that there was an easy solution.
NeilBrown
Quoted 65 lines · NeilBrown
Quoted 60 lines · Patrick Goetz
Quoted 40 lines
Sadly, it took me a couple of days to track this down. The
/etc/netgroupfile I'm using works perfectly on another NFS server (Ubuntu 18.04) in production, so this wasn't an immediate suspicion. However, if I use this/etc/exports:
/srv/nfs@cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check)/srv/nfs/cryosparc@cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)Client mounts fail:
root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021 mount.nfs4: trying text-based options 'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx' mount.nfs4: mount(2): Permission denied mount.nfs4: access denied by server while mounting cerebro:/cryosparcand if I switch to specifying the host explicitly:
/srv/nfsjavelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)
/srv/nfs/cryosparcjavelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)the mount just works. The tcpdump error message isn't terribly helpful here:
11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 0
11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth Bogus Credentials (seal broken)but after figuring out the cause of the problem, I did find a corroborating RHEL error report (which you'll need a RHEL account to access):
Quoted 9 lines
https://access.redhat.com/solutions/3563601I couldn't figure out how to determine the exact version of the NFS server that ships with Ubuntu 20.04. Maybe someone could explain how to do this. Running
/usr/sbin/rpc.nfsd --versiondoesn't do it.
The problem is unlikely to be the implementation of netgroups - that hasn't changed in a long time. It is more likely to be some subtle configuration difference.
Can you provide the verbatim
/etc/netgroupsfile, and the extact host name that a DNS lookup of the client IP adress results in?NeilBrown