Skip to contentSkip to conversation

Use of /etc/netgroup appears to be broken in the NFS server version which ships with Ubuntu 20.04

DISCUSSION Patrick Goetz 2 participants 5 messages Jun 15, 2021 – Jun 17, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Sadly, it took me a couple of days to track this down. The /etc/netgroup file I'm using works perfectly on another NFS server (Ubuntu 18.04) in production, so this wasn't an immediate suspicion. However, if I use this /etc/exports:

/srv/nfs @cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check) /srv/nfs/cryosparc @cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)

Client mounts fail:

root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc
mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021
mount.nfs4: trying text-based options 'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting cerebro:/cryosparc

and if I switch to specifying the host explicitly:

/srv/nfs javelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)

/srv/nfs/cryosparc
javelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)

the mount just works. The tcpdump error message isn't terribly helpful here:

11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 0
11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth Bogus Credentials (seal broken)

but after figuring out the cause of the problem, I did find a corroborating RHEL error report (which you'll need a RHEL account to access):

https://access.redhat.com/solutions/3563601

I couldn't figure out how to determine the exact version of the NFS server that ships with Ubuntu 20.04. Maybe someone could explain how to do this. Running
/usr/sbin/rpc.nfsd --version
doesn't do it.

#1 lore raw
Quoted 40 lines · Patrick Goetz

Sadly, it took me a couple of days to track this down. The /etc/netgroup file I'm using works perfectly on another NFS server (Ubuntu 18.04) in production, so this wasn't an immediate suspicion. However, if I use this /etc/exports:

/srv/nfs @cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check) /srv/nfs/cryosparc @cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)

Client mounts fail:

root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc
mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021
mount.nfs4: trying text-based options 
'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting cerebro:/cryosparc

and if I switch to specifying the host explicitly:

/srv/nfs javelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)

/srv/nfs/cryosparc
javelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)

the mount just works. The tcpdump error message isn't terribly helpful here:

11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 0
11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth Bogus Credentials (seal broken)

but after figuring out the cause of the problem, I did find a corroborating RHEL error report (which you'll need a RHEL account to access):

Quoted 9 lines
   https://access.redhat.com/solutions/3563601

I couldn't figure out how to determine the exact version of the NFS server that ships with Ubuntu 20.04. Maybe someone could explain how to do this. Running

    /usr/sbin/rpc.nfsd --version

doesn't do it.

The problem is unlikely to be the implementation of netgroups - that hasn't changed in a long time. It is more likely to be some subtle configuration difference.

Can you provide the verbatim /etc/netgroups file, and the extact host name that a DNS lookup of the client IP adress results in?

NeilBrown

#2 lore raw

Hi Neil -

This is extremely embarrassing, but chalk this one up to user error (technically, PEBKAC). I'm used to /etc/nsswitch.conf always including the files option, so didn't think to check this when in fact on Ubuntu 20.04 they ship the nsswitch.conf with this netgroup entry.

netgroup: nis

Who still uses NIS? Beats me; but once I added files to the option list it starting working as advertised. Very sorry to dump random noise onto the list.

Quoted 60 lines · NeilBrown
Quoted 40 lines · Patrick Goetz

Sadly, it took me a couple of days to track this down. The /etc/netgroup file I'm using works perfectly on another NFS server (Ubuntu 18.04) in production, so this wasn't an immediate suspicion. However, if I use this /etc/exports:

/srv/nfs @cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check) /srv/nfs/cryosparc @cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)

Client mounts fail:

root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc
mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021
mount.nfs4: trying text-based options
'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting cerebro:/cryosparc

and if I switch to specifying the host explicitly:

/srv/nfs javelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)

/srv/nfs/cryosparc
javelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)

the mount just works. The tcpdump error message isn't terribly helpful here:

11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 0
11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth Bogus Credentials (seal broken)

but after figuring out the cause of the problem, I did find a corroborating RHEL error report (which you'll need a RHEL account to access):

Quoted 9 lines
    https://access.redhat.com/solutions/3563601

I couldn't figure out how to determine the exact version of the NFS server that ships with Ubuntu 20.04. Maybe someone could explain how to do this. Running

     /usr/sbin/rpc.nfsd --version

doesn't do it.

The problem is unlikely to be the implementation of netgroups - that hasn't changed in a long time. It is more likely to be some subtle configuration difference.

Can you provide the verbatim /etc/netgroups file, and the extact host name that a DNS lookup of the client IP adress results in?

NeilBrown

#3 lore raw
Quoted 7 lines · Patrick Goetz

Hi Neil -

This is extremely embarrassing, but chalk this one up to user error (technically, PEBKAC). I'm used to /etc/nsswitch.conf always including the files option, so didn't think to check this when in fact on Ubuntu 20.04 they ship the nsswitch.conf with this netgroup entry.

Quoted 5 lines

netgroup: nis

Who still uses NIS? Beats me; but once I added files to the option list it starting working as advertised. Very sorry to dump random noise onto the list.

:-)
NIS has a certain elegant simplicity. LDAP is probably objective better in every way ... except the elegant simplicity.

Glad to know I wasn't missing anything important, and that there was an easy solution.

NeilBrown

Quoted 65 lines · NeilBrown
Quoted 60 lines · Patrick Goetz
Quoted 40 lines

Sadly, it took me a couple of days to track this down. The /etc/netgroup file I'm using works perfectly on another NFS server (Ubuntu 18.04) in production, so this wasn't an immediate suspicion. However, if I use this /etc/exports:

/srv/nfs @cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check) /srv/nfs/cryosparc @cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)

Client mounts fail:

root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc
mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021
mount.nfs4: trying text-based options
'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting cerebro:/cryosparc

and if I switch to specifying the host explicitly:

/srv/nfs javelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)

/srv/nfs/cryosparc
javelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)

the mount just works. The tcpdump error message isn't terribly helpful here:

11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 0
11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth Bogus Credentials (seal broken)

but after figuring out the cause of the problem, I did find a corroborating RHEL error report (which you'll need a RHEL account to access):

Quoted 9 lines
    https://access.redhat.com/solutions/3563601

I couldn't figure out how to determine the exact version of the NFS server that ships with Ubuntu 20.04. Maybe someone could explain how to do this. Running

     /usr/sbin/rpc.nfsd --version

doesn't do it.

The problem is unlikely to be the implementation of netgroups - that hasn't changed in a long time. It is more likely to be some subtle configuration difference.

Can you provide the verbatim /etc/netgroups file, and the extact host name that a DNS lookup of the client IP adress results in?

NeilBrown

#4 lore raw
Quoted 10 lines · NeilBrown
Quoted 9 lines · Patrick Goetz

Hi Neil -

This is extremely embarrassing, but chalk this one up to user error (technically, PEBKAC). I'm used to /etc/nsswitch.conf always including the files option, so didn't think to check this when in fact on Ubuntu 20.04 they ship the nsswitch.conf with this netgroup entry.

netgroup: nis

Quoted 8 lines
Quoted 3 lines

Who still uses NIS? Beats me; but once I added files to the option list it starting working as advertised. Very sorry to dump random noise onto the list.

:-)
NIS has a certain elegant simplicity. LDAP is probably objective better in every way ... except the elegant simplicity.

I agree. I hate LDAP (so inelegant) and I find ASN.1 to be especially inelegant. (I've never seen anything to indicate this is anything other than an anachronism.)

Quoted 71 lines

Glad to know I wasn't missing anything important, and that there was an easy solution.

NeilBrown

Quoted 65 lines · NeilBrown
Quoted 60 lines · Patrick Goetz
Quoted 40 lines

Sadly, it took me a couple of days to track this down. The /etc/netgroup file I'm using works perfectly on another NFS server (Ubuntu 18.04) in production, so this wasn't an immediate suspicion. However, if I use this /etc/exports:

/srv/nfs @cryo_em(rw,sync,fsid=0,crossmnt,no_subtree_check) /srv/nfs/cryosparc @cryo_em(rw,sync,fsid=2,crossmnt,no_subtree_check)

Client mounts fail:

root@javelina:~# mount -vvvt nfs4 cerebro:/cryosparc /cryosparc
mount.nfs4: timeout set for Tue Jun 15 11:53:22 2021
mount.nfs4: trying text-based options
'vers=4.2,addr=128.xx.xx.xxx,clientaddr=129.xxx.xxx.xx'
mount.nfs4: mount(2): Permission denied
mount.nfs4: access denied by server while mounting cerebro:/cryosparc

and if I switch to specifying the host explicitly:

/srv/nfs javelina.my.domain(rw,sync,fsid=0,crossmnt,no_subtree_check)

/srv/nfs/cryosparc
javelina.mydomain(rw,sync,fsid=2,crossmnt,no_subtree_check)

the mount just works. The tcpdump error message isn't terribly helpful here:

11:14:02.856094 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [.], ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 0
11:14:02.856178 IP cerebro.my.domain.nfs > javelina.my.domain.741: Flags [P.], seq 1:25, ack 281, win 507, options [nop,nop,TS val 791638255 ecr 2576087678], length 24: NFS reply xid 2752089303 reply ERR 20: Auth Bogus Credentials (seal broken)

but after figuring out the cause of the problem, I did find a corroborating RHEL error report (which you'll need a RHEL account to access):

Quoted 9 lines
     https://access.redhat.com/solutions/3563601

I couldn't figure out how to determine the exact version of the NFS server that ships with Ubuntu 20.04. Maybe someone could explain how to do this. Running

      /usr/sbin/rpc.nfsd --version

doesn't do it.

The problem is unlikely to be the implementation of netgroups - that hasn't changed in a long time. It is more likely to be some subtle configuration difference.

Can you provide the verbatim /etc/netgroups file, and the extact host name that a DNS lookup of the client IP adress results in?

NeilBrown