NFS problems across reboot

After rebooting my NFS server, the clients can no longer access the mounts.

A trimmed df shows:

james:/home/james            0         1         0   0% /home/james
james:/home/hnc              0         1         0   0% /home/hnc

# su - xyz
su: warning: cannot change directory to /home/james/x/xyz: Permission denied
bash: /home/james/x/xyz/.bash_profile: Permission denied

However:

# rpcinfo -p james
   program vers proto   port
    100000    2   tcp    111  portmapper
    100000    2   udp    111  portmapper
    100005    1   udp  10000  mountd
    100005    1   tcp  10000  mountd
    100005    2   udp  10000  mountd
    100005    2   tcp  10000  mountd
    100005    3   udp  10000  mountd
    100005    3   tcp  10000  mountd
    100003    2   udp   2049  nfs
    100003    3   udp   2049  nfs
    100021    1   udp  10001  nlockmgr
    100021    3   udp  10001  nlockmgr
    100021    4   udp  10001  nlockmgr
    100024    1   udp  10002  status
    100024    1   tcp  10001  status

# umount /home/hnc
# mount /home/hnc

And the hnc mount is fine again. After repeating with /home/james, that mount is also fine.

Why do I need to umount and remount? This is pretty brutal when I need to shutdown services so the mounts aren't in use.

These are 2.4.18+ kernels with nfs utils from about a month ago.

David

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Can't offer any solutions, but I've had the same problem - (see
http://marc.theaimsgroup.com/?l=linux-nfs&m=101610950230938&w=2 )

In my case, one particular application (accessing files via NFS on the server at the time) was causing the server machine to crash, when the server rebooted, Linux clients got "Permission denied" problems on the mount points from the server (IRIX clients reported "I/O Error" on the same mount points).

I thought I had fixed the problem by using a newer kernel on the server
- but all this fixed was the application in question crashing the machine ... I still see the "Permission denied" problem from time to time (df reports a similar output to yours as well) - however I don't see it every time a 'server' reboots - most of our Linux workstations NFS export local disks - which can be automounted by other workstations
- the workstations tend to get rebooted 'fairly' frequently.

The workaround is to umount/mount ...

I'm using kernels 2.4.7 and 2.4.14 (both with XFS).

It doesn't seem to be a Linux NFS client problem - as I mentioned above, SGI IRIX clients have a similar problem if the Linux server crashes/reboots.

James Pearson

Quoted 44 lines · David Ford

After rebooting my NFS server, the clients can no longer access the mounts.

A trimmed df shows:

james:/home/james            0         1         0   0% /home/james
james:/home/hnc              0         1         0   0% /home/hnc

# su - xyz
su: warning: cannot change directory to /home/james/x/xyz: Permission denied
bash: /home/james/x/xyz/.bash_profile: Permission denied

However:

# rpcinfo -p james
  program vers proto   port
   100000    2   tcp    111  portmapper
   100000    2   udp    111  portmapper
   100005    1   udp  10000  mountd
   100005    1   tcp  10000  mountd
   100005    2   udp  10000  mountd
   100005    2   tcp  10000  mountd
   100005    3   udp  10000  mountd
   100005    3   tcp  10000  mountd
   100003    2   udp   2049  nfs
   100003    3   udp   2049  nfs
   100021    1   udp  10001  nlockmgr
   100021    3   udp  10001  nlockmgr
   100021    4   udp  10001  nlockmgr
   100024    1   udp  10002  status
   100024    1   tcp  10001  status

# umount /home/hnc
# mount /home/hnc

And the hnc mount is fine again. After repeating with /home/james, that mount is also fine.

Why do I need to umount and remount? This is pretty brutal when I need to shutdown services so the mounts aren't in use.

These are 2.4.18+ kernels with nfs utils from about a month ago.

David

Quoted 4 lines

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Well, I'm an all Linux shop here, here are a few points to consider, Trond, please help us out.

a) All machines are Linux
b) this is 100% repeatable
c) this is entirely unacceptable, every user (/home is mounted) has to log out of their machine in order to remount, daemons need shut down.

Here's the scoop:

a) Linux 2.4.18-pre6 on one nfs server, 2.4.19-pre6 on the other
b) 2.4.18-pre7 through 2.4.19-pre6 on clients
c) mount options are all like: defaults,rsize=8192,wsize=8192
d) programs run/ning, portmap, kmountd, knfsd 5, lockd, and statd.
e) no firewall rules, /etc/exports is setup using individual IPs since the friggen system can't deal with ranges or hostnames properly, "Hostname is the same as "hostname" in DNS land, it should be in NFS land as well. exports is like: /path 1.2.3.4(rw,no_root_squash).
f) hosts.allow is basically *:all

When the server reboots (cleanly), the clients get the following, from rebooting to running:
nfs: server james not responding, still trying
nfs: server james not responding, still trying
nfs: server james OK
nfs: server james OK
nfs_statfs: statfs error = 13
nfs_statfs: statfs error = 13
nfs_statfs: statfs error = 13

13 == EACCES. Looks like the filesystem isn't exported.

Show us the relevent /etc/init.d file.
My guess is that "exportfs -a" is being run *After* rpc.nfsd. It must must run *before* for correct operation.

NeilBrown

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Quoted 6 lines · On Sunday April 7, [email protected] wrote:

13 == EACCES. Looks like the filesystem isn't exported.

Show us the relevent /etc/init.d file.
My guess is that "exportfs -a" is being run *After* rpc.nfsd. It must must run *before* for correct operation.

Actually, that would cause ESTALE, not EACCES.

EACCES seems to suggest an XFS problem.
Can you duplicate this without using XFS????

NeilBrown

Quoted 6 lines

NeilBrown

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Quoted 7 lines · On Sunday April 7, [email protected] wrote:

13 == EACCES. Looks like the filesystem isn't exported.

Show us the relevent /etc/init.d file.
My guess is that "exportfs -a" is being run *After* rpc.nfsd. It must must run *before* for correct operation.

Actually, that would cause ESTALE, not EACCES.

EACCES seems to suggest an XFS problem.
Can you duplicate this without using XFS????

NeilBrown

Quoted 5 lines

NeilBrown

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

I'm using XFS on my root file system - I'm using a "stock" RedHat 7.1 or 7.2 initscripts - so exportfs -r is run before rpc.mountd and rpc.nfsd

I can reproduce my problem by resetting an NFS server just after automounting one of its exported disks. When the server comes up, the client reports 'Permission denied' on that mount point.

If I repeat the test, but wait a short while before resetting the server, everything is OK on reboot - and rmtab contains an entry for the client).

It looks like the update to /var/lib/nfs/rmtab of the clients mount hasn't been sync'd to disk before the server is reset.

In fact I can simulate this on a running server and client by:

On the client, mount one of the server's exported file systems and cd to it.
Runs 'ls', everything fine. Then:

On the server:

cd /var/lib/nfs
cp rmtab rmtab.save
cat /dev/null > rmtab
/etc/init.d/nfs stop
/etc/init.d/nfs start

Then on the client, attempt to 'ls' etc. gives 'Permission denied'

If I then copy back the saved rmtab file and restart nfs, file access on the client is OK.

I've also put /var/lib/nfs on an ext2 file system, but that didn't make any difference - it seems to be that fact that the 'crash' happens before the file data has been flushed to disk.

The man page for rpc.mountd says the rmtab file is 'mostly ornamental', however in this case, it seems to be a bit more important. ...

So, is it possible to have a client 'survive' a server crash even if its entry in rmtab hasn't been flushed? i.e. somehow allow the client access even if there is no entry for the client in rmtab?

Thanks

James Pearson

Quoted 19 lines · Neil Brown

Quoted 7 lines · On Sunday April 7, [email protected] wrote:

13 == EACCES. Looks like the filesystem isn't exported.

Show us the relevent /etc/init.d file.
My guess is that "exportfs -a" is being run *After* rpc.nfsd. It must must run *before* for correct operation.

Actually, that would cause ESTALE, not EACCES.

EACCES seems to suggest an XFS problem.
Can you duplicate this without using XFS????

NeilBrown

Quoted 2 lines

NeilBrown

Quoted 3 lines

Quoted 3 lines

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

_______________________________________________
NFS maillist - [email protected]
https://lists.sourceforge.net/lists/listinfo/nfs

Having a quick look through the nfs-utils source, as a test, I changed fendrmtabent() in nfs-utils-0.3.3/support/nfs/rmtab.c (used by mountd) to:

void
fendrmtabent(FILE *fp)

{
        if (fp) {
                fflush(fp);
                fdatasync(fileno(fp));
                fclose(fp);
        }
}

which appears to help - I can reset my test server immediately after a client mounts a disk - without getting the 'Permission denied' problems.

However, I have no idea if this is a 'sensible' thing to do ... if it is OK to do this, would using fdatasync() elsewhere be helpful?

Show us the relevent /etc/init.d file.
My guess is that "exportfs -a" is being run *After* rpc.nfsd. It must must run *before* for correct operation.

NeilBrown

Hi,

I just checked the default NFS server init script from Debian stable (2.2) as found in '/etc/init.d/nfs-kernel-server'. It appears to violate the above order, by starting exportfs *after* rpc.nfsd. Am I missing something, please?

Could this explain why I've seen the same behaviour as described in an earlier post in this thread: in other words, when the NFS server cleanly reboots, all clients with open file handles develop difficulty with NFS access (sorry I can't remember exact error, it hasn't happened in months); come to think of it, though, I don't have this problem with my Debian woody NFS server, which has the correct 'exportfs'/'rpc.nfsd' order in the init-scripts.

Quoted 15 lines · On Saturday April 6, [email protected] wrote:

Well, I'm an all Linux shop here, here are a few points to consider, Trond, please help us out.

a) All machines are Linux
b) this is 100% repeatable
c) this is entirely unacceptable, every user (/home is mounted) has to log out of their machine in order to remount, daemons need shut down.

Here's the scoop:

a) Linux 2.4.18-pre6 on one nfs server, 2.4.19-pre6 on the other
b) 2.4.18-pre7 through 2.4.19-pre6 on clients
c) mount options are all like: defaults,rsize=8192,wsize=8192
d) programs run/ning, portmap, kmountd, knfsd 5, lockd, and statd.
e) no firewall rules, /etc/exports is setup using individual IPs since the friggen system can't deal with ranges or hostnames properly, "Hostname is the same as "hostname" in DNS land, it should be in NFS land as well. exports is like: /path 1.2.3.4(rw,no_root_squash).
f) hosts.allow is basically *:all

Quoted 9 lines

When the server reboots (cleanly), the clients get the following, from rebooting to running:
nfs: server james not responding, still trying
nfs: server james not responding, still trying
nfs: server james OK
nfs: server james OK
nfs_statfs: statfs error = 13
nfs_statfs: statfs error = 13
nfs_statfs: statfs error = 13

13 == EACCES. Looks like the filesystem isn't exported.

Show us the relevent /etc/init.d file.
My guess is that "exportfs -a" is being run *After* rpc.nfsd. It must must run *before* for correct operation.

NeilBrown

That was the problem. Everything mounts just fine normally. When the server reboots, the client loses access. All the client has to do is unmount and remount and everything is fine again. If it were a matter of what was exported, then the remount should fail, but it doesn't. Why does the client lose the access but with no action on the server part, the client regains access? (to whom know the answer, please don't let this thread just die away, ambiguity like this doesn't do anyone any good)

=20
"Exporting" a filesystem means doing two things:
1/ telling mountd that it is allow to respond positively to mount
requests.
2/ telling the kernel nfsd that it is allow to respond to access

requests.
=20
When mountd responds to a mount request, it also tells the kernel to respond to access requests.
But after a reboot, you need to run exportfs. It reads /var/lib/nfs/rmtab to find out which clients have which filesystems mounted, and it tells the kernel to allow those clients to access those filesystem.

Quoted 7 lines

"Exporting" a filesystem means doing two things:
1/ telling mountd that it is allow to respond positively to mount
requests.
2/ telling the kernel nfsd that it is allow to respond to access
requests.

Quoted 6 lines

When mountd responds to a mount request, it also tells the kernel to respond to access requests.
But after a reboot, you need to run exportfs. It reads /var/lib/nfs/rmtab to find out which clients have which filesystems mounted, and it tells the kernel to allow those clients to access those filesystem.

Is it sufficent, if exportfs -r is run before rpc.nfsd are started instead of exportfs -a?
The startscripts of redhat 6.2 uses exportfs -r (and we sometimes have troubles with stale mounts).