Has replicated mount failover been implemented?
We are in the process of switching from Sun blades to Linux blades for our compute farm and are running into issues with automount not failing over when an NFS server goes down. I am wondering if it is a configuration issue, if I am using a version of autofs that doesn't support failover or if it hasn't been implemented yet.
Currently I am running SLES 10 with autofs 4.1.4. Here are my configs. The server will pick the next host in line at mount but if the failure happens after mount the mount does NOT failover. Any information would be greatly appreciated.:
/etc/auto_appl
test -rsize=32768,wsize=32768,nfsvers=3,tcp,retrans=5,timeo=600 test-ap01,test-ap02,test-ap03:/export/appl/test
/etc/auto.master
/appl /etc/auto_appl -rw,intr,nosuid,nobrowse /apps /etc/auto_apps -ro,intr,nosuid,nobrowse /home /etc/auto_home -rw,intr,nosuid,nobrowse
/etc/sysconfig/autofs
AUTOFS_OPTIONS="--timeout 3600"
____________________________________________________________________________________
Tonight's top picks. What will you watch tonight? Preview the hottest shows on Yahoo! TV.
http://tv.yahoo.com/
Ian Kent
maintainerQuoted 8 lines · John Simon
We are in the process of switching from Sun blades to Linux blades for our compute farm and are running into issues with automount not failing over when an NFS server goes down. I am wondering if it is a configuration issue, if I am using a version of autofs that doesn't support failover or if it hasn't been implemented yet.
Quoted 5 lines
Currently I am running SLES 10 with autofs 4.1.4. Here are my configs. The server will pick the next host in line at mount but if the failure happens after mount the mount does NOT failover. Any information would be greatly appreciated.:
Failover of active NFS mounts isn't something that autofs can do. When I say that I don't mean it hasn't been implemented, I mean, as far as I can see, it's not possible for autofs to do it. I've thought about quite a bit and I just can't see a way to implement it.
Once autofs has selected a server and performed the mount it has no knowledge of what is happening in the mount itself. All that autofs could do (if it was implemented in the NFS client) is to pass an ordered list of servers to mount so it can then pass that to the kernel. This is essentially what the Solaris automount does.
This has to be supported in the kernel NFS client and it's not.
Ian
Since client-side failover is not currently possible with Linux autofs does anyone have any recommendations for minimizing server side impact during when a HA-NFS server fails over. Right now what happens with HA-NFS is all the clients retry so much they basically end up DOS'ing the NFS server and causing it to failover again and again. We have hundreds of clients.
Quoted 10 lines · --- John Simon
We are in the process of switching from Sun blades to
Linux blades for our compute farm and are running into
issues with automount not failing over when an NFS server goes down. I am wondering if it is a configuration issue, if I am using a version of autofs
that doesn't support failover or if it hasn't been implemented yet.
____________________________________________________________________________________
Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase.
http://farechase.yahoo.com/
Quoted 7 lines · John Simon
Since client-side failover is not currently possible with Linux autofs does anyone have any recommendations for minimizing server side impact during when a HA-NFS server fails over. Right now what happens with HA-NFS is all the clients retry so much they basically end up DOS'ing the NFS server and causing it to failover again and again. We have hundreds of clients.
See the retry, retrans, and timeo nfs mount options.
Cheers,
Jeff
Quoted 7 lines · John Simon
Since client-side failover is not currently possible with Linux autofs does anyone have any recommendations for minimizing server side impact during when a HA-NFS server fails over. Right now what happens with HA-NFS is all the clients retry so much they basically end up DOS'ing the NFS server and causing it to failover again and again. We have hundreds of clients.
I don't think that I understand what the client side failover of Solaris was being used for in this configuration. If the server is truly HA, then shouldn't the NFS service be able to failover from one server to the next with minimal interruption on the clients?
The Solaris client side failover required relooking up all file handles which referred to the dead server, so it wasn't cheap either.
Thanx...
psWe have two pairs of VCS HA-NFS servers (one in each data center), so each entry in auto_appl is the VIP for the HA pair. Should one portion of the cluster in the data center local to the compute engines fail it will just fail over to the other cluster node and client-side failover would not be used. If however the entire cluster fails or is unavailable for whatever reason the duplicate VCS HA-NFS server in the other data center resumes serving data, albeit at a slightly slower rate due to latency over the MAN. This site to site failover is where we currently depend on Solaris client-side failover.
Quoted 16 lines · --- Peter Staubach
I don't think that I understand what the client side failover
of Solaris was being used for in this configuration.
If the
server is truly HA, then shouldn't the NFS service be able to
failover from one server to the next with minimal interruption
on the clients?The Solaris client side failover required relooking up all
file handles which referred to the dead server, so it wasn't
cheap either.
Quoted 4 lines
Thanx...
ps____________________________________________________________________________________
Need a vacation? Get great deals
to amazing places on Yahoo! Travel.
http://travel.yahoo.com/
Quoted 4 lines · John Simon
We have two pairs of VCS HA-NFS servers (one in each data center), so each entry in
auto_applis the VIP for the HA pair. Should one portion of the cluster in the data center local to the compute engines fail it
Quoted 8 lines
will just fail over to the other cluster node and client-side failover would not be used. If however the entire cluster fails or is unavailable for whatever reason the duplicate VCS HA-NFS server in the other data center resumes serving data, albeit at a slightly slower rate due to latency over the MAN. This site to site failover is where we currently depend on Solaris client-side failover.
Ahhh. Okay, thanx!
ps
John Simon wrote, On 10/11/2007 01:50 PM:
Quoted 9 lines
Since client-side failover is not currently possible with Linux autofs does anyone have any recommendations for minimizing server side impact during when a HA-NFS server fails over. Right now what happens with HA-NFS is all the clients retry so much they basically end up DOS'ing the NFS server and causing it to failover again and again. We have hundreds of clients.
Do you mean an HA Linux[1] server?
You might try running nslookup|dig in a for loop that has all of your client machines listed, if that for loop takes very long (greater than 30-60 seconds to lookup every client in your domain) then you may want to maintain a full /etc/hosts table on the server. When I had some problems with name service taking a while to respond[3] HA would give up on nfs getting started and fall back to the other machine (repeatedly), and that was when all the client machines were shutdown.
Unfortunately, you will need to maintain this hosts table so I would suggest, that like me, you write a script to build looked up information and let you know when that differs from the current /etc/hosts file.
Assuming that the failover again and again is caused by HA getting impatient with 'service nfs start', you might ask on the HA list[2] if there is a timeout value you could increase to give it a better chance of coming up in your environment (been a _long_ time since I messed with HA's timeouts and that was for ver 1.2.X).
Jeff's timeo nfs mount option might help too.
[1] http://www.linux-ha.org/
which IIRC works on solaris too.
[2] http://lists.linux-ha.org/mailman/listinfo/linux-ha
[3] it WAS very sick DNS hardware.
Quoted 16 lines
Quoted 10 lines · --- John Simon
We are in the process of switching from Sun blades to
Linux blades for our compute farm and are running into
issues with automount not failing over when an NFS server goes down. I am wondering if it is a configuration issue, if I am using a version of autofs
that doesn't support failover or if it hasn't been implemented yet.
Signature · 3 lines
Todd Denniston Crane Division, Naval Surface Warfare Center (NSWC Crane) Harnessing the Power of Technology for the Warfighter