Replicated hosts in autofs v4.
I've previously posted about this problem, but I still haven't found an answer, although I think I better understand what's happening.
We have an automount map called "auto_group" that looks like this:
icdes ausfiler,minnfiler:/vol/vol0/data/group/&
ausfiler lives in Texas, minnfiler lives in Minnesota. Between ausfiler and minnfiler lies a T1 line.
On the MInnesota office network, from a RedHat Enterprise 3.0 Update 3 client (using autofs-4.1.3-12), cd'ing into /group/icdes mounts ausfiler:/vol/vol0/data/group/icdes. No matter what...it always mounts ausfiler. The problem is that ausfiler is on the other end of a T1 from the client, whereas minnfiler is on the same subnet, on the same network switch.
I filed a support ticket with RedHat Global support, and an engineer there confirmed that the client does an RPC call with a .1 second timeout. However, the client's behavior indicates to me that the comparison being made in the reponse times between the replicated mount servers isn't fine-grained enough to make the correct choice. A coworker made the comment that our T1 line could be "too robust", making the server farther away appear to be good enough for the client's purposes.
Solaris doesn't seem to have any issues with doing this properly. A Solaris client in Minnesota will always mount the "closest" server in a replicated host map, which is important because we're trying to use RHEL3 to replace some of our Solaris infrastructure. The RH Global support engineer suggested reversing the order of the servers in the map, however that would cause clients in Texas to exhibit the same behavior that clients in Minneapolis are.
Can somebody explain to me whether I'm not the right track in understanding the problem, with regards to the comparison not being fine-grained enough for the client to make the correct choice? Or, might there be something else going on? I previously had assumed that the behavior I was seeing was related to the bug that caused the client to always mount the first-listed server, but I no longer think that's the case.
Thanks,
-- jeremy
Quoted 10 lines · Jeremy Rosengren
I've previously posted about this problem, but I still haven't found an answer, although I think I better understand what's happening.
We have an automount map called "
auto_group" that looks like this:icdes ausfiler,minnfiler:/vol/vol0/data/group/&
ausfiler lives in Texas, minnfiler lives in Minnesota. Between ausfiler and minnfiler lies a T1 line.
Quoted 6 lines
On the MInnesota office network, from a RedHat Enterprise 3.0 Update 3 client (using autofs-4.1.3-12), cd'ing into /group/icdes mounts ausfiler:/vol/vol0/data/group/icdes. No matter what...it always mounts ausfiler. The problem is that ausfiler is on the other end of a T1 from the client, whereas minnfiler is on the same subnet, on the same network switch.
Your config does not let the system know that you prefer minnfiler over ausfiler if both are available.
Quoted 10 lines
I filed a support ticket with RedHat Global support, and an engineer there confirmed that the client does an RPC call with a .1 second timeout. However, the client's behavior indicates to me that the comparison being made in the reponse times between the replicated mount servers isn't fine-grained enough to make the correct choice. A coworker made the comment that our T1 line could be "too robust", making the server farther away appear to be good enough for the client's purposes.
Solaris doesn't seem to have any issues with doing this properly. A
Prolly Solaris does not use first server from the set of those which answered, it uses server which _answers first_.
--
vda
Ian Kent
maintainerQuoted 34 lines · Jeremy Rosengren
I've previously posted about this problem, but I still haven't found an answer, although I think I better understand what's happening.
We have an automount map called "
auto_group" that looks like this:icdes ausfiler,minnfiler:/vol/vol0/data/group/&
ausfiler lives in Texas, minnfiler lives in Minnesota. Between ausfiler and minnfiler lies a T1 line.
On the MInnesota office network, from a RedHat Enterprise 3.0 Update 3 client (using autofs-4.1.3-12), cd'ing into /group/icdes mounts ausfiler:/vol/vol0/data/group/icdes. No matter what...it always mounts ausfiler. The problem is that ausfiler is on the other end of a T1 from the client, whereas minnfiler is on the same subnet, on the same network switch.
I filed a support ticket with RedHat Global support, and an engineer there confirmed that the client does an RPC call with a .1 second timeout. However, the client's behavior indicates to me that the comparison being made in the reponse times between the replicated mount servers isn't fine-grained enough to make the correct choice. A coworker made the comment that our T1 line could be "too robust", making the server farther away appear to be good enough for the client's purposes.
Solaris doesn't seem to have any issues with doing this properly. A Solaris client in Minnesota will always mount the "closest" server in a replicated host map, which is important because we're trying to use RHEL3 to replace some of our Solaris infrastructure. The RH Global support engineer suggested reversing the order of the servers in the map, however that would cause clients in Texas to exhibit the same behavior that clients in Minneapolis are.
Quoted 7 lines
Can somebody explain to me whether I'm not the right track in understanding the problem, with regards to the comparison not being fine-grained enough for the client to make the correct choice? Or, might there be something else going on? I previously had assumed that the behavior I was seeing was related to the bug that caused the client to always mount the first-listed server, but I no longer think that's the case.
Yes. This behavior is not what I'm after. The Solaris case is what I want it to do.
Please bear in mind that this is still fairly new code and a number of problems have been identified and solved so far.
I'll have a look at the state of the patches that Jeff has in release 12 and the code and see what I can find out.
You'll need to bear with me as I'm way busy atm.
Ian
==> Regarding Re: [autofs] Replicated hosts in autofs v4.; Denis Vlasenko <[email protected]> adds:
Quoted 5 lines · Jeremy Rosengren
Quoted 5 lines
I've previously posted about this problem, but I still haven't found an answer, although I think I better understand what's happening.
We have an automount map called "
auto_group" that looks like this:
Quoted 7 lines
Quoted 7 lines
icdes ausfiler,minnfiler:/vol/vol0/data/group/&
ausfiler lives in Texas, minnfiler lives in Minnesota. Between ausfiler and minnfiler lies a T1 line.
On the MInnesota office network, from a RedHat Enterprise 3.0 Update 3 client (using autofs-4.1.3-12), cd'ing into /group/icdes mounts
ausfiler:/vol/vol0/data/group/icdes. No matter what...it always
Quoted 3 lines
Quoted 3 lines
mounts ausfiler. The problem is that ausfiler is on the other end of a T1 from the client, whereas minnfiler is on the same subnet, on the same network switch.
vda> Your config does not let the system know that you prefer minnfiler vda> over ausfiler if both are available.
Quoted 10 lines
Quoted 10 lines
I filed a support ticket with RedHat Global support, and an engineer there confirmed that the client does an RPC call with a .1 second timeout. However, the client's behavior indicates to me that the comparison being made in the reponse times between the replicated mount servers isn't fine-grained enough to make the correct choice. A coworker made the comment that our T1 line could be "too robust", making the server farther away appear to be good enough for the client's purposes.
Solaris doesn't seem to have any issues with doing this properly. A
vda> Prolly Solaris does not use first server from the set of those which vda> answered, it uses server which _answers first_. -- vda
Not true. Solaris behaves quite as you would expect in these cases:
Server proximity takes priority in the selection process.
In the example above, if the server delta is on the same network segment as the client, but the others are on dif- ferent network segments, then delta will be selected; the weighting value is ignored. The weighting has effect only when selecting between servers with the same network proxim- ity.
Above taken from the Solaris automount(1m) man page.
-Jeff
Ian Kent
maintainerQuoted 10 lines · Jeff Moyer
Not true. Solaris behaves quite as you would expect in these cases:
Server proximity takes priority in the selection process.
In the example above, if the server delta is on the same network segment as the client, but the others are on dif- ferent network segments, then delta will be selected; the weighting value is ignored. The weighting has effect only when selecting between servers with the same network proxim- ity.
Mike, can we define "proximity" and "same network segment" as used in the Solaris automounter please (if that's possible).
Ian
Quoted 5 lines · Ian Kent
Quoted 1 line · Jeff Moyer
Server proximity takes priority in the selection process.
Mike, can we define "proximity" and "same network segment" as used in the Solaris automounter please (if that's possible).
The desired behavior (and I think this is what our Sun boxes actually do) is that if the client has an interface such that (clientIP & netmask) == (serverIP & netmask) (using the client's same netmask for both), then that server is preferred because the traffic goes direct, not through a router. The client might have multiple interfaces.
If there are multiple servers and one is down, another can be tried after a brief timeout. But what really interests me is hot failover: you already mounted server "A", which dies. The automounter is magically aware of this, and mounts server "B". Can of worms: Processes with open files (readonly of course) in the dead filesystem, you would like to transfer them transparently to presumably identical files from the other server. In a map row with multiple servers, all share the same mount point, so how do you dispose of the corpse so you can mount the other server on the same-named mount point? None of this will be easy.
James F. Carter Voice 310 825 2897 FAX 310 206 6673
UCLA-Mathnet; 6115 MSA; 405 Hilgard Ave.; Los Angeles, CA, USA 90095-1555
Email: [email protected] http://www.math.ucla.edu/~jimc (q.v. for PGP key)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Quoted 10 lines · Jim Carter
Quoted 6 lines · Ian Kent
Quoted 1 line · Jeff Moyer
Server proximity takes priority in the selection process.
Mike, can we define "proximity" and "same network segment" as used in the Solaris automounter please (if that's possible).
Quoted 6 lines
The desired behavior (and I think this is what our Sun boxes actually do) is that if the client has an interface such that (clientIP & netmask) == (serverIP & netmask) (using the client's same netmask for both), then that server is preferred because the traffic goes direct, not through a router. The client might have multiple interfaces.
I believe this is the right interpretation of the 'weight override' bit that Jeff posted from the Solaris automount(1m) manpage.
However, I believe that algorithm uses some sort of traceroute/rpcping to figure out distances and weights are used to prioritize given equal distances. I'll ask around to see what the actual method used is. (a tcpdump of this may prove useful ;)
Quoted 10 lines
If there are multiple servers and one is down, another can be tried after a brief timeout. But what really interests me is hot failover: you already mounted server "A", which dies. The automounter is magically aware of this, and mounts server "B". Can of worms: Processes with open files (readonly of course) in the dead filesystem, you would like to transfer them transparently to presumably identical files from the other server. In a map row with multiple servers, all share the same mount point, so how do you dispose of the corpse so you can mount the other server on the same-named mount point? None of this will be easy.
Note, this 'hot failover' functionality is only prescribed to work when the filesystem is mounted _read only_. I believe this is handled by the NFS subsystem proper as the stateless nature of NFS2/3 would make this feasible for the read-only case. Again, I'll ask around to see how this is handled and will post back when I get an answer.
- --
Mike Waychison
Sun Microsystems, Inc.
1 (650) 352-5299 voice
1 (416) 202-8336 voice
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
NOTICE: The opinions expressed in this email are held by me, and may not represent the views of Sun Microsystems, Inc.
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFBa56ddQs4kOxk3/MRApIMAJ47NUyh7yoZucN8YKXvtu4AnR9JTQCgl7Ny 99axGIQugUaU3fBuggDb0iQ=
=0gDc
-----END PGP SIGNATURE-----
[email protected]
maintainerQuoted 21 lines · Mike Waychison
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1Quoted 10 lines · Jim Carter
Quoted 6 lines · Ian Kent
Quoted 1 line · Jeff Moyer
Server proximity takes priority in the selection process.
Mike, can we define "proximity" and "same network segment" as used in the Solaris automounter please (if that's possible).
Quoted 6 lines
The desired behavior (and I think this is what our Sun boxes actually do) is that if the client has an interface such that (clientIP & netmask) == (serverIP & netmask) (using the client's same netmask for both), then that server is preferred because the traffic goes direct, not through a router. The client might have multiple interfaces.
Quoted 7 lines
I believe this is the right interpretation of the 'weight override' bit that Jeff posted from the Solaris automount(1m) manpage.
However, I believe that algorithm uses some sort of traceroute/rpcping to figure out distances and weights are used to prioritize given equal distances. I'll ask around to see what the actual method used is. (a tcpdump of this may prove useful ;)
OK. The traceroute idea is good but we have had people with port blocking so this would have to be quite smart.
Ian
[email protected]
maintainerQuoted 4 lines · Jim Carter
Quoted 3 lines · Ian Kent
Quoted 1 line · Jeff Moyer
Server proximity takes priority in the selection process.
Quoted 9 lines
Quoted 2 lines
Mike, can we define "proximity" and "same network segment" as used in the Solaris automounter please (if that's possible).
The desired behavior (and I think this is what our Sun boxes actually do) is that if the client has an interface such that (clientIP & netmask) == (serverIP & netmask) (using the client's same netmask for both), then that server is preferred because the traffic goes direct, not through a router. The client might have multiple interfaces.
Yes, I thought of that but it occured to me that we may not want to use a server on the the same subnet. For example, on a multiple subnet gigabit network we might want to find the least loaded server. So this approach might be unreliable.
Quoted 9 lines
If there are multiple servers and one is down, another can be tried after a brief timeout. But what really interests me is hot failover: you already mounted server "A", which dies. The automounter is magically aware of this, and mounts server "B". Can of worms: Processes with open files (readonly of course) in the dead filesystem, you would like to transfer them transparently to presumably identical files from the other server. In a map row with multiple servers, all share the same mount point, so how do you dispose of the corpse so you can mount the other server on the same-named mount point? None of this will be easy.
Mmmm. NFS kernel land I think.
We'll have to be extra polite to the NFS gents!
Ian