hang on existing systems when exporting NFS share to new systems
Subsystems (2)
Hi.
My list of NFS exports has been gradually growing over the years. Right now, for example, my home directories are exported to around 800 hosts. (although only a relatively small subset of those will mount at the same time...). I used to just add hosts to /etc/exports on the file server, and run "exportfs -r", and everything would be fine. New systems would be able to mount everything perfectly, and existing systems would not be affected at all. As the list has grown, I've been noticing a problem. Now, when I run exportfs -r, there is an approximate 7-10 second hang on the systems that have already mounted the share, and then everything returns to normal. This doesn't happen *while* exportfs -r is running, but just after it exits. I figured that maybe exportfs was "unexporting"/re-exporting to hosts that already had the share in use which might have caused the problem, so I tried to manually add/remove hosts thinking that this would only affect those hosts, but it did not. Exporting to one new host still causes the hang on all existing hosts.
Since I have multiple exports to all of the hosts, adding one new host can hang things for a while. I can see that reducing the list of exports, or hosts would reduce the delay. What I am wondering is if there is a better way that I can add hosts without affecting connectivity to existing hosts?
The NFS server itself is pretty powerful -- dual quad core box, lots of memory, many NFS threads, exclusive NFS server, etc... I am running an older RHEL4 release though, so it would have an older kernel/NFS system.
Maybe this issue has been solved in newer releases.
Any feedback would be helpful.
Thanks!
Jason.
Quoted 26 lines · Jason Keltz
Hi.
My list of NFS exports has been gradually growing over the years. Right now, for example, my home directories are exported to around 800 hosts. (although only a relatively small subset of those will mount at the same time...). I used to just add hosts to
/etc/exportson the file server, and run "exportfs -r", and everything would be fine. New systems would be able to mount everything perfectly, and existing systems would not be affected at all. As the list has grown, I've been noticing a problem. Now, when I run exportfs -r, there is an approximate 7-10 second hang on the systems that have already mounted the share, and then everything returns to normal. This doesn't happen *while* exportfs -r is running, but just after it exits. I figured that maybe exportfs was "unexporting"/re-exporting to hosts that already had the share in use which might have caused the problem, so I tried to manually add/remove hosts thinking that this would only affect those hosts, but it did not. Exporting to one new host still causes the hang on all existing hosts.Since I have multiple exports to all of the hosts, adding one new host can hang things for a while. I can see that reducing the list of exports, or hosts would reduce the delay. What I am wondering is if there is a better way that I can add hosts without affecting connectivity to existing hosts?
Quoted 5 lines
The NFS server itself is pretty powerful -- dual quad core box, lots of memory, many NFS threads, exclusive NFS server, etc... I am running an older RHEL4 release though, so it would have an older kernel/NFS system. Maybe this issue has been solved in newer releases.
There have been fixes in this area, though I don't see any that I'm sure would address your problem. If you could test with the latest nfs-utils (ideally, with the latest nfs-utils and kernel) and let us know the result, that would be helpful.
The -t option to rpc.mountd (may need a newer nfs-utils?) may also help.
Also worth filing an RHEL bug.
--b.
Quoted 31 lines · J. Bruce Fields
On Wed, Jul 28, 2010 at 09:44:48AM -0400, Jason Keltz wrote My list of NFS exports has been gradually growing over the years. Right now, for example, my home directories are exported to around 800 hosts. (although only a relatively small subset of those will mount at the same time...). I used to just add hosts to
/etc/exportson the file server, and run "exportfs -r", and everything would be fine. New systems would be able to mount everything perfectly, and existing systems would not be affected at all. As the list has grown, I've been noticing a problem. Now, when I run exportfs -r, there is an approximate 7-10 second hang on the systems that have already mounted the share, and then everything returns to normal. This doesn't happen *while* exportfs -r is running, but just after it exits. I figured that maybe exportfs was "unexporting"/re-exporting to hosts that already had the share in use which might have caused the problem, so I tried to manually add/remove hosts thinking that this would only affect those hosts, but it did not. Exporting to one new host still causes the hang on all existing hosts.Since I have multiple exports to all of the hosts, adding one new host can hang things for a while. I can see that reducing the list of exports, or hosts would reduce the delay. What I am wondering is if there is a better way that I can add hosts without affecting connectivity to existing hosts?
The NFS server itself is pretty powerful -- dual quad core box, lots of memory, many NFS threads, exclusive NFS server, etc... I am running an older RHEL4 release though, so it would have an older kernel/NFS system. Maybe this issue has been solved in newer releases.
Quoted 9 lines
There have been fixes in this area, though I don't see any that I'm sure would address your problem. If you could test with the latest nfs-utils (ideally, with the latest nfs-utils and kernel) and let us know the result, that would be helpful.
The -t option to rpc.mountd (may need a newer nfs-utils?) may also help.
Also worth filing an RHEL bug.
Hi Bruce,
I backported the -t option to RHEL4 by looking at the latest nfs-utils, but it didn't fix the problem. I'm having trouble compiling the latest nfs-utils for RHEL4 because a couple of changed libraries...
What I have learned:
1) whether exportfs -r, or manually add a single host with exportfs, or even remove a host with exportfs -u, the delay to all the clients is the same. The delay doesn't change depending on the share.
2) the delay doesn't happen while exportfs is running. It happens immediately afterwards, and when it does happen, an strace of rpc.mountd shows that rpc.mountd is busy resolving every single hostname in etab.. on one of our NFS servers, this means a total of 13,000 DNS requests... on another system, that's over 30,000 DNS requests (and around a 30 second delay to all shares). Once rpc.mountd stops burdening the DNS, that's exactly when activity on all the shares returns.
3) I've tried to change /etc/exports to use just IP... but exportfs happily switches etab back to using hostnames, and then mountd does all the lookups again...
I suppose that the reason why exportfs doesn't convert etab to just use IPs in the first place is because a name can resolve to multiple IPs... but if I start with a list of IPs in /etc/exports, it would be nice if they just stayed like that in etab, and if mountd could use them as is... what's the point of all the DNS requests? (first to generate etab, then from mountd a second time!)
The only thing I can think to try at this point would be to see if I populated /etc/hosts locally on the file server to see if the timing works better than the DNS requests.
If someone has any suggestions, I'd love to hear them.
Thanks!
Jason.
A message that is not in the archive (sent off-list or not yet received): 4C53855F.5060201-Au2wK/[email protected]
Quoted 71 lines · Jason Keltz
Quoted 35 lines · J. Bruce Fields
On Wed, Jul 28, 2010 at 09:44:48AM -0400, Jason Keltz wrote My list of NFS exports has been gradually growing over the years. Right now, for example, my home directories are exported to around 800 hosts. (although only a relatively small subset of those will mount at the same time...). I used to just add hosts to
/etc/exportson the file server, and run "exportfs -r", and everything would be fine. New systems would be able to mount everything perfectly, and existing systems would not be affected at all. As the list has grown, I've been noticing a problem. Now, when I run exportfs -r, there is an approximate 7-10 second hang on the systems that have already mounted the share, and then everything returns to normal. This doesn't happen *while* exportfs -r is running, but just after it exits. I figured that maybe exportfs was "unexporting"/re-exporting to hosts that already had the share in use which might have caused the problem, so I tried to manually add/remove hosts thinking that this would only affect those hosts, but it did not. Exporting to one new host still causes the hang on all existing hosts.Since I have multiple exports to all of the hosts, adding one new host can hang things for a while. I can see that reducing the list of exports, or hosts would reduce the delay. What I am wondering is if there is a better way that I can add hosts without affecting connectivity to existing hosts?
The NFS server itself is pretty powerful -- dual quad core box, lots of memory, many NFS threads, exclusive NFS server, etc... I am running an older RHEL4 release though, so it would have an older kernel/NFS system. Maybe this issue has been solved in newer releases.
There have been fixes in this area, though I don't see any that I'm sure would address your problem. If you could test with the latest nfs-utils (ideally, with the latest nfs-utils and kernel) and let us know the result, that would be helpful.Quoted 3 lines
The -t option to rpc.mountd (may need a newer nfs-utils?) may also help.
Also worth filing an RHEL bug.
Hi Bruce,
I backported the -t option to RHEL4 by looking at the latest nfs-utils, but it didn't fix the problem.
I'm having trouble compiling the latest nfs-utils for RHEL4 because a couple of changed libraries...What I have learned:
1) whether exportfs -r, or manually add a single host with exportfs, or even remove a host with exportfs -u, the delay to all the clients is the same. The delay doesn't change depending on the share.
2) the delay doesn't happen while exportfs is running. It happens immediately afterwards, and when it does happen, an strace of rpc.mountd shows that rpc.mountd is busy resolving every single hostname in etab.. on one of our NFS servers, this means a total of 13,000 DNS requests... on another system, that's over 30,000 DNS requests (and around a 30 second delay to all shares). Once rpc.mountd stops burdening the DNS, that's exactly when activity on all the shares returns.
3) I've tried to change/etc/exportsto use just IP... but exportfs happily switches etab back to using hostnames, and then mountd does all the lookups again...I suppose that the reason why exportfs doesn't convert etab to just use IPs in the first place is because a name can resolve to multiple IPs... but if I start with a list of IPs in
/etc/exports, it would be nice if they just stayed like that in etab, and if mountd could use them as is... what's the point of all the DNS requests? (first to generate etab, then from mountd a second time!)
Quoted 5 lines
The only thing I can think to try at this point would be to see if I populated
/etc/hostslocally on the file server to see if the timing works better than the DNS requests.If someone has any suggestions, I'd love to hear them.
Did you ever figure out anything more about the problem?
--b.
Quoted 71 lines · J. Bruce Fields
Quoted 70 lines · Jason Keltz
Quoted 35 lines · J. Bruce Fields
On Wed, Jul 28, 2010 at 09:44:48AM -0400, Jason Keltz wrote My list of NFS exports has been gradually growing over the years. Right now, for example, my home directories are exported to around 800 hosts. (although only a relatively small subset of those will mount at the same time...). I used to just add hosts to
/etc/exportson the file server, and run "exportfs -r", and everything would be fine. New systems would be able to mount everything perfectly, and existing systems would not be affected at all. As the list has grown, I've been noticing a problem. Now, when I run exportfs -r, there is an approximate 7-10 second hang on the systems that have already mounted the share, and then everything returns to normal. This doesn't happen *while* exportfs -r is running, but just after it exits. I figured that maybe exportfs was "unexporting"/re-exporting to hosts that already had the share in use which might have caused the problem, so I tried to manually add/remove hosts thinking that this would only affect those hosts, but it did not. Exporting to one new host still causes the hang on all existing hosts.Since I have multiple exports to all of the hosts, adding one new host can hang things for a while. I can see that reducing the list of exports, or hosts would reduce the delay. What I am wondering is if there is a better way that I can add hosts without affecting connectivity to existing hosts?
The NFS server itself is pretty powerful -- dual quad core box, lots of memory, many NFS threads, exclusive NFS server, etc... I am running an older RHEL4 release though, so it would have an older kernel/NFS system. Maybe this issue has been solved in newer releases.
There have been fixes in this area, though I don't see any that I'm sure would address your problem. If you could test with the latest nfs-utils (ideally, with the latest nfs-utils and kernel) and let us know the result, that would be helpful.Quoted 3 lines
The -t option to rpc.mountd (may need a newer nfs-utils?) may also help.
Also worth filing an RHEL bug.
Hi Bruce,
I backported the -t option to RHEL4 by looking at the latest nfs-utils, but it didn't fix the problem.
I'm having trouble compiling the latest nfs-utils for RHEL4 because a couple of changed libraries...What I have learned:
1) whether exportfs -r, or manually add a single host with exportfs, or even remove a host with exportfs -u, the delay to all the clients is the same. The delay doesn't change depending on the share.
2) the delay doesn't happen while exportfs is running. It happens immediately afterwards, and when it does happen, an strace of rpc.mountd shows that rpc.mountd is busy resolving every single hostname in etab.. on one of our NFS servers, this means a total of 13,000 DNS requests... on another system, that's over 30,000 DNS requests (and around a 30 second delay to all shares). Once rpc.mountd stops burdening the DNS, that's exactly when activity on all the shares returns.
3) I've tried to change/etc/exportsto use just IP... but exportfs happily switches etab back to using hostnames, and then mountd does all the lookups again...I suppose that the reason why exportfs doesn't convert etab to just use IPs in the first place is because a name can resolve to multiple IPs... but if I start with a list of IPs in
/etc/exports, it would be nice if they just stayed like that in etab, and if mountd could use them as is... what's the point of all the DNS requests? (first to generate etab, then from mountd a second time!)
Quoted 7 lines
Quoted 5 lines
The only thing I can think to try at this point would be to see if I populated
/etc/hostslocally on the file server to see if the timing works better than the DNS requests.If someone has any suggestions, I'd love to hear them.
Did you ever figure out anything more about the problem?
Hi Bruce,
Actually, since I was sharing out over a private network, I did not need to include every address in the list, and was able to use an IP range. Now, I share out to pretty much the same host list, but export is instantaneous. I still believe that I *should* have been able to export to the large number of hosts without the tremendous delay, but I was not able to solve the problem with my current (RHEL4) installation. We will upgrade to RHEL6 when it's available, so I might try again with that just for fun.
Jason.
A message that is not in the archive (sent off-list or not yet received): 4CA9D4BC.1060201-Au2wK/[email protected]
Quoted 8 lines · Jason Keltz
Actually, since I was sharing out over a private network, I did not need to include every address in the list, and was able to use an IP range. Now, I share out to pretty much the same host list, but export is instantaneous. I still believe that I *should* have been able to export to the large number of hosts without the tremendous delay, but I was not able to solve the problem with my current (RHEL4) installation. We will upgrade to RHEL6 when it's available, so I might try again with that just for fun.
OK, thanks--let us know of any results you get, positive or negative.
--b.