latest autofs from centos 4
it looks like some recent changes to both autofs v4 and v5 has caused the siogcifconf buffer to be only 128 bytes. this means that on our servers, it can't correctly match a local address since that is only enough room for 3 addresses and we're in the middle of a renumbering so we have 5-10 addresses per machine. this was working fine until i just updated to centos 4.7. from looking at the code, it looks like someone created a MAX_IF_BUF #define at 2048, but didn't use it for the ioctl call. is that a mistake?
i'll try a patched rpm locally and make sure this is the source of my woes.
Ian Kent
maintainerQuoted 11 lines · Joe Pruett
it looks like some recent changes to both autofs v4 and v5 has caused the siogcifconf buffer to be only 128 bytes. this means that on our servers, it can't correctly match a local address since that is only enough room for 3 addresses and we're in the middle of a renumbering so we have 5-10 addresses per machine. this was working fine until i just updated to centos 4.7. from looking at the code, it looks like someone created a
MAX_IF_BUF#define at 2048, but didn't use it for the ioctl call. is that a mistake?i'll try a patched rpm locally and make sure this is the source of my woes.
Yeah, known issue.
It's resolved in RHEL-5.3.
Not sure if this will apply to your package source but give it a try.
---
autofs-5.0.3 - fix interface config buffer size
From: Ian Kent <[email protected]>
When getting the interface configuration information autofs uses a fixed size buffer for the interface information. If there are many interfaces this causes the check to fail.
---
modules/replicated.c | 51 ++++++++++++++++++++++++++++++++++++++++++-------- 1 files changed, 43 insertions(+), 8 deletions(-)
modules/replicated.c +43 -8
@@ -95,6 +95,41 @@ void seed_random(void)return ; } +static int alloc_ifreq (struct ifconf *ifc,int sock) +{ +int ret, lastlen =0 , len =MAX_IFC_BUF ; +char err_buf[MAX_ERR_BUF ], *buf; + +while (1 ) { + buf =malloc (len); +if (!buf) { +char *estr =strerror_r (errno, err_buf,MAX_ERR_BUF ); +logerr ("malloc: %s" , estr); +return 0 ; + } + + ifc->ifc_len = len; + ifc->ifc_req = (struct ifreq *) buf; + + ret =ioctl (sock,SIOCGIFCONF , ifc); +if (ret ==-1 ) { +char *estr =strerror_r (errno, err_buf,MAX_ERR_BUF ); +logerr ("ioctl: %s" , estr); +free (buf); +return 0 ; + } + +if (ifc->ifc_len == lastlen) +break ; + + lastlen = ifc->ifc_len; + len +=MAX_IFC_BUF ; +free (buf); + } + +return 1 ; +} +static unsigned int get_proximity (const char *host_addr,int addr_len) {struct sockaddr_in *msk_addr, *if_addr;
@@ -122,12 +157,7 @@ static unsigned int get_proximity(const char *host_addr, int addr_len)fcntl (sock,F_SETFD , cl_flags); } - ifc.ifc_len =sizeof (buf); - ifc.ifc_req = (struct ifreq *) buf; - ret =ioctl (sock,SIOCGIFCONF , &ifc); -if (ret ==-1 ) { -char *estr =strerror_r (errno, buf,MAX_ERR_BUF ); -logerr ("ioctl: %s" , estr); +if (!alloc_ifreq (&ifc, sock)) {close (sock);return PROXIMITY_ERROR ; }
@@ -138,7 +168,7 @@ static unsigned int get_proximity(const char *host_addr, int addr_len) i =0 ; ptr = (char *) &ifc.ifc_buf[0 ]; -while (ptr < buf + ifc.ifc_len) { +while (ptr < (char *) ifc.ifc_req + ifc.ifc_len) { ifr = (struct ifreq *) ptr;switch (ifr->ifr_addr.sa_family) {
@@ -147,6 +177,7 @@ static unsigned int get_proximity(const char *host_addr, int addr_len) ret =memcmp (&if_addr->sin_addr, hst_addr, addr_len);if (!ret) {close (sock); +free (ifc.ifc_req);return PROXIMITY_LOCAL ; }break ;
@@ -162,7 +193,7 @@ static unsigned int get_proximity(const char *host_addr, int addr_len) i =0 ; ptr = (char *) &ifc.ifc_buf[0 ]; -while (ptr < buf + ifc.ifc_len) { +while (ptr < (char *) ifc.ifc_req + ifc.ifc_len) { ifr = (struct ifreq *) ptr;switch (ifr->ifr_addr.sa_family) {
@@ -178,6 +209,7 @@ static unsigned int get_proximity(const char *host_addr, int addr_len)char *estr =strerror_r (errno, buf,MAX_ERR_BUF );logerr ("ioctl: %s" , estr);close (sock); +free (ifc.ifc_req);return PROXIMITY_ERROR ; }
@@ -186,6 +218,7 @@ static unsigned int get_proximity(const char *host_addr, int addr_len)if ((ia & mask) == (ha & mask)) {close (sock); +free (ifc.ifc_req);return PROXIMITY_SUBNET ; }
@@ -208,6 +241,7 @@ static unsigned int get_proximity(const char *host_addr, int addr_len)if ((ia & mask) == (ha & mask)) {close (sock); +free (ifc.ifc_req);return PROXIMITY_NET ; }break ;
@@ -221,6 +255,7 @@ static unsigned int get_proximity(const char *host_addr, int addr_len) }close (sock); +free (ifc.ifc_req);return PROXIMITY_OTHER ; }
Quoted 10 lines · Ian Kent
Quoted 9 lines · Joe Pruett
it looks like some recent changes to both autofs v4 and v5 has caused the siogcifconf buffer to be only 128 bytes. this means that on our servers, it can't correctly match a local address since that is only enough room for 3 addresses and we're in the middle of a renumbering so we have 5-10 addresses per machine. this was working fine until i just updated to centos 4.7. from looking at the code, it looks like someone created a
MAX_IF_BUF#define at 2048, but didn't use it for the ioctl call. is that a mistake?
Quoted 5 lines
Quoted 2 lines
i'll try a patched rpm locally and make sure this is the source of my woes.
Yeah, known issue.
It's resolved in RHEL-5.3.
it seems like it got worse with the latest 4.7 stuff. maybe i just didn't notice machines talking to themselves over nfs previously. has it always been 128 bytes?
i'm concerned about the patch for my web hosting machine. right now it has over 1024 interfaces. the amount of time all that malloc/free is going to take for every mount seems pretty expensive. would any kind of caching make sense?
Quoted 15 lines · Joe Pruett
Quoted 13 lines · Ian Kent
Quoted 11 lines · Joe Pruett
it looks like some recent changes to both autofs v4 and v5 has caused the siogcifconf buffer to be only 128 bytes. this means that on our servers, it can't correctly match a local address since that is only enough room for 3 addresses and we're in the middle of a renumbering so we have 5-10 addresses per machine. this was working fine until i just updated to centos 4.7. from looking at the code, it looks like someone created a
MAX_IF_BUF#define at 2048, but didn't use it for the ioctl call. is that a mistake?i'll try a patched rpm locally and make sure this is the source of my woes.
Quoted 6 lines
Quoted 2 lines
Yeah, known issue.
It's resolved in RHEL-5.3.it seems like it got worse with the latest 4.7 stuff. maybe i just didn't notice machines talking to themselves over nfs previously. has it always been 128 bytes?
It may have regressed, if that was the update that pulled in the v5 version of the replicated server handling code.
Quoted 4 lines
i'm concerned about the patch for my web hosting machine. right now it has over 1024 interfaces. the amount of time all that malloc/free is going to take for every mount seems pretty expensive. would any kind of caching make sense?
There's no doubt that we could be smarter about this. It may be enough to make last_len in alloc_ifreq a static. Ian, what do you think about doing that?
But honestly, malloc/free is pretty cheap. I doubt you'll see performance problems due to that.
Cheers,
Jeff
Ian Kent
maintainerQuoted 17 lines · Jeff Moyer
Quoted 15 lines · Joe Pruett
Quoted 13 lines · Ian Kent
Quoted 11 lines · Joe Pruett
it looks like some recent changes to both autofs v4 and v5 has caused the siogcifconf buffer to be only 128 bytes. this means that on our servers, it can't correctly match a local address since that is only enough room for 3 addresses and we're in the middle of a renumbering so we have 5-10 addresses per machine. this was working fine until i just updated to centos 4.7. from looking at the code, it looks like someone created a
MAX_IF_BUF#define at 2048, but didn't use it for the ioctl call. is that a mistake?i'll try a patched rpm locally and make sure this is the source of my woes.
Quoted 6 lines
Quoted 6 lines
Quoted 2 lines
Yeah, known issue.
It's resolved in RHEL-5.3.it seems like it got worse with the latest 4.7 stuff. maybe i just didn't notice machines talking to themselves over nfs previously. has it always been 128 bytes?
Oops, yes, forgot that we have this in v4 too now.
Quoted 4 lines
It may have regressed, if that was the update that pulled in the v5 version of the replicated server handling code.
Quoted 8 lines
Quoted 4 lines
i'm concerned about the patch for my web hosting machine. right now it has over 1024 interfaces. the amount of time all that malloc/free is going to take for every mount seems pretty expensive. would any kind of caching make sense?
There's no doubt that we could be smarter about this. It may be enough to make
last_leninalloc_ifreqa static. Ian, what do you think about doing that?
Ahhh .. that's a good idea. I'll think about that.
Quoted 7 lines
But honestly, malloc/free is pretty cheap. I doubt you'll see performance problems due to that.
Cheers,
Jeff
Ian Kent
maintainerQuoted 15 lines · Joe Pruett
Quoted 13 lines · Ian Kent
Quoted 11 lines · Joe Pruett
it looks like some recent changes to both autofs v4 and v5 has caused the siogcifconf buffer to be only 128 bytes. this means that on our servers, it can't correctly match a local address since that is only enough room for 3 addresses and we're in the middle of a renumbering so we have 5-10 addresses per machine. this was working fine until i just updated to centos 4.7. from looking at the code, it looks like someone created a
MAX_IF_BUF#define at 2048, but didn't use it for the ioctl call. is that a mistake?i'll try a patched rpm locally and make sure this is the source of my woes.
Quoted 6 lines
Quoted 2 lines
Yeah, known issue.
It's resolved in RHEL-5.3.it seems like it got worse with the latest 4.7 stuff. maybe i just didn't notice machines talking to themselves over nfs previously. has it always been 128 bytes?
Only the autofs5 package in 4.7 would be affected by this. Version 5 has had this mistake from the beginning.
Quoted 5 lines
i'm concerned about the patch for my web hosting machine. right now it has over 1024 interfaces. the amount of time all that malloc/free is going to take for every mount seems pretty expensive. would any kind of caching make sense?
Yeah, that's not good but I have to do something to fix it.
We could double the buffer size each time we allocate it but that would lead to considerable waste. Or maybe increasing the increment to 4k would be better. The current 2k will lead to 10 or 11 allocations.
Ian
i was to onervous to try the real patch for now. so i just quickly hacked in another buffer using the MAX_IFC_LEN. so far it seems to be happy. i guess i'll find out about the new malloc setup when i go to 5.3.
modules/replicated.c 2008-09-17 08:01:58.000000000 -0700 renamed from modules/replicated.c 2008-09-16 12:08:03.000000000 -0700 +5 -5
@@ -99,7 +99,7 @@ { struct sockaddr_in *msk_addr, *if_addr; struct in_addr *hst_addr; - char tmp[20], buf[MAX_ERR_BUF], *ptr; + char tmp[20], buf[MAX_ERR_BUF], ibuf[MAX_IFC_BUF], *ptr; struct ifconf ifc; struct ifreq *ifr, nmptr; int sock, cl_flags, ret, i;
@@ -122,8 +122,8 @@ fcntl(sock, F_SETFD, cl_flags); } - ifc.ifc_len = sizeof(buf); - ifc.ifc_req = (struct ifreq *) buf; + ifc.ifc_len = sizeof(ibuf); + ifc.ifc_req = (struct ifreq *) ibuf; ret = ioctl(sock, SIOCGIFCONF, &ifc); if (ret == -1) { char *estr = strerror_r(errno, buf, MAX_ERR_BUF);
@@ -138,7 +138,7 @@ i = 0; ptr = (char *) &ifc.ifc_buf[0]; - while (ptr < buf + ifc.ifc_len) { + while (ptr < ibuf + ifc.ifc_len) { ifr = (struct ifreq *) ptr; switch (ifr->ifr_addr.sa_family) {
@@ -162,7 +162,7 @@ i = 0; ptr = (char *) &ifc.ifc_buf[0]; - while (ptr < buf + ifc.ifc_len) { + while (ptr < ibuf + ifc.ifc_len) { ifr = (struct ifreq *) ptr; switch (ifr->ifr_addr.sa_family) {
Ian Kent
maintainerQuoted 4 lines · Joe Pruett
i was to onervous to try the real patch for now. so i just quickly hacked in another buffer using the
MAX_IFC_LEN. so far it seems to be happy. i guess i'll find out about the new malloc setup when i go to 5.3.
But that shouldn't work because the ifc struct is 40 chars so MAX_IFC_BUF will allow 51 such buffers, not nearly close enough to 1000.
Also the patch starts with an alloc of MAX_IFC_BUF and add MAX_IFC_BUF each time it find it not large enough.
The static buffer length variable is the way to go I think.
Quoted 41 lines
diff -ru
autofs-5.0.1/modules/replicated.cautofs-new/modules/replicated.cmodules/replicated.c 2008-09-17 08:01:58.000000000 -0700 renamed from modules/replicated.c 2008-09-16 12:08:03.000000000 -0700 +5
-5@@ -99,7 +99,7 @@ { struct sockaddr_in *msk_addr, *if_addr; struct in_addr *hst_addr; - char tmp[20], buf[MAX_ERR_BUF], *ptr; + char tmp[20], buf[MAX_ERR_BUF], ibuf[MAX_IFC_BUF], *ptr; struct ifconf ifc; struct ifreq *ifr, nmptr; int sock, cl_flags, ret, i;@@ -122,8 +122,8 @@ fcntl(sock, F_SETFD, cl_flags); } - ifc.ifc_len = sizeof(buf); - ifc.ifc_req = (struct ifreq *) buf; + ifc.ifc_len = sizeof(ibuf); + ifc.ifc_req = (struct ifreq *) ibuf; ret = ioctl(sock, SIOCGIFCONF, &ifc); if (ret == -1) { char *estr = strerror_r(errno, buf, MAX_ERR_BUF);@@ -138,7 +138,7 @@ i = 0; ptr = (char *) &ifc.ifc_buf[0]; - while (ptr < buf + ifc.ifc_len) { + while (ptr < ibuf + ifc.ifc_len) { ifr = (struct ifreq *) ptr; switch (ifr->ifr_addr.sa_family) {@@ -162,7 +162,7 @@ i = 0; ptr = (char *) &ifc.ifc_buf[0]; - while (ptr < buf + ifc.ifc_len) { + while (ptr < ibuf + ifc.ifc_len) { ifr = (struct ifreq *) ptr; switch (ifr->ifr_addr.sa_family) {
Ian Kent
maintainerQuoted 3 lines · Joe Pruett
i was to onervous to try the real patch for now. so i just quickly hacked in another buffer using the
MAX_IFC_LEN. so far it seems to be happy. i guess i'll find out about the new malloc setup when i go to 5.3.
Don't get me wrong, I think your caution is a sensible and responsible approach to an unknown patch. I understand this only too well.
Quoted 42 lines
diff -ru
autofs-5.0.1/modules/replicated.cautofs-new/modules/replicated.cmodules/replicated.c 2008-09-17 08:01:58.000000000 -0700 renamed from modules/replicated.c 2008-09-16 12:08:03.000000000 -0700 +5
-5@@ -99,7 +99,7 @@ { struct sockaddr_in *msk_addr, *if_addr; struct in_addr *hst_addr; - char tmp[20], buf[MAX_ERR_BUF], *ptr; + char tmp[20], buf[MAX_ERR_BUF], ibuf[MAX_IFC_BUF], *ptr; struct ifconf ifc; struct ifreq *ifr, nmptr; int sock, cl_flags, ret, i;@@ -122,8 +122,8 @@ fcntl(sock, F_SETFD, cl_flags); } - ifc.ifc_len = sizeof(buf); - ifc.ifc_req = (struct ifreq *) buf; + ifc.ifc_len = sizeof(ibuf); + ifc.ifc_req = (struct ifreq *) ibuf; ret = ioctl(sock, SIOCGIFCONF, &ifc); if (ret == -1) { char *estr = strerror_r(errno, buf, MAX_ERR_BUF);@@ -138,7 +138,7 @@ i = 0; ptr = (char *) &ifc.ifc_buf[0]; - while (ptr < buf + ifc.ifc_len) { + while (ptr < ibuf + ifc.ifc_len) { ifr = (struct ifreq *) ptr; switch (ifr->ifr_addr.sa_family) {@@ -162,7 +162,7 @@ i = 0; ptr = (char *) &ifc.ifc_buf[0]; - while (ptr < buf + ifc.ifc_len) { + while (ptr < ibuf + ifc.ifc_len) { ifr = (struct ifreq *) ptr; switch (ifr->ifr_addr.sa_family) {
Quoted 6 lines
Quoted 3 lines
i was to onervous to try the real patch for now. so i just quickly hacked in another buffer using the
MAX_IFC_LEN. so far it seems to be happy. i guess i'll find out about the new malloc setup when i go to 5.3.Don't get me wrong, I think your caution is a sensible and responsible approach to an unknown patch. I understand this only too well.
on my server with 1000 ips, the ip i care about is near the front, so 2048 is plenty big enough to find it.
i am still having some kind of issue where at boot time, it can't seem to find a server or talk to mountd or something like that and then won't try again. it looks like maybe this is the negative caching feature? is that new as well?
Ian Kent
maintainerQuoted 9 lines · Joe Pruett
Quoted 6 lines
Quoted 3 lines
i was to onervous to try the real patch for now. so i just quickly hacked in another buffer using the
MAX_IFC_LEN. so far it seems to be happy. i guess i'll find out about the new malloc setup when i go to 5.3.Don't get me wrong, I think your caution is a sensible and responsible approach to an unknown patch. I understand this only too well.
on my server with 1000 ips, the ip i care about is near the front, so 2048 is plenty big enough to find it.
Hehe, but the ioctl will fail with that many interfaces so you'll still have the same problem. You would need to make the buffer larger than 42k for this case.
The patch I sent was for version 5 but should be the same for recent versions of RHEL-4 v4. It may need some work.
Quoted 5 lines
i am still having some kind of issue where at boot time, it can't seem to find a server or talk to mountd or something like that and then won't try again. it looks like maybe this is the negative caching feature? is that new as well?
Jeff is more familiar with the RHEL-4 v4 code than I am, I'll have to leave this to him. In any case we'd need to see a debug log.
Ian
Quoted 8 lines · Ian Kent
Quoted 4 lines · Joe Pruett
i am still having some kind of issue where at boot time, it can't seem to find a server or talk to mountd or something like that and then won't try again. it looks like maybe this is the negative caching feature? is that new as well?
Jeff is more familiar with the RHEL-4 v4 code than I am, I'll have to leave this to him. In any case we'd need to see a debug log.
Yep, gather a debug log and include the other pertinent information from:
http://people.redhat.com/~jmoyer/
Thanks,
Jeff
Quoted 3 lines
Yep, gather a debug log and include the other pertinent information from:
http://people.redhat.com/~jmoyer/
well, it's a production box and only seems to show the problem at boot time. from looking at the logs, i think this is the underlying culprit:
Sep 17 09:18:08 hyperion kernel: RPC: Can't bind to reserved port (98). Sep 17 09:18:08 hyperion kernel: RPC: can't bind to reserved port. Sep 17 09:18:08 hyperion kernel: RPC: error 5 connecting to server jupiter.spiretech.com Sep 17 09:18:08 hyperion kernel: RPC: Can't bind to reserved port (98). Sep 17 09:18:08 hyperion kernel: RPC: can't bind to reserved port. Sep 17 09:18:08 hyperion kernel: RPC: error 5 connecting to server jupiter.spiretech.com
that is from the client trying to talk to the server.
Quoted 8 lines · Joe Pruett
Quoted 3 lines
Yep, gather a debug log and include the other pertinent information from:
http://people.redhat.com/~jmoyer/well, it's a production box and only seems to show the problem at boot time. from looking at the logs, i think this is the underlying culprit:
Quoted 8 lines
Sep 17 09:18:08 hyperion kernel: RPC: Can't bind to reserved port (98). Sep 17 09:18:08 hyperion kernel: RPC: can't bind to reserved port. Sep 17 09:18:08 hyperion kernel: RPC: error 5 connecting to server jupiter.spiretech.com Sep 17 09:18:08 hyperion kernel: RPC: Can't bind to reserved port (98). Sep 17 09:18:08 hyperion kernel: RPC: can't bind to reserved port. Sep 17 09:18:08 hyperion kernel: RPC: error 5 connecting to server jupiter.spiretech.comthat is from the client trying to talk to the server.
That's strange. Errno 98 is EADDRINUSE. So, this means that you cycled through the entire port range and all of the ports were bound. Do you do a lot of mounts upon boot? Can you try tweaking sunrpc.min_resvport and sunrpc.max_resvport? Max can't be more than 1023. Try pushing min down to 500 or so.
Cheers,
Jeff
Quoted 11 lines
Quoted 10 lines
well, it's a production box and only seems to show the problem at boot time. from looking at the logs, i think this is the underlying culprit:
Sep 17 09:18:08 hyperion kernel: RPC: Can't bind to reserved port (98). Sep 17 09:18:08 hyperion kernel: RPC: can't bind to reserved port. Sep 17 09:18:08 hyperion kernel: RPC: error 5 connecting to server jupiter.spiretech.com Sep 17 09:18:08 hyperion kernel: RPC: Can't bind to reserved port (98). Sep 17 09:18:08 hyperion kernel: RPC: can't bind to reserved port. Sep 17 09:18:08 hyperion kernel: RPC: error 5 connecting to server jupiter.spiretech.com
Quoted 7 lines
Quoted 1 line
that is from the client trying to talk to the server.
That's strange. Errno 98 is EADDRINUSE. So, this means that you cycled through the entire port range and all of the ports were bound. Do you do a lot of mounts upon boot? Can you try tweaking sunrpc.min_resvport and sunrpc.max_resvport? Max can't be more than 1023. Try pushing min down to 500 or so.
i have run into this kind of problem in the past (running out of privileged ports). i found some code in glibc that seemed to be the biggest problem and never got around to trying to fix it. i'll look at what my current settings are and see about tweaking them.
Quoted 19 lines · Joe Pruett
Quoted 11 lines
Quoted 10 lines
well, it's a production box and only seems to show the problem at boot time. from looking at the logs, i think this is the underlying culprit:
Sep 17 09:18:08 hyperion kernel: RPC: Can't bind to reserved port (98). Sep 17 09:18:08 hyperion kernel: RPC: can't bind to reserved port. Sep 17 09:18:08 hyperion kernel: RPC: error 5 connecting to server jupiter.spiretech.com Sep 17 09:18:08 hyperion kernel: RPC: Can't bind to reserved port (98). Sep 17 09:18:08 hyperion kernel: RPC: can't bind to reserved port. Sep 17 09:18:08 hyperion kernel: RPC: error 5 connecting to server jupiter.spiretech.comQuoted 7 lines
Quoted 1 line
that is from the client trying to talk to the server.
That's strange. Errno 98 is EADDRINUSE. So, this means that you cycled through the entire port range and all of the ports were bound. Do you do a lot of mounts upon boot? Can you try tweaking sunrpc.min_resvport and sunrpc.max_resvport? Max can't be more than 1023. Try pushing min down to 500 or so.
Quoted 4 lines
i have run into this kind of problem in the past (running out of privileged ports). i found some code in glibc that seemed to be the biggest problem and never got around to trying to fix it. i'll look at what my current settings are and see about tweaking them.
Details would be welcome. The biggest offender we found was that the portmapper interfaces would use reserved ports. That is simply not necessary, and those issues have been fixed in autofs and mount.nfs. In fact, we've got regression tests that make sure that this is the case.
Cheers,
Jeff
Quoted 4 lines
Details would be welcome. The biggest offender we found was that the portmapper interfaces would use reserved ports. That is simply not necessary, and those issues have been fixed in autofs and mount.nfs. In fact, we've got regression tests that make sure that this is the case.
yes, i have found that as well. i think the biggest consumer is ypserv tcp connections for doing a full map dump.
here is my rpcinfo:
program vers proto port
100000 2 tcp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 844 status
100024 1 tcp 847 status
100004 2 udp 994 ypserv
100004 1 udp 994 ypserv
100004 2 tcp 997 ypserv
100004 1 tcp 997 ypserv
100007 2 udp 1007 ypbind
100007 1 udp 1007 ypbind
100007 2 tcp 1010 ypbind
100007 1 tcp 1010 ypbind
100011 1 udp 835 rquotad
100011 2 udp 835 rquotad
100011 1 tcp 838 rquotad
100011 2 tcp 838 rquotad
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100021 1 udp 39755 nlockmgr
100021 3 udp 39755 nlockmgr
100021 4 udp 39755 nlockmgr
100021 1 tcp 32773 nlockmgr
100021 3 tcp 32773 nlockmgr
100021 4 tcp 32773 nlockmgr
100005 1 udp 852 mountd
100005 1 tcp 855 mountd
100005 2 udp 852 mountd
100005 2 tcp 855 mountd
100005 3 udp 852 mountd
100005 3 tcp 855 mountdQuoted 26 lines · Jeff Moyer
Quoted 19 lines · Joe Pruett
Quoted 11 lines
Quoted 10 lines
well, it's a production box and only seems to show the problem at boot time. from looking at the logs, i think this is the underlying culprit:
Sep 17 09:18:08 hyperion kernel: RPC: Can't bind to reserved port (98). Sep 17 09:18:08 hyperion kernel: RPC: can't bind to reserved port. Sep 17 09:18:08 hyperion kernel: RPC: error 5 connecting to server jupiter.spiretech.com Sep 17 09:18:08 hyperion kernel: RPC: Can't bind to reserved port (98). Sep 17 09:18:08 hyperion kernel: RPC: can't bind to reserved port. Sep 17 09:18:08 hyperion kernel: RPC: error 5 connecting to server jupiter.spiretech.comQuoted 7 lines
Quoted 1 line
that is from the client trying to talk to the server.
That's strange. Errno 98 is EADDRINUSE. So, this means that you cycled through the entire port range and all of the ports were bound. Do you do a lot of mounts upon boot? Can you try tweaking sunrpc.min_resvport and sunrpc.max_resvport? Max can't be more than 1023. Try pushing min down to 500 or so.
Quoted 4 lines
i have run into this kind of problem in the past (running out of privileged ports). i found some code in glibc that seemed to be the biggest problem and never got around to trying to fix it. i'll look at what my current settings are and see about tweaking them.
Quoted 4 lines
Details would be welcome. The biggest offender we found was that the portmapper interfaces would use reserved ports. That is simply not necessary, and those issues have been fixed in autofs and mount.nfs. In fact, we've got regression tests that make sure that this is the case.
since i had my hang last night i have dug back into this. the big culprit is getgrouplist (or so it seems) that opens the group.byname map with a yp all call and it does that from a privileged port. watching my mail server, i see it burn through priv ports at quite a rapid clip (5 per second or so). i am going to see why it doesn't use the netid map. it could just be glibc not having that feature.
Quoted 11 lines
Quoted 4 lines
Details would be welcome. The biggest offender we found was that the portmapper interfaces would use reserved ports. That is simply not necessary, and those issues have been fixed in autofs and mount.nfs. In fact, we've got regression tests that make sure that this is the case.
since i had my hang last night i have dug back into this. the big culprit is getgrouplist (or so it seems) that opens the group.byname map with a yp all call and it does that from a privileged port. watching my mail server, i see it burn through priv ports at quite a rapid clip (5 per second or so). i am going to see why it doesn't use the netid map. it could just be glibc not having that feature.
aha! there is a file /etc/default/nss where you can enable nis to use the netid map. NETID_AUTHORITATIVE=TRUE is the key. and that has made a huge difference on my server.
Quoted 3 lines
Hehe, but the ioctl will fail with that many interfaces so you'll still have the same problem. You would need to make the buffer larger than 42k for this case.
the ioctl soft fails and just returns as much as it can. the autofs code just scans the ones that are returned. so my patch works for me. the ioctl must have other hard failure options, but a small buffer isn't one of them. the official patch compares the returned length for call n and call n+1 and exits once they are the same.