iproute uses too small of a receive buffer

I have a very busy system with a bunch of xorp router processes (mis)configured.

This thing is rapidly making route changes for whatever reason.

The 'ip monitor route' command was failing:

[root@i7-dqc-1 ]# ip monitor route
netlink receive error No buffer space available (105) Dump terminated

It is only using a 32k rcv buffer, and it seems the OS was overdriving it.

Please consider making the rcv buffer larger, perhaps something like this (inline is white-space damaged...attachment should apply if deemed useful.):

Signed-off-by Ben Greear <[email protected]>

lib/libnetlink.c +1 -1

index b68e2fd..95a7d1d 100644
@@ -38,7 +38,7 @@ int rtnl_open_byproto(struct rtnl_handle *rth, unsigned subscriptions,
  {
         socklen_t addr_len;
         int sndbuf = 32768;
-       int rcvbuf = 32768;
+       int rcvbuf = 3276800;
 
         memset(rth, 0, sizeof(*rth));
 

Thanks,
Ben

Quoted 34 lines · On Tue, 27 Oct 2009 16:16:52 -0700 Ben Greear<[email protected]> wrote:

I have a very busy system with a bunch of xorp router processes (mis)configured.

This thing is rapidly making route changes for whatever reason.

The 'ip monitor route' command was failing:

[root@i7-dqc-1 ]# ip monitor route
netlink receive error No buffer space available (105) Dump terminated

It is only using a 32k rcv buffer, and it seems the OS was overdriving it.

Please consider making the rcv buffer larger, perhaps something like this (inline is white-space damaged...attachment should apply if deemed useful.):

Signed-off-by Ben Greear<[email protected]>

lib/libnetlink.c +1 -1

index b68e2fd..95a7d1d 100644
@@ -38,7 +38,7 @@ int rtnl_open_byproto(struct rtnl_handle *rth, unsigned subscriptions,
   {
          socklen_t addr_len;
          int sndbuf = 32768;
-       int rcvbuf = 32768;
+       int rcvbuf = 3276800;
 
          memset(rth, 0, sizeof(*rth));
 

Quoted 3 lines

Thanks,
Ben

Just having larger buffer isn't guarantee of success. Allocating a huge buffer is not going to work on embedded.

Why not have it continue after one error.

Probably the right way is to give a cmd-line arg to set the buffer size and also continue if the error is ENOBUFs (but print some error out so users know they have issues). I can make the attempt if that sounds good to you.

Quoted 4 lines · Ben Greear

Probably the right way is to give a cmd-line arg to set the buffer size and also continue if the error is ENOBUFs (but print some error out so users know they have issues). I can make the attempt if that sounds good to you.

Real fix is to realloc buffer at receive time, no need for user setting.

Quoted 4 lines · Ben Greear

Probably the right way is to give a cmd-line arg to set the buffer size and also continue if the error is ENOBUFs (but print some error out so users know they have issues). I can make the attempt if that sounds good to you.

Real fix is to realloc buffer at receive time, no need for user setting.

In my testings I saw it reaching 1 Mbyte write(2, "REALLOC buflen 8192\n"..., 20) = 20 write(2, "REALLOC buflen 16384\n"..., 21) = 21 write(2, "REALLOC buflen 32768\n"..., 21) = 21 write(2, "REALLOC buflen 65536\n"..., 21) = 21 write(2, "REALLOC buflen 131072\n"..., 22) = 22 write(2, "REALLOC buflen 262144\n"..., 22) = 22 write(2, "REALLOC buflen 524288\n"..., 22) = 22

[iproute2] realloc buffer in rtnl_listen

# ip monitor route
netlink receive error No buffer space available (105)
Dump terminated 

Reported-by Ben Greear<[email protected]>

Signed-off-by Eric Dumazet <[email protected]>

Just having larger buffer isn't guarantee of success. Allocating a huge buffer is not going to work on embedded.

Why not have it continue after one error.

Quoted 4 lines · Stephen Hemminger

Just having larger buffer isn't guarantee of success. Allocating a huge buffer is not going to work on embedded.

Please note we do not allocate a big buffer, only allow more small skbs to be queued on socket receive queue.

If memory is not available, skb allocation will eventually fail and be reported as well, embedded or not.

I vote for allowing 1024*1024 bytes instead of 32768, and eventually user should be warned that it is capped by /proc/sys/net/core/rmem_max

Quoted 3 lines · Stephen Hemminger

Just having larger buffer isn't guarantee of success. Allocating a huge buffer is not going to work on embedded.

Please note we do not allocate a big buffer, only allow more small skbs to be queued on socket receive queue.

If memory is not available, skb allocation will eventually fail and be reported as well, embedded or not.

I vote for allowing 1024*1024 bytes instead of 32768, and eventually user should be warned that it is capped by /proc/sys/net/core/rmem_max

Quoted 8 lines · Eric Dumazet

Quoted 3 lines · Stephen Hemminger

Just having larger buffer isn't guarantee of success. Allocating a huge buffer is not going to work on embedded.

Please note we do not allocate a big buffer, only allow more small skbs to be queued on socket receive queue.

Quoted 6 lines

If memory is not available, skb allocation will eventually fail and be reported as well, embedded or not.

I vote for allowing 1024*1024 bytes instead of 32768, and eventually user should be warned that it is capped by /proc/sys/net/core/rmem_max

How about this? It will double the receive queue limit on ENOBUFS up to 1024 * 1024b, then bail out with the normal error message on further ENOBUFS.

Signed-off-by Patrick McHardy<[email protected]>

Quoted 20 lines · Patrick McHardy

Quoted 8 lines · Eric Dumazet

Quoted 3 lines · Stephen Hemminger

Just having larger buffer isn't guarantee of success. Allocating a huge buffer is not going to work on embedded.

Please note we do not allocate a big buffer, only allow more small skbs to be queued on socket receive queue.

Quoted 6 lines

If memory is not available, skb allocation will eventually fail and be reported as well, embedded or not.

I vote for allowing 1024*1024 bytes instead of 32768, and eventually user should be warned that it is capped by /proc/sys/net/core/rmem_max

How about this? It will double the receive queue limit on ENOBUFS up to 1024 * 1024b, then bail out with the normal error message on further ENOBUFS.

Quoted 1 line

Signed-off-by Patrick McHardy<[email protected]>

First: This still pretty much guarantees that messages will be lost when the program starts (when messages are coming in too large of chunks for small buffers)
If you are debugging something tricky, having lost messages will be very annoying!

Second: Why bail on ENOBUFS at all? I don't see how it helps the user since they will probably just have to start it again, and will miss more messages than keeping going would have.

And, even 1MB may not be enough for some scenarios. So, probably best to let users over-ride the initial setting on cmd-line. If not, then use a large value to start with.

Quoted 3 lines

And, even 1MB may not be enough for some scenarios. So, probably best to let users over-ride the initial setting on cmd-line. If not, then use a large value to start with.

How about this? It uses 1MB as receive buf limit by default (without increasing /proc/sys/net/core/rmem_max it will be limited by less however) and allows to specify the size manually using "-rcvbuf X" (-r is already used, so you need to specify at least -rc).

Additionally rtnl_listen() continues on ENOBUFS after printing the error message.

Quoted 10 lines · Patrick McHardy

Quoted 4 lines

And, even 1MB may not be enough for some scenarios. So, probably best to
let users over-ride the initial setting on cmd-line. If not, then use a large value to start with.

How about this? It uses 1MB as receive buf limit by default (without increasing /proc/sys/net/core/rmem_max it will be limited by less however) and allows to specify the size manually using "-rcvbuf X" (-r is already used, so you need to specify at least -rc).

Quoted 2 lines

Additionally rtnl_listen() continues on ENOBUFS after printing the error message.

Looks good..except:

If rmem_max is smaller than 1M, will that cause setsocktopt to fail and thus fail early out of rtnl_open_byproto?

Maybe we should only print errors but not return in that method when setsockopt fails?

In another project, I ended up trying ever smaller values until one worked in order to get near what the user wanted even if rmem_max was configured smaller. Not sure if that is worth doing here or not.

Quoted 15 lines · Ben Greear

Quoted 12 lines · Patrick McHardy

Quoted 4 lines

And, even 1MB may not be enough for some scenarios. So, probably best to
let users over-ride the initial setting on cmd-line. If not, then use a large value to start with.

How about this? It uses 1MB as receive buf limit by default (without increasing /proc/sys/net/core/rmem_max it will be limited by less however) and allows to specify the size manually using "-rcvbuf X" (-r is already used, so you need to specify at least -rc).

Additionally rtnl_listen() continues on ENOBUFS after printing the error message.

Quoted 4 lines

Looks good..except:

If rmem_max is smaller than 1M, will that cause setsocktopt to fail and thus fail early out of rtnl_open_byproto?

No, the kernel takes the value as a hint and only uses the maximum allowable value:

How about this? It uses 1MB as receive buf limit by default (without increasing /proc/sys/net/core/rmem_max it will be limited by less however) and allows to specify the size manually using "-rcvbuf X" (-r is already used, so you need to specify at least -rc).

Additionally rtnl_listen() continues on ENOBUFS after printing the error message.

Second: Why bail on ENOBUFS at all? I don't see how it helps the user since they will probably just have to start it again, and will miss more messages than keeping going would have.

And, even 1MB may not be enough for some scenarios. So, probably best to let users over-ride the initial setting on cmd-line. If not, then use a large value to start with.

How about this? It will double the receive queue limit on ENOBUFS up to 1024 * 1024b, then bail out with the normal error message on further ENOBUFS.

Signed-off-by Patrick McHardy <[email protected]>