lib/libnetlink: ensure a minimum of 32KB for the buffer used in rtnl_recvmsg()
Subsystems (3)
PATCH 1/1 lib/libnetlink: ensure a minimum of 32KB for the buffer used in rtnl_recvmsg()
In the past, we tried to increase the buffer size up to 32 KB in order to reduce number of syscalls per dump.
Commit 2d34851cd341 ("lib/libnetlink: re malloc buff if size is not enough") brought the size back to 4KB because the kernel can not know the application is ready to receive bigger requests.
See kernel commits 9063e21fb026 ("netlink: autosize skb lengthes") and d35c99ff77ec ("netlink: do not enter direct reclaim from netlink_dump()") for more details.
--- lib/libnetlink.c | 2 ++ 1 file changed, 2 insertions(+)
lib/libnetlink.c +2 -0
@@ -718,6 +718,8 @@ static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer)if (len <0 )return len; +if (len <32768 ) + len =32768 ; buf =malloc (len);if (!buf) {fprintf (stderr,"malloc error: not enough buffer\n" );
Signature · 1 line
2.21.0.rc0.258.g878e2cd30e-goog
Quoted 25 lines · Eric Dumazet
In the past, we tried to increase the buffer size up to 32 KB in order to reduce number of syscalls per dump.
Commit
2d34851cd341("lib/libnetlink: re malloc buff if size is not enough") brought the size back to 4KB because the kernel can not know the application is ready to receive bigger requests.See kernel commits
9063e21fb026("netlink: autosize skb lengthes") andd35c99ff77ec("netlink: do not enter direct reclaim fromnetlink_dump()") for more details.Fixes 2d34851cd341 ("lib/libnetlink: re malloc buff if size is not enough")Signed-off-by Eric Dumazet <[email protected]>Cc Hangbin Liu <[email protected]>Cc Phil Sutter <[email protected]>--- lib/libnetlink.c | 2 ++ 1 file changed, 2 insertions(+)lib/libnetlink.c +0
-0
lib/libnetlink.c static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer) · line 725
Quoted 8 lines
lib/libnetlink.c +2
-0@@ -718,6 +718,8 @@ static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer)if (len <0 )return len; +if (len <32768 ) + len =32768 ; buf =malloc (len);if (!buf) {fprintf (stderr,"malloc error: not enough buffer\n" );
I believe that negates the whole point of 2d34851cd341 - which I have no problem with. 2 recvmsg calls per message is overkill.
Do we know of any single message sizes > 32k? 2d34851cd341 cites increasing VF's but at some point there is a limit. If not, the whole PEEK thing should go away and we just malloc 32k (or 64k) buffers for each recvmsg.
Quoted 28 lines · David Ahern
Quoted 27 lines · Eric Dumazet
In the past, we tried to increase the buffer size up to 32 KB in order to reduce number of syscalls per dump.
Commit
2d34851cd341("lib/libnetlink: re malloc buff if size is not enough") brought the size back to 4KB because the kernel can not know the application is ready to receive bigger requests.See kernel commits
9063e21fb026("netlink: autosize skb lengthes") andd35c99ff77ec("netlink: do not enter direct reclaim fromnetlink_dump()") for more details.Fixes 2d34851cd341 ("lib/libnetlink: re malloc buff if size is not enough")Signed-off-by Eric Dumazet <[email protected]>Cc Hangbin Liu <[email protected]>Cc Phil Sutter <[email protected]>--- lib/libnetlink.c | 2 ++ 1 file changed, 2 insertions(+)lib/libnetlink.c +0
-0@@ -718,6 +718,8 @@ static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer)if (len <0 )return len;
Quoted 10 lines
lib/libnetlink.c static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer) · line 725
Quoted 6 lines
lib/libnetlink.c +2
-0+if (len <32768 ) + len =32768 ; buf =malloc (len);if (!buf) {fprintf (stderr,"malloc error: not enough buffer\n" );I believe that negates the whole point of
2d34851cd341- which I have no problem with. 2 recvmsg calls per message is overkill.
It does not negates the point at all.
The main point was to eventually be able to allocate more than 32KB.
We need to have a minimum size of 32KB so that the kernel can cook reasonably sized skbs
Because trying to allocate 4KB only in 2019 is kind of stupid...
( Especially considering ss currently buffers the whole thing before calling render() !!! )
Quoted 5 lines
Do we know of any single message sizes > 32k?
2d34851cd341cites increasing VF's but at some point there is a limit. If not, the whole PEEK thing should go away and we just malloc 32k (or 64k) buffers for each recvmsg.
Hi David,
Quoted 8 lines · Eric Dumazet
Quoted 8 lines
Quoted 8 lines
lib/libnetlink.c +0
-0@@ -718,6 +718,8 @@ static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer)if (len <0 )return len;
Quoted 9 lines
Quoted 9 lines
lib/libnetlink.c static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer) · line 725
Quoted 6 lines
lib/libnetlink.c +2
-0+if (len <32768 ) + len =32768 ; buf =malloc (len);if (!buf) {fprintf (stderr,"malloc error: not enough buffer\n" );I believe that negates the whole point of
2d34851cd341- which I have no problem with. 2 recvmsg calls per message is overkill.
It should not affects ip cmd too much. But for ss, as Eric pointed, it cause performance issue.
Quoted 11 lines
Quoted 1 line
It does not negates the point at all.
The main point was to eventually be able to allocate more than 32KB.
We need to have a minimum size of 32KB so that the kernel can cook reasonably sized skbs
Because trying to allocate 4KB only in 2019 is kind of stupid...
( Especially considering ss currently buffers the whole thing before calling
render()!!! )
This makes sense to me.
Quoted 4 lines
Quoted 4 lines
Do we know of any single message sizes > 32k?
2d34851cd341cites increasing VF's but at some point there is a limit. If not, the whole PEEK thing should go away and we just malloc 32k (or 64k) buffers for each recvmsg.
Apart from the 200 VFs example, I think there will be more and more virtual interfaces be used in cloud environment, like openstack/OVS, so useing 32K or 64K is still not safe.
What do you think.
Thanks
Hangbin
Quoted 9 lines · Hangbin Liu
Quoted 5 lines
Quoted 5 lines
Do we know of any single message sizes > 32k?
2d34851cd341cites increasing VF's but at some point there is a limit. If not, the whole PEEK thing should go away and we just malloc 32k (or 64k) buffers for each recvmsg.Apart from the 200 VFs example, I think there will be more and more virtual interfaces be used in cloud environment, like openstack/OVS, so useing 32K or 64K is still not safe.
Many intefraces are not a problem. Those will have separate messages so that they don't need to fit into one buffer.
Michal Kubecek
Quoted 5 lines · David Ahern
Do we know of any single message sizes > 32k?
2d34851cd341cites increasing VF's but at some point there is a limit. If not, the whole PEEK thing should go away and we just malloc 32k (or 64k) buffers for each recvmsg.
IFLA_VF_LIST is by far the biggest thing I have seen so far. I don't remember exact numbers but the issue with 32KB buffer (for the whole RTM_NELINK message) was encountered by some of our customers with NICs having 120 or 128 VFs.
There is a bigger issue with IFLA_VFINFO_LIST, though, as it's an attribute so that netlink limits its size to 64 KB. IIRC with current size of IFLA_VF_INFO this would be reached with 270-280 VFs (I'm sure the number was higer than 256 but not too much higher.)
This would mean unless we let something else grow too much, the whole message shouldn't get much bigger than 64 KB. And if we can find some other solution (e.g. passing VF information in separate messages if client declares support), even 32 KB would be more than enough.
Michal Kubecek
Quoted 17 lines · Michal Kubecek
Quoted 5 lines · David Ahern
Do we know of any single message sizes > 32k?
2d34851cd341cites increasing VF's but at some point there is a limit. If not, the whole PEEK thing should go away and we just malloc 32k (or 64k) buffers for each recvmsg.
IFLA_VF_LISTis by far the biggest thing I have seen so far. I don't remember exact numbers but the issue with 32KB buffer (for the wholeRTM_NELINKmessage) was encountered by some of our customers with NICs having 120 or 128 VFs.There is a bigger issue with
IFLA_VFINFO_LIST, though, as it's an attribute so that netlink limits its size to 64 KB. IIRC with current size ofIFLA_VF_INFOthis would be reached with 270-280 VFs (I'm sure the number was higer than 256 but not too much higher.)
Quoted 4 lines
This would mean unless we let something else grow too much, the whole message shouldn't get much bigger than 64 KB. And if we can find some other solution (e.g. passing VF information in separate messages if client declares support), even 32 KB would be more than enough.
That's what I was asking, thanks. So 32kB today is sufficient, 64kB has future buffer. So this whole PEEK and allocate the message size is overkill. It could just as easily been bumped from 32kB to 64kB in the original patch and been good for a while.
Quoted 13 lines · David Ahern
Quoted 12 lines · Michal Kubecek
Quoted 5 lines · David Ahern
Do we know of any single message sizes > 32k?
2d34851cd341cites increasing VF's but at some point there is a limit. If not, the whole PEEK thing should go away and we just malloc 32k (or 64k) buffers for each recvmsg.
IFLA_VF_LISTis by far the biggest thing I have seen so far. I don't remember exact numbers but the issue with 32KB buffer (for the wholeRTM_NELINKmessage) was encountered by some of our customers with NICs having 120 or 128 VFs.
Quoted 4 lines
Quoted 4 lines
There is a bigger issue with
IFLA_VFINFO_LIST, though, as it's an attribute so that netlink limits its size to 64 KB. IIRC with current size ofIFLA_VF_INFOthis would be reached with 270-280 VFs (I'm sure the number was higer than 256 but not too much higher.)
Using netdevsim, 'ip link show' becomes unusable after enabling more than 244 VFs. I guess it depends on how much info per VF is available.
Quoted 9 lines
Quoted 4 lines
This would mean unless we let something else grow too much, the whole message shouldn't get much bigger than 64 KB. And if we can find some other solution (e.g. passing VF information in separate messages if client declares support), even 32 KB would be more than enough.
That's what I was asking, thanks. So 32kB today is sufficient, 64kB has future buffer. So this whole PEEK and allocate the message size is overkill. It could just as easily been bumped from 32kB to 64kB in the original patch and been good for a while.
Yes, I think the real problem is how VF-related messages are structured currently.
Cheers, Phil
Quoted 38 lines · David Ahern
Quoted 25 lines · Eric Dumazet
In the past, we tried to increase the buffer size up to 32 KB in order to reduce number of syscalls per dump.
Commit
2d34851cd341("lib/libnetlink: re malloc buff if size is not enough") brought the size back to 4KB because the kernel can not know the application is ready to receive bigger requests.See kernel commits
9063e21fb026("netlink: autosize skb lengthes") andd35c99ff77ec("netlink: do not enter direct reclaim fromnetlink_dump()") for more details.Fixes 2d34851cd341 ("lib/libnetlink: re malloc buff if size is not enough")Signed-off-by Eric Dumazet <[email protected]>Cc Hangbin Liu <[email protected]>Cc Phil Sutter <[email protected]>--- lib/libnetlink.c | 2 ++ 1 file changed, 2 insertions(+)lib/libnetlink.c +0
-0lib/libnetlink.c static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer) · line 725
Quoted 8 lines
lib/libnetlink.c +2
-0@@ -718,6 +718,8 @@ static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer)if (len <0 )return len; +if (len <32768 ) + len =32768 ; buf =malloc (len);if (!buf) {fprintf (stderr,"malloc error: not enough buffer\n" );I believe that negates the whole point of
2d34851cd341- which I have no problem with. 2 recvmsg calls per message is overkill.
Quoted 5 lines
Do we know of any single message sizes > 32k?
2d34851cd341cites increasing VF's but at some point there is a limit. If not, the whole PEEK thing should go away and we just malloc 32k (or 64k) buffers for each recvmsg.
Hey,
Sorry for reviving this old thread, but I see this topic was already discussed here :).
I have a system where the large number of VFs result in a message greater than 32k, which makes a simple 'ip link' command return an error.
Should we change the kernel's 'max_recvmsg_len' to 64k? Any other (more robust) ideas to resolve this issue?
Quoted 47 lines · On Mon, 29 May 2023 15:29:51 +0300 Gal Pressman <[email protected]> wrote:
Quoted 38 lines · David Ahern
Quoted 25 lines · Eric Dumazet
In the past, we tried to increase the buffer size up to 32 KB in order to reduce number of syscalls per dump.
Commit
2d34851cd341("lib/libnetlink: re malloc buff if size is not enough") brought the size back to 4KB because the kernel can not know the application is ready to receive bigger requests.See kernel commits
9063e21fb026("netlink: autosize skb lengthes") andd35c99ff77ec("netlink: do not enter direct reclaim fromnetlink_dump()") for more details.Fixes 2d34851cd341 ("lib/libnetlink: re malloc buff if size is not enough")Signed-off-by Eric Dumazet <[email protected]>Cc Hangbin Liu <[email protected]>Cc Phil Sutter <[email protected]>--- lib/libnetlink.c | 2 ++ 1 file changed, 2 insertions(+)lib/libnetlink.c +0
-0lib/libnetlink.c static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer) · line 725
Quoted 8 lines
lib/libnetlink.c +2
-0@@ -718,6 +718,8 @@ static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer)if (len <0 )return len; +if (len <32768 ) + len =32768 ; buf =malloc (len);if (!buf) {fprintf (stderr,"malloc error: not enough buffer\n" );I believe that negates the whole point of
2d34851cd341- which I have no problem with. 2 recvmsg calls per message is overkill.Quoted 5 lines
Do we know of any single message sizes > 32k?
2d34851cd341cites increasing VF's but at some point there is a limit. If not, the whole PEEK thing should go away and we just malloc 32k (or 64k) buffers for each recvmsg.Hey,
Quoted 7 lines
Sorry for reviving this old thread, but I see this topic was already discussed here :).
I have a system where the large number of VFs result in a message greater than 32k, which makes a simple 'ip link' command return an error.Should we change the kernel's '
max_recvmsg_len' to 64k? Any other (more robust) ideas to resolve this issue?
No matter what the size, someone will always have too many VF's to fit in the response. There is no way to get a stable solution without doing some API changes.
It is possible to dump millions of routes, so it is not directly a netlink issue more that the current API is slamming all the VF's as info blocks under a single message response.
That would mean replacing IFLA_VFINFO_LIST with a separate query
Quoted 62 lines · Stephen Hemminger
Quoted 47 lines · On Mon, 29 May 2023 15:29:51 +0300 Gal Pressman <[email protected]> wrote:
Quoted 38 lines · David Ahern
Quoted 25 lines · Eric Dumazet
In the past, we tried to increase the buffer size up to 32 KB in order to reduce number of syscalls per dump.
Commit
2d34851cd341("lib/libnetlink: re malloc buff if size is not enough") brought the size back to 4KB because the kernel can not know the application is ready to receive bigger requests.See kernel commits
9063e21fb026("netlink: autosize skb lengthes") andd35c99ff77ec("netlink: do not enter direct reclaim fromnetlink_dump()") for more details.Fixes 2d34851cd341 ("lib/libnetlink: re malloc buff if size is not enough")Signed-off-by Eric Dumazet <[email protected]>Cc Hangbin Liu <[email protected]>Cc Phil Sutter <[email protected]>--- lib/libnetlink.c | 2 ++ 1 file changed, 2 insertions(+)lib/libnetlink.c +0
-0lib/libnetlink.c static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer) · line 725
Quoted 8 lines
lib/libnetlink.c +2
-0@@ -718,6 +718,8 @@ static int rtnl_recvmsg(int fd, struct msghdr *msg, char **answer)if (len <0 )return len; +if (len <32768 ) + len =32768 ; buf =malloc (len);if (!buf) {fprintf (stderr,"malloc error: not enough buffer\n" );I believe that negates the whole point of
2d34851cd341- which I have no problem with. 2 recvmsg calls per message is overkill.Quoted 5 lines
Do we know of any single message sizes > 32k?
2d34851cd341cites increasing VF's but at some point there is a limit. If not, the whole PEEK thing should go away and we just malloc 32k (or 64k) buffers for each recvmsg.Hey,
Quoted 7 lines
Sorry for reviving this old thread, but I see this topic was already discussed here :).
I have a system where the large number of VFs result in a message greater than 32k, which makes a simple 'ip link' command return an error.Should we change the kernel's '
max_recvmsg_len' to 64k? Any other (more robust) ideas to resolve this issue?No matter what the size, someone will always have too many VF's to fit in the response. There is no way to get a stable solution without doing some API changes.
Quoted 5 lines
It is possible to dump millions of routes, so it is not directly a netlink issue more that the current API is slamming all the VF's as info blocks under a single message response.
That would mean replacing
IFLA_VFINFO_LISTwith a separate query
Thanks Stephen!
How would you imagine it? Changing the userspace to split each (PF, VF) to a separate netlink call instead of a single call for each PF?
Quoted 9 lines · Gal Pressman
Quoted 5 lines
It is possible to dump millions of routes, so it is not directly a netlink issue more that the current API is slamming all the VF's as info blocks under a single message response.
That would mean replacing
IFLA_VFINFO_LISTwith a separate queryThanks Stephen!
How would you imagine it? Changing the userspace to split each (PF, VF) to a separate netlink call instead of a single call for each PF?
This is the last attempt that I recall to address this issue:
Quoted 7 lines · David Ahern
Quoted 6 lines · Gal Pressman
Quoted 5 lines
It is possible to dump millions of routes, so it is not directly a netlink issue more that the current API is slamming all the VF's as info blocks under a single message response.
That would mean replacing
IFLA_VFINFO_LISTwith a separate query
Quoted 8 lines
Quoted 3 lines
Thanks Stephen!
How would you imagine it? Changing the userspace to split each (PF, VF) to a separate netlink call instead of a single call for each PF?This is the last attempt that I recall to address this issue:
Thanks for the pointer David, this is helpful.
Hi Eric,
Quoted 10 lines · Eric Dumazet
In the past, we tried to increase the buffer size up to 32 KB in order to reduce number of syscalls per dump.
Commit
2d34851cd341("lib/libnetlink: re malloc buff if size is not enough") brought the size back to 4KB because the kernel can not know the application is ready to receive bigger requests.See kernel commits
9063e21fb026("netlink: autosize skb lengthes") andd35c99ff77ec("netlink: do not enter direct reclaim fromnetlink_dump()") for more details.
Wouldn't it be better if the kernel recognized MSG_TRUNC and allocated a buffer large enough to hold the full message in that case? I have no idea how hard that would be to implement, but calling recvmsg() with MSG_TRUNC set and not getting the full message length in return is not quite what one expects after reading recvmsg(2).
Cheers, Phil
Quoted 14 lines · Phil Sutter
Hi Eric,
Quoted 10 lines · Eric Dumazet
In the past, we tried to increase the buffer size up to 32 KB in order to reduce number of syscalls per dump.
Commit
2d34851cd341("lib/libnetlink: re malloc buff if size is not enough") brought the size back to 4KB because the kernel can not know the application is ready to receive bigger requests.See kernel commits
9063e21fb026("netlink: autosize skb lengthes") andd35c99ff77ec("netlink: do not enter direct reclaim fromnetlink_dump()") for more details.
Quoted 6 lines
Wouldn't it be better if the kernel recognized
MSG_TRUNCand allocated a buffer large enough to hold the full message in that case? I have no idea how hard that would be to implement, but callingrecvmsg()withMSG_TRUNCset and not getting the full message length in return is not quite what one expects after reading recvmsg(2).
I am not sure what you are suggesting.
The buffer is already in the kernel, this is the skb in the receive queue. Its size is unknown... We only can guess.
We can not change old kernels, and new iproute2/ss commands need to work with old kernels.
We can not change MSG_TRUNC semantic.
Adding another MSG_be_gentle_do_not_consume_skb_if_user_size_is_too_small wont solve the issue for old kernels.
Quoted 7 lines · On Tue, 12 Feb 2019 17:58:41 -0800 Eric Dumazet <[email protected]> wrote:
In the past, we tried to increase the buffer size up to 32 KB in order to reduce number of syscalls per dump.
Commit
2d34851cd341("lib/libnetlink: re malloc buff if size is not enough") brought the size back to 4KB because the kernel can not know the application is ready to receive bigger requests.
Quoted 8 lines
See kernel commits
9063e21fb026("netlink: autosize skb lengthes") andd35c99ff77ec("netlink: do not enter direct reclaim fromnetlink_dump()") for more details.Fixes 2d34851cd341 ("lib/libnetlink: re malloc buff if size is not enough")Signed-off-by Eric Dumazet <[email protected]>Cc Hangbin Liu <[email protected]>Cc Phil Sutter <[email protected]>
Applied, although maybe we should bump it to 64K or bigger?
Quoted 15 lines · Stephen Hemminger
Quoted 11 lines · On Tue, 12 Feb 2019 17:58:41 -0800 Eric Dumazet <[email protected]> wrote:
In the past, we tried to increase the buffer size up to 32 KB in order to reduce number of syscalls per dump.
Commit
2d34851cd341("lib/libnetlink: re malloc buff if size is not enough") brought the size back to 4KB because the kernel can not know the application is ready to receive bigger requests.See kernel commits
9063e21fb026("netlink: autosize skb lengthes") andd35c99ff77ec("netlink: do not enter direct reclaim fromnetlink_dump()") for more details.
Quoted 6 lines
Quoted 4 lines
Fixes 2d34851cd341 ("lib/libnetlink: re malloc buff if size is not enough")Signed-off-by Eric Dumazet <[email protected]>Cc Hangbin Liu <[email protected]>Cc Phil Sutter <[email protected]>Applied, although maybe we should bump it to 64K or bigger?
Note the kernel does not yet try 64KB allocations, so I do not see an urgent need for that :)