netlink: handle EMSGSIZE errors in the core
Subsystems (3)
Jakub Kicinski
maintainerv2 cover netlink: handle EMSGSIZE errors in the core
Ido discovered some time back that we usually force NLMSG_DONE to be delivered in a separate recv() syscall, even if it would fit into the same skb as data messages. He made nexthop try to fit DONE with data in commit 8743aeff5bc4 ("nexthop: Fix infinite nexthop bucket dump when using maximum nexthop ID"), and nobody has complained so far.
We have since also tried to follow the same pattern in new genetlink families, but explaining to people, or even remembering the correct handling ourselves is tedious.
Let the netlink socket layer consume -EMSGSIZE errors. Practically speaking most families use this error code as "dump needs more space", anyway.
v2:
- init err to 0 in last patch
v1: https://lore.kernel.org/all/[email protected]/
Jakub Kicinski (3):
netlink: handle EMSGSIZE errors in the core
netdev: let netlink core handle -EMSGSIZE errors
genetlink: fit NLMSG_DONE into same read() as families
net/core/netdev-genl.c | 15 +++------------ net/core/page_pool_user.c | 2 -- net/netlink/af_netlink.c | 9 +++++++++ net/netlink/genetlink.c | 12 +++++++----- 4 files changed, 19 insertions(+), 19 deletions(-)
Signature · 1 line
2.44.0
Jakub Kicinski
maintainerPATCH v2 1/3 netlink: handle EMSGSIZE errors in the core
Eric points out that our current suggested way of handling EMSGSIZE errors ((err == -EMSGSIZE) ? skb->len : err) will break if we didn't fit even a single object into the buffer provided by the user. This should not happen for well behaved applications, but we can fix that, and free netlink families from dealing with that completely by moving error handling into the core.
Let's assume from now on that all EMSGSIZE errors in dumps are because we run out of skb space. Families can now propagate the error nla_put_*() etc generated and not worry about any return value magic. If some family really wants to send EMSGSIZE to user space, assuming it generates the same error on the next dump iteration the skb->len should be 0, and user space should still see the EMSGSIZE.
This should simplify families and prevent mistakes in return values which lead to DONE being forced into a separate recv() call as discovered by Ido some time ago.
---
CC: [email protected]
--- net/netlink/af_netlink.c | 9 +++++++++ 1 file changed, 9 insertions(+)
net/netlink/af_netlink.c +9 -0
@@ -2267,6 +2267,15 @@ static int netlink_dump(struct sock *sk, bool lock_taken)if (extra_mutex)mutex_unlock (extra_mutex); +/* EMSGSIZE plus something already in the skb means +* that there's more to dump but current skb has filled up. +* If the callback really wants to return EMSGSIZE to user space +* it needs to do so again, on the next cb->dump() call, +* without putting data in the skb. +*/ +if (nlk->dump_done_errno == -EMSGSIZE && skb->len) + nlk->dump_done_errno = skb->len; + cb->extack =NULL ; }
Signature · 1 line
2.44.0
Quoted 16 lines · Jakub Kicinski
Eric points out that our current suggested way of handling EMSGSIZE errors ((err ==
-EMSGSIZE) ?skb->len: err) will break if we didn't fit even a single object into the buffer provided by the user. This should not happen for well behaved applications, but we can fix that, and free netlink families from dealing with that completely by moving error handling into the core.Let's assume from now on that all EMSGSIZE errors in dumps are because we run out of skb space. Families can now propagate the error nla_put_*() etc generated and not worry about any return value magic. If some family really wants to send EMSGSIZE to user space, assuming it generates the same error on the next dump iteration the
skb->lenshould be 0, and user space should still see the EMSGSIZE.
Quoted 6 lines
This should simplify families and prevent mistakes in return values which lead to DONE being forced into a separate
recv()call as discovered by Ido some time ago.Reviewed-by Eric Dumazet <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>
I read your comment here [1] and I believe this patch [2] is needed to avoid another pass in case of an error code other than EMSGSIZE. I can submit it after your series is accepted.
[1] https://lore.kernel.org/netdev/[email protected]/
[2]
net/ipv4/nexthop.c +0 -9
@@ -3241,10 +3241,6 @@ static int rtm_dump_nexthop(struct sk_buff *skb, struct netlink_callback *cb) err =rtm_dump_walk_nexthops (skb, cb, root, ctx, &rtm_dump_nexthop_cb, &filter); -if (err <0 ) { -if (likely (skb->len)) - err = skb->len; - } cb->seq = net->nexthop.seq;nl_dump_check_consistent (cb,nlmsg_hdr (skb));
@@ -3439,11 +3435,6 @@ static int rtm_dump_nexthop_bucket(struct sk_buff *skb, &rtm_dump_nexthop_bucket_cb, &dd); } -if (err <0 ) { -if (likely (skb->len)) - err = skb->len; - } - cb->seq = net->nexthop.seq;nl_dump_check_consistent (cb,nlmsg_hdr (skb));return err;
Jakub Kicinski
maintainerPATCH v2 2/3 netdev: let netlink core handle -EMSGSIZE errors
In reply to Jakub KicinskiPrevious change added -EMSGSIZE handling to af_netlink, we don't have to hide these errors any longer.
Theoretically the error handling changes from:
if (err == -EMSGSIZE)
to
if (err == -EMSGSIZE && skb->len)
everywhere, but in practice it doesn't matter.
All messages fit into NLMSG_GOODSIZE, so overflow of an empty skb cannot happen.
---
CC: [email protected]
CC: [email protected]
CC: [email protected]
--- net/core/netdev-genl.c | 15 +++------------ net/core/page_pool_user.c | 2 -- 2 files changed, 3 insertions(+), 14 deletions(-)
net/core/netdev-genl.c +3 -12
@@ -152,10 +152,7 @@ int netdev_nl_dev_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb) }rtnl_unlock (); -if (err != -EMSGSIZE ) -return err; - -return skb->len; +return err; } static int
@@ -287,10 +284,7 @@ int netdev_nl_napi_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb) }rtnl_unlock (); -if (err != -EMSGSIZE ) -return err; - -return skb->len; +return err; } static int
@@ -463,10 +457,7 @@ int netdev_nl_queue_get_dumpit(struct sk_buff *skb, struct netlink_callback *cb) }rtnl_unlock (); -if (err != -EMSGSIZE ) -return err; - -return skb->len; +return err; } static int netdev_genl_netdevice_event(struct notifier_block *nb,
net/core/page_pool_user.c +0 -2
@@ -103,8 +103,6 @@ netdev_nl_page_pool_get_dump(struct sk_buff *skb, struct netlink_callback *cb,mutex_unlock (&page_pools_lock);rtnl_unlock (); -if (skb->len && err == -EMSGSIZE ) -return skb->len;return err; }
Signature · 1 line
2.44.0
Re: [PATCH net-next v2 2/3] netdev: let netlink core handle -EMSGSIZE errors
Quoted 8 lines · Jakub Kicinski
Previous change added
-EMSGSIZEhandling toaf_netlink, we don't have to hide these errors any longer.Theoretically the error handling changes from:
if (err ==-EMSGSIZE)
to
if (err ==-EMSGSIZE&&skb->len)
Quoted 6 lines
everywhere, but in practice it doesn't matter.
All messages fit intoNLMSG_GOODSIZE, so overflow of an empty skb cannot happen.Reviewed-by Eric Dumazet <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>
Jakub Kicinski
maintainerPATCH v2 3/3 genetlink: fit NLMSG_DONE into same read() as families
In reply to Jakub KicinskiMake sure ctrl_fill_info() returns sensible error codes and propagate them out to netlink core. Let netlink core decide when to return skb->len and when to treat the exit as an error. Netlink core does better job at it, if we always return skb->len the core doesn't know when we're done dumping and NLMSG_DONE ends up in a separate read().
---
CC: [email protected]
--- net/netlink/genetlink.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)
net/netlink/genetlink.c +7 -5
@@ -1232,7 +1232,7 @@ static int ctrl_fill_info(const struct genl_family *family, u32 portid, u32 seq, hdr =genlmsg_put (skb, portid, seq, &genl_ctrl, flags, cmd);if (hdr ==NULL ) -return -1 ; +return -EMSGSIZE ; if (nla_put_string(skb, CTRL_ATTR_FAMILY_NAME, family->name) || nla_put_u16(skb, CTRL_ATTR_FAMILY_ID, family->id) ||
@@ -1355,6 +1355,7 @@ static int ctrl_dumpfamily(struct sk_buff *skb, struct netlink_callback *cb)struct net *net =sock_net (skb->sk);int fams_to_skip = cb->args[0 ];unsigned int id; +int err =0 ;idr_for_each_entry (&genl_fam_idr, rt, id) {if (!rt->netnsok && !net_eq (net, &init_net))
@@ -1363,16 +1364,17 @@ static int ctrl_dumpfamily(struct sk_buff *skb, struct netlink_callback *cb)if (n++ < fams_to_skip)continue ; -if (ctrl_fill_info (rt,NETLINK_CB (cb->skb).portid, - cb->nlh->nlmsg_seq,NLM_F_MULTI , - skb,CTRL_CMD_NEWFAMILY ) <0 ) { + err =ctrl_fill_info (rt,NETLINK_CB (cb->skb).portid, + cb->nlh->nlmsg_seq,NLM_F_MULTI , + skb,CTRL_CMD_NEWFAMILY ); +if (err) { n--;break ; } } cb->args[0 ] = n; -return skb->len; +return err; } static struct sk_buff *ctrl_build_family_msg(const struct genl_family *family,
Signature · 1 line
2.44.0
Re: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 9 lines · Jakub Kicinski
Make sure
ctrl_fill_info()returns sensible error codes and propagate them out to netlink core. Let netlink core decide when to returnskb->lenand when to treat the exit as an error. Netlink core does better job at it, if we always returnskb->lenthe core doesn't know when we're done dumping andNLMSG_DONEends up in a separateread().Reviewed-by Eric Dumazet <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>
Re: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
In reply to Jakub KicinskiHi,
Quoted 6 lines · On Sat, 2 Mar 2024 21:24:08 -0800 Jakub Kicinski <[email protected]> wrote:
Make sure
ctrl_fill_info()returns sensible error codes and propagate them out to netlink core. Let netlink core decide when to returnskb->lenand when to treat the exit as an error. Netlink core does better job at it, if we always returnskb->lenthe core doesn't know when we're done dumping andNLMSG_DONEends up in a separateread().
While this change is obviously correct, it breaks... well, broken applications that _wrongly_ rely on the fact that NLMSG_DONE is delivered in a separate datagram.
This was the (embarrassing) case for passt(1), which I just fixed:
https://archives.passt.top/passt-dev/[email protected]/
but the "separate" NLMSG_DONE is such an established behaviour, I think, that this might raise a more general concern.
From my perspective, I'm just happy that this change revealed the issue, but I wanted to report this anyway in case somebody has similar possible breakages in mind.
Quoted 51 lines
Reviewed-by Eric Dumazet <[email protected]>Signed-off-by Jakub Kicinski <[email protected]>---
CC: [email protected]--- net/netlink/genetlink.c | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-)net/netlink/genetlink.c +7
-5@@ -1232,7 +1232,7 @@ static int ctrl_fill_info(const struct genl_family *family, u32 portid, u32 seq, hdr =genlmsg_put (skb, portid, seq, &genl_ctrl, flags, cmd);if (hdr ==NULL ) -return -1 ; +return -EMSGSIZE ; if (nla_put_string(skb, CTRL_ATTR_FAMILY_NAME, family->name) || nla_put_u16(skb, CTRL_ATTR_FAMILY_ID, family->id) ||@@ -1355,6 +1355,7 @@ static int ctrl_dumpfamily(struct sk_buff *skb, struct netlink_callback *cb)struct net *net =sock_net (skb->sk);int fams_to_skip = cb->args[0 ];unsigned int id; +int err =0 ;idr_for_each_entry (&genl_fam_idr, rt, id) {if (!rt->netnsok && !net_eq (net, &init_net))@@ -1363,16 +1364,17 @@ static int ctrl_dumpfamily(struct sk_buff *skb, struct netlink_callback *cb)if (n++ < fams_to_skip)continue ; -if (ctrl_fill_info (rt,NETLINK_CB (cb->skb).portid, - cb->nlh->nlmsg_seq,NLM_F_MULTI , - skb,CTRL_CMD_NEWFAMILY ) <0 ) { + err =ctrl_fill_info (rt,NETLINK_CB (cb->skb).portid, + cb->nlh->nlmsg_seq,NLM_F_MULTI , + skb,CTRL_CMD_NEWFAMILY ); +if (err) { n--;break ; } } cb->args[0 ] = n; -return skb->len; +return err; } static struct sk_buff *ctrl_build_family_msg(const struct genl_family *family,
Signature · 1 line
Stefano
Jakub Kicinski
maintainerRe: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 14 lines · On Fri, 15 Mar 2024 12:48:08 +0100 Stefano Brivio wrote:
Quoted 6 lines
Make sure
ctrl_fill_info()returns sensible error codes and propagate them out to netlink core. Let netlink core decide when to returnskb->lenand when to treat the exit as an error. Netlink core does better job at it, if we always returnskb->lenthe core doesn't know when we're done dumping andNLMSG_DONEends up in a separateread().While this change is obviously correct, it breaks... well, broken applications that _wrongly_ rely on the fact that
NLMSG_DONEis delivered in a separate datagram.This was the (embarrassing) case for passt(1), which I just fixed:
https://archives.passt.top/passt-dev/[email protected]/
Quoted 6 lines
but the "separate"
NLMSG_DONEis such an established behaviour, I think, that this might raise a more general concern.From my perspective, I'm just happy that this change revealed the issue, but I wanted to report this anyway in case somebody has similar possible breakages in mind.
Hi Stefano! I was worried this may happen :( I think we should revert offending commits, but I'd like to take it on case by case basis. I'd imagine majority of netlink is only exercised by iproute2 and libmnl-based tools. Does passt hang specifically on genetlink family dump? Your commit also mentions RTM_GETROUTE. This is not the only commit which removed DONE:
$ git log --since='1 month ago' --grep=NLMSG_DONE --no-merges --oneline
9cc4cc329d30 ipv6: use xa_array iterator to implement inet6_dump_addr() 87d381973e49 genetlink: fit NLMSG_DONE into same read() as families 4ce5dc9316de inet: switch inet_dump_fib() to RCU protection 6647b338fc5c netlink: fix netlink_diag_dump() return value
Re: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 30 lines · Jakub Kicinski
Quoted 14 lines · On Fri, 15 Mar 2024 12:48:08 +0100 Stefano Brivio wrote:
Quoted 6 lines
Make sure
ctrl_fill_info()returns sensible error codes and propagate them out to netlink core. Let netlink core decide when to returnskb->lenand when to treat the exit as an error. Netlink core does better job at it, if we always returnskb->lenthe core doesn't know when we're done dumping andNLMSG_DONEends up in a separateread().While this change is obviously correct, it breaks... well, broken applications that _wrongly_ rely on the fact that
NLMSG_DONEis delivered in a separate datagram.This was the (embarrassing) case for passt(1), which I just fixed:
https://archives.passt.top/passt-dev/[email protected]/Quoted 6 lines
but the "separate"
NLMSG_DONEis such an established behaviour, I think, that this might raise a more general concern.From my perspective, I'm just happy that this change revealed the issue, but I wanted to report this anyway in case somebody has similar possible breakages in mind.
Hi Stefano! I was worried this may happen :( I think we should revert offending commits, but I'd like to take it on case by case basis. I'd imagine majority of netlink is only exercised by iproute2 and libmnl-based tools. Does passt hang specifically on genetlink family dump? Your commit also mentions
RTM_GETROUTE. This is not the only commit which removed DONE:
Quoted 6 lines
$ git log --since='1 month ago' --grep=NLMSG_DONE --no-merges --oneline
9cc4cc329d30ipv6: usexa_arrayiterator to implementinet6_dump_addr()87d381973e49genetlink: fitNLMSG_DONEinto sameread()as families4ce5dc9316deinet: switchinet_dump_fib()to RCU protection6647b338fc5cnetlink: fixnetlink_diag_dump()return value
Lets not bring back more RTNL locking please for the handlers that still require it.
The core can generate an NLMSG_DONE by itself, if we decide this needs to be done.
I find this discussion a bit strange, because NLMSG_DONE being piggybacked has been a long established behavior.
Jason patch was from 2017
Jakub Kicinski
maintainerRe: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 9 lines · On Tue, 19 Mar 2024 18:17:47 +0100 Eric Dumazet wrote:
Quoted 9 lines
Hi Stefano! I was worried this may happen :( I think we should revert offending commits, but I'd like to take it on case by case basis. I'd imagine majority of netlink is only exercised by iproute2 and libmnl-based tools. Does passt hang specifically on genetlink family dump? Your commit also mentions
RTM_GETROUTE. This is not the only commit which removed DONE:$ git log --since='1 month ago' --grep=NLMSG_DONE --no-merges --oneline
Quoted 7 lines
Quoted 4 lines
9cc4cc329d30ipv6: usexa_arrayiterator to implementinet6_dump_addr()87d381973e49genetlink: fitNLMSG_DONEinto sameread()as families4ce5dc9316deinet: switchinet_dump_fib()to RCU protection6647b338fc5cnetlink: fixnetlink_diag_dump()return valueLets not bring back more RTNL locking please for the handlers that still require it.
Definitely. My git log copy/paste is pretty inaccurate, these two are better examples:
5d9b7cb383bb nexthop: Simplify dump error handling 02e24903e5a4 netlink: let core handle error cases in dump operations
I was trying to point out that we merged a handful of DONE "coalescing" patches, and if we need to revert - let's only do that for the exact commands needed. The comment was raised on my genetlink patch while the discussion in the link points to RTM_GETROUTE.
Quoted 2 lines
The core can generate an
NLMSG_DONEby itself, if we decide this needs to be done.
Exactly.
Re: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 29 lines · Jakub Kicinski
Quoted 9 lines · On Tue, 19 Mar 2024 18:17:47 +0100 Eric Dumazet wrote:
Quoted 9 lines
Hi Stefano! I was worried this may happen :( I think we should revert offending commits, but I'd like to take it on case by case basis. I'd imagine majority of netlink is only exercised by iproute2 and libmnl-based tools. Does passt hang specifically on genetlink family dump? Your commit also mentions
RTM_GETROUTE. This is not the only commit which removed DONE:$ git log --since='1 month ago' --grep=NLMSG_DONE --no-merges --onelineQuoted 7 lines
Quoted 4 lines
9cc4cc329d30ipv6: usexa_arrayiterator to implementinet6_dump_addr()87d381973e49genetlink: fitNLMSG_DONEinto sameread()as families4ce5dc9316deinet: switchinet_dump_fib()to RCU protection6647b338fc5cnetlink: fixnetlink_diag_dump()return valueLets not bring back more RTNL locking please for the handlers that still require it.
Definitely. My git log copy/paste is pretty inaccurate, these two are better examples:
5d9b7cb383bbnexthop: Simplify dump error handling02e24903e5a4netlink: let core handle error cases in dump operationsI was trying to point out that we merged a handful of DONE "coalescing" patches, and if we need to revert - let's only do that for the exact commands needed. The comment was raised on my genetlink patch while the discussion in the link points to
RTM_GETROUTE.
Quoted 5 lines
Quoted 2 lines
The core can generate an
NLMSG_DONEby itself, if we decide this needs to be done.Exactly.
We've encountered a new issue recently which I believe is related to this discussion.
Following Eric's patch:9cc4cc329d30 ("ipv6: use xa_array iterator to implement inet6_dump_addr()")
Setting the interface mtu to < 1280 results in 'ip addr show eth2' returning an error, because the ipv6 dump fails. This is a degradation from the user's perspective.
# ip addr show eth2
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 24:42:53:21:52:44 brd ff:ff:ff:ff:ff:ff
altname enp6s0f0np0
# ip link set dev eth2 mtu 1000
# ip addr show eth2
RTNETLINK answers: No such device
Dump terminatedRe: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 10 lines · Gal Pressman
We've encountered a new issue recently which I believe is related to this discussion.
Following Eric's patch:
9cc4cc329d30("ipv6: usexa_arrayiterator to implementinet6_dump_addr()")Setting the interface mtu to < 1280 results in 'ip addr show eth2' returning an error, because the ipv6 dump fails. This is a degradation from the user's perspective.
Quoted 9 lines
# ip addr show eth2
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group
default qlen 1000
link/ether 24:42:53:21:52:44 brd ff:ff:ff:ff:ff:ff
altname enp6s0f0np0
# ip link set dev eth2 mtu 1000
# ip addr show eth2
RTNETLINK answers: No such device
Dump terminatedI don't think it's the same issue. Original issue was about user space not knowing how to handle NLMSG_DONE being sent together with dump responses. The issue you reported seems to be related to an unintentional change in the return code when IPv6 is disabled on an interface. Can you please test the following patch?
net/ipv6/addrconf.c +3 -2
@@ -5416,10 +5416,11 @@ static int inet6_dump_addr(struct sk_buff *skb, struct netlink_callback *cb, err =0 ;if (fillargs.ifindex) { - err = -ENODEV ; dev =dev_get_by_index_rcu (tgt_net, fillargs.ifindex); -if (!dev) +if (!dev) { + err = -ENODEV ; goto done; + } idev =__in6_dev_get (dev);if (idev) err = in6_dump_addrs(idev, skb, cb,
Re: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 35 lines · Ido Schimmel
Quoted 10 lines · Gal Pressman
We've encountered a new issue recently which I believe is related to this discussion.
Following Eric's patch:
9cc4cc329d30("ipv6: usexa_arrayiterator to implementinet6_dump_addr()")Setting the interface mtu to < 1280 results in 'ip addr show eth2' returning an error, because the ipv6 dump fails. This is a degradation from the user's perspective.
Quoted 9 lines
# ip addr show eth2 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 24:42:53:21:52:44 brd ff:ff:ff:ff:ff:ff altname enp6s0f0np0 # ip link set dev eth2 mtu 1000 # ip addr show eth2 RTNETLINK answers: No such device Dump terminatedI don't think it's the same issue. Original issue was about user space not knowing how to handle
NLMSG_DONEbeing sent together with dump responses. The issue you reported seems to be related to an unintentional change in the return code when IPv6 is disabled on an interface. Can you please test the following patch?net/ipv6/addrconf.c +0
-0@@ -5416,10 +5416,11 @@ static int inet6_dump_addr(struct sk_buff *skb, struct netlink_callback *cb, err =0 ;if (fillargs.ifindex) {
net/ipv6/addrconf.c static int inet6_dump_addr(struct sk_buff *skb, struct netlink_callback *cb, · line 5426
Quoted 10 lines
net/ipv6/addrconf.c +3
-2- err = -ENODEV ; dev =dev_get_by_index_rcu (tgt_net, fillargs.ifindex); -if (!dev) +if (!dev) { + err = -ENODEV ; goto done; + } idev =__in6_dev_get (dev);if (idev) err = in6_dump_addrs(idev, skb, cb,
This seems to fix it, thanks!
Will you submit a patch?
Eric Dumazet
Reviewed-byRe: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
In reply to Ido SchimmelQuoted 36 lines · Ido Schimmel
Quoted 10 lines · Gal Pressman
We've encountered a new issue recently which I believe is related to this discussion.
Following Eric's patch:
9cc4cc329d30("ipv6: usexa_arrayiterator to implementinet6_dump_addr()")Setting the interface mtu to < 1280 results in 'ip addr show eth2' returning an error, because the ipv6 dump fails. This is a degradation from the user's perspective.
Quoted 9 lines
# ip addr show eth2 4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000 link/ether 24:42:53:21:52:44 brd ff:ff:ff:ff:ff:ff altname enp6s0f0np0 # ip link set dev eth2 mtu 1000 # ip addr show eth2 RTNETLINK answers: No such device Dump terminatedI don't think it's the same issue. Original issue was about user space not knowing how to handle
NLMSG_DONEbeing sent together with dump responses. The issue you reported seems to be related to an unintentional change in the return code when IPv6 is disabled on an interface. Can you please test the following patch?net/ipv6/addrconf.c +0
-0@@ -5416,10 +5416,11 @@ static int inet6_dump_addr(struct sk_buff *skb, struct netlink_callback *cb, err =0 ;if (fillargs.ifindex) {
net/ipv6/addrconf.c static int inet6_dump_addr(struct sk_buff *skb, struct netlink_callback *cb, · line 5426
Quoted 10 lines
net/ipv6/addrconf.c +3
-2- err = -ENODEV ; dev =dev_get_by_index_rcu (tgt_net, fillargs.ifindex); -if (!dev) +if (!dev) { + err = -ENODEV ; goto done; + } idev =__in6_dev_get (dev);if (idev) err = in6_dump_addrs(idev, skb, cb,
Good catch, thanks !
Not sure why ip command is so picky about the error code here.
There is no IPv6 on this device after all.
The following seems to react quite differently :
# ip addr show lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host proto kernel_lo
valid_lft forever preferred_lft forever
# ip link set dev lo mtu 1000
# ip addr show lo
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 1000 qdisc noqueue state UNKNOWN
group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft foreverRe: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 17 lines · Eric Dumazet
The following seems to react quite differently :
# ip addr show lo 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host proto kernel_lo valid_lft forever preferred_lft forever # ip link set dev lo mtu 1000 # ip addr show lo 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 1000 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever
Yes, for loopback the NETDEV_CHANGEMTU event is treated as NETDEV_DOWN rather than NETDEV_UNREGISTER:
if (dev->mtu < IPV6_MIN_MTU) {
addrconf_ifdown(dev, dev != net->loopback_dev);
break;
}vim net/ipv6/addrconf.c +3656
Quoted 1 line
Reviewed-by Eric Dumazet <[email protected]>
Sorry Eric, already sent the patch without your tag:
Re: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
In reply to Jakub KicinskiQuoted 29 lines · Jakub Kicinski
Quoted 9 lines · On Tue, 19 Mar 2024 18:17:47 +0100 Eric Dumazet wrote:
Quoted 9 lines
Hi Stefano! I was worried this may happen :( I think we should revert offending commits, but I'd like to take it on case by case basis. I'd imagine majority of netlink is only exercised by iproute2 and libmnl-based tools. Does passt hang specifically on genetlink family dump? Your commit also mentions
RTM_GETROUTE. This is not the only commit which removed DONE:$ git log --since='1 month ago' --grep=NLMSG_DONE --no-merges --onelineQuoted 7 lines
Quoted 4 lines
9cc4cc329d30ipv6: usexa_arrayiterator to implementinet6_dump_addr()87d381973e49genetlink: fitNLMSG_DONEinto sameread()as families4ce5dc9316deinet: switchinet_dump_fib()to RCU protection6647b338fc5cnetlink: fixnetlink_diag_dump()return valueLets not bring back more RTNL locking please for the handlers that still require it.
Definitely. My git log copy/paste is pretty inaccurate, these two are better examples:
5d9b7cb383bbnexthop: Simplify dump error handling02e24903e5a4netlink: let core handle error cases in dump operationsI was trying to point out that we merged a handful of DONE "coalescing" patches, and if we need to revert - let's only do that for the exact commands needed. The comment was raised on my genetlink patch while the discussion in the link points to
RTM_GETROUTE.
Quoted 4 lines
Quoted 2 lines
The core can generate an
NLMSG_DONEby itself, if we decide this needs to be done.Exactly.
FWIW, it seems that Libreswan is suffering from the same issue on RTM_GETROUTE dump.
On 6.9.0-rc1 I see:
/usr/sbin/ipsec auto --config ipsec.conf --ctlsocket pluto.ctl \
--start --asynchronous tun-in-1recvfrom(7,
[
[{nlmsg_len=52, nlmsg_type=RTM_NEWROUTE, nlmsg_flags=NLM_F_MULTI, ...],
...
[{nlmsg_len=52, nlmsg_type=RTM_NEWROUTE, nlmsg_flags=NLM_F_MULTI, ...], [{nlmsg_len=20, nlmsg_type=NLMSG_DONE, nlmsg_flags=NLM_F_MULTI, ...]
], 40960, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [12])
recvfrom(7, <-- Stuck here forever
On 6.8.0 the output is following:
recvfrom(7,
[
[{nlmsg_len=52, nlmsg_type=RTM_NEWROUTE, nlmsg_flags=NLM_F_MULTI, ...],
...
[{nlmsg_len=52, nlmsg_type=RTM_NEWROUTE, nlmsg_flags=NLM_F_MULTI, ...]
], 40960, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [12])
recvfrom(7,
[{nlmsg_len=20, nlmsg_type=NLMSG_DONE, nlmsg_flags=NLM_F_MULTI,}, 0], 40728, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [12])
close(7)
So, it seems like it explicitly waits for NLMSG_DONE to be in a separate message.
I reported the issue to Libreswan:
https://github.com/libreswan/libreswan/issues/1675
but just wanted to let you know as well.
Found this since it breaks IPsec system tests in Open vSwitch.
Best regards, Ilya Maximets.
Jakub Kicinski
maintainerRe: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 11 lines · On Thu, 4 Apr 2024 00:52:15 +0200 Ilya Maximets wrote:
/usr/sbin/ipsec auto --config ipsec.conf --ctlsocket pluto.ctl \ --start --asynchronous tun-in-1recvfrom(7,
[
[{nlmsg_len=52, nlmsg_type=RTM_NEWROUTE, nlmsg_flags=NLM_F_MULTI, ...],
...
[{nlmsg_len=52, nlmsg_type=RTM_NEWROUTE, nlmsg_flags=NLM_F_MULTI, ...], [{nlmsg_len=20, nlmsg_type=NLMSG_DONE, nlmsg_flags=NLM_F_MULTI, ...]
], 40960, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [12])
Quoted 1 line
recvfrom(7, <-- Stuck here forever
I think we should probably fix this..
Would you mind sharing the sendmsg() call? Eyeballing rtnl_dump_all() - it does seem to coalesce DONE..
Re: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 12 lines · Jakub Kicinski
Quoted 11 lines · On Thu, 4 Apr 2024 00:52:15 +0200 Ilya Maximets wrote:
/usr/sbin/ipsec auto --config ipsec.conf --ctlsocket pluto.ctl \ --start --asynchronous tun-in-1recvfrom(7,
[
[{nlmsg_len=52, nlmsg_type=RTM_NEWROUTE, nlmsg_flags=NLM_F_MULTI, ...],
...
[{nlmsg_len=52, nlmsg_type=RTM_NEWROUTE, nlmsg_flags=NLM_F_MULTI, ...], [{nlmsg_len=20, nlmsg_type=NLMSG_DONE, nlmsg_flags=NLM_F_MULTI, ...]
], 40960, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [12])
Quoted 5 lines
Quoted 1 line
recvfrom(7, <-- Stuck here forever
I think we should probably fix this..
Would you mind sharing thesendmsg()call? Eyeballingrtnl_dump_all()- it does seem to coalesce DONE..
Sure, the whole sequence looks like this in strace:
socket(AF_NETLINK, SOCK_DGRAM|SOCK_CLOEXEC, NETLINK_ROUTE) = 7
sendto(7, [
{
nlmsg_len=36, nlmsg_type=0x1a /* NLMSG_??? */,
nlmsg_flags=NLM_F_REQUEST|0x300, nlmsg_seq=0, nlmsg_pid=138040 }, "\x02\x20\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x01\x00\x0a\x01\x01\x02"
], 36, 0, NULL, 0) = 36
recvfrom(7, [
[
{nlmsg_len=52, nlmsg_type=RTM_NEWROUTE, nlmsg_flags=NLM_F_MULTI, nlmsg_seq=0, nlmsg_pid=138040}, {rtm_family=AF_INET, rtm_dst_len=0, rtm_src_len=0, rtm_tos=0, rtm_table=RT_TABLE_MAIN,
rtm_protocol=RTPROT_BOOT, rtm_scope=RT_SCOPE_UNIVERSE, rtm_type=RTN_UNICAST, rtm_flags=0},
[
[{nla_len=8, nla_type=RTA_TABLE}, RT_TABLE_MAIN],
[{nla_len=8, nla_type=RTA_GATEWAY}, inet_addr("10.1.1.2")],
[{nla_len=8, nla_type=RTA_OIF}, if_nametoindex("p0")]
]
],
[
{nlmsg_len=60, nlmsg_type=RTM_NEWROUTE, nlmsg_flags=NLM_F_MULTI, nlmsg_seq=0, nlmsg_pid=138040},
{rtm_family=AF_INET, rtm_dst_len=24, rtm_src_len=0, rtm_tos=0, rtm_table=RT_TABLE_MAIN,
rtm_protocol=RTPROT_KERNEL, rtm_scope=RT_SCOPE_LINK, rtm_type=RTN_UNICAST, rtm_flags=0},
[
[{nla_len=8, nla_type=RTA_TABLE}, RT_TABLE_MAIN],
[{nla_len=8, nla_type=RTA_DST}, inet_addr("10.1.1.0")],
[{nla_len=8, nla_type=RTA_PREFSRC}, inet_addr("10.1.1.1")],
[{nla_len=8, nla_type=RTA_OIF}, if_nametoindex("p0")]
]
],
[
{nlmsg_len=60, nlmsg_type=RTM_NEWROUTE, nlmsg_flags=NLM_F_MULTI, nlmsg_seq=0, nlmsg_pid=138040},
{rtm_family=AF_INET, rtm_dst_len=32, rtm_src_len=0, rtm_tos=0, rtm_table=RT_TABLE_LOCAL,
rtm_protocol=RTPROT_KERNEL, rtm_scope=RT_SCOPE_HOST, rtm_type=RTN_LOCAL, rtm_flags=0},
[
[{nla_len=8, nla_type=RTA_TABLE}, RT_TABLE_LOCAL],
[{nla_len=8, nla_type=RTA_DST}, inet_addr("10.1.1.1")],
[{nla_len=8, nla_type=RTA_PREFSRC}, inet_addr("10.1.1.1")],
[{nla_len=8, nla_type=RTA_OIF}, if_nametoindex("p0")]
]
],
[
{nlmsg_len=60, nlmsg_type=RTM_NEWROUTE, nlmsg_flags=NLM_F_MULTI, nlmsg_seq=0, nlmsg_pid=138040},
{rtm_family=AF_INET, rtm_dst_len=32, rtm_src_len=0, rtm_tos=0, rtm_table=RT_TABLE_LOCAL,
rtm_protocol=RTPROT_KERNEL, rtm_scope=RT_SCOPE_LINK, rtm_type=RTN_BROADCAST, rtm_flags=0},
[
[{nla_len=8, nla_type=RTA_TABLE}, RT_TABLE_LOCAL],
[{nla_len=8, nla_type=RTA_DST}, inet_addr("10.1.1.255")],
[{nla_len=8, nla_type=RTA_PREFSRC}, inet_addr("10.1.1.1")],
[{nla_len=8, nla_type=RTA_OIF}, if_nametoindex("p0")] ]
],
[
{nlmsg_len=20, nlmsg_type=NLMSG_DONE, nlmsg_flags=NLM_F_MULTI, nlmsg_seq=0, nlmsg_pid=138040}, 0
]
], 40960, 0, {sa_family=AF_NETLINK, nl_pid=0, nl_groups=00000000}, [12]) = 252
recvfrom(7, <unfinished ...>
nlmsg_type=0x1a /* NLMSG_??? */ --> RTM_GETROUTE nlmsg_flags=NLM_F_REQUEST|0x300 --> NLM_F_REQUEST|NLM_F_DUMP
Best regards, Ilya Maximets.
Jakub Kicinski
maintainerRe: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 2 lines · On Thu, 11 Apr 2024 17:39:20 +0200 Ilya Maximets wrote:
nlmsg_type=0x1a /*
NLMSG_??? */ -->RTM_GETROUTEnlmsg_flags=NLM_F_REQUEST|0x300 --> NLM_F_REQUEST|NLM_F_DUMP
Thanks!
Also:
"\x02\x20\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x01\x00\x0a\x01\x01\x02"
^^^^
So it's dumping AF_INET. I'm guessing its also going to dump v6, separately? To fix v4 I think something like this would do (untested):
net/ipv4/fib_frontend.c +5 -0
@@ -1050,6 +1050,11 @@ static int inet_dump_fib(struct sk_buff *skb, struct netlink_callback *cb) e++; } } + +/* Don't let NLM_DONE coalesce into a message, even if it could. +* Some user space expects NLM_DONE in a separate recv(). +*/ + err = skb->len; out: cb->args[1 ] = e;
Re: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 6 lines · Jakub Kicinski
Quoted 2 lines · On Thu, 11 Apr 2024 17:39:20 +0200 Ilya Maximets wrote:
nlmsg_type=0x1a /*
NLMSG_??? */ -->RTM_GETROUTEnlmsg_flags=NLM_F_REQUEST|0x300 --> NLM_F_REQUEST|NLM_F_DUMPThanks!
Quoted 7 lines
Also:
"\x02\x20\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x08\x00\x01\x00\x0a\x01\x01\x02"
^^^^So it's dumping
AF_INET. I'm guessing its also going to dump v6, separately?
AFAICT, it dumps the family according to the tunnel configuration. If ipsec is set up for IPv4, then it dumps IPv4, if v6 - v6.
Quoted 8 lines
To fix v4 I think something like this would do (untested):
net/ipv4/fib_frontend.c +0
-0@@ -1050,6 +1050,11 @@ static int inet_dump_fib(struct sk_buff *skb, struct netlink_callback *cb) e++;
net/ipv4/fib_frontend.c static int inet_dump_fib(struct sk_buff *skb, struct netlink_callback *cb) · line 1060
Quoted 10 lines
net/ipv4/fib_frontend.c +5
-0} } + +/* Don't let NLM_DONE coalesce into a message, even if it could. +* Some user space expects NLM_DONE in a separate recv(). +*/ + err = skb->len; out: cb->args[1 ] = e;
I tried that and IPv4 tests with Libreswan are passing with this change on latest net-next/main.
IPv6 tests are stuck in the same way as IPv4 did before. The sendto as captured by strace is following:
sendto(7, [
{
nlmsg_len=48, nlmsg_type=0x1a /* NLMSG_??? */,
nlmsg_flags=NLM_F_REQUEST|0x300, nlmsg_seq=0, nlmsg_pid=17084}, "\x0a\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x14\x00\x01\x00\xfd\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x02"], 48, 0, NULL, 0) = 48
NLMSG_DONE is part of the first recvfrom and the process is stuck waiting for something from the second one.
Best regards, Ilya Maximets.
Jakub Kicinski
maintainerRe: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 7 lines · On Thu, 11 Apr 2024 18:38:19 +0200 Ilya Maximets wrote:
I tried that and IPv4 tests with Libreswan are passing with this change on latest net-next/main.
IPv6 tests are stuck in the same way as IPv4 did before. The sendto as captured by strace is following:
sendto(7, [
Quoted 8 lines
{ nlmsg_len=48, nlmsg_type=0x1a /* NLMSG_??? */, nlmsg_flags=NLM_F_REQUEST|0x300, nlmsg_seq=0, nlmsg_pid=17084}, "\x0a\x80\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x14\x00\x01\x00\xfd\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01\x02"], 48, 0, NULL, 0) = 48
NLMSG_DONEis part of the first recvfrom and the process is stuck waiting for something from the second one.
Perfect, thank you! I just sent the v4 fix, just to be clear you were testing on -next right? Because AFAICT the v6 change hasn't made it to Linus.
Jakub Kicinski
maintainerRe: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
Quoted 6 lines · On Thu, 11 Apr 2024 11:03:13 -0700 Jakub Kicinski wrote:
Quoted 2 lines
NLMSG_DONEis part of the first recvfrom and the process is stuck waiting for something from the second one.Perfect, thank you! I just sent the v4 fix, just to be clear you were testing on -next right? Because AFAICT the v6 change hasn't made it to Linus.
Oh, you said net-next/main earlier in your message, all makes sense.
Re: [PATCH net-next v2 3/3] genetlink: fit NLMSG_DONE into same read() as families
In reply to Jakub KicinskiQuoted 22 lines · Jakub Kicinski
Quoted 14 lines · On Fri, 15 Mar 2024 12:48:08 +0100 Stefano Brivio wrote:
Quoted 6 lines
Make sure
ctrl_fill_info()returns sensible error codes and propagate them out to netlink core. Let netlink core decide when to returnskb->lenand when to treat the exit as an error. Netlink core does better job at it, if we always returnskb->lenthe core doesn't know when we're done dumping andNLMSG_DONEends up in a separateread().While this change is obviously correct, it breaks... well, broken applications that _wrongly_ rely on the fact that
NLMSG_DONEis delivered in a separate datagram.This was the (embarrassing) case for passt(1), which I just fixed:
https://archives.passt.top/passt-dev/[email protected]/Quoted 6 lines
but the "separate"
NLMSG_DONEis such an established behaviour, I think, that this might raise a more general concern.From my perspective, I'm just happy that this change revealed the issue, but I wanted to report this anyway in case somebody has similar possible breakages in mind.
Quoted 6 lines
Hi Stefano! I was worried this may happen :( I think we should revert offending commits, but I'd like to take it on case by case basis. I'd imagine majority of netlink is only exercised by iproute2 and libmnl-based tools. Does passt hang specifically on genetlink family dump? Your commit also mentions
RTM_GETROUTE. This is not the only commit which removed DONE:
I don't think there's anything specirfic to RTM_GETROUTE here from the kernel side. We've looked at the problem in passt more closely now, and it turns out we handled a merged NLMSG_DONE correctly in most cases. For various reasons internal to passt, our handling of RTM_GETROUTE on one path is more complex, and we had a subtle error there which broke the handling of a merged NLMSG_DONE.
Quoted 8 lines
$ git log --since='1 month ago' --grep=NLMSG_DONE --no-merges --oneline
9cc4cc329d30ipv6: usexa_arrayiterator to implementinet6_dump_addr()87d381973e49genetlink: fitNLMSG_DONEinto sameread()as families4ce5dc9316deinet: switchinet_dump_fib()to RCU protection6647b338fc5cnetlink: fixnetlink_diag_dump()return value
Signature · 4 lines
David Gibson | I'll have my music baroque, and my code david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ | _way_ _around_! http://www.ozlabs.org/~dgibson
This series was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:
Hello:
This series was applied to netdev/net-next.git (main) by David S. Miller <[email protected]>:
Quoted 8 lines · On Sat, 2 Mar 2024 21:24:05 -0800 you wrote:
Ido discovered some time back that we usually force
NLMSG_DONEto be delivered in a separaterecv()syscall, even if it would fit into the same skb as data messages. He made nexthop try to fit DONE with data in commit8743aeff5bc4("nexthop: Fix infinite nexthop bucket dump when using maximum nexthop ID"), and nobody has complained so far.[...]
Here is the summary with links:
- [net-next,v2,1/3] netlink: handle EMSGSIZE errors in the core
https://git.kernel.org/netdev/net-next/c/b5a899154aa9 - [net-next,v2,2/3] netdev: let netlink core handle -EMSGSIZE errors https://git.kernel.org/netdev/net-next/c/0b11b1c5c320 - [net-next,v2,3/3] genetlink: fit NLMSG_DONE into same read() as families https://git.kernel.org/netdev/net-next/c/87d381973e49
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html