net: gro: fix misuse of CB in udp socket lookup
Subsystems (3)
v3 cover net: gro: fix misuse of CB in udp socket lookup
GRO stack uses udp_lib_lookup_skb which relies on IP/IPv6 CB's info, and at the GRO stage, CB holds napi_gro_cb info. Specifically, udp_lib_lookup_skb tries to fetch iff and flags information from the CB, to find the relevant udp tunnel socket (GENEVE/VXLAN/..). Up until a patch I submitted recently [0], it worked merely by luck, due to the layouts of napi_gro_cb and IP6CB.
AFAIU it worked because:IP6CB(skb)->flags is at offset 16 inside IP6CB:
- Before the patch: flags was mapped to flush.
- After the patch: flags was mapped to data_offset.
IP6CB(skb)->iff is at offset 0 inside IP6CB:
- Before the patch: iif was mapped to frag0.
- After the patch: iif was mapped to a union of frag0 and last.
After my patch, on the receive phase, while data_offset is 40 (since IPv6 header is 40 bytes), inet_iif calls ipv6_l3mdev_skb, which checks whether IP6CB(skb)->flags's IP6SKB_L3SLAVE bit is on or off (in our case its off). If it is off, inet_iif returns IP6CB(skb)->iif, which is mapped to napi_gro_cb->frag0, making inet_iif return 0 most of the times. inet_sdif returns zero due to a similar reason caused by data_offset being equal to 40 (and less than 64).
On the other hand, the complete phase behaves differently. data_offset is usually greater than 64 and less than 128 so the IP6SKB_L3SLAVE flag is on. Thus, inet_sdif returns IP6CB(skb)->iif, which is mapped to last which contains a pointer. This causes udp_sk_bound_dev_eq to fail, which leads to udp6_lib_lookup2 failing and not returning a socket. This leads the receive phase of GRO to find the right socket, and on the complete phase, it fails to find it and makes the throughput go down to nearly zero.
Before [0] flags was mapped to flush. flush's possible values were 1 and 0, making inet6_iff always returning skb->skb_iif and inet6_sdif returning 0, and leading to udp_sk_bound_dev_eq returning true.
A fix is to not rely on CB, and get iff and sdif using skb->dev. l3mdev case requires special attention since it has a master and a slave device.
[0] https://lore.kernel.org/netdev/20230601160924.GA9194@debian/
Changelog:
v2 -> v3:
* change functions to static inline in the header file
v1 -> v2:
* make functions inline
* fix logical bug
* add a comment when we can use the new functions
* checkpatch fixes
Richard Gobert (1):
net: gro: fix misuse of CB in udp socket lookup
include/net/gro.h | 43 ++++++++++++++++++++++++++++++++++++++++++ net/ipv4/udp.c | 8 ++++++-- net/ipv4/udp_offload.c | 7 +++++-- net/ipv6/udp.c | 8 ++++++-- net/ipv6/udp_offload.c | 7 +++++-- 5 files changed, 65 insertions(+), 8 deletions(-)
Signature · 1 line
2.36.1
PATCH v3 1/1 net: gro: fix misuse of CB in udp socket lookup
This patch fixes a misuse of IP{6}CB(skb) in GRO, while calling to udp6_lib_lookup2 when handling udp tunnels. udp6_lib_lookup2 fetch the device from CB. The fix changes it to fetch the device from skb->dev. l3mdev case requires special attention since it has a master and a slave device.
--- include/net/gro.h | 43 ++++++++++++++++++++++++++++++++++++++++++ net/ipv4/udp.c | 8 ++++++-- net/ipv4/udp_offload.c | 7 +++++-- net/ipv6/udp.c | 8 ++++++-- net/ipv6/udp_offload.c | 7 +++++-- 5 files changed, 65 insertions(+), 8 deletions(-)
include/net/gro.h +43 -0
@@ -452,6 +452,49 @@ static inline void gro_normal_one(struct napi_struct *napi, struct sk_buff *skb, gro_normal_list(napi); } +/* This function is the alternative of 'inet_iif' and 'inet_sdif' +* functions in case we can not rely on fields of IPCB. +* +* The caller must verify skb_valid_dst(skb) is false and skb->dev is initialized. +* The caller must hold the RCU read lock. +*/ +static inline void inet_get_iif_sdif (const struct sk_buff *skb,int *iif,int *sdif) +{ + *iif =inet_iif (skb) ?: skb->dev->ifindex; + *sdif =0 ; + +#if IS_ENABLED (CONFIG_NET_L3_MASTER_DEV ) +if (netif_is_l3_slave (skb->dev)) { +struct net_device *master =netdev_master_upper_dev_get_rcu (skb->dev); + + *sdif = *iif; + *iif = master ? master->ifindex :0 ; + } +#endif +} + +/* This function is the alternative of 'inet6_iif' and 'inet6_sdif' +* functions in case we can not rely on fields of IP6CB. +* +* The caller must verify skb_valid_dst(skb) is false and skb->dev is initialized. +* The caller must hold the RCU read lock. +*/ +static inline void inet6_get_iif_sdif (const struct sk_buff *skb,int *iif,int *sdif) +{ +/* using skb->dev->ifindex because skb_dst(skb) is not initialized */ + *iif = skb->dev->ifindex; + *sdif =0 ; + +#if IS_ENABLED (CONFIG_NET_L3_MASTER_DEV ) +if (netif_is_l3_slave (skb->dev)) { +struct net_device *master =netdev_master_upper_dev_get_rcu (skb->dev); + + *sdif = *iif; + *iif = master ? master->ifindex :0 ; + } +#endif +} +extern struct list_head offload_base;#endif /* _NET_IPV6_GRO_H */
net/ipv4/udp.c +6 -2
@@ -114,6 +114,7 @@#include <net/sock_reuseport.h> #include <net/addrconf.h> #include <net/udp_tunnel.h> +#include <net/gro.h> #if IS_ENABLED (CONFIG_IPV6 )#include <net/ipv6_stubs.h> #endif
@@ -555,10 +556,13 @@ struct sock *udp4_lib_lookup_skb(const struct sk_buff *skb, {const struct iphdr *iph =ip_hdr (skb);struct net *net =dev_net (skb->dev); +int iif, sdif; + +inet_get_iif_sdif (skb, &iif, &sdif);return __udp4_lib_lookup (net, iph->saddr, sport, - iph->daddr, dport,inet_iif (skb), -inet_sdif (skb), net->ipv4.udp_table,NULL ); + iph->daddr, dport, iif, + sdif, net->ipv4.udp_table,NULL ); }/* Must be called under rcu_read_lock().
net/ipv4/udp_offload.c +5 -2
@@ -603,10 +603,13 @@ static struct sock *udp4_gro_lookup_skb(struct sk_buff *skb, __be16 sport, {const struct iphdr *iph =skb_gro_network_header (skb);struct net *net =dev_net (skb->dev); +int iif, sdif; + +inet_get_iif_sdif (skb, &iif, &sdif);return __udp4_lib_lookup (net, iph->saddr, sport, - iph->daddr, dport,inet_iif (skb), -inet_sdif (skb), net->ipv4.udp_table,NULL ); + iph->daddr, dport, iif, + sdif, net->ipv4.udp_table,NULL ); }INDIRECT_CALLABLE_SCOPE
net/ipv6/udp.c +6 -2
@@ -51,6 +51,7 @@#include <net/inet6_hashtables.h> #include <net/busy_poll.h> #include <net/sock_reuseport.h> +#include <net/gro.h> #include <linux/proc_fs.h> #include <linux/seq_file.h>
@@ -300,10 +301,13 @@ struct sock *udp6_lib_lookup_skb(const struct sk_buff *skb, {const struct ipv6hdr *iph =ipv6_hdr (skb);struct net *net =dev_net (skb->dev); +int iif, sdif; + +inet6_get_iif_sdif (skb, &iif, &sdif);return __udp6_lib_lookup (net, &iph->saddr, sport, - &iph->daddr, dport,inet6_iif (skb), -inet6_sdif (skb), net->ipv4.udp_table,NULL ); + &iph->daddr, dport, iif, + sdif, net->ipv4.udp_table,NULL ); }/* Must be called under rcu_read_lock().
net/ipv6/udp_offload.c +5 -2
@@ -119,10 +119,13 @@ static struct sock *udp6_gro_lookup_skb(struct sk_buff *skb, __be16 sport, {const struct ipv6hdr *iph =skb_gro_network_header (skb);struct net *net =dev_net (skb->dev); +int iif, sdif; + +inet6_get_iif_sdif (skb, &iif, &sdif);return __udp6_lib_lookup (net, &iph->saddr, sport, - &iph->daddr, dport,inet6_iif (skb), -inet6_sdif (skb), net->ipv4.udp_table,NULL ); + &iph->daddr, dport, iif, + sdif, net->ipv4.udp_table,NULL ); }INDIRECT_CALLABLE_SCOPE
Signature · 1 line
2.36.1
Quoted 9 lines · Richard Gobert
This patch fixes a misuse of IP{6}CB(skb) in GRO, while calling to
udp6_lib_lookup2when handling udp tunnels.udp6_lib_lookup2fetch the device from CB. The fix changes it to fetch the device fromskb->dev. l3mdev case requires special attention since it has a master and a slave device.Fixes a6024562ffd7 ("udp: Add GRO functions to UDP socket")Reported-by Gal Pressman <[email protected]>Signed-off-by Richard Gobert <[email protected]>
Quoted 8 lines
--- include/net/gro.h | 43 ++++++++++++++++++++++++++++++++++++++++++ net/ipv4/udp.c | 8 ++++++-- net/ipv4/udp_offload.c | 7 +++++-- net/ipv6/udp.c | 8 ++++++-- net/ipv6/udp_offload.c | 7 +++++-- 5 files changed, 65 insertions(+), 8 deletions(-)
I still think the text in the cover letter should be in this commit message.
Code wise, it looks ok to me:
This patch was applied to netdev/net.git (main) by David S. Miller <[email protected]>:
Hello:
This patch was applied to netdev/net.git (main)
by David S. Miller <[email protected]>:
Quoted 8 lines · On Thu, 27 Jul 2023 17:25:11 +0200 you wrote:
GRO stack uses
udp_lib_lookup_skbwhich relies on IP/IPv6 CB's info, and at the GRO stage, CB holdsnapi_gro_cbinfo. Specifically,udp_lib_lookup_skbtries to fetchiffandflagsinformation from the CB, to find the relevant udp tunnel socket (GENEVE/VXLAN/..). Up until a patch I submitted recently [0], it worked merely by luck, due to the layouts ofnapi_gro_cband IP6CB.[...]
Here is the summary with links:
- [v3,1/1] net: gro: fix misuse of CB in udp socket lookup
https://git.kernel.org/netdev/net/c/7938cd154368
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html