net: fix lwtunnel reentry loops
Subsystems (5)
v2 cover net: fix lwtunnel reentry loops
v2:
- removed some patches from the -v1 series
- added a patch that was initially sent separately
- code style for the selftest (thanks Paolo)
v1:
- https://lore.kernel.org/all/[email protected]/
When the destination is the same after the transformation, we enter a lwtunnel loop. This is true for most of lwt users: ioam6, rpl, seg6, seg6_local, ila_lwt, and lwt_bpf. It can happen in their input() and output() handlers respectively, where either dst_input() or dst_output() is called at the end. It can also happen in xmit() handlers.
Here is an example for rpl_input():
dump_stack_lvl+0x60/0x80 rpl_input+0x9d/0x320 lwtunnel_input+0x64/0xa0 lwtunnel_input+0x64/0xa0 lwtunnel_input+0x64/0xa0 lwtunnel_input+0x64/0xa0 lwtunnel_input+0x64/0xa0 [...] lwtunnel_input+0x64/0xa0 lwtunnel_input+0x64/0xa0 lwtunnel_input+0x64/0xa0 lwtunnel_input+0x64/0xa0 lwtunnel_input+0x64/0xa0 ip6_sublist_rcv_finish+0x85/0x90 ip6_sublist_rcv+0x236/0x2f0
... until rpl_do_srh() fails, which means skb_cow_head() failed.
This series provides a fix at the core level of lwtunnel to catch such loops when they're not caught by the respective lwtunnel users, and handle the loop case in ioam6 which is one of the users. This series also comes with a new selftest to detect some dst cache reference loops in lwtunnel users.
Justin Iurman (3):
net: lwtunnel: fix recursion loops
net: ipv6: ioam6: fix lwtunnel_output() loop
selftests: net: test for lwtunnel dst ref loops
net/core/lwtunnel.c | 65 ++++- net/ipv6/ioam6_iptunnel.c | 8 +- tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/config | 2 + .../selftests/net/lwt_dst_cache_ref_loop.sh | 246 ++++++++++++++++++ 5 files changed, 306 insertions(+), 16 deletions(-) create mode 100755 tools/testing/selftests/net/lwt_dst_cache_ref_loop.sh
Signature · 1 line
2.34.1
PATCH v2 1/3 net: lwtunnel: fix recursion loops
This patch acts as a parachute, catch all solution, by detecting recursion loops in lwtunnel users and taking care of them (e.g., a loop between routes, a loop within the same route, etc). In general, such loops are the consequence of pathological configurations. Each lwtunnel user is still free to catch such loops early and do whatever they want with them. It will be the case in a separate patch for, e.g., seg6 and seg6_local, in order to provide drop reasons and update statistics. Another example of a lwtunnel user taking care of loops is ioam6, which has valid use cases that include loops (e.g., inline mode), and which is addressed by the next patch in this series. Overall, this patch acts as a last resort to catch loops and drop packets, since we don't want to leak something unintentionally because of a pathological configuration in lwtunnels.
The solution in this patch reuses dev_xmit_recursion(), dev_xmit_recursion_inc(), and dev_xmit_recursion_dec(), which seems fine considering the context.
---
Cc: Roopa Prabhu <[email protected]>
Cc: Andrea Mayer <[email protected]>
Cc: Stefano Salsano <[email protected]>
Cc: Ahmed Abdelsalam <[email protected]>
Cc: Ido Schimmel <[email protected]>
--- net/core/lwtunnel.c | 65 ++++++++++++++++++++++++++++++++++++--------- 1 file changed, 53 insertions(+), 12 deletions(-)
net/core/lwtunnel.c +53 -12
@@ -23,6 +23,8 @@#include <net/ip6_fib.h> #include <net/rtnh.h> +#include "dev.h" +DEFINE_STATIC_KEY_FALSE (nf_hooks_lwtunnel_enabled);EXPORT_SYMBOL_GPL (nf_hooks_lwtunnel_enabled);
@@ -325,13 +327,23 @@ EXPORT_SYMBOL_GPL(lwtunnel_cmp_encap);int lwtunnel_output (struct net *net,struct sock *sk,struct sk_buff *skb) { -struct dst_entry *dst =skb_dst (skb);const struct lwtunnel_encap_ops *ops;struct lwtunnel_state *lwtstate; -int ret = -EINVAL ; +struct dst_entry *dst; +int ret; + +if (dev_xmit_recursion ()) { +net_crit_ratelimited ("%s(): recursion limit reached on datapath\n" , + __func__); + ret = -ENETDOWN ; + goto drop; + } -if (!dst) + dst =skb_dst (skb); +if (!dst) { + ret = -EINVAL ; goto drop; + } lwtstate = dst->lwtstate; if (lwtstate->type == LWTUNNEL_ENCAP_NONE ||
@@ -341,8 +353,11 @@ int lwtunnel_output(struct net *net, struct sock *sk, struct sk_buff *skb) ret = -EOPNOTSUPP ;rcu_read_lock (); ops =rcu_dereference (lwtun_encaps[lwtstate->type]); -if (likely (ops && ops->output)) +if (likely (ops && ops->output)) { +dev_xmit_recursion_inc (); ret = ops->output (net, sk, skb); +dev_xmit_recursion_dec (); + }rcu_read_unlock ();if (ret == -EOPNOTSUPP )
@@ -359,13 +374,23 @@ EXPORT_SYMBOL_GPL(lwtunnel_output);int lwtunnel_xmit (struct sk_buff *skb) { -struct dst_entry *dst =skb_dst (skb);const struct lwtunnel_encap_ops *ops;struct lwtunnel_state *lwtstate; -int ret = -EINVAL ; +struct dst_entry *dst; +int ret; + +if (dev_xmit_recursion ()) { +net_crit_ratelimited ("%s(): recursion limit reached on datapath\n" , + __func__); + ret = -ENETDOWN ; + goto drop; + } -if (!dst) + dst =skb_dst (skb); +if (!dst) { + ret = -EINVAL ; goto drop; + } lwtstate = dst->lwtstate;
@@ -376,8 +401,11 @@ int lwtunnel_xmit(struct sk_buff *skb) ret = -EOPNOTSUPP ;rcu_read_lock (); ops =rcu_dereference (lwtun_encaps[lwtstate->type]); -if (likely (ops && ops->xmit)) +if (likely (ops && ops->xmit)) { +dev_xmit_recursion_inc (); ret = ops->xmit (skb); +dev_xmit_recursion_dec (); + }rcu_read_unlock ();if (ret == -EOPNOTSUPP )
@@ -394,13 +422,23 @@ EXPORT_SYMBOL_GPL(lwtunnel_xmit);int lwtunnel_input (struct sk_buff *skb) { -struct dst_entry *dst =skb_dst (skb);const struct lwtunnel_encap_ops *ops;struct lwtunnel_state *lwtstate; -int ret = -EINVAL ; +struct dst_entry *dst; +int ret; -if (!dst) +if (dev_xmit_recursion ()) { +net_crit_ratelimited ("%s(): recursion limit reached on datapath\n" , + __func__); + ret = -ENETDOWN ; goto drop; + } + + dst =skb_dst (skb); +if (!dst) { + ret = -EINVAL ; + goto drop; + } lwtstate = dst->lwtstate; if (lwtstate->type == LWTUNNEL_ENCAP_NONE ||
@@ -410,8 +448,11 @@ int lwtunnel_input(struct sk_buff *skb) ret = -EOPNOTSUPP ;rcu_read_lock (); ops =rcu_dereference (lwtun_encaps[lwtstate->type]); -if (likely (ops && ops->input)) +if (likely (ops && ops->input)) { +dev_xmit_recursion_inc (); ret = ops->input (skb); +dev_xmit_recursion_dec (); + }rcu_read_unlock ();if (ret == -EOPNOTSUPP )
Signature · 1 line
2.34.1
PATCH v2 2/3 net: ipv6: ioam6: fix lwtunnel_output() loop
In reply to Justin IurmanFix the lwtunnel_output() reentry loop in ioam6_iptunnel when the destination is the same after transformation. Note that a check on the destination address was already performed, but it was not enough. This is the example of a lwtunnel user taking care of loops without relying only on the last resort detection offered by lwtunnel.
---
Cc: Ido Schimmel <[email protected]>
--- net/ipv6/ioam6_iptunnel.c | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
net/ipv6/ioam6_iptunnel.c +4 -4
@@ -337,7 +337,6 @@ static int ioam6_do_encap(struct net *net, struct sk_buff *skb,static int ioam6_output (struct net *net,struct sock *sk,struct sk_buff *skb) {struct dst_entry *dst =skb_dst (skb), *cache_dst =NULL ; -struct in6_addr orig_daddr;struct ioam6_lwt *ilwt;int err = -EINVAL ;u32 pkt_cnt;
@@ -352,8 +351,6 @@ static int ioam6_output(struct net *net, struct sock *sk, struct sk_buff *skb)if (pkt_cnt % ilwt->freq.n >= ilwt->freq.k) goto out; - orig_daddr =ipv6_hdr (skb)->daddr; -local_bh_disable (); cache_dst =dst_cache_get (&ilwt->cache);local_bh_enable ();
@@ -422,7 +419,10 @@ static int ioam6_output(struct net *net, struct sock *sk, struct sk_buff *skb) goto drop; } -if (!ipv6_addr_equal (&orig_daddr, &ipv6_hdr (skb)->daddr)) { +/* avoid lwtunnel_output() reentry loop when destination is the same +* after transformation (e.g., with the inline mode) +*/ +if (dst->lwtstate != cache_dst->lwtstate) {skb_dst_drop (skb);skb_dst_set (skb, cache_dst);return dst_output (net, sk, skb);
Signature · 1 line
2.34.1
PATCH v2 3/3 selftests: net: test for lwtunnel dst ref loops
In reply to Justin IurmanAs recently specified by commit 0ea09cbf8350 ("docs: netdev: add a note on selftest posting") in net-next, the selftest is therefore shipped in this series. However, this selftest does not really test this series. It needs this series to avoid crashing the kernel. What it really tests, thanks to kmemleak, is what was fixed by the following commits:
- commit c71a192976de ("net: ipv6: fix dst refleaks in rpl, seg6 and ioam6 lwtunnels")
- commit 92191dd10730 ("net: ipv6: fix dst ref loops in rpl, seg6 and ioam6 lwtunnels")
- commit c64a0727f9b1 ("net: ipv6: fix dst ref loop on input in seg6 lwt")
- commit 13e55fbaec17 ("net: ipv6: fix dst ref loop on input in rpl lwt")
- commit 0e7633d7b95b ("net: ipv6: fix dst ref loop in ila lwtunnel")
- commit 5da15a9c11c1 ("net: ipv6: fix missing dst ref drop in ila lwtunnel")
---
Cc: Shuah Khan <[email protected]>
Cc: [email protected]
--- tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/config | 2 + .../selftests/net/lwt_dst_cache_ref_loop.sh | 246 ++++++++++++++++++ 3 files changed, 249 insertions(+) create mode 100755 tools/testing/selftests/net/lwt_dst_cache_ref_loop.sh
tools/testing/selftests/net/Makefile +1 -0
@@ -101,6 +101,7 @@ TEST_PROGS += vlan_bridge_binding.sh TEST_PROGS += bpf_offload.py TEST_PROGS += ipv6_route_update_soft_lockup.sh TEST_PROGS += busy_poll_test.sh +TEST_PROGS += lwt_dst_cache_ref_loop.sh # YNL files, must be before "include ..lib.mk" YNL_GEN_FILES := busy_poller netlink-dumps
tools/testing/selftests/net/config +2 -0
@@ -107,3 +107,5 @@ CONFIG_XFRM_INTERFACE=m CONFIG_XFRM_USER=m CONFIG_IP_NF_MATCH_RPFILTER=m CONFIG_IP6_NF_MATCH_RPFILTER=m +CONFIG_IPV6_ILA=m +CONFIG_IPV6_RPL_LWTUNNEL=y
tools/testing/selftests/net/lwt_dst_cache_ref_loop.sh new file +246 -0
246 lines not on this page · raw message
Signature · 1 line
2.34.1
Paolo Abeni
maintainerRe: [PATCH net v2 3/3] selftests: net: test for lwtunnel dst ref loops
Quoted 5 lines · Justin Iurman
As recently specified by commit
0ea09cbf8350("docs: netdev: add a note on selftest posting") in net-next, the selftest is therefore shipped in this series. However, this selftest does not really test this series. It needs this series to avoid crashing the kernel. What it really tests, thanks to kmemleak,
As a net-next follow-up you could force a kmemleak scan and check the result after each test case to really output a pass/fail message.
Also, still for net-next, please investigate if dropping or reducing the many sleep below could be possible (it's not clear to me why they are needed).
I'll take is as-is to avoid blocking the fixes for trivial matters.
Thanks,
Paolo
Re: [PATCH net v2 3/3] selftests: net: test for lwtunnel dst ref loops
Quoted 9 lines · Paolo Abeni
Quoted 5 lines · Justin Iurman
As recently specified by commit
0ea09cbf8350("docs: netdev: add a note on selftest posting") in net-next, the selftest is therefore shipped in this series. However, this selftest does not really test this series. It needs this series to avoid crashing the kernel. What it really tests, thanks to kmemleak,As a net-next follow-up you could force a kmemleak scan and check the result after each test case to really output a pass/fail message.
Sure. Could you point me to an example doing that? Not sure how I'd force it within the selftest.
Quoted 3 lines
Also, still for net-next, please investigate if dropping or reducing the many sleep below could be possible (it's not clear to me why they are needed).
Of course, will do.
Quoted 1 line
I'll take is as-is to avoid blocking the fixes for trivial matters.
Thanks,
Justin
Quoted 4 lines
Thanks,
Paolo
Re: [PATCH net v2 3/3] selftests: net: test for lwtunnel dst ref loops: manual merge
In reply to Justin IurmanHi Justin,
Quoted 45 lines · Justin Iurman
As recently specified by commit
0ea09cbf8350("docs: netdev: add a note on selftest posting") in net-next, the selftest is therefore shipped in this series. However, this selftest does not really test this series. It needs this series to avoid crashing the kernel. What it really tests, thanks to kmemleak, is what was fixed by the following commits:
- commitc71a192976de("net: ipv6: fix dst refleaks in rpl, seg6 and ioam6 lwtunnels")
- commit92191dd10730("net: ipv6: fix dst ref loops in rpl, seg6 and ioam6 lwtunnels")
- commitc64a0727f9b1("net: ipv6: fix dst ref loop on input in seg6 lwt")
- commit13e55fbaec17("net: ipv6: fix dst ref loop on input in rpl lwt")
- commit0e7633d7b95b("net: ipv6: fix dst ref loop in ila lwtunnel")
- commit5da15a9c11c1("net: ipv6: fix missing dst ref drop in ila lwtunnel")Signed-off-by Justin Iurman <[email protected]>---
Cc: Shuah Khan <[email protected]>
Cc: [email protected]--- tools/testing/selftests/net/Makefile | 1 + tools/testing/selftests/net/config | 2 + .../selftests/net/lwt_dst_cache_ref_loop.sh | 246 ++++++++++++++++++ 3 files changed, 249 insertions(+) create mode 100755 tools/testing/selftests/net/lwt_dst_cache_ref_loop.shtools/testing/selftests/net/Makefile +1
-0@@ -101,6 +101,7 @@ TEST_PROGS += vlan_bridge_binding.sh TEST_PROGS += bpf_offload.py TEST_PROGS += ipv6_route_update_soft_lockup.sh TEST_PROGS += busy_poll_test.sh +TEST_PROGS += lwt_dst_cache_ref_loop.sh # YNL files, must be before "include ..lib.mk" YNL_GEN_FILES := busy_poller netlink-dumpstools/testing/selftests/net/config +0
-0
tools/testing/selftests/net/config CONFIG_XFRM_INTERFACE=m · line 111
Quoted 5 lines
tools/testing/selftests/net/config +2
-0@@ -107,3 +107,5 @@ CONFIG_XFRM_INTERFACE=m CONFIG_XFRM_USER=m CONFIG_IP_NF_MATCH_RPFILTER=m CONFIG_IP6_NF_MATCH_RPFILTER=m +CONFIG_IPV6_ILA=m +CONFIG_IPV6_RPL_LWTUNNEL=y
FYI, we got a small conflict when merging 'net' in 'net-next' in the MPTCP tree due to this patch applied in 'net':
3ed61b8938c6 ("selftests: net: test for lwtunnel dst ref loops")
and these ones from 'net-next':
03544faad761 ("selftest: net: add proc_net_pktgen") 85cb3711acb8 ("selftests: net: Add test cases for link and peer netns")
----- Generic Message -----
The best is to avoid conflicts between 'net' and 'net-next' trees but if they cannot be avoided when preparing patches, a note about how to fix them is much appreciated.
The conflict has been resolved on our side[1] and the resolution we suggest is attached to this email. Please report any issues linked to this conflict resolution as it might be used by others. If you worked on the mentioned patches, don't hesitate to ACK this conflict resolution.
---------------------------
Regarding these conflicts, that's because the new test and configs have been added at the end. It is usually recommended to avoid that for long list, e.g. inserting new entries at the right place if the list is sorted by alphabetical order (not the case here), grouping by theme, or keeping the same order as in the .config, etc. For the resolutions here, I simply kept the new tests and configs from both sides at the same place, adding the new ones from net at the end.
Rerere cache is available in [2].
Cheers,
Matt
[1] https://github.com/multipath-tcp/mptcp_net-next/commit/89220b9164c7
[2] https://github.com/multipath-tcp/mptcp-upstream-rr-cache/commit/d240
Cheers,
Matt
Signature · 1 line
Sponsored by the NGI0 Core fund.
tools/testing/selftests/net/Makefile +0 -0
@@@ -31,15 -31,11 +31,14 @@@ TEST_PROGS += veth.s TEST_PROGS += ioam6.sh TEST_PROGS += gro.sh TEST_PROGS += gre_gso.sh - TEST_PROGS += gre_ipv6_lladdr.sh TEST_PROGS += cmsg_so_mark.sh TEST_PROGS += cmsg_so_priority.sh -TEST_PROGS += cmsg_time.sh cmsg_ipv6.sh +TEST_PROGS += test_so_rcv.sh +TEST_PROGS += cmsg_time.sh cmsg_ip.sh TEST_PROGS += netns-name.sh +TEST_PROGS += link_netns.py TEST_PROGS += nl_netdev.py +TEST_PROGS += rtnetlink.py TEST_PROGS += srv6_end_dt46_l3vpn_test.sh TEST_PROGS += srv6_end_dt4_l3vpn_test.sh TEST_PROGS += srv6_end_dt6_l3vpn_test.sh TEST_PROGS += bpf_offload.py TEST_PROGS += ipv6_route_update_soft_lockup.sh TEST_PROGS += busy_poll_test.sh +TEST_GEN_PROGS += proc_net_pktgen + TEST_PROGS += lwt_dst_cache_ref_loop.sh # YNL files, must be before "include ..lib.mk" YNL_GEN_FILES := busy_poller netlink-dumps
tools/testing/selftests/net/config +0 -0
@@@ -109,9 -107,5 +109,11 @@@ CONFIG_XFRM_INTERFACE= CONFIG_XFRM_USER=m CONFIG_IP_NF_MATCH_RPFILTER=m CONFIG_IP6_NF_MATCH_RPFILTER=m +CONFIG_IPVLAN=m +CONFIG_CAN=m +CONFIG_CAN_DEV=m +CONFIG_CAN_VXCAN=m +CONFIG_NETKIT=y +CONFIG_NET_PKTGEN=m + CONFIG_IPV6_ILA=m + CONFIG_IPV6_RPL_LWTUNNEL=y
This series was applied to netdev/net.git (main) by Paolo Abeni <[email protected]>:
Hello:
This series was applied to netdev/net.git (main)
by Paolo Abeni <[email protected]>:
Quoted 8 lines · On Fri, 14 Mar 2025 13:00:45 +0100 you wrote:
v2:
- removed some patches from the -v1 series
- added a patch that was initially sent separately
- code style for the selftest (thanks Paolo)
v1:
- https://lore.kernel.org/all/[email protected]/[...]
Here is the summary with links:
- [net,v2,1/3] net: lwtunnel: fix recursion loops
https://git.kernel.org/netdev/net/c/986ffb3a57c5 - [net,v2,2/3] net: ipv6: ioam6: fix lwtunnel_output() loop https://git.kernel.org/netdev/net/c/3e7a60b368ea - [net,v2,3/3] selftests: net: test for lwtunnel dst ref loops https://git.kernel.org/netdev/net/c/3ed61b8938c6
You are awesome, thank you!
Signature · 2 lines
Deet-doot-dot, I am a bot. https://korg.docs.kernel.org/patchwork/pwbot.html
Quoted 6 lines · Justin Iurman
v2:
- removed some patches from the -v1 series
- added a patch that was initially sent separately
- code style for the selftest (thanks Paolo)
v1:
- https://lore.kernel.org/all/[email protected]/
Hi Justin,
I've noticed a BUG splat likely introduced by this patch. The splat is reported when executing some BPF selftests, e.g. lwt_ip_encap_ipv4/egress
(defined in tools/testing/selftests/bpf/prog_tests/lwt_ip_encap.c and
tools/testing/selftests/bpf/progs/test_lwt_ip_encap.c).
Decoded splat is at the end of the email.
Line numbers correspond to commita27a97f71394 ("Merge branch 'bpf-support-atomic-update-for-htab-of-maps'") from the kernel/git/bpf/bpf-next.git tree.
Thanks,
Eduard
---
[ 193.993893] BUG: using __this_cpu_add() in preemptible [00000000] code: test_progs/206 [ 193.994292] caller is lwtunnel_xmit (net/core/dev.h:340 net/core/lwtunnel.c:408) [ 193.994601] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE [ 193.994603] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-4.el9 04/01/2014 [ 193.994605] Call Trace: [ 193.994608] <TASK> [ 193.994611] dump_stack_lvl (lib/dump_stack.c:122) [ 193.994622] check_preemption_disabled (lib/smp_processor_id.c:0) [ 193.994630] ? lwtunnel_xmit (./include/linux/rcupdate.h:331 ./include/linux/rcupdate.h:841 net/core/lwtunnel.c:403) [ 193.994637] lwtunnel_xmit (net/core/dev.h:340 net/core/lwtunnel.c:408) [ 193.994648] ip_finish_output2 (net/ipv4/ip_output.c:222) [ 193.994655] ? ip_skb_dst_mtu (./include/net/ip.h:517) [ 193.994659] ? ip_skb_dst_mtu (./include/linux/rcupdate.h:331 ./include/linux/rcupdate.h:841 ./include/net/ip.h:471 ./include/net/ip.h:512) [ 193.994669] ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:182) [ 193.994672] ? __ip_finish_output (net/ipv4/ip_output.c:306) [ 193.994683] ? __ip_queue_xmit (./include/linux/rcupdate.h:331 ./include/linux/rcupdate.h:841 net/ipv4/ip_output.c:470) [ 193.994688] __ip_queue_xmit (net/ipv4/ip_output.c:527) [ 193.994693] ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:182) [ 193.994711] ? __ip_queue_xmit (./include/linux/rcupdate.h:331 ./include/linux/rcupdate.h:841 net/ipv4/ip_output.c:470) [ 193.994726] __tcp_transmit_skb (net/ipv4/tcp_output.c:1479) [ 193.994800] ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:182) [ 193.994804] ? __asan_memset (mm/kasan/shadow.c:84) [ 193.994810] ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:182) [ 193.994824] tcp_connect (net/ipv4/tcp_output.c:0 net/ipv4/tcp_output.c:4155) [ 193.994890] tcp_v4_connect (net/ipv4/tcp_ipv4.c:343) [ 193.994926] __inet_stream_connect (net/ipv4/af_inet.c:678) [ 193.994944] ? __local_bh_enable_ip (./arch/x86/include/asm/irqflags.h:42 ./arch/x86/include/asm/irqflags.h:119 kernel/softirq.c:412) [ 193.994950] ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:182) [ 193.994953] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4473) [ 193.994967] inet_stream_connect (net/ipv4/af_inet.c:748) [ 193.994976] ? __pfx_inet_stream_connect (net/ipv4/af_inet.c:744) [ 193.994981] __sys_connect (./include/linux/file.h:62 ./include/linux/file.h:83 net/socket.c:2058) [ 193.995013] __x64_sys_connect (net/socket.c:2063 net/socket.c:2060 net/socket.c:2060) [ 193.995022] do_syscall_64 (arch/x86/entry/syscall_64.c:0) [ 193.995026] ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:182) [ 193.995030] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4473) [ 193.995038] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130) [ 193.995042] RIP: 0033:0x7faec2d0f9cb [ 193.995047] Code: 83 ec 18 89 54 24 0c 48 89 34 24 89 7c 24 08 e8 4b 70 f7 ff 8b 54 24 0c 48 8b 34 24 41 89 c0 8b 7c 24 08 b8 2a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 08 e8 a1 70 f7 ff 8b 44 All code ======== 0: 83 ec 18 sub $0x18,%esp 3: 89 54 24 0c mov %edx,0xc(%rsp) 7: 48 89 34 24 mov %rsi,(%rsp) b: 89 7c 24 08 mov %edi,0x8(%rsp) f: e8 4b 70 f7 ff call 0xfffffffffff7705f 14: 8b 54 24 0c mov 0xc(%rsp),%edx 18: 48 8b 34 24 mov (%rsp),%rsi 1c: 41 89 c0 mov %eax,%r8d 1f: 8b 7c 24 08 mov 0x8(%rsp),%edi 23: b8 2a 00 00 00 mov $0x2a,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 35 ja 0x67 32: 44 89 c7 mov %r8d,%edi 35: 89 44 24 08 mov %eax,0x8(%rsp) 39: e8 a1 70 f7 ff call 0xfffffffffff770df 3e: 8b .byte 0x8b 3f: 44 rex.R
Code starting with the faulting instruction
===========================================
0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 35 ja 0x3d 8: 44 89 c7 mov %r8d,%edi b: 89 44 24 08 mov %eax,0x8(%rsp) f: e8 a1 70 f7 ff call 0xfffffffffff770b5 14: 8b .byte 0x8b 15: 44 rex.R [ 193.995050] RSP: 002b:00007fff992d3a20 EFLAGS: 00000293 ORIG_RAX: 000000000000002a [ 193.995054] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007faec2d0f9cb [ 193.995057] RDX: 0000000000000010 RSI: 00007fff992d3ad8 RDI: 0000000000000035 [ 193.995059] RBP: 00007fff992d3ac0 R08: 0000000000000000 R09: 0000000000000004 [ 193.995062] R10: 00007fff992d39b0 R11: 0000000000000293 R12: 00007fff992d7b78 [ 193.995064] R13: 000000000095f760 R14: 0000000002e38b90 R15: 00007faec373d000 [ 193.995091] </TASK>
Quoted 17 lines · Eduard Zingerman
Quoted 6 lines · Justin Iurman
v2:
- removed some patches from the -v1 series
- added a patch that was initially sent separately
- code style for the selftest (thanks Paolo)
v1:
- https://lore.kernel.org/all/[email protected]/Hi Justin,
I've noticed a BUG splat likely introduced by this patch. The splat is reported when executing some BPF selftests, e.g. lwt_ip_encap_ipv4/egress
(defined intools/testing/selftests/bpf/prog_tests/lwt_ip_encap.cand
tools/testing/selftests/bpf/progs/test_lwt_ip_encap.c).
Quoted 4 lines
Decoded splat is at the end of the email.
Line numbers correspond to commita27a97f71394("Merge branch 'bpf-support-atomic-update-for-htab-of-maps'") from thekernel/git/bpf/bpf-next.gittree.
Hi Eduard,
This is currently discussed in [1]. We have a solution, but no consensus on the fix yet.
[1]
https://lore.kernel.org/netdev/[email protected]/T/#t
Thanks for the report,
Justin
Quoted 82 lines
Thanks,
Eduard---
[ 193.993893] BUG: using __this_cpu_add() in preemptible [00000000] code: test_progs/206 [ 193.994292] caller is lwtunnel_xmit (net/core/dev.h:340 net/core/lwtunnel.c:408) [ 193.994601] Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE [ 193.994603] Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.3-4.el9 04/01/2014 [ 193.994605] Call Trace: [ 193.994608] <TASK> [ 193.994611] dump_stack_lvl (lib/dump_stack.c:122) [ 193.994622] check_preemption_disabled (lib/smp_processor_id.c:0) [ 193.994630] ? lwtunnel_xmit (./include/linux/rcupdate.h:331 ./include/linux/rcupdate.h:841 net/core/lwtunnel.c:403) [ 193.994637] lwtunnel_xmit (net/core/dev.h:340 net/core/lwtunnel.c:408) [ 193.994648] ip_finish_output2 (net/ipv4/ip_output.c:222) [ 193.994655] ? ip_skb_dst_mtu (./include/net/ip.h:517) [ 193.994659] ? ip_skb_dst_mtu (./include/linux/rcupdate.h:331 ./include/linux/rcupdate.h:841 ./include/net/ip.h:471 ./include/net/ip.h:512) [ 193.994669] ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:182) [ 193.994672] ? __ip_finish_output (net/ipv4/ip_output.c:306) [ 193.994683] ? __ip_queue_xmit (./include/linux/rcupdate.h:331 ./include/linux/rcupdate.h:841 net/ipv4/ip_output.c:470) [ 193.994688] __ip_queue_xmit (net/ipv4/ip_output.c:527) [ 193.994693] ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:182) [ 193.994711] ? __ip_queue_xmit (./include/linux/rcupdate.h:331 ./include/linux/rcupdate.h:841 net/ipv4/ip_output.c:470) [ 193.994726] __tcp_transmit_skb (net/ipv4/tcp_output.c:1479) [ 193.994800] ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:182) [ 193.994804] ? __asan_memset (mm/kasan/shadow.c:84) [ 193.994810] ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:182) [ 193.994824] tcp_connect (net/ipv4/tcp_output.c:0 net/ipv4/tcp_output.c:4155) [ 193.994890] tcp_v4_connect (net/ipv4/tcp_ipv4.c:343) [ 193.994926] __inet_stream_connect (net/ipv4/af_inet.c:678) [ 193.994944] ? __local_bh_enable_ip (./arch/x86/include/asm/irqflags.h:42 ./arch/x86/include/asm/irqflags.h:119 kernel/softirq.c:412) [ 193.994950] ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:182) [ 193.994953] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4473) [ 193.994967] inet_stream_connect (net/ipv4/af_inet.c:748) [ 193.994976] ? __pfx_inet_stream_connect (net/ipv4/af_inet.c:744) [ 193.994981] __sys_connect (./include/linux/file.h:62 ./include/linux/file.h:83 net/socket.c:2058) [ 193.995013] __x64_sys_connect (net/socket.c:2063 net/socket.c:2060 net/socket.c:2060) [ 193.995022] do_syscall_64 (arch/x86/entry/syscall_64.c:0) [ 193.995026] ? srso_alias_return_thunk (arch/x86/lib/retpoline.S:182) [ 193.995030] ? lockdep_hardirqs_on (kernel/locking/lockdep.c:4473) [ 193.995038] entry_SYSCALL_64_after_hwframe (arch/x86/entry/entry_64.S:130) [ 193.995042] RIP: 0033:0x7faec2d0f9cb [ 193.995047] Code: 83 ec 18 89 54 24 0c 48 89 34 24 89 7c 24 08 e8 4b 70 f7 ff 8b 54 24 0c 48 8b 34 24 41 89 c0 8b 7c 24 08 b8 2a 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 35 44 89 c7 89 44 24 08 e8 a1 70 f7 ff 8b 44 All code ======== 0: 83 ec 18 sub $0x18,%esp 3: 89 54 24 0c mov %edx,0xc(%rsp) 7: 48 89 34 24 mov %rsi,(%rsp) b: 89 7c 24 08 mov %edi,0x8(%rsp) f: e8 4b 70 f7 ff call 0xfffffffffff7705f 14: 8b 54 24 0c mov 0xc(%rsp),%edx 18: 48 8b 34 24 mov (%rsp),%rsi 1c: 41 89 c0 mov %eax,%r8d 1f: 8b 7c 24 08 mov 0x8(%rsp),%edi 23: b8 2a 00 00 00 mov $0x2a,%eax 28: 0f 05 syscall 2a:* 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax <-- trapping instruction 30: 77 35 ja 0x67 32: 44 89 c7 mov %r8d,%edi 35: 89 44 24 08 mov %eax,0x8(%rsp) 39: e8 a1 70 f7 ff call 0xfffffffffff770df 3e: 8b .byte 0x8b 3f: 44 rex.RCode starting with the faulting instruction
===========================================0: 48 3d 00 f0 ff ff cmp $0xfffffffffffff000,%rax 6: 77 35 ja 0x3d 8: 44 89 c7 mov %r8d,%edi b: 89 44 24 08 mov %eax,0x8(%rsp) f: e8 a1 70 f7 ff call 0xfffffffffff770b5 14: 8b .byte 0x8b 15: 44 rex.R [ 193.995050] RSP: 002b:00007fff992d3a20 EFLAGS: 00000293 ORIG_RAX: 000000000000002a [ 193.995054] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007faec2d0f9cb [ 193.995057] RDX: 0000000000000010 RSI: 00007fff992d3ad8 RDI: 0000000000000035 [ 193.995059] RBP: 00007fff992d3ac0 R08: 0000000000000000 R09: 0000000000000004 [ 193.995062] R10: 00007fff992d39b0 R11: 0000000000000293 R12: 00007fff992d7b78 [ 193.995064] R13: 000000000095f760 R14: 0000000002e38b90 R15: 00007faec373d000 [ 193.995091] </TASK>
Justin Iurman <[email protected]> writes:
[...]
Quoted 5 lines
This is currently discussed in [1]. We have a solution, but no consensus on the fix yet.
Hi Justin,
Thank you for the link, apologies for the noise.
I didn't look for existing reports properly.
[...]