ipvlan: add xmit recursion protection
Subsystems (3)
PATCH 1/1 ipvlan: add xmit recursion protection
From: Tristan Madani <[email protected]>
ipvlan devices can enter infinite transmit recursion when combined with packet forwarding configurations (such as IPVS) that route traffic back through the same ipvlan interface.
The recursion path is:
ipvlan_start_xmit -> ipvlan_queue_xmit -> ipvlan_xmit_mode_l3 -> ipvlan_process_outbound -> ip_local_out -> netfilter hooks -> dev_queue_xmit -> ipvlan_start_xmit (recurse)
The existing per-CPU xmit recursion counter in __dev_queue_xmit() (XMIT_RECURSION_LIMIT = 8) does detect the loop, but fires too late:
each recursion level consumes roughly 2KB of stack space through ip_local_out and netfilter, and at 8 levels the cumulative usage exceeds the 16KB kernel stack on x86_64. The resulting stack overflow hits the VMAP_STACK guard page and causes a kernel panic.
Add a per-CPU counter that prevents any re-entry into ipvlan_queue_xmit() while it is already executing on the same CPU. This mirrors the approach used by tunnel devices (see IP_TUNNEL_RECURSION_LIMIT in ip_tunnels.h) but with a stricter limit appropriate for ipvlan.
--- drivers/net/ipvlan/ipvlan_core.c | 25 +++++++++++++++++++++---- 1 file changed, 21 insertions(+), 4 deletions(-)
drivers/net/ipvlan/ipvlan_core.c +21 -4
@@ -9,6 +9,8 @@static u32 ipvlan_jhash_secret__read_mostly ; +static DEFINE_PER_CPU(int, ipvlan_xmit_depth); +void ipvlan_init_secret (void ) {net_get_random_once (&ipvlan_jhash_secret,sizeof (ipvlan_jhash_secret));
@@ -676,6 +678,7 @@ int ipvlan_queue_xmit(struct sk_buff *skb, struct net_device *dev) {struct ipvl_dev *ipvlan =netdev_priv (dev);struct ipvl_port *port =ipvlan_port_get_rcu_bh (ipvlan->phy_dev); +int ret =NET_XMIT_DROP ;if (!port) goto out;
@@ -683,18 +686,32 @@ int ipvlan_queue_xmit(struct sk_buff *skb, struct net_device *dev)if (unlikely (!pskb_may_pull (skb,sizeof (struct ethhdr )))) goto out; +if (this_cpu_read (ipvlan_xmit_depth)) { +net_crit_ratelimited ("ipvlan: xmit recursion detected on dev %s\n" , + dev->name); + goto out; + } + +this_cpu_inc (ipvlan_xmit_depth);switch (port->mode) {case IPVLAN_MODE_L2 : -return ipvlan_xmit_mode_l2 (skb, dev); + ret =ipvlan_xmit_mode_l2 (skb, dev); +break ;case IPVLAN_MODE_L3 :#ifdef CONFIG_IPVLAN_L3S case IPVLAN_MODE_L3S :#endif -return ipvlan_xmit_mode_l3 (skb, dev); + ret =ipvlan_xmit_mode_l3 (skb, dev); +break ; +default : +WARN_ONCE (true,"%s called for mode = [%x]\n" , + __func__, port->mode); +kfree_skb (skb); +break ; } +this_cpu_dec (ipvlan_xmit_depth); +return ret; -/* Should not reach here */ -WARN_ONCE (true,"%s called for mode = [%x]\n" , __func__, port->mode); out:kfree_skb (skb);return NET_XMIT_DROP ;
Signature · 1 line
2.47.3
Quoted 13 lines · Tristan Madani
From: Tristan Madani <[email protected]>
ipvlan devices can enter infinite transmit recursion when combined with packet forwarding configurations (such as IPVS) that route traffic back through the same ipvlan interface.
The recursion path is:
ipvlan_start_xmit->ipvlan_queue_xmit->ipvlan_xmit_mode_l3->ipvlan_process_outbound->ip_local_out-> netfilter hooks ->dev_queue_xmit->ipvlan_start_xmit(recurse)
Quoted 7 lines
The existing per-CPU xmit recursion counter in
__dev_queue_xmit()(XMIT_RECURSION_LIMIT= 8) does detect the loop, but fires too late:
each recursion level consumes roughly 2KB of stack space throughip_local_outand netfilter, and at 8 levels the cumulative usage exceeds the 16KB kernel stack onx86_64. The resulting stack overflow hits theVMAP_STACKguard page and causes a kernel panic.
Please fix the existing mechanism. 2KB of stack space seems excessive.
If this is caused by some DEBUG option, it will be just fine to limit XMIT_RECURSION_LIMIT to 3 for such debug kernels.
Adding a workaround in every virtual driver is not an option.
PATCH v2 1/1 net: reduce XMIT_RECURSION_LIMIT under KASAN
In reply to Tristan MadaniFrom: Tristan Madani <[email protected]>
Virtual network devices (ipvlan, macvlan, bonding) can enter legitimate transmit recursion when combined with packet forwarding configurations such as IPVS NAT. The existing XMIT_RECURSION_LIMIT (8) in __dev_queue_xmit() detects and breaks these loops, but the allowed depth is too high for KASAN-instrumented kernels: each recursion level consumes significantly more stack due to KASAN inline instrumentation, and the cumulative usage overflows the kernel stack before the limit fires.
On x86_64, CONFIG_KASAN doubles THREAD_SIZE from 16KB to 32KB (KASAN_STACK_ORDER=1), but KASAN per-access checks inflate individual function frames by roughly 2-3x. For an ipvlan L3 + IPVS NAT routing loop, objdump measurements on a non-KASAN kernel show ~1.4KB of stack consumed per recursion level (across 17 functions from __dev_queue_xmit through the full IP output path and back). At KASAN ~2.3x inflation factor that becomes ~3.3KB per level. Nine levels -- reached before the current limit fires -- total ~30KB plus the initial call chain, which exceeds the 32KB KASAN stack. The overflow hits the VMAP_STACK guard page and causes a non-recoverable kernel panic (BUG: stack guard page was hit).
On non-KASAN kernels the same loop is safely caught by the existing
limit: the "Dead loop on virtual device" message fires and the packet is dropped without any stack overflow.
Reduce XMIT_RECURSION_LIMIT to 3 when CONFIG_KASAN is enabled. This keeps the recursion counter well within the 32KB KASAN stack budget while preserving the established limit of 8 for production kernels.
The recursion path triggering this is:
__dev_queue_xmit -> dev_hard_start_xmit -> ipvlan_start_xmit -> ipvlan_queue_xmit -> ipvlan_process_outbound -> ip_local_out -> nf_hook (IPVS) -> ip_vs_in_hook -> ip_vs_nat_xmit -> ip_output -> ip_finish_output2 -> neigh_resolve_output -> __dev_queue_xmit
Tested:
- KASAN kernel (6.8.12 x86_64): panic before fix, "Dead loop" drop after fix.
- Non-KASAN kernel (6.8.12 x86_64): "Dead loop" drop both before and after fix (no behavior change for production kernels).
---
v2: Switch from per-driver recursion guard in ipvlan_core.c to reducing
the global XMIT_RECURSION_LIMIT under CONFIG_KASAN, as suggested by Eric Dumazet.
include/linux/netdevice.h | 4 ++++ 1 file changed, 4 insertions(+)
include/linux/netdevice.h +4 -0
@@ -3640,7 +3640,11 @@ struct page_pool_bh { }; DECLARE_PER_CPU(struct page_pool_bh, system_page_pool); +#ifdef CONFIG_KASAN +#define XMIT_RECURSION_LIMIT 3 +#else #define XMIT_RECURSION_LIMIT 8 +#endif #ifndef CONFIG_PREEMPT_RT static inline int dev_recursion_level (void )
Signature · 1 line
2.47.3
Jakub Kicinski
maintainerRe: [PATCH net v2] net: reduce XMIT_RECURSION_LIMIT under KASAN
Quoted 8 lines · On Sat, 11 Jul 2026 20:47:00 +0000 Tristan Madani wrote:
Virtual network devices (ipvlan, macvlan, bonding) can enter legitimate transmit recursion when combined with packet forwarding configurations such as IPVS NAT. The existing
XMIT_RECURSION_LIMIT(8) in__dev_queue_xmit()detects and breaks these loops, but the allowed depth is too high for KASAN-instrumented kernels: each recursion level consumes significantly more stack due to KASAN inline instrumentation, and the cumulative usage overflows the kernel stack before the limit fires.
This patch was marked as changes requested, I'm not sure why. If it is still valid please repost (please start a new thread, please add a link to the previous posting under --- like this:
v3:
- no changes
v2: https://lore.kernel.org/[email protected]