Skip to contentSkip to conversation

ipvlan: add xmit recursion protection

PATCH In review Tristan Madani 3 participants 4 messages Jul 11 – Jul 21

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 ipvlan: add xmit recursion protection

From: Tristan Madani <[email protected]>

ipvlan devices can enter infinite transmit recursion when combined with packet forwarding configurations (such as IPVS) that route traffic back through the same ipvlan interface.

The recursion path is:

ipvlan_start_xmit -> ipvlan_queue_xmit -> ipvlan_xmit_mode_l3 -> ipvlan_process_outbound -> ip_local_out -> netfilter hooks -> dev_queue_xmit -> ipvlan_start_xmit (recurse)

The existing per-CPU xmit recursion counter in __dev_queue_xmit() (XMIT_RECURSION_LIMIT = 8) does detect the loop, but fires too late:
each recursion level consumes roughly 2KB of stack space through ip_local_out and netfilter, and at 8 levels the cumulative usage exceeds the 16KB kernel stack on x86_64. The resulting stack overflow hits the VMAP_STACK guard page and causes a kernel panic.

Add a per-CPU counter that prevents any re-entry into ipvlan_queue_xmit() while it is already executing on the same CPU. This mirrors the approach used by tunnel devices (see IP_TUNNEL_RECURSION_LIMIT in ip_tunnels.h) but with a stricter limit appropriate for ipvlan.

Fixes 2ad7bf363841 ("ipvlan: Initial check-in of the IPVLAN driver.")
---
 drivers/net/ipvlan/ipvlan_core.c | 25 +++++++++++++++++++++----
 1 file changed, 21 insertions(+), 4 deletions(-)
drivers/net/ipvlan/ipvlan_core.c +21 -4
index 835e04835..ab99eb624 100644
@@ -9,6 +9,8 @@
 
 static u32 ipvlan_jhash_secret __read_mostly;
 
+static DEFINE_PER_CPU(int, ipvlan_xmit_depth);
+
 void ipvlan_init_secret(void)
 {
 	net_get_random_once(&ipvlan_jhash_secret, sizeof(ipvlan_jhash_secret));
@@ -676,6 +678,7 @@ int ipvlan_queue_xmit(struct sk_buff *skb, struct net_device *dev)
 {
 	struct ipvl_dev *ipvlan = netdev_priv(dev);
 	struct ipvl_port *port = ipvlan_port_get_rcu_bh(ipvlan->phy_dev);
+	int ret = NET_XMIT_DROP;
 
 	if (!port)
 		goto out;
@@ -683,18 +686,32 @@ int ipvlan_queue_xmit(struct sk_buff *skb, struct net_device *dev)
 	if (unlikely(!pskb_may_pull(skb, sizeof(struct ethhdr))))
 		goto out;
 
+	if (this_cpu_read(ipvlan_xmit_depth)) {
+		net_crit_ratelimited("ipvlan: xmit recursion detected on dev %s\n",
+				     dev->name);
+		goto out;
+	}
+
+	this_cpu_inc(ipvlan_xmit_depth);
 	switch(port->mode) {
 	case IPVLAN_MODE_L2:
-		return ipvlan_xmit_mode_l2(skb, dev);
+		ret = ipvlan_xmit_mode_l2(skb, dev);
+		break;
 	case IPVLAN_MODE_L3:
 #ifdef CONFIG_IPVLAN_L3S
 	case IPVLAN_MODE_L3S:
 #endif
-		return ipvlan_xmit_mode_l3(skb, dev);
+		ret = ipvlan_xmit_mode_l3(skb, dev);
+		break;
+	default:
+		WARN_ONCE(true, "%s called for mode = [%x]\n",
+			  __func__, port->mode);
+		kfree_skb(skb);
+		break;
 	}
+	this_cpu_dec(ipvlan_xmit_depth);
+	return ret;
 
-	/* Should not reach here */
-	WARN_ONCE(true, "%s called for mode = [%x]\n", __func__, port->mode);
 out:
 	kfree_skb(skb);
 	return NET_XMIT_DROP;
Signature · 1 line
2.47.3
#1 lore raw
Quoted 13 lines · Tristan Madani

From: Tristan Madani <[email protected]>

ipvlan devices can enter infinite transmit recursion when combined with packet forwarding configurations (such as IPVS) that route traffic back through the same ipvlan interface.

The recursion path is:

ipvlan_start_xmit -> ipvlan_queue_xmit -> ipvlan_xmit_mode_l3 -> ipvlan_process_outbound -> ip_local_out -> netfilter hooks -> dev_queue_xmit -> ipvlan_start_xmit (recurse)

Quoted 7 lines

The existing per-CPU xmit recursion counter in __dev_queue_xmit() (XMIT_RECURSION_LIMIT = 8) does detect the loop, but fires too late:
each recursion level consumes roughly 2KB of stack space through ip_local_out and netfilter, and at 8 levels the cumulative usage exceeds the 16KB kernel stack on x86_64. The resulting stack overflow hits the VMAP_STACK guard page and causes a kernel panic.

Please fix the existing mechanism. 2KB of stack space seems excessive.

If this is caused by some DEBUG option, it will be just fine to limit XMIT_RECURSION_LIMIT to 3 for such debug kernels.

Adding a workaround in every virtual driver is not an option.

#2 lore raw

PATCH v2 1/1 net: reduce XMIT_RECURSION_LIMIT under KASAN

In reply to Tristan Madani

From: Tristan Madani <[email protected]>

Virtual network devices (ipvlan, macvlan, bonding) can enter legitimate transmit recursion when combined with packet forwarding configurations such as IPVS NAT. The existing XMIT_RECURSION_LIMIT (8) in __dev_queue_xmit() detects and breaks these loops, but the allowed depth is too high for KASAN-instrumented kernels: each recursion level consumes significantly more stack due to KASAN inline instrumentation, and the cumulative usage overflows the kernel stack before the limit fires.

On x86_64, CONFIG_KASAN doubles THREAD_SIZE from 16KB to 32KB (KASAN_STACK_ORDER=1), but KASAN per-access checks inflate individual function frames by roughly 2-3x. For an ipvlan L3 + IPVS NAT routing loop, objdump measurements on a non-KASAN kernel show ~1.4KB of stack consumed per recursion level (across 17 functions from __dev_queue_xmit through the full IP output path and back). At KASAN ~2.3x inflation factor that becomes ~3.3KB per level. Nine levels -- reached before the current limit fires -- total ~30KB plus the initial call chain, which exceeds the 32KB KASAN stack. The overflow hits the VMAP_STACK guard page and causes a non-recoverable kernel panic (BUG: stack guard page was hit).

On non-KASAN kernels the same loop is safely caught by the existing
limit: the "Dead loop on virtual device" message fires and the packet is dropped without any stack overflow.

Reduce XMIT_RECURSION_LIMIT to 3 when CONFIG_KASAN is enabled. This keeps the recursion counter well within the 32KB KASAN stack budget while preserving the established limit of 8 for production kernels.

The recursion path triggering this is:

__dev_queue_xmit -> dev_hard_start_xmit -> ipvlan_start_xmit -> ipvlan_queue_xmit -> ipvlan_process_outbound -> ip_local_out -> nf_hook (IPVS) -> ip_vs_in_hook -> ip_vs_nat_xmit -> ip_output -> ip_finish_output2 -> neigh_resolve_output -> __dev_queue_xmit

Tested:
- KASAN kernel (6.8.12 x86_64): panic before fix, "Dead loop" drop after fix.
- Non-KASAN kernel (6.8.12 x86_64): "Dead loop" drop both before and after fix (no behavior change for production kernels).

Fixes 2ad7bf363841 ("ipvlan: Initial check-in of the IPVLAN driver.")

---
v2: Switch from per-driver recursion guard in ipvlan_core.c to reducing
the global XMIT_RECURSION_LIMIT under CONFIG_KASAN, as suggested by Eric Dumazet.

 include/linux/netdevice.h | 4 ++++
 1 file changed, 4 insertions(+)
include/linux/netdevice.h +4 -0
index 9981d637f8b54..bdcb61d352afb 100644
@@ -3640,7 +3640,11 @@ struct page_pool_bh {
 };
 DECLARE_PER_CPU(struct page_pool_bh, system_page_pool);
 
+#ifdef CONFIG_KASAN
+#define XMIT_RECURSION_LIMIT	3
+#else
 #define XMIT_RECURSION_LIMIT	8
+#endif
 
 #ifndef CONFIG_PREEMPT_RT
 static inline int dev_recursion_level(void)
Signature · 1 line
2.47.3
maintainer
#3 lore raw

Re: [PATCH net v2] net: reduce XMIT_RECURSION_LIMIT under KASAN

Quoted 8 lines · On Sat, 11 Jul 2026 20:47:00 +0000 Tristan Madani wrote:

Virtual network devices (ipvlan, macvlan, bonding) can enter legitimate transmit recursion when combined with packet forwarding configurations such as IPVS NAT. The existing XMIT_RECURSION_LIMIT (8) in __dev_queue_xmit() detects and breaks these loops, but the allowed depth is too high for KASAN-instrumented kernels: each recursion level consumes significantly more stack due to KASAN inline instrumentation, and the cumulative usage overflows the kernel stack before the limit fires.

This patch was marked as changes requested, I'm not sure why. If it is still valid please repost (please start a new thread, please add a link to the previous posting under --- like this:

v3:
- no changes
v2: https://lore.kernel.org/[email protected]