Skip to contentSkip to conversation

fou: ensure GUE headers have enough headroom

PATCH Superseded Chengfeng Ye 3 participants 5 messages Aug 1 – Aug 24

Subsystems (4)
lore
Share & tools
lore
#0 lore raw

PATCH 1/1 fou: ensure GUE headers have enough headroom

ipgre_changelink() installs GUE encapsulation before it publishes the new GRE header length and updates dev->needed_headroom. The transmit path does not serialize with RTNL, so it can interleave as follows:

CPU 0 (ipgre_changelink) CPU 1 (ipgre_xmit)
install GUE encapsulation
reserve the old needed_headroom
publish larger GRE flags
update tunnel->tun_hlen
push the larger GRE header push the GUE and UDP headers
update dev->needed_headroom

With REMCSUM, the new layout can push 16 bytes of GRE and 20 bytes of GUE/UDP headers into an skb with only 32 bytes of actual headroom. The final UDP push writes four bytes before skb->head.

With the update window widened, the kernel reported:

skbuff: skb_under_panic: ... len:128 put:8 ... dev:gre0poc kernel BUG at net/core/skbuff.c:214!

  Oops: invalid opcode: 0000 [#1] SMP KASAN NOPTI
  Call Trace:
   skb_push
   fou_build_udp
   gue_build_header
   ip_tunnel_xmit
   __gre_xmit
   ipgre_xmit

Make __gue_build_header() ensure space for both the GUE header it is about to push and the UDP header that follows. On normally sized skbs the check is a no-op. If configuration changes race with transmission, skb_cow_head() expands the head before either GUE write, or returns an error without modifying the packet.

Fixes dd9d598c6657 ("ip_gre: add the support for i/o_flags update via netlink")
Assisted-by Codex:gpt-5
---
 net/ipv4/fou_core.c | 2 ++
 1 file changed, 2 insertions(+)
net/ipv4/fou_core.c +2 -0
index ab09dfcdecbd..8cf0d43acb41 100644
@@ -980,6 +980,8 @@ int __gue_build_header(struct sk_buff *skb, struct ip_tunnel_encap *e,
 						skb, 0, 0, false);
 
 	hdrlen = sizeof(struct guehdr) + optlen;
+	if (skb_cow_head(skb, hdrlen + sizeof(struct udphdr)))
+		return -ENOMEM;
 
 	skb_push(skb, hdrlen);
 
Signature · 1 line
2.43.0
#1 lore raw
Quoted 49 lines · Chengfeng Ye

ipgre_changelink() installs GUE encapsulation before it publishes the new GRE header length and updates dev->needed_headroom. The transmit path does not serialize with RTNL, so it can interleave as follows:

CPU 0 (ipgre_changelink) CPU 1 (ipgre_xmit)
install GUE encapsulation
reserve the old needed_headroom
publish larger GRE flags
update tunnel->tun_hlen
push the larger GRE header push the GUE and UDP headers
update dev->needed_headroom

With REMCSUM, the new layout can push 16 bytes of GRE and 20 bytes of GUE/UDP headers into an skb with only 32 bytes of actual headroom. The final UDP push writes four bytes before skb->head.

With the update window widened, the kernel reported:

skbuff: skb_under_panic: ... len:128 put:8 ... dev:gre0poc kernel BUG at net/core/skbuff.c:214!

  Oops: invalid opcode: 0000 [#1] SMP KASAN NOPTI
  Call Trace:
   skb_push
   fou_build_udp
   gue_build_header
   ip_tunnel_xmit
   __gre_xmit
   ipgre_xmit

Make __gue_build_header() ensure space for both the GUE header it is about to push and the UDP header that follows. On normally sized skbs the check is a no-op. If configuration changes race with transmission, skb_cow_head() expands the head before either GUE write, or returns an error without modifying the packet.

Fixes dd9d598c6657 ("ip_gre: add the support for i/o_flags update via netlink")
Assisted-by Codex:gpt-5
---
 net/ipv4/fou_core.c | 2 ++
 1 file changed, 2 insertions(+)
net/ipv4/fou_core.c +0 -0
index ab09dfcdecbd..8cf0d43acb41 100644

net/ipv4/fou_core.c int __gue_build_header(struct sk_buff *skb, struct ip_tunnel_encap *e, · line 984

Quoted 5 lines
net/ipv4/fou_core.c +2 -0
@@ -980,6 +980,8 @@ int __gue_build_header(struct sk_buff *skb, struct ip_tunnel_encap *e,
 						skb, 0, 0, false);
 
 	hdrlen = sizeof(struct guehdr) + optlen;
+	if (skb_cow_head(skb, hdrlen + sizeof(struct udphdr)))
+		return -ENOMEM;

My understanding is this takes into account the UDP header pushed in fou_build_udp. Isn't fou_build_header also affected by the same issue then? (If so check the IPv6 paths too).

Also please check Sashiko's output,
https://sashiko.dev/#/patchset/20260801060115.3538849-1-nicoyip.dev%40gmail.com

#2 lore raw

PATCH v2 1/1 fou: ensure GUE headers have enough headroom

In reply to Chengfeng Ye

ipgre_changelink() installs GUE encapsulation before it publishes the new GRE header length and updates dev->needed_headroom. The transmit path does not serialize with RTNL, so it can interleave as follows:

CPU 0 (ipgre_changelink) CPU 1 (ipgre_xmit)
install GUE encapsulation
reserve the old needed_headroom
publish larger GRE flags
update tunnel->tun_hlen
push the larger GRE header push the GUE and UDP headers
update dev->needed_headroom

With REMCSUM, the new layout can push 16 bytes of GRE and 20 bytes of GUE/UDP headers into an skb with only 32 bytes of actual headroom. The final UDP push writes four bytes before skb->head.

With the update window widened, the kernel reported:

skbuff: skb_under_panic: ... len:128 put:8 ... dev:gre0poc kernel BUG at net/core/skbuff.c:214!

  Oops: invalid opcode: 0000 [#1] SMP KASAN NOPTI
  Call Trace:
   skb_push
   fou_build_udp
   gue_build_header
   ip_tunnel_xmit
   __gre_xmit
   ipgre_xmit

Make __gue_build_header() ensure space for both the GUE header it is about to push and the UDP header that follows. On normally sized skbs the check is a no-op. If configuration changes race with transmission, skb_cow_head() expands the head before either GUE write, or returns an error without modifying the packet.

skb_cow_head() may move skb->head. Refresh skb-derived inner_iph pointers after encapsulation, and read the tunnel fragment and TTL fields before ip_tunnel_encap() so ip_tunnel_xmit() does not dereference tnl_params after a possible skb head reallocation.

Fixes dd9d598c6657 ("ip_gre: add the support for i/o_flags update via netlink")

---
Changes in v2:
- Refresh skb-derived inner_iph pointers after ip_tunnel_encap() and read frag_off and ttl from tnl_params before ip_tunnel_encap(), because the new skb_cow_head() in the GUE builder can move skb->head.

 net/ipv4/fou_core.c  | 2 ++
 net/ipv4/ip_tunnel.c | 7 +++++--
 2 files changed, 7 insertions(+), 2 deletions(-)
net/ipv4/fou_core.c +2 -0
index ab09dfcdecbd..8cf0d43acb41 100644
@@ -980,6 +980,8 @@ int __gue_build_header(struct sk_buff *skb, struct ip_tunnel_encap *e,
 						skb, 0, 0, false);
 
 	hdrlen = sizeof(struct guehdr) + optlen;
+	if (skb_cow_head(skb, hdrlen + sizeof(struct udphdr)))
+		return -ENOMEM;
 
 	skb_push(skb, hdrlen);
 
net/ipv4/ip_tunnel.c +5 -2
index 9d114bd575f9..2bd684eb2ba2 100644
@@ -606,6 +606,7 @@ void ip_md_tunnel_xmit(struct sk_buff *skb, struct net_device *dev,
 
 	if (ip_tunnel_encap(skb, &tun_info->encap, &proto, &fl4) < 0)
 		goto tx_error;
+	inner_iph = (const struct iphdr *)skb_inner_network_header(skb);
 
 	use_cache = ip_tunnel_dst_cache_usable(skb, tun_info);
 	if (use_cache)
@@ -765,8 +766,12 @@ void ip_tunnel_xmit(struct sk_buff *skb, struct net_device *dev,
 			    tunnel->net, READ_ONCE(tunnel->parms.link),
 			    tunnel->fwmark, skb_get_hash(skb), 0);
 
+	df = tnl_params->frag_off;
+	ttl = tnl_params->ttl;
+
 	if (ip_tunnel_encap(skb, &tunnel->encap, &protocol, &fl4) < 0)
 		goto tx_error;
+	inner_iph = (const struct iphdr *)skb_inner_network_header(skb);
 
 	if (connected && md) {
 		use_cache = ip_tunnel_dst_cache_usable(skb, tun_info);
@@ -799,7 +804,6 @@ void ip_tunnel_xmit(struct sk_buff *skb, struct net_device *dev,
 		goto tx_error;
 	}
 
-	df = tnl_params->frag_off;
 	if (payload_protocol == htons(ETH_P_IP) && !tunnel->ignore_df)
 		df |= (inner_iph->frag_off & htons(IP_DF));
 
@@ -821,7 +825,6 @@ void ip_tunnel_xmit(struct sk_buff *skb, struct net_device *dev,
 	}
 
 	tos = ip_tunnel_ecn_encap(tos, inner_iph, skb);
-	ttl = tnl_params->ttl;
 	if (ttl == 0) {
 		if (payload_protocol == htons(ETH_P_IP))
 			ttl = inner_iph->ttl;
Signature · 1 line
2.43.0
maintainer
#3 lore raw
Quoted 36 lines · Chengfeng Ye

ipgre_changelink() installs GUE encapsulation before it publishes the new GRE header length and updates dev->needed_headroom. The transmit path does not serialize with RTNL, so it can interleave as follows:

CPU 0 (ipgre_changelink) CPU 1 (ipgre_xmit)
install GUE encapsulation
reserve the old needed_headroom
publish larger GRE flags
update tunnel->tun_hlen
push the larger GRE header push the GUE and UDP headers
update dev->needed_headroom

With REMCSUM, the new layout can push 16 bytes of GRE and 20 bytes of GUE/UDP headers into an skb with only 32 bytes of actual headroom. The final UDP push writes four bytes before skb->head.

With the update window widened, the kernel reported:

skbuff: skb_under_panic: ... len:128 put:8 ... dev:gre0poc kernel BUG at net/core/skbuff.c:214!

  Oops: invalid opcode: 0000 [#1] SMP KASAN NOPTI
  Call Trace:
   skb_push
   fou_build_udp
   gue_build_header
   ip_tunnel_xmit
   __gre_xmit
   ipgre_xmit

Make __gue_build_header() ensure space for both the GUE header it is about to push and the UDP header that follows. On normally sized skbs the check is a no-op. If configuration changes race with transmission, skb_cow_head() expands the head before either GUE write, or returns an error without modifying the packet.

Quoted 4 lines

skb_cow_head() may move skb->head. Refresh skb-derived inner_iph pointers after encapsulation, and read the tunnel fragment and TTL fields before ip_tunnel_encap() so ip_tunnel_xmit() does not dereference tnl_params after a possible skb head reallocation.

It is up to the caller of ip_tunnel_encap() to make sure that there is enough headroom in the packet. Otherwise, what is the point of the encap_hlen() callback? Also, adding skb_cow_head() in the build_header() callback forces every caller to refresh pointers to skb->head.

There's a WIP patch to fix the same problem in IPv6 [1]. I suggest doing something similar in ip_tunnel_xmit() and ip_md_tunnel_xmit(). Note that IPv6 reserves the headroom before the encap push, so the snapshot is enough there. IPv4 reserves it after, so we also need an skb_cow_head() before ip_tunnel_encap().

[1] https://lore.kernel.org/netdev/[email protected]/

#4 lore raw
Quoted 6 lines · Ido Schimmel

It is up to the caller of ip_tunnel_encap() to make sure that there is enough headroom in the packet. Otherwise, what is the point of the encap_hlen() callback? Also, adding skb_cow_head() in the build_header() callback forces every caller to refresh pointers to skb->head.

Quoted 7 lines

There's a WIP patch to fix the same problem in IPv6 [1]. I suggest doing something similar in ip_tunnel_xmit() and ip_md_tunnel_xmit(). Note that IPv6 reserves the headroom before the encap push, so the snapshot is enough there. IPv4 reserves it after, so we also need an skb_cow_head() before ip_tunnel_encap().

[1] https://lore.kernel.org/netdev/[email protected]/

Thanks for the guide, I just created a v3 to fix the problem using the similar way as that ipv6 patch by adjusting the callsite.

Best regards,
Chengfeng