Skip to contentSkip to conversation

vlan: fix skb_under_panic and races when toggling HW VLAN offload

PATCH Superseded Eric Dumazet 3 participants 12 messages Jul 24 – Aug 11

Subsystems (2)

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

PATCH v2 1/1 vlan: fix skb_under_panic and races when toggling HW VLAN offload

Toggling hardware VLAN TX offload (NETIF_F_HW_VLAN_CTAG_TX or NETIF_F_HW_VLAN_STAG_TX) on a lower device invokes vlan_transfer_features(), which dynamically changed vlandev->hard_header_len.

This causes two issues:
1. Lockless TX paths (e.g. packet_snd in af_packet.c, ip6_finish_output2) read dev->hard_header_len without holding RTNL lock. Mutating hard_header_len dynamically under RTNL creates a data race where upper layers reserve insufficient headroom based on a stale hard_header_len, resulting in skb_under_panic when vlan_dev_hard_header() is called.
2. In addition, vlan_transfer_features() updated hard_header_len without updating header_ops, causing a mismatch between allocated headroom and header creation.

Always setting dev->hard_header_len = real_dev->hard_header_len and dev->needed_headroom = real_dev->needed_headroom + VLAN_HLEN unconditionally ensures:
- dev->hard_header_len remains 100% static and immutable at real_dev->hard_header_len, eliminating all dynamic runtime updates and data races on hard_header_len.
- Upper layers allocating skbs via LL_RESERVED_SPACE() will always reserve sufficient headroom for software VLAN tag insertion (real_dev->hard_header_len + real_dev->needed_headroom + VLAN_HLEN).
- AF_PACKET SOCK_RAW network header offsets remain correctly aligned at real_dev->hard_header_len.
- vlan_header_ops is used unconditionally.

---
v2: apply the VLAN_HLEN provision on needed_headroom
v1: https://lore.kernel.org/netdev/[email protected]/T/#t

 net/8021q/vlan_dev.c | 37 +++++--------------------------------
 1 file changed, 5 insertions(+), 32 deletions(-)
net/8021q/vlan_dev.c +0 -0
index ec2569b3f8dac629027b4344bc89402decf026d1..aaf2051a434cfef30d52e0434a9c372142f49195 100644
@@ -502,26 +502,6 @@ static const struct header_ops vlan_header_ops = {

.parse_protocol = vlan_parse_protocol,
};

-static int vlan_passthru_hard_header(struct sk_buff *skb, struct net_device *dev,
- unsigned short type,
- const void *daddr, const void *saddr,
- unsigned int len)
-{
- struct vlan_dev_priv *vlan = vlan_dev_priv(dev);
- struct net_device *real_dev = vlan->real_dev;
-
- if (saddr == NULL)
- saddr = dev->dev_addr;
-
- return dev_hard_header(skb, real_dev, type, daddr, saddr, len);
-}
-
-static const struct header_ops vlan_passthru_header_ops = {
- .create = vlan_passthru_hard_header,
- .parse = eth_header_parse,
- .parse_protocol = vlan_parse_protocol,
-};
-

 static const struct device_type vlan_type = {
	.name	= "vlan",
 };
@@ -580,14 +560,9 @@ static int vlan_dev_init(struct net_device *dev)
	dev->fcoe_ddp_xid = real_dev->fcoe_ddp_xid;

#endif

- dev->needed_headroom = real_dev->needed_headroom;
- if (vlan_hw_offload_capable(real_dev->features, vlan->vlan_proto)) {
- dev->header_ops = &vlan_passthru_header_ops;
- dev->hard_header_len = real_dev->hard_header_len;
- } else {
- dev->header_ops = &vlan_header_ops;
- dev->hard_header_len = real_dev->hard_header_len + VLAN_HLEN;
- }
+ dev->needed_headroom = real_dev->needed_headroom + VLAN_HLEN;
+ dev->header_ops = &vlan_header_ops;
+ dev->hard_header_len = real_dev->hard_header_len;

	dev->netdev_ops = &vlan_netdev_ops;

@@ -1029,10 +1004,8 @@ static void vlan_transfer_features(struct net_device *dev,

	netif_inherit_tso_max(vlandev, dev);

- if (vlan_hw_offload_capable(dev->features, vlan->vlan_proto))
- vlandev->hard_header_len = dev->hard_header_len;
- else
- vlandev->hard_header_len = dev->hard_header_len + VLAN_HLEN;
+ vlandev->needed_headroom = dev->needed_headroom + VLAN_HLEN;
+ vlandev->hard_header_len = dev->hard_header_len;

 #if IS_ENABLED(CONFIG_FCOE)
	vlandev->fcoe_ddp_xid = dev->fcoe_ddp_xid;

--
2.55.0.229.g6434b31f56-goog

#1 lore raw
Quoted 100 lines · Eric Dumazet

Toggling hardware VLAN TX offload (NETIF_F_HW_VLAN_CTAG_TX or NETIF_F_HW_VLAN_STAG_TX) on a lower device invokes vlan_transfer_features(), which dynamically changed vlandev->hard_header_len.

This causes two issues:
1. Lockless TX paths (e.g. packet_snd in af_packet.c, ip6_finish_output2) read dev->hard_header_len without holding RTNL lock. Mutating hard_header_len dynamically under RTNL creates a data race where upper layers reserve insufficient headroom based on a stale hard_header_len, resulting in skb_under_panic when vlan_dev_hard_header() is called.
2. In addition, vlan_transfer_features() updated hard_header_len without updating header_ops, causing a mismatch between allocated headroom and header creation.

Always setting dev->hard_header_len = real_dev->hard_header_len and dev->needed_headroom = real_dev->needed_headroom + VLAN_HLEN unconditionally ensures:
- dev->hard_header_len remains 100% static and immutable at real_dev->hard_header_len, eliminating all dynamic runtime updates and data races on hard_header_len.
- Upper layers allocating skbs via LL_RESERVED_SPACE() will always reserve sufficient headroom for software VLAN tag insertion (real_dev->hard_header_len + real_dev->needed_headroom + VLAN_HLEN).
- AF_PACKET SOCK_RAW network header offsets remain correctly aligned at real_dev->hard_header_len.
- vlan_header_ops is used unconditionally.

---
v2: apply the VLAN_HLEN provision on needed_headroom
v1: https://lore.kernel.org/netdev/[email protected]/T/#t

 net/8021q/vlan_dev.c | 37 +++++--------------------------------
 1 file changed, 5 insertions(+), 32 deletions(-)
net/8021q/vlan_dev.c +5 -33
index ec2569b3f8dac629027b4344bc89402decf026d1..aaf2051a434cfef30d52e0434a9c372142f49195 100644
@@ -502,26 +502,6 @@ static const struct header_ops vlan_header_ops = {
 	.parse_protocol = vlan_parse_protocol,
 };
 
-static int vlan_passthru_hard_header(struct sk_buff *skb, struct net_device *dev,
-					 unsigned short type,
-					 const void *daddr, const void *saddr,
-					 unsigned int len)
-{
-	struct vlan_dev_priv *vlan = vlan_dev_priv(dev);
-	struct net_device *real_dev = vlan->real_dev;
-
-	if (saddr == NULL)
-		saddr = dev->dev_addr;
-
-	return dev_hard_header(skb, real_dev, type, daddr, saddr, len);
-}
-
-static const struct header_ops vlan_passthru_header_ops = {
-	.create	 = vlan_passthru_hard_header,
-	.parse	 = eth_header_parse,
-	.parse_protocol = vlan_parse_protocol,
-};
-
 static const struct device_type vlan_type = {
 	.name	= "vlan",
 };
@@ -580,14 +560,9 @@ static int vlan_dev_init(struct net_device *dev)
 	dev->fcoe_ddp_xid = real_dev->fcoe_ddp_xid;
 #endif
 
-	dev->needed_headroom = real_dev->needed_headroom;
-	if (vlan_hw_offload_capable(real_dev->features, vlan->vlan_proto)) {
-		dev->header_ops 	 = &vlan_passthru_header_ops;
-		dev->hard_header_len = real_dev->hard_header_len;
-	} else {
-		dev->header_ops 	 = &vlan_header_ops;
-		dev->hard_header_len = real_dev->hard_header_len + VLAN_HLEN;
-	}
+	dev->needed_headroom = real_dev->needed_headroom + VLAN_HLEN;
+	dev->header_ops 	 = &vlan_header_ops;
+	dev->hard_header_len = real_dev->hard_header_len;
 
 	dev->netdev_ops = &vlan_netdev_ops;
 
@@ -1029,10 +1004,8 @@ static void vlan_transfer_features(struct net_device *dev,
 
 	netif_inherit_tso_max(vlandev, dev);
 
-	if (vlan_hw_offload_capable(dev->features, vlan->vlan_proto))
-		vlandev->hard_header_len = dev->hard_header_len;
-	else
-		vlandev->hard_header_len = dev->hard_header_len + VLAN_HLEN;
+	vlandev->needed_headroom = dev->needed_headroom + VLAN_HLEN;
+	vlandev->hard_header_len = dev->hard_header_len;
 
 #if IS_ENABLED(CONFIG_FCOE)
 	vlandev->fcoe_ddp_xid = dev->fcoe_ddp_xid;
--
Quoted 2 lines

2.55.0.229.g6434b31f56-goog

Hi Eric,

While testing this backport in our local environment, we hit a similar skb_under_panic issue when layering macvlan on top of a VLAN device.

macvlan does't sync vlan's need_headroom, so skb head align to 16 bytes. when skb arrive vlan, call vlan_header_ops -> vlan_dev_hard_header, push VLAN_HLEN 4 + ETH_LEN 14 to skb, oops.

Call trace
==========

[  103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan
[  103.088344] ------------[ cut here ]------------
[  103.088346] kernel BUG at net/core/skbuff.c:214!
[  103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY
[  103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  103.099725] Workqueue: mld mld_ifc_work
[  103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  103.100898] pc : skb_panic+0x58/0x68
[  103.101310] lr : skb_panic+0x58/0x68
[  103.101738] sp : ffff800083d1b950
[  103.109858] Call trace:
[  103.110161]  skb_panic+0x58/0x68 (P)
[  103.110578]  skb_push+0x58/0x60
[  103.110971]  eth_header+0x3c/0xe0
[  103.111359]  vlan_dev_hard_header+0x7c/0x1a8 [8021q]
[  103.111904]  macvlan_hard_header+0x38/0x60 [macvlan]
[  103.112455]  neigh_resolve_output.part.0+0xa8/0x168
[  103.113081]  neigh_resolve_output+0x4c/0x90
[  103.113639]  ip6_finish_output2+0x2c0/0x858
[  103.114182]  ip6_finish_output+0x23c/0x398
[  103.114741]  ip6_output+0x90/0x250
[  103.115211]  NF_HOOK.constprop.0+0x54/0x118
[  103.115699]  mld_sendpack+0x1cc/0x3d0
[  103.116202]  mld_send_cr+0x1cc/0x318
[  103.116696]  mld_ifc_work+0x38/0x128

Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up

ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up

# macvlan does't sync vlan's need_headroom,
ip link add link vlan_test name macvlan type macvlan
ip link set macvlan up

ethtool -K veth0 tx-vlan-hw-insert off

# skb_under_panic
ip -6 addr add fe80::1/64 dev macvlan
Signature · 2 lines
Best regards,
Tangxin Xie
#2 lore raw
Quoted 50 lines · xietangxin

Hi Eric,

While testing this backport in our local environment, we hit a similar skb_under_panic issue when layering macvlan on top of a VLAN device.

macvlan does't sync vlan's need_headroom, so skb head align to 16 bytes. when skb arrive vlan, call vlan_header_ops -> vlan_dev_hard_header, push VLAN_HLEN 4 + ETH_LEN 14 to skb, oops.

Call trace
==========

[  103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan
[  103.088344] ------------[ cut here ]------------
[  103.088346] kernel BUG at net/core/skbuff.c:214!
[  103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY
[  103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  103.099725] Workqueue: mld mld_ifc_work
[  103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  103.100898] pc : skb_panic+0x58/0x68
[  103.101310] lr : skb_panic+0x58/0x68
[  103.101738] sp : ffff800083d1b950
[  103.109858] Call trace:
[  103.110161]  skb_panic+0x58/0x68 (P)
[  103.110578]  skb_push+0x58/0x60
[  103.110971]  eth_header+0x3c/0xe0
[  103.111359]  vlan_dev_hard_header+0x7c/0x1a8 [8021q]
[  103.111904]  macvlan_hard_header+0x38/0x60 [macvlan]
[  103.112455]  neigh_resolve_output.part.0+0xa8/0x168
[  103.113081]  neigh_resolve_output+0x4c/0x90
[  103.113639]  ip6_finish_output2+0x2c0/0x858
[  103.114182]  ip6_finish_output+0x23c/0x398
[  103.114741]  ip6_output+0x90/0x250
[  103.115211]  NF_HOOK.constprop.0+0x54/0x118
[  103.115699]  mld_sendpack+0x1cc/0x3d0
[  103.116202]  mld_send_cr+0x1cc/0x318
[  103.116696]  mld_ifc_work+0x38/0x128

Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up

ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up

Quoted 8 lines
# macvlan does't sync vlan's need_headroom,
ip link add link vlan_test name macvlan type macvlan
ip link set macvlan up

ethtool -K veth0 tx-vlan-hw-insert off

# skb_under_panic
ip -6 addr add fe80::1/64 dev macvlan

Thanks for testing and for providing a clean repro.

This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero needed_headroom.

Maybe the bug was hidden because skb->head was properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().

drivers/net/macvlan.c +0 -0
index c40fa331836bb2395267914807542ae5094e1a3c..0fb619413d94c89e472efed4a6a9f0dd7fb69940

100644

drivers/net/macvlan.c +1 -0
@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev)
        dev->lltx               = true;
        netif_inherit_tso_max(dev, lowerdev);
        dev->hard_header_len    = lowerdev->hard_header_len;
+       dev->needed_headroom    = lowerdev->needed_headroom;
        macvlan_set_lockdep_class(dev);
 
        vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);
@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct

notifier_block *unused,

        case NETDEV_FEAT_CHANGE:
                list_for_each_entry(vlan, &port->vlans, list) {
                        netif_inherit_tso_max(vlan->dev, dev);
+                       vlan->dev->needed_headroom = dev->needed_headroom;
                        netdev_update_features(vlan->dev);
                }
                break;
#3 lore raw
Quoted 90 lines · Eric Dumazet
Quoted 50 lines · xietangxin

Hi Eric,

While testing this backport in our local environment, we hit a similar skb_under_panic issue when layering macvlan on top of a VLAN device.

macvlan does't sync vlan's need_headroom, so skb head align to 16 bytes. when skb arrive vlan, call vlan_header_ops -> vlan_dev_hard_header, push VLAN_HLEN 4 + ETH_LEN 14 to skb, oops.

Call trace
==========

[  103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan
[  103.088344] ------------[ cut here ]------------
[  103.088346] kernel BUG at net/core/skbuff.c:214!
[  103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY
[  103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  103.099725] Workqueue: mld mld_ifc_work
[  103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  103.100898] pc : skb_panic+0x58/0x68
[  103.101310] lr : skb_panic+0x58/0x68
[  103.101738] sp : ffff800083d1b950
[  103.109858] Call trace:
[  103.110161]  skb_panic+0x58/0x68 (P)
[  103.110578]  skb_push+0x58/0x60
[  103.110971]  eth_header+0x3c/0xe0
[  103.111359]  vlan_dev_hard_header+0x7c/0x1a8 [8021q]
[  103.111904]  macvlan_hard_header+0x38/0x60 [macvlan]
[  103.112455]  neigh_resolve_output.part.0+0xa8/0x168
[  103.113081]  neigh_resolve_output+0x4c/0x90
[  103.113639]  ip6_finish_output2+0x2c0/0x858
[  103.114182]  ip6_finish_output+0x23c/0x398
[  103.114741]  ip6_output+0x90/0x250
[  103.115211]  NF_HOOK.constprop.0+0x54/0x118
[  103.115699]  mld_sendpack+0x1cc/0x3d0
[  103.116202]  mld_send_cr+0x1cc/0x318
[  103.116696]  mld_ifc_work+0x38/0x128

Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up

ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up

Quoted 8 lines
# macvlan does't sync vlan's need_headroom,
ip link add link vlan_test name macvlan type macvlan
ip link set macvlan up

ethtool -K veth0 tx-vlan-hw-insert off

# skb_under_panic
ip -6 addr add fe80::1/64 dev macvlan

Thanks for testing and for providing a clean repro.

This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero needed_headroom.

Maybe the bug was hidden because skb->head was properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().

drivers/net/macvlan.c +0 -0
index c40fa331836bb2395267914807542ae5094e1a3c..0fb619413d94c89e472efed4a6a9f0dd7fb69940

100644

drivers/net/macvlan.c +2 -0
@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev)
        dev->lltx               = true;
        netif_inherit_tso_max(dev, lowerdev);
        dev->hard_header_len    = lowerdev->hard_header_len;
+       dev->needed_headroom    = lowerdev->needed_headroom;
        macvlan_set_lockdep_class(dev);
 
        vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);
@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct
 notifier_block *unused,
        case NETDEV_FEAT_CHANGE:
                list_for_each_entry(vlan, &port->vlans, list) {
                        netif_inherit_tso_max(vlan->dev, dev);
+                       vlan->dev->needed_headroom = dev->needed_headroom;
                        netdev_update_features(vlan->dev);
                }
Quoted 1 line
                break;

Hi Eric,

I tested this patch with two reproducers, and it indeed fixes. However, are there other similar issues where devices on vlan?

Signature · 2 lines
Best regards,
Tangxin Xie
#4 lore raw
Quoted 97 lines · xietangxin
Quoted 90 lines · Eric Dumazet
Quoted 50 lines · xietangxin

Hi Eric,

While testing this backport in our local environment, we hit a similar skb_under_panic issue when layering macvlan on top of a VLAN device.

macvlan does't sync vlan's need_headroom, so skb head align to 16 bytes. when skb arrive vlan, call vlan_header_ops -> vlan_dev_hard_header, push VLAN_HLEN 4 + ETH_LEN 14 to skb, oops.

Call trace
==========

[  103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan
[  103.088344] ------------[ cut here ]------------
[  103.088346] kernel BUG at net/core/skbuff.c:214!
[  103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY
[  103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  103.099725] Workqueue: mld mld_ifc_work
[  103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  103.100898] pc : skb_panic+0x58/0x68
[  103.101310] lr : skb_panic+0x58/0x68
[  103.101738] sp : ffff800083d1b950
[  103.109858] Call trace:
[  103.110161]  skb_panic+0x58/0x68 (P)
[  103.110578]  skb_push+0x58/0x60
[  103.110971]  eth_header+0x3c/0xe0
[  103.111359]  vlan_dev_hard_header+0x7c/0x1a8 [8021q]
[  103.111904]  macvlan_hard_header+0x38/0x60 [macvlan]
[  103.112455]  neigh_resolve_output.part.0+0xa8/0x168
[  103.113081]  neigh_resolve_output+0x4c/0x90
[  103.113639]  ip6_finish_output2+0x2c0/0x858
[  103.114182]  ip6_finish_output+0x23c/0x398
[  103.114741]  ip6_output+0x90/0x250
[  103.115211]  NF_HOOK.constprop.0+0x54/0x118
[  103.115699]  mld_sendpack+0x1cc/0x3d0
[  103.116202]  mld_send_cr+0x1cc/0x318
[  103.116696]  mld_ifc_work+0x38/0x128

Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up

ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up

Quoted 8 lines
# macvlan does't sync vlan's need_headroom,
ip link add link vlan_test name macvlan type macvlan
ip link set macvlan up

ethtool -K veth0 tx-vlan-hw-insert off

# skb_under_panic
ip -6 addr add fe80::1/64 dev macvlan

Thanks for testing and for providing a clean repro.

This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero needed_headroom.

Maybe the bug was hidden because skb->head was properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().

drivers/net/macvlan.c +0 -0
index c40fa331836bb2395267914807542ae5094e1a3c..0fb619413d94c89e472efed4a6a9f0dd7fb69940

100644

drivers/net/macvlan.c +2 -0
@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev)
        dev->lltx               = true;
        netif_inherit_tso_max(dev, lowerdev);
        dev->hard_header_len    = lowerdev->hard_header_len;
+       dev->needed_headroom    = lowerdev->needed_headroom;
        macvlan_set_lockdep_class(dev);
 
        vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);
@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct
 notifier_block *unused,
        case NETDEV_FEAT_CHANGE:
                list_for_each_entry(vlan, &port->vlans, list) {
                        netif_inherit_tso_max(vlan->dev, dev);
+                       vlan->dev->needed_headroom = dev->needed_headroom;
                        netdev_update_features(vlan->dev);
                }
Quoted 1 line
                break;

Hi Eric,

Quoted 2 lines

I tested this patch with two reproducers, and it indeed fixes. However, are there other similar issues where devices on vlan?

Tell me :)

At least our selftests passed.

Note that many devices have belts and suspenders. They use skb_cow(skb, min_headroom) or pskb_expand_head(), because we have numerous bugs yet to be fixed.

#5 lore raw
Quoted 101 lines · Eric Dumazet
Quoted 97 lines · xietangxin
Quoted 90 lines · Eric Dumazet
Quoted 50 lines · xietangxin

Hi Eric,

While testing this backport in our local environment, we hit a similar skb_under_panic issue when layering macvlan on top of a VLAN device.

macvlan does't sync vlan's need_headroom, so skb head align to 16 bytes. when skb arrive vlan, call vlan_header_ops -> vlan_dev_hard_header, push VLAN_HLEN 4 + ETH_LEN 14 to skb, oops.

Call trace
==========

[  103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan
[  103.088344] ------------[ cut here ]------------
[  103.088346] kernel BUG at net/core/skbuff.c:214!
[  103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY
[  103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  103.099725] Workqueue: mld mld_ifc_work
[  103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  103.100898] pc : skb_panic+0x58/0x68
[  103.101310] lr : skb_panic+0x58/0x68
[  103.101738] sp : ffff800083d1b950
[  103.109858] Call trace:
[  103.110161]  skb_panic+0x58/0x68 (P)
[  103.110578]  skb_push+0x58/0x60
[  103.110971]  eth_header+0x3c/0xe0
[  103.111359]  vlan_dev_hard_header+0x7c/0x1a8 [8021q]
[  103.111904]  macvlan_hard_header+0x38/0x60 [macvlan]
[  103.112455]  neigh_resolve_output.part.0+0xa8/0x168
[  103.113081]  neigh_resolve_output+0x4c/0x90
[  103.113639]  ip6_finish_output2+0x2c0/0x858
[  103.114182]  ip6_finish_output+0x23c/0x398
[  103.114741]  ip6_output+0x90/0x250
[  103.115211]  NF_HOOK.constprop.0+0x54/0x118
[  103.115699]  mld_sendpack+0x1cc/0x3d0
[  103.116202]  mld_send_cr+0x1cc/0x318
[  103.116696]  mld_ifc_work+0x38/0x128

Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up

ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up

Quoted 8 lines
# macvlan does't sync vlan's need_headroom,
ip link add link vlan_test name macvlan type macvlan
ip link set macvlan up

ethtool -K veth0 tx-vlan-hw-insert off

# skb_under_panic
ip -6 addr add fe80::1/64 dev macvlan

Thanks for testing and for providing a clean repro.

This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero needed_headroom.

Maybe the bug was hidden because skb->head was properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().

drivers/net/macvlan.c +0 -0
index c40fa331836bb2395267914807542ae5094e1a3c..0fb619413d94c89e472efed4a6a9f0dd7fb69940

100644

drivers/net/macvlan.c +2 -0
@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev)
        dev->lltx               = true;
        netif_inherit_tso_max(dev, lowerdev);
        dev->hard_header_len    = lowerdev->hard_header_len;
+       dev->needed_headroom    = lowerdev->needed_headroom;
        macvlan_set_lockdep_class(dev);
 
        vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);
@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct
 notifier_block *unused,
        case NETDEV_FEAT_CHANGE:
                list_for_each_entry(vlan, &port->vlans, list) {
                        netif_inherit_tso_max(vlan->dev, dev);
+                       vlan->dev->needed_headroom = dev->needed_headroom;
                        netdev_update_features(vlan->dev);
                }
Quoted 1 line
                break;

Hi Eric,

Quoted 2 lines

I tested this patch with two reproducers, and it indeed fixes. However, are there other similar issues where devices on vlan?

Quoted 7 lines

Tell me :)

At least our selftests passed.

Note that many devices have belts and suspenders. They use skb_cow(skb, min_headroom) or pskb_expand_head(), because we have numerous bugs yet to be fixed.

Hi Eric,

I ran few tests on other devices over vlan,
and ipvlan triggers the exact same skb_under_panic issue. I haven't tested many devices, and there might be others.

Call trace
==========

[  113.414244] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000c716d000 data:ffff0000c716cffe tail:0x70 end:0x6c0 dev:ipvlan_test
[  113.414286] ------------[ cut here ]------------
[  113.414288] kernel BUG at net/core/skbuff.c:214!
[  113.414290] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  113.424840] CPU: 1 UID: 0 PID: 315 Comm: kworker/1:4 Kdump: loaded Not tainted 7.2.0-rc4+ #59 PREEMPTLAZY
[  113.425832] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  113.426573] Workqueue: mld mld_ifc_work
[  113.427039] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  113.427788] pc : skb_panic+0x58/0x68
[  113.428224] lr : skb_panic+0x58/0x68
[  113.428638] sp : ffff800083d1b950
[  113.437079]  skb_panic+0x58/0x68 (P)
[  113.437558]  skb_push+0x58/0x60
[  113.437949]  eth_header+0x3c/0xe0
[  113.438344]  vlan_dev_hard_header+0xa0/0x1d0 [8021q]
[  113.438913]  ipvlan_hard_header+0x34/0x58 [ipvlan]
[  113.439491]  neigh_resolve_output.part.0+0xa8/0x168
[  113.440027]  neigh_resolve_output+0x4c/0x90
[  113.440505]  ip6_finish_output2+0x2c0/0x858
[  113.440976]  ip6_finish_output+0x23c/0x398
[  113.441442]  ip6_output+0x90/0x250
[  113.441835]  NF_HOOK.constprop.0+0x54/0x118
[  113.442305]  mld_sendpack+0x1cc/0x3d0
[  113.442726]  mld_send_cr+0x1cc/0x318
[  113.443157]  mld_ifc_work+0x38/0x128

Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up

ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up

ip link add link vlan_test name ipvlan_test type ipvlan mode l2 ip link set ipvlan_test up

ethtool -K veth0 tx-vlan-hw-insert off

ip -6 addr add fe80::1/64 dev ipvlan_test

Signature · 2 lines
Best regards,
Tangxin Xie
#6 lore raw
Quoted 155 lines · xietangxin
Quoted 101 lines · Eric Dumazet
Quoted 97 lines · xietangxin
Quoted 90 lines · Eric Dumazet
Quoted 50 lines · xietangxin

Hi Eric,

While testing this backport in our local environment, we hit a similar skb_under_panic issue when layering macvlan on top of a VLAN device.

macvlan does't sync vlan's need_headroom, so skb head align to 16 bytes. when skb arrive vlan, call vlan_header_ops -> vlan_dev_hard_header, push VLAN_HLEN 4 + ETH_LEN 14 to skb, oops.

Call trace
==========

[  103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan
[  103.088344] ------------[ cut here ]------------
[  103.088346] kernel BUG at net/core/skbuff.c:214!
[  103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY
[  103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  103.099725] Workqueue: mld mld_ifc_work
[  103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  103.100898] pc : skb_panic+0x58/0x68
[  103.101310] lr : skb_panic+0x58/0x68
[  103.101738] sp : ffff800083d1b950
[  103.109858] Call trace:
[  103.110161]  skb_panic+0x58/0x68 (P)
[  103.110578]  skb_push+0x58/0x60
[  103.110971]  eth_header+0x3c/0xe0
[  103.111359]  vlan_dev_hard_header+0x7c/0x1a8 [8021q]
[  103.111904]  macvlan_hard_header+0x38/0x60 [macvlan]
[  103.112455]  neigh_resolve_output.part.0+0xa8/0x168
[  103.113081]  neigh_resolve_output+0x4c/0x90
[  103.113639]  ip6_finish_output2+0x2c0/0x858
[  103.114182]  ip6_finish_output+0x23c/0x398
[  103.114741]  ip6_output+0x90/0x250
[  103.115211]  NF_HOOK.constprop.0+0x54/0x118
[  103.115699]  mld_sendpack+0x1cc/0x3d0
[  103.116202]  mld_send_cr+0x1cc/0x318
[  103.116696]  mld_ifc_work+0x38/0x128

Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up

ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up

Quoted 8 lines
# macvlan does't sync vlan's need_headroom,
ip link add link vlan_test name macvlan type macvlan
ip link set macvlan up

ethtool -K veth0 tx-vlan-hw-insert off

# skb_under_panic
ip -6 addr add fe80::1/64 dev macvlan

Thanks for testing and for providing a clean repro.

This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero needed_headroom.

Maybe the bug was hidden because skb->head was properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().

drivers/net/macvlan.c +0 -0
index c40fa331836bb2395267914807542ae5094e1a3c..0fb619413d94c89e472efed4a6a9f0dd7fb69940

100644

drivers/net/macvlan.c +2 -0
@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev)
        dev->lltx               = true;
        netif_inherit_tso_max(dev, lowerdev);
        dev->hard_header_len    = lowerdev->hard_header_len;
+       dev->needed_headroom    = lowerdev->needed_headroom;
        macvlan_set_lockdep_class(dev);
 
        vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);
@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct
 notifier_block *unused,
        case NETDEV_FEAT_CHANGE:
                list_for_each_entry(vlan, &port->vlans, list) {
                        netif_inherit_tso_max(vlan->dev, dev);
+                       vlan->dev->needed_headroom = dev->needed_headroom;
                        netdev_update_features(vlan->dev);
                }
Quoted 1 line
                break;

Hi Eric,

Quoted 2 lines

I tested this patch with two reproducers, and it indeed fixes. However, are there other similar issues where devices on vlan?

Quoted 7 lines

Tell me :)

At least our selftests passed.

Note that many devices have belts and suspenders. They use skb_cow(skb, min_headroom) or pskb_expand_head(), because we have numerous bugs yet to be fixed.

Hi Eric,

I ran few tests on other devices over vlan,
and ipvlan triggers the exact same skb_under_panic issue. I haven't tested many devices, and there might be others.

Call trace
==========

[  113.414244] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000c716d000 data:ffff0000c716cffe tail:0x70 end:0x6c0 dev:ipvlan_test
[  113.414286] ------------[ cut here ]------------
[  113.414288] kernel BUG at net/core/skbuff.c:214!
[  113.414290] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  113.424840] CPU: 1 UID: 0 PID: 315 Comm: kworker/1:4 Kdump: loaded Not tainted 7.2.0-rc4+ #59 PREEMPTLAZY
[  113.425832] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  113.426573] Workqueue: mld mld_ifc_work
[  113.427039] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  113.427788] pc : skb_panic+0x58/0x68
[  113.428224] lr : skb_panic+0x58/0x68
[  113.428638] sp : ffff800083d1b950
[  113.437079]  skb_panic+0x58/0x68 (P)
[  113.437558]  skb_push+0x58/0x60
[  113.437949]  eth_header+0x3c/0xe0
[  113.438344]  vlan_dev_hard_header+0xa0/0x1d0 [8021q]
[  113.438913]  ipvlan_hard_header+0x34/0x58 [ipvlan]
[  113.439491]  neigh_resolve_output.part.0+0xa8/0x168
[  113.440027]  neigh_resolve_output+0x4c/0x90
[  113.440505]  ip6_finish_output2+0x2c0/0x858
[  113.440976]  ip6_finish_output+0x23c/0x398
[  113.441442]  ip6_output+0x90/0x250
[  113.441835]  NF_HOOK.constprop.0+0x54/0x118
[  113.442305]  mld_sendpack+0x1cc/0x3d0
[  113.442726]  mld_send_cr+0x1cc/0x318
[  113.443157]  mld_ifc_work+0x38/0x128

Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up

ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up

Quoted 6 lines

ip link add link vlan_test name ipvlan_test type ipvlan mode l2 ip link set ipvlan_test up

ethtool -K veth0 tx-vlan-hw-insert off

ip -6 addr add fe80::1/64 dev ipvlan_test

I am not surprised, ipvlan has been copy/pasted from macvlan :)

drivers/net/ipvlan/ipvlan_main.c +0 -0
index ed46439a9f4eb1dd8bfaf6c83476ba4f2aff31bc..7513c005a5121676d3f8edb0898492ad434a9fab

100644

drivers/net/ipvlan/ipvlan_main.c +1 -0
@@ -146,6 +146,7 @@ static int ipvlan_init(struct net_device *dev)
        dev->lltx = true;
        netif_inherit_tso_max(dev, phy_dev);
        dev->hard_header_len = phy_dev->hard_header_len;
+       dev->needed_headroom = phy_dev->needed_headroom;
 
        netdev_lockdep_set_classes(dev);
 
@@ -773,6 +774,7 @@ static int ipvlan_device_event(struct

notifier_block *unused,

        case NETDEV_FEAT_CHANGE:
                list_for_each_entry(ipvlan, &port->ipvlans, pnode) {
                        netif_inherit_tso_max(ipvlan->dev, dev);
+                       ipvlan->dev->needed_headroom = dev->needed_headroom;
                        netdev_update_features(ipvlan->dev);
                }
                break;
#7 lore raw
Quoted 186 lines · Eric Dumazet
Quoted 155 lines · xietangxin
Quoted 101 lines · Eric Dumazet
Quoted 97 lines · xietangxin
Quoted 90 lines · Eric Dumazet
Quoted 50 lines · xietangxin

Hi Eric,

While testing this backport in our local environment, we hit a similar skb_under_panic issue when layering macvlan on top of a VLAN device.

macvlan does't sync vlan's need_headroom, so skb head align to 16 bytes. when skb arrive vlan, call vlan_header_ops -> vlan_dev_hard_header, push VLAN_HLEN 4 + ETH_LEN 14 to skb, oops.

Call trace
==========

[  103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan
[  103.088344] ------------[ cut here ]------------
[  103.088346] kernel BUG at net/core/skbuff.c:214!
[  103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY
[  103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  103.099725] Workqueue: mld mld_ifc_work
[  103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  103.100898] pc : skb_panic+0x58/0x68
[  103.101310] lr : skb_panic+0x58/0x68
[  103.101738] sp : ffff800083d1b950
[  103.109858] Call trace:
[  103.110161]  skb_panic+0x58/0x68 (P)
[  103.110578]  skb_push+0x58/0x60
[  103.110971]  eth_header+0x3c/0xe0
[  103.111359]  vlan_dev_hard_header+0x7c/0x1a8 [8021q]
[  103.111904]  macvlan_hard_header+0x38/0x60 [macvlan]
[  103.112455]  neigh_resolve_output.part.0+0xa8/0x168
[  103.113081]  neigh_resolve_output+0x4c/0x90
[  103.113639]  ip6_finish_output2+0x2c0/0x858
[  103.114182]  ip6_finish_output+0x23c/0x398
[  103.114741]  ip6_output+0x90/0x250
[  103.115211]  NF_HOOK.constprop.0+0x54/0x118
[  103.115699]  mld_sendpack+0x1cc/0x3d0
[  103.116202]  mld_send_cr+0x1cc/0x318
[  103.116696]  mld_ifc_work+0x38/0x128

Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up

ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up

Quoted 8 lines
# macvlan does't sync vlan's need_headroom,
ip link add link vlan_test name macvlan type macvlan
ip link set macvlan up

ethtool -K veth0 tx-vlan-hw-insert off

# skb_under_panic
ip -6 addr add fe80::1/64 dev macvlan

Thanks for testing and for providing a clean repro.

This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero needed_headroom.

Maybe the bug was hidden because skb->head was properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().

drivers/net/macvlan.c +0 -0
index c40fa331836bb2395267914807542ae5094e1a3c..0fb619413d94c89e472efed4a6a9f0dd7fb69940

100644

drivers/net/macvlan.c +2 -0
@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev)
        dev->lltx               = true;
        netif_inherit_tso_max(dev, lowerdev);
        dev->hard_header_len    = lowerdev->hard_header_len;
+       dev->needed_headroom    = lowerdev->needed_headroom;
        macvlan_set_lockdep_class(dev);
 
        vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);
@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct
 notifier_block *unused,
        case NETDEV_FEAT_CHANGE:
                list_for_each_entry(vlan, &port->vlans, list) {
                        netif_inherit_tso_max(vlan->dev, dev);
+                       vlan->dev->needed_headroom = dev->needed_headroom;
                        netdev_update_features(vlan->dev);
                }
Quoted 1 line
                break;

Hi Eric,

Quoted 2 lines

I tested this patch with two reproducers, and it indeed fixes. However, are there other similar issues where devices on vlan?

Quoted 7 lines

Tell me :)

At least our selftests passed.

Note that many devices have belts and suspenders. They use skb_cow(skb, min_headroom) or pskb_expand_head(), because we have numerous bugs yet to be fixed.

Hi Eric,

I ran few tests on other devices over vlan,
and ipvlan triggers the exact same skb_under_panic issue. I haven't tested many devices, and there might be others.

Call trace
==========

[  113.414244] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000c716d000 data:ffff0000c716cffe tail:0x70 end:0x6c0 dev:ipvlan_test
[  113.414286] ------------[ cut here ]------------
[  113.414288] kernel BUG at net/core/skbuff.c:214!
[  113.414290] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  113.424840] CPU: 1 UID: 0 PID: 315 Comm: kworker/1:4 Kdump: loaded Not tainted 7.2.0-rc4+ #59 PREEMPTLAZY
[  113.425832] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  113.426573] Workqueue: mld mld_ifc_work
[  113.427039] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  113.427788] pc : skb_panic+0x58/0x68
[  113.428224] lr : skb_panic+0x58/0x68
[  113.428638] sp : ffff800083d1b950
[  113.437079]  skb_panic+0x58/0x68 (P)
[  113.437558]  skb_push+0x58/0x60
[  113.437949]  eth_header+0x3c/0xe0
[  113.438344]  vlan_dev_hard_header+0xa0/0x1d0 [8021q]
[  113.438913]  ipvlan_hard_header+0x34/0x58 [ipvlan]
[  113.439491]  neigh_resolve_output.part.0+0xa8/0x168
[  113.440027]  neigh_resolve_output+0x4c/0x90
[  113.440505]  ip6_finish_output2+0x2c0/0x858
[  113.440976]  ip6_finish_output+0x23c/0x398
[  113.441442]  ip6_output+0x90/0x250
[  113.441835]  NF_HOOK.constprop.0+0x54/0x118
[  113.442305]  mld_sendpack+0x1cc/0x3d0
[  113.442726]  mld_send_cr+0x1cc/0x318
[  113.443157]  mld_ifc_work+0x38/0x128

Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up

ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up

Quoted 6 lines

ip link add link vlan_test name ipvlan_test type ipvlan mode l2 ip link set ipvlan_test up

ethtool -K veth0 tx-vlan-hw-insert off

ip -6 addr add fe80::1/64 dev ipvlan_test

I am not surprised, ipvlan has been copy/pasted from macvlan :)

drivers/net/ipvlan/ipvlan_main.c +0 -0
index ed46439a9f4eb1dd8bfaf6c83476ba4f2aff31bc..7513c005a5121676d3f8edb0898492ad434a9fab

100644

drivers/net/ipvlan/ipvlan_main.c +2 -0
@@ -146,6 +146,7 @@ static int ipvlan_init(struct net_device *dev)
        dev->lltx = true;
        netif_inherit_tso_max(dev, phy_dev);
        dev->hard_header_len = phy_dev->hard_header_len;
+       dev->needed_headroom = phy_dev->needed_headroom;
 
        netdev_lockdep_set_classes(dev);
 
@@ -773,6 +774,7 @@ static int ipvlan_device_event(struct
 notifier_block *unused,
        case NETDEV_FEAT_CHANGE:
                list_for_each_entry(ipvlan, &port->ipvlans, pnode) {
                        netif_inherit_tso_max(ipvlan->dev, dev);
+                       ipvlan->dev->needed_headroom = dev->needed_headroom;
                        netdev_update_features(ipvlan->dev);
                }
Quoted 1 line
                break;

Hi Eric,

It works fine.
Thanks for the quick fix!

Signature · 2 lines
Best regards,
Tangxin Xie
#8 lore raw
Quoted 192 lines · xietangxin
Quoted 186 lines · Eric Dumazet
Quoted 155 lines · xietangxin
Quoted 101 lines · Eric Dumazet
Quoted 97 lines · xietangxin
Quoted 90 lines · Eric Dumazet
Quoted 50 lines · xietangxin

Hi Eric,

While testing this backport in our local environment, we hit a similar skb_under_panic issue when layering macvlan on top of a VLAN device.

macvlan does't sync vlan's need_headroom, so skb head align to 16 bytes. when skb arrive vlan, call vlan_header_ops -> vlan_dev_hard_header, push VLAN_HLEN 4 + ETH_LEN 14 to skb, oops.

Call trace
==========

[  103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan
[  103.088344] ------------[ cut here ]------------
[  103.088346] kernel BUG at net/core/skbuff.c:214!
[  103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY
[  103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  103.099725] Workqueue: mld mld_ifc_work
[  103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  103.100898] pc : skb_panic+0x58/0x68
[  103.101310] lr : skb_panic+0x58/0x68
[  103.101738] sp : ffff800083d1b950
[  103.109858] Call trace:
[  103.110161]  skb_panic+0x58/0x68 (P)
[  103.110578]  skb_push+0x58/0x60
[  103.110971]  eth_header+0x3c/0xe0
[  103.111359]  vlan_dev_hard_header+0x7c/0x1a8 [8021q]
[  103.111904]  macvlan_hard_header+0x38/0x60 [macvlan]
[  103.112455]  neigh_resolve_output.part.0+0xa8/0x168
[  103.113081]  neigh_resolve_output+0x4c/0x90
[  103.113639]  ip6_finish_output2+0x2c0/0x858
[  103.114182]  ip6_finish_output+0x23c/0x398
[  103.114741]  ip6_output+0x90/0x250
[  103.115211]  NF_HOOK.constprop.0+0x54/0x118
[  103.115699]  mld_sendpack+0x1cc/0x3d0
[  103.116202]  mld_send_cr+0x1cc/0x318
[  103.116696]  mld_ifc_work+0x38/0x128

Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up

ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up

Quoted 8 lines
# macvlan does't sync vlan's need_headroom,
ip link add link vlan_test name macvlan type macvlan
ip link set macvlan up

ethtool -K veth0 tx-vlan-hw-insert off

# skb_under_panic
ip -6 addr add fe80::1/64 dev macvlan

Thanks for testing and for providing a clean repro.

This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero needed_headroom.

Maybe the bug was hidden because skb->head was properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().

drivers/net/macvlan.c +0 -0
index c40fa331836bb2395267914807542ae5094e1a3c..0fb619413d94c89e472efed4a6a9f0dd7fb69940

100644

drivers/net/macvlan.c +2 -0
@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev)
        dev->lltx               = true;
        netif_inherit_tso_max(dev, lowerdev);
        dev->hard_header_len    = lowerdev->hard_header_len;
+       dev->needed_headroom    = lowerdev->needed_headroom;
        macvlan_set_lockdep_class(dev);
 
        vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);
@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct
 notifier_block *unused,
        case NETDEV_FEAT_CHANGE:
                list_for_each_entry(vlan, &port->vlans, list) {
                        netif_inherit_tso_max(vlan->dev, dev);
+                       vlan->dev->needed_headroom = dev->needed_headroom;
                        netdev_update_features(vlan->dev);
                }
Quoted 1 line
                break;

Hi Eric,

Quoted 2 lines

I tested this patch with two reproducers, and it indeed fixes. However, are there other similar issues where devices on vlan?

Quoted 7 lines

Tell me :)

At least our selftests passed.

Note that many devices have belts and suspenders. They use skb_cow(skb, min_headroom) or pskb_expand_head(), because we have numerous bugs yet to be fixed.

Hi Eric,

I ran few tests on other devices over vlan,
and ipvlan triggers the exact same skb_under_panic issue. I haven't tested many devices, and there might be others.

Call trace
==========

[  113.414244] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000c716d000 data:ffff0000c716cffe tail:0x70 end:0x6c0 dev:ipvlan_test
[  113.414286] ------------[ cut here ]------------
[  113.414288] kernel BUG at net/core/skbuff.c:214!
[  113.414290] Internal error: Oops - BUG: 00000000f2000800 [#1]  SMP
[  113.424840] CPU: 1 UID: 0 PID: 315 Comm: kworker/1:4 Kdump: loaded Not tainted 7.2.0-rc4+ #59 PREEMPTLAZY
[  113.425832] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015
[  113.426573] Workqueue: mld mld_ifc_work
[  113.427039] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--)
[  113.427788] pc : skb_panic+0x58/0x68
[  113.428224] lr : skb_panic+0x58/0x68
[  113.428638] sp : ffff800083d1b950
[  113.437079]  skb_panic+0x58/0x68 (P)
[  113.437558]  skb_push+0x58/0x60
[  113.437949]  eth_header+0x3c/0xe0
[  113.438344]  vlan_dev_hard_header+0xa0/0x1d0 [8021q]
[  113.438913]  ipvlan_hard_header+0x34/0x58 [ipvlan]
[  113.439491]  neigh_resolve_output.part.0+0xa8/0x168
[  113.440027]  neigh_resolve_output+0x4c/0x90
[  113.440505]  ip6_finish_output2+0x2c0/0x858
[  113.440976]  ip6_finish_output+0x23c/0x398
[  113.441442]  ip6_output+0x90/0x250
[  113.441835]  NF_HOOK.constprop.0+0x54/0x118
[  113.442305]  mld_sendpack+0x1cc/0x3d0
[  113.442726]  mld_send_cr+0x1cc/0x318
[  113.443157]  mld_ifc_work+0x38/0x128

Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up

ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up

Quoted 6 lines

ip link add link vlan_test name ipvlan_test type ipvlan mode l2 ip link set ipvlan_test up

ethtool -K veth0 tx-vlan-hw-insert off

ip -6 addr add fe80::1/64 dev ipvlan_test

I am not surprised, ipvlan has been copy/pasted from macvlan :)

drivers/net/ipvlan/ipvlan_main.c +0 -0
index ed46439a9f4eb1dd8bfaf6c83476ba4f2aff31bc..7513c005a5121676d3f8edb0898492ad434a9fab

100644

drivers/net/ipvlan/ipvlan_main.c +2 -0
@@ -146,6 +146,7 @@ static int ipvlan_init(struct net_device *dev)
        dev->lltx = true;
        netif_inherit_tso_max(dev, phy_dev);
        dev->hard_header_len = phy_dev->hard_header_len;
+       dev->needed_headroom = phy_dev->needed_headroom;
 
        netdev_lockdep_set_classes(dev);
 
@@ -773,6 +774,7 @@ static int ipvlan_device_event(struct
 notifier_block *unused,
        case NETDEV_FEAT_CHANGE:
                list_for_each_entry(ipvlan, &port->ipvlans, pnode) {
                        netif_inherit_tso_max(ipvlan->dev, dev);
+                       ipvlan->dev->needed_headroom = dev->needed_headroom;
                        netdev_update_features(ipvlan->dev);
                }
Quoted 1 line
                break;

Hi Eric,

Quoted 2 lines

It works fine.
Thanks for the quick fix!

Hi Eric,

Gentle ping on this thread.

Just wanted to check if you have time to post the v3 patch, or if there's anything else needed on my end?

Signature · 2 lines
Best regards,
Tangxin Xie
#9 lore raw
Quoted 9 lines · xietangxin

Hi Eric,

Gentle ping on this thread.

Just wanted to check if you have time to post the v3 patch, or if there's anything else needed on my end?

I have been on PTO, I will send a series soon. Thanks.

#10 lore raw
Quoted 5 lines · Eric Dumazet
Quoted 2 lines · xietangxin
Quoted 8 lines
Quoted 6 lines

Hi Eric,

Gentle ping on this thread.

Just wanted to check if you have time to post the v3 patch, or if there's anything else needed on my end?

I have been on PTO, I will send a series soon. Thanks.

Prereq patches have been merged, I am sending the V3 today.

maintainer
#11 lore raw
In reply to Eric Dumazet
Quoted 3 lines · On Fri, 24 Jul 2026 03:54:19 +0000 Eric Dumazet wrote:

Toggling hardware VLAN TX offload (NETIF_F_HW_VLAN_CTAG_TX or NETIF_F_HW_VLAN_STAG_TX) on a lower device invokes vlan_transfer_features(), which dynamically changed vlandev->hard_header_len.

This one needs a rebase