vlan: fix skb_under_panic and races when toggling HW VLAN offload
Subsystems (2)
PATCH v2 1/1 vlan: fix skb_under_panic and races when toggling HW VLAN offload
Toggling hardware VLAN TX offload (NETIF_F_HW_VLAN_CTAG_TX or NETIF_F_HW_VLAN_STAG_TX) on a lower device invokes vlan_transfer_features(), which dynamically changed vlandev->hard_header_len.
This causes two issues:
1. Lockless TX paths (e.g. packet_snd in af_packet.c, ip6_finish_output2) read dev->hard_header_len without holding RTNL lock. Mutating hard_header_len dynamically under RTNL creates a data race where upper layers reserve insufficient headroom based on a stale hard_header_len, resulting in skb_under_panic when vlan_dev_hard_header() is called.
2. In addition, vlan_transfer_features() updated hard_header_len without updating header_ops, causing a mismatch between allocated headroom and header creation.
Always setting dev->hard_header_len = real_dev->hard_header_len and dev->needed_headroom = real_dev->needed_headroom + VLAN_HLEN unconditionally ensures:
- dev->hard_header_len remains 100% static and immutable at real_dev->hard_header_len, eliminating all dynamic runtime updates and data races on hard_header_len.
- Upper layers allocating skbs via LL_RESERVED_SPACE() will always reserve sufficient headroom for software VLAN tag insertion (real_dev->hard_header_len + real_dev->needed_headroom + VLAN_HLEN).
- AF_PACKET SOCK_RAW network header offsets remain correctly aligned at real_dev->hard_header_len.
- vlan_header_ops is used unconditionally.
---
v2: apply the VLAN_HLEN provision on needed_headroom
v1: https://lore.kernel.org/netdev/[email protected]/T/#t
net/8021q/vlan_dev.c | 37 +++++-------------------------------- 1 file changed, 5 insertions(+), 32 deletions(-)
net/8021q/vlan_dev.c +0 -0
@@ -502,26 +502,6 @@ static const struct header_ops vlan_header_ops = {
.parse_protocol = vlan_parse_protocol,
};
-static int vlan_passthru_hard_header(struct sk_buff *skb, struct net_device *dev,
- unsigned short type,
- const void *daddr, const void *saddr,
- unsigned int len)
-{
- struct vlan_dev_priv *vlan = vlan_dev_priv(dev);
- struct net_device *real_dev = vlan->real_dev;
-
- if (saddr == NULL)
- saddr = dev->dev_addr;
-
- return dev_hard_header(skb, real_dev, type, daddr, saddr, len);
-}
-
-static const struct header_ops vlan_passthru_header_ops = {
- .create = vlan_passthru_hard_header,
- .parse = eth_header_parse,
- .parse_protocol = vlan_parse_protocol,
-};
-
static const struct device_type vlan_type = {
.name = "vlan",
};
@@ -580,14 +560,9 @@ static int vlan_dev_init(struct net_device *dev)
dev->fcoe_ddp_xid = real_dev->fcoe_ddp_xid;#endif
- dev->needed_headroom = real_dev->needed_headroom;
- if (vlan_hw_offload_capable(real_dev->features, vlan->vlan_proto)) {
- dev->header_ops = &vlan_passthru_header_ops;
- dev->hard_header_len = real_dev->hard_header_len;
- } else {
- dev->header_ops = &vlan_header_ops;
- dev->hard_header_len = real_dev->hard_header_len + VLAN_HLEN;
- }
+ dev->needed_headroom = real_dev->needed_headroom + VLAN_HLEN;
+ dev->header_ops = &vlan_header_ops;
+ dev->hard_header_len = real_dev->hard_header_len;
dev->netdev_ops = &vlan_netdev_ops;
@@ -1029,10 +1004,8 @@ static void vlan_transfer_features(struct net_device *dev,
netif_inherit_tso_max(vlandev, dev);
- if (vlan_hw_offload_capable(dev->features, vlan->vlan_proto))
- vlandev->hard_header_len = dev->hard_header_len;
- else
- vlandev->hard_header_len = dev->hard_header_len + VLAN_HLEN;
+ vlandev->needed_headroom = dev->needed_headroom + VLAN_HLEN;
+ vlandev->hard_header_len = dev->hard_header_len;
#if IS_ENABLED(CONFIG_FCOE) vlandev->fcoe_ddp_xid = dev->fcoe_ddp_xid;
--
2.55.0.229.g6434b31f56-goog
Quoted 100 lines · Eric Dumazet
Toggling hardware VLAN TX offload (
NETIF_F_HW_VLAN_CTAG_TXorNETIF_F_HW_VLAN_STAG_TX) on a lower device invokesvlan_transfer_features(), which dynamically changedvlandev->hard_header_len.This causes two issues:
1. Lockless TX paths (e.g.packet_sndinaf_packet.c,ip6_finish_output2) readdev->hard_header_lenwithout holding RTNL lock. Mutatinghard_header_lendynamically under RTNL creates a data race where upper layers reserve insufficient headroom based on a stalehard_header_len, resulting inskb_under_panicwhenvlan_dev_hard_header()is called.
2. In addition,vlan_transfer_features()updatedhard_header_lenwithout updatingheader_ops, causing a mismatch between allocated headroom and header creation.Always setting
dev->hard_header_len=real_dev->hard_header_lenanddev->needed_headroom=real_dev->needed_headroom+VLAN_HLENunconditionally ensures:
-dev->hard_header_lenremains 100% static and immutable atreal_dev->hard_header_len, eliminating all dynamic runtime updates and data races onhard_header_len.
- Upper layers allocating skbs viaLL_RESERVED_SPACE()will always reserve sufficient headroom for software VLAN tag insertion (real_dev->hard_header_len+real_dev->needed_headroom+VLAN_HLEN).
-AF_PACKETSOCK_RAWnetwork header offsets remain correctly aligned atreal_dev->hard_header_len.
-vlan_header_opsis used unconditionally.Fixes 1da177e4c3f4 ("Linux-2.6.12-rc2")Reported-by Tangxin Xie <[email protected]>Signed-off-by Eric Dumazet <[email protected]>---
v2: apply theVLAN_HLENprovision onneeded_headroom
v1: https://lore.kernel.org/netdev/[email protected]/T/#tnet/8021q/vlan_dev.c | 37 +++++-------------------------------- 1 file changed, 5 insertions(+), 32 deletions(-)net/8021q/vlan_dev.c +5
-33@@ -502,26 +502,6 @@ static const struct header_ops vlan_header_ops = { .parse_protocol = vlan_parse_protocol, }; -static int vlan_passthru_hard_header (struct sk_buff *skb,struct net_device *dev, -unsigned short type, -const void *daddr,const void *saddr, -unsigned int len) -{ -struct vlan_dev_priv *vlan =vlan_dev_priv (dev); -struct net_device *real_dev = vlan->real_dev; - -if (saddr ==NULL ) - saddr = dev->dev_addr; - -return dev_hard_header (skb, real_dev, type, daddr, saddr, len); -} - -static const struct header_ops vlan_passthru_header_ops = { - .create = vlan_passthru_hard_header, - .parse = eth_header_parse, - .parse_protocol = vlan_parse_protocol, -}; -static const struct device_type vlan_type = { .name ="vlan" , };@@ -580,14 +560,9 @@ static int vlan_dev_init(struct net_device *dev) dev->fcoe_ddp_xid = real_dev->fcoe_ddp_xid;#endif - dev->needed_headroom = real_dev->needed_headroom; -if (vlan_hw_offload_capable (real_dev->features, vlan->vlan_proto)) { - dev->header_ops = &vlan_passthru_header_ops; - dev->hard_header_len = real_dev->hard_header_len; - }else { - dev->header_ops = &vlan_header_ops; - dev->hard_header_len = real_dev->hard_header_len +VLAN_HLEN ; - } + dev->needed_headroom = real_dev->needed_headroom +VLAN_HLEN ; + dev->header_ops = &vlan_header_ops; + dev->hard_header_len = real_dev->hard_header_len; dev->netdev_ops = &vlan_netdev_ops;@@ -1029,10 +1004,8 @@ static void vlan_transfer_features(struct net_device *dev,netif_inherit_tso_max (vlandev, dev); -if (vlan_hw_offload_capable (dev->features, vlan->vlan_proto)) - vlandev->hard_header_len = dev->hard_header_len; -else - vlandev->hard_header_len = dev->hard_header_len +VLAN_HLEN ; + vlandev->needed_headroom = dev->needed_headroom +VLAN_HLEN ; + vlandev->hard_header_len = dev->hard_header_len;#if IS_ENABLED (CONFIG_FCOE ) vlandev->fcoe_ddp_xid = dev->fcoe_ddp_xid; --
Quoted 2 lines
2.55.0.229.g6434b31f56-goog
Hi Eric,
While testing this backport in our local environment, we hit a similar skb_under_panic issue when layering macvlan on top of a VLAN device.
macvlan does't sync vlan's need_headroom, so skb head align to 16 bytes. when skb arrive vlan, call vlan_header_ops -> vlan_dev_hard_header, push VLAN_HLEN 4 + ETH_LEN 14 to skb, oops.
Call trace
==========
[ 103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan [ 103.088344] ------------[ cut here ]------------ [ 103.088346] kernel BUG at net/core/skbuff.c:214! [ 103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY [ 103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 103.099725] Workqueue: mld mld_ifc_work [ 103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 103.100898] pc : skb_panic+0x58/0x68 [ 103.101310] lr : skb_panic+0x58/0x68 [ 103.101738] sp : ffff800083d1b950 [ 103.109858] Call trace: [ 103.110161] skb_panic+0x58/0x68 (P) [ 103.110578] skb_push+0x58/0x60 [ 103.110971] eth_header+0x3c/0xe0 [ 103.111359] vlan_dev_hard_header+0x7c/0x1a8 [8021q] [ 103.111904] macvlan_hard_header+0x38/0x60 [macvlan] [ 103.112455] neigh_resolve_output.part.0+0xa8/0x168 [ 103.113081] neigh_resolve_output+0x4c/0x90 [ 103.113639] ip6_finish_output2+0x2c0/0x858 [ 103.114182] ip6_finish_output+0x23c/0x398 [ 103.114741] ip6_output+0x90/0x250 [ 103.115211] NF_HOOK.constprop.0+0x54/0x118 [ 103.115699] mld_sendpack+0x1cc/0x3d0 [ 103.116202] mld_send_cr+0x1cc/0x318 [ 103.116696] mld_ifc_work+0x38/0x128
Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up
ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up
# macvlan does't sync vlan's need_headroom, ip link add link vlan_test name macvlan type macvlan ip link set macvlan up
ethtool -K veth0 tx-vlan-hw-insert off
# skb_under_panic ip -6 addr add fe80::1/64 dev macvlan
Signature · 2 lines
Best regards, Tangxin Xie
Quoted 50 lines · xietangxin
Hi Eric,
While testing this backport in our local environment, we hit a similar
skb_under_panicissue when layering macvlan on top of a VLAN device.macvlan does't sync vlan's
need_headroom, so skb head align to 16 bytes. when skb arrive vlan, callvlan_header_ops->vlan_dev_hard_header, pushVLAN_HLEN4 +ETH_LEN14 to skb, oops.Call trace
==========[ 103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan [ 103.088344] ------------[ cut here ]------------ [ 103.088346] kernel BUG at net/core/skbuff.c:214! [ 103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY [ 103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 103.099725] Workqueue: mld mld_ifc_work [ 103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 103.100898] pc : skb_panic+0x58/0x68 [ 103.101310] lr : skb_panic+0x58/0x68 [ 103.101738] sp : ffff800083d1b950 [ 103.109858] Call trace: [ 103.110161] skb_panic+0x58/0x68 (P) [ 103.110578] skb_push+0x58/0x60 [ 103.110971] eth_header+0x3c/0xe0 [ 103.111359] vlan_dev_hard_header+0x7c/0x1a8 [8021q] [ 103.111904] macvlan_hard_header+0x38/0x60 [macvlan] [ 103.112455] neigh_resolve_output.part.0+0xa8/0x168 [ 103.113081] neigh_resolve_output+0x4c/0x90 [ 103.113639] ip6_finish_output2+0x2c0/0x858 [ 103.114182] ip6_finish_output+0x23c/0x398 [ 103.114741] ip6_output+0x90/0x250 [ 103.115211] NF_HOOK.constprop.0+0x54/0x118 [ 103.115699] mld_sendpack+0x1cc/0x3d0 [ 103.116202] mld_send_cr+0x1cc/0x318 [ 103.116696] mld_ifc_work+0x38/0x128Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 upip link add link veth0 name
vlan_testtype vlan id 10reorder_hdroff ip link setvlan_testup
Quoted 8 lines
# macvlan does't sync vlan's need_headroom, ip link add link vlan_test name macvlan type macvlan ip link set macvlan upethtool -K veth0 tx-vlan-hw-insert off
# skb_under_panic ip -6 addr add fe80::1/64 dev macvlan
Thanks for testing and for providing a clean repro.
This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero needed_headroom.
Maybe the bug was hidden because skb->head was properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().
drivers/net/macvlan.c +0 -0
100644
drivers/net/macvlan.c +1 -0
@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev) dev->lltx = true;netif_inherit_tso_max (dev, lowerdev); dev->hard_header_len = lowerdev->hard_header_len; + dev->needed_headroom = lowerdev->needed_headroom;macvlan_set_lockdep_class (dev); vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);
@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct
notifier_block *unused,
case NETDEV_FEAT_CHANGE:
list_for_each_entry(vlan, &port->vlans, list) {
netif_inherit_tso_max(vlan->dev, dev);
+ vlan->dev->needed_headroom = dev->needed_headroom;
netdev_update_features(vlan->dev);
}
break;Quoted 90 lines · Eric Dumazet
Quoted 50 lines · xietangxin
Hi Eric,
While testing this backport in our local environment, we hit a similar
skb_under_panicissue when layering macvlan on top of a VLAN device.macvlan does't sync vlan's
need_headroom, so skb head align to 16 bytes. when skb arrive vlan, callvlan_header_ops->vlan_dev_hard_header, pushVLAN_HLEN4 +ETH_LEN14 to skb, oops.Call trace
==========[ 103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan [ 103.088344] ------------[ cut here ]------------ [ 103.088346] kernel BUG at net/core/skbuff.c:214! [ 103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY [ 103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 103.099725] Workqueue: mld mld_ifc_work [ 103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 103.100898] pc : skb_panic+0x58/0x68 [ 103.101310] lr : skb_panic+0x58/0x68 [ 103.101738] sp : ffff800083d1b950 [ 103.109858] Call trace: [ 103.110161] skb_panic+0x58/0x68 (P) [ 103.110578] skb_push+0x58/0x60 [ 103.110971] eth_header+0x3c/0xe0 [ 103.111359] vlan_dev_hard_header+0x7c/0x1a8 [8021q] [ 103.111904] macvlan_hard_header+0x38/0x60 [macvlan] [ 103.112455] neigh_resolve_output.part.0+0xa8/0x168 [ 103.113081] neigh_resolve_output+0x4c/0x90 [ 103.113639] ip6_finish_output2+0x2c0/0x858 [ 103.114182] ip6_finish_output+0x23c/0x398 [ 103.114741] ip6_output+0x90/0x250 [ 103.115211] NF_HOOK.constprop.0+0x54/0x118 [ 103.115699] mld_sendpack+0x1cc/0x3d0 [ 103.116202] mld_send_cr+0x1cc/0x318 [ 103.116696] mld_ifc_work+0x38/0x128Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 upip link add link veth0 name
vlan_testtype vlan id 10reorder_hdroff ip link setvlan_testupQuoted 8 lines
# macvlan does't sync vlan's need_headroom, ip link add link vlan_test name macvlan type macvlan ip link set macvlan upethtool -K veth0 tx-vlan-hw-insert off
# skb_under_panic ip -6 addr add fe80::1/64 dev macvlanThanks for testing and for providing a clean repro.
This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero
needed_headroom.Maybe the bug was hidden because
skb->headwas properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().drivers/net/macvlan.c +0
-0100644
drivers/net/macvlan.c +2
-0@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev) dev->lltx = true;netif_inherit_tso_max (dev, lowerdev); dev->hard_header_len = lowerdev->hard_header_len; + dev->needed_headroom = lowerdev->needed_headroom;macvlan_set_lockdep_class (dev); vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct notifier_block *unused, case NETDEV_FEAT_CHANGE: list_for_each_entry(vlan, &port->vlans, list) {netif_inherit_tso_max (vlan->dev, dev); + vlan->dev->needed_headroom = dev->needed_headroom;netdev_update_features (vlan->dev); }
Quoted 1 line
break;
Hi Eric,
I tested this patch with two reproducers, and it indeed fixes. However, are there other similar issues where devices on vlan?
Signature · 2 lines
Best regards, Tangxin Xie
Quoted 97 lines · xietangxin
Quoted 90 lines · Eric Dumazet
Quoted 50 lines · xietangxin
Hi Eric,
While testing this backport in our local environment, we hit a similar
skb_under_panicissue when layering macvlan on top of a VLAN device.macvlan does't sync vlan's
need_headroom, so skb head align to 16 bytes. when skb arrive vlan, callvlan_header_ops->vlan_dev_hard_header, pushVLAN_HLEN4 +ETH_LEN14 to skb, oops.Call trace
==========[ 103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan [ 103.088344] ------------[ cut here ]------------ [ 103.088346] kernel BUG at net/core/skbuff.c:214! [ 103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY [ 103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 103.099725] Workqueue: mld mld_ifc_work [ 103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 103.100898] pc : skb_panic+0x58/0x68 [ 103.101310] lr : skb_panic+0x58/0x68 [ 103.101738] sp : ffff800083d1b950 [ 103.109858] Call trace: [ 103.110161] skb_panic+0x58/0x68 (P) [ 103.110578] skb_push+0x58/0x60 [ 103.110971] eth_header+0x3c/0xe0 [ 103.111359] vlan_dev_hard_header+0x7c/0x1a8 [8021q] [ 103.111904] macvlan_hard_header+0x38/0x60 [macvlan] [ 103.112455] neigh_resolve_output.part.0+0xa8/0x168 [ 103.113081] neigh_resolve_output+0x4c/0x90 [ 103.113639] ip6_finish_output2+0x2c0/0x858 [ 103.114182] ip6_finish_output+0x23c/0x398 [ 103.114741] ip6_output+0x90/0x250 [ 103.115211] NF_HOOK.constprop.0+0x54/0x118 [ 103.115699] mld_sendpack+0x1cc/0x3d0 [ 103.116202] mld_send_cr+0x1cc/0x318 [ 103.116696] mld_ifc_work+0x38/0x128Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 upip link add link veth0 name
vlan_testtype vlan id 10reorder_hdroff ip link setvlan_testupQuoted 8 lines
# macvlan does't sync vlan's need_headroom, ip link add link vlan_test name macvlan type macvlan ip link set macvlan upethtool -K veth0 tx-vlan-hw-insert off
# skb_under_panic ip -6 addr add fe80::1/64 dev macvlanThanks for testing and for providing a clean repro.
This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero
needed_headroom.Maybe the bug was hidden because
skb->headwas properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().drivers/net/macvlan.c +0
-0100644
drivers/net/macvlan.c +2
-0@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev) dev->lltx = true;netif_inherit_tso_max (dev, lowerdev); dev->hard_header_len = lowerdev->hard_header_len; + dev->needed_headroom = lowerdev->needed_headroom;macvlan_set_lockdep_class (dev); vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct notifier_block *unused, case NETDEV_FEAT_CHANGE: list_for_each_entry(vlan, &port->vlans, list) {netif_inherit_tso_max (vlan->dev, dev); + vlan->dev->needed_headroom = dev->needed_headroom;netdev_update_features (vlan->dev); }Quoted 1 line
break;Hi Eric,
Quoted 2 lines
I tested this patch with two reproducers, and it indeed fixes. However, are there other similar issues where devices on vlan?
Tell me :)
At least our selftests passed.
Note that many devices have belts and suspenders. They use skb_cow(skb, min_headroom) or pskb_expand_head(), because we have numerous bugs yet to be fixed.
Quoted 101 lines · Eric Dumazet
Quoted 97 lines · xietangxin
Quoted 90 lines · Eric Dumazet
Quoted 50 lines · xietangxin
Hi Eric,
While testing this backport in our local environment, we hit a similar
skb_under_panicissue when layering macvlan on top of a VLAN device.macvlan does't sync vlan's
need_headroom, so skb head align to 16 bytes. when skb arrive vlan, callvlan_header_ops->vlan_dev_hard_header, pushVLAN_HLEN4 +ETH_LEN14 to skb, oops.Call trace
==========[ 103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan [ 103.088344] ------------[ cut here ]------------ [ 103.088346] kernel BUG at net/core/skbuff.c:214! [ 103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY [ 103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 103.099725] Workqueue: mld mld_ifc_work [ 103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 103.100898] pc : skb_panic+0x58/0x68 [ 103.101310] lr : skb_panic+0x58/0x68 [ 103.101738] sp : ffff800083d1b950 [ 103.109858] Call trace: [ 103.110161] skb_panic+0x58/0x68 (P) [ 103.110578] skb_push+0x58/0x60 [ 103.110971] eth_header+0x3c/0xe0 [ 103.111359] vlan_dev_hard_header+0x7c/0x1a8 [8021q] [ 103.111904] macvlan_hard_header+0x38/0x60 [macvlan] [ 103.112455] neigh_resolve_output.part.0+0xa8/0x168 [ 103.113081] neigh_resolve_output+0x4c/0x90 [ 103.113639] ip6_finish_output2+0x2c0/0x858 [ 103.114182] ip6_finish_output+0x23c/0x398 [ 103.114741] ip6_output+0x90/0x250 [ 103.115211] NF_HOOK.constprop.0+0x54/0x118 [ 103.115699] mld_sendpack+0x1cc/0x3d0 [ 103.116202] mld_send_cr+0x1cc/0x318 [ 103.116696] mld_ifc_work+0x38/0x128Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 upip link add link veth0 name
vlan_testtype vlan id 10reorder_hdroff ip link setvlan_testupQuoted 8 lines
# macvlan does't sync vlan's need_headroom, ip link add link vlan_test name macvlan type macvlan ip link set macvlan upethtool -K veth0 tx-vlan-hw-insert off
# skb_under_panic ip -6 addr add fe80::1/64 dev macvlanThanks for testing and for providing a clean repro.
This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero
needed_headroom.Maybe the bug was hidden because
skb->headwas properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().drivers/net/macvlan.c +0
-0100644
drivers/net/macvlan.c +2
-0@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev) dev->lltx = true;netif_inherit_tso_max (dev, lowerdev); dev->hard_header_len = lowerdev->hard_header_len; + dev->needed_headroom = lowerdev->needed_headroom;macvlan_set_lockdep_class (dev); vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct notifier_block *unused, case NETDEV_FEAT_CHANGE: list_for_each_entry(vlan, &port->vlans, list) {netif_inherit_tso_max (vlan->dev, dev); + vlan->dev->needed_headroom = dev->needed_headroom;netdev_update_features (vlan->dev); }Quoted 1 line
break;Hi Eric,
Quoted 2 lines
I tested this patch with two reproducers, and it indeed fixes. However, are there other similar issues where devices on vlan?
Quoted 7 lines
Tell me :)
At least our selftests passed.
Note that many devices have belts and suspenders. They use
skb_cow(skb, min_headroom)orpskb_expand_head(), because we have numerous bugs yet to be fixed.
Hi Eric,
I ran few tests on other devices over vlan,
and ipvlan triggers the exact same skb_under_panic issue. I haven't tested many devices, and there might be others.
Call trace
==========
[ 113.414244] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000c716d000 data:ffff0000c716cffe tail:0x70 end:0x6c0 dev:ipvlan_test [ 113.414286] ------------[ cut here ]------------ [ 113.414288] kernel BUG at net/core/skbuff.c:214! [ 113.414290] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 113.424840] CPU: 1 UID: 0 PID: 315 Comm: kworker/1:4 Kdump: loaded Not tainted 7.2.0-rc4+ #59 PREEMPTLAZY [ 113.425832] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 113.426573] Workqueue: mld mld_ifc_work [ 113.427039] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 113.427788] pc : skb_panic+0x58/0x68 [ 113.428224] lr : skb_panic+0x58/0x68 [ 113.428638] sp : ffff800083d1b950 [ 113.437079] skb_panic+0x58/0x68 (P) [ 113.437558] skb_push+0x58/0x60 [ 113.437949] eth_header+0x3c/0xe0 [ 113.438344] vlan_dev_hard_header+0xa0/0x1d0 [8021q] [ 113.438913] ipvlan_hard_header+0x34/0x58 [ipvlan] [ 113.439491] neigh_resolve_output.part.0+0xa8/0x168 [ 113.440027] neigh_resolve_output+0x4c/0x90 [ 113.440505] ip6_finish_output2+0x2c0/0x858 [ 113.440976] ip6_finish_output+0x23c/0x398 [ 113.441442] ip6_output+0x90/0x250 [ 113.441835] NF_HOOK.constprop.0+0x54/0x118 [ 113.442305] mld_sendpack+0x1cc/0x3d0 [ 113.442726] mld_send_cr+0x1cc/0x318 [ 113.443157] mld_ifc_work+0x38/0x128
Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 up
ip link add link veth0 name vlan_test type vlan id 10 reorder_hdr off ip link set vlan_test up
ip link add link vlan_test name ipvlan_test type ipvlan mode l2 ip link set ipvlan_test up
ethtool -K veth0 tx-vlan-hw-insert off
ip -6 addr add fe80::1/64 dev ipvlan_test
Signature · 2 lines
Best regards, Tangxin Xie
Quoted 155 lines · xietangxin
Quoted 101 lines · Eric Dumazet
Quoted 97 lines · xietangxin
Quoted 90 lines · Eric Dumazet
Quoted 50 lines · xietangxin
Hi Eric,
While testing this backport in our local environment, we hit a similar
skb_under_panicissue when layering macvlan on top of a VLAN device.macvlan does't sync vlan's
need_headroom, so skb head align to 16 bytes. when skb arrive vlan, callvlan_header_ops->vlan_dev_hard_header, pushVLAN_HLEN4 +ETH_LEN14 to skb, oops.Call trace
==========[ 103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan [ 103.088344] ------------[ cut here ]------------ [ 103.088346] kernel BUG at net/core/skbuff.c:214! [ 103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY [ 103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 103.099725] Workqueue: mld mld_ifc_work [ 103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 103.100898] pc : skb_panic+0x58/0x68 [ 103.101310] lr : skb_panic+0x58/0x68 [ 103.101738] sp : ffff800083d1b950 [ 103.109858] Call trace: [ 103.110161] skb_panic+0x58/0x68 (P) [ 103.110578] skb_push+0x58/0x60 [ 103.110971] eth_header+0x3c/0xe0 [ 103.111359] vlan_dev_hard_header+0x7c/0x1a8 [8021q] [ 103.111904] macvlan_hard_header+0x38/0x60 [macvlan] [ 103.112455] neigh_resolve_output.part.0+0xa8/0x168 [ 103.113081] neigh_resolve_output+0x4c/0x90 [ 103.113639] ip6_finish_output2+0x2c0/0x858 [ 103.114182] ip6_finish_output+0x23c/0x398 [ 103.114741] ip6_output+0x90/0x250 [ 103.115211] NF_HOOK.constprop.0+0x54/0x118 [ 103.115699] mld_sendpack+0x1cc/0x3d0 [ 103.116202] mld_send_cr+0x1cc/0x318 [ 103.116696] mld_ifc_work+0x38/0x128Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 upip link add link veth0 name
vlan_testtype vlan id 10reorder_hdroff ip link setvlan_testupQuoted 8 lines
# macvlan does't sync vlan's need_headroom, ip link add link vlan_test name macvlan type macvlan ip link set macvlan upethtool -K veth0 tx-vlan-hw-insert off
# skb_under_panic ip -6 addr add fe80::1/64 dev macvlanThanks for testing and for providing a clean repro.
This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero
needed_headroom.Maybe the bug was hidden because
skb->headwas properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().drivers/net/macvlan.c +0
-0100644
drivers/net/macvlan.c +2
-0@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev) dev->lltx = true;netif_inherit_tso_max (dev, lowerdev); dev->hard_header_len = lowerdev->hard_header_len; + dev->needed_headroom = lowerdev->needed_headroom;macvlan_set_lockdep_class (dev); vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct notifier_block *unused, case NETDEV_FEAT_CHANGE: list_for_each_entry(vlan, &port->vlans, list) {netif_inherit_tso_max (vlan->dev, dev); + vlan->dev->needed_headroom = dev->needed_headroom;netdev_update_features (vlan->dev); }Quoted 1 line
break;Hi Eric,
Quoted 2 lines
I tested this patch with two reproducers, and it indeed fixes. However, are there other similar issues where devices on vlan?
Quoted 7 lines
Tell me :)
At least our selftests passed.
Note that many devices have belts and suspenders. They use
skb_cow(skb, min_headroom)orpskb_expand_head(), because we have numerous bugs yet to be fixed.Hi Eric,
I ran few tests on other devices over vlan,
and ipvlan triggers the exact sameskb_under_panicissue. I haven't tested many devices, and there might be others.Call trace
==========[ 113.414244] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000c716d000 data:ffff0000c716cffe tail:0x70 end:0x6c0 dev:ipvlan_test [ 113.414286] ------------[ cut here ]------------ [ 113.414288] kernel BUG at net/core/skbuff.c:214! [ 113.414290] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 113.424840] CPU: 1 UID: 0 PID: 315 Comm: kworker/1:4 Kdump: loaded Not tainted 7.2.0-rc4+ #59 PREEMPTLAZY [ 113.425832] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 113.426573] Workqueue: mld mld_ifc_work [ 113.427039] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 113.427788] pc : skb_panic+0x58/0x68 [ 113.428224] lr : skb_panic+0x58/0x68 [ 113.428638] sp : ffff800083d1b950 [ 113.437079] skb_panic+0x58/0x68 (P) [ 113.437558] skb_push+0x58/0x60 [ 113.437949] eth_header+0x3c/0xe0 [ 113.438344] vlan_dev_hard_header+0xa0/0x1d0 [8021q] [ 113.438913] ipvlan_hard_header+0x34/0x58 [ipvlan] [ 113.439491] neigh_resolve_output.part.0+0xa8/0x168 [ 113.440027] neigh_resolve_output+0x4c/0x90 [ 113.440505] ip6_finish_output2+0x2c0/0x858 [ 113.440976] ip6_finish_output+0x23c/0x398 [ 113.441442] ip6_output+0x90/0x250 [ 113.441835] NF_HOOK.constprop.0+0x54/0x118 [ 113.442305] mld_sendpack+0x1cc/0x3d0 [ 113.442726] mld_send_cr+0x1cc/0x318 [ 113.443157] mld_ifc_work+0x38/0x128Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 upip link add link veth0 name
vlan_testtype vlan id 10reorder_hdroff ip link setvlan_testup
Quoted 6 lines
ip link add link
vlan_testnameipvlan_testtype ipvlan mode l2 ip link setipvlan_testupethtool -K veth0 tx-vlan-hw-insert off
ip -6 addr add fe80::1/64 dev
ipvlan_test
I am not surprised, ipvlan has been copy/pasted from macvlan :)
drivers/net/ipvlan/ipvlan_main.c +0 -0
100644
drivers/net/ipvlan/ipvlan_main.c +1 -0
@@ -146,6 +146,7 @@ static int ipvlan_init(struct net_device *dev) dev->lltx = true;netif_inherit_tso_max (dev, phy_dev); dev->hard_header_len = phy_dev->hard_header_len; + dev->needed_headroom = phy_dev->needed_headroom;netdev_lockdep_set_classes (dev);
@@ -773,6 +774,7 @@ static int ipvlan_device_event(struct
notifier_block *unused,
case NETDEV_FEAT_CHANGE:
list_for_each_entry(ipvlan, &port->ipvlans, pnode) {
netif_inherit_tso_max(ipvlan->dev, dev);
+ ipvlan->dev->needed_headroom = dev->needed_headroom;
netdev_update_features(ipvlan->dev);
}
break;Quoted 186 lines · Eric Dumazet
Quoted 155 lines · xietangxin
Quoted 101 lines · Eric Dumazet
Quoted 97 lines · xietangxin
Quoted 90 lines · Eric Dumazet
Quoted 50 lines · xietangxin
Hi Eric,
While testing this backport in our local environment, we hit a similar
skb_under_panicissue when layering macvlan on top of a VLAN device.macvlan does't sync vlan's
need_headroom, so skb head align to 16 bytes. when skb arrive vlan, callvlan_header_ops->vlan_dev_hard_header, pushVLAN_HLEN4 +ETH_LEN14 to skb, oops.Call trace
==========[ 103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan [ 103.088344] ------------[ cut here ]------------ [ 103.088346] kernel BUG at net/core/skbuff.c:214! [ 103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY [ 103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 103.099725] Workqueue: mld mld_ifc_work [ 103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 103.100898] pc : skb_panic+0x58/0x68 [ 103.101310] lr : skb_panic+0x58/0x68 [ 103.101738] sp : ffff800083d1b950 [ 103.109858] Call trace: [ 103.110161] skb_panic+0x58/0x68 (P) [ 103.110578] skb_push+0x58/0x60 [ 103.110971] eth_header+0x3c/0xe0 [ 103.111359] vlan_dev_hard_header+0x7c/0x1a8 [8021q] [ 103.111904] macvlan_hard_header+0x38/0x60 [macvlan] [ 103.112455] neigh_resolve_output.part.0+0xa8/0x168 [ 103.113081] neigh_resolve_output+0x4c/0x90 [ 103.113639] ip6_finish_output2+0x2c0/0x858 [ 103.114182] ip6_finish_output+0x23c/0x398 [ 103.114741] ip6_output+0x90/0x250 [ 103.115211] NF_HOOK.constprop.0+0x54/0x118 [ 103.115699] mld_sendpack+0x1cc/0x3d0 [ 103.116202] mld_send_cr+0x1cc/0x318 [ 103.116696] mld_ifc_work+0x38/0x128Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 upip link add link veth0 name
vlan_testtype vlan id 10reorder_hdroff ip link setvlan_testupQuoted 8 lines
# macvlan does't sync vlan's need_headroom, ip link add link vlan_test name macvlan type macvlan ip link set macvlan upethtool -K veth0 tx-vlan-hw-insert off
# skb_under_panic ip -6 addr add fe80::1/64 dev macvlanThanks for testing and for providing a clean repro.
This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero
needed_headroom.Maybe the bug was hidden because
skb->headwas properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().drivers/net/macvlan.c +0
-0100644
drivers/net/macvlan.c +2
-0@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev) dev->lltx = true;netif_inherit_tso_max (dev, lowerdev); dev->hard_header_len = lowerdev->hard_header_len; + dev->needed_headroom = lowerdev->needed_headroom;macvlan_set_lockdep_class (dev); vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct notifier_block *unused, case NETDEV_FEAT_CHANGE: list_for_each_entry(vlan, &port->vlans, list) {netif_inherit_tso_max (vlan->dev, dev); + vlan->dev->needed_headroom = dev->needed_headroom;netdev_update_features (vlan->dev); }Quoted 1 line
break;Hi Eric,
Quoted 2 lines
I tested this patch with two reproducers, and it indeed fixes. However, are there other similar issues where devices on vlan?
Quoted 7 lines
Tell me :)
At least our selftests passed.
Note that many devices have belts and suspenders. They use
skb_cow(skb, min_headroom)orpskb_expand_head(), because we have numerous bugs yet to be fixed.Hi Eric,
I ran few tests on other devices over vlan,
and ipvlan triggers the exact sameskb_under_panicissue. I haven't tested many devices, and there might be others.Call trace
==========[ 113.414244] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000c716d000 data:ffff0000c716cffe tail:0x70 end:0x6c0 dev:ipvlan_test [ 113.414286] ------------[ cut here ]------------ [ 113.414288] kernel BUG at net/core/skbuff.c:214! [ 113.414290] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 113.424840] CPU: 1 UID: 0 PID: 315 Comm: kworker/1:4 Kdump: loaded Not tainted 7.2.0-rc4+ #59 PREEMPTLAZY [ 113.425832] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 113.426573] Workqueue: mld mld_ifc_work [ 113.427039] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 113.427788] pc : skb_panic+0x58/0x68 [ 113.428224] lr : skb_panic+0x58/0x68 [ 113.428638] sp : ffff800083d1b950 [ 113.437079] skb_panic+0x58/0x68 (P) [ 113.437558] skb_push+0x58/0x60 [ 113.437949] eth_header+0x3c/0xe0 [ 113.438344] vlan_dev_hard_header+0xa0/0x1d0 [8021q] [ 113.438913] ipvlan_hard_header+0x34/0x58 [ipvlan] [ 113.439491] neigh_resolve_output.part.0+0xa8/0x168 [ 113.440027] neigh_resolve_output+0x4c/0x90 [ 113.440505] ip6_finish_output2+0x2c0/0x858 [ 113.440976] ip6_finish_output+0x23c/0x398 [ 113.441442] ip6_output+0x90/0x250 [ 113.441835] NF_HOOK.constprop.0+0x54/0x118 [ 113.442305] mld_sendpack+0x1cc/0x3d0 [ 113.442726] mld_send_cr+0x1cc/0x318 [ 113.443157] mld_ifc_work+0x38/0x128Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 upip link add link veth0 name
vlan_testtype vlan id 10reorder_hdroff ip link setvlan_testupQuoted 6 lines
ip link add link
vlan_testnameipvlan_testtype ipvlan mode l2 ip link setipvlan_testupethtool -K veth0 tx-vlan-hw-insert off
ip -6 addr add fe80::1/64 dev
ipvlan_testI am not surprised, ipvlan has been copy/pasted from macvlan :)
drivers/net/ipvlan/ipvlan_main.c +0
-0100644
drivers/net/ipvlan/ipvlan_main.c +2
-0@@ -146,6 +146,7 @@ static int ipvlan_init(struct net_device *dev) dev->lltx = true;netif_inherit_tso_max (dev, phy_dev); dev->hard_header_len = phy_dev->hard_header_len; + dev->needed_headroom = phy_dev->needed_headroom;netdev_lockdep_set_classes (dev);@@ -773,6 +774,7 @@ static int ipvlan_device_event(struct notifier_block *unused, case NETDEV_FEAT_CHANGE: list_for_each_entry(ipvlan, &port->ipvlans, pnode) {netif_inherit_tso_max (ipvlan->dev, dev); + ipvlan->dev->needed_headroom = dev->needed_headroom;netdev_update_features (ipvlan->dev); }
Quoted 1 line
break;
Hi Eric,
It works fine.
Thanks for the quick fix!
Signature · 2 lines
Best regards, Tangxin Xie
Quoted 192 lines · xietangxin
Quoted 186 lines · Eric Dumazet
Quoted 155 lines · xietangxin
Quoted 101 lines · Eric Dumazet
Quoted 97 lines · xietangxin
Quoted 90 lines · Eric Dumazet
Quoted 50 lines · xietangxin
Hi Eric,
While testing this backport in our local environment, we hit a similar
skb_under_panicissue when layering macvlan on top of a VLAN device.macvlan does't sync vlan's
need_headroom, so skb head align to 16 bytes. when skb arrive vlan, callvlan_header_ops->vlan_dev_hard_header, pushVLAN_HLEN4 +ETH_LEN14 to skb, oops.Call trace
==========[ 103.088286] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000d2055800 data:ffff0000d20557fe tail:0x70 end:0x6c0 dev:macvlan [ 103.088344] ------------[ cut here ]------------ [ 103.088346] kernel BUG at net/core/skbuff.c:214! [ 103.088350] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 103.098003] CPU: 3 UID: 0 PID: 160 Comm: kworker/3:3 Kdump: loaded Not tainted 7.2.0-rc4+ #56 PREEMPTLAZY [ 103.099007] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 103.099725] Workqueue: mld mld_ifc_work [ 103.100169] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 103.100898] pc : skb_panic+0x58/0x68 [ 103.101310] lr : skb_panic+0x58/0x68 [ 103.101738] sp : ffff800083d1b950 [ 103.109858] Call trace: [ 103.110161] skb_panic+0x58/0x68 (P) [ 103.110578] skb_push+0x58/0x60 [ 103.110971] eth_header+0x3c/0xe0 [ 103.111359] vlan_dev_hard_header+0x7c/0x1a8 [8021q] [ 103.111904] macvlan_hard_header+0x38/0x60 [macvlan] [ 103.112455] neigh_resolve_output.part.0+0xa8/0x168 [ 103.113081] neigh_resolve_output+0x4c/0x90 [ 103.113639] ip6_finish_output2+0x2c0/0x858 [ 103.114182] ip6_finish_output+0x23c/0x398 [ 103.114741] ip6_output+0x90/0x250 [ 103.115211] NF_HOOK.constprop.0+0x54/0x118 [ 103.115699] mld_sendpack+0x1cc/0x3d0 [ 103.116202] mld_send_cr+0x1cc/0x318 [ 103.116696] mld_ifc_work+0x38/0x128Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 upip link add link veth0 name
vlan_testtype vlan id 10reorder_hdroff ip link setvlan_testupQuoted 8 lines
# macvlan does't sync vlan's need_headroom, ip link add link vlan_test name macvlan type macvlan ip link set macvlan upethtool -K veth0 tx-vlan-hw-insert off
# skb_under_panic ip -6 addr add fe80::1/64 dev macvlanThanks for testing and for providing a clean repro.
This definitely is a bug in macvlan :/, I am pretty sure it can be stacked on top of devices that have a non zero
needed_headroom.Maybe the bug was hidden because
skb->headwas properly reallocated for such lower devices incurring an expensive kmalloc()/kfree().drivers/net/macvlan.c +0
-0100644
drivers/net/macvlan.c +2
-0@@ -950,6 +950,7 @@ static int macvlan_init(struct net_device *dev) dev->lltx = true;netif_inherit_tso_max (dev, lowerdev); dev->hard_header_len = lowerdev->hard_header_len; + dev->needed_headroom = lowerdev->needed_headroom;macvlan_set_lockdep_class (dev); vlan->pcpu_stats = netdev_alloc_pcpu_stats(struct vlan_pcpu_stats);@@ -1824,6 +1825,7 @@ static int macvlan_device_event(struct notifier_block *unused, case NETDEV_FEAT_CHANGE: list_for_each_entry(vlan, &port->vlans, list) {netif_inherit_tso_max (vlan->dev, dev); + vlan->dev->needed_headroom = dev->needed_headroom;netdev_update_features (vlan->dev); }Quoted 1 line
break;Hi Eric,
Quoted 2 lines
I tested this patch with two reproducers, and it indeed fixes. However, are there other similar issues where devices on vlan?
Quoted 7 lines
Tell me :)
At least our selftests passed.
Note that many devices have belts and suspenders. They use
skb_cow(skb, min_headroom)orpskb_expand_head(), because we have numerous bugs yet to be fixed.Hi Eric,
I ran few tests on other devices over vlan,
and ipvlan triggers the exact sameskb_under_panicissue. I haven't tested many devices, and there might be others.Call trace
==========[ 113.414244] skbuff: skb_under_panic: text:ffff800080d2159c len:114 put:14 head:ffff0000c716d000 data:ffff0000c716cffe tail:0x70 end:0x6c0 dev:ipvlan_test [ 113.414286] ------------[ cut here ]------------ [ 113.414288] kernel BUG at net/core/skbuff.c:214! [ 113.414290] Internal error: Oops - BUG: 00000000f2000800 [#1] SMP [ 113.424840] CPU: 1 UID: 0 PID: 315 Comm: kworker/1:4 Kdump: loaded Not tainted 7.2.0-rc4+ #59 PREEMPTLAZY [ 113.425832] Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 [ 113.426573] Workqueue: mld mld_ifc_work [ 113.427039] pstate: 60400005 (nZCv daif +PAN -UAO -TCO -DIT -SSBS BTYPE=--) [ 113.427788] pc : skb_panic+0x58/0x68 [ 113.428224] lr : skb_panic+0x58/0x68 [ 113.428638] sp : ffff800083d1b950 [ 113.437079] skb_panic+0x58/0x68 (P) [ 113.437558] skb_push+0x58/0x60 [ 113.437949] eth_header+0x3c/0xe0 [ 113.438344] vlan_dev_hard_header+0xa0/0x1d0 [8021q] [ 113.438913] ipvlan_hard_header+0x34/0x58 [ipvlan] [ 113.439491] neigh_resolve_output.part.0+0xa8/0x168 [ 113.440027] neigh_resolve_output+0x4c/0x90 [ 113.440505] ip6_finish_output2+0x2c0/0x858 [ 113.440976] ip6_finish_output+0x23c/0x398 [ 113.441442] ip6_output+0x90/0x250 [ 113.441835] NF_HOOK.constprop.0+0x54/0x118 [ 113.442305] mld_sendpack+0x1cc/0x3d0 [ 113.442726] mld_send_cr+0x1cc/0x318 [ 113.443157] mld_ifc_work+0x38/0x128Reproducer
==========
ip link add veth0 type veth peer name veth1
ip link set veth0 up
ip link set veth1 upip link add link veth0 name
vlan_testtype vlan id 10reorder_hdroff ip link setvlan_testupQuoted 6 lines
ip link add link
vlan_testnameipvlan_testtype ipvlan mode l2 ip link setipvlan_testupethtool -K veth0 tx-vlan-hw-insert off
ip -6 addr add fe80::1/64 dev
ipvlan_testI am not surprised, ipvlan has been copy/pasted from macvlan :)
drivers/net/ipvlan/ipvlan_main.c +0
-0100644
drivers/net/ipvlan/ipvlan_main.c +2
-0@@ -146,6 +146,7 @@ static int ipvlan_init(struct net_device *dev) dev->lltx = true;netif_inherit_tso_max (dev, phy_dev); dev->hard_header_len = phy_dev->hard_header_len; + dev->needed_headroom = phy_dev->needed_headroom;netdev_lockdep_set_classes (dev);@@ -773,6 +774,7 @@ static int ipvlan_device_event(struct notifier_block *unused, case NETDEV_FEAT_CHANGE: list_for_each_entry(ipvlan, &port->ipvlans, pnode) {netif_inherit_tso_max (ipvlan->dev, dev); + ipvlan->dev->needed_headroom = dev->needed_headroom;netdev_update_features (ipvlan->dev); }Quoted 1 line
break;Hi Eric,
Quoted 2 lines
It works fine.
Thanks for the quick fix!
Hi Eric,
Gentle ping on this thread.
Just wanted to check if you have time to post the v3 patch, or if there's anything else needed on my end?
Signature · 2 lines
Best regards, Tangxin Xie
Quoted 9 lines · xietangxin
Hi Eric,
Gentle ping on this thread.
Just wanted to check if you have time to post the v3 patch, or if there's anything else needed on my end?
I have been on PTO, I will send a series soon. Thanks.
Quoted 5 lines · Eric Dumazet
Quoted 2 lines · xietangxin
Quoted 8 lines
Quoted 6 lines
Hi Eric,
Gentle ping on this thread.
Just wanted to check if you have time to post the v3 patch, or if there's anything else needed on my end?
I have been on PTO, I will send a series soon. Thanks.
Prereq patches have been merged, I am sending the V3 today.
Jakub Kicinski
maintainerQuoted 3 lines · On Fri, 24 Jul 2026 03:54:19 +0000 Eric Dumazet wrote:
Toggling hardware VLAN TX offload (
NETIF_F_HW_VLAN_CTAG_TXorNETIF_F_HW_VLAN_STAG_TX) on a lower device invokesvlan_transfer_features(), which dynamically changedvlandev->hard_header_len.
This one needs a rebase