Skip to contentSkip to conversation

vxlan: duplicate ICMP replies with nexthop FDB entries after 3fa64d86fc64

REGRESSION Sascha Frey 2 participants 4 messages Oct 28 – Oct 29

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Hi,

I am seeing a VXLAN forwarding regression with Linux 7.2.7 and later, apparently related to commit 3fa64d86fc64 ("vxlan: reject dynamic fdb entries that reference a nexthop id").

Environment

Linux kernel: 7.2.7, tested from the upstream source tree Also affected: Debian backports 7.2.9 and 7.2.11 Known good: Linux 7.1.8 and 7.2.6

The affected systems are Debian-based KVM virtualization hosts. Each host uses Linux bridges with VxLAN devices attached as bridge ports, acting as VTEPs.

The overall network uses EVPN. Some VTEPs run on the Debian virtualization hosts, while other VTEPs are implemented on Mellanox switches running Cumulus Linux 5.18 with EVPN multihoming (EVPN-MH).

FRR/Zebra installs nexthop groups and VxLAN FDB entries on the Linux hosts.

Symptoms

ICMP replies are duplicated. A packet capture on the source host's physical interface carrying the outer VXLAN UDP traffic shows the same ICMP reply encapsulated towards all remote VTEP addresses.

On a remote VTEP, a capture on the vxlan interface shows only one request and one reply per sequence number.

Disabling GRO did not help, and the problem is independent of the ICMP packet size.

Relevant FDB state

With the unmodified 7.2.7 kernel, the following entry was missing:

90:e2:ba:33:3b:04 nhid 536870949 self extern_learn

The corresponding nexthop group existed:

id 536870949 group 268435494/268435495 scope global proto zebra fdb id 268435494 via 10.224.90.1 scope link proto zebra fdb id 268435495 via 10.224.90.2 scope link proto zebra fdb

Revert test

I reverted commit 3fa64d86fc64 on top of Linux 7.2.7, without other source changes.

With this kernel:

The missing VXLAN FDB entry reappears.
The duplicate ICMP replies disappear.

The revert removes the checks in vxlan_fdb_add() and vxlan_fdb_update_existing() that reject dynamic FDB entries referencing a nexthop ID.

This strongly suggests that the commit is involved, although I have not yet established the exact mechanism by which the FDB entry goes missing.

Could someone help investigate whether the new checks can interfere with EVPN/FRR-installed nexthop FDB entries, or whether this exposes another issue in the VXLAN FDB handling?

Best regards,
Sascha

#1 lore raw
Quoted 3 lines · Sascha Frey

Could someone help investigate whether the new checks can interfere with EVPN/FRR-installed nexthop FDB entries, or whether this exposes another issue in the VXLAN FDB handling?

I believe we need to relax the checks to allow "extern_learn" entries. They are not subject to ageing (vxlan_cleanup() skips them), so they cannot trigger the bug that commit 98fc57d16744 ("vxlan: reject dynamic fdb entries that reference a nexthop id") fixed. Let me do some testing.

#2 lore raw
Quoted 9 lines · Ido Schimmel
Quoted 3 lines · Sascha Frey

Could someone help investigate whether the new checks can interfere with EVPN/FRR-installed nexthop FDB entries, or whether this exposes another issue in the VXLAN FDB handling?

I believe we need to relax the checks to allow "extern_learn" entries. They are not subject to ageing (vxlan_cleanup() skips them), so they cannot trigger the bug that commit 98fc57d16744 ("vxlan: reject dynamic fdb entries that reference a nexthop id") fixed. Let me do some testing.

Please test the following fix:

https://github.com/idosch/linux/commit/8d98fad34ed4ec2b5e0253a68ead7e436ceae6d6.patch

It's on top of current net.git. It won't apply cleanly to 7.2.y, but the conflict resolution is trivial.

#3 lore raw
Quoted 9 lines · Ido Schimmel
Quoted 4 lines · Ido Schimmel

I believe we need to relax the checks to allow "extern_learn" entries. They are not subject to ageing (vxlan_cleanup() skips them), so they cannot trigger the bug that commit 98fc57d16744 ("vxlan: reject dynamic fdb entries that reference a nexthop id") fixed. Let me do some testing.

Please test the following fix:

https://github.com/idosch/linux/commit/8d98fad34ed4ec2b5e0253a68ead7e436ceae6d6.patch

The fix works for me.

Thanks!