Packet reflection breaks Linux bridge

Hello,
I want to use OpenVPN to extend a layer 2 segment over a routed connection. I was not able to ping a system on the same subnet, so I tried to reproduce it with a different network and was _not_ able to: It worked out of the box. It turned out that I had some Exchange sytem that sends every packet it gets (including broadcast packets) out again. After 3 hours of debugging with Michael Gernoth (on CC), it turned out that the bridge was thinking that the MAC address of the device was on eth1 instead of 'vpn' and the bridge in Linux was not forwarding any packet. We we're able to fight the symptoms by issuing the following command:

ebtables -t nat -A POSTROUTING -d ff:ff:ff:ff:ff:ff -o eth1 -j snat --to-source 00:50:56:98:0a:22

With this command we can access all systems using Layer 2, but not the two Exchange systems which replicate every packet they're seeing.

This is what the Linux box sends:

(miniwheezy64) [~] tcpdump -e -n -i vpn ether host 00:50:56:98:12:ed
tcpdump: WARNING: vpn: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vpn, link-type EN10MB (Ethernet), capture size 65535 bytes
09:35:35.402501 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 35, length 64 09:35:36.410454 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 36, length 64 09:35:37.418526 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 37, length 64 ^C
3 packets captured
3 packets received by filter
0 packets dropped by kernel

This is how it looks, after the Microsoft Exchange settings saw the
package: As you can see, the packets are duplicated at least once, sometimes twice. This also happens with arp who-has and is-at packets which confused the Linux bridge because it thinks that the mac is now at another port:

(miniwheezy64) [~] tcpdump -e -n -i br0 ether host 00:50:56:98:12:ed
tcpdump: WARNING: br0: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 65535 bytes
09:35:42.458571 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 42, length 64 09:35:42.458693 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 42, length 64 09:35:42.459019 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 42, length 64 09:35:42.459220 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 42, length 64 09:35:42.459340 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 42, length 64 09:35:43.466531 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 43, length 64 09:35:43.466758 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 43, length 64 09:35:43.467162 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 43, length 64 09:35:43.467169 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 43, length 64 09:35:43.467366 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 43, length 64 09:35:44.474551 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 44, length 64 09:35:44.474774 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 2973, seq 44, length 64 09:35:44.475108 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 44, length 64 09:35:44.475150 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 44, length 64 09:35:44.475160 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 2973, seq 44, length 64 ^C
15 packets captured
19 packets received by filter
0 packets dropped by kernel
(miniwheezy64) [~] brctl show

bridge name     bridge id               STP enabled     interfaces
br0             8000.005056980a22       no              eth1

vpn

(miniwheezy64) [~] brctl showmacs br0 | grep -i 00:50:56:98:12:ed
1 00:50:56:98:12:ed no 0.08

As you can see the mac address of the system is learned on port 1 but should be learned on port 2.

This is the setup:

Exchange and other systems - VLAN 103 - eth1 - br0 - vpn - OpenVPN - vpn - br0 - eth1 - virtual switch - client (00:50:56:98:12:ed)

It seems that the Linux bridge behaves different than a physical switch. Because with a physical switch it still forwards the packet. But the Linux bridge gets offbeat by the reflected arp who-has and is-at packets and thinks that the mac has moved to a different port even that it hasn't. So I wonder why does the linux bridge gets confused by that, but the physical switch not?

I found one other reference which confirms this problem.

http://serverfault.com/questions/518254/linux-container-bridge-filters-arp-reply

Also I wonder what else than source natting broadcast request we could do to fight the symptoms? For example I would like to drop all packets with the source mac address from 00:50:56:98:12:ed which are comming in via eth1. I tried this using ebtables, arptables and iptables but did not get the syntax right. Has someone, something I could try?

I'm running Debian wheezy with Linux minisqueeze 3.2.0-4-686-pae #1 SMP Debian 3.2.51-1 i686 GNU/Linux.

If I set the MAC adress of a system static on the system behind the layer 2 bridge, the system gets connectivity. If I remove OpenVPN from the equation and use a layer 2 gre tunnel or just another physical network card, I have the same behaviour.

Cheers,
Thomas

Bridging doesn't like loops, and you have created a loop. If you are goin to mess around using ebtables, just write another rule to drop the reflections.

Hello Stephan,

I agree. It was actually a Microsoft Load Balancing cluster that created the loop. Michael told me that the physical switch works because it:

a) broadcasts
b) sends it to both ports.

Michael said that ebtables only after the bridge has seen the packets and so is not applicable. Is that true? So should I use arptables. I tried to block the looped packages by:

- iptables
- arptables
- eptables

But somehow I never made it. Do you have an example for me or can tell me which of the 3 tools should work, so that I can try again. The NATing fought the symptoms of my problem.

Cheers,
Thomas

No example.

You can also define static fdb entries with lastest kernel/iproute and that would problably pin the entry.

Thomas Glanzmann schreef op 30/11/2013 9:54:

The author answered his own question stating it was an issue in his network :-)

ebtables -t nat -A PREROUTING -s 00:50:56:98:12:ed -i eth1 -j DROP

Use PREROUTING so that the Linux bridge fdb isn't updated. A better solution is probably to fix your network.

cheers,
Bart

Hello Bart,

that is true.

Thank you a lot for this line. I just tried it and it works:

(miniwheezy64) [~] ebtables -t nat -A PREROUTING -s 00:50:56:98:12:ed -i eth1 -j DROP
(miniwheezy64) [~] tcpdump -e -n -i vpn ether host 00:50:56:98:12:ed
tcpdump: WARNING: vpn: no IPv4 address assigned
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on vpn, link-type EN10MB (Ethernet), capture size 65535 bytes
05:27:59.786228 00:50:56:98:12:ed > 03:bf:0a:40:67:2a, ethertype IPv4 (0x0800), length 98: 10.64.103.250 > 10.64.103.42: ICMP echo request, id 14490, seq 29, length 64 05:27:59.786768 00:15:5d:b1:45:71 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 14490, seq 29, length 64 05:27:59.786787 00:15:5d:67:ac:93 > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 14490, seq 29, length 64 05:27:59.787041 00:15:5d:67:ab:5f > 00:50:56:98:12:ed, ethertype IPv4 (0x0800), length 98: 10.64.103.42 > 10.64.103.250: ICMP echo reply, id 14490, seq 29, length 64
...

As you can see the system is totally broken. It even sends its unicast packets three times back. A wonder that anything works in this broadcast domain. And the MAC is learned on the rigth port:

(miniwheezy64) [~] brctl showmacs br0 | grep -i 00:50:56:98:12:ed
2 00:50:56:98:12:ed no 0.28

I agree. The problem is that it is _not_ _my_ network. I identified two Microsoft Exchange servers which are part of two different load balancing clusters running on Hyper-V as the culprit. They do not only reflect all broadcast packets that they see, but also the unicast packets. So when I do a unicast connection to them using ICMP or TCP, all packets get reflected. So what I'm looking for now is a workaround to carry out the migration. Because of the feedback provided by Michael, Stephen and you, now I have three:

- SNAT the broadcast packets that come from the far side of the bridge, and SNAT the unicast packets going to the systems which do reflect all packages that they can see.

ebtables -t nat -F
ebtables -t nat -A POSTROUTING -d ff:ff:ff:ff:ff:ff -o eth1 -j snat --to-source 00:50:56:98:0a:22 ebtables -t nat -A POSTROUTING -d 03:bf:0a:40:67:80 -o eth1 -j snat --to-source 00:50:56:98:0a:22 ebtables -t nat -A POSTROUTING -d 03:bf:0a:40:67:2a -o eth1 -j snat --to-source 00:50:56:98:0a:22

- Use the newest code and pin the mac address of the far side machines to the right interface.

- Use the ebtables PREROUTING statement to filter the packets out before they reach the bridge:

ebtables -t nat -A PREROUTING -s 00:50:56:98:12:ed -i eth1 -j DROP

Far side I call systems behind the OpenVPN bridge on the virtual network.

I agree that the above is fighting the symptoms and not resolving the real problem. I don't like that. However I'm not allowed to reconfigure the Exchange Microsoft Cluster, so I work with that. I have informed the people reasonable for it and the network administrators and hope that they take care of the problem.

Cheers,
Thomas