bridge, vlan and *no* stp/bpdu
Hello list,
I've posted here about this before, but I realise that it may have been assumed that the bridged vlans simply put a switch port in a blocking state and left my question ignored. So to recap.
I have two tg3 interfaces named 'in' and 'out' and a bridge named 'br0'
My vlan trunk is on the 'in' side of the network, and set as in.2, in.3 ... The 'out' side goes straight to an ipv4 gateway on untagged plain ethernet.
Putting 'in.2' and 'out' on the bridge works quite well and is roughly what I've been using so far.
# brctl show
bridge name bridge id STP enabled interfaces
br0 8000.00e081342870 no out
in.2If I add in.3 to the bridge, trouble starts. The bridge keeps forwarding packets just like it should, with the exception of ARP replies from the gateway to machines in vlan 2. Machine that had ARPed the gateway prior to adding in.3 to the bridge keep working fine.
Here's the strange thing however. Running a tcpdump on 'out' 'br0' or in.2 shows me the arp requests *and replies* for the machines that do not work, however, if I look on the wire leaving the 'in' interface itself (using a hub and another box), the arp replies are nowhere to be found.
So the arp replies get eaten *before* they make it onto the wire, but *after* tcpdump saw them on in.2. It's driving me nuts... I thought it might have to do with the tg3 hardware doing some funky vlan acceleration, but I've seen the same on plain dumb NICs too.
I'm willing to pay for a solution to this... Or even for just someone knowledgeable enough with the code taking interest in the issue.
Thanks,
Jonathan
Quoted 5 lines · Jonathan Thibault
Hello list,
I've posted here about this before, but I realise that it may have been assumed that the bridged vlans simply put a switch port in a blocking state and left my question ignored. So to recap.
It might have been ignored because you didn't tell us what kernel were using. Things change so fast that a bug 2.6.23 might be fixed in 2.6.24 -- someone can easily tell you that if they know the version info.
Quoted 14 lines
I have two tg3 interfaces named 'in' and 'out' and a bridge named 'br0'
My vlan trunk is on the 'in' side of the network, and set as in.2, in.3 ... The 'out' side goes straight to an ipv4 gateway on untagged plain ethernet.
Putting 'in.2' and 'out' on the bridge works quite well and is roughly what I've been using so far.
# brctl show bridge name bridge id STP enabled interfaces br0 8000.00e081342870 no out in.2
Quoted 5 lines
If I add in.3 to the bridge, trouble starts. The bridge keeps forwarding packets just like it should, with the exception of ARP replies from the gateway to machines in vlan 2. Machine that had ARPed the gateway prior to adding in.3 to the bridge keep working fine.
Are the ARPs from the 'different' machines on in.3 and in.2 using the same MAC address (are they even the same machine but with different VLANs)?
Quoted 22 lines
Here's the strange thing however. Running a tcpdump on 'out' 'br0' or in.2 shows me the arp requests *and replies* for the machines that do not work, however, if I look on the wire leaving the 'in' interface itself (using a hub and another box), the arp replies are nowhere to be found.
So the arp replies get eaten *before* they make it onto the wire, but *after* tcpdump saw them on in.2. It's driving me nuts... I thought it might have to do with the tg3 hardware doing some funky vlan acceleration, but I've seen the same on plain dumb NICs too.
I'm willing to pay for a solution to this... Or even for just someone knowledgeable enough with the code taking interest in the issue.
Thanks,
Jonathan
_______________________________________________
Bridge mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/bridge
A message that is not in the archive (sent off-list or not yet received): [email protected]
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 9 lines · Andy Gospodarek
Configurations like this can get tricky at times. Though it seems to make perfect sense to put two interfaces like this in the same bridge the current setup of the linux networking stack makes it hard to get things working sometimes. Let me make sure I can understand what you are trying to accomplish. Do you want to prevent traffic from vlan 2 and vlan 3 from ever talking while still allowing them to talk to the gateway? Will you want vlan 2 and vlan3 to communicate with each other at all?
Well, the plan is to let the vlans communicate together 'freely', for now. I'm essentially using the vlan tag as a way to 'figure out' where the packet is coming from. The network spans a little over 100km in a large, flat /22 subnet. I wish I could do this the usual way with subnets and what not, but we don't really control the routing and gateway side of things. So basically, we want to assign a vlan to each branch of the network and avoid having to add a new NIC to the bridge each time we add a new branch. Feeding it a trunk into it and adding vlan interfaces as needed works better for us since bandwidth is not really the limiting factor. Well, it would be better if it worked :P
Now I fully understand that it's not how 90% of people would approach this, including myself, but it's the only option we have given what's on either side of the bridge, which we can't change right now. The alternative is having a switch split the trunk and then feed that into real (as opposed to vlan) interfaces on the bridge, wich I haven't fully tested yet but am sure would work fine.
I'd rather focus on solving the bridged vlan thing though, it just might be useful to someone else someday and definitely falls into the 'should work' category as far as I'm concerned.
Jonathan
P.S.: Sorry, forgot to reply to the list.
Quoted 28 lines · Jonathan Thibault
Quoted 9 lines · Andy Gospodarek
Configurations like this can get tricky at times. Though it seems to make perfect sense to put two interfaces like this in the same bridge the current setup of the linux networking stack makes it hard to get things working sometimes. Let me make sure I can understand what you are trying to accomplish. Do you want to prevent traffic from vlan 2 and vlan 3 from ever talking while still allowing them to talk to the gateway? Will you want vlan 2 and vlan3 to communicate with each other at all?
Well, the plan is to let the vlans communicate together 'freely', for now. I'm essentially using the vlan tag as a way to 'figure out' where the packet is coming from. The network spans a little over 100km in a large, flat /22 subnet. I wish I could do this the usual way with subnets and what not, but we don't really control the routing and gateway side of things. So basically, we want to assign a vlan to each branch of the network and avoid having to add a new NIC to the bridge each time we add a new branch. Feeding it a trunk into it and adding vlan interfaces as needed works better for us since bandwidth is not really the limiting factor. Well, it would be better if it worked :P
Now I fully understand that it's not how 90% of people would approach this, including myself, but it's the only option we have given what's on either side of the bridge, which we can't change right now. The alternative is having a switch split the trunk and then feed that into real (as opposed to vlan) interfaces on the bridge, wich I haven't fully tested yet but am sure would work fine.
Quoted 3 lines
I'd rather focus on solving the bridged vlan thing though, it just might be useful to someone else someday and definitely falls into the 'should work' category as far as I'm concerned.
You still didn't tell us any version numbers, and I've got a similar setup which "works for me". The only real difference is that my box routes between two logical bridges, and the bridged interfaces are multiple vlans in the same trunk. I can even protect individual vlans from each other with netfilter rules.
What if you routed "out" as a new vlan on the same cable as "in"?
Quoted 11 lines
Jonathan
P.S.: Sorry, forgot to reply to the list._______________________________________________
Bridge mailing list
[email protected]
https://lists.linux-foundation.org/mailman/listinfo/bridge
Sorry, that's was in another mistakenly off-list reply.
Kernel is 2.6.24, been seeing this problem since 2.6.16 when I started the setup.
Quoted 10 lines · [email protected] wrote:
You still didn't tell us any version numbers, and I've got a similar setup which "works for me". The only real difference is that my box routes between two logical bridges, and the bridged interfaces are multiple vlans in the same trunk. I can even protect individual vlans from each other with netfilter rules.
What if you routed "out" as a new vlan on the same cable as "in"?
That's an interesting idea which I haven't tried yet. An interesting tidbit is that there is a handful of machines in the lot which are affected right away (as soon as I add a second vlan interface to the bridge). It may just be that they just have a very short arp timeout. I tried to find a pattern in their MAC addresses or hardware but there isn't really one. I first assumed there was a problem with those machines but given that the ARP reply never gets to the trunk cable going their way, I concluded otherwise.
Thanks a lot for the help. The fact that you have a setup that works gives me some confidence that I'm not just trying to do something insane ;)
Jonathan