Multiple subnets of the same IP space, different VLANs, same box...

Okay, first let me admit that what I'm trying to do is a bit on the weird side, and stems from some historical requirements that I can not easily change. Having said that, here's the scoop.

I have numerous subnets, which are all identical. Each has devices in the 10.1.1.0/24 range that need to talk to a server at 10.1.1.1. All subnets for the sake of argument will be Class C ranges. (1) They are all connected through Cisco switches, via 802.1q VLANs, back to a single Linux box. Each subnet comes in on a virtual interface (courtesy of vconfig), such as eth1.10, eth1.11, etc. I need to be able to respond to each of those interfaces as 10.1.1.1, with (so far) only ICMP and UDP traffic.

At first, I'd hoped to be able to simply assign that same IP, 10.1.1.1, over and over to each of the VLAN interfaces, and everything would work itself out. Unfortunately, that doesn't work. It works fine for the first box, but not for the second. The second IP, on the second VLAN, is unable to correctly ARP with the second client. I can beat the ARP problem temporarily by forcing a ping out the VLAN interface (ping -I eth1.11 10.1.1.2, for example), or by adding a static ARP entry, but even after ARP is handled, traffic simply doesn't flow up the IP stack (or more realistically I think it's not flowing back *down* the IP stack correctly).

In discussing this problem with a friend, he suggests that the problem is that when Linux attempts to take a packet from the responding service (ICMP from the kernel, or tftp from tftpd, for example), it starts fresh to determine how to send out that packet, attempts to look up the destination in the ARP cache, regardless of the interface it came in on, and sends to the first entry it finds. I currently believe that this is why the simple / ideal solution I describe above is failing.

I've considered that some magic with iptables DNAT, SNAT, or REDIRECT targets, perhaps combined with the mangle table's ROUTE target, may allow this to be possible, but I can't wrap my head around how to get the packets properly mangled, and I'm still not 100% convinced that will bypass the ARP problem. Another possible (very ugly) solution would be to run virtual machines attached to each of those VLAN interfaces, which would separate out the problem of one "box" having lots of identical IP addresses and ARP entries, but it seems like an amazing waste of resources.

Any input anyone can offer about possible solutions, or even providing further insight into the problem, would be most appreciated. :)

Aaron S. Joyner

(1) IP addresses have, of course, been changed to protect the innocent.

I don't see how you can make this work without SNAT'ing all the incoming traffic on each VLAN interface into a unique IP address range. In addition, as you already mentioned, you'll have to mess with the routing tables as well; you can apply a "mark" to the connections in the connection table for each VLAN interface, then use that to select an outbound routing table.

It's pretty complex; realistically, you might actually be better off creating virtual machines to NAT the traffic onto the "internal" network in the box, then hide it all from the services on that box.

Alternatively, you could buy a whole boatload of cheap NAT routers, and attach each one of them to a non-tagged VLAN port on your switch (one for each VLAN), then connect the "upstream" side of them to a non-VLAN-enabled switch with your Linux box on it.

Third alternative would be to do the VLAN switching with a device that is actually a Layer 3 router as well, that can do the NAT for you.

All of these solutions, though, involve NAT, which will bring along other problems if you are running applications/protocols that are not NAT friendly.

nah--this request comes up weekly...

i can't *believe* i'm actually going to link to this, but the last time someone asked something similar (k--so it was over a week ago), i posted this:

http://marc.theaimsgroup.com/?l=netfilter&m=110737820716813&w=2

it doesn't exactly match your situation, but the concept is the same--as long as the machine in question only has to respond to requests you can MARK the connection as it comes in, and route the reply based on the mark, which is restored using CONNMARK.

as the OP never responded as to whether this suggestion worked, or ate his dog, i still have no idea as to whether it works or not; as i never tried it myself...so YMMV.

as someone else already suggested, you can also do this using NETMAP to nat entire subnets to something unique.

IIRC--there was someone that asked about this, and i came up with:

http://marc.theaimsgroup.com/?l=netfilter&m=110027573811157&w=2

again--no idea if this works or not... but you should be able to get the general idea of what's involved in this sort of chicanery.

-j

--
"Me lose brain? Uh, oh! Ha ha ha! Why I laugh?"
--The Simpsons

[...]

The contents of that post will work. We do iproute2 policy routing of that nature around my office all the time with configurations that are similar and even-more whacked out. ;-)

[ Tobias DiPasquale ]
0x636f6465736c696e67657240676d61696c2e636f6d