BUG: 2 NICs on same network

Hi,

We recently stummed across a rather annoying bug when 2 nics are on the same network.

Our situation is this: we have a server with 2 nics, each with a different IP on the same network, connected to the same switch. Let's assume eth0 has ip 1.2.3.1 and eth1 has 1.2.3.2, with a both with a netmask of 255.255.255.0.

Now the strange thing is that traffic for 1.2.3.2 arrives at eth0 no matter what!

Even if we disconnect the cable for eth1, 1.2.3.2 still replies to pings, ssh, web, ...

We tested this on IA32 architecture, different 2.4.x kernels and different nics ...

Is this a bug or a known issue? If it is not a bug, how can it be solved?

Kind Regards,
Frank Louwers

Openminds bvba                www.openminds.be
Tweebruggenstraat 16  -  9000 Gent  -  Belgium

check the MAC address of the two cards cards !
i am also using similar configuration but am not getting any thing like that !

rpm

Mac addresses are different! (In my first setup the addresses only differed 1 bit (server with 2 nics onboard)), but later I tried with 2 different brand of nics, so they should have been different enough.

When i disconnect the cable of eth1, and ping it's ip from another machine and check the arp table for that machine, I get the mac address for eth0 ...

Frank

[CCing back to list as Vincent saw the same behaviour]

Well, load balancing or bridging is an option, but not the one I want... I will use the first nic as "normal" (firewalled, traffic shaped, ...) interface for my customer's websites, and eth1 as the "backup" and maintenance nic in case something goes wrong ...

I don't understand why it should be illegal to have to nics on the same server on the same subnet ...

Regarding static arps: am I correct these should be added to all machines on the subnet and to the router? (That last is not possible, I don't have permission to change stuff on the router)

Vriendelijke groeten,
Frank Louwers

Openminds bvba                www.openminds.be
Tweebruggenstraat 16  -  9000 Gent  -  Belgium

try disconnecting eth0 interface ( with eth1 connected ) and then pinging it eth0 !

Do you have ip forwarding enabled? If yes, kernel just forwards packets:
network -> eth0 -> kernel

Try traceroute to eth1' ip. Examine arp tables. What MAC is listed there for eth1 IP?
--
vda

ipforwarding is disabled, arp proxy is disabled.

The mac address is that of eth0 ...

Vriendelijke groeten,
Frank Louwers

Openminds bvba                www.openminds.be
Tweebruggenstraat 16  -  9000 Gent  -  Belgium

Kernel bug I think. Sorry can't help you here.
I'm afraid you have to dig deeper...
--
vda

--------- Received message begins Here ---------

I would suspect the default route is directing the reply (netstat -r should show if this is the case).
This happens whenever there are two NICs to the same network. An incoming connection will go out the default route IF the default route is the same as the network address. This gives preferential activity to the designated route. We've done this when phasing out one interface in favor of another (fddi->GEthernet)..

The usual reccomendation is not to have both interfaces active simultaneiously as this causes (relatively minor) confusion. The connections should work, but one will be relayed to the other interface since there is no distinguishing characteristic about which interface to use - they are equivalent. Once the default route is switched, the inverse situation occurs.

If you want only one port, just down the undesired interface. Use it only when the primary fails. If you need both IPs active, then use ifconfig eth0:1 and have both on the same interface (I think that still works).

Load balancing really calls for two different subnets. Otherwise you get collisions between the two interfaces...

-------------------------------------------------------------------------
Jesse I Pollard, II
Email: [email protected]

Any opinions expressed are solely my own.

If the two NICs are on the same network, the kernel will send the packets to the first available interface.

The kernel has acted like this for years...

Jeff

done based on network address. So if you first configure eth0 with mask 255.255.255.0, and a subnet of 1.2.3.0,
and eth1 is configured just the same (ip adresses only differ in last number), the kernel just uses the first interface it founds for a subnet. To prove this, you could try to load the interfaces in reverse order, and all traffic should go to eth1.

Do you really need the two interfaces to be in the same subnet ? I use tw parallel nets for a cluster, but configured both as independent subnets, 10.0.0.0 and 10.0.1.0. So you can drive all nfs through one interface mounting the server as 10.0.0.1, and all the bproc traffic through the other (or all the ssh through the other connecting always to 10.0.1.1).

Hope this helps.

J.A. Magallon                           #  Let the source be with you...        
mailto:[email protected]
Mandrake Linux release 8.3 (Cooker) for i586
Linux werewolf 2.4.19-pre7-jam5 #1 SMP mar abr 23 01:29:38 CEST 2002 i686

Well, I was planning on using it as a backup link: use eth0 as my "normal" interface card, with a traffic shaper on it, firewalled etc.

If however, something goes wrong with either the firewall, or traffic shaper, or so, I have to drive 60 miles to my server to correct things on the console.

I was hoping on configuring that second nic as a "backup" nic ...

Frank

. . .

It is a known issue with 2.4 Linux kernels and ARP. Linux will respond to arp requests for any address configured on the box on any interface that receives the arp request. Patches have been proposed in the past, but the maintainers have elected to not accept the patches on the basis that the current behavior is RFC-compliant. The behavior you describe is also RFC-compliant, and is in fact what the other OSes that I'm familiar with do.

In your situation, where you have a single nic that you want to use as backup only, you could set noarp on the backup nic. Then, when you want to talk to the machine on the backup nic, you could use a static arp entry specifying the MAC address of the backup and any IP address on the box.

Alternatively, you could put the NICs on different physical segments or you could dig up the proposed patches (2.4.0,2,and 12 if I remember correctly) and port them forward and apply them.

--Harley Stenzel
[email protected]

This is actually standards compliant behaviour, as silly as it sounds. However, if you want stricter arp behaviour I *think* that the following will fix it. At least it used to...

echo 1 > /proc/sys/net/ipv4/conf/all/arp_filter

If this is no longer correct I'd love to hear it.

Chris

Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]

Hi,

I'm looking at that as well and it doesn't look as it solves the problem, in fact I think that the arp_filter is meant to insure that the reply will be sent out to the interface where linux would route it, that is to say, the other way around. It should be something like that instead :

ip_route_output(&rt,tip, sip, 0, 0)

But this function will route tip to eth0 ( the wrong one in that context). But in fact, it comes back to the fact that the two interfaces are put on the same subnet. I know that on some routers, this not even possible to configure two interfaces on the same subnet, it is supposed to be possible on linux ? ok, from what I see in the reply, this is possible but linux will take the first one into account and install a local route to that subnet pointing to the first interface , making the other one useless as we see in this example.

static int arp_filter(__u32 sip, __u32 tip, struct net_device *dev)

{
        struct rtable *rt;
        int flag = 0;
        /*unsigned long now; */

        
        if (ip_route_output(&rt, sip, tip, 0, 0) < 0)
                return 1;             
        if (rt->u.dst.dev != dev) {
                NET_INC_STATS_BH(ArpFilter);
                flag = 1;
        } 

        ip_rt_put(rt);

		Vincent Guffens
		Unite d'automatique, de dynamique et d'analyse des systemes, UCL

the powers that be consider this a feature, for reasons i've never quite understood. your question comes up regularly, and typically the suggestion is some convoluted filtering thing, and i've never been sufficiently convinced the filter is worth the overhead. (especially in my case when i wanted two interfaces for performance reasons.)

last i investigated this problem the kernel actually sent arp responses out *both* interfaces, and it was basically a race condition as to which the recipient would use.

anyhow, below is a patch i created for 2.4.2... hopefully it still works. apply this and also disable proxy arp in /proc/somewhere.

-dean

@@ -737,10 +737,12 @@
 		addr_type = rt->rt_type;
 
 		if (addr_type == RTN_LOCAL) {
+			if ((rt->rt_flags&RTCF_DIRECTSRC) || IN_DEV_PROXY_ARP(in_dev)) {
 			n = neigh_event_ns(&arp_tbl, sha, &sip, dev);
 			if (n) {
 				arp_send(ARPOP_REPLY,ETH_P_ARP,sip,dev,tip,sha,dev->dev_addr,sha);
 				neigh_release(n);
+			}
 			}
 			goto out;
 		} else if (IN_DEV_FORWARD(in_dev)) {

Actually, as long as you have your system set up to route based on source address (which you probably should if you care about strict arp replies) arp_filter seems to work properly. I just tested with a 2.4.18 box, with the following address configuration:

[root@pcary1k5 /]# ip add
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:30:65:bf:46:ba brd ff:ff:ff:ff:ff:ff
    inet 47.129.82.58/24 brd 47.129.82.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
    link/ether 00:50:ff:90:04:44 brd ff:ff:ff:ff:ff:ff
    inet 47.129.82.107/24 brd 47.129.82.255 scope global eth1
[root@pcary1k5 /]# ip ru
0:      from all lookup local
32764:  from 47.129.82.107 lookup 101
32765:  from 47.129.82.58 lookup 100
32766:  from all lookup main
32767:  from all lookup 253
[root@pcary1k5 /]# ip ro
47.129.82.0/24 dev eth0  proto kernel  scope link  src 47.129.82.58
47.129.82.0/24 dev eth1  proto kernel  scope link  src 47.129.82.107

127.0.0.0/8 dev lo scope link
default via 47.129.82.1 dev eth0
[root@pcary1k5 /]# ip ro li table 100
47.129.82.0/24 dev eth0 scope link
default via 47.129.82.1 dev eth0
[root@pcary1k5 /]# ip ro li table 101
47.129.82.0/24 dev eth1 scope link
default via 47.129.82.1 dev eth1

With this setup, arp requests for 47.129.82.58 will only be answered by eth0, and similarly for 47.129.82.107 and eth1.

Chris

Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]

ok, I see, so both interfaces are replying to the arp request but this time, the first interface to reply, eth0, will have its arp reply filtered by the arp_filter as ip_route_output will indicate eth1 as the outgoing interface for a packet that has its source set to the ip to resolve, which has a source route policy entry to point to eth1, is it ?

But then, by curiosity, is there some reasons why two different interfaces shouldn't be in the same subnet ? I was in this impression having configured some cisco routers which don't even accept this configuration. Is it possible on other network devices ? Is there any theoretical justification ?

thanks for your explanations,

--
Vincent Guffens

Correct. Both NICs receive the arp request, only one of them actually sends a reply.

I don't think there are any theoretical objections. I'm pretty sure windows will let you do it, but they don't give you nearly as much control over the routing.

Without source routing or arp filter, traffic to any of the interfaces on the same subnet can go through any of the interfaces on that subnet--whichever arp reply gets there last is the winner. This can be useful for quasi-redundant type communications, but causes big problems if you're trying to monitor the state of your network communications paths. This is where the more strict behaviour becomes useful.

Chris

Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]

Let me say that this is not a kernel question, but a networking question, there's no bug, you just have several configuration issues. However, having been beaten to death and the corpe dragged through the street, hopefully this might help.

The primary issue is that you are responding to arp and sending out the wrong MAC address. Don't wait for the router to ask, tell it where the MAC is for that IP.

ifconfig eth0 10.8.0.5
ifconfig eth1 10.8.0.6
route add www.favorite.com dev eth1
ping -c2 www.favorite.com

The last two lines are the charm... force the route to any host which goes via the router to use the "other" IP address. Good, now the router knows that the IP amd MAC belong together. BTW: if you do an IP takeover (add an alias IP for a server which fails) you also want to do this, or the router will send packets to the old MAC until the ARP tables timeout.

This takes care of incoming connections, the traffic should go to the right NIC and be sent back through the right NIC. To balance outgoing connections you can try this instead of defining a default route:

route add -net 0.0.0.0 netmask 0.0.0.1 dev eth0
route add -net 0.0.0.1 netmask 0.0.0.1 dev eth1

which used to work to send odd IP destinations out one NIC and even out the other. If that fails and you care, just get iproute2 and claw your way up the learning curve.

If you just want to use this for backup, when the machine detects that the NIC has failed (blessedly unusual, I have 15 dual NIC machines in four timezones) it can just ifconfig the first down, the second up, and send out a ping, or better yet use one which allows the MAC address to be set.

Some of this should get you going, or at least looking in the right place.

bill davidsen <[email protected]>
  CTO, TMR Associates, Inc
Doing interesting things with little computers since 1979.

Hi!

This is in perfect agreement with rfc1122, section 3.3.4.2. Linux uses the weak ES model, so the interface doesn't matter. There was once a patch floating around for 2.4.6 that made the kernel accept only arp coming from the 'right' interface. arp_self_only, or something. I have attached it, but I have no idea if it still applies/works.

Ciao,
Roland

+---------------------------+-------------------------+
|    TU Muenchen            |                         |
|    Physik-Department E18  |  Raum    3558           |
|    James-Franck-Str.      |  Telefon 089/289-12592  |
|    85747 Garching         |                         |
+---------------------------+-------------------------+

diff -Naur --exclude=.* --exclude=*.o linux-2.4.6.1/net/ipv4/arp.c linux-2.4.6/net/ipv4/arp.c
@@ -760,6 +760,9 @@
                                int dont_send = 0;
                                if (IN_DEV_ARPFILTER(in_dev))
                                  dont_send |= arp_filter(sip,tip,dev);
+                               if (IN_DEV_ARP_SELF_ONLY(in_dev))
+                                 if (dev != ip_dev_find(tip))
+                                       dont_send = 1;
                                if (!dont_send)
                                  arp_send(ARPOP_REPLY,ETH_P_ARP,sip,dev,tip,sha,dev->dev_addr,sha);
 

diff -Naur --exclude=.* --exclude=*.o linux-2.4.6.1/net/ipv4/devinet.c linux-2.4.6/net/ipv4/devinet.c
@@ -1016,7 +1016,7 @@
 static struct devinet_sysctl_table
 {
        struct ctl_table_header *sysctl_header;
-       ctl_table devinet_vars[14];
+       ctl_table devinet_vars[15];
        ctl_table devinet_dev[2];
        ctl_table devinet_conf_dir[2];
        ctl_table devinet_proto_dir[2];

@@ -1062,6 +1062,9 @@
        {NET_IPV4_CONF_ARPFILTER, "arp_filter",
         &ipv4_devconf.arp_filter, sizeof(int), 0644, NULL,
         &proc_dointvec},
+        {NET_IPV4_CONF_ARP_SELF_ONLY, "arp_self_only",
+         &ipv4_devconf.arp_self_only, sizeof(int), 0644, NULL,
+         &proc_dointvec},
         {0}},
 
        {{NET_PROTO_CONF_ALL, "all", NULL, 0, 0555, devinet_sysctl.devinet_vars},{0}},