ARP responses broken!

Hello-

This is a known 'feature' of the Linux kernel, and can help with load sharing and fault tolerance. However, it can also cause problems (such as when one nic in a multi-nic machine fails and you don't know right away).

There are three 'solutions' I know of:

* In recent 2.2 kernels, it was possible to fix this by doing the following as
root:
# Start the hiding interface functionality
echo 1 > /proc/sys/net/ipv4/conf/all/hidden
# Hide all addresses for this interface
echo 1 > /proc/sys/net/ipv4/conf/<interface_name>/hidden
but 2.4 doesn't have that option, for technical reasons.

* Use 'ifconfig -arp ...' to force an interface not to respond to ARP
requests. Hosts which want to send to that interface may need to manually add the proper mac address to their ARP tables with 'arp -s'.

* Use a packet filtering tool (iptables arp filter module, for example) and
just filter the ARP requests and ARP replies so that only the proper set get through, i.e. when an arp request for the mac address of an interface arrives, filter out arp replies from all the other interfaces.

There have been a few threads on this on the linux-kernel mailing list. Search your favorite archive for them.

-Eric

--------------------------------------------

 Eric H. Weigle   CCS-1, RADIANT team
 [email protected]     Los Alamos National Lab
 (505) 665-4937   http://home.lanl.gov/ehw/

--------------------------------------------

I tought this for a while and this does not help load sharing neighter or fault tolerance. Causes problem with router environment. I use different cards to load the problem by assigning different addresses to these and by pointing these addresses with routes so I use the IP to mark a device.

Only case where this would help with "fault tolerance" is if I assign address to other device that is not marked as up, it would still be possible to see the address via other device, and this goes way off.

I don't see good valid reasons, I implemented this by myself to the kernel with very little effort, only thing is that my patch makes it not possible to have the same address on multible interfaces, this is altought very easy to fix. FIB code lacks a way to detect if there are multible interfaces for a single IP.

Manual ARP, uh, sounds as nasty as can be.

There is no ARP module for netfilter and this is such low level issue that would see requires another way to be solved by all means.

It is not working correctly if there are two ARP replies for a ARP query, another one from interface that DOES NOT have the address assigned to it. Basically I should have a firewall rule that would instantly DROP all packets that come to the interface with address that it does not have if I don't have forwarding enabled.

Well, this is not solved so we probaply need one more.

The code I used to do the trick at my network was as simple as this, in function arp_rcv, the problem is ip_dev_find that does know if there are other devices with same IP address.

@@ -571,6 +571,7 @@
        int addr_type;
        struct in_device *in_dev = in_dev_get(dev);
        struct neighbour *n;
+       struct net_device *devx = NULL;
 
 /*
  *     The hardware length of the packet should match the hardware length

@@ -728,8 +729,10 @@
 
                rt = (struct rtable*)skb->dst;
                addr_type = rt->rt_type;
 
-               if (addr_type == RTN_LOCAL) {
+               devx = ip_dev_find(tip);
+               if (addr_type == RTN_LOCAL &&
+                   devx == dev) {
                        n = neigh_event_ns(&arp_tbl, sha, &sip, dev);
                        if (n) {
                                arp_send(ARPOP_REPLY,ETH_P_ARP,sip,dev,tip,sha,dev->dev_addr,sha);

- Sampsa Ranta
[email protected]

Or use arpfilter in even newer 2.2 kernels; which filters based on the routing table. "hidden" is quite a sledgehammer which often does more harm than good.

-Andi

Does arpfilter exist in 2.4 kernels?

/Martin

Not yet, will be merged very soon. I can send you a patch if you need it urgently.

-Andi

On Tue, 17 Apr 2001, Andi Kleen wrote:

[snip]

No I don't need it urgently.
I was asking because I had this problem before (router with two cards against one physical subnet) and arpwatch complained that the router kept switching MACaddresses all the time.

/Martin

Ok, I was ignorant of the arp filter functionality in 2.2. I found an old (probably painfully out-of-date) posting the patch Andi Kleen was referring to in the archive, but I've not used it.
http://www.uwsg.indiana.edu/hypermail/linux/kernel/0101.2/1198.html

Indeed, the default behavior does cause problems in a router environment, but this only happens when multiple nics are on the same subnet; in a 'true' router each nic would be on a separate subnet. Regardless, I personally use FreeBSD when I need a router (Horrors! ;)

What I meant by load sharing was implicit sharing rather than explicit sharing; when an ARP request comes the reply the host gets may have the MAC address of NICs other than the one explicitly bound to the given IP-- thus different hosts will semi-randomly get different MAC addresses and thus send to different NICs; this implicit sharing completely hoses explicit load sharing.

And as for fault tolerance, here's what happened to me, as I mentioned in another message: We have a 8-node cluster with 2 nics, a eepro and a gig-e acenic in each node. A while back the acenic driver had some problems and would silently fail after a while; the arp reponse behavior allowed the cluster to remain 'up' long enough to finish the jobs we assigned to it (although performance sucked since all traffic went over the eepros). After we were done, we could ifdown/ifup the interfaces and all was good. Again, this is a sort of 'implicit' fault tolerance, rather than a more explicit form where the card goes down, we get some sort of notification, and it fails over to the other card explicitly.

Well, Yes, but that's not really the issue. The problem is 'what is the proper *default* behavior of the Linux ARP subsystem'... this code changes it, which is probably more of a political than technical decision. Where you (and I) see 'broken' others see 'feature' :/

-Eric

--------------------------------------------

 Eric H. Weigle   CCS-1, RADIANT team
 [email protected]     Los Alamos National Lab
 (505) 665-4937   http://home.lanl.gov/ehw/

--------------------------------------------

oh heck, why not post a third patch for this problem. here's the one i wrote.

-dean

@@ -737,10 +737,12 @@
 		addr_type = rt->rt_type;
 
 		if (addr_type == RTN_LOCAL) {
+			if ((rt->rt_flags&RTCF_DIRECTSRC) || IN_DEV_PROXY_ARP(in_dev)) {
 			n = neigh_event_ns(&arp_tbl, sha, &sip, dev);
 			if (n) {
 				arp_send(ARPOP_REPLY,ETH_P_ARP,sip,dev,tip,sha,dev->dev_addr,sha);
 				neigh_release(n);
+			}
 			}
 			goto out;
 		} else if (IN_DEV_FORWARD(in_dev)) {

That sounds like a bug in arpwatch. A box can have multiple mac addresses. Its probably a tricky one to handle but arpwatch I guess should spot and cope with repeated transitions between the same set of addresses as one warning

Well, two. Or three.

- Hey, IP x changed from mac X to mac Y.
- Hey, IP x changed back again to X.
- Hmm. IP X seems to be using both Mac X and and Mac Y. No further warnings will be issued about this.

If someone is taking over an IP address (which is especially what arpwatch should be looking for), this is exactly what you'll see. Having the issue be ignored after one warning is bad.

Oh, and I know people who swear that this would be an invalid configuration, so that it is good for arpwatch to should loud and clear about it...

Roger.

** [email protected] ** http://www.BitWizard.nl/ ** +31-15-2137555 **
*-- BitWizard writes Linux device drivers for any device you may have! --*
* There are old pilots, and there are bold pilots. 
* There are also old, bald pilots.