Skip to contentSkip to conversation

Ethernet NIC dual homing

DISCUSSION willy tarreau 6 participants 14 messages Oct 29, 2001 – Oct 30, 2001

lore
Share & tools
lore
#0 lore raw

Hi Laurent,

Quoted 1 line

Does someone know if there is some work in the area

of NIC

Quoted 1 line

dual homing ?

I have implemented this for 2.2 kernel a while ago, and Chad
Tindel has completed the port to 2.4. Some other contributors
have added features such as XOR distribution. You can take
a look at it, kernel 2.4 patches are on :

http://sf.net/projects/bonding/

and 2.2 patches are on :

http://www-miaif.lip6.fr/willy/linux-patches/bonding/

Regards,
Willy

___________________________________________________________
Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français ! Yahoo! Courrier : http://courrier.yahoo.fr

#1 lore raw
Quoted 11 lines · willy tarreau

Hi Laurent,

Quoted 2 lines

Does someone know if there is some work in the area of NIC dual homing ?

I have implemented this for 2.2 kernel a while ago,
and Chad Tindel has completed the port to 2.4. Some other contributors have added features such as XOR distribution. You can take a look at it, kernel 2.4 patches are on :

Quoted 6 lines

http://sf.net/projects/bonding/

and 2.2 patches are on :

http://www-miaif.lip6.fr/willy/linux-patches/bonding/

Thanks for the pointers.

Currently only the link status is used to monitor a NIC.
So it would be nice if an ioctl was available to force a NIC switch-over (especially in active-backup policy). This could be used by a user-space daemon in case for instance no traffic is detected.

I see that the bonding driver is included in 2.2.18, what is its status in 2.4.x ?

Regards,

Laurent

#2 lore raw
Quoted 7 lines

Currently only the link status is used to monitor a NIC.
So it would be nice if an ioctl was available to force a NIC switch-over
(especially in active-backup policy). This could be used by a user-space
daemon in case for instance no traffic is detected.

the 2.4 driver provides a mode which sends ARP packets to test the link (far more reliable than MII), and the appropriate ioctl for the NIC switch-over you need. It is available to user through ifenslave -c bond0 eth0 for example.

Quoted 2 lines

I see that the bonding driver is included in 2.2.18, what is its status in 2.4.x ?

well, it works for me each time I download a new release, but I don't have prod servers on it to test for a long time, nor have I passed all the intensive tests that Constantine Gavrilov did when I was working on 2.2. BTW, the last release (20011026) seems to have a buggy ifenslave which activates all the NICs flags (to be confirmed, just seen this today and replaced with the previous one). If this is confirmed, I'll mail
Chad about this problem.

If there are still people interested in 2.2, I'll try to find some time to back-port the switch-over ioctl from 2.4.

Regards,
Willy

___________________________________________________________
Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français ! Yahoo! Courrier : http://courrier.yahoo.fr

#3 lore raw
Quoted 5 lines · willy tarreau

the 2.4 driver provides a mode which sends ARP packets to test the link (far more reliable than MII), and the appropriate ioctl for the NIC switch-over you need. It is available to user through ifenslave -c bond0 eth0 for example.

Are there issues with using MII to detect link state? I thought it was fairly reliable...

How are you using arp packets to detect if the link is up? Sending it out to your own MAC address?

Chris

Signature · 4 lines
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]
#4 lore raw
Quoted 2 lines

Are there issues with using MII to detect link
state? I thought it was fairly reliable...

no, there are examples with too long lines, or scratched wires where the links stay up, but even arp doesn't work.

Quoted 2 lines

How are you using arp packets to detect if the link is up? Sending it out to your own MAC address?

no, simply sending ARP request for a known IP address which will reply, so generate traffic that can be counted to tell wether a NIC seems working or not.

I didn't try to set my own IP address though. Perhaps with arp_filter properly set this could be usefull...

Regards,
Willy

___________________________________________________________
Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français ! Yahoo! Courrier : http://courrier.yahoo.fr

#5 lore raw
In reply to Christopher Friesen
Quoted 2 lines · Christopher Friesen

Are there issues with using MII to detect link state? I thought it was fairly reliable...

It doesn't work to detect link state through bridging device (say, bridged ethernet over T3). The T3 might go down, but your MII link to the local router will remain "up", so you will never know about the loss of link and your packets will happily go into the void...

-Dan

Signature · 1 line
[-] Omae no subete no kichi wa ore no mono da. [-]
#6 lore raw
Quoted 9 lines · Dan Hollis
Quoted 3 lines · Christopher Friesen

Are there issues with using MII to detect link state? I thought
it was fairly
reliable...

It doesn't work to detect link state through bridging device (say, bridged ethernet over T3). The T3 might go down, but your MII link to the local router will remain "up", so you will never know about the loss of link and your packets will happily go into the void...

ARP isn't going to do much for you once the failure is beyond the local segment, is it?

Signature · 1 line
/Jonathan Lundell.
#7 lore raw
Quoted 11 lines · Jonathan Lundell
Quoted 8 lines · Dan Hollis
Quoted 3 lines · Christopher Friesen

Are there issues with using MII to detect link state? I thought
it was fairly
reliable...

It doesn't work to detect link state through bridging device (say, bridged ethernet over T3). The T3 might go down, but your MII link to the local router will remain "up", so you will never know about the loss of link and your packets will happily go into the void...

ARP isn't going to do much for you once the failure is beyond the local segment, is it?

But you can use it to determine end-to-end link status. MII is useless for that when you're going through a bridge.

So ARP is *perfect* for this situation and is *exactly* what is needed. When you determine link is down using end-to-end status like ARP then you can take the device out of the bonding queue. Presto, 100% perfect failover.

-Dan

Signature · 1 line
[-] Omae no subete no kichi wa ore no mono da. [-]
#8 lore raw
In reply to Dan Hollis
Followup to:  <p05100304b803c6908755@[10.128.7.49]>
By author:    Jonathan Lundell <[email protected]>
In newsgroup: linux.dev.kernel
Quoted 9 lines
Quoted 5 lines

It doesn't work to detect link state through bridging device (say, bridged ethernet over T3). The T3 might go down, but your MII link to the local router will remain "up", so you will never know about the loss of link and your packets will happily go into the void...

ARP isn't going to do much for you once the failure is beyond the local segment, is it?

ARP is broadcast to the layer 2 local segment; link detection refers to the layer 1 local segment, which is not necessarily the same.

On the other hand, doing link detection is extremely useful for a portable computer: when I plug in my Ethernet cable in a portable system I want it to try to start doing DHCP detection and anything else that is normally associated with the interface being "up" at that time.

-hpa

Signature · 3 lines
<[email protected]> at work, <[email protected]> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<[email protected]>
#9 lore raw
Quoted 4 lines · H. Peter Anvin
Quoted 3 lines

ARP isn't going to do much for you once the failure is beyond the
local segment, is it?

Quoted 8 lines

ARP is broadcast to the layer 2 local segment; link detection refers to the layer 1 local segment, which is not necessarily the same.

On the other hand, doing link detection is extremely useful for a portable computer: when I plug in my Ethernet cable in a portable system I want it to try to start doing DHCP detection and anything else that is normally associated with the interface being "up" at that time.

I'm not planning to use bonding on my notebook any time soon.

But what I meant was bonding's use of ARP to determine whether the connection is good (or rather, bad, even when the link is up), when the connection is routed via level 3. Seems to me you'd need a level 3 protocol (say ICMP) rather than ARP.

Signature · 1 line
/Jonathan Lundell.
#10 lore raw
Quoted 4 lines · Jonathan Lundell

But what I meant was bonding's use of ARP to determine whether the connection is good (or rather, bad, even when the link is up), when the connection is routed via level 3. Seems to me you'd need a level 3 protocol (say ICMP) rather than ARP.

bonding isn't for layer 3. it's layer 2. layer 3 you use equal cost multipath or other method for load balancing.

-Dan

Signature · 1 line
[-] Omae no subete no kichi wa ore no mono da. [-]
#11 lore raw
In reply to Dan Hollis
Quoted 4 lines · Jonathan Lundell

But what I meant was bonding's use of ARP to determine whether the connection is good (or rather, bad, even when the link is up), when the connection is routed via level 3. Seems to me you'd need a level 3 protocol (say ICMP) rather than ARP.

This is what we've done here at work. We use a combination of MII for fast detection of local link loss, and ICMP ping packets to highly available hosts to test the network path (with somewhat slower response time).

Chris

Signature · 4 lines
Chris Friesen                    | MailStop: 043/33/F10  
Nortel Networks                  | work: (613) 765-0557
3500 Carling Avenue              | fax:  (613) 765-2986
Nepean, ON K2H 8E9 Canada        | email: [email protected]
#12 lore raw
In reply to willy tarreau
Quoted 6 lines · Laurent Deniel

Currently only the link status is used to monitor a NIC.
So it would be nice if an ioctl was available to force a NIC switch-over (especially in active-backup policy). This could be used by a user-space daemon in case for instance no traffic is detected.

Quoted 6 lines

I see that the bonding driver is included in 2.2.18, what is its status in 2.4.x ?

Regards,

Laurent

Hmm, it seems that a lot of good stuff (e.g. ARP monitoring and SIOCBONDCHANGEACTIVE ioctl) are implemented in the bonding patch for 2.4.13. Will it be included in the mainstream 2.4.x kernel or is it a 2.5 thing ?

#13 lore raw
Quoted 6 lines

Hmm, it seems that a lot of good stuff (e.g. ARP monitoring and
SIOCBONDCHANGEACTIVE ioctl) are implemented in the bonding patch for 2.4.13.
Will it be included in the mainstream 2.4.x kernel or is it a 2.5 thing ?

Personnaly, I don't know. Chad now maintains the project so he may have better opinions about this. But I'd like to see it in the 2.4 once well tested since it interests lots of people, and it allows us to put linux boxes in more critical environments.

Regards,
Willy

___________________________________________________________
Do You Yahoo!? -- Une adresse @yahoo.fr gratuite et en français ! Yahoo! Courrier : http://courrier.yahoo.fr