Skip to contentSkip to conversation

quick help with bonding?

DISCUSSION Chris Friesen 4 participants 14 messages Mar 29, 2007 – Apr 2, 2007

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

I"m doing some experimenting with a new network driver that receives jumbo frames into multiple separate pages that are then joined together in a single sk_buff using skb_fill_page_desc().

It behaved fairly well with standard networking, but its behaving strangely with bonding added to the mix.

Could someone either point me to the bonding high level design document (couldn't find one at the sourceforge project page) or else give me a quick overview of the code path followed by an incoming packet when bonding is involved?

Thanks,

Chris Friesen

#1 lore raw
Quoted 4 lines · Chris Friesen

Could someone either point me to the bonding high level design document (couldn't find one at the sourceforge project page) or else give me a quick overview of the code path followed by an incoming packet when bonding is involved?

There really isn't a high level design document.

The input path goes from the driver, which (probably) calls
netif_receive_skb. That function does its processing whatnot, the only special step for bonding is the processing done by skb_bond() which assigns the packet to the bonding device. In the current mainline, skb_bond() also does some stuff to drop traffic on inactive slaves as the like.

After that, the packet follows the regular input path in
netif_skb_receive.

	-J

---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]

#2 lore raw
In reply to Chris Friesen
Quoted 7 lines · Chris Friesen

I"m doing some experimenting with a new network driver that receives jumbo frames into multiple separate pages that are then joined together in a single sk_buff using skb_fill_page_desc().

It behaved fairly well with standard networking, but its behaving strangely with bonding added to the mix.

Can you elaborate on what isn't going well with this driver/hardware?

#3 lore raw
Quoted 1 line · Andy Gospodarek

Can you elaborate on what isn't going well with this driver/hardware?

I have a ppc64 blade running a customized 2.6.10. At init time, two of our gigE links (eth4 and eth5) are bonded together to form bond0. This link has an MTU of 9000, and uses arp monitoring. We're using an ethernet driver with a modified RX path for jumbo frames[1]. With the stock driver, it seems to work fine.

The problem is that eth5 seems to be bouncing up and down every 15 sec or so (see the attached log excerpt). Also, "ifconfig" shows that only 3 packets totalling 250 bytes have gone out eth5, when I know that the arp monitoring code from the bond layer is sending 10 arps/sec out the link.

eth5 Link encap:Ethernet HWaddr 00:03:CC:51:01:3E

          inet6 addr: fe80::203:ccff:fe51:13e/64 Scope:Link
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:9000  Metric:1
          RX packets:119325 errors:90283 dropped:90283 overruns:90283 
frame:0
          TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:8978310 (8.5 MiB)  TX bytes:250 (250.0 b)
          Base address:0x3840 Memory:92220000-92240000

I had initially suspected that it might be due to the "u32 jiffies" stuff in bonding.h, but changing that doesn't seem to fix the issue.

If I boot the system and then log in and manually create the bond link (rather than it happening at init time) then I don't see the problem.

If it matters at all, normally the system boots from eth4. I'm going to try booting from eth6 and see if the problem still occurs.

Chris

[1] I'm not sure if I'm supposed to mention the specific driver, as it hasn't been officially released yet, so I'll keep this high-level. Normally for jumbo frames you need to allocate a large physically contiguous buffer. With the modified driver, rather than receiving into a contiguous buffer the incoming packet is split across multiple pages which are then reassembled into an sk_buff and passed up the link.

Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: ARP monitoring set to 100 ms with 2 target(s): 172.24.136.0 172.24.137.0
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: ARP monitoring set to 100 ms with 2 target(s): 172.25.136.0 172.25.137.0
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: Warning: failed to get speed/duplex from eth4, speed forced to 100Mbps, duplex forced to Full.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: enslaving eth4 as an active interface with an up link.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: Warning: failed to get speed/duplex from eth5, speed forced to 100Mbps, duplex forced to Full.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: enslaving eth5 as an active interface with an up link.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth5 to be reset in 30000 msec.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now down.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth4 to be reset in 30000 msec.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth4 is now down.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: now running without any active interface !
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: link status definitely up for interface eth5
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth4
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth4 is now up
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth5 to be reset in 30000 msec.
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now down.
Mar 29 20:54:09 base0-0-0-5-0-11-1 kernel: bonding: interface eth4 reset delay set to 600 msec.
Mar 29 20:54:59 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5
Mar 29 20:54:59 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now up
Mar 29 20:54:59 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth5 to be reset in 30000 msec.
Mar 29 20:54:59 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now down.
Mar 29 20:55:15 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5
Mar 29 20:55:15 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now up
Mar 29 20:55:15 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth5 to be reset in 30000 msec.
Mar 29 20:55:15 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now down.
Mar 29 20:55:30 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5
Mar 29 20:55:30 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now up
Mar 29 20:55:30 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth5 to be reset in 30000 msec.
Mar 29 20:55:30 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now down.
Mar 29 20:55:45 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5
Mar 29 20:55:45 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now up
Mar 29 20:55:46 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth5 to be reset in 30000 msec.
Mar 29 20:55:46 base0-0-0-5-0-11-1 kernel: bonding: bond0: interface eth5 is now down.
#4 lore raw

Chris Friesen <[email protected]> wrote:
[...]

Quoted 5 lines

I have a ppc64 blade running a customized 2.6.10. At init time, two of our gigE links (eth4 and eth5) are bonded together to form bond0. This link has an MTU of 9000, and uses arp monitoring. We're using an ethernet driver with a modified RX path for jumbo frames[1]. With the stock driver, it seems to work fine.

2.6.10 is pretty old, and there have been a number of fixes to
the bonding ARP monitor since then, so it may be that it is simply misbehaving (presuming that you're running the 2.6.10 bonding driver). Are you in a position to test against a more recent kernel (and/or bonding driver)? Does the miimon misbehave in a similar fashion?

Quoted 4 lines

The problem is that eth5 seems to be bouncing up and down every 15 sec or so (see the attached log excerpt). Also, "ifconfig" shows that only 3 packets totalling 250 bytes have gone out eth5, when I know that the arp monitoring code from the bond layer is sending 10 arps/sec out the link.

[...]

Quoted 1 line
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth4 to be reset in 30000 msec.

[...]

Quoted 1 line
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5

These two messages (which appear a number of times in your log
excerpt) are not from the standard mainline bonding driver, even in 2.6.10. I don't know what this is all about.

Quoted 2 lines

If I boot the system and then log in and manually create the bond link (rather than it happening at init time) then I don't see the problem.

I would hazard to guess that it's an ARP monitor problem; older
versions of the ARP monitor had less than intelligent means to figure out what the bond's IP address is (to use for the probes). This, along with some logic problems in the monitor code itself, led to various problems with the ARP probes and the sort of "up / down" cycle of behavior you seem to be seeing.

	-J

---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]

#5 lore raw
Quoted 47 lines · Jay Vosburgh

Chris Friesen <[email protected]> wrote:
[...]

Quoted 6 lines

I have a ppc64 blade running a customized 2.6.10. At init time, two of our gigE links (eth4 and eth5) are bonded together to form bond0. This link has an MTU of 9000, and uses arp monitoring. We're using an ethernet driver with a modified RX path for jumbo frames[1]. With the stock driver, it seems to work fine.

2.6.10 is pretty old, and there have been a number of fixes to
the bonding ARP monitor since then, so it may be that it is simply misbehaving (presuming that you're running the 2.6.10 bonding driver). Are you in a position to test against a more recent kernel (and/or bonding driver)? Does the miimon misbehave in a similar fashion?

Quoted 5 lines

The problem is that eth5 seems to be bouncing up and down every 15 sec or so (see the attached log excerpt). Also, "ifconfig" shows that only 3 packets totalling 250 bytes have gone out eth5, when I know that the arp monitoring code from the bond layer is sending 10 arps/sec out the link.

[...]

Quoted 2 lines
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: scheduling interface eth4 to be reset in 30000 msec.

[...]

Quoted 2 lines
Mar 29 20:54:08 base0-0-0-5-0-11-1 kernel: bonding: bond0: cancelled scheduled reset of interface eth5

These two messages (which appear a number of times in your log
excerpt) are not from the standard mainline bonding driver, even in 2.6.10. I don't know what this is all about.

Quoted 3 lines

If I boot the system and then log in and manually create the bond link (rather than it happening at init time) then I don't see the problem.

I would hazard to guess that it's an ARP monitor problem; older
versions of the ARP monitor had less than intelligent means to figure out what the bond's IP address is (to use for the probes). This, along with some logic problems in the monitor code itself, led to various problems with the ARP probes and the sort of "up / down" cycle of behavior you seem to be seeing.

Quoted 6 lines
	-J

---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]
-

I'll second what Jay said. I support a version of the 2.6.10 kernel with bonding, and I needed to upgrade the bonding that was native to 2.6.10 to get reasonable behavior. You may also need a newer ifenslave.

It also looks like the mii interface is not well-behaved, because of the initialization messages related to link speed.

#6 lore raw
In reply to Jay Vosburgh
Quoted 5 lines · Jay Vosburgh

2.6.10 is pretty old, and there have been a number of fixes to
the bonding ARP monitor since then, so it may be that it is simply misbehaving (presuming that you're running the 2.6.10 bonding driver). Are you in a position to test against a more recent kernel (and/or bonding driver)? Does the miimon misbehave in a similar fashion?

Testing a more recent kernel is problematic. A new bonding driver could be possible, assuming the code hasn't changed too much.

I just did another experiment. Normally we boot via eth4 (which then becomes part of the bond with eth5 at init time). If I boot via eth6 instead, it appears as though the problem doesn't show up.

Chris

#7 lore raw
Quoted 8 lines · Chris Friesen
Quoted 5 lines · Jay Vosburgh

2.6.10 is pretty old, and there have been a number of fixes to
the bonding ARP monitor since then, so it may be that it is simply misbehaving (presuming that you're running the 2.6.10 bonding driver). Are you in a position to test against a more recent kernel (and/or bonding driver)? Does the miimon misbehave in a similar fashion?

Quoted 6 lines

Testing a more recent kernel is problematic. A new bonding driver could be possible, assuming the code hasn't changed too much.

I just did another experiment. Normally we boot via eth4 (which then becomes part of the bond with eth5 at init time). If I boot via eth6 instead, it appears as though the problem doesn't show up.

Well, if you're still inclined to investigate, you may want to
inspect the ARP probes generated by bonding in the "bad" situation. I don't really have any evidence to back it up, but one guess is that the IP detection stuff in the ARP monitor is getting messed up. I'd check to see if the ARP probes have the correct source IP address (which, in the 2.6.10 era bonding, is determined only once by inspection of outbound ARP traffic, and never updated). If you're not using active-backup mode (you didn't say, and I can't tell from your log excerpt), then the ARP monitor may not work at all (since it will send ARP probes with an IP source of all zeros).

If bad ARP probe source addresses are your problem, then that is
fixed in a later version of bonding, although the changes would require some rework to backport to 2.6.10 (if they can be backported).

	-J

---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]

#8 lore raw
In reply to Chris Friesen
Quoted 10 lines · Chris Friesen
Quoted 5 lines · Jay Vosburgh

2.6.10 is pretty old, and there have been a number of fixes to
the bonding ARP monitor since then, so it may be that it is simply misbehaving (presuming that you're running the 2.6.10 bonding driver). Are you in a position to test against a more recent kernel (and/or bonding driver)? Does the miimon misbehave in a similar fashion?

Testing a more recent kernel is problematic. A new bonding driver could be possible, assuming the code hasn't changed too much.

If you are looking for a decent source for patches you could consider downloading the latest source-rpm from RHEL4/CentOS4. The bonding driver in those releases have been updated to much later code and I can tell you from personal experience they work pretty well. You may need to do some backporting to get the latest arp-monitoring features, but let me know if you need a hand with that, I might have some laying around. ;)

Quoted 3 lines

I just did another experiment. Normally we boot via eth4 (which then becomes part of the bond with eth5 at init time). If I boot via eth6 instead, it appears as though the problem doesn't show up.

Does eth6 use the same hardware/driver as eth4/5? (Sorry if I missed that in the thread, but didn't see if you indicated that it did.)

#9 lore raw
Quoted 7 lines · Andy Gospodarek

If you are looking for a decent source for patches you could consider downloading the latest source-rpm from RHEL4/CentOS4. The bonding driver in those releases have been updated to much later code and I can tell you from personal experience they work pretty well. You may need to do some backporting to get the latest arp-monitoring features, but let me know if you need a hand with that, I might have some laying around. ;)

I'm just about to load a kernel with a backport of bonding from 2.6.14.
I'll try it out and if it doesn't help I'll try the RHEL4 one.

Quoted 2 lines

Does eth6 use the same hardware/driver as eth4/5? (Sorry if I missed that in the thread, but didn't see if you indicated that it did.)

No, eth6 is an AMD-8111.

Chris

#10 lore raw
Quoted 9 lines · Chris Friesen
Quoted 4 lines · Andy Gospodarek

If you are looking for a decent source for patches you could consider downloading the latest source-rpm from RHEL4/CentOS4. The bonding driver in those releases have been updated to much later code and I can tell you from personal experience they work pretty well.

I'm just about to load a kernel with a backport of bonding from 2.6.14.
I'll try it out and if it doesn't help I'll try the RHEL4 one.

No joy on the 2.6.14 backport, so I guess I'll try the RHEL4 route.

Chris

#11 lore raw
Quoted 8 lines · Chris Friesen
Quoted 6 lines · Chris Friesen
Quoted 4 lines · Andy Gospodarek

If you are looking for a decent source for patches you could consider downloading the latest source-rpm from RHEL4/CentOS4. The bonding driver in those releases have been updated to much later code and I can tell you from personal experience they work pretty well.

Quoted 5 lines
Quoted 2 lines

I'm just about to load a kernel with a backport of bonding from 2.6.14.
I'll try it out and if it doesn't help I'll try the RHEL4 one.

No joy on the 2.6.14 backport, so I guess I'll try the RHEL4 route.

Ah, ok. I'm not too sure how different the 2.6.9 and 2.6.10 bonding code was, so it might take a little tweaking but I'm guessing there won't be significant differences.

#12 lore raw
In reply to Chris Friesen
Quoted 1 line · Chris Friesen

No joy on the 2.6.14 backport, so I guess I'll try the RHEL4 route.

Bonding driver from 2.6.9-42.0.8.EL doesn't help at all, at least with the module parms I was using before.

Switching to miimon doesn't help either.

Chris

#13 lore raw
In reply to Chris Friesen
Quoted 1 line · Chris Friesen

It behaved fairly well with standard networking, but its behaving strangely with bonding added to the mix.

Thanks to everyone for the suggestions and help.

It turns out it was actually a driver problem that was more easily manifested by the timing of the way bonding changed the link MTU.

Chris