Skip to contentSkip to conversation

dhcp/bonding interaction question

DISCUSSION Chris Friesen 2 participants 5 messages Sep 23, 2008 – Sep 24, 2008

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

We've recently run into an interesting issue, and I was hoping to get some suggestions.

We've got a boot server configured with two bonded links using the XOR tx mode. Another board comes up. It's got two links which will eventually be bonded. It issues a DHCP request to the server over one of the links, which arrives on one of the bonded slaves. The server sends a reply, but the XOR results in the packet being sent back via the other slave. On the booting blade the xid in the packet doesn't match the xid for the device on which the packet was received, so the packet is dropped.

What's the proper solution here? Should dhcpd be forcing reply packets out the slave on which the packet was received?

Chris

#1 lore raw
Quoted 7 lines · Chris Friesen

We've got a boot server configured with two bonded links using the XOR tx mode. Another board comes up. It's got two links which will eventually be bonded. It issues a DHCP request to the server over one of the links, which arrives on one of the bonded slaves. The server sends a reply, but the XOR results in the packet being sent back via the other slave. On the booting blade the xid in the packet doesn't match the xid for the device on which the packet was received, so the packet is dropped.

Presumably the blade is matching xid on a per-interface basis.

I'm guessing your problem is really due to the nature of the
intra-chassis network on the blade system. If it's like the ones I'm familiar with (eth0 of all blades to a single switch, eth1 of all blades to a different switch, etc), then you can't configure the switches into an etherchannel group as can be done with the usual configuration (multiple ports on one switch). If the other blades in the chassis aren't also configured for etherchannel (balance-xor is an etherchannel compatible mode), then they'll see the same MAC arriving on multiple interfaces and not really be prepared to deal with it.

Quoted 2 lines

What's the proper solution here? Should dhcpd be forcing reply packets out the slave on which the packet was received?

Why is the blade issuing DHCP before the bond is configured?

I'm unsure as to whether dhcpd should force replies as
suggested, but recent changes to bonding/net core would probably permit dhcpd to run directly on the slave (it might need tweaking, I'm not sure) and accomplish that result. I don't think it would work on older versions of bonding, since there's no way to tell which slave a packet arrived on.

In that case, dhcpd would have to run on all slaves, since a
bond that uses DHCP for its address assignment might use any of its interfaces to issue the request, and it would accept a response on any interface.

	-J

---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]

#2 lore raw
Quoted 7 lines · Jay Vosburgh
Quoted 3 lines · Chris Friesen

On the
booting blade the xid in the packet doesn't match the xid for the device on which the packet was received, so the packet is dropped.

Presumably the blade is matching xid on a per-interface basis.

Yes, the dhcp code in the kernel does this.

Quoted 6 lines

I'm guessing your problem is really due to the nature of the
intra-chassis network on the blade system. If it's like the ones I'm familiar with (eth0 of all blades to a single switch, eth1 of all blades to a different switch, etc), then you can't configure the switches into an etherchannel group as can be done with the usual configuration (multiple ports on one switch).

This has been handled already. The two switches are a single etherchannel group and all blades (except the booting one) have etherchannel enabled.

Quoted 4 lines
Quoted 2 lines

What's the proper solution here? Should dhcpd be forcing reply packets out the slave on which the packet was received?

Why is the blade issuing DHCP before the bond is configured?

The blade is net-booting. The BIOS does DHCP to download the kernel, and this works. The kernel comes up and does DHCP to obtain an IP address so it can download the boot image, and this fails as described.

The etherchannel/bond link isn't set up until userspace is loaded and the bonding driver is insmodded (under multiple aliases with different sets of slaves).

Quoted 6 lines

I'm unsure as to whether dhcpd should force replies as suggested, but recent changes to bonding/net core would probably permit dhcpd to run directly on the slave (it might need tweaking, I'm not sure) and accomplish that result. I don't think it would work on older versions of bonding, since there's no way to tell which slave a packet arrived on.

That's not a problem. We've got an older version, but support for querying which slave a packet arrived on has been added.

Chris

#3 lore raw
Quoted 2 lines · Chris Friesen
Quoted 1 line · Jay Vosburgh

Chris Friesen <[email protected]> wrote:

[...]

Quoted 1 line

Yes, the dhcp code in the kernel does this.

Ah, I didn't know that.

Quoted 9 lines
Quoted 6 lines

I'm guessing your problem is really due to the nature of the
intra-chassis network on the blade system. If it's like the ones I'm familiar with (eth0 of all blades to a single switch, eth1 of all blades to a different switch, etc), then you can't configure the switches into an etherchannel group as can be done with the usual configuration (multiple ports on one switch).

This has been handled already. The two switches are a single etherchannel group and all blades (except the booting one) have etherchannel enabled.

Ok, so your problem (if I'm understanding things correctly) is
really a chicken-and-egg type of deal. Your entire network is all etherchannel, except for this blade, which needs to connect, temporarily, in a non-etherchannel manner in order to boot up and become etherchannel-ified (at which point it'll work).

One possible problem with hacking the dhcpd to send back out on
the receiving interface is that (because of the etherchannel) there's no general guarantee that the switch paths are symmetrical, particularly if a port is down somewhere.

Quoted 2 lines

That's not a problem. We've got an older version, but support for querying which slave a packet arrived on has been added.

If you're already rolling your own kernels, would it be easier
to remove the xid interface check from the kernel's dhcp client (i.e., accept the DHCP reply as if it had arrived on the proper interface for its xid)? You could make it into some kind of "netboot_on_etherchannel" option.

	-J

---
-Jay Vosburgh, IBM Linux Technology Center, [email protected]

#4 lore raw
Quoted 6 lines · Jay Vosburgh

Chris Friesen <[email protected]> wrote:
Ok, so your problem (if I'm understanding things correctly) is
really a chicken-and-egg type of deal. Your entire network is all etherchannel, except for this blade, which needs to connect, temporarily, in a non-etherchannel manner in order to boot up and become etherchannel-ified (at which point it'll work).

Basically, yes.

Quoted 4 lines

One possible problem with hacking the dhcpd to send back out on
the receiving interface is that (because of the etherchannel) there's no general guarantee that the switch paths are symmetrical, particularly if a port is down somewhere.

True.

Quoted 5 lines

If you're already rolling your own kernels, would it be easier
to remove the xid interface check from the kernel's dhcp client (i.e., accept the DHCP reply as if it had arrived on the proper interface for its xid)? You could make it into some kind of "netboot_on_etherchannel" option.

Yes, this is being considered as well. Initially I was against it due to the possibility of introducing unexpected behaviour, but making it a boot argument makes a lot of sense, especially since it would allow for more robust behaviour on certain fault scenarios.

Thanks,

Chris