kvm problem: bonding network interface breaks dhcp

Hi folks,

I am trying to use a bonding network interface as a bridge for a virtual machine (kvm). Host and guest are both running 2.6.31.5. Problem: The guest does not receive the DHCPOFFER reply sent by my dhcp server. There is no such problem if the host uses just a single network interface instead of bond0.

Looking at tcpdump on the Linux guest there are several dhcp discover packages like

15:17:44.005306 00:16:36:2f:f1:d2 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328) 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:36:2f:f1:d2, length 300, xid 0x4c31213d, secs 10, Flags [none]

          Client-Ethernet-Address 00:16:36:2f:f1:d2 [|bootp]

The dhcp server receives these packages, and sends out a reply

15:17:45.927589 00:16:36:2f:f1:d2 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 328) 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:36:2f:f1:d2, length 300, xid 0x4c31213d, secs 10, Flags [none]

          Client-Ethernet-Address 00:16:36:2f:f1:d2 [|bootp]
15:17:45.927658 00:15:17:94:16:65 > 00:16:36:2f:f1:d2, ethertype IPv4 (0x0800), length 364: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 350) 172.19.96.123.67 > 172.19.97.243.68: BOOTP/DHCP, Reply, length 322, xid 0x4c31213d, secs 10, Flags [none]
          Your-IP 172.19.97.243
          Client-Ethernet-Address 00:16:36:2f:f1:d2 [|bootp]

This reply never shows up on the guest.

iptable is not set, of course. sysctl.conf says

net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Any helpful comment would be highly appreciated.

Many thanx

Harri

The output of brctl show, ip addr list, and cat /proc/net/bonding/bond* might be helpful.

- Matt

Hi Matt,

Sure. Using the bridge on the bonding interface (while the guest was running) I got:

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.001517ab0a59       no              bond0
                                                        vnet0
# ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth2: <BROADCAST,MULTICAST,PROMISC,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 00:15:17:ab:0a:59 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:30:48:c6:e0:98 brd ff:ff:ff:ff:ff:ff
4: eth3: <BROADCAST,MULTICAST,PROMISC,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 00:15:17:ab:0a:59 brd ff:ff:ff:ff:ff:ff
5: _rename: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:30:48:c6:e0:99 brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST,PROMISC,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 00:15:17:ab:0a:59 brd ff:ff:ff:ff:ff:ff
7: eth5: <BROADCAST,MULTICAST,PROMISC,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 00:15:17:ab:0a:59 brd ff:ff:ff:ff:ff:ff
8: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 00:15:17:ab:0a:59 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::215:17ff:feab:a59/64 scope link
       valid_lft forever preferred_lft forever
51: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 00:15:17:ab:0a:59 brd ff:ff:ff:ff:ff:ff
    inet 172.19.96.25/23 brd 172.19.97.255 scope global br0
    inet6 fe80::215:17ff:feab:a59/64 scope link
       valid_lft forever preferred_lft forever
52: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether c6:d7:7b:fb:02:35 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::c4d7:7bff:fefb:235/64 scope link
       valid_lft forever preferred_lft forever
# cat /proc/net/bonding/bond*
Ethernet Channel Bonding Driver: v3.5.0 (November 4, 2008)

Bonding Mode: load balancing (round-robin) MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth2
MII Status: up
Link Failure Count: 0 Permanent HW addr: 00:15:17:ab:0a:59

Slave Interface: eth3
MII Status: up
Link Failure Count: 0 Permanent HW addr: 00:15:17:ab:0a:58

Slave Interface: eth4
MII Status: up
Link Failure Count: 0 Permanent HW addr: 00:15:17:ab:0a:5b

Slave Interface: eth5
MII Status: up
Link Failure Count: 0 Permanent HW addr: 00:15:17:ab:0a:5a

For not using bonding I got

# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.001517ab0a59       no              eth2
                                                        vnet0
# ip addr list
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 00:15:17:ab:0a:59 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::215:17ff:feab:a59/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:30:48:c6:e0:98 brd ff:ff:ff:ff:ff:ff
4: eth3: <BROADCAST,MULTICAST,PROMISC> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:15:17:ab:0a:58 brd ff:ff:ff:ff:ff:ff
5: _rename: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
    link/ether 00:30:48:c6:e0:99 brd ff:ff:ff:ff:ff:ff
6: eth4: <BROADCAST,MULTICAST,PROMISC> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:15:17:ab:0a:5b brd ff:ff:ff:ff:ff:ff
7: eth5: <BROADCAST,MULTICAST,PROMISC> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 00:15:17:ab:0a:5a brd ff:ff:ff:ff:ff:ff
8: bond0: <BROADCAST,MULTICAST,MASTER> mtu 1500 qdisc noqueue state DOWN
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
53: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    link/ether 00:15:17:ab:0a:59 brd ff:ff:ff:ff:ff:ff
    inet 172.19.96.25/23 brd 172.19.97.255 scope global br0
    inet6 fe80::215:17ff:feab:a59/64 scope link
       valid_lft forever preferred_lft forever
54: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 500
    link/ether fe:2f:ce:cc:ec:ac brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc2f:ceff:fecc:ecac/64 scope link
       valid_lft forever preferred_lft forever

Hope this helps. Regards

Harri

Can you tcpdump on bond0, br0, vnet0, and the guest's interface to see where the packet is lost?

error compiling committee.c: too many arguments to function

Sure. Using the tcpdump command line:

tcpdump -i br0 -w /var/tmp/tcpdump.br0 ether host 00:16:36:2f:f1:d2

(similar for other interfaces) I can see the DHCPOFFER coming from my dhcp server on bond0:

11:00:08.237350 00:15:17:91:3f:59 > 00:16:36:2f:f1:d2, ethertype IPv4 (0x0800), length 364: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 350)

    172.19.96.124.67 > 172.19.97.250.68: BOOTP/DHCP, Reply, length 322, xid 0x78fb274e, secs 3, Flags [none]
          Your-IP 172.19.97.250
          Client-Ethernet-Address 00:16:36:2f:f1:d2 [|bootp]

It is also visible on br0:

11:00:08.237350 00:15:17:91:3f:59 > 00:16:36:2f:f1:d2, ethertype IPv4 (0x0800), length 364: (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 350)

    172.19.96.124.67 > 172.19.97.250.68: BOOTP/DHCP, Reply, length 322, xid 0x78fb274e, secs 3, Flags [none]
          Your-IP 172.19.97.250
          Client-Ethernet-Address 00:16:36:2f:f1:d2 [|bootp]

But it is not visible on vnet0, and of course not on the guest. All I see there are the DHCPDISCOVER calls sent by the guest, and some IPv6 traffic:

:
11:00:05.245090 00:16:36:2f:f1:d2 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:36:2f:f1:d2, length 300 11:00:05.245247 00:16:36:2f:f1:d2 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:36:2f:f1:d2, length 300 11:00:08.237025 00:16:36:2f:f1:d2 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:36:2f:f1:d2, length 300 11:00:08.237135 00:16:36:2f:f1:d2 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:36:2f:f1:d2, length 300 11:00:08.237147 00:16:36:2f:f1:d2 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:36:2f:f1:d2, length 300 11:00:08.237196 00:16:36:2f:f1:d2 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:36:2f:f1:d2, length 300 11:00:08.883308 00:16:36:2f:f1:d2 > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 11:00:08.883381 00:16:36:2f:f1:d2 > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 11:00:08.883411 00:16:36:2f:f1:d2 > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 11:00:08.883419 00:16:36:2f:f1:d2 > 33:33:00:00:00:16, ethertype IPv6 (0x86dd), length 90: :: > ff02::16: HBH ICMP6, multicast listener report v2, 1 group record(s), length 28 11:00:14.238455 00:16:36:2f:f1:d2 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:36:2f:f1:d2, length 300 11:00:14.238523 00:16:36:2f:f1:d2 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:36:2f:f1:d2, length 300 11:00:14.238544 00:16:36:2f:f1:d2 > ff:ff:ff:ff:ff:ff, ethertype IPv4 (0x0800), length 342: 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP, Request from 00:16:36:2f:f1:d2, length 300 :

I can send you the complete tcpdumps, if you are interested?

Regards

Harri

So, it looks like a bridging problem. Can you send this the bridge maintainers (Stephen Hemminger <[email protected]>, [email protected])?

error compiling committee.c: too many arguments to function

The thread can be found here:

https://lists.linux-foundation.org/pipermail/bridge/2009-November/006749.html

Regards

Harri

Perhaps its related to your kernel version?

With RHEL5 as the host OS I have not seen any problems with bonding and dhcp in either the host or the guest. The stack is:

   ------        ------
  | tapX |  ... | tapY |
   ------        ------
        \        /
        ---------
       |   br0   |
        ---------
            |
         -------
        | bond0 |
         -------
        /       \
     ------    ------
    | eth0 |  | eth1 |
     ------    ------

With the following bonding options:

BONDING_OPTS="mode=active-backup primary=eth0 miimon=100"

David Ahern