Skip to contentSkip to conversation

bonding can't change to another slave if you ifdown the active slave

DISCUSSION Weiping Pan 2 participants 5 messages Mar 8, 2011 – Mar 9, 2011

Subsystems (2)
Networking Network drivers
lore
Share & tools
lore

Weiping Pan

author
#0 lore raw
Quoted 60 lines · Nicolas de Pesloüan
Quoted 53 lines · Weiping Pan
Quoted 26 lines · Nicolas de Pesloüan
Quoted 11 lines · Andy Gospodarek
Quoted 10 lines · Weiping Pan

Hi,

I'm doing some Linux bonding driver test, and I find a problem in balance-rr mode.
That's it can't change to another slave if you ifdown the active slave. Any comments are warmly welcomed!

regards
Weiping Pan

Quoted 7 lines
Quoted 2 lines

My host is Fedora 14, and I install VirtualBox (4.0.2), and enable 4 nics for the guest system.

Does this mean you are passing 4 NICs from your host to your guest (maybe via direct pci-device assignment to the guest) or are you creating 4 virtual devices on the host that are in a bridge group on the host?

VirtualBox does not allow assignment of pci-device to the guest. The network interfaces on the guest are pure virtual one, with several modes available. In order to help you trouble shooting this problem, we need to know the mode form each of the virtual interfaces. Possible modes are NAT, bridged, internal-network, and host-only-network.

Quoted 7 lines

Please provide the output of the following command:

VBoxManage showvminfo <your-vm-uuid> | grep ^NIC

To display your vm uuid, use the following command:

VBoxManage list vms

[root@localhost ~]# VBoxManage showvminfo 67b83c47-0ee2-46bc-b0ff-e0eb43edc1c2 |grep ^NIC
NIC 1: MAC: 0800270481A8, Attachment: Bridged Interface 'eth0', Cable
connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 2: MAC: 08002778F641, Attachment: Bridged Interface 'eth0', Cable
connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 3: MAC: 080027C408BA, Attachment: Bridged Interface 'eth0', Cable
connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 4: MAC: 080027DB339A, Attachment: Bridged Interface 'eth0', Cable
connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 5: disabled
NIC 6: disabled
NIC 7: disabled
NIC 8: disabled

Quoted 5 lines

And when guest starts, i find that:
NIC 1: eth7
NIC 2: eth6
NIC 3: eth9
NIC 4: eth8

Quoted 5 lines

Would you mind testing with "Host-only Interface 'vboxnet0'", instead of "Bridged Interface 'eth0'"?

All the bonding tests I do use this setup and the link failure detection work well.

Nicolas.

ok, I use "Host-only mode" and get

an conclusion, that if the first enslaved nic is pulled out, bonding can't handle well.

First test.
I first enslave eth6, then pull it out, bonding doesn't work. on host,
[root@localhost ~]# VBoxManage -v
4.0.2r69518

[root@localhost ~]# VBoxManage showvminfo 67b83c47-0ee2-46bc-b0ff-e0eb43edc1c2|grep ^NIC

NIC 1:           MAC: 0800270481A8, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 2:           MAC: 08002778F641, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 3:           MAC: 080027C408BA, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 4:           MAC: 080027DB339A, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 5:           disabled
NIC 6:           disabled
NIC 7:           disabled
NIC 8:           disabled

[root@localhost ~]# ifconfig vboxnet0
vboxnet0 Link encap:Ethernet HWaddr 0A:00:27:00:00:00
inet addr:192.168.56.1 Bcast:192.168.56.255 Mask:255.255.255.0 inet6 addr: fe80::800:27ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:268 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:52456 (51.2 KiB)

restart guest,
[root@localhost ~]# uname -a
Linux localhost.localdomain 2.6.35.11-83.fc14.i686 #1 SMP Mon Feb 7 07:04:18 UTC 2011 i686 i686 i386 GNU/Linux

[root@localhost ~]# ifconfig
eth6 Link encap:Ethernet HWaddr 08:00:27:78:F6:41

          inet addr:192.168.56.101  Bcast:192.168.56.255

Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe78:f641/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:6772 (6.6 KiB) TX bytes:1152 (1.1 KiB)

eth7 Link encap:Ethernet HWaddr 08:00:27:04:81:A8

          inet addr:192.168.56.102  Bcast:192.168.56.255

Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe04:81a8/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:6772 (6.6 KiB) TX bytes:1152 (1.1 KiB)

eth8 Link encap:Ethernet HWaddr 08:00:27:DB:33:9A

          inet addr:192.168.56.104  Bcast:192.168.56.255

Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fedb:339a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:63 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:19456 (19.0 KiB) TX bytes:1152 (1.1 KiB)

eth9 Link encap:Ethernet HWaddr 08:00:27:C4:08:BA

          inet addr:192.168.56.103  Bcast:192.168.56.255

Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fec4:8ba/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:63 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:19456 (19.0 KiB) TX bytes:1152 (1.1 KiB)

lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:16 errors:0 dropped:0 overruns:0 frame:0 TX packets:16 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:880 (880.0 b) TX bytes:880 (880.0 b)

so

NIC 1:           eth7
NIC 2:           eth6
NIC 3:           eth9
NIC 4:           eth8

on guest,
[root@localhost ~]# for i in eth{6..9}; do ifconfig $i down; done [root@localhost ~]# modprobe bonding mode=0 miimon=100 [root@localhost ~]# ifconfig bond0 192.168.56.2 netmask 255.255.255.0 up [root@localhost ~]# ifenslave bond0 eth6
[root@localhost ~]# ifenslave bond0 eth7
[root@localhost ~]# ping 192.168.56.1 -c 5
PING 192.168.56.1 (192.168.56.1) 56(84) bytes of data.
64 bytes from 192.168.56.1: icmp_req=1 ttl=64 time=1.83 ms 64 bytes from 192.168.56.1: icmp_req=2 ttl=64 time=0.149 ms 64 bytes from 192.168.56.1: icmp_req=3 ttl=64 time=0.204 ms 64 bytes from 192.168.56.1: icmp_req=4 ttl=64 time=0.294 ms 64 bytes from 192.168.56.1: icmp_req=5 ttl=64 time=0.412 ms

--- 192.168.56.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4003ms rtt min/avg/max/mdev = 0.149/0.578/1.832/0.633 ms [root@localhost ~]# arp

Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.56.1             ether   0a:00:27:00:00:00   C                     bond0

on host,
[root@localhost ~]# arp

Address                  HWtype  HWaddress           Flags Mask            Iface
192.168.56.2             ether   08:00:27:78:f6:41   C                     vboxnet0
corerouter.nay.redhat.c  ether   00:1d:45:20:d5:ff   C                     eth0

[root@localhost ~]# VBoxManage controlvm 67b83c47-0ee2-46bc-b0ff-e0eb43edc1c2 setlinkstate2 off

[root@localhost ~]# ping 192.168.56.1 -c 5
PING 192.168.56.1 (192.168.56.1) 56(84) bytes of data. From 192.168.56.2 icmp_seq=2 Destination Host Unreachable From 192.168.56.2 icmp_seq=3 Destination Host Unreachable From 192.168.56.2 icmp_seq=4 Destination Host Unreachable From 192.168.56.2 icmp_seq=5 Destination Host Unreachable

--- 192.168.56.1 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4001ms pipe 3
[root@localhost ~]# dmesg

[  513.093249] e1000: eth6 NIC Link is Down
[  513.129123] bonding: bond0: link status definitely down for interface eth6, disabling it

[root@localhost ~]# ip route show
192.168.56.0/24 dev bond0 proto kernel scope link src 192.168.56.2 192.168.56.0/24 dev eth7 proto kernel scope link src 192.168.56.101 metric 1 [root@localhost ~]# ip neigh show
192.168.56.1 dev bond0 FAILED

on host,
[root@localhost ~]# ip route show
192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.100 192.168.56.0/24 dev vboxnet0 proto kernel scope link src 192.168.56.1 10.66.64.0/23 dev eth0 proto kernel scope link src 10.66.65.228 metric 1 default via 10.66.65.254 dev eth0 proto static
[root@localhost ~]# ip neigh show
192.168.56.2 dev vboxnet0 lladdr 08:00:27:78:f6:41 STALE 10.66.65.254 dev eth0 lladdr 00:1d:45:20:d5:ff REACHABLE

Second test
I first enslave eth6, then pull eth7 out, bonding works well. on guest,
[root@localhost ~]# modprobe bonding mode=0 miimon=100 [root@localhost ~]# ifconfig bond0 192.168.56.2 netmask 255.255.255.0 up [root@localhost ~]# ifenslave bond0 eth6
[root@localhost ~]# ifenslave bond0 eth7
[root@localhost ~]# ping 192.168.56.1 -c 5
PING 192.168.56.1 (192.168.56.1) 56(84) bytes of data.
64 bytes from 192.168.56.1: icmp_req=1 ttl=64 time=0.902 ms 64 bytes from 192.168.56.1: icmp_req=2 ttl=64 time=0.260 ms 64 bytes from 192.168.56.1: icmp_req=3 ttl=64 time=0.237 ms 64 bytes from 192.168.56.1: icmp_req=4 ttl=64 time=0.335 ms 64 bytes from 192.168.56.1: icmp_req=5 ttl=64 time=0.170 ms

--- 192.168.56.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms rtt min/avg/max/mdev = 0.170/0.380/0.902/0.267 ms

[root@localhost ~]# dmesg

[ 1162.524825] bonding: MII link monitoring set to 100 ms
[ 1165.845586] ADDRCONF(NETDEV_UP): bond0: link is not ready
[ 1174.505912] e1000: eth6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 1174.512372] bonding: bond0: enslaving eth6 as an active interface with an up link.
[ 1174.512897] ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
[ 1178.857322] ICMPv6 NA: someone advertises our address fe80:0000:0000:0000:0a00:27ff:fe78:f641 on bond0!
[ 1178.858649] e1000: eth7 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 1178.861391] bonding: bond0: enslaving eth7 as an active interface with an up link.
[ 1184.682110] bond0: no IPv6 routers present

on host,
[root@localhost ~]# VBoxManage controlvm 67b83c47-0ee2-46bc-b0ff-e0eb43edc1c2 setlinkstate1 off

on guest,
[root@localhost ~]# ping 192.168.56.1 -c 5
PING 192.168.56.1 (192.168.56.1) 56(84) bytes of data. 64 bytes from 192.168.56.1: icmp_req=1 ttl=64 time=0.599 ms 64 bytes from 192.168.56.1: icmp_req=2 ttl=64 time=0.150 ms 64 bytes from 192.168.56.1: icmp_req=3 ttl=64 time=0.224 ms 64 bytes from 192.168.56.1: icmp_req=4 ttl=64 time=0.154 ms 64 bytes from 192.168.56.1: icmp_req=5 ttl=64 time=0.189 ms

--- 192.168.56.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4000ms rtt min/avg/max/mdev = 0.150/0.263/0.599/0.170 ms [root@localhost ~]# dmesg

[ 1281.421231] e1000: eth7 NIC Link is Down
[ 1281.492178] bonding: bond0: link status definitely down for interface eth7, disabling it

many thanks
Weiping Pan

#1 lore raw
Quoted 9 lines · Weiping Pan

ok, I use "Host-only mode" and get

an conclusion, that if the first enslaved nic is pulled out, bonding can't handle well.

First test.
I first enslave eth6, then pull it out, bonding doesn't work. on host,

...

Quoted 4 lines

Second test
I first enslave eth6, then pull eth7 out, bonding works well. on guest,

Can you show me your /proc/net/bonding/bond0 before and after pulling down eth6 or eth7? And what does ip link show say?

author
#2 lore raw
Quoted 9 lines · WANG Cong
Quoted 7 lines · Weiping Pan

ok, I use "Host-only mode" and get

an conclusion, that if the first enslaved nic is pulled out, bonding can't handle well.

Quoted 9 lines
Quoted 2 lines

First test.
I first enslave eth6, then pull it out, bonding doesn't work. on host,

...

Quoted 3 lines

Second test
I first enslave eth6, then pull eth7 out, bonding works well. on guest,

Can you show me your /proc/net/bonding/bond0 before and after pulling down eth6 or eth7? And what does ip link show say?

Ok, let me repeat my test, and gather more information.
on host,
[root@localhost ~]# VBoxManage -v
4.0.2r69518
[root@localhost ~]# VBoxManage showvminfo 67b83c47-0ee2-46bc-b0ff-e0eb43edc1c2|grep ^NIC

NIC 1:           MAC: 0800270481A8, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 2:           MAC: 08002778F641, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 3:           MAC: 080027C408BA, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0
NIC 4:           MAC: 080027DB339A, Attachment: Host-only Interface 'vboxnet0', Cable connected: on, Trace: off (file: none), Type: 82540EM, Reported speed: 0 Mbps, Boot priority: 0

[root@localhost ~]# ifconfig
vboxnet0 Link encap:Ethernet HWaddr 0A:00:27:00:00:00
inet addr:192.168.56.1 Bcast:192.168.56.255 Mask:255.255.255.0 inet6 addr: fe80::800:27ff:fe00:0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:3438 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:0 (0.0 b) TX bytes:296716 (289.7 KiB)

vboxnet0:0 Link encap:Ethernet HWaddr 0A:00:27:00:00:00
inet addr:192.168.1.100 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

vboxnet0:1 Link encap:Ethernet HWaddr 0A:00:27:00:00:00
inet addr:192.168.1.101 Bcast:192.168.1.255 Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
[root@localhost ~]# route
Kernel IP routing table

Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
192.168.1.0     *               255.255.255.0   U     0      0        0 vboxnet0
192.168.56.0    *               255.255.255.0   U     0      0        0 vboxnet0
10.66.64.0      *               255.255.254.0   U     1      0        0 eth0
default         corerouter.nay. 0.0.0.0         UG    0      0        0 eth0

[root@localhost ~]# ip neigh show
10.66.65.254 dev eth0 lladdr 00:1d:45:20:d5:ff REACHABLE

then restart guest,
[root@localhost ~]# ifconfig
eth6 Link encap:Ethernet HWaddr 08:00:27:78:F6:41

          inet addr:192.168.56.101  Bcast:192.168.56.255

Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe78:f641/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:6772 (6.6 KiB) TX bytes:1152 (1.1 KiB)

eth7 Link encap:Ethernet HWaddr 08:00:27:04:81:A8

          inet addr:192.168.56.102  Bcast:192.168.56.255

Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fe04:81a8/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:6772 (6.6 KiB) TX bytes:1152 (1.1 KiB)

eth8 Link encap:Ethernet HWaddr 08:00:27:DB:33:9A

          inet addr:192.168.56.104  Bcast:192.168.56.255

Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fedb:339a/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:6772 (6.6 KiB) TX bytes:1152 (1.1 KiB)

eth9 Link encap:Ethernet HWaddr 08:00:27:C4:08:BA

          inet addr:192.168.56.103  Bcast:192.168.56.255

Mask:255.255.255.0
inet6 addr: fe80::a00:27ff:fec4:8ba/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:14 errors:0 dropped:0 overruns:0 frame:0 TX packets:8 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:6772 (6.6 KiB) TX bytes:1152 (1.1 KiB)

so according to mac address,

NIC 1:           eth7
NIC 2:           eth6
NIC 3:           eth9
NIC 4:           eth8

[root@localhost ~]# ifconfig eth6 down
[root@localhost ~]# ifconfig eth7 down
[root@localhost ~]# ifconfig eth8 down
[root@localhost ~]# ifconfig eth9 down
[root@localhost ~]# modprobe bonding mode=0 miimon=100 [root@localhost ~]# ifconfig bond0 192.168.1.2 netmask 255.255.255.0 up [root@localhost ~]# ifenslave bond0 eth6
[root@localhost ~]# ifenslave bond0 eth7
[root@localhost ~]# dmesg

[ 1436.344751] bonding: Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
[ 1436.344756] bonding: MII link monitoring set to 100 ms
[ 1480.485933] ADDRCONF(NETDEV_UP): bond0: link is not ready
[ 1490.087608] e1000: eth6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 1490.091795] bonding: bond0: enslaving eth6 as an active interface with an up link.
[ 1490.092326] ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
[ 1492.676643] e1000: eth7 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 1492.684839] bonding: bond0: enslaving eth7 as an active interface with an up link.

[root@localhost ~]# cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth6
MII Status: up
Link Failure Count: 0
Permanent HW addr: 08:00:27:78:f6:41

Slave Interface: eth7
MII Status: up
Link Failure Count: 0
Permanent HW addr: 08:00:27:04:81:a8
[root@localhost ~]# ifconfig
bond0 Link encap:Ethernet HWaddr 08:00:27:78:F6:41
inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe78:f641/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:40 errors:0 dropped:0 overruns:0 frame:0 TX packets:25 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:19632 (19.1 KiB) TX bytes:4070 (3.9 KiB)

eth6 Link encap:Ethernet HWaddr 08:00:27:78:F6:41

          inet addr:192.168.56.101  Bcast:192.168.56.255

Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:21 errors:0 dropped:0 overruns:0 frame:0 TX packets:13 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:10158 (9.9 KiB) TX bytes:1802 (1.7 KiB)

eth7 Link encap:Ethernet HWaddr 08:00:27:78:F6:41

          inet addr:192.168.56.101  Bcast:192.168.56.255

Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:19 errors:0 dropped:0 overruns:0 frame:0 TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:9474 (9.2 KiB) TX bytes:2268 (2.2 KiB)

[root@localhost ~]# ping 192.168.1.100 -c 5
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data. 64 bytes from 192.168.1.100: icmp_req=1 ttl=64 time=3.69 ms 64 bytes from 192.168.1.100: icmp_req=2 ttl=64 time=0.242 ms 64 bytes from 192.168.1.100: icmp_req=3 ttl=64 time=0.154 ms 64 bytes from 192.168.1.100: icmp_req=4 ttl=64 time=0.270 ms 64 bytes from 192.168.1.100: icmp_req=5 ttl=64 time=0.278 ms

--- 192.168.1.100 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4004ms rtt min/avg/max/mdev = 0.154/0.928/3.696/1.384 ms

[root@localhost ~]# dmesg -c &>/dev/null

on host,
[root@localhost ~]# VBoxManage controlvm 67b83c47-0ee2-46bc-b0ff-e0eb43edc1c2 setlinkstate2 off

on guest,
[root@localhost ~]# dmesg

[ 1731.945272] e1000: eth6 NIC Link is Down
[ 1732.026207] bonding: bond0: link status definitely down for interface eth6, disabling it

[root@localhost ~]# cat /proc/net/bonding/bond0 Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: load balancing (round-robin)
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth6
MII Status: down
Link Failure Count: 1
Permanent HW addr: 08:00:27:78:f6:41

Slave Interface: eth7
MII Status: up
Link Failure Count: 0
Permanent HW addr: 08:00:27:04:81:a8 [root@localhost ~]# ping 192.168.1.100 -c 5 PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data. From 192.168.1.2 icmp_seq=2 Destination Host Unreachable From 192.168.1.2 icmp_seq=3 Destination Host Unreachable From 192.168.1.2 icmp_seq=4 Destination Host Unreachable From 192.168.1.2 icmp_seq=5 Destination Host Unreachable

--- 192.168.1.100 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4001ms pipe 3
[root@localhost ~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth7: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
    link/ether 08:00:27:78:f6:41 brd ff:ff:ff:ff:ff:ff
3: eth6: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master bond0 state DOWN qlen 1000
    link/ether 08:00:27:78:f6:41 brd ff:ff:ff:ff:ff:ff
4: eth9: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 08:00:27:c4:08:ba brd ff:ff:ff:ff:ff:ff
5: eth8: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
    link/ether 08:00:27:db:33:9a brd ff:ff:ff:ff:ff:ff
6: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP
    link/ether 08:00:27:78:f6:41 brd ff:ff:ff:ff:ff:ff

[root@localhost ~]# ip neigh show
192.168.1.100 dev bond0 FAILED

on host,
[root@localhost ~]# ip neigh show
10.66.65.254 dev eth0 lladdr 00:1d:45:20:d5:ff STALE 192.168.1.2 dev vboxnet0 lladdr 08:00:27:78:f6:41 STALE

on guest, ping while tcpdump
[root@localhost ~]# ping 192.168.1.100
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data.
From 192.168.1.2 icmp_seq=11 Destination Host Unreachable
From 192.168.1.2 icmp_seq=12 Destination Host Unreachable
From 192.168.1.2 icmp_seq=13 Destination Host Unreachable
From 192.168.1.2 icmp_seq=15 Destination Host Unreachable
From 192.168.1.2 icmp_seq=16 Destination Host Unreachable
From 192.168.1.2 icmp_seq=17 Destination Host Unreachable
From 192.168.1.2 icmp_seq=19 Destination Host Unreachable
From 192.168.1.2 icmp_seq=20 Destination Host Unreachable
From 192.168.1.2 icmp_seq=21 Destination Host Unreachable
From 192.168.1.2 icmp_seq=23 Destination Host Unreachable
From 192.168.1.2 icmp_seq=24 Destination Host Unreachable
From 192.168.1.2 icmp_seq=25 Destination Host Unreachable
From 192.168.1.2 icmp_seq=27 Destination Host Unreachable
From 192.168.1.2 icmp_seq=28 Destination Host Unreachable
From 192.168.1.2 icmp_seq=29 Destination Host Unreachable
^C
--- 192.168.1.100 ping statistics ---
30 packets transmitted, 0 received, +15 errors, 100% packet loss, time 29011ms pipe 3
[root@localhost ~]# tcpdump -i bond0 -p arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on bond0, link-type EN10MB (Ethernet), capture size 65535 bytes 21:22:40.694058 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:41.695875 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:42.698067 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:44.689068 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:45.689837 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:46.692076 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:48.691080 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:49.693828 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:50.696074 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:52.693070 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:53.693837 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:54.696075 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:56.694072 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:57.696063 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:22:58.698096 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:23:00.696065 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:23:01.698072 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:23:02.700063 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:23:04.698071 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:23:05.700504 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 21:23:06.702009 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 ^C
21 packets captured
21 packets received by filter
0 packets dropped by kernel

meanwhile on host,
[root@localhost ~]# tcpdump -i vboxnet0 -p arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vboxnet0, link-type EN10MB (Ethernet), capture size 65535 bytes 10:22:56.936833 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:22:56.936839 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:22:57.938415 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:22:57.938422 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:22:58.939875 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:22:58.939881 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:23:00.937292 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:23:00.937299 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:23:01.939242 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:23:01.939249 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:23:02.940559 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:23:02.940565 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:23:04.938148 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:23:04.938156 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:23:05.939688 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:23:05.939695 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:23:06.941288 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 10:23:06.941295 ARP, Request who-has 192.168.1.100 tell 192.168.1.2, length 28 ^C
18 packets captured
18 packets received by filter
0 packets dropped by kernel

Maybe this is the cause of the problem.
The guest can't receive correct ARP reply, maybe the virtual network of VirtualBox doesn't transfer it.

on host,
[root@localhost ~]# VBoxManage controlvm 67b83c47-0ee2-46bc-b0ff-e0eb43edc1c2 setlinkstate2 on

on guest,
[root@localhost ~]# dmesg

[ 2392.304591] e1000: eth6 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
[ 2392.381223] bonding: bond0: link status definitely up for interface eth6.

[root@localhost ~]# ping 192.168.1.100 -c5
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data. 64 bytes from 192.168.1.100: icmp_req=1 ttl=64 time=0.648 ms 64 bytes from 192.168.1.100: icmp_req=2 ttl=64 time=0.208 ms 64 bytes from 192.168.1.100: icmp_req=3 ttl=64 time=0.187 ms 64 bytes from 192.168.1.100: icmp_req=4 ttl=64 time=0.274 ms 64 bytes from 192.168.1.100: icmp_req=5 ttl=64 time=0.216 ms

--- 192.168.1.100 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4002ms rtt min/avg/max/mdev = 0.187/0.306/0.648/0.174 ms

[root@localhost ~]# ip neigh show
192.168.1.100 dev bond0 lladdr 0a:00:27:00:00:00 REACHABLE

thanks
Weiping Pan

#3 lore raw

On Wed, Mar 09, 2011 at 10:40:36AM +0800, Weiping Pan wrote:

<...>

Quoted 19 lines

[root@localhost ~]# ifconfig
bond0 Link encap:Ethernet HWaddr 08:00:27:78:F6:41
inet addr:192.168.1.2 Bcast:192.168.1.255 Mask:255.255.255.0 inet6 addr: fe80::a00:27ff:fe78:f641/64 Scope:Link UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1 RX packets:40 errors:0 dropped:0 overruns:0 frame:0 TX packets:25 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:19632 (19.1 KiB) TX bytes:4070 (3.9 KiB)

eth6 Link encap:Ethernet HWaddr 08:00:27:78:F6:41

         inet addr:192.168.56.101  Bcast:192.168.56.255

Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:21 errors:0 dropped:0 overruns:0 frame:0 TX packets:13 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:10158 (9.9 KiB) TX bytes:1802 (1.7 KiB)

Quoted 8 lines

eth7 Link encap:Ethernet HWaddr 08:00:27:78:F6:41

         inet addr:192.168.56.101  Bcast:192.168.56.255

Mask:255.255.255.0
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1 RX packets:19 errors:0 dropped:0 overruns:0 frame:0 TX packets:12 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000
RX bytes:9474 (9.2 KiB) TX bytes:2268 (2.2 KiB)

It's weird that you bond two nic's into another different subnet.

Quoted 9 lines

[root@localhost ~]# ping 192.168.1.100 -c 5
PING 192.168.1.100 (192.168.1.100) 56(84) bytes of data. 64 bytes from 192.168.1.100: icmp_req=1 ttl=64 time=3.69 ms 64 bytes from 192.168.1.100: icmp_req=2 ttl=64 time=0.242 ms 64 bytes from 192.168.1.100: icmp_req=3 ttl=64 time=0.154 ms 64 bytes from 192.168.1.100: icmp_req=4 ttl=64 time=0.270 ms 64 bytes from 192.168.1.100: icmp_req=5 ttl=64 time=0.278 ms

What is your route table in your guest?

Quoted 8 lines

[root@localhost ~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth7: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master bond0 state UP qlen 1000
link/ether 08:00:27:78:f6:41 brd ff:ff:ff:ff:ff:ff
3: eth6: <NO-CARRIER,BROADCAST,MULTICAST,SLAVE,UP> mtu 1500 qdisc pfifo_fast master bond0 state DOWN qlen 1000

Clearly this equals to pulling off the cable from eth6, which is your first slave.

author
#4 lore raw
In reply to WANG Cong
Quoted 15 lines · WANG Cong
Quoted 9 lines · Weiping Pan

ok, I use "Host-only mode" and get

an conclusion, that if the first enslaved nic is pulled out, bonding can't handle well.

First test.
I first enslave eth6, then pull it out, bonding doesn't work. on host,

...

Quoted 3 lines

Second test
I first enslave eth6, then pull eth7 out, bonding works well. on guest,

Quoted 7 lines

Can you show me your /proc/net/bonding/bond0 before and after pulling down eth6 or eth7? And what does ip link show say?

--
To unsubscribe from this list: send the line "unsubscribe netdev" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

And I test it again, there is no problem with kvm. Maybe there is bug in virtualbox.

thanks
Weiping Pan