macvlan/macvtap: guest/host cannot communicate when network cable is unplugged

From 192.168.1.2: icmp_seq=10 Destination Host Unreachable From 192.168.1.2: icmp_seq=11 Destination Host Unreachable

Hi all,

I try to setup guest network with macvlan, as guest cannot communicate with host directly, i try to workaround by configuring host to using a macvlan interface

host: Gentoo x64, kernel 3.5.2, qemu-kvm 1.1.1-r1, libvirt 0.9.13
guest: Ubuntu 12.04, kernel 3.4.9, virtio-net

The host is using macvlan (interface name: znet0, ip: 192.168.1.2, bridge mode)

The guest is using macvtap managed by libvirt (interface name: macvtap0, ip: 192.168.1.184, bridge mode, with vhost)

I am facing a strange problem that the guest/host can communicate only when the network cable of the ethernet adapter is connected.

The network config of the host

    $ifconfig

    eth0      Link encap:Ethernet  HWaddr f4:6d:xx:xx:xx:xx
              inet6 addr: fe80::xx:xx:xx:xx/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:86507 errors:0 dropped:0 overruns:0 frame:0
              TX packets:55940 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:126005746 (120.1 MiB)  TX bytes:4394225 (4.1 MiB)

    macvtap0  Link encap:Ethernet  HWaddr 52:54:xx:xx:xx:xx
              inet6 addr: fe80::xx:xx:xx:xx/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:70 errors:0 dropped:0 overruns:0 frame:0
              TX packets:84 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:500
              RX bytes:9036 (8.8 KiB)  TX bytes:14734 (14.3 KiB)

    znet0     Link encap:Ethernet  HWaddr 00:60:xx:xx:xx:xx
              inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
              inet6 addr: 2002:xx:xx:xx:xx/64 Scope:Global
              inet6 addr: fe80:xx:xx:xx:xx/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:4463190 errors:0 dropped:0 overruns:0 frame:0
              TX packets:12527522 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:3959213697 (3.6 GiB)  TX bytes:18590336476 (17.3 GiB)

$ip -d link show

10: znet0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
link/ether 00:60:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
macvlan mode bridge
17: macvtap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT qlen 500
link/ether 52:54:xx:xx:xx:xx brd ff:ff:ff:ff:ff:ff
macvtap mode bridge

libvirt config of the guest network

    <interface type='direct'>
      <mac address='52:54:xx:xx:xx:xx'/>
      <source dev='eth0' mode='bridge'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </interface>

The network config of the guest

eth0 Link encap:Ethernet HWaddr 52:54:xx:xx:xx:xx

          inet addr:192.168.1.184  Bcast:192.168.1.255  Mask:255.255.255.0
          inet6 addr: 2002:xx:xx:xx:xxx/64 Scope:Global
          inet6 addr: 2002:xx:xx:xx:xx/64 Scope:Global
          inet6 addr: fe80:xx:xx:xx:xx/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:25 errors:0 dropped:0 overruns:0 frame:0
          TX packets:66 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:3108 (3.1 KB)  TX bytes:13066 (13.0 KB)

when network cable of host is plugged in

$ ping 192.168.1.184
PING 192.168.1.184 (192.168.1.184) 56(84) bytes of data.
64 bytes from 192.168.1.184: icmp_req=1 ttl=64 time=0.314 ms
64 bytes from 192.168.1.184: icmp_req=2 ttl=64 time=0.253 ms
64 bytes from 192.168.1.184: icmp_req=3 ttl=64 time=0.230 ms

Afterward, i unplugged the network cable to ensure that packet is not routed via external router, but the test seems failed.

$ ping 192.168.1.184
PING 192.168.1.184 (192.168.1.184) 56(84) bytes of data.
From 192.168.1.2: icmp_seq=10 Destination Host Unreachable
From 192.168.1.2: icmp_seq=11 Destination Host Unreachable

Do anyone have similar problem? Am i missing something?

Can you try the same test with two macvlan interfaces on the host (no macvtap)? You may need to use the ping -I <interface-address> argument to force the ping source address to a specific macvlan interface.

If you see the same problem, it may just be the macvlan design - it is stacked on top of eth0 and might not work when eth0 is down. CCing macvlan/macvtap folks.

Stefan

Quoted 6 lines

Can you try the same test with two macvlan interfaces on the host (no macvtap)? You may need to use the ping -I <interface-address> argument to force the ping source address to a specific macvlan interface.

Quoted 6 lines

If you see the same problem, it may just be the macvlan design - it is stacked on top of eth0 and might not work when eth0 is down. CCing macvlan/macvtap folks.

Stefan

tested as below

$ifconfig

    eth0      Link encap:Ethernet  HWaddr f4:6d:xx:xx:xx:xx
              inet6 addr: fe80::xx:xx:xx:xx/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:86507 errors:0 dropped:0 overruns:0 frame:0
              TX packets:55940 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:1000
              RX bytes:126005746 (120.1 MiB)  TX bytes:4394225 (4.1 MiB)

    macvtap0  Link encap:Ethernet  HWaddr 52:54:xx:xx:xx:xx
              inet6 addr: fe80::xx:xx:xx:xx/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:70 errors:0 dropped:0 overruns:0 frame:0
              TX packets:84 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:500
              RX bytes:9036 (8.8 KiB)  TX bytes:14734 (14.3 KiB)

    znet0     Link encap:Ethernet  HWaddr 00:60:xx:xx:xx:92
              inet addr:192.168.1.2  Bcast:192.168.1.255  Mask:255.255.255.0
              inet6 addr: 2002:xx:xx:xx:xx/64 Scope:Global
              inet6 addr: fe80:xx:xx:xx:xx/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:4463190 errors:0 dropped:0 overruns:0 frame:0
              TX packets:12527522 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:3959213697 (3.6 GiB)  TX bytes:18590336476 (17.3 GiB)

znet1 Link encap:Ethernet HWaddr 00:60:xx:xx:xx:99

              inet addr:192.168.1.177  Bcast:192.168.1.255  Mask:255.255.255.0
              inet6 addr: 2002:xx:xx:xx:xx64 Scope:Global
              inet6 addr: fe80:xx:xx:xx:xx/64 Scope:Link
              UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
              RX packets:8 errors:0 dropped:0 overruns:0 frame:0
              TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
              collisions:0 txqueuelen:0
              RX bytes:1399 (1.3 KiB)  TX bytes:1522 (1.4 KiB)

$ ip -d link show

10: znet0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT
link/ether 00:60:xx:xx:xx:92 brd ff:ff:ff:ff:ff:ff
macvlan mode bridge
15: znet1@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT
link/ether 00:60:xx:xx:xx:99 brd ff:ff:ff:ff:ff:ff
macvlan mode bridge

the macvlan interface cannot ping each other no matter network cable is plugged or not

$ ping -I 192.168.1.2 192.168.1.177
PING 192.168.1.177 (192.168.1.177) from 192.168.1.2 : 56(84) bytes of data.

--- 192.168.1.177 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 4999ms

I also perform an additional test: the guests (macvtap bridge mode) CAN communicate each other no matter network cable is plugged or not.

Quoted 2 lines

I also perform an additional test: the guests (macvtap bridge mode) CAN communicate each other no matter network cable is plugged or not.

Strange. I thought the original problem was that the macvtap guests cannot communicate with each other when the network cable is unplugged?

Hopefully someone else can help you, I'm not familiar enough with macvlan/macvtap.

Stefan

Quoted 11 lines

Quoted 2 lines

I also perform an additional test: the guests (macvtap bridge mode) CAN communicate each other no matter network cable is plugged or not.

Strange. I thought the original problem was that the macvtap guests cannot communicate with each other when the network cable is unplugged?

Hopefully someone else can help you, I'm not familiar enough with macvlan/macvtap.

Stefan

My word may be a little bit confusing, it should be

I also perform an additional test: I created two qemu guests (macvtap bridge mode on eth0), they CAN communicate each other no matter network cable of the host is unplugged or not.

and my problem is specified in the subject already: guest/host cannot communicate when network cable is unplugged

if my configuration seems correct, i may file a kernel bugzilla and seek for their advice.