macvlan on top mlx4 fails
Subsystems (2)
Hi all,
I am trying to have a macvlan on top of an ethernet driver infiniband emulation communicating with the macvlan on anoher host with the same configuration. But I am not able to ping them through the ip address assigned to each macvlan.
On the host1 (s1):
s1:~> ifconfig
eth1 Link encap:Ethernet HWaddr 00:1A:64:44:B5:FB
inet addr:9.114.244.95 Bcast:9.114.247.255 Mask:255.255.248.0
inet6 addr: fe80::21a:64ff:fe44:b5fb/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:5355123 errors:0 dropped:0 overruns:0 frame:0
TX packets:2193023 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:791518554 (754.8 Mb) TX bytes:18432872954 (17578.9 Mb)ib0 Link encap:UNSPEC HWaddr 80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.0.95 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::202:c903:1:1b1d/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 RX packets:306 errors:0 dropped:0 overruns:0 frame:0 TX packets:272 errors:0 dropped:5 overruns:0 carrier:0 collisions:0 txqueuelen:256
RX bytes:27870 (27.2 Kb) TX bytes:27150 (26.5 Kb)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:170697 errors:0 dropped:0 overruns:0 frame:0
TX packets:170697 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:61678890 (58.8 Mb) TX bytes:61678890 (58.8 Mb)mc1 Link encap:Ethernet HWaddr 86:D7:44:D4:DC:C0
inet addr:1.2.3.5 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::84d7:44ff:fed4:dcc0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:11903 errors:0 dropped:0 overruns:0 frame:0
TX packets:6 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:851808 (831.8 Kb) TX bytes:468 (468.0 b)s2:~ # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN qlen 1000
link/ether 00:1a:64:44:b5:22 brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:1a:64:44:b5:23 brd ff:ff:ff:ff:ff:ff4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256
link/infiniband
80:00:00:48:fe:80:00:00:00:00:00:41:00:02:c9:03:00:01:1e:bd brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 5: ib1: <BROADCAST,MULTICAST> mtu 2044 qdisc noop state DOWN qlen 256
link/infiniband
80:00:00:49:fe:80:00:00:00:00:00:00:00:02:c9:03:00:01:1e:be brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 49: mc1@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether be:7e:6e:dd:58:c0 brd ff:ff:ff:ff:ff:ff
On the host2 (s2):
s2:~ # ifconfig
eth1 Link encap:Ethernet HWaddr 00:1A:64:44:B5:23
inet addr:9.114.244.96 Bcast:9.114.247.255 Mask:255.255.248.0
inet6 addr: fe80::21a:64ff:fe44:b523/64 Scope:Link
UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500 Metric:1
RX packets:967848 errors:0 dropped:0 overruns:0 frame:0
TX packets:100977 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:1535677477 (1464.5 Mb) TX bytes:61077758 (58.2 Mb)ib0 Link encap:UNSPEC HWaddr 80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00
inet addr:192.168.0.96 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::202:c903:1:1ebd/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:2044 Metric:1 RX packets:58 errors:0 dropped:0 overruns:0 frame:0 TX packets:34 errors:0 dropped:5 overruns:0 carrier:0 collisions:0 txqueuelen:256
RX bytes:7305 (7.1 Kb) TX bytes:4145 (4.0 Kb)
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:580 errors:0 dropped:0 overruns:0 frame:0 TX packets:580 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0
RX bytes:43888 (42.8 Kb) TX bytes:43888 (42.8 Kb)
mc1 Link encap:Ethernet HWaddr BE:7E:6E:DD:58:C0
inet addr:1.2.3.4 Bcast:0.0.0.0 Mask:255.255.255.0
inet6 addr: fe80::bc7e:6eff:fedd:58c0/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:12411 errors:0 dropped:0 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:886732 (865.9 Kb) TX bytes:594 (594.0 b)s1:~ # ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast state DOWN qlen 1000
link/ether 00:1a:64:44:b5:fa brd ff:ff:ff:ff:ff:ff
3: eth1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000
link/ether 00:1a:64:44:b5:fb brd ff:ff:ff:ff:ff:ff4: ib0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 2044 qdisc pfifo_fast state UP qlen 256
link/infiniband
80:00:00:48:fe:80:00:00:00:00:00:41:00:02:c9:03:00:01:1b:1d brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 5: ib1: <BROADCAST,MULTICAST> mtu 2044 qdisc noop state DOWN qlen 256
link/infiniband
80:00:00:49:fe:80:00:00:00:00:00:00:00:02:c9:03:00:01:1b:1e brd 00:ff:ff:ff:ff:12:40:1b:ff:ff:00:00:00:00:00:00:ff:ff:ff:ff 85: mc1@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
link/ether 86:d7:44:d4:dc:c0 brd ff:ff:ff:ff:ff:ff
The ping result:
s2:~ # ping 1.2.3.5
PING 1.2.3.5 (1.2.3.5) 56(84) bytes of data.
From 1.2.3.4: icmp_seq=2 Destination Host Unreachable
From 1.2.3.4 icmp_seq=2 Destination Host Unreachable
From 1.2.3.4 icmp_seq=3 Destination Host Unreachable
From 1.2.3.4 icmp_seq=4 Destination Host Unreachable
^C
--- 1.2.3.5 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4010ms , pipe 3
The arp cache:
Address HWtype HWaddress Flags Mask
Iface
1.2.3.5 (incomplete)
mc1
When doing a tcpdump on s2 host, I have arp who-as request:
09:14:20.764389 arp who-has 1.2.3.5 tell 1.2.3.4
09:14:20.764394 arp who-has 1.2.3.5 tell 1.2.3.4
09:14:20.764427 arp who-has 1.2.3.5 tell 1.2.3.4
But doing the same tcpdump on the s1 host I don't see there arp request.
The output of lscpi:
s2:~ # lspci
0000:00:01.0 RAID bus controller: IBM Obsidian chipset SCSI controller (rev 02) 0001:00:01.0 USB Controller: NEC Corporation USB (rev 43)
0001:00:01.1 USB Controller: NEC Corporation USB (rev 43)
0001:00:01.2 USB Controller: NEC Corporation USB 2.0 (rev 04)
0002:00:01.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev 02) 0003:01:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0)
I am a newbie on infiniband so I was wondering if I did something wrong or if this is unsupported.
Thanks in advance.
-- Daniel
ps : I was not able to find the email of the mlx4 driver maintainer.
Quoted 51 lines · Daniel Lezcano
Hi all,
I am trying to have a macvlan on top of an ethernet driver infiniband emulation communicating with the macvlan on anoher host with the same configuration. But I am not able to ping them through the ip address assigned to each macvlan.
On the host1 (s1):
...
s2:~ # ping 1.2.3.5
PING 1.2.3.5 (1.2.3.5) 56(84) bytes of data.
From 1.2.3.4: icmp_seq=2 Destination Host Unreachable
From 1.2.3.4 icmp_seq=2 Destination Host Unreachable
From 1.2.3.4 icmp_seq=3 Destination Host Unreachable
From 1.2.3.4 icmp_seq=4 Destination Host Unreachable
^C
--- 1.2.3.5 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4010ms , pipe 3The arp cache:
Address HWtype HWaddress Flags Mask Iface
1.2.3.5 (incomplete) mc1When doing a tcpdump on s2 host, I have arp who-as request:
09:14:20.764389 arp who-has 1.2.3.5 tell 1.2.3.4
09:14:20.764394 arp who-has 1.2.3.5 tell 1.2.3.4
09:14:20.764427 arp who-has 1.2.3.5 tell 1.2.3.4But doing the same tcpdump on the s1 host I don't see there arp request.
The output of lscpi:
s2:~ # lspci
0000:00:01.0 RAID bus controller: IBM Obsidian chipset SCSI controller (rev 02)
0001:00:01.0 USB Controller: NEC Corporation USB (rev 43) 0001:00:01.1 USB Controller: NEC Corporation USB (rev 43) 0001:00:01.2 USB Controller: NEC Corporation USB 2.0 (rev 04) 0002:00:01.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev
02)
0003:01:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0)
Quoted 2 lines
I am a newbie on infiniband so I was wondering if I did something wrong or if this is unsupported.
Well, I don't know much about infiniband myself. On which device are you running tcpdump? In case its the eth-device, does running tcpdump directly on top of the ib-devices make any difference? Perhaps its not properly propagating the promiscous mode flag or the secondary unicast addresses.
Quoted 52 lines · Patrick McHardy
Quoted 51 lines · Daniel Lezcano
Hi all,
I am trying to have a macvlan on top of an ethernet driver infiniband emulation communicating with the macvlan on anoher host with the same configuration. But I am not able to ping them through the ip address assigned to each macvlan.
On the host1 (s1):
...
s2:~ # ping 1.2.3.5
PING 1.2.3.5 (1.2.3.5) 56(84) bytes of data.
From 1.2.3.4: icmp_seq=2 Destination Host Unreachable
From 1.2.3.4 icmp_seq=2 Destination Host Unreachable
From 1.2.3.4 icmp_seq=3 Destination Host Unreachable
From 1.2.3.4 icmp_seq=4 Destination Host Unreachable
^C
--- 1.2.3.5 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4010ms , pipe 3The arp cache:
Address HWtype HWaddress Flags Mask Iface
1.2.3.5 (incomplete) mc1When doing a tcpdump on s2 host, I have arp who-as request:
09:14:20.764389 arp who-has 1.2.3.5 tell 1.2.3.4
09:14:20.764394 arp who-has 1.2.3.5 tell 1.2.3.4
09:14:20.764427 arp who-has 1.2.3.5 tell 1.2.3.4But doing the same tcpdump on the s1 host I don't see there arp request.
The output of lscpi:
s2:~ # lspci
0000:00:01.0 RAID bus controller: IBM Obsidian chipset SCSI controller (rev 02)
0001:00:01.0 USB Controller: NEC Corporation USB (rev 43) 0001:00:01.1 USB Controller: NEC Corporation USB (rev 43) 0001:00:01.2 USB Controller: NEC Corporation USB 2.0 (rev 04) 0002:00:01.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev
02)
0003:01:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0)
Quoted 5 lines
Quoted 2 lines
I am a newbie on infiniband so I was wondering if I did something wrong or if this is unsupported.
Well, I don't know much about infiniband myself. On which device are you running tcpdump?
I did:
tcpdump -i any arp dst or src 1.2.3.5
In case its the eth-device, does running
Quoted 1 line
tcpdump directly on top of the ib-devices make any difference?
Yes, right. The arp request are seen on eth1 but not on the ib.
Quoted 2 lines
Perhaps its not properly propagating the promiscous mode flag or the secondary unicast addresses.
Is it possible to check that ?
Thanks.
Quoted 60 lines · Daniel Lezcano
Quoted 53 lines · Patrick McHardy
Quoted 52 lines · Daniel Lezcano
Hi all,
I am trying to have a macvlan on top of an ethernet driver infiniband emulation communicating with the macvlan on anoher host with the same configuration. But I am not able to ping them through the ip address assigned to each macvlan.
On the host1 (s1):
...
s2:~ # ping 1.2.3.5
PING 1.2.3.5 (1.2.3.5) 56(84) bytes of data.
From 1.2.3.4: icmp_seq=2 Destination Host Unreachable From 1.2.3.4 icmp_seq=2 Destination Host Unreachable
From 1.2.3.4 icmp_seq=3 Destination Host Unreachable
From 1.2.3.4 icmp_seq=4 Destination Host Unreachable
^C
--- 1.2.3.5 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4010ms
, pipe 3The arp cache:
Address HWtype HWaddress Flags Mask Iface
1.2.3.5 (incomplete) mc1When doing a tcpdump on s2 host, I have arp who-as request:
09:14:20.764389 arp who-has 1.2.3.5 tell 1.2.3.4
09:14:20.764394 arp who-has 1.2.3.5 tell 1.2.3.4
09:14:20.764427 arp who-has 1.2.3.5 tell 1.2.3.4But doing the same tcpdump on the s1 host I don't see there arp request.
The output of lscpi:
s2:~ # lspci
0000:00:01.0 RAID bus controller: IBM Obsidian chipset SCSI controller (rev 02)
0001:00:01.0 USB Controller: NEC Corporation USB (rev 43) 0001:00:01.1 USB Controller: NEC Corporation USB (rev 43) 0001:00:01.2 USB Controller: NEC Corporation USB 2.0 (rev 04) 0002:00:01.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev
02)
0003:01:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0)Quoted 5 lines
Quoted 2 lines
I am a newbie on infiniband so I was wondering if I did something wrong or if this is unsupported.
Well, I don't know much about infiniband myself. On which device are you running tcpdump?
Quoted 8 lines
I did:
tcpdump -i any arp dst or src 1.2.3.5
In case its the eth-device, does running
Quoted 1 line
tcpdump directly on top of the ib-devices make any difference?
Yes, right. The arp request are seen on eth1 but not on the ib.
That would be fine unless I'm misunderstanding your setup - with macvlan bound to eth1 they should be visible on both eth1 any the macvlan device.
I'm guessing that you have a filter misconfiguration. You need arp_ignore=1 and possibly rp_filter=0.
Quoted 4 lines
Quoted 2 lines
Perhaps its not properly propagating the promiscous mode flag or the secondary unicast addresses.
Is it possible to check that ?
If the devices are not already in promiscous mode, you should see "device XXX entered promiscuous mode" for both devices in the ring buffer.
Quoted 74 lines · Patrick McHardy
Quoted 58 lines · Daniel Lezcano
Quoted 53 lines · Patrick McHardy
Quoted 52 lines · Daniel Lezcano
Hi all,
I am trying to have a macvlan on top of an ethernet driver infiniband emulation communicating with the macvlan on anoher host with the same configuration. But I am not able to ping them through the ip address assigned to each macvlan.
On the host1 (s1):
...
s2:~ # ping 1.2.3.5
PING 1.2.3.5 (1.2.3.5) 56(84) bytes of data.
From 1.2.3.4: icmp_seq=2 Destination Host Unreachable From 1.2.3.4 icmp_seq=2 Destination Host Unreachable
From 1.2.3.4 icmp_seq=3 Destination Host Unreachable
From 1.2.3.4 icmp_seq=4 Destination Host Unreachable
^C
--- 1.2.3.5 ping statistics ---
5 packets transmitted, 0 received, +4 errors, 100% packet loss, time 4010ms
, pipe 3The arp cache:
Address HWtype HWaddress Flags Mask Iface
1.2.3.5 (incomplete) mc1When doing a tcpdump on s2 host, I have arp who-as request:
09:14:20.764389 arp who-has 1.2.3.5 tell 1.2.3.4
09:14:20.764394 arp who-has 1.2.3.5 tell 1.2.3.4
09:14:20.764427 arp who-has 1.2.3.5 tell 1.2.3.4But doing the same tcpdump on the s1 host I don't see there arp request.
The output of lscpi:
s2:~ # lspci
0000:00:01.0 RAID bus controller: IBM Obsidian chipset SCSI controller (rev 02)
0001:00:01.0 USB Controller: NEC Corporation USB (rev 43) 0001:00:01.1 USB Controller: NEC Corporation USB (rev 43) 0001:00:01.2 USB Controller: NEC Corporation USB 2.0 (rev 04) 0002:00:01.0 VGA compatible controller: ATI Technologies Inc ES1000 (rev
02)
0003:01:00.0 InfiniBand: Mellanox Technologies MT25418 [ConnectX IB DDR, PCIe 2.0 2.5GT/s] (rev a0)Quoted 4 lines
Quoted 2 lines
I am a newbie on infiniband so I was wondering if I did something wrong or if this is unsupported.
Well, I don't know much about infiniband myself. On which device are you running tcpdump?
Quoted 7 lines
I did:
tcpdump -i any arp dst or src 1.2.3.5
In case its the eth-device, does running
Quoted 1 line
tcpdump directly on top of the ib-devices make any difference?
Yes, right. The arp request are seen on eth1 but not on the ib.
That would be fine unless I'm misunderstanding your setup - with macvlan bound to eth1 they should be visible on both eth1 any the macvlan device.
I'm guessing that you have a filter misconfiguration. You need arp_ignore=1 and possibly rp_filter=0.
Quoted 7 lines
Quoted 3 lines
Quoted 2 lines
Perhaps its not properly propagating the promiscous mode flag or the secondary unicast addresses.
Is it possible to check that ?
If the devices are not already in promiscous mode, you should see "device XXX entered promiscuous mode" for both devices in the ring buffer.
Hi patrick,
finally I was puzzled with the nic on the machines, they are not inifinband emulated ethernet but powerpc lpar virtual ethernet card.
It seems they drop the packets when the mac address is not the one assigned to the virtual ethernet :(
I am afraid macvlan neither veth or any virtual interface with its own mac address will be able to communicate between lpars. In other words, the lpar seems to be incompatible with the container technology today.
Thanks for your help.
-- Daniel